Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005003 IMPLEMENTATION COMPLETION AND RESULTS REPORT LOAN NUMBER 8188-ID ON A LOAN FROM THE INTERNATIONAL BANK FOR RECONSTRUCTION AND DEVELOPMENT IN THE AMOUNT OF US$80 MILLION TO THE REPUBLIC OF INDONESIA FOR THE SUSTAINABLE MANAGEMENT OF AGRICULTURAL RESEARCH AND TECHNOLOGY DISSEMINATION (SMARTD) PROJECT DECEMBER 23, 2019 Agriculture and Food Global Practice East Asia and Pacific Region This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. CURRENCY EQUIVALENTS (Exchange Rate Effective June 30, 2019) Currency Unit = Indonesian Rupiah (IDR) IDR 14,128 = US$1 FISCAL YEAR July 1 –June 30 ABBREVIATIONS AND ACRONYMS ACIAR Australian Center for International Agricultural Research AIAT Assessment Institute for Agricultural Technology CPF Country Partnership Framework CPS Country Partnership Strategy ECOP Environmental Code of Practices EOP End of Project ERR Economic Rate of Return FAO Food and Agriculture Organization of the United Nations GoI Government of Indonesia HR Human Resource IAARD Indonesian Agency for Agricultural Research and Development ICATAD Indonesian Centre for Agricultural Technology Assessment and Development ICR Implementation Completion and Results Report IPR Intellectual Property Rights ISM Implementation Support Mission KM Knowledge Management M&E Monitoring and Evaluation MIS Management Information System MoA Ministry of Agriculture MoF Ministry of Finance MP3MI Multi-channel Pengembangan Pertanian Perdesaan Model Berbasis Inovasi (Multi-channel Research, Development and Innovation Model) MTR Midterm Review NPV Net Present Value O&M Operation and Maintenance OHS Occupational Health and Safety PAD Project Appraisal Document PDO Project Development Objective PEA Political Economy Analysis PMU Project Management Unit POM Project Operations Manual R&D Research and Development RPJMN Rencana Pembangunan Jangka Menengah Nasional (National Medium-term Development Plan) SMARTD Sustainable Management of Agricultural Research and Technology Dissemination STI Science, Technology, and Innovation TFP Total Factor Productivity VAT Value Added Tax Regional Vice President: Victoria Kwakwa Country Director: Rolande Simone Pryce Regional Director: Benoit Bosquet Practice Manager: Dina Umali-Deininger Task Team Leader(s): Jan Joost Nijhoff ICR Main Contributor: Ir Mariam Rikhana TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5 A. CONTEXT AT APPRAISAL .........................................................................................................5 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION ................................................................9 II. OUTCOME .................................................................................................................... 10 A. RELEVANCE OF PDOs ............................................................................................................ 10 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 13 C. EFFICIENCY ........................................................................................................................... 22 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 25 E. OTHER OUTCOMES AND IMPACTS ......................................................................................... 26 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 28 A. KEY FACTORS DURING PREPARATION ................................................................................... 28 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 29 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 30 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 30 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 33 C. BANK PERFORMANCE ........................................................................................................... 37 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 38 V. LESSONS AND RECOMMENDATIONS ............................................................................. 40 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 43 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 53 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 55 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 56 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 65 ANNEX 6. SUPPORTING DOCUMENTS .................................................................................. 67 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name Sustainable Management of Agricultural Research and P117243 Technology Dissemination (SMARTD) Country Financing Instrument Indonesia Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency Indonesian Agency for Agricultural Research and IAARD - SMARTD Development (IAARD) Project Development Objective (PDO) Original PDO The objective of the Project is to improve the institutional capacity and performance of the Indonesian Agency for Agricultural Research and Development (IAARD) to develop and disseminate relevant and demand-driven innovative technologies, meeting the needs of producers and of the agri-food system. Page 1 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 80,000,000 77,093,168 77,093,168 IBRD-81880 Total 80,000,000 77,093,168 77,093,168 Non-World Bank Financing 0 0 0 Borrower/Recipient 20,000,000 20,000,000 18,200,729 AUSTRALIA, Government of 6,000,000 0 0 Total 26,000,000 20,000,000 18,200,729 Total Project Cost 106,000,000 97,093,168 95,293,897 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 02-Aug-2012 25-Oct-2012 28-Mar-2015 30-Sep-2017 30-Jun-2019 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 09-Jun-2016 30.94 Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Reallocation between Disbursement Categories Change in Implementation Schedule KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Moderately Satisfactory Modest Page 2 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 24-Nov-2012 Satisfactory Satisfactory 0 02 27-Jun-2013 Moderately Satisfactory Moderately Satisfactory 4.12 03 03-Jan-2014 Moderately Satisfactory Moderately Satisfactory 5.51 04 30-Aug-2014 Moderately Satisfactory Moderately Unsatisfactory 11.72 05 22-Dec-2014 Moderately Satisfactory Moderately Satisfactory 14.64 Moderately 06 25-Jun-2015 Moderately Unsatisfactory 22.92 Unsatisfactory Moderately 07 30-Dec-2015 Moderately Unsatisfactory 27.26 Unsatisfactory 08 01-Jun-2016 Moderately Satisfactory Moderately Satisfactory 30.94 09 12-Dec-2016 Moderately Satisfactory Satisfactory 40.73 10 25-May-2017 Moderately Satisfactory Satisfactory 52.65 11 30-Dec-2017 Moderately Satisfactory Moderately Satisfactory 63.12 12 15-Jun-2018 Moderately Satisfactory Moderately Satisfactory 70.12 13 16-Dec-2018 Moderately Satisfactory Moderately Satisfactory 79.41 SECTORS AND THEMES Sectors Major Sector/Sector (%) Agriculture, Fishing and Forestry 95 Agricultural Extension, Research, and Other Support 80 Activities Public Administration - Agriculture, Fishing & Forestry 15 Industry, Trade and Services 5 Agricultural markets, commercialization and agri- 5 business Page 3 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Private Sector Development 100 Jobs 100 Finance 2 Finance for Development 2 Agriculture Finance 2 Human Development and Gender 11 Nutrition and Food Security 11 Food Security 11 Urban and Rural Development 87 Rural Development 87 Rural Markets 2 Rural Infrastructure and service delivery 85 Environment and Natural Resource Management 17 Climate change 17 Mitigation 16 Adaptation 1 ADM STAFF Role At Approval At ICR Regional Vice President: Pamela Cox Victoria Kwakwa Country Director: Stefan G. Koeberle Rolande Simone Pryce Director: John A. Roome Benoit Bosquet Practice Manager: Franz R. Drees-Gross Dina Umali-Deininger Task Team Leader(s): Paulus Van Hofwegen Jan Joost Nijhoff ICR Contributing Author: Ir Mariam Rikhana Page 4 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. The appraisal of the Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) project took place in 2012, a time when Indonesia’s economy was growing at a rate of above 6 percent annually. Fresh out of the global recession of 2008–2009 (after weathering the 1997–1998 Asian financial crisis), the middle-income country got back on track the previous year, when its investment rating was upgraded following improvements in the banking sector and in domestic consumption. Following a similarly steady pattern, both relative and absolute poverty have continued to decline, with a rural poverty rate that stood at 14.3 percent in 2012—down from above 20 percent only a few years earlier. Inequality, measured by the Gini coefficient, had already plateaued at slightly above 40 percent at the time of appraisal and has remained slightly above that level ever since. The context of this trend is one in which growth had been accelerating fast since 2010 particularly in sectors such as the extractive industries and tertiary sectors, both characterized by high concentrations of ownership of assets and capital. 2. In 2012, rising inequality had to some extent been offset by agriculture sector growth and job creation in both urban and rural areas. Agriculture accounted for around 14 percent of gross domestic product and 35 percent of employment, as it finally saw a reversal of the decline in public investment of previous years. At the time of appraisal, the reference document for Indonesian agriculture was the National Master Plan for Acceleration and Expansion of Indonesia's Economic Development (Masterplan Percepatan dan Perluasan Pembangunan Ekonomi Indonesia, published in 2011), which assigned a central role to agriculture. With a rapidly growing population, food security concerns translated into policy attention to rice and to a dozen other ‘strategic’ food and cash crops toward which the bulk of the ministry’s budget was allocated as part of a new paradigm of fostering inclusive socioeconomic growth. Government policy priorities included enhancing the productivity and efficiency of rice production systems as one of the main outcomes to be pursued. 3. Although public spending and institutional support to the agricultural sector had been increasing for a few years by the time of appraisal, funding earmarked specifically for research and development (R&D) had not. Indonesia, one of the largest agricultural economies in Asia, spent proportionally less on agricultural R&D than most countries in the region. BRIC countries1 such as Brazil and India have been investing substantially more and have become dependable suppliers of solid agricultural research; their R&D institutions have been able to achieve higher research productivity and their systems have been generating higher fees received from royalties and licenses or from patents granted. The urgency for a new middle-income country such as Indonesia to invest more heavily in agricultural R&D was underscored at the time by a World Bank Agricultural Public Expenditure Review in Indonesia, as well as by an Australian study (ACIAR and ANU 2011)2 on enhancing productivity growth which concluded that this 1BRIC = Brazil, Russia, India, China. 2ACIAR (Australian Centre for International Agricultural Research) and ANU (Australian National University). 2011. “The Effect of Research on Agricultural Productivity in Indonesia.” AGB/2010/018, prepared by Professor Peter Warr. Page 5 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) would not happen with current levels of underinvestment in agricultural research. Biotechnology, for example, was singled out as a domain within which Indonesia would need to get up to speed, linking innovations to a new knowledge economy. Against this backdrop, the Indonesian Agency for Agricultural Research and Development (IAARD) was to become globally competent and locally relevant. Theory of Change (Results Chain) 4. Problem statement. In 2012, there was widespread consensus that to sustain and accelerate economic growth and poverty reduction, Indonesia needed to get ready to become more competitive by enabling more sophisticated value addition in agriculture. To do so, a concomitant shift to bring about the required new investments in human capacity was necessary to step up national capacity to innovate in its evolving and modernizing agri-food sector. “The rationale for investment in SMARTD lies in the need to address short-term constraints, especially on the HR side, while providing support for further transforming research management and establishing an overall policy framework that supports effective innovation through medium- and long-term reforms” (Project Appraisal Document [PAD], page 5). The country’s prime public institution specifically mandated to address this challenge, IAARD, was thus identified as the implementing agency. IAARD had been struggling to keep up with an evolving agri-food system that needed to respond to the innovation requirements that were driven by rapidly evolving consumption patterns that are typical of a populous middle-income country in transition, such as Indonesia. National food security and import substitution concerns pointed to the continued necessity of sustainable increases in agricultural productivity and value chain efficiency of some of the country’s main food crops. 5. A Theory of Change was not explicitly presented at the time of project appraisal. An ex post reconstruction of the Theory of Change is presented in figure 1. Decades of experience in this context have shown that technology transfer is not a linear process or chain of events but a sociopolitical process involving a tangled vertical and horizontal web of multi-stakeholder groups of actors. These are not necessarily subject to any one single overriding organizing force that would allow the channeling of all or most of these disparate resources toward a single envisaged agricultural development outcome. Of course, government priority strategies provide such a meta framework, but these remain imperfectly articulated when it comes to their operationalization on the ground, especially when it comes to agricultural research and extension. In the case of SMARTD, a consensus has however emerged, over the years, about what could be regarded to be a ‘SMARTD trademark’, a relatively simple formula applied in a complex context. It consists of the threefold combination of investing in human resource (HR) development, facilities upgrading, and research management improvement. The expectation was that synergistic planning among these clusters of activities would have allowed achieving the Project Development Objective (PDO). Page 6 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Figure 1. SMARTD Theory of Change Page 7 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Project Development Objectives 6. The PDO is “to improve the institutional capacity and performance of the Indonesian Agency for Agricultural Research and Development (IAARD) to develop and disseminate relevant and demand-driven innovative technologies, meeting the needs of producers and of the agri-food system.”3 Key Expected Outcomes and Outcome Indicators 7. Four outcome indicators, listed both in the PAD4 as well as the Loan Agreement, were to be measured: (a) Percentage of farmers, disaggregated by gender, adopting IAARD’s research results disseminated under the project; (b) Number of technologies developed and/or refined and are disseminated to users; (c) Number of articles relating to agricultural research and development carried out by IAARD published in internationally peer-reviewed journals; and (d) Percentage and number of sources of external funding in IAARD’s research and dissemination budget and percentage of IAARD's operational budget allocated to research and dissemination. Components 8. Component A: Human Resource Development and Management (Original allocation: US$40 million, actual: US$17.93 million). This component aimed at strengthening the scientific skills and research capacities of IAARD's professionals for a total of close to 1,400 researchers, technicians, and support staff. Its four subcomponents were Subcomponent A1: Scientific training (degree and other) and networking; Subcomponent A2: Technical and management training; Subcomponent A3: Human resources management (mobility program, incentives/promotion, and recruitment); and Subcomponent A4: Short-term expertise and support for implementation of Subcomponents A1, A2, and A3. 9. Component B: Improvement in Research Infrastructure and Facilities (Original allocation: US$35 million, actual: US$51.82 million). This component aimed at rehabilitating, improving, and upgrading the physical infrastructure of some of the operational units within IAARD and at developing a fully comprehensive strategic master plan to guide future investments in research infrastructure. Its three subcomponents were Subcomponent B1: Laboratory improvement, Subcomponent B2: Field station improvement, and Subcomponent B3: Building construction and renovation. 10. Component C: Research Management and Policy Support (Original allocation: US$15 million; actual: US$22.6 million). This component aimed at enhancing the efficiency and effectiveness in the use 3 The PDO is the same in the PAD and the legal agreement, save for the omission of “of” as in “and the agri-food system” in the legal agreement. 4 There is a slight, non-essential, difference in the wording of the indicators in the PAD. Page 8 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) of research resources through the implementation of improved research management strategies, processes, and instruments. Its four subcomponents were Subcomponent C1: Improvement of priority- setting, planning, and programming systems with more effective stakeholder participation; Subcomponent C2: Scaling-up of competitive research funds to enhance collaborative programs with other national and international research organizations and foster local innovation; Subcomponent C3: Support to the commercialization of research products, leading to the establishment of more diversified and sustainable research funding strategies; and Subcomponent C4: Improvement of technology dissemination systems. SMARTD also assisted in the generation of information and studies in support of strategic policy decision making. 11. Component D: Project Management and Monitoring and Evaluation (Original allocation: US$10 million; actual: US$2.95 million). This component aimed at facilitating project implementation, providing the necessary administrative support, and carrying out monitoring and evaluation (M&E) activities. 12. The Government contribution of US$20 million (of which US$18.2 million was disbursed) was programmed as follows: Component A: US$600,000 (3 percent), Component B: US$5 million (25 percent), Component C: US$11.4 million (57 percent), and Component D: US$3 million (15 percent). Original allocations and actual allocations are summarized below: Original allocation (US$m) Post-MTR allocation (US$m) Actual (US$m) IBRD+IAARD of which IAARD IBRD IAARD IBRD IAARD Total Component A 40 0.6 22.81 17.42 0.51 17.93 Component B 35 5.0 43.71 47.22 4.60 51.82 Component C 15 11.4 12.63 12.29 10.31 22.6 Component D 10 3.0 0.85 0.17 2.78 2.95 Total 100 20.0 80.00 20.00 77.10 18.20 95.30 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION Revised PDO and Outcome Targets 13. The PDO was not revised during implementation and neither were the outcome targets. Revised PDO Indicators 14. The PDO indicators were not revised. Revised Components 15. The components were not revised per se but, as noted, the financing envelopes allocated to them underwent changes. Other Changes 16. The March 2015 MTR recommendations and follow-up implementation support missions were reflected in a project restructuring (June 2016) which included an extension of the project closing date by Page 9 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) 21 months until June 30, 2019 and revision of some activities which required a reallocation of funds between components and corresponding adjustments to the disbursement estimates and indicator target dates. Rationale for Changes and Their Implication on the Original Theory of Change 17. The rationale for the changes introduced at the MTR and the subsequent restructuring was threefold: (a) a new government had recently set a revamped policy agenda for the agricultural sector and had tasked IAARD with refocusing its strategy on five priority commodities, namely, rice, corn, soybean, sugarcane, and beef; (b) with some of the initial delays in project start-up, there was less time available to allow PhD candidates to finish their studies before the loan closing date, therefore more emphasis was placed on increasing the number of MScs supported under the project; and (c) the low disbursement rate called for a reconsideration of some activities. The extension of the closing date was welcome because typically, several years are necessary before a new technology can be safely disseminated and adopted more widely within farming communities. The additional time thus allowed SMARTD to develop and disseminate a higher number of technology packets (see the second outcome indicator described in the following paragraphs). The budget reallocation between components was cost- neutral (see the section on financial management). Until the MTR, SMARTD had generated considerable cost savings, because of a decision by IAARD to give preference to more modest designs for buildings and facilities, and because of the appreciation of the U.S. dollar against the Indonesian rupiah (from US$1 = IDR 9,000 in 2012 to IDR 12,000 in 2015). The reduction of Ph.D. candidates in favor of M.Sc. candidates under Component A also generated savings that were reallocated to Component C. The implications of these changes on the Theory of Change do not involve any substantial deviation and are therefore deemed to be negligible. The changes above were timely and critical in allowing SMARTD to reach satisfactory overall performance, including in terms of physical progress and disbursement. II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 18. The relevance of the PDO of SMARTD is rated ‘High’. 19. At the time of appraisal, Indonesia had just reemerged from the global financial crisis, as a middle- income country ready to renew its commitments to democracy and economic growth, with agriculture playing a major role in generating employment and incomes and in feeding the nation. The Government's national medium-term development plan (Rencana Pembangunan Jangka Menengah Nasional, RPJMN) for 2010–2014 indicated institutional and governance reform and capacity building as national priorities. On ‘science and technology’, the RPJMN called for improving ‘the quality and utilization of science and technology to support the overall goal of improving the country's competitiveness and a move toward a more knowledge-based economy’—at the time, Indonesia ranked 103 out of 146 countries on the Knowledge Economy Index, lagging behind its neighbors, most of which also fared better in the ‘Global Competitiveness Index’, where Indonesia occupied rank 54, well behind Malaysia (24), China (29), Page 10 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Thailand (36), and India (49). Against this backdrop, underinvestment in agricultural R&D was rightly identified by the SMARTD appraisal team as a key issue to be addressed. 20. As described in Indonesia’s Strategic Plan of Agriculture for 2009–2014 published in 2011, the primary goals of agricultural development during that period were to (a) achieve and maintain self- sufficiency of five strategic commodities (rice, maize, soybean, sugar, and beef); (b) enhance food diversification; (c) increase value added, export, and competitiveness; and (d) increase farmers’ welfare. The design of SMARTD sought to contribute equally to all four of these primary goals, especially during the first years of project implementation. In 2015, a new government shifted policy priorities, paying even greater attention to the goal of self-sufficiency in strategic commodities. To achieve the Strategic Plan’s primary goals, the Ministry of Agriculture (MoA) focused on the revitalization of (a) land resources, (b) seed and breeds, (c) infrastructure, (d) HR, (e) agricultural finance, (f) farmer institutions, and (g) technology and downstream industry. Except points (e) and (f), SMARTD activities can be viewed as having played an important role in supporting the MoA with this ‘revitalization agenda’. 21. The PDO remained relevant throughout the life of the project. The relevance of the PDO was illustrated most evidently in the Country Partnership Strategy (CPS) for FY09–FY12. Its theme ‘Investing in Indonesia's Institutions’ sought to address some of the country’s critical governance and institutional challenges, while supporting ‘success stories of replicable institutional reform’. This focus continued in even more depth during the CPS for FY13-FY15, which saw the World Bank contributing to improve the performance of public institutions and enhance HR capacity and providing advice to “improve the public R&D institutes’ performance and science, technology, and innovation (STI) human resource capacity, including technical assistance for policy reforms to reduce fragmentation and increase competitive funding for research”. All of these elements are present in SMARTD, which furthermore, had a Results Framework conducive to gender mainstreaming, a topic rising to prominence at the time. The project was highlighted in that CPS under Pillar II as one of the Education Development Results envisaged by the World Bank, titled PRO-JOBS ‘Enhancing Skills and Technology, and Improving Social Protection’. 22. The vision contained in Indonesia’s Strategic Plan of Agriculture for 2015–2019 is “[the]…realization of sustainable agriculture-bioindustry systems producing various healthy foods and high value-added products based on local resources for food sovereignty and farmers welfare.” Consequently, the mission of the Indonesian MoA is to achieve food sovereignty, sustainable agriculture- bioindustry system, farmers welfare, and bureaucracy reform. The MoA’s objectives include (a) improving the availability and diversification of food toward food sovereignty, (b) increasing the value-added and competitiveness-enhancing agricultural food products, (c) developing raw material availabilities for bioindustry and bioenergy, (d) improving the income and welfare of farmers, and (e) improving the quality of the performance of the agricultural government apparatus. SMARTD has been able to provide timely and well-structured support to MoA’s mission and objectives during this period. Page 11 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Table 1. Consistency between PDO and World Bank Group Strategies for Indonesia Strategic World Bank World Bank Group Group Objectives to PDO and Longer-term Impact Strategy which Project Outcomes have Contributed Number of technologies developed and/or refined and CPS 09–12: ‘Investing in Strengthening that are disseminated to users Indonesia's Institutions’ institutional Percentage and number of sources of external funding accountability in IAARD research budget and percentage of IAARD's Strengthening operational budget allocated to research and institutional capacity dissemination Number of articles in internationally peer-reviewed CPS 2013–15: Education PRO-JOBS - Enhancing journals on agriculture R&D results developed by the Development Results Skills and Technology, IAARD system and Improving Social Protection Percentage of farmers, disaggregated by gender, CPF 2016–20: Food Modernized client and adopting IAARD’s research results disseminated under security and rural market-oriented the project development extension services accessible by smallholders Note: CPF = Country Partnership Framework. 23. The FY16-FY20 CPF (in effect at the time of project completion) underscored some of the persisting governance challenges affecting a country as large and diverse as Indonesia and that these remain a key impediment to eliminating poverty and enhancing shared prosperity. It was therefore decided to keep promoting the effectiveness of public institutions, improving service delivery and responsiveness and accountability to citizens at the local level; in this context, IAARD remained a relevant institution for World Bank support. The CPF contained a section (sub-annex 3:1b) which rated the World Bank’s pro-jobs engagements as ‘satisfactory’. In this context, it mentions that SMARTD is playing its role to enhance staff capacity by promoting research and innovation as well as technology development and uptake and that the project is on track to reach its targets; at the same time, the CPF acknowledged the fact that it was too early to judge the impact of SMARTD on the pro-jobs agenda. Table 2. SMARTD Contribution to the Total Budget of IAARD (IDR, million) Year 2011 2012 2013 2014 2015 2016 2017 2018 IAARD 1,147,110 1,288,620 1,745,280 1,582,600 1,876,600 1,925,500 1,658,700 2,092,700 budget SMARTD budget 67,635 129,735 135,810 266,436 200,745 220,995 SMARTD budget as share of IAARD 4 8 7 14 12 10.5 budget (%) 24. As illustrated in table 1, over the last decade, the World Bank partnered with the Government of Indonesia (GoI) through institutional support to building national managerial and technical capacities. The GoI and the World Bank decided to invest in IAARD, through a lending program, to strategically address Page 12 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) key institutional issues in agricultural R&D. As shown in table 2, the project provided up to 14 percent of IAARD’s total budget in 2015. Project interventions were strategic and contributed to the World Bank’s focus on implementing institutional strengthening and pro-jobs interventions in Indonesia over the years. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 25. The PDO has been ‘unpacked’ into two outcomes, as follows: 1. IAARD’s (physical and human) institutional capacity to develop and disseminate relevant demand-driven innovative technologies; 2. IAARD’s performance to develop and disseminate technologies meeting the needs of producers (farmers) and the agri-food system. 26. The assessment in this section follows the ‘reverse flow’ of the Theory of Change (see above), from outcomes at the farm level in the field toward intermediate outcomes (performance) and outputs and activities (capacity to perform). For illustrative purposes, a number of case studies are presented in summary form in the sections below. The full versions of the case studies are attached in annex 6. 27. Outcome 2: IAARD’s performance to develop and disseminate technologies meeting the needs of farmers and other users. The central performance indicator relates to technology adoption, the ‘penultimate’ link in the results chain. Adoption was measured through the first PDO indicator (‘percentage of farmers, disaggregated by gender, adopting IAARD research results disseminated under the project’), which was assessed using two studies by the Indonesian Centre for Agricultural Technology Assessment and Development (ICATAD, or ‘BBP2TP’ by its Indonesian acronym) on two separate technology dissemination programs, and information and data from Component C. With respect to the PDO, the indicator reflects the effectiveness of IAARD as an institution. 28. The key performance indicators for this outcome were PDO indicator P1 ‘percentage of farmers adopting IAARD research results, disaggregated by gender’ (see Annex 6.3); PDO indicator P2 ‘number of technologies developed and/or refined and that are disseminated to users’ (also used to assess Outcome 1); and number of dissemination areas under MP3MI (Multi-channel Pengembangan Pertanian Perdesaan Model Berbasis Inovasi, output indicator C4). By midterm, these had reached, respectively: 49 percent (starting from a baseline of 30 percent in 2012), of which 95 percent male and only 5 percent were female; 350 technologies (during 2015, up from a baseline of 32 in 2012); and 10 areas (last measured in 2015 after which that specific dissemination method was discontinued, and starting from a baseline of zero as this indicator was not measured in 2012). 29. Adoption rates reached 73–81 percent at project completion (77 percent average), disaggregated as follows: 74–84 (79 percent average) percent male farmers and 60-58 (59 percent average) female farmers adopting the technologies (the first figure refers to the ‘Jarwo Super’ technology packages, Page 13 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) whereas the second to the ‘MP3MI’ technology packages. Generally, agricultural technology adoption rates throughout the world tend to be below 50 percent on average.5 30. Adoption rates of IAARD technologies were measured in two phases of technology development funded under SMARTD: (a) the Multi-channel Research, Development, and Innovation Model (Multi- channel Pengembangan Pertanian Perdesaan Model berbasis Inovasi, MP3MI), a preexisting flagship initiative promoting a ‘model of rural agricultural development through innovation’, implemented from 2013 until 2015 and (b) ‘Jarwo Super’, a strategic priority initiative by the MoA focusing on integrated rice cultivation techniques, implemented in 2016. The former applied an approach combining various methods of technology dissemination with a ‘multichannel dissemination spectrum’ that had been piloted by IAARD since 2011. It consisted of on-farm participatory demonstrations done not only by extension workers and local agricultural service (Dinas Pertanian) staff, but also by private sector actors and progressive (‘cooperator’) farmers. SMARTD scaled it up, reaching 33 locations in all 33 Indonesian provinces in which IAARD could count on the presence of its outreach centers, the Assessment Institutes for Agricultural Technology (AIATs, or ‘BPTP’ by their Indonesian acronym). These institutes have the mandate of adapting and fine-tuning research results to local conditions by ‘repackaging’ them in collaboration with different stakeholders (as happened under MP3MI) or, at a minimum, through close interaction with ‘lead’ farmers. 31. The ICATAD studies6 covered a representative sample of different agroecosystems and estimated adoption levels of a total of 54 technology components, divided into (a) Three to six technology packages under MP3MI, including rice-livestock integration, an organic farming technology, cocoa cultivation, mechanization, potato seedlings production, and ‘KUB’ chicken (a ‘superior native chicken’) rearing, with a different focus in every province and (b) Nine major technology packages under Jarwo Super, consisting essentially of a new spacing method for rice crop establishment and wider area of dissemination. 32. Cooperator farmers were provided with free inputs and technical guidance and their adoption level was 89 percent for MP3MI and 72 percent for Jarwo Super. In the case of non-cooperator farmers in the same communities (receiving neither free inputs nor technical guidance), the adoption rate was 72 percent for MP3MI and 69 percent for Jarwo Super. For non-cooperator farmers who were not in the same communities (but in ‘neighboring villages’), the adoption rate was 89 percent for MP3MI and 83 5 See, for example, G. Kuehne et al. 2017. “Predicting Farmer Uptake of New Agricultural Practices: A Tool for Research, Extension and Policy.” Agricultural Systems Volume 156: 115–125, for a discussion of the many factors (in this case, 22) determining farmers’ decisions on technology adoption. Available at https://doi.org/10.1016/j.agsy.2017.06.007. 6 Measurement has not taken place on a continuous basis and reflects a heterogeneous range of technologies and dissemination environments which does however not allow for any straightforward assessment of the general picture that emerges (these shortcomings, as well as some good spin-off effects triggered by the project, are discussed in the M&E sections below). The ICATAD studies suffer from (a) a relatively small sample universe not representative of farmers at national scale across 33 provinces (8 provinces were covered in relation to the MP3MI program and 7 provinces in relation to the Jarwo Super Program) and (b) the absence of control groups to compare yields obtained, production costs incurred, and so on, with technologies other than the newly introduced ones. Nonetheless, the studies are scientifically rigorous, and their analytical methodologies are clearly explained. Page 14 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) percent for Jarwo Super (see Annex 6.3 for further details). Overall, the progress under this PDO outcome indicator shows a positive trend, with high adoption rates of IAARD research results. 33. The above adoption rates, measured among a cross-section of farming communities spread throughout approximately one-fourth of the country (7 or 8 out of 33 provinces), indicate that the technologies in question do meet the needs of producers. A discussion of their relevance and especially their demand-driven character is less straightforward because while MP3MI was a highly participatory program, Jarwo Super was the result of a central government decision to radically boost domestic rice production. 34. Insofar as the private sector is concerned, the strongest indication of IAARD technologies meeting the needs of the agri-food system comes from the successful commercialization of its research products. A corresponding activity promoted under the project (under Subcomponent C3) had the objective of increasing funding levels and diversifying funding sources—with the explicit aim of boosting nontax revenue. This was meant to take place through the identification of technology/knowledge commercialization opportunities, the development of a business plan for IAARD’s research product commercialization, and support to the implementation of strategies that seek to better disseminate research products to the benefit of end-users. 35. By project completion, IAARD commercialized its products through 586 IPR certificates, well above the baseline of 29 and an original target of 37 patents or licenses. This, however, is by no means attributable to SMARTD alone—rather, it should be viewed as part of the project’s significant leverage effect across IAARD as an institution. In sum, over the course of the project implementation period, 586 Intellectual Property Rights (IPR) certificates were newly issued or extended in connection with research activities: 240 licenses, 54 plant varieties, 168 patents, 115 copyrights, and 9 trademarks. Particularly noteworthy is the increasing number of patents received by IAARD as these are indicative of some of the potential for further commercialization of innovative technologies and other research products through the private sector. Box 1. Unlocking the Overlooked Potential of Sugar Palm Takes Several Years With productivity levels that are twice as high as that of the much better-known sugarcane, palm sugar holds the potential of supplying a much-increased share of Indonesia’s sugar needs. In the Province of Bengkulu on the western coast of Sumatra, sugar palm plants are intercropped with coffee as a source of daily income of the local communities from trade on the island and export as sugar and palm fiber. Collaborative research funded by SMARTD on the local variety called kapur began in 2016. It was registered with the name Smulen ST1 in 2018 and flagged as a superior national variety for further dissemination. Farmers can sell an average of 12 kg per day of molded sugar which provides an average income of IDR 192,000 per day (equivalent to approximately four to five times the daily wage for agricultural labor). The quality of Smulen ST1 palm sugar produced by local farmer groups has already met SNI 01-3743-1995 standards. With lobbying from the Bengkulu provincial government, IAARD is now working on the official nationwide release of this variety. 36. SMARTD’s support to innovation through research collaboration and competitive grant-funded projects contributed to improving its ability to register IPRs, which triggered a policy debate and the establishment of a dedicated working group, leading to the passing of the Ministry of Finance (MoF) Regulation No. 72 in 2015. This regulation states that royalties should accrue to the individual researcher(s) who has (have) developed an innovation for which a patent has been issued. Although Page 15 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) relatively recent, the regulation is already affecting the incentive structure, motivation, and work style of researchers positively, prompting them to pay much attention to the quality and marketability of their research results. Networking has become more prominent in IAARD, to foster closer ties with the private sector with a view to attract private financial, technical, and human resources for commercialization or further R&D. As can be seen in table 1, royalties accruing to IAARD and its researchers have risen steeply, from IDR 229 million in 2012 to IDR 5.107 million in 2018, corresponding to an increase that is more than twentyfold. 37. Business plans for research product commercialization were prepared annually as planned, and this was part of a broader quest for diversifying the funding sources of IAARD (as captured by the fourth PDO outcome indicator P4 ‘percentage and number of sources of external funding in IAARD research budget’ and indicator A3 ‘percentage of IAARD's operational budget allocated to research and dissemination’), the achievement of which was assessed using the institution’s statistical yearbooks. Table 3. External Funding in IAARD’s Research Budget (IDR , million) 2012 2013 2014 2015 2016 2017 2018 Research budget 191,754 293,195 173,622 204,095 273,223 100,292 207,509 Royalties 229 1,230 815 767 3,137 4,587 5,107 Foreign grants 17,674 24,509 21,184 9,386 6,221 7,372 11,158 In-country grants 885 1,122 954 4,246 Royalties and grants as 9.3 8.8 12.7 5.0 3.4 11.9 9.9 percentage of research budget 38. At project completion, external funding available to IAARD was 9.9 percent of its total budget, a figure that is lower than the original appraisal target of 12.5 percent (see table 3). Part of the explanation for this is that not long after SMARTD became effective, GoI passed a new MoF Decree—often referred to as the ‘one-door’ regulation—requiring all external funds to be channeled through the national budget system. In anticipation of steep increases in bureaucratic transaction costs, a restriction was imposed on the minimum external grant amounts that could be received, from then on fixed at IRD 500 million (approximately US$35,600). 39. The demand responsiveness of technologies generated under the project can be summarized as follows: On the supply side of technology development, the project supported a prioritization process by identifying research requirements through a feedback loop from farmers and agribusinesses, enabling the development of guidelines by the IAARD institutes to prioritize bottom-up research requirements. The project’s competitive research grants represent a significant contribution to influencing agricultural research priorities through better articulation between supply and demand. They have strengthened the responsiveness to local demands from farmers and agri-food system actors at the decentralized level of the single IAARD centers scattered throughout Indonesia, which are now better equipped to follow up on such demands and translate them into relevant and appropriate technological innovations. The available evidence shows relatively high adoption rates among farmers, including female farmers, and high adoption rates among ‘non-cooperator’ farmers in and outside the community. In addition, a high number of technologies were patented and licensed, and there is a demonstrated willingness by agribusiness operators to pay royalties. Page 16 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) 40. Outcome 1: IAARD’s (physical and human) institutional capacity to develop and disseminate relevant demand-driven innovative technologies. The step before technology adoption, namely technology development and dissemination, was measured by the second PDO outcome indicator (‘number of technologies developed and/or refined and that are disseminated to users’), which was assessed using IAARD’s statistical yearbooks, and information and data from Components A, B, and C. This PDO outcome indicator lies at the core of the official mandate and of the main activities of IAARD and as such, most project activities across all three components, contributed directly or indirectly to its achievement. Components A and C can be conceived as contributing to the part of the PDO relating to IAARD’s institutional performance (the ‘software’). The performance of research and extension staff in developing and refining a large number of technologies to prepare them for adoption, is in part a function of their capacities and incentives, as well as of the dedicated budget at their disposal.7 41. Many of the technology packets developed within IAARD over the last six years have been generated by SMARTD’s R&D partnership collaboration initiatives. Measured against the original target of 72 technology packets to be developed every year by project completion, and starting from a baseline of 32, the project achievement is an impressive 165 percent (equivalent to 119 technology packets developed in 2018). Two caveats are that (a) the baseline figure was a stopgap figure because of a lack of data at the time of appraisal and should therefore not be considered accurate and (b) SMARTD funded only part of the total number of technology packets developed. As can be seen in table 4, the number of packets has actually decreased after 2015–2016, mostly as a consequence of the new central government strategy that selected seven priority crops, toward which researchers’ attention and resources needed to shift. Table 4. Summary of IAARD’s Technology Packets and Dissemination through Various Media Year 2012 2013 2014 2015 2016 2017 2018 No. of technology packets 32 403 385 350 340 234 119 Table 5. Summary of IAARD’s Dissemination Format Dissemination 2013 2014 2015 2016 2017 2018 Format Printed 1,263 1,308 1,169 1,011 973 842 Electronic 2,756 2,922 2,840 2,169 2,948 4,576 Other 464 481 1,017 955 871 775 Total 4,483 4,711 5,026 4,138 4,792 6,193 Source: IAARD statistical yearbooks 7 See the annex for a snapshot of how the budget allocations for AIATs have evolved. Page 17 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Box 2. Harnessing the Multi-Functionality Potential of Sorghum, Including for Improving Nutrition Sweet sorghum (sorghum bicolor (l). moench) is a drought and salinity resistant crop. Its stems can be used for crystal sugar, bio-ethanol, animal feeds, and compost, while its grains can produce intermediate products such as granulate and oat flour, as well as final products such as instant rice and instant porridge oat, instant noodles, and beverage products. Sweet sorghum syrup is used for making bread, pancakes, bakery produce and ice cream, and pharmaceutical products. One of IAARD’s most successful research collaboration activities funded under SMARTD involved a holistic approach targeting several varieties of local sorghum. Research focused on planting equipment, threshers and post-harvest handling, and technology for processing sweet sorghum into food and liquid sugar. By project closing, sorghum cultivation in East Nusa Tenggara had reached a total of almost 500 ha, indicating how diets in East Nusa Tenggara are rapidly integrating this crop as a staple. In the future, this trend is also expected to reduce stunting in children. ‘Sorghum is nutritious, healthy, and profitable’ has become a familiar motto among local communities. 42. The project rightly featured additional activities aimed at further improving technology dissemination systems (under Subcomponent C4), for example, through the organization of periodic events at provincial and local levels for showcasing and demonstration purposes. IAARD took advantage of these occasions to strengthen the MP3MI system in the field and as vehicles to mobilize funding from local governments and partners. The dissemination system was also enriched by further diversifying and multiplying the different formats deployed for communication with IAARD beneficiaries and clients, notably placing more emphasis on electronic media; this choice is justified by the fact that Indonesians are among the world’s most arduous users of social media and electronic information. Table 5 shows IAARD’s turn toward electronic media, the use of which has seen a massive boost over the last year in particular. 43. In the context of contributing to improve IAARD’s performance, SMARTD also strove to improve research management (under Component C), encouraging an enhanced orientation toward results through a renewed focus on end users and clients. One perceived shortcoming at the time of appraisal was the lack of clear priority setting and sound planning and programming mechanisms within IAARD. Through the project, technological needs and priority commodities were successfully identified in all provinces and guidelines prepared on ‘priority setting in agricultural research and development programs’. SMARTD also supported the drafting of two ‘grand design’ reference documents, one titled ‘Human Resource and Development at IAARD’, finalized in 2015, and a second one titled ‘Facilities for IAARD’, a forward-looking strategy covering the period from 2015 to 2045 (not yet finalized). The document on the grand design of facilities responded to a need for guidance on the planning and procurement of infrastructure and equipment at the level of the decentralized IAARD centers. 44. With a view to foster innovations, 1,098 national and 44 international R&D research partnership collaboration activities were implemented over the lifetime of the project – refer to PDO indicator P3 ‘number of articles in internationally peer reviewed journals and publications on agricultural R&D results developed by the IAARD’. The national partnership collaboration activities were funded entirely by SMARTD until 2015 (by which time there were 111) and thereafter by the GoI (see table 5). Page 18 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Table 6. Number of R&D Partnership Collaboration Initiatives, 2013–2019 45. The allocation of competitive research grants followed clearly communicated selection criteria that were transparent and unambiguous (they were contained in a regularly revised and updated publication in Indonesian, titled Strategic Agricultural Research Assessment and Development Collaboration (Kerja sama Penelitian, Pengkajian, dan Pengembangan Pertanian Strategis, KP4S - 2018, available online).8 The criteria were applied as part of a well-conceived process, which constitutes a thorough project contribution to the good governance of agricultural research in Indonesia. Development partners and foundations, as well as private sector corporations (corporate social responsibility and commercial) interested in providing research funding to IAARD are expected to benefit from this. 46. Some structural issues impeded even stronger performance on certain collaborative partnership initiatives, for example, those involving laboratories (see also below, and indicators B1 ‘number of laboratories supported under SMARTD accredited national and internationally’ and B3 ‘number of laboratories supported under SMARTD involved in international research networking’). Indonesian government regulation stipulates that such types of collaboration must invariably be channeled through IAARD at the central level, which has slowed down the pace and suppressed the uptake of such ‘lab-to- lab’ collaboration. A hiring freeze (see section III.B) and the age structure of IAARD staff—it is estimated that more than 100 researchers will be retiring within the next five years—has detracted from what overall remains a combined positive trend in the number of national and international R&D partnership collaboration activities (see also indicator C1 ‘number of national and international research collaborations’). Interviews conducted with resource persons across the IAARD hierarchy have confirmed that since the onset of SMARTD, ‘collaboration has become the name of the game’, and that this motto has effectively been ‘exported’ from the central level to the provincial and district levels (through decentralization of some of the competitive funding schemes). Given the well-known ‘silo culture’ of a lot of national research institutions around the world, and the fact that work-related habits and comfort zones can be rather deeply entrenched and behavioral change elusive, this should be viewed as a rather significant positive achievement induced by the project. 8 https://scholar.google.com/scholar?hl=en&as_sdt=0,9&as_vis=1&q=Kerjasama+Penelitian,+Pengkajian,+dan+Pengembangan+Pertanian+Strategis. Page 19 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Figure 2. The New Laboratory of the Indonesian Centre for Estate Crops Research and Development (ICECRD) in Bogor 47. The above relates to IAARD’s scientific and intellectual capacity. The intermediate outcome indicator A1 ‘percent of researchers in IAARD’ sought to gauge the extent to which the institution can count on a large enough pool of researchers who are capable of developing innovative technologies for dissemination and attract and retain them in the short to medium term. This percentage has remained stable at about 26 percent throughout the project implementation period; in fact, it has increased slightly in relative terms, from 25.2 percent (of which 44 percent were women) in 2012 to 26.9 percent (of which 52 percent were women) in 2018. It must be borne in mind that there has been a general decline in staff numbers because a general public sector hiring freeze in Indonesia has affected IAARD, especially since 2015; the total number of staff at the institution has decreased from 7,780 in 2012 to 6,161 in 2018. The target (contained in the PAD, indicator A1) was 33 percent after five years of project implementation. 48. By providing a budget earmarked specifically for the purpose, SMARTD has pioneered collaboration between IAARD and institutions inside and outside of the MoA. In future, researchers will need to engage more proactively in networking and proposal writing to obtain new knowledge and funding from external sources, for example, the Indonesian Research Fund (DIPI, by its Indonesian acronym Dana Ilmu Pengetahuan Indonesia), and from nontraditional funding sources (including philanthropic organizations). Through the many technical and managerial training sessions it has facilitated, the project has done an excellent job of helping researchers adjust to the new situation. Interviews and anecdotal evidence suggest that indeed SMARTD trainee researchers are getting a head start in what has become a more competitive, transparent, rigorous, and accountable agricultural research environment that is no longer isolated from the international scene. Page 20 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Box 3. SMARTD Investments in IAARD Researchers’ Higher Education Contributing to the part of the PDO relating to IAARD’s human resource capacity (the ‘software’), SMARTD has also invested significant financial and technical resources to make available more and better educational opportunities for IAARD researchers. These ranged from relatively basic levels to Ph.D. and post doctorate training and research and were associated with varying scientific publication requirements. A total of 68 Ph.D. students, 32 male and 36 female, were sponsored by the project, along with a total of 93 M.Sc. students, 50 male and 43 female. The baseline for intermediate outcome indicator A2 ( ‘percent of IAARD researchers having M.Sc. and Ph.D. degrees’) contained in the PAD was 1,474 staff, which was to increase to 1,624 after five years of project implementation (an increase of about 10 percent). At project completion, there were only 1,204 such staff (761 Ph.D. and 443 M.Sc.); in relative terms, however, the percentage (according to the original indicator) increased from a baseline value of 54 percent to 72.5 percent Support for participation of IAARD staff in international seminars, workshops, and conferences was a very successful and popular activity and targets were vastly exceeded on all counts (see a comment on this in section V). An internship program established under SMARTD encouraged competent and experienced staff to mentor a next generation of researchers. Management training also exceeded its target at 103 percent, whereas technical training, admittedly much needed according to many senior IAARD staff, performed less well with a target met only at 55 percent. 49. Project activities on infrastructure upgrading (under Component B) can be conceived of as contributing to the part of the PDO relating to IAARD’s physical infrastructure capacity (the ‘hardware’). This consideration is based on the assumption made at project appraisal that once more and better facilities—upgraded and accredited—would be available to IAARD centers, these would be able to increase the quantity and quality of their research collaboration initiatives at the national and international levels. 50. The project enabled improvements in the capacity of 19 laboratories, with new ISO 17025 accreditation for 7 laboratories, ISO 17025 re-accreditation for 6 laboratories, and accreditation extension for a further 6 laboratories. Data in the Results Framework show the steady increase of laboratories and experimental stations accredited through SMARTD, from a baseline of 15 to 36 (in addition, 4 are undergoing the accreditation process). The target was 21 laboratories. Certain laboratories newly built with project funding were approved as reference laboratories, including the veterinary laboratory of the Indonesian Research Centre for Veterinary Sciences, the Soil Lab of the Indonesian Soil Research Institute, and the Post-harvest Lab of the Indonesian Centre for Agricultural Post Harvest Research and Development. 51. Further results include the updating of ISO 9001 certification from the 2008 to the 2015 version for 41 experimental stations. Although not completely aligned with the original target of stations selected on a case-by-case basis, the total number of improved stations at completion exceeds the original target of eight stations. 52. In terms of ISO 17025 accreditation for non-accredited laboratories, out of 159 laboratories situated in research institutes and AIATs, 46 laboratories were supported by the project and had finalized this process at project completion (not a results indicator). Available evidence (from third parties, publications, and field visits) point to good usage and maintenance of the facilities funded under SMARTD. The nanotechnology laboratory, for example, is piloting a range of innovations with commercial potential, Page 21 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) such as the development of breadfruit flour as a substitute in the processing of noodles or the development of plastic-type bags made from (widely available) rice crop residues. 53. Overall, the project invested substantially in infrastructure, for a total of US$27.8 million (about 61 percent of expenditures under Component B) and in equipment, for a total of US$18.0 million (about 39 percent of expenditures). Importantly, out of this budget, US$17.7 million (39 percent of the total) were spent at decentralized level, on infrastructure and equipment for 22 research centers and 28 AIATs. Results to be highlighted include the procurement of lab equipment (36 items; original target: 12 items) and experimental station equipment (14 items; original target: 7 items), as well as lab and office construction (52 buildings; original target: 6 buildings) and experimental station infrastructure construction (36 buildings; original target: 14 buildings). As part of the field-level focus, attention was paid to seed production units and machinery, further allowing the project to enhance its impact within farming communities throughout Indonesia. It seems safe to conclude that the physical upgrading of IAARD infrastructure through the project has played an important role in the research collaboration initiatives and technology packets that the institution has been able to develop and roll out since 2012. Justification of Overall Efficacy Rating 54. The overall efficacy rating for SMARTD is ‘Substantial’. The project achieved its main intended outcomes. While a ‘Substantial’ rating can be justified based on the available evidence, a ‘High’ efficacy rating cannot be justified due to limitations in the project’s ability to generate a fully comprehensive evidence base that tracked all aspects of the (complex) PDO. C. EFFICIENCY Assessment of Efficiency and Rating 55. The methodology used to assess the efficiency of SMARTD needs to be adapted to two specificities of the project. The first is that the causal chain and the time frame for the returns on investment for SMARTD are longer than in most projects. A second specificity is that it is very difficult to isolate the returns of SMARTD activities. The process of developing and mainstreaming a technology involves different steps, the research to develop the technology, the dissemination of the technology by AIATs, and the mainstreaming of the technology through the extension networks. SMARTD did not systematically finance all these steps in the process (from beginning to end), so the investment costs of a particular technology development do not correspond to SMARTD’s investment in that technology component. 56. To assess the returns of SMARTD activities, project-specific evidence of economic and financial impacts are assessed based on a subset of SMARTD investments and their expected outcomes. Because the time frame and data are insufficient to assess the extent to which technologies are adopted, a break- even analysis is used to determine how many units of production need to adopt the new technology to recover the investment costs. 57. Technology-level impact. IAARD developed a wide range of technologies, ranging from new seed varieties to the development of products based on nanotechnology. The AIATs will typically disseminate a package of different technologies. The technologies modelled for this analysis were constrained by available data but nonetheless represent different sub-sectors and types of technologies. Page 22 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Table 7. Models Model Sub-sector Technology Rice PATBO super Food crops, lowland rainfed rice Technologies for rainfed lowland rice fields JARWO Super Food crops, lowland rice A package of technologies for rice TSS shallots Horticulture, shallots Shallot seeds Garlic seeds Horticulture, garlic Garlic seeds Cow feeding Livestock, cows Improved cow feeding integrating horticulture crops Sugar cane Estate crops, sugar cane Improved seeds Bio-nano Silica Post-harvest technologies, rice Development of bio-nano silica from rice husks 58. A traditional financial analysis would not be adequate for these models because the underlying investment and the cost of research and technology dissemination are public investments that are not borne by producers. Regardless, the question of whether adopting a new technology is financially attractive for producers remains relevant, because it is a precondition for adoption and for generating benefits from the research. Thus, the producer margin for the new technology must be greater than the producer margin with the old technology. Table 8 summarizes the incremental margin per technology or technology package. Two incremental margins are presented. The first column A comes from the publications of the AIATs and is based on optimal conditions for implementation, including access to the full technology package and technical assistance. The second column B corresponds to 70 percent of that margin, based on the assumption that incremental margins might be lower for producers outside of the assessment and dissemination program context, without access to all inputs and technical assistance. In the case of cow feeding and the Bio-nano silica, the data from the incremental margin do not come from publications, so it is considered to be the producers’ incremental margin. Table 8. Incremental Margins 59. For the economic analysis, the technology-level analysis assesses adoption needed to break-even on the investment costs. The break-even analysis considers the full cost of development and dissemination of a technology. The cost of investment is the key variable in this analysis, because it is the most difficult to estimate and the results are very sensitive to assumptions. The cost used is the average cost for R&D and dissemination for each technology package disseminated through an AIAT in one province. The average cost per disseminated technology package is IDR 11.6 billion, corresponding to US$0.51 million. 60. The benefits from the investment correspond to the incremental producer margin determined in the financial analysis. Because a lag can be expected from the moment the research is conducted until the Page 23 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) technology is more widely disseminated and adopted, the adjusted additional margin is discounted by four years using an economic discount rate of 8 percent, on the basis of an annual rate of interest of 7.23 percent for Indonesia 10-year treasury bonds. 61. The analysis shows that the research technology packages assessed are likely to bring significant returns. For the models assessed, the adoptions needed per dissemination are rather low. For instance, in the case of PATBO Super rice, an adoption on 2,804 ha would suffice to break even from the cost of R&D and dissemination in one province. If this package was disseminated in two provinces, the total break-even adoption level would be about 6,000 ha in both provinces. In the case of Jarwo Super, the analysis takes into consideration that the technology package was adopted in seven provinces, so the investment costs are higher, US$3.5 million, and the break-even adoption is about 10,500 ha. Rice is cultivated on over 15,000,000 ha in Indonesia, and about 80 percent of these lands are lowland rice fields. The adoptions needed for break-even for Jarwo Super and PATBO super are therefore very small in proportion to rice growing areas. For sugar cane, the total area cultivated was 430,112 ha. Table 9. Break-Even Analysis Note: TSS = 62. There are several limitations to the methodology and results described above. The main limitation is that the cost of the research is an estimation. Using an average cost of investment does not capture the fact that the cost of R&D is likely to vary significantly from one technology to another. In addition, while the methodology captures the economies of scale on average, it cannot prove the hypothesis that research technologies disseminated in many provinces are more cost-effective than those disseminated only in one province. Moreover, the estimates of the investment costs are based on the IAARD budget and do not include the cost of extension workers. Another important limitation to the methodology is the bias in the availability of data and the subsequent choice of models. Indeed, the technologies modelled here are subject to a ‘publication bias’, because successful research endeavors are more likely to result in a publication than less successful ones. 63. Macro-level impact. A study carried out by the ACIAR in 2011 sought to quantify the impact of research on agricultural productivity in Indonesia. The study examined the extent to which agricultural research within Indonesia contributes to the enhancement of productivity growth and attempted to explain determinants of total factor productivity (TFP) growth, in particular, the contribution of agricultural research. The resulting estimated annual real rate of return from a marginal increase in investment in Indonesian agricultural research is 13 percent. Page 24 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) 64. Based on the assumption that the results from the 2011 study remain valid, a 13 percent return on the SMARTD investments would correspond to a net present value (NPV) of US$12.2 million (IDR 173 billion), corresponding to an economic rate of return (ERR) of 11 percent. The ERR is 11 percent and not 13 percent because Component D is not assumed to generate a 13 percent return, while it is included in the project costs. If all project costs, including Component D, are assumed to generate a 13 percent return, the NPV reaches US$21.8 million (IDR 308 billion) and the ERR is 13 percent. It is not clear whether the original study computing the 13 percent return considered overhead costs for IAARD as investments. Both NPVs are computed using a discount rate of 8 percent, as above. Other Efficiency Considerations 65. Revenues from licenses and royalties. Beyond the analysis of the return on investment of SMARTD, some broader conclusions can be made on the project efficiency. While a PDO indicator considers the external funding in IAARD research budget, it is interesting to also consider the revenue from licenses and royalties more specifically. Revenue from royalties has increased significantly over the project lifetime, from IDR 1.2 billion in 2013 to IDR 5.1 billion in 2018. Similarly, the IAARD nontax revenue has increased from IDR 12.8 billion in 2014 (US$0.90 million) to IDR 31.6 billion in 2018 (US$2.23 million). This shows that the organization is increasingly successful at developing technologies that are ultimately licensed and commercialized and at directly commercializing services. 66. Implementation considerations. Finally, it is important to note that efficiency was affected by the initial slow start of activities. Indeed, by December 2015, the disbursement rate only reached 34 percent of the total loan, because of slow procurement processes, and the project was extended, partly also to allow for time for students enrolled in Ph.D.s to finish their studies. In the end, the slow start led to reforms in the procurement system of the institution, which was decentralized to the different research agencies in January 2016, thereby increasing the efficiency of procurement throughout IAARD. 67. Rating. The low adoption rates required for break-even of investments on technologies generated under the project and the broader 13 percent ERR based on empirical approaches to computation of ERR for research type projects demonstrate the viability of the project. Although the implementation period was extended by 21 months, the realization of almost all of the end-of-project (EOP) targets and the dramatic increase in IAARD royalties and nontax revenues are additional indicators of efficient project implementation. 68. The efficiency of SMARTD is therefore rated ‘Substantial’. D. JUSTIFICATION OF OVERALL OUTCOME RATING 69. Based on the combined ratings in the relevance (High), efficacy (Substantial), and efficiency (Substantial) categories, the overall outcome rating of SMARTD is ‘Satisfactory’. Page 25 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) E. OTHER OUTCOMES AND IMPACTS Gender 70. IAARD’s HR policies do not feature any proactive measures to promote gender equality and neither did the project design at appraisal, which adhered to the ‘bare minimum’ standard of requesting that any data collected would be gender disaggregated, if possible. At project completion, female staffing numbers at all levels of this institution were impressive, making for a close to perfect gender balance. To illustrate this claim, the following may be noted: out of a total of 68 Ph.D. students sponsored by SMARTD, 36 were female; out of 93 Master’s students, 43 were female. Researchers and extension staff supported by the project to actively participate in international seminars, workshops, and conferences, by presenting their work, was also almost completely evenly split between women and men. Research and technical management trainees comprised 111 male and 132 female staff and 73 male and 91 female participants in the context of professional ability training. At the outcome level, one of the IAARD technologies aimed at improving local poultry breeds. The breed referred to as ‘KUB chicken’ is nutritious and fetches good prices in the market; because chicken rearing is an activity mostly carried out by women and female- headed households, it is reasonable to assume that there has been substantial positive impact on rural women in this connection. Institutional Strengthening 71. Institutional strengthening outcomes will not be assessed separately as such, given that building the capacity of IAARD was a central element in this PDO. The project’s infrastructure, equipment, and training focus on the field (research institutes and AIAT) level was appropriate and is likely to pay off considerably in the future. SMARTD’s injection of relatively sizeable financial resources to be spent within a relatively short time frame prompted the institution to improve its procurement practices. Particularly noteworthy was the decentralization of procurement and contract management to the different research agencies that are part of the IAARD system. This resulted in enhanced local ownership, accountability, rigor, and efficiency—decentralized procurement has now become the norm. 72. Activities aimed at the improvement of promotion and incentive systems and recruitment procedures were less effective than anticipated, owing to the fact that in Indonesia, these aspects are mostly standardized for all public sector employees and thus beyond the authority of IAARD/MoA. However, the design of the project did include several activities that had an influence on the incentives and motivation of IAARD staff. For example, the number of awards assigned to distinguished, particularly productive and high-performance, researchers increased, as it is to some extent correlated with the number of patents registered by these researchers, which are linked to the technologies they have been able to develop or refine with SMARTD funding and for which IPRs have been filed. 73. Several additional considerations can however be made to position an institutional strengthening project such as SMARTD in the broader perspective of the current opportunities and threats faced by the agricultural sector in Indonesia. The first one relates to IAARD’s responsiveness when it comes to dealing with emergencies such as the fall army worm or African swine fever. This aspect appears to have been somewhat overlooked and it may warrant greater attention in the future. The second aspect relates to the low visibility of climate change as a concern for the future of farming in Southeast Asia. Although several of the competitive research grants have addressed this issue directly or indirectly and two Page 26 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) technical trainings dealt with this topic, relatively little prominence seems to be given to this thematic area of work under IAARD.9 Finally, another very important task for professionals working with global agri-food systems is the reduction of post-harvest losses, which does not feature very prominently in IAARD at present (for a noteworthy exception, see box 4). Yet, all things considered, SMARTD has contributed to making IAARD a stronger (and more gender-sensitive) institution than it was in 2012. Mobilizing Private Sector Financing 74. While a PDO indicator considers the external funding in the IAARD research budget, it is also interesting to consider the revenue from licenses and royalties of IAARD more specifically, because it reflects revenues from non-public and non-grant funding.10 This shows that the organization is increasingly successful at developing technologies that are ultimately licensed and commercialized and at directly commercializing some services. This increase in revenues also occurred at the level of research institutions, as illustrates a case study of the Field Station for Tropical Fruit in Subang, which increased its revenues from IDR 10 million in 2014 to IDR 100 million in 2019 (as documented in the SAMRTD Project Completion Report by IAARD). Nonetheless, it is important to note that research remains a public good and, while the increase in external funding represents positive momentum, public funding of IAARD remains necessary and desirable. Box 4. Reducing Post-harvest Losses for Snake Fruit Exporters One of the international research collaboration initiatives under a scheme referred to as KKP3I ( Kegiatan Kerjasama Penelitian dan Pengembangan Pertanian Internasional ) sought to improve the packaging technology for snake fruits in the Sleman District of Java. In collaboration with two snake fruit exporters, the local AIAT introduced the use of innovative equipment for grading and wrapping snake fruit in specialized plastic, considerably lengthening its short shelf life. The activity was also supported by New Zealand, one of the main destination countries of snake fruit exported from Indonesia. The new grading equipment was created by AIAT researchers and has already been awarded an IPR license. Use of this equipment leads to a 25 percent reduction in post-harvest damages. The research project also worked on Good Agriculture Practice certification for snake fruit farmers employing environment-friendly technology. This has been followed up on by the local government, which promotes and supports the GAP program snake fruit farmers producing for the export market. Poverty Reduction and Shared Prosperity 75. Some of the types of R&D activities promoted under SMARTD are ‘pro-poor’ in nature. Worth flagging, for example, is research on improved animal feed, which allows for cost reductions of up to 25 percent. Such savings can be of extreme importance to poor and vulnerable households, who are often 9 On the other hand, ‘environment-friendly technology’ is a frequent tag of IAARD’s research themes. The Indonesia Peat-land Research Institute in Banjarbaru, South Kalimantan, for example, has been equipped with ISO accredited laboratory facilities through SMARTD. Since becoming operational, the facilities have been used mostly to conduct research on biopesticides, biofilters for water quality improvement/management, and biocharges for emission reduction, with the aim of improving local pesticide use. 10 Revenue from royalties has increased significantly over the project lifetime from IDR 1,230 million in 2013 to IDR 5,107 million in 2018. Similarly, the IAARD nontax revenue has increased from IDR 12.8 billion in 2014 (US$0.90 million) to IDR 31.6 billion in 2018 (US$2.23 million). Page 27 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) landless (particularly women), but own two or three milk cows—as is the case, for example, in much of rural Java. Especially since 2015, the project started to pay much attention to research on strategic food security crops such as rice and maize and over the last few years, also to coffee and cocoa (typical ‘pro - poor’ crops grown by smallholder farmers all across the country). Regarding the shared prosperity agenda, SMARTD has focused on a government institution providing public services to promote agricultural development. IAARD has begun important efforts to make its products easily accessible and widely available throughout Indonesia. According to current legislation on the matter, its research on rice is and will remain in the public domain. Other Unintended Outcomes and Impacts 76. SMARTD has had a positive influence on the incentives system of researchers. Even though changing the rules on incentives including promotions, special assignments, and so on, was and still is beyond the remit of IAARD or even the MoA (these are applied across all ministries and other government agencies as part of the national HR system), the project has had the indirect impact of helping some of the researchers involved move up the career ladder. This is because one of the requirements for a promotion to the next level consists of a minimum number of published scientific articles, which SMARTD has been supporting through the research grants. It has had the welcome and unintended effect of bringing in more young professionals as senior scientists at IAARD.11 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 77. The project design was rather ambitious as it sought to turn IAARD into a different institution, one that would be much more innovative, demand-driven, results oriented, and international. This required institutional transformation based on important internal changes and strengthened linkages with stakeholders, mainly in the domain of research management, which needed to become more strategic and relevant to Indonesia’s modernizing economy and fast evolving food system. Against this backdrop, the objectives set for SMARTD were realistic, but hard to achieve as well as dependent on many factors beyond the remit of the project. The project design was straightforward enough, with two ‘soft’ components and one component dedicated to infrastructure, and a light management structure fully integrated within IAARD. The operational logic was clear and to the point, and the timing and sequencing of tasks to be implemented was pertinent and logical. The Government’s commitment to the project was strong, also evidenced by the US$20 million own contribution that was committed. 78. A key risk in project design identified at appraisal was that having more qualified researchers is a necessary but not sufficient condition for higher quality research outcomes. Civil servant researchers did not have enough incentives to do better research, particularly in terms of better market orientation and industry linkages. The substantial investment in scholarships for advanced overseas training of 11On the other hand, about 75 researchers currently ages between 33 and 41 years have no chance of upwards mobility within the next five years due to a recent IAARD regulation indicating that to qualify they must possess a Ph.D. degree, while at the same time, their age makes them ineligible to enroll in higher education. Page 28 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) researchers supported by SMARTD may not automatically lead to better research outcomes. The mitigation measure entailed addressing institutional management issues to provide better incentives for outcome, as planned under component C of the project. This mitigation measure by itself did not address incentives as part of career management because that falls outside the remit of MoA (it is handled by a central government unit). However, incentives in the form of a reward system linked to IPF-related royalties were developed with support from SMARTD. A planned complementary activity – and a risk mitigation measure – was the likely funding by AusAID of a technical assistance activity to complement SMARTD implementation with an indicative amount of US$6.0-6.5 million. This activity envisaged supporting a revision of research priority-setting mechanisms, the scaling up of the competitive grant schemes, and the development of IAARD's business plan for research product commercialization. However, the technical assistance did not materialize. 79. The Results Framework included several indicators with questionable baselines, as well as, more importantly, indicators measuring results that could not be attributed to SMARTD alone. Targets seemed reasonable, achievable, and aligned with operational objectives, but PDO outcome indicators were not always clear or always directly relevant. A monitoring plan including realistic measures to collect information that constitutes evidence of achievement of outcomes, was not prepared at appraisal and it took the PMU a long time to get it ready. B. KEY FACTORS DURING IMPLEMENTATION 80. Certain factors that were under the control of the MoA affected SMARTD implementation. These included a number of policy changes at the central government level, some wide ranging and affecting the agriculture sector overall and some limited to research at IAARD. The MoA’s priorities changed when a new government came to power in 2015 and tasked the MoA with refocusing its work on seven strategic commodities (rice, maize, soybean, sugar cane, shallot, cattle beef, and garlic; later increased by two more: coffee and cocoa). These began to be given precedence over work on any other commodity, which affected the project-sponsored competitive research grant schemes that were not previously driven by parameters such as priority commodities. Another factor affecting implementation was an MoA restriction on sending staff overseas on scientific visits, contributing to the rather low level of achievement of the non-degree program under Component A. The government contribution of US$20 million was adequately budgeted for and has been made available to the project. 81. The project experienced a slow start-up phase, in part caused by an initial lack of administrative capacity to manage procurement. An MTR was carried out in March 2015 which recommended a restructuring, including an extension of the project until June 30, 2019 and a cost-neutral budget reallocation between components (see section I B on changes during implementation and IV B on financial management). The additional time allowed SMARTD to develop and disseminate a higher number of technology packets. Administrative arrangements were made to decentralize procurement and decentralized staff received procurement training. 82. Although IAARD remained strongly committed to implementing SMARTD successfully, there was a significant turnover of staffing in the PMU, including managers. Leadership was not constant but improved after the MTR, allowing the project to achieve most of its targets. IAARD considered that SMARTD provided important management experience to appointed PMU staff, some of whom were indeed promoted to director (Echelons 2 and 3) level. Staffing levels within the PMU were appropriate Page 29 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) and the skills set of the unit’s members were broadly satisfactory, either immediately or after being trained by the project and/or experts from the World Bank. The PMU was housed in adequate office space and could avail itself of good information and communication technology and transportation logistics. Initially, some of the environmental safeguard measures (mostly related to the use and disposal of chemicals in laboratories) were inadequate, but this could be successfully addressed by the project, especially in the case of the laboratories that were upgraded and accredited. 83. Factors under the control of the World Bank included proactive implementation support and appropriate follow-up which was provided regularly and on time. This contributed to the resolution of most implementation issues, which were reported regularly and candidly in implementation support mission (ISM) reports. When the dissemination program that formed the basis for measuring technology adoption was discontinued in 2015, the World Bank team suggested that an adoption survey be developed and implemented (the ICATAD studies referenced previously). 84. Factors outside of the control of the MoA affecting project implementation included (a) a general moratorium on new hiring of public servants, which has been affecting IAARD as well, especially since 2015 and (b) an MoF Decree requiring all external funds to be channeled through the national budget system, adding further bureaucratic burden to an already rather slow system; in this context, a new restriction was also imposed on minimum external grant amounts (IDR 500 million or approximately US$35,600) that can be received, including those destined to research at IAARD. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 85. A Results Framework provided the main reference for M&E in SMARTD. Baseline information for the framework was difficult to obtain for the first two outcome indicators on technology dissemination and adoption. As a result, the design of the M&E system needed to accommodate the rather limited availability of data and data collection mechanisms at the time. In addition, the PDO is complex and has many nuances, and the PDO indicators did not capture all the PDO aspects (innovation and demand- driven, for example). 86. While baselines and targets were provided in the Results Framework, these were presented without a methodology and in some cases appeared to bear little relation to the data that were collected during program implementation. Additional gaps/flaws of the M&E design were that no outputs had been defined at appraisal and no standard ‘working definitions’ of certain indicators (where IAARD did not dispose of such definitions and/or where these were deemed unsuitable, that is, unmeasurable or imprecise) had been defined either. Some of the indicators that were presented in the PAD were revised during the negotiation of the Loan Agreement, leading to arguably better definitions, but also creating uncertainty about which version of each indicator was to be applied. In addition, abbreviated versions of the indicators were used (including in the World Bank ISRs). However, despite slight differences in wording, the substance was not altered. Page 30 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) 87. Following the MTR, a simple methodology for measuring technology adoption was developed, but the dissemination program that formed its basis was discontinued in the following year, and a new approach was needed. For measuring the indicator on adoption, a survey was developed and implemented in 2018/2019. A few original indicators were flawed, either in terms of their wording (for example, indicators C2 and C312 - see annex 1), or in terms of how directly relevant they were to the project (for example, the indicator C4). 88. Intermediate outcome indicators were defined by the PMU with support from the World Bank. Details on how to implement the M&E system, from where and how to collect data, type of information, and so on, were finally elaborated and agreed upon in 2016 as part of the Project Operations Manual (POM). This document did refer to outputs, both by component and subcomponent and also at ‘subproject’ level, meaning that it is in the context of implementing one of the subcomponents (C1: Priority setting, planning, and program[ming]) that a methodology for research output identification and output measurement was spelled out, to be customized for the different participating IAARD centers. M&E Implementation 89. As noted by the MTR in March 2015, the absence of a results chain has made it more difficult to develop or present a logical structure of how the various project components interlink and integrate. As noted, no outputs had been defined at appraisal and although the MTR mission did remind the PMU of the importance of outputs as an integral part of a well-structured M&E system, their absence was deemed less of a priority at this stage. In their place, the MTR called for the preparation of M&E plans and the operationalization of the management information system (MIS). The MTR also recommended to the PMU to conduct two monitoring studies; of these, only one was carried out: the study on the percentage of farmers that have adopted IAARD research results was split into two, as follows: (a) adoption of technology (percentage of farmers and (b) effectiveness of innovation dissemination (both available in Bahasa Indonesia). A timelier engagement of the M&E focal point would have strengthened the performance of the M&E system. 90. As described in the POM, data collection was arranged by component, and standard annual project reports were prepared. In spite of the problems related to the definitions of indicators, the program was able to continuously collect and present information related to program performance. Data on most indicators were collected on time and project reports were presented during ISMs, with reporting improving significantly over the program’s duration. Early reports were somewhat limited to the presentation of data with little analysis of links between outputs or components. However, following the MTR, the reports showed improved analysis of links and strategic thinking and helped to identify strengths and weaknesses. The reports provided information on the progress in implementation, the outputs of each component, the progress on procurement and disbursement; as well as proposals for project 12Indicator C3 (IAARD annual work programs take into account the results of the priority setting planning and programming systems developed under the project) was not rigorously measured. However, the finalized guidelines for priority setting, research collaboration, and dissemination models were prepared and distributed in 2017/18. The extent to which they were taken into account in the multitude of annual plans was difficult to determine. Page 31 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) adjustment and reallocation of funds to improve project performance. Sound planning in Component B contributed to its good performance. M&E Utilization 91. With the links and interactions between the various project components not well defined, the task of implementing an M&E system that is able to inform project management (the PMU manager and the three coordinators of components) decision making and higher-level operation managers and policy makers was difficult. 92. On the other hand, there is no evidence of any major shortcomings in terms of late or ‘wrong’ decisions taken by the PMU. Information collected as part of M&E and subsequent analysis was able to guide and adjust project planning and implementation. Overall there was a significant improvement in strategic implementation with increased focus on the ‘grand design’ and on matching and linking activities. Issues identified during ISMs were continuously addressed and delivery of the various components improved throughout, which was in part reflected in increased disbursement following the MTR and restructuring of the project. 93. The financial management dashboard of the MIS did work well enough to allow for a timely reallocation of funds at midterm, based on cost savings, and prompted IAARD to decentralize SMARTD implementation. The MIS can be retrieved from the pulldown menu on the SMARTD website (fully embedded in the IAARD institutional website and accessible here: http://www.litbang.pertanian.go.id/smartd/), which has been monitoring the number and nationality of its visitors and has had formatting and software problems ever since its creation that were never fixed. IAARD centers ended up submitting information and data manually or through e-mail rather than uploading it to the site, which was widely considered as too slow to respond and unreliable. 94. As a contributor to knowledge management (KM), the M&E system performed below expectations and most KM efforts on information and data were sourced directly from contributing research collaborators. SMARTD has been treating KM much in the same fashion as it has been treating technology development/transfer, namely, as a unidirectional transfer from one expert to another. A KM strategy, fed with quantitative and qualitative information and data from the M&E system, would have helped prepare standardized and stylized products for a nonscientific lay audience, such as success stories and infographics on innovations. Justification of Overall Rating of Quality of M&E 95. Limited capacity development on M&E had been offered to project staff before the MTR, which, coupled with a lack of clear responsibilities within the PMU (even though a recommendation to this effect had been made on several occasions, no M&E officer had been recruited until project closing), led to the poor performance of the M&E system. Although reports tended to have been submitted on time, the quality of their contents was not always up to expectations. By way of illustration, data inconsistencies required a major effort of ‘reconciliation’ during the ICR mission. 96. On a positive note, the World Bank’s insistence on the development of a project-level M&E capability did leave some positive impact on organizational development within IAARD in general, and on Page 32 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) its work culture, in particular. The importance of measuring and reporting both quantitative and qualitative results has been recognized by a larger group of professionals within IAARD, beyond the PMU. Especially the many discussions on how to measure technology dissemination and adoption have contributed toward an enhanced results-orientation and accountability, as well as an enhanced awareness of why ‘value for money’ should be one of the main preoccupations of research management. 97. The overall rating of the quality of M&E under SMARTD is ‘Modest’. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE Environmental and Social Compliance 98. The project is a Category B – partial assessment project which triggered OP 4.01 on Environmental Assessment. The project’s environmental and social risk rating was assessed between Moderate and Low depending on the scale of construction and upgrading works. Impacts were mostly related to construction works and operation of facilities, particularly laboratories, with occupational health and safety (OHS) and public health risks (Component B). No fatalities or major incidents were identified or reported during project implementation. An Environmental Code of Practice (ECOP) was prepared during project preparation. Supplemental OHS and laboratory safety guidelines were subsequently prepared during project implementation, along with necessary capacity building measures (that is, training, supervision, and coaching) to contractors, supervision consultants, and IAARD staff. 99. Environment. The project was rated as a Category B project due to the scale of potential environmental impacts and triggered the Bank's Operational Policy (OP) 4.01 on Environmental Assessment. Potential environment impacts would have come from the improvement of research infrastructure (civil works for new laboratories, rehabilitation and/or the upgrading of laboratories, and field stations). Laboratories and field stations were to meet world standards (ISO 17025). Environmental impacts associated directly with the project were insignificant, site specific, and noncumulative due to the scale and scope of the investments. Investments were below the Indonesian environmental permission threshold and, therefore, no environmental permits were required under the project. The project did not require a stand-alone environmental impact assessment or Environmental Management Plan (EMP). The project applied environmental codes of practice (ECOPs) for construction activities, based on World Bank standards. During appraisal, the environmental safeguards team revised the ECOPs and provided specific recommendations to improve the existing version. In order to meet disclosure requirements, the ECOPs were disclosed on IAARD's website on January 30, 2012. 100. Social. No land acquisition was required for the project/subproject activities. Construction and rehabilitation were implemented within the existing facility premises that belong to MoA. Land due diligence to assess the ‘clean and clear’ status of the proposed location, followed by submission of copies of land certificates was required before proposal selection under Component B. No overlapping claims or conflicts were reported. However, land documentation was not systematically archived at the beginning of project implementation, resulting in incomplete understanding of the overall land due diligence process. Labor influx risks were assessed as low based on a supplemental diagnostic labor risk assessment (archived May 10, 2017). Page 33 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) 101. Key issues. The main issues concerning safeguards were related to project management, summarized as follows: (a) Lack of dedicated personnel to oversee safeguards implementation and supervision budget during the initial years of project implementation, resulting in ineffective enforcement of ECOPs for civil works (b) Limited capacity on OHS and laboratory safety aspects among project implementers (Satkers), contractors, and supervision consultants (c) OHS issues typically applicable to small- to medium-scale construction works by local contractors such as poor management of labor risks, construction waste, and public safety (that is, construction site demarcation) (d) Weak contract management to ensure adequate provisions for OHS along with its budget allocation in the Bill of Quantity (e) Limited awareness of laboratory safety requirements and budgetary constraints for their long-term maintenance (f) The project’s Grievance Redress Mechanism relied on a direct communication and formal elevation procedure to supervisors, senior officers, contractors, and/or supervision consultants with limited accessibility to the broader public (g) An additional requirement to assess potential environmental and social implications of research activities (Component C) was only introduced in the research proposal guidelines at a later stage of project implementation and hence suffered from lack of documentation pertaining to its implementation 102. Despite these challenges, there were no major issues reported during implementation. Essential technical support in the form of guidance, mentorship, and supervision was provided to the PMU to improve its safeguards management. Some turnover of World Bank safeguards staff and consultants supporting the project during the initial years of implementation prevented knowledge accumulation and engagement at the subnational level. 103. Compliance. The project demonstrated substantial efforts to strengthen the project safeguards management to address the above issues. These included the following: (a) Establishment of a dedicated safeguards team at the PMU, consisting of staff representatives and safeguards focal points at the Satker level with requisite intermittent support from an OHS consultant (b) Establishment of an OHS taskforce consisting of the administration and management division heads (echelon 2 of IAARD institutions), chaired by the IAARD executive secretary to provide overall OHS oversight (c) Steady budget allocation of safeguards supervision and capacity building (d) Development of operational guidelines for OHS and laboratory safety and delivery of required training/workshops of OHS and laboratory safety requirements. IAARD committed to adopting the laboratory safety guidelines and training as part of the accreditation capacity Page 34 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) building package, which provides assurance for safeguards continuity over the long term beyond the project (e) Inclusion of the ECOP checklist and OHS guideline in bidding documents (f) Inclusion of OHS indicators in the project’s M&E to ensure regular safeguards monitoring and reporting (g) Retrofitting of existing facilities to meet laboratory safety requirements financed by budget from the GoI (h) Systematic improvements in the land record inventory Fiduciary Compliance and Procurement 104. Project implementation has complied with the World Bank’s Procurement Guidelines and the design foresaw delay in the selection of various individual positions and firms in the early stage of project implementation and delay in procurement of civil works in local working units. The Procurement Committee was familiar with the GoI’s procurement regulations but less familiar with the World Bank’s Procurement Guidelines. In the case of selection of consultants, the responsiveness of candidates to calls for proposals and decision making on the selection of responses received also contributed to the delay in the selection process. Delay in technical assistance procurement had an important impact on the development of the more strategic activities in the project such as the development of the strategy for selection of candidates and their placement for postgraduate education, support to the overall strategy of development of infrastructure and facilities and the M&E system for the project, and IAARD’s overall research effectiveness. 105. A major challenge in procurement was the delay in procurement of works due to budget management at IAARD’s head office while the procurement process was managed by the respective local working unit (Satuan Kerja/Satker), combined with the capacity of procurement staff at the local working unit level using World Bank Procurement Guidelines. To improve procurement performance, the following action was taken following the MTR: (a) The budget was decentralized from IAARD’s head office to local working units starting January 2016. (b) The World Bank provided a procurement training program and support during project implementation. (c) The PMU at IAARD head office provided hands-on support to local working units (Satker) during procurement processes. Financial Management 106. The project generally maintained good financial management performance standards during implementation. All financial management covenants were met, including timely submission of audit reports and interim financial reports. The audit reports consistently provided an unqualified opinion on the project’s annual financial reports. The project also followed up on the findings noted in the audit report on time, including the refunding of ineligible expenditures. The internal control system was Page 35 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) sufficient. Adequate supporting documents were generally available to support the reported expenditures as indicated from the World Bank’s supervision mission and audit reports. 107. Early on during implementation, the project experienced slow progress on expenditures. Up until December 2015, three years into project implementation, the disbursement rate stood at only 34 percent of the total loan. Delays in the procurement process, particularly under Component B, were one of the main reasons for the slow progress. Disbursement picked up starting in 2016 and reached 96.4 percent by the project closing date. The decentralization of the budget under Component B to the research centers was one of the main reasons that the project could accelerate its disbursement progress. 108. The MTR revision of output targets led to a cost neutral budget reallocation between components (see table 10). Table 7. Budget Reallocation and Output Target Revision at Midterm Component/Sat- Change in Proposed Change New Targets if Applicable ker/Balai Cost Component A Reduction of Ph.D. and sandwich programs Reduction of 74 M.Sc., 76 Ph.D., and 45 and increase of M.Sc. programs US$2.06 sandwich programs millions Component A Increase in scientific exchange and visiting Reduction of 44 postdoctorates, 378 scientific scientist and reduction of postdoctoral US$0.33 exchange, and 109 visiting scholarships millions scientists Component A Reduction of conference, seminar, and Reduction of 93 seminars, 98 conferences, and workshop US$1.78 88 workshops millions Component A Reduction of research management and Reduction of 235 trainees for research technical training US$0.11 management and 297 for millions technical training Component A Increase targets for the detachering Reduction of 124 young scientists and 15 program US$80,000 mentors participate in the detachering program Component A Reduction of budget allocation to review of Reduction of Target unchanged the recruitment and promotion processes US$200,000 Component C Reduction of budget allocation for priority Reduction of Target unchanged setting, planning, and programming US$500,000 Component C Increased allocation to the competitive Increase of 74 ‘KKP3I’, 315 ‘KKP3N’, 337 funds and the dissemination schemes US$3.78 ‘KKP3SL’ and 159 dissemination million schemes Component C Reduction in budget allocation for Reduction of Unchanged targets commercialization of research products US$800,000 109. Further funding of US$7.03 million or 46.87 percent of the initial budget allocation, was transferred from Component A to Component C, which allowed for additional activities, in particular Subcomponent C2 on Research Collaboration, Assessment, and Dissemination. 110. As shown in table 11, the remaining budget reallocations benefitted Component B. Page 36 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Table 8. Component B, Implementation of Infrastructure-related Procurement Initially Planned Infrastructure Infrastructure Target after the Activities Target MTR Laboratory equipment 12 36 Experimental station equipment 7 14 Experimental station infrastructure 6 36 Construction for laboratory and 14 52 office Total planned 39 138 Total achieved 100% 111. Throughout the project lifetime, 12 ISMs were conducted by the World Bank team and the project consistently received Moderately Satisfactory ratings for its financial management performance and was upgraded to Satisfactory during the last two missions. Some good measures were taken to address financial management aspects after the closing date including the provision of GoI budget to cover scholarship-related expenditures for recipients who had not concluded their studies at the closing date and for the maintenance of facilities/equipment under Component B. With the generally good financial management performance of the project, it is also worth acknowledging that financial management under the project was entirely conducted by government staff without support from external consultants. SMARTD has twice earned an award from the MoF for good financial management. C. BANK PERFORMANCE Quality at Entry 112. The PAD was a well-written document that clearly spelt out why investing in IAARD was timely and forward looking, what this investment would consist of, and how it would be implemented. Making such a strong case was no easy feat, given the size of this institution, the fast-paced middle-income country it serves, and the relative scarcity of information and data on how R&D can translate into economic growth in the agricultural sector. The project was designed to leverage other IAARD funds, with a vision of a ‘sum that is greater than its parts’ because the intended major leap in capacity was expected to position the institution on a higher plane of effectiveness and sophistication. 113. Some of the details on the outputs, outcomes, and results to be achieved by SMARTD and the indicators to be used to measure them, were left to be decided upon later, to be included in the POM. This undoubtedly not only fostered enhanced ownership by the PMU, but also proved to be a difficult task for them, given that their backgrounds as researchers has not provided them with much exposure to the ‘nuts and bolts’ of implementing and reporting on an externally funded project. The main issue dragging down the overall quality at entry rating is the relatively weak Results Framework containing several questionable baseline indicators. Similarly, some of the definitions were not agreed upon and were left open to ambiguity (for example, how to define a ‘researcher’). In sum, quality at entry can be rated ‘Moderately Satisfactory’. Page 37 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Quality of Supervision 114. The World Bank provided regular implementation support on both technical and fiduciary aspects through monthly meetings and 12 ISMs, contributing to flag and helping resolve emerging issues effectively. ISM Aide Memoires have been timely and candid, providing enough detail on action points, with roles and responsibilities clearly spelt out. The World Bank facilitated participation of PMU staff at a procurement training program. Fiduciary issues and financial management have performed up to expectations and ISMs have always led to timely action on jointly preparing and revising disbursement projections. In sum, no major problems arose during the life of the project, except for the quality of M&E, which represented one area of longer-lasting concern. By the seventh ISM in 2016, M&E started to be treated separately during a second leg of the mission, to allow for it to be looked at more closely, together with fiduciary issues. The MTR in 2015 represented an opportunity that was seized successfully, leading to agreement on budget reallocations that have allowed the project to achieve most of its results, while making efficient use of loan resources. Justification of Overall Rating of Bank Performance 115. The World Bank has been proactive throughout project implementation, including seeking to address flaws in the M&E system by hiring an international consultant. The overall rating of World Bank performance is ‘Moderately Satisfactory’. D. RISK TO DEVELOPMENT OUTCOME 116. The main risk to development outcomes resides in the fact that SMARTD was designed to leverage resources from within IAARD and beyond. This meant that for the most part, the results achieved by the project cannot be isolated from the broader results achieved by its host institution. This complicates the design and implementation of an exit strategy specially geared toward the sustainability of SMARTD- funded activities. A ‘sustainability strategy’ is nonetheless included in the Project Completion Report prepared by the PMU. The strategy is divided into immediate and medium- to long-term recommendations, and calls on the GoI to provide funding from the national budget for Master’s or Ph.D. students after the closing date of the project, given that there are 13 M.Sc. and 42 Ph.D. students who have not graduated yet. 117. The GoI also needs to provide funding toward the maintenance costs of infrastructure and equipment. One of the MTR findings was that in operational budgets, especially with top-end research equipment, the routine cost for test materials adds significantly to operation and maintenance (O&M) budgets; the latter is included in the current budget of IAARD. See table 12 for a compilation of some of the issues to be followed up on. 118. It has not been possible to estimate the total costs of O&M of the infrastructure and equipment procured through SMARTD and compare them to current funding levels. O&M budgets are prepared at a decentralized level by each of the IAARD centers; in the case of the Rice Research Center, for example (one of the most important IAARD centers), the total budget allocated to O&M every year corresponds to approximately 1 percent of its total budget (about IDR 200–300 million, out of a total annual budget of about IDR 20 billion). Page 38 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Table 9. ‘Pending Issues’, their Link to Outcomes, and Possible Mitigation Measures Activity Unfinished at SMARTD Contribution to Project Outcomes Possible Mitigation Measures Completion Date In an effort to increase impact (P1) Percentage of farmers, World Bank and other recognition, AIATs should further disaggregated by gender, adopting development partners to intensify their efforts at disseminating IAARD research results highlight the importance of the IAARD research results and disseminated under the project research-extension nexus and technologies continue to support the GoI in this respect The GoI to provide enough funding (P2) Number of technologies World Bank to continue to lobby from the national budget to see 13 developed and/or refined and that for such funding and to continue M.Sc. and 42 Ph.D. students through are disseminated to users to support efforts at raising to graduation IAARD’s nontax revenue through its ongoing projects and programs The GoI to provide enough funding (P2) Number of technologies As above from the national budget for O&M of developed and/or refined and that infrastructure and equipment are disseminated to users IAARD to carry out an optimal use (P2) Number of technologies World Bank to offer technical assessment of newly built and developed and/or refined and that assistance on this if necessary equipped laboratories and for optimal are disseminated to users use and self-funding of field stations IAARD’s strategy plan should draw on (P2) Number of technologies As above the ‘match and link’ method to developed and/or refined and that ensure R&D initiatives are well are disseminated to users aligned with available infrastructure and human resources The accreditation of laboratories and (P2) Number of technologies field stations and the establishment developed and/or refined and that of safeguard management units are disseminated to users should be continued so as to achieve 100 percent coverage The finalized guidelines for priority (P2) Number of technologies setting, research collaboration, and developed and/or refined and that dissemination models have been are disseminated to users distributed and follow-up is needed to ensure that they are used IAARD should further consolidate and (P2) Number of technologies nurture its newly established developed and/or refined and that international research collaboration are disseminated to users networks IAARD needs to continue raising its (P3) Number of articles in profile at national and international internationally peer reviewed levels, by promoting the continued journals on agricultural R&D results publication of its research results and developed by the IAARD system by advertising these results more widely Page 39 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Activity Unfinished at SMARTD Contribution to Project Outcomes Possible Mitigation Measures Completion Date IAARD should ensure lessons learned (P4) Percentage and number of World Bank to support this effort from SMARTD are applied, especially sources of external funding in through future interaction with on M&E and accountability IAARD research budget and IAARD percentage of IAARD's operational budget allocated to research and dissemination V. LESSONS AND RECOMMENDATIONS 119. When PMUs are fully embedded in broader government structures, the importance of crafting robust and relevant M&E systems becomes even more pronounced. The advantages of fully embedding the PMU of SMARTD in IAARD outweigh the drawbacks—especially in the case of an intervention essentially aimed at institutional capacity building. Among the disadvantages of this type of implementation arrangement, one fundamental consideration is that attributing results to SMARTD alone will be much more difficult or impossible, as most activities are fully mainstreamed within the day-to-day work of the government agency hosting the project. This calls for major attention to M&E, with the need for a solid system that reports on project results and that ideally feeds into the government agency’s overall M&E and reporting system. M&E indicators should be realistic and relevant as well as easy to measure and understand. 120. Reliance on government staff rather than external consultants builds ownership and enhances capacities by providing important learning opportunities and a platform for management training and career development. Governments understandably prefer not to use loan funding to hire international or even national technical assistance, unless they are certain that such assistance will be difficult to source from within government institutions. The fact that SMARTD resorted to only one international consultant (paid for by the World Bank and Food and Agriculture Organization of the United Nations [FAO]) as well as few national consultants, is a source of pride for the PMU, and is something that was commended by the Planning and Finance Ministries, where the project is seen as a ‘model for project implementation’. Flexible project implementation and regular dialogue with PMUs, taking time to create rapport and build trust can pay off in often unexpected ways and can allow for cost-effective ‘creative problem-solving’. 121. When projects generate less immediately ‘tangible’ outcomes that take time to develop and whose impact does not become visible in the field until later, the importance of designing a good KM strategy is even greater. The lack of a KM strategy for projects can lead to missed opportunities in terms of influencing key institutions such as the Ministry of National Development Planning and the MoF. This is exacerbated in the case of agricultural research, which often suffers from low visibility among decision makers giving priority to ‘value for money’ and demonstrable immediate results. Although short of an economic and financial analysis, the success stories presented in the GoI Project Completion Report, for example, do show that much thought has been given to showcasing a good sample of technologies covering a range of Indonesia’s many different agroecological settings; they would have made for an even more powerful case if they were guided by a KM strategy and were formatted in a similar way, capturing some of the same types and kinds of information and data, making the success stories easier to compare, learn from, and share more widely. Page 40 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) 122. Behavioral and cultural change is possible even in relatively large public sector institutions which can and do evolve, are difficult to achieve and measure, but are both possible, including in the type of public sector institutions that from the outside may look too large and too entrenched to be influenced successfully through time-bound interventions such as projects. By focusing on transparency and fair competition among researchers and providing an earmarked budget for grant-funded schemes at national and international levels, SMARTD managed to find the right incentive mix that nudged IAARD out of its relative isolation. Once researchers got involved in international research collaboration initiatives, a demand was created that is so strong that senior managers in this institution refer to collaboration as their new leitmotif. SMARTD was an ambitious project that sought to transform IAARD and exert influence on how national agricultural research priorities are set and translated into action. While its impact at the central level has arguably been rather modest, it has been significant at the decentralized level throughout Indonesia’s 33 provinces. The project has demonstrated a different way of doing things among the country’s community of researchers, based on multi-stakeholder collaboration that puts farmers and agri- food system actors in the driver’s seat. This sort of good practice has already become engrained in the daily routines of research and extension staff working in the field, who will continue to follow up on both centrally determined and locally determined agricultural research priorities. To do justice to the project, it is necessary to look beyond expected outputs at the central and formal level and turn attention to how activities are conducted in IAARD’s many decentralized centers. 123. Ambitious objectives in terms of reforming and modernizing bureaucracies can be appropriate but should be based on assessments of what power holders involved in project implementation can and cannot influence and change. A rapid institutional analysis and political economy analysis (PEA) should be carried out during project appraisal to shed light on levels of decision-making authority within government structures. In the case of SMARTD, this would have illustrated the fact that many aspects that the project tried to have an impact on (for example, researchers’ incentive systems and career pathways, choice of crops other than ‘priority crops’, and so on), were beyond the remit of and mandate of IAARD and often even the MoA. Of course, it can and should be expected that policy dialogue would influence some of these ‘givens’; a rapid institutional analysis and PEA at appraisal would add value in getting a more precise picture of how realistic and achievable some of these ambitions are. In the case of large government systems covering all public servants, for example, there should an ex ante agreement with the government on what could be changed, why, and how. 124. Strengthening synergies and functional links between public agricultural research and public agricultural extension is fundamental. If these government services are to remain relevant in Indonesia’s fast-paced economy, they must be able to address market failures by targeting segments of the rural population that are unable or unwilling to pay for private sector services. On the other hand, as food systems evolve, they must also step up to the challenge of competing and collaborating with modern value chain actors in the agri-food industry. The daunting issue of an efficient agricultural research-extension nexus, and of the broader institutional setting within which this type of interface is ideally located, as well as of an enabling policy environment that is conducive to operationalizing and sustaining it over time, is something that many countries continue to struggle with. Against this backdrop, SMARTD managed to creatively and successfully contribute to further bridge the institutional divide between research and extension, by, for example, deciding to open up new (strictly speaking, research- related) opportunities for extension staff to get international exposure by attending seminars and workshops and presenting their work there, allowing them to take stock of good practice and latest developments in approaches to extension (for example, digital agriculture extension methods). Page 41 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) . Page 42 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: PDO indicators Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Clients who have adopted an Number 30.00 50.00 77.00 improved agr. technology promoted by the project 01-Jul-2012 01-Jul-2012 28-Jun-2019 Comments (achievements against targets): The full indicator description is: Percentage of farmers, disaggregated by gender, adopting IAARD research results disseminated under the project. The end value of adoption is based on a study on technology adoption by farmers covering two technology dissemination models promoted with support of the project. The study was carried out by ICATAD. At the restructuring in 2015/16, this indicator became gender disaggregated. The average technology adoption rate for these models was estimated at 77 percent in 2018. Adoption among female farmers was 59 percent versus 79 percent among male farmers. The baseline suggested that only 10% of technology adoption involved female farmers. Page 43 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Technologies demonstrated Number 32.00 72.00 119.00 in the project areas (number) 01-Jul-2012 01-Jul-2012 28-Jun-2019 Comments (achievements against targets): The full indicator description is: Number of technologies developed and/or refined and disseminated to users. The actual achieved value represents the number of technologies that were reported as having been demonstrated during the final year of the project, based on available information. The baseline was underestimated, and the number of technologies produced annually fluctuated between 403 and 234, with 119 being the lowest recorded value. Source: IAARD. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Publications in international Number 9.00 41.00 1422.00 peer reviewed journals 01-Jul-2012 01-Jul-2012 28-Jun-2019 Comments (achievements against targets): The full indicator description is: Number of articles relating to agricultural research and development carried out by IAARD published in internationally peer reviewed journals. The reported number of publications exceeds the target. Both baseline and target were underestimated. The baseline value is now estimated at 468 and the cumulative number of publications has steadily increased to 1,422. Source: MoA Central Library. Page 44 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion External Funding Percentage 3.80 12.50 9.90 01-Jul-2012 01-Jul-2012 28-Jun-2019 Comments (achievements against targets): The full indicator description is: Percentage and number of sources of external funding in IAARD's research and dissemination budget, and percentage of IAARD's operational budget allocated to research and dissemination. The level of external funding has fluctuated between 3.4 and 12.7 percent and stood at 9.9 percent in 2018/19. A policy change that discontinued implementation of small projects with relatively small external funding amounts had an impact on the otherwise positive trend (a drop from 12.7 percent in 2014 to 3.4 percent in 2016). Source: IAARD. A.2 Intermediate Results Indicators Component: Development and Management of Human Resources Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion percentage of IAARD staff Percentage 25.23 33.00 26.94 that are researchers 01-Jul-2012 01-Jul-2012 28-Jun-2019 percentage of IAARD Percentage 44.00 50.00 52.00 research staff that are female 01-Jul-2012 01-Jul-2012 28-Jun-2019 Page 45 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Comments (achievements against targets): The full indicator description is: Percentage of researchers in IAARD staff, disaggregated by gender. The achieved result should be viewed against the background that there has been a general decline in staff numbers since a general public sector hiring freeze in Indonesia that has affected IAARD, especially since 2015. In 2018, 1,660 of the 6,161 employees were researchers. Source: IAARD. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IAARD Researchers with MSc Percentage 54.00 67.00 72.53 and/or PhD 30-Oct-2013 01-Jul-2012 28-Jun-2019 Comments (achievements against targets): The full indicator description is: Percentage of IAARD researchers with master's and doctorate degrees, disaggregated by gender. In 2018, IAARD employed 1,660 researchers, of whom 761 were trained at Master level, and 443 at PhD level. Source: IAARD. Component: Improvement in Research Infrastructure and Facilities Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of laboratories Number 15.00 21.00 36.00 supported under SMARTD accredited nationally and 01-Jul-2012 31-Dec-2015 28-Jun-2019 internationally Page 46 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Comments (achievements against targets): The full indicator description is: Number of laboratories supported under the project having been accredited nationally or internationally. At the time of final reporting, 4 additional laboratories were undergoing accreditation. Source: IAARD. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of Experimental Number 0.00 8.00 41.00 Stations supported under SMARTD accredited 01-Jul-2012 01-Jul-2012 28-Jun-2019 nationally and internationally. Comments (achievements against targets): The full indicator description is: Number of field stations supported under the project having been accredited nationally or internationally. Source: IAARD. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of laboratories Number 0.00 10.00 11.00 supported under SMARTD involved in international 01-Jul-2012 01-Jul-2012 28-Jun-2019 research networking Page 47 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Comments (achievements against targets): The full indicator description is: Number of laboratories supported under the project involved in international research networking Source: IAARD Component: Research Management and Policy Support Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Collaborative Number 181.00 668.00 503.00 research/extension sub- projects under 01-Jul-2012 31-Dec-2017 31-Dec-2017 implementation/completed Comments (achievements against targets): The full indicator description is: Number of national and international research collaborations. No value is available for 2018/19. Source: IAARD Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Patents and Licences Number 29.00 37.00 586.00 01-Jul-2012 01-Jul-2012 28-Jun-2019 Comments (achievements against targets): Page 48 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) The full indicator description is: Number of inventions having been patented or licensed to industry. Intellectual Property Rights (IPR) certificates were newly issued or extended in connection with research activities: 240 licenses, 54 plant varieties, 168 patents, 115 copyrights, and 9 trademarks. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IAARD Annual Program takes Yes/no No Yes Not available into account the results of the program and budget 01-Jul-2012 01-Jul-2012 28-Jun-2019 priority setting developed under SMARTD Comments (achievements against targets): This indicator was not rigorously measured. However, the finalized guidelines for priority setting, research collaboration, and dissemination models were prepared and distributed in 2017/18. The extent to which they were taken into account in the multitude of annual plans by the various institutes was difficult to determine. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Dissemination area Number 1.00 4.00 10.00 01-Jul-2012 01-Jul-2012 31-Jul-2015 Comments (achievements against targets): Page 49 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) The full indicator description is: Number of areas under the IAARD multi-channel technology dissemination program that are priorities consistent with Provincial Technology Commission recommendations. The multi-channel dissemination program underlying this indicator was discontinued in 2015. Source: IAARD-ICATAD Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IAARD Budget devoted to Percentage 26.12 35.00 23.00 research and dissemination 01-Jul-2012 01-Jul-2012 31-Dec-2017 Comments (achievements against targets): This indicator is part of the fourth PDO Indicator on external funding. The proportion of research and dissemination budget against IAARD's total budget has not been measured since 2017 due to difficulties with isolating research and dissemination within the overall budget structure. Page 50 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1 P1. Percentage of farmers, disaggregated by gender, adopting IAARD’s Outcome Indicators research results disseminated under the project Intermediate Results Indicators - Number of national and international research collaborations A total of 503 research collaborations were completed or under implementation. Many of these were region or province specific Key Outputs by Component projects that produced locally relevant technology packages. (linked to the achievement of the Objective/Outcome 1) Overall estimated adoption rates are 77 percent. Adoption among female farmers was 59 percent versus 79 percent among male farmers. Objective/Outcome 2 P2. Number of technologies developed and/or refined and that are Outcome Indicators disseminated to users - Percentage of researchers in IAARD - Percentage of IAARD researchers with Master’s and doctorate degrees Intermediate Results Indicators - Number of laboratories supported under the project having been accredited nationally and internationally - Number of field stations supported under the project having been accredited nationally and internationally IAARD’s capacity to develop and disseminate technology improved Key Outputs by Component over time as a result of physical improvements to infrastructure and (linked to the achievement of the Objective/Outcome 2) facilities, the development of human capacity that generated new knowledge in the process, and funding for demand-driven research. Page 51 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Objective/Outcome 3 P3. Number of articles relating to agricultural research and Outcome Indicators development carried out by IAARD published in internationally peer- reviewed journals - Number of laboratories supported under the project involved in Intermediate Results Indicators international networking - Number of national and international research collaborations The cumulative number of IAARD journal publications has increased significantly, although attribution to SMARTD is not possible. Key Outputs by Component International collaboration funded by the project will have (linked to the achievement of the Objective/Outcome 1) contributed to that result, for which the target was 41 publications. The number of laboratories supported by SMARTD that were involved in international research collaboration was 11. Objective/Outcome 4 P4. Percentage and number of sources of external funding in IAARD Outcome Indicators research and dissemination budget and percentage of IAARD's operational budget allocated to research and dissemination - Number of inventions having been patented or licensed to industry - IAARD annual work programs take into account the results of the Intermediate Results Indicators priority setting planning and programming systems developed under the project External funding levels increased yet fluctuated with a level of 9.9% in 2018 (against a baseline of 3.8%). This increase was partly attributable Key Outputs by Component to a sharp increase in patents and licenses. In general, improved (linked to the achievement of the Objective/Outcome 1) planning would have resulted in increased external funding for research. However, smaller grants were discontinued due to high administrative costs. Page 52 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Paulus Van Hofwegen Task Team Leader(s) Enggar Prasetyaningsih Procurement Specialist(s) Novira Kusdarti Financial Management Specialist Sumaryo Soemardjo Team Member Surhid P. Gautam Social Specialist Panneerselvam Lakshminarayanan Social Specialist Dely P. Gapasin Team Member Juan Martinez Social Specialist Andrew Daniel Sembel Social Specialist Supervision/ICR Jan Joost Nijhoff Task Team Leader(s) Angelia B. Nurwihapsari .S Procurement Specialist(s) I Gusti Ngurah Wijaya Kusuma Financial Management Specialist Then Nina Herawati Team Member Ahsan Ali Procurement Team Ir Mariam Rikhana Team Member Indira Janaki Ekanayake Team Member Krisnan Pitradjaja Isomartana Environmental Specialist Fajar Argo Djati Social Specialist AMY CHUA FANG LIM Team Member Page 53 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY10 11.388 78,899.30 FY11 15.170 139,836.15 FY12 21.357 164,783.08 FY13 4.950 30,950.17 Total 52.87 414,468.70 Supervision/ICR FY13 14.035 56,355.26 FY14 26.519 85,140.54 FY15 24.319 100,387.07 FY16 18.286 77,116.06 FY17 18.132 106,262.47 FY18 13.738 93,322.10 FY19 13.684 105,827.87 FY20 4.688 41,839.87 Total 133.40 666,251.24 Page 54 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) ANNEX 3. PROJECT COST BY COMPONENT Actual at Project Amount at Approval13 Components Closing (US$, Percentage of Approval (US$, millions) millions) Development and Management of Human 40 18 45% Resources Improvement in Research 35 52 149% Infrastructure and Facilities Research Management and 15 22 147% Policy Support Project Management and 10 3 30% Monitoring and Evaluation Total 100 95 95% 13Likely financing of US$6 million by the Government of Australia for complementary technical assistance as foreseen at appraisal is mentioned in the Data Sheet on page 2 and in section III A. This technical assistance did not materialize. Page 55 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) ANNEX 4. EFFICIENCY ANALYSIS 1. The methodology used to assess the efficiency of the project SMARTD needs to be adapted to two specificities of the project. The first specificity is that the causal chain and the time frame for the returns on investment for SMARTD are longer than in most projects. The PDO is improving the IAARD’s institutional capacity and performance in developing and disseminating innovative, demand-driven, and relevant technologies and meeting the needs of producers and agri-food industries. If the project succeeds in meeting this objective, the wider expected impact is the adoption of these technologies by actors in the agricultural sector. This in turn will lead to (a) reduced production costs; (b) increases in production; (c) increases in productivity; and/or (d) increases in the value added of products, depending on the technology package and sector. In addition, the process from research to adoption is likely to take at least three years for a given technology, considering the time necessary to develop, disseminate, and mainstream a technology. 2. A second specificity is that it is very difficult to isolate the returns of SMARTD activities. Even within the IAARD’s mandate, it is not possible to isolate SMARTD investments from IAARD investments. For instance, while SMARTD financed infrastructure, research programs, and training, it did not finance the salaries of researchers. Moreover, depending on the technology developed, SMARTD financed equipment, research grants, and/or dissemination grants, but not necessarily all of these. In addition, even if it was possible to attribute the development and dissemination of specific technologies through IAARD to SMARTD, the successful mainstreaming of these technologies relies on processes that are beyond IAARD and SMARTD’s mandates, in particular, adequate extension services. 3. No economic and financial analysis of the project was conducted at design and during the MTR. The analysis at design cited a study conducted by ACIAR14 in 2011, which “examined the extent to which agricultural research within Indonesia contributes to the enhancement of productivity growth, while allowing for other possible determinants of agricultural productivity growth, including international agricultural research, infrastructure investments, extension, weather changes and epidemics.” The study estimates an annual real rate of return of 13 percent for investments in agricultural research in Indonesia. The project design also notes that Indonesia was underinvesting in agricultural research at the time of design. Hence, this ICR is the first attempt to quantify the benefits of the SMARTD project. 4. The economic and financial analysis uses a two-pronged approach. In the first stage, project- specific evidence of economic and financial impacts are assessed based on a subset of SMARTD investments and their expected outcomes. Because the time frame and data are insufficient to assess the extent to which technologies are adopted, a break-even analysis is used to determine how many units of production need to adopt the new technology to recover the investment costs. At the macro level, the program’s economic benefits are estimated based on the empirical relationship between investment in agricultural research and TFP in Indonesia identified by the ACIAR study. 5. The data used for this analysis come primarily from research publications from the different 14ACIAR (Australian Centre for International Agricultural Research and ANU (Australian National University). 2011. “The Effect of Research on Agricultural Productivity in Indonesia.” AGB/2010/018, prepared by Professor Peter Warr. Page 56 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) IAARD institutes, in particular from the AIATs. These AIATs provide a link between the research centers developing the technologies and the extension workers who will mainstream some of these technologies. They assess the technologies in the local context, choose the most appropriate technologies based on local needs and disseminate the technologies locally. Hence, they provide the key link between research and adoption. For some technologies, publications included cost-benefit analysis of new technologies from the perspective of the producers expected to adopt the technology. Additional data come from the presentations delivered in the context of the field visits for the ICR that took place in October 2019, the IAARD statistics book, and the ACIAR study. A. Technology Level Impact 6. In the first stage of the efficiency analysis, returns on specific research outputs are analyzed. Research outputs, also referred to as technologies in this annex, correspond to new technologies developed by the different IAARD research centers. IAARD developed a wide range of technologies, ranging from new seed varieties to the development of products based on nanotechnology, such as bio- nano silica. The AIATs will typically disseminate a package of different technologies. The technologies modelled for this analysis were constrained by available data but nonetheless represent different subsectors and types of technologies. Table 4.1. Models Model Sub-sector Technology Rice PATBO super Food crops, lowland rainfed rice Technologies for rainfed lowland rice fields JARWO Super Food crops, lowland rice A package of technologies for rice TSS shallots Horticulture, shallots Shallot seeds Garlic seeds Horticulture, garlic Garlic seeds Cow feeding Livestock, cows Improved cow feeding integrating horticulture crops Sugar cane Estate crops, sugar cane Improved seeds Bio-nano Silica Post-harvest technologies, rice Development of bio-nano silica from rice husks Financial Analysis 7. A traditional financial analysis would not be adequate for these models because the underlying investment, the cost of research, and the cost of technology dissemination are public investments that are not borne by producers. 8. Regardless, the question of whether adopting a new technology is financially attractive for producers remains relevant, because it is a precondition for adoption and for generating benefits from the research. Thus, the producer margin for the new technology must be greater than the producer margin with the old technology. Table 4.2.4.2 summarizes the yearly margin with the status quo technology and the yearly margin with the improved technology. 9. Two incremental margins are presented. The first (Column A, Table 4.2) comes from the AIATs’ publications and is based on optimal conditions for implementation, including access to the full technology package and technical assistance. The second (Column B) corresponds to 70 percent of that margin, based on the assumption that incremental margins might be lower for producers outside of the assessment and dissemination program context, without access to all inputs and technical assistance. In Page 57 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) the case of cow feeding and bio-nano silica, the data from incremental margin does not come from publications, so it is considered to be the producers’ incremental margin. Table 4.2. Margins per Unit of Production Margin with the Margin with the Incremental margin Incremental Incremental Model Unit of production status quo improved based on research margin for wider margin for wider technology, IDR technology, IDR data, IDR (A) adoption, IDR (B) adoption, USD Rice PATBO super hectare 10,905,500 18,042,500 7,137,000 4,995,900 352.49 JARWO Super hectare 14,280,000 23,571,000 9,291,000 6,503,700 459 TSS shallots hectare 93,787,000 224,860,000 131,073,000 91,751,100 6,474 Garlic seeds hectare 74,144,400 120,843,800 46,699,400 32,689,580 2,306 Cow feeding Cow 2,409,000 170 Sugar cane hectare 21,700,000 31,010,000 9,310,000 6,517,000 460 Bio-nano Silica Litre 70,000 5 10. For rice PATBO Super, Jarwo Super, ‘TSS’ shallots, and garlic seeds, the data comes from publications. In the case of cow feeding, the data comes directly from producers’ experience in the field. The producers noted that their feed costs decreased while productivity remained constant. Hence, the incremental margin corresponds to the decrease in production costs. In the case of sugarcane, the new variety increases the yield in sugarcane and crystalline sugar. Because no data are available on the production costs, the analysis assumes that production costs remain constant. The incremental margin therefore corresponds to the value of the incremental yield. In the case of bio-nano silica, the incremental margin corresponds to the production cost of bio-nano silica compared to the imported alternative. Economic Analysis 11. For the economic analysis, the technology-level analysis assesses the extent to which the technology outputs need to be adopted to break even on the investment costs. 12. The return on SMARTD investments cannot be assessed separately from the return on research spending more generally, because SMARTD was very much embedded in the IAARD system. For each technology, SMARTD will have financed different research costs, but not the full amount of the research costs. SMARTD contributed to buildings and equipment for some research centers, it provided some research grants and, in other cases, grants for dissemination. It also contributed to human capital development by financing trainings for researchers and IAARD staff members. However, there are no cases where SMARTD contributed to the full cost of research, as SMARTD did not finance researchers’ salaries. 13. Investment costs. The break-even analysis considers the full cost of development and dissemination of a technology. The cost of investment is the key variable in this analysis because it is the most difficult to estimate and the results are very sensitive to assumptions. The cost of the development and dissemination of a technology package will vary depending on (a) how many technologies are included in the package, (b) the cost of developing each of these technologies, (c) how many AIATs disseminate the technology, and (d) the cost of dissemination. The following options were considered to estimate the investment costs, using data from IAARD Statistics 2014–2018: • Total cost per copyright granted. Option 1 consists of computing a cost per copyright, corresponding to the full IAARD budget divided by the number of copyrights granted. This results in a relatively high cost of investment, with an average of US$0.84 million per Page 58 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) copyright between 2014 and 2018. This approach seemed overly conservative because it overlooks the fact that some research findings might not result in copyrights and still be disseminated and adopted. • Total cost per technology package. Option 2 consists of computing a cost per technology package, corresponding to the full IAARD budget divided by the number of technology packages reported in the IAARD statistics book. This approach overlooks the fact that one technology package can be found both at the level of research centers and AIATs. Hence, this approach might double count technology packages for technologies developed at the research center and then disseminated in one or more AIATs. This results in a high number of technology packages, and a low cost per package, US$0.26 million per technology on average between 2014 and 2018. • Total cost per technology package, adjusted. Option 3 is the same as Option 2, but it avoids double counting by only using the cost of technology packages at the level of AIATs. This means the cost is the average cost for R&D and dissemination for each technology package disseminated through an AIAT.15 This is the cost used in the analysis. The average cost per disseminated technology package is IDR 11.6 billion, corresponding to US$0.51 million. Table 4.3. Investment Cost per Technology Package Disseminated (IDR billion) IDR, million 2014 2015 2016 2017 2018 TOTAL IAARD BUDGET 1,582,600 1,876,600 1,925,500 1,658,700 2,092,700 Intellectual Property Rights and Certificates 91 77 106 111 105 IAARD technology packages 385 351 354 257 267 Technology packages, AIATs only 222 164 187 137 121 Option 1: Total cost per copyright 17,391 24,371 18,165 14,943 19,930 Option 2: Total cost per technology package 4,111 5,346 5,439 6,454 7,838 Option 3: Total cost per technology package, adjusted 7,129 11,443 10,297 12,107 17,295 14. For each investment, the cost refers to the average cost in the year in which either the research or the dissemination was conducted (the earliest date of the SMARTD grant is used). 15. For information purposes, the R&D cost per copyright was also estimated. It is computed by dividing the budget of research centers16 by the number of copyrights. The cost averages IDR 11.7 billion between 2014 and 2018, corresponding US$0.52 million. This cost is surprisingly similar to the average cost of disseminated technology packages per AIAT per province. It might seem counterintuitive that R&D costs are as high as R&D plus dissemination costs per province. The reasons behind this are at least twofold. First, the average cost of disseminated technology packages reflects for economies of scale in research; a new technology can be developed once and disseminated in many provinces, so the cost of R&D is split. Second, some technologies do not result in copyrights but are nonetheless valuable and disseminated, so using copyrights as a proxy for research technologies is a conservative assumption. 16. Benefits. The benefits from the investment correspond to the incremental producer margin 15 Assuming that the statistics book’s data on technology packages for AIATs refer to dissemination, which is what the AIATs are mandated to do. 16 This was computed by subtracting the AIATs budget from the total IAARD budget. Page 59 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) determined in the financial analysis. Because a lag can be expected from the moment the research is conducted until the technology is more widely disseminated and adopted, the adjusted incremental margin is discounted by four years. An economic discount rate of 8 percent is used on the basis of an annual rate of interest of 7.23 percent for Indonesia 10-year treasury bonds. The results for each model are summarized below. Table 4.4. Incremental Margin per Model, Economic Analysis NPV of Incremental Incremental NPV of incremental incremental Unit of production margin based on margin for wider margin, discounting margin, research data, IDR adoption, IDR at 8%, IDR discounting at 8%, USD Rice PATBO super hectare 7,137,000 4,995,900 3,672,136 259 JARWO Super hectare 9,291,000 6,503,700 4,780,414 337 TSS shallots hectare 131,073,000 91,751,100 67,439,798 4,758 Garlic seeds hectare 46,699,400 32,689,580 24,027,817 1,695 Cow feeding Cow 2,409,000 1,770,687 125 Sugar cane hectare 9,310,000 6,517,000 4,790,190 338 Bio-nano Silica Litre - 70,000 51,452 4 17. Economic prices. The prices used for the economic analysis are the same as the ones used for the financial analysis. In Indonesia, generally, the agricultural sector is not significantly taxed. In terms of value added tax (VAT), animal husbandry products; seeds and seedlings for agricultural crops and plantations; and cattle, poultry, and fish feed are considered strategic goods and are exempted from VAT. Basic commodities including rice, corn, and soybeans are nontaxable. In terms of import duties, they can vary from 0 percent to 30 percent based on the product. In any case, this project covers a wide range of subsectors, focused on both exports and import substitution, and types of technologies. Hence, the impact of price distortions on the project varies from one subsector to another. 18. Results. The analysis shows that the research technology packages assessed are likely to bring significant returns. For the models assessed, the adoptions needed per dissemination are rather low. For instance, in the case of rice PATBO Super, an adoption on 2,804 ha would suffice to break even from the cost of R&D and dissemination in one province. If this package was disseminated in two provinces, the total break-even adoption level would be about 5,600 ha in both provinces. In the case of Jarwo Super, the analysis considers that the technology package was disseminated in seven provinces, so the investment costs are higher, about US$3.5 million, and the break-even adoption is about 10,500 ha. Rice is cultivated on over 15,000,000 ha in Indonesia17 and about 80 percent of these lands are lowland rice fields. The adoptions needed for breaking even for Jarwo Super and PATBO super are therefore very small. For sugarcane, the total area cultivated was 430,112 ha.18 17 FAO stats 2017. 18 FAO stats 2017. Page 60 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) Table 4.5. Break-Even Analysis NPV of incremental Number of Cost of Research Cost of Investment, NPV of additional Adoption needed Unit of production margin, Provinces Investment, IDR USD margin, IDR for breaking-even discounting at 8%, USD Rice PATBO super hectare 1 10,296,791,444 726,508 3,672,136 259 2,804 JARWO Super hectare 7 49,901,801,802 3,520,906 4,780,414 337 10,439 TSS shallots hectare 1 7,128,828,829 502,987 67,439,798 4,758 106 Garlic seeds hectare 1 10,296,791,444 726,508 24,027,817 1,695 429 Cow feeding Cow 1 11,442,682,927 807,358 1,770,687 125 6,462 Sugar cane hectare 1 10,296,791,444 726,508 4,790,190 338 2,150 Bio-nano Silica Litre 1 7,128,828,829 502,987 51,452 4 138,553 19. The break-even adoption scale is particularly low for the technologies that result in a significantly higher producer margin, in particular, the production of TSS shallots and of garlic seeds, where an adoption on 106 ha and 429 ha per province, respectively, would suffice to break even. The adoption needed is much higher for cow feeding, where the incremental producer margin is lower. However, this technology is already both financially attractive and easy to adopt for producers, because it is a cost- reducing technology. 20. The cost of production and commercialization of the new technologies is also included indirectly in the analysis. Indeed, when the producer buys an input, the input price covers the cost of producing and commercializing the input for the supplier, as well as the profit margin of the supplier. Therefore, only considering the producer’s incremental margin is a conservative assumption, because it does not consider the incremental margins of other actors in the value chain. 21. Sensitivity analysis. As a sensitivity analysis, the results were computed using the alternative investment costs. As above, option 1 corresponds to using the total cost per copyright granted (the highest investment cost); option 2 corresponds to using the total cost per technology package (the lowest investment cost); and option 3 corresponds to the adjusted cost per technology package, the base case. Table 4.6. presents the results. Table 4.6. Sensitivity Analysis Cost of Research Adoption needed Cost of Research Adoption needed Cost of Research Adoption needed Unit of production Investment, option for breaking-even, Investment, IDR, for breaking-even, Investment, IDR, for breaking-even, 1 option 1 option 2 option 2 total option 3 option 3 Rice PATBO super hectare 18,165,094,340 4,947 5,439,265,537 1,481 10,296,791,444 2,804 JARWO Super hectare 121,738,461,538 25,466 28,774,545,455 6,019 49,901,801,802 10,439 TSS shallots hectare 17,391,208,791 258 4,110,649,351 61 7,128,828,829 106 Garlic seeds hectare 18,165,094,340 756 5,439,265,537 226 10,296,791,444 429 Cow feeding Cow 24,371,428,571 13,764 5,346,438,746 3,019 11,442,682,927 6,462 Sugar cane hectare 18,165,094,340 3,792 5,439,265,537 1,136 10,296,791,444 2,150 Bio-nano Silica Litre 17,391,208,791 338,008 4,110,649,351 79,893 7,128,828,829 138,553 22. There are several limitations to the methodology and results described above. The main limitation is that the cost of the research is an estimation and the results of the analysis are quite sensitive to the assumptions. Using an average cost of investment does not capture the fact that the cost of R&D is likely to vary significantly from one technology to another. In addition, while the methodology captures the Page 61 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) economies of scale on average,19 it cannot prove the hypothesis that research technologies disseminated in many provinces are more cost-effective than research technologies disseminated only in one province. Finally, the estimates of the investment costs are based on the IAARD budget and do not include the cost of extension workers, which are not financed by IAARD. These extension workers are still key to the agricultural research system, as they are responsible for ensuring the wider adoption of different technology packages. On the other hand, the IAARD budget in itself also finances activities that go beyond research, such as the production of certain seeds and conducting quality testing. The cost of technology development and dissemination are therefore overestimated, because it is based on the full IAARD budget. 23. A second limitation is that all adoption is assumed to take place four years after the investment cost, while in practice, adoption might be more phased. However, since the adoption needed for breaking even is quite low, this level of adoption is likely to be achieved during the dissemination phase or shortly after. 24. An important limitation to the methodology is the bias in the availability of data and the subsequent choice of models. Indeed, the technologies modelled here are subject to a ‘publication bias’. In other words, while the work of IAARD bodies includes both successful research endeavors and unsuccessful ones, successful research endeavors are more likely to result in a publication. Hence, the models here are likely to represent the more successful research findings and disseminations. In addition to the publication bias, there is also a geographical bias; the models overwhelmingly come from research disseminated by the West Java AIAT, visited in the context of the field mission for the project ICR. Nonetheless, some of the models also present research disseminated in other provinces. For instance, the Jarwo Super technology package was disseminated in seven provinces and bio-nano sillica has been applied in several provinces including Bali, Sumatra, and Lampung. B. Macro-level Impact 25. A macro-level assessment is appropriate for the SMARTD program due to the clear links between investment in agricultural research, poverty alleviation, and economic growth. Innovation in agricultural technologies and the dissemination of improved technologies lead to increased agricultural productivity, which is a key driver of poverty reduction in Indonesia. Sustaining agricultural growth is important for maintaining Indonesia’s food security and improving the living standards of the majority of poor people residing in rural areas and directly involved in agricultural production. This occurs by raising incomes within the agricultural sector and through reducing the price of food available to consumers, both of which are important determinants of the overall level of poverty incidence in the country. Agricultural research promotes technical improvements, which lead to a growth in TFP. This link is supported by the economic literature, which shows very high past rates of return on government investments in R&D (Alston and Pardey 2000; ACIAR 2011). 19If IAARD research centers developed 50 technologies in 2010 and AIATs disseminated 200 technology packages, each technology was disseminated four times on average. Therefore, the average cost of dissemination includes only one-fourth of the total R&D cost. Page 62 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) 26. A study carried out by the ACIAR in 2011 sought to quantify the impact of research on agricultural productivity in Indonesia. The study examined the extent to which agricultural research within Indonesia contributes to the enhancement of productivity growth and attempted to explain determinants of TFP growth, in particular the contribution of agricultural research. The study provides a statistical analysis of the relationship between government expenditure in agricultural research, expressed in constant prices, and the level of TFP in Indonesian agriculture (1974–2006). The results showed a significant effect of expenditure on agricultural research on TFP in Indonesian agricultural production. The analysis converts the increase in TFP to value terms, using the level of value added in agriculture in 2006. The results show that a 1 percent increase in agricultural research leads to an increase in TFP of 0.02 percent in the short run. In the long run, the corresponding increase is 0.08 percent. The resulting estimated annual real rate of return from a marginal increase in investment in Indonesian agricultural research is 13 percent. 27. Based on the assumption that the results from the 2011 study remain valid, a 13 percent return on the SMARTD investments would correspond to an NPV of US$12.2 million (IDR 173 billion), corresponding to an ERR of 11 percent. The ERR is 11 percent and not 13 percent because Component D is not assumed to generate a 13 percent return, while it is included in the project costs. If all project costs, including Component D, are assumed to generate a 13 percent return, the NPV reaches US$21.8 million (IDR 308 billion). It is not clear whether the original study computing the 13 percent return considered overhead costs for IAARD as investments. Both NPVs are computed using a discount rate of 8 percent, as above. 28. The robustness of the methodology rests on the assumption that there have been no significant changes to the relationship between investment in agricultural research and TFP since the ACIAR study. In other words, it assumes that neither the research intensity nor the efficiency of research have changed significantly. To the extent that SMARTD succeeded in increasing the efficiency of research by focusing research more on adoption and demand-driven technologies, the return might be higher than 13 percent. C. Other Efficiency Considerations 29. Beyond the analysis of the return on investment of SMARTD, some broader conclusions can be made on the project efficiency. 30. Costs for trainings. The project provided some data on the trainings financed as part of one of the two ‘soft’ components. The data provided is incomplete and only corresponds to US$4.1 million, about one fourth of Component A expenditures. Nonetheless, it provides an initial idea of activities under this component. The average budget per student was highest for Ph.D.s, with an average of IDR 1,104 million (about US$78,000). For Master’s degrees, the average cost per student was IDR 486 million (a bout US$34,000). The cheapest forms of trainings were workshops and seminars, which respectively cost IDR 32 million and IDR 41 million on average per participant. There are no major gender variations in the cost per student, with the exception of Master’s degrees, where the cost per student was IDR 384 million for women and IDR 548 million for men (the average budget for scientific exchanges also varies a lot for men and women, with IDR 220 million for women versus IDR 45 million for men; however, this seems to be due to an outlier). 31. Revenues from licenses and royalties. Beyond the analysis of the return on investment of SMARTD, some broader conclusions can be made on the project efficiency. While a PDO indicator Page 63 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) considers the external funding in IAARD research budget, it is interesting to also consider the revenue from licenses and royalties more specifically. Revenue from royalties has increased significantly over the project lifetime, from IDR 1.2 billion in 2013 to IDR 5.1 billion in 2018. Similarly, the IAARD nontax revenue has increased from IDR 12.8 billion in 2014 (US$0.90 million) to IDR 31.6 billion in 2018 (US$2.23 million). This shows that the organization is increasingly successful at developing technologies that are ultimately licensed and commercialized and at directly commercializing services. 32. Implementation considerations. Finally, it is important to note that efficiency was affected by the initial slow start of activities. Indeed, by December 2015, the disbursement rate only reached 34 percent of the total loan, because of slow procurement processes and the project was extended, partly also to allow for time for students enrolled in Ph.D.s to finish their studies. In the end, the slow start led to reforms in the procurement system of the institution, which was decentralized to the different research agencies in January 2016, thereby increasing the efficiency of procurement throughout IAARD. D. Rating 33. The low adoption rates required for breakeven of investments on technologies generated under the project and the broader 13 percent ERR based on empirical approaches to computation of ERR for research type projects demonstrate the viability of the project. Although the implementation period was extended by 21 months, the realization of almost all of the EOP targets and the dramatic increase in IAARD royalties and nontax revenues are additional indicators of efficient project implementation. The efficiency of SMARTD is therefore rated ‘Substantial’. Page 64 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS 1. Senior staff at IAARD who were involved in the implementation of SMARTD provided the following feedback. 2. The ICR is written in an objective manner, covering the achievements, weaknesses, and limitations of SMARTD implementation. A number of detailed comments on the ICR are as follows: (a) Please ensure that the actual disbursed amounts are reflected on the Data Sheet, also reflecting the recent cancelation of the unused loan amount. (b) Figure 1 - Theory of Change: the links between outputs and intermediate outcomes are not quite clear. The consolidated outputs of all components should contribute to all intermediate outcomes, requiring a clarification. (c) Regarding the budget allocation to each component (page 9), a clarification is required to indicate the initial allocation versus the allocation after restructuring; (d) Outcome assessment of relevance of PDO and rating. The reference in paragraph 1 to Indonesia’s performance in the ‘Global Competitiveness Index’ from 103 to 54 requires a caveat that this achievement is the result of many factors. (e) Regarding the observed high turnover of staff at the SMARTD PMU, it should be noted that IAARD management considered that PMU positions provided a valuable management learning experience. Some of the former PMU staff have been promoted to directors at the Echelon 2 level, and some even at the Echelon 3 level. Another reason for the turnover of PMU staff was retirement. Lessons Learned (a) Key indicators in the Results Framework. IAARD experienced difficulties in understanding some key indicators and data to be collected. IAARD also found some inconsistencies between the indicators in the Loan Agreement and indicators in the PAD. (b) Utilization of M&E. Despite the lack of M&E performance, IAARD/PMU used the results of M&E to improve the strategy of SMARTD implementation. For example, the change from a centralized project budget to a decentralized budget to improve procurement of infrastructure and equipment was informed by information collected by the M&E system. (c) Capacity of PMU staff. It is important to consider the previous experience of staff appointed as a key staff of a PMU. For example, for M&E—no appointed staff responsible for M&E had actual experience in M&E, which partly explains the weak performance of the M&E system of SMARTD. On the other hand, staff appointed to coordinate research collaboration had a long experience in managing research collaboration activities of IAARD. Page 65 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) (d) Coordination and collaboration with other key stakeholders. IAARD recognizes the importance of good coordination with key partners such as the MoF, Bappenas, and the Secretary General’s office of the MoA. This helped the PMU in resolving various challenges quickly during the life of the project. Especially support and assistance by the MoF to enable disbursement are very much appreciated by IAARD. (e) KM. It is felt by IAARD that there was no clear strategy on KM within SMARTD. It is suggested that for future projects, a KM strategy and implementation plan should be included in the design. (f) SMARTD management model. The MoF and IAARD have noted that the SMARTD management model could be considered as a model for future projects. Without technical assistance from outside, the Government is able to manage a project well, provided the host institution and appointed staff have a high commitment to dedicate their time and knowledge to the project. (g) It is suggested that the outcomes of the SMARTD project will lead to a future engagement between the MoA and the World Bank. Page 66 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) ANNEX 6. SUPPORTING DOCUMENTS 6.1 Profile of IAARD researchers (Dec. 2018) 6.2 List of topics of technical training funded under SMARTD 6.3 Technical Note: Measuring Adoption for PDO Indicator P1, May 2019 6.4 SMARTD Case Study 1: Innovating on marginal dry lands in West Kalimantan 6.5 SMARTD Case Study 2: Harnessing the multi-functionality potential of sorghum 6.6 SMARTD Case Study 3: Unlocking the overlooked potential of sugar palm takes several years 6.7 SMARTD Case Study 4: Reducing post-harvest losses for snake fruit exporters 6.8 SMARTD Case Study 4: Reducing post-harvest losses for snake fruit exporters 6.9 Evolution of IAARD funding allocation to assessment and dissemination 6.1 Profile of IAARD Researchers (December 2018) Researcher Age Education Total Ranks 26–30 31–35 36–40 41–45 46–50 51–57 58–63 64–65 First level - Bachelor 15 122 141 64 19 10 371 researchers (retired 58 Master's 3 19 49 22 12 4 109 years) (Pertama) Ph.D. 14 42 12 22 41 131 Junior Bachelor 30 100 109 76 62 377 researchers (retired 58 Master's 1 2 9 23 32 22 89 years) (Muda) Ph.D. 1 42 33 2 78 Middle Bachelor 8 10 28 100 41 6 193 level- researchers (retired 65 Master's 1 3 20 55 102 24 205 years) (Madya) Ph.D. Senior - Diploma 1 1 researchers Bachelor 5 21 2 28 (retired 70 years) Master's 19 48 13 80 (Utama) Ph.D. 2 7 43 83 16 151 Total 19 188 353 262 251 450 251 39 1,813 Source: IAARD HR Division. Page 67 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) 6.2 List of Topics of Technical Training Funded under SMARTD List of Topics of Technical Training Funded under SMARTD No of Participants 2013 Technical Management Training in Singapore 1 Technical Management Training in Australia 2 Rice Breeding Training at IRRI, Philippines 5 Research Management Training in Australia 21 Research and Technical Management Training in Belgium 2 Research and Technical Management Training in Korea 4 Research and Technical Management Training in Thailand 3 Research and Technical Management Training in Beijing 2 Research and Technical Management Training in the Philippines 1 Research and Technical Management Training in Japan 8 Research and Technical Management Training in Taiwan, China 1 Research and Technical Management Training in USA 1 Training of Digital Soil Mapping in Australia 9 Professional Training in Australia, USA 7 ‘Digital Library’ Training in Thailand 15 Training of ‘Management Development Program’ in the Philippines 23 2014 The Rice Post-Production to Market Training Course 1 Postharvest Technology Short Course 1 Training in Science, Technology and Information 1 Training on ‘International Food Safety’ 2 Operator Training and Maintenance Course TruPunch 1000 2 Training on Quality Seed Production 2 International Vegetable Training 1 Rice Molecular Breeding Course 1 Training on Market Access for Sustainable Development 1 Training on ‘Influenza Division Laboratories: Identification, Characterization and Bioinformatics Analysis of Avian Influenza’ 1 Training on Research Management Program Training Course on Digital Library Management 2015 International Course ‘Food Security in Urbanizing Society’ 4 ‘Linking Research to Local Rural Innovation’ 1 Page 68 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) List of Topics of Technical Training Funded under SMARTD No of Participants Training Course on ‘Smart Use of Fertilizers to Improve Crop Production and Soil 3 Conservation’ International Training Workshop on Postharvest Management Technology for Horticultural 2 Crops International Training Program on Marker Assisted Breeding 6 Training on Agricultural Extension Practices 15 Regional Training on Prevention and Control of Mycotoxin in Food and Feedstuff 2 International Vegetable Training Course: From Harvest to Table 7 International Training Program and Study Tour Entitled Technology Advances in 2 Agricultural Production, Water and Nutrient Management Training on Molecular Sugarcane Disease 1 Fundamentals of SAR for Agriculture and ASAR Workshop 3 Training on ‘Ecological Management of Rodents, Insects and Weeds, in Rice Agro - 5 Ecosystems’ The Second International Workshop for Regulation of Animal Biotechnology 1 2016 Training on The Measurement of and Understanding of Greenhouse Gas Emissions from 2 Agriculture in Thailand Training on Rice: Post-production to Market 1 Training on ‘The Course Horticulture Sector Development for Emerging Markets’ 1 Training on Wagyu Cattle 2 Training on Management of Genetic Resources and Impatiens Genetic Engineering 2 Training on ‘Climate Change and Food Security Nexus’ 2 Training on ‘Assessing Crop Production, Nutrient Management, Climate Risk and 4 Environmental Sustainability with Simulation Models’ Training on ‘International Food Safety Program’ 3 Training on Lost Harvest and Waste Food-Managing the Crop Supply Chain for Increased 1 Food Security International Hybrid Rice Technology Training Course 1 Regional Expert Consultation on Hybrid Rice Development on Asia: Constrains and 2 Opportunities Regional Training Course on Capacity Development of Hybrid Rice in Asia 2 Training on ‘Climate Change’ 3 2017 Training on ‘Project Management, Monitoring and Evaluation’ 3 Training on Molecular Biology in Poultry 1 Training on ‘Project Management, Monitoring and Evaluation’ 7 Page 69 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) List of Topics of Technical Training Funded under SMARTD No of Participants On the Job Training Program for Data Management, Analysis and Publication of Indonesia 4 Global Yield Gap Analysis Training on ‘Public Sector Capacity Building’ 7 International Training Program on Intellectual Property and Genetic Resources 2 Training on ‘Development of Water Harvesting Technique and Irrigation Infrastructure 2 Upgrade’ Training on ‘Integrated Seed Sector Development’ 5 Training on ‘Food Security in an Urbanizing Society’ 1 Technology Transfer, Intellectual Property Management, Technology Commercialization 4 and Product Stewardship Program Training on ‘Delivering Quality Extension Services’ 10 Training on ‘Advances in Feed Evaluation Science’ 1 Training on ‘Scientific Exchange on Bio-Industry’ 7 2018 Training on Measurement of Ideal Nutrient Digestibility in Feedstuffs for Poultry 1 Training on Plant Genetic Resources and Resilient Seed Systems for Sustainable Food 2 Security Training on Proximal Sensing System as Smart Sensor for Rapid Analysis of Soils 2 Training Course on Technology Transfer and ABS for Genetic Resources 6 Training on Strategic Planning & Management Sustainable Agricultural Technology Transfer 7 Page 70 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) 6.3 Technical Note: Measuring Adoption for PDO Indicator P1, May 2019 Results Framework Indicator P1 1. The SMARTD Results Framework Indicator P1, as defined in the SMARTD Loan Agreement, measures the percentage of farmers, disaggregated by gender, adopting IAARD research results disseminated under the project.20 The indicator measures the effectiveness of the dissemination activities that are directly supported by SMARTD. Up to 2015 this was the MP3MI which was replaced by the Jarwo Super Program. 2. The methodology for measuring Indicator P1 was first properly defined after the MTR. At that time, it was decided to base the indicator on the adoption of technologies that had been disseminated through SMARTD-supported MP3MI activities. Measurement was carried out through surveys of farmers who had participated in MP3MI activities. The surveys found that in 2015, 49 percent of participating farmers had adopted technologies from the MP3MI programs. 3. With the ending of SMARTD-funded MP3MI activities in 2015, it was not possible to continue with the previous surveys, and a new approach was developed in 2018 that covered both the MP3MI and the Jarwo Super dissemination programs. The ISM team noted that IAARD’s working units do not have a systematic way of measuring the adoption of their research or of the technologies that they are disseminating. There is therefore, a strong rationale for developing survey tools that can be institutionalized by IAARD. This would improve IAARD’s institutional M&E system and increase the accountability of its working units. A survey tool was developed and implemented in 2018/2019 as part of a broader initiative by ICATAD to better understand the process of technology dissemination and adoption by farmers and to provide data for indicator P1. This technical note provides a brief overview of the methodology and of the relevant results of the survey, ending with a brief discussion. Survey Methodology 4. ICATAD carried out a survey to measure adoption of IAARD technologies by farmers, in and near areas that had previously been sites for Jarwo Super or MP3MI dissemination activities that had been funded by SMARTD. The survey covered seven Jarwo Super sites in seven provinces and eight MP3MI sites in eight provinces. Site selection aimed to include a broad selection of agroecosystems, commodities, and regions. Sites were distributed across the following eight provinces: Sumatra, Lampung, South Kalimantan, Central Java, East Java, West Nusa Tenggara, South Sulawesi, and the Special Region of Yogyakarta. 5. There were 36 to 40 respondents per site, and, in total, 594 farmers were surveyed. Farmers were categorized as one of three types: (a) farmers that had been directly involved in the dissemination activity (co-operators), (b) farmers from the same village that were members of farmer groups that were not part 20 Note that the indicator’s definition in the Loan Agreement (which is the binding definition) is more restrictive than the one presented in the PAD, with the latter not being limited to research results that are disseminated under the project. The change in definition reflected concerns by the PMU and the World Bank Task Team about relevance and program attribution, with the original indicator in the PAD being too broad. Page 71 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) of the program (non-cooperators), and (c) farmers from neighboring villages within the same district (table 6.2). The gender of respondents was recorded to allow disaggregation of results by gender. Table 6.2. Survey Respondents by Type Non-Cooperators Program Cooperators Non-Cooperators in Neighboring Total Villages Jarwo Super 90 136 54 280 MP3MI 111 149 54 314 Overall 201 285 108 594 6. For the purpose of enumerating SMARTD’s Results Framework indicator on adoption, the analysis defined adoption as the continuing use of at least one of the technology components that had previously been disseminated. The survey covered 54 technology components, comprising those which had been included in the previous dissemination program. The Jarwo Super Program had covered 9 technology components related to rice cultivation, while the MP3MI had covered 45 technology components related to the following farming areas: integrated rice-cattle farming, organic farming, cocoa cultivation, mechanization, potato farming, and chicken farming. The collected information also covered the level of technology adoption, which ranged from technology awareness to continuing use of the technology (‘confirmation stage’). SMARTD’s program management decided that the confirmation stage corresponds best to the general definition of adoption that is used in the SMARTD Results Framework. Summary of Relevant Survey Results 7. Of the 594 respondents, 453 had adopted at least one of the technology components that had been disseminated through the previous dissemination program, corresponding to an overall adoption rate of 77 percent. The rate of adoption achieved by the MP3MI program was 81 percent, while that achieved by Jarwo Super activities was 73 percent (table 6.3). Farmers, who had adopted at least one technology, had on average adopted 4.1 and 2.5 technologies for MP3MI and Jarwo Super respectively. Table 6.3. Observed Adoption Rates Program No. of Respondents No. of Adopters Adoption Rate Jarwo Super 280 205 73 MP3MI 314 255 81 Overall 594 453 77 8. Through the survey, it was possible to rank the technology components according to their adoption rate. Technologies with high adoption rates were Inpari 32; the Jarwo 2:1 rice planting system; Inpari 30, a composting process; a high-yield paddy variety; and an organic compost application system. Technologies that had a medium level of adoption were a pest management approach, fertilizer and organic urea production, Jajar Legowo systems, Jarwo transplanting technology, and the use of a combine harvester. 9. Adoption rates varied between the three types of farmers. For the MP3MI program, cooperator farmers had an adoption rate of 89 percent, non-cooperator farmers in the program village had an adoption rate of 72 percent, and non-cooperator farmers in neighboring villages had an adoption rate of Page 72 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) 89 percent. For the Jarwo Super activities, cooperator farmers had an adoption rate of 72 percent, non- cooperator farmers in the program village had an adoption rate of 69 percent, and non-cooperator farmers in neighboring villages had an adoption rate of 83 percent (table 6.4). Table 6.4. Adoption Rates by Respondent Type (%) Cooperators Non-Cooperators Neighboring Village Overall Jarwo Super 72 69 83 73 MP3MI 89 72 89 81 10. The adoption rates differed between male and female farmers with observed adoption rates for women respondents being lower. Only 48, or 8 percent of the 594 respondents, were women. For Jarwo Super activities, the female adoption rate was found to be 60 percent compared to 74 percent for males. For MP3MI, the female adoption rate was 58 percent compared to 84 percent for males. A Z-test indicated that the adoption difference for Jarwo Super was not statistically significant, while for MP3MI the survey results do indicate that adoption rates among male and female farmers differ (table 6.5). Table 6.5. Difference between Male and Female Adoption Rates, Z-test Adoption Adoption Adoption Probability. of No. No. Rate for. Rate Rate for z-value Observing Male Female Females Difference Males (%) Differencea (%) (%) Jarwo 265 15 74 60 14 1.1881 0.1170 Super MP3MI 281 33 84 58 26 3.6016 0.0001 Note: a. The probability refers to the likelihood of observing the recorded difference when there is no difference in the adoption rates. Discussion 11. The ICATAD study provides valuable data on the adoption of technologies that have been developed by IAARD and that have been disseminated with support from the SMARTD program. As such, the overall adoption rate of 77 percent identified by the study provides a measurement for SMARTD’s PDO indicator. However, as there is no compatible baseline data available, it is not possible to draw conclusions concerning SMARTD’s effectiveness in improving adoption rates. SMARTD is believed to contribute to an increased adoption of IAARD’s research results by strengthening research capacity through Components A, B, and C, and by supporting the dissemination programs that were the subject of the ICATAD survey. Most of SMARTD’s impacts, however, are unlikely to become measurable during the project’s duration, as they relate mainly to new research which will take several years before leading to increased adoption. It should be noted that the ICATAD survey is part of an effort to better understand and measure adoption and it is possible that similar surveys will be carried out in the future. It is recommended that at that time, results of the current survey will be compared to the new results and changes are analyzed. 6.4 SMARTD Case Study 1: Innovating on Marginal Dry Lands in West Kalimantan 12. IAARD's innovative crop-livestock integration system has proven to improve farming efficiency Page 73 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) due to the newly enhanced role of animals as a source of meat, traction power, organic fertilizer, biogas, and waste converters. Crops in this system have included corn, rice, and peanuts, commodities for which there are good local and export marketing opportunities across the nearby border from West Kalimantan into Malaysia. The technology package features a strong emphasis on agroecological approaches to dryland farming and minimizes external inputs by catalyzing the complementarity of plants, animals, soil, water, climate, and human beings. 13. The technology package included fertilization, pest control, and post-harvest and handling practices, a new hybrid high-yielding variety of corn created by IAARD in 2017, Bima 19, as well as new composite varieties such as Sukmaraga and Lamuru. Bima 19 seeds were sold at IDR 30,000 per kg, while Lamuru and Sukmaraga seeds were sold at IDR 10,000 per kg. These prices are much more competitive than the hybrid corn seed prices fixed until then by private companies at IDR 80,000 to 100,000 per kg. This quickly acted as an incentive for local corn farmers to continue to grow corn using the seeds introduced by IAARD, thus speeding up technology dissemination. 14. Improvements in the currently available corn breeder seed coupled with the introduction of new kinds of intercropping increased land use efficiency and productivity. The spread of corn seeds of the new varieties from one farmer to the other was due to their excellent performance on farm, their resistance to diseases, and their affordability. SMARTD support prompted the establishment of a collaboration network of IAARD centers, local government, cooperatives, financial institutions, commodity associations, and others. 15. Technology introduced in 2018 focused on corn, dry land rice, and peanuts. A similar approach was applied to peanut seed growers who aimed at responding to domestic and international market demand from Malaysia. An already established seed reimbursement scheme was strengthened: at harvesting time, some of the new high-yielding variety seeds were returned to farmer group members growing the seeds, while the rest could then be distributed to all group members. Freshly harvested peanuts grown from peanut seeds promoted by IAARD were sold at IDR 8,000 to IDR 10,000 per kg. At project closing, R&D activities had started on rice planted in the tidal swamp area in the District of Sambas in West Kalimantan. 6.5 SMARTD Case Study 2: Harnessing the Multi-functionality Potential of Sorghum 16. Sorghum is one of the most versatile crops as a staple food, animal feed, and energy source, while its environmental adaptability allows it to be grown on many different soils including marginal and degraded land. It has higher food fiber, protein, and mineral content than rice. In the eastern part of the island of Flores, which receives only two months of rainfall per year, a traditional sorghum variety called Kuali is well suited to local conditions and can easily yield 3–4 ton per ha without fertilizer. With SMARTD funding, seed production was initiated in six locations, among which Kawalelo village, where the total area under cultivation has reached some 100 ha, making it now look green even during the dry season. Farmers in Kawalelo also grow grain sorghum, also called ‘rice sorghum’, due to its high demand by buyers from Java and Bali, where buyers consider it to be an organic food plant because they know that only organic fertilizers are used to grow it. The project has embarked on obtaining a Geographical Information Certificate for this much appreciated local variety of sorghum. 17. Indonesia’s contemporary quest for national sugar self-sufficiency has been inspiring researchers Page 74 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) to look for new sources of sugar, such as sweet sorghum (sorghum bicolor (l). moench), which is drought and salinity resistant and holds great development potential. Its stem can also be used for crystal sugar, bio-ethanol, animal feeds, and compost, while the grain can be processed to produce intermediate products such as granulate and oat flour, as well as final products such as instant rice and instant porridge oat, instant noodles, and beverage products. Sweet sorghum syrup is used for making bread, pancake, bakery produce and ice cream, and pharmacy products. 18. Project support led to a multidisciplinary approach on sorghum development involving the Indonesian Cereal Research Institute in Maros, South Sulawesi, the AIAT, and Local Agriculture Services (Dinas) of East Nusa Tenggara, and sorghum farmer groups. Successful elements of this IAARD-supported research are the comprehensiveness of the research components, including characterization of physiochemistry of several varieties of local sorghum and its development as staple food, processing technology for sweet sorghum as food and liquid sugar, integrated agribusiness model for sorghum, development of planting equipment and thresher, and machinery and post-harvest handling and processing. Equipment tested consisted of a sorghum thresher which was modified and redesigned from a rice thresher (performing at 99.16 percent threshing efficiency) and semi-mechanical maize and soybean planting tools developed by IAARD's Agricultural Mechanization Institute. 19. By project closing, sorghum cultivation in East Nusa Tenggara had reached a total of almost 500 ha, indicating how diets in East Nusa Tenggara are rapidly integrating this crop as a staple and diminishing the dependency on rice—'sorghum is nutritious, healthy, and profitable’ has become a familiar motto among local communities. Sorghum-related activities were also supported in Manggarai Barat (Komodo Island), Pula di Bajawa, Ende, Sikka, and East Flores, where typically about 60 percent of production goes toward home consumption while the remainder is marketed. Coca-Cola Inc. has also provided assistance in the context of a research collaboration initiative on sorghum post-harvest equipment with Yayasan Pembangunan Ekonomi (Yaspensel) Foundation. Yaspensel farmer partners have received training on the use of sorghum waste for animal fodder and for improving human nutrition—reducing stunting in children—in cooperation with the local health department services. 6.6 SMARTD Case Study 3: Unlocking the Overlooked Potential of Sugar Palm Takes Several Years 20. With productivity levels that are double those of the much better-known sugarcane, palm sugar holds the potential of supplying a much-increased share of Indonesia’s sugar needs. In the Province of Bengkulu on the western coast of Sumatra, sugar palm plants are intercropped with coffee as a source of daily income of the local communities. They can easily be marketed all year long in the form of sugar and palm fiber which are traded all over the island and even exported. 21. Collaborative research funded by SMARTD on the local variety of sugar palm called kapur began in 2016. In 2018, it was registered with the name Smulen ST1 and flagged as a superior national variety for further dissemination throughout the country. The superiority of Smulen ST1 includes high sap production up to 15.4 L per person per day, high sugar content, long tapping period of 4–6 months, low stem height, production starting already after seven years, high potential seed production per tree, human health benefits due to its low glycemic index, and resistance to pests and diseases. 22. With a production of 20 L sap per day after five days farmers can process 100 L of sap into 18 kg of semut sugar using solar dryers. Fetching a price of IDR 16,000 per kg for molded sugar and IDR 25,000 Page 75 of 76 The World Bank Sustainable Management of Agricultural Research and Technology Dissemination (SMARTD) (P117243) per kg for semut sugar, farmers can sell an average of 12 kg per day of molded sugar which provides an average income of IDR 192,000 per day (equivalent to approximately four to five times the daily wage for agricultural labor). 23. The quality of Smulen ST1 palm sugar produced by local farmer groups has met SNI 01-3743-1995 standards. With lobbying from the Bengkulu provincial government, IAARD is now working on the official release of this variety. 6.7 SMARTD Case Study 4: Reducing Post-Harvest Losses for Snake Fruit Exporters 24. One of the international research collaboration initiatives under a scheme referred to as KKP3I sought to improve the packaging technology for snake fruits in the Sleman District of Java. In collaboration with two snake fruit exporters, the local AIAT introduced the use of innovative equipment for grading and wrapping snake fruit in special plastic, considerably lengthening its short shelf life. The activity was also supported by New Zealand, one of the main destination countries of snake fruit exported from Indonesia. The new grading equipment was created by AIAT researchers and has already been awarded an IPR license. Use of this equipment leads to a 25 percent reduction in post-harvest damages. The research project also worked on Good Agriculture Practices certification for snake fruit farmers employing environment-friendly technology. This has been followed up on by the local government, which promotes and supports the GAP program snake fruit farmers producing for the export market. 6.8 Evolution of IAARD Funding Allocation to Assessment and Dissemination 25. The AIAT institutes are the critical link at the research-extension interface, and (as shown in table 6.6) their funding levels have been increasing over the years—although in uneven fashion. Their technology dissemination, and adoption method is, somewhat simplistically, called ‘technology assessment’, which is a process that is for the most part determined on a case-by-case basis according to programmatic priorities and funding levels. In practice, such ‘assessments’ can vary across the spectrum from being highly participatory, iterative, and adaptive (farmer driven), to being basically top-down, uniform, and technocentric (externally driven). In this system, AIAT-based extension staff are not directly responsible for interacting with farmers, which is the prerogative of district- and subdistrict- level extension agents. Monthly meetings are typically held to foster interactive dialogue on technology dissemination and adoption among these different layers. Table 6.6. Evolution of Funding of AIATs (IDR, millions) 2011 2012 2013 2014 2015 2016 2017 2018 368,186 440,404 532,789 481,448 744,412 755,731 701,104 824,437 Source: IAARD statistical yearbooks Page 76 of 76