GPR 57645 INTERNATIONAL ASSESSMENT OF AGRICULTURAL KNOWLEDGE, SCIENCE, AND TECHNOLOGY FOR DEVELOPMENT GLOBAL PROGRAM REVIEW Volume 4 Issue 2 THE WORLD BANK GROUP WORKING FOR A WORLD FREE OF POVERTY The World Bank Group consists of five institutions--the International Bank for Reconstruction and Development (IBRD), the International Finance Corporation (IFC), the International Development Association (IDA), the Multilateral Investment Guarantee Agency (MIGA), and the International Centre for the Settlement of Investment Disputes (ICSID). Its mission is to fight poverty for lasting results and to help people help themselves and their envi- ronment by providing resources, sharing knowledge, building capacity, and forging partnerships in the public and private sectors. THE INDEPENDENT EVALUATION GROUP IMPROVING DEVELOPMENT RESULTS THROUGH EXCELLENCE IN EVALUATION The Independent Evaluation Group (IEG) is an independent, three-part unit within the World Bank Group. IEG-World Bank is charged with evaluating the activities of the IBRD (The World Bank) and IDA, IEG-IFC focuses on assessment of IFC's work toward private sector development, and IEG-MIGA evaluates the contributions of MIGA guarantee projects and services. IEG reports directly to the Bank's Board of Directors through the Director-General, Evaluation. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank Group's work, and to provide accountability in the achievement of its objectives. It also improves Bank Group work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. International Assessment of Agricultural Knowledge, Science and Technology for Development (IAASTD) June 28, 2010 Corporate and Global Evaluations and Methods http://www.globalevaluations.org ii ©2010 Independent Evaluation Group, The World Bank Group 1818 H Street NW Washington DC 20433 Telephone: 202-458-4497 Internet: http://www.globalevaluations.org E-mail: grpp@worldbank.org All rights reserved This volume is a product of the staff of the Independent Evaluation Group (IEG) of the World Bank Group. It is part of an ongoing series that reviews global and regional partnership programs in which the World Bank is engaged as one of the partners. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. IEG does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of IEG concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. IEG encourages the dissemination of its work and permits these reviews to be copied or otherwise transmitted, with appropriate credit given to IEG as the authoring body. Cover image: Harvesting rice fields in a White Thai village in Mai Chau, Hoa Binh province, northern Vietnam. Photo by Tran Thi Hoa, courtesy of the World Bank Photo Library. ISBN-13: 978-1-60244-145-3 ISBN-10: 1-60244-145-6 Printed on Recycled Paper Independent Evaluation Group Communication, Learning, and Strategy (IEGCS) E-mail: grpp@worldbank.org Telephone: 202-458-4497 IEG Mission: Improving Development Results Through Excellence in Evaluation The Independent Evaluation Group (IEG) of the World Bank reviews global and regional partnership programs (GRPPs) in which the Bank is engaged as one partner among many for two main purposes: (a) to provide accountability in the achievement of the program's objectives by providing an independent opinion of the program's effectiveness, and (b) to identify and disseminate lessons learned from the experience of individual GRPPs. The preparation of a global or regional program review (GPR) is contingent on a recently completed evaluation of the program, typically commissioned by the governing body of the program. The first purpose above includes validating the findings of the GRPP evaluation with respect to the effectiveness of the program, and assessing the Bank's performance as a partner in the program. The second purpose includes assessing the independence and quality of the GRPP evaluation itself and drawing implications for the Bank's continued involvement in the program. Assessing the quality of GRPP evaluations is an important aspect of GPRs, since encouraging high quality evaluation methodology and practice more uniformly across Bank-supported GRPPs is one of the reasons why IEG embarked on this new product in 2005. IEG annually reviews a number of GRPPs in which the Bank is a partner. In selecting programs for review, preference is given to those that are innovative, large, or complex; those that are relevant to upcoming sector studies; those for which the Executive Directors or Bank management have requested reviews; and those that are likely to generate important lessons. IEG also aims for a representative distribution of GPRs across sectors in each fiscal year. A GPR is a "review" and not a full-fledged "evaluation." It assesses the independence and quality of the relevant evaluation; provides a second opinion on the effectiveness of the program; assesses the performance of the Bank as a partner in the program; and draws lessons for the Bank's engagement in global and regional programs. The GPR does not formally rate the various attributes of the program. A GPR seeks to add value to the program and to the World Bank beyond what is contained in the external evaluation, while also drawing upon IEG's experience in reviewing a growing number of programs. It reports on key program developments since the evaluation was completed, including the progress in implementing the recommendations of the evaluation. A GPR involves a desk review of key documents, consultations with key stakeholders, and a mission to the program management unit (secretariat) of the program if this is located outside the World Bank or Washington, DC. Key stakeholders include the Bank's representative on the governing body of the program, the Bank's task team leader (if separate from the Bank's representative), the program chair, the head of the secretariat, other program partners (at the governance and implementing levels), and other Bank operational staff involved with the program. The writer of a GPR may also consult with the person(s) who conducted the evaluation of the GRPP. Each GPR is subject to internal IEG peer review, Panel review, and management approval. Once cleared internally, the GPR is reviewed by the responsible Bank department and the secretariat of the program. Comments received are taken into account in finalizing the document, and the formal management response from the program is attached as an annex to the final report. After the document has been distributed to the Bank's Board of Executive Directors, it is disclosed to the public on IEG's external Web site. Abbreviations and Acronyms ACTS African Centre for Technology Studies AKST Agricultural knowledge, science and technology ARD Agriculture and Rural Development Department (of the World Bank) CGIAR Consultative Group for International Agricultural Research CPM Computerized Project Management CSO Civil Society Organization CWANA Central and West Asia and North Africa DGF Development Grant Facility (of the World Bank) ESAP East and South Asia and the Pacific FAO Food and Agriculture Organization GA Global Assessment GEF Global Environment Facility GMO genetically modified organism GPR Global Program Review GR Global Report (of IAASTD) GSDM Global Summary for Decision-Makers IAASTD International Assessment of Agricultural Knowledge, Science and Technology for Development IARC International Agricultural Research Centre ICARDA International Centre of Agricultural Research in Dry Areas IE Independent Evaluation IEG Independent Evaluation Group (of the World Bank) IFAD International Fund for Agricultural Development IICA Inter-American Institute for Cooperation on Agriculture IPCC International Panel on Climate Change LAC Latin America and the Caribbean MA Millennium Ecosystem Assessment M&E Monitoring and Evaluation NAE North America and Europe NGO Non-Governmental Organization OECD Organization for Economic Cooperation and Development PRA Participatory Rural Appraisal SR Synthesis Report SSA Sub-Sahara Africa UNEP United Nations Environment Program UNESCO United Nations Educational, Scientific and Cultural Organization WDR World Development Report (of the World Bank, referring here to the 2008 Agriculture WDR) WTO World Trade Organization Fiscal Year of Program January to December Director-General, Evaluation Mr. Vinod Thomas Director, Independent Evaluation Group (World Bank) Ms. Cheryl Gray Manager, Corporate and Global Evaluations and Methods Unit Mr. Mark Sundberg Global Programs Coordinator Mr. Chris Gerrard Consultant Mr. Ridley Nelson i Contents Preface .................................................................................................................................................. iii IAASTD: Program at a Glance............................................................................................................... v Key Bank Staff Responsible during Period under Review................................................................... vii IAASTD Governance and Management .............................................................................................. vii Principal Ratings* ............................................................................................................................... viii Glossary ................................................................................................................................................. ix Summary ............................................................................................................................................... xi 1. The Program ....................................................................................................................................... 1 Program Origin and Objectives .......................................................................................................... 1 Activities ............................................................................................................................................ 3 Financial Resources ........................................................................................................................... 4 IAASTD Governance Structure ......................................................................................................... 4 2. The Independent Evaluation of IAASTD ........................................................................................... 5 Design and Methodology ................................................................................................................... 5 Independence and Quality .................................................................................................................. 5 Impact of the Evaluation .................................................................................................................... 8 3. The Effectiveness of IAASTD ........................................................................................................... 8 Relevance ........................................................................................................................................... 9 Efficacy ............................................................................................................................................ 11 Outputs......................................................................................................................................... 12 Outcomes ..................................................................................................................................... 18 Unintended Outcomes ................................................................................................................. 25 Efficiency ......................................................................................................................................... 25 Overall Program Rating ................................................................................................................... 27 4. Governance, Monitoring and Evaluation, and Sustainability ........................................................... 28 Governance, Management, and Resources ...................................................................................... 28 Monitoring and Evaluation (M&E) .................................................................................................. 30 Sustainability of Outcomes .............................................................................................................. 31 5. World Bank Performance as a Partner ............................................................................................. 33 The Multiple Roles of the World Bank ............................................................................................ 33 Reputational Risks to the Bank ........................................................................................................ 34 Exit Strategy ..................................................................................................................................... 35 6. Lessons Learned ............................................................................................................................... 35 References ............................................................................................................................................ 41 ii Annex A. Evaluation Framework for Global Program Reviews ......................................................... 43 Annex B. Timeline ............................................................................................................................... 51 Annex C. Terms of Reference for Independent Evaluation ................................................................. 52 Annex D. The External Evaluation Findings and a Summary of the IAASTD Director's Response .. 59 Annex E. A Comparison of IAASTD SDM Findings with the Conclusions Chapter of Agriculture and the Environmenta .................................................................................................................... 64 Annex F. Persons Consulted ................................................................................................................ 68 Annex G. Memorandum on Evaluation of GRPPs Supported by Both the DGF and the GEF ........... 70 Annex H. Response of the World Bank's Agricultural and Rural Development Department to IEG's Global Program Review .......................................................................................................... 72 Boxes Box 1. IAASTD Structure ...................................................................................................................... 5 Box 2. IAASTD's Objectives Were Consistent with the Bank's Rural Sector Strategy ..................... 11 Box 3. Summarized Key Findings from IAASTD Global Summary for Decision Makers................. 13 Tables Table 1. Usefulness of IAASTD Reports for Discussion with Policy Makers, by Work Affiliation of Respondents (in descending order of usefulness ratings) .................................................................... 14 Table 2. Presentations of IAASTD Findings by Participant Type and Forum (up to June 2009) ....... 22 Table 3. IAASTD Expenditures IEG Estimates as of End October 2009 ............................................ 26 Table 4. Overall Program Outcome ..................................................................................................... 27 Table 5. GEF Project Document: Key Performance Indicators ........................................................... 31 iii Preface The International Assessment of Agricultural Knowledge, Science and Technology for Development (IAASTD) was a multidisciplinary and multi-stakeholder assessment by about 400 experts which had four primary goals: (a) to assess the effects of agricultural knowledge, science and technology policy and institutional environments, as well as practices, in the context of sustainable development; (b) to identify where critically important information gaps exist in order to more effectively target research; (c) to make the resulting state of the art, objective, analyses accessible to decision makers at all levels from small producers to those who create international policy; and (d) to further the capacity of developing country nationals and institutions to generate, access, and use agricultural knowledge, science and technology that promote sustainable development. This Global Program Review (GPR) assesses the quality and independence of the 2009 Independent Evaluation (IE) completed under contract with the Agriculture and Rural Development Department of the World Bank; provides a second opinion on the effectiveness of IAASTD's work; assesses the performance of the Bank as a contributor to IAASTD; and draws lessons for the future for global assessments and for the Bank's role in such assessments. This review follows the current IEG evaluative methodology outlined in Annex A. It is titled a review as opposed to an evaluation since it is an assessment with modest resources building on the existing and more substantial IAASTD Independent Evaluation of 2009. A number of partners, participants and other observers were interviewed (see Annex F). These interviews were tailored to the respondent's role and therefore did not follow a formal questionnaire. However, three main lines of questioning were probed in nearly all cases: the quality of the process, the quality of the products, and the impact or expected impact of the assessment. Text analysis of the reports was undertaken and a range of associated literature consulted, including particularly the references used in the IAASTD reports. A reference search was conducted and comparisons were undertaken of references used by IAASTD with other reports covering related areas. The IAASTD was financed primarily by grants from the World Bank's Development Grant Facility (DGF), the Global Environment Facility (GEF), and other donors who contributed to Bank-administered trust funds. IAASTD was chosen for a GPR as part of an agreement between the World Bank's Operational Policies and Country Services (OPCS) Vice-Presidency, the Agriculture and Rural Development Department, the Independent Evaluation Group, the DGF, and the GEF Evaluation Office in order to the eliminate overlapping evaluation requirements for global and regional partnership programs (GRPPs) that are supported by both the DGF and the GEF. OPCS and the GEF Evaluation Office agreed to waive the usual requirement for an end-of-project Implementation Completion Report (ICR) followed by an IEG ICR Review, provided that IEG prepared a Global Program Review based on the 2009 evaluation of the program, which also included ratings for (a) project outcome, (b) sustainability of project outcome, (c) project monitoring and evaluation, and (d) the quality of the external evaluation. (See Annex G.) iv IAASTD also provides an example of a global assessment in a sectoral area of considerable controversy and in a public policy area with large and broad stakeholder involvement. Its process drew upon the earlier experience with two other large assessments -- the Intergovernmental Panel on Climate Change (IPCC) and the Millennium Ecosystem Assessment (MEA). In addition, IAASTD is an example of a program with an agreed output and exit point at the outset, which was not intended to become a sustained knowledge network over time. IEG gratefully acknowledges all those who made time for interviews, in particular the IAASTD Director and a number of participants and observers. v IAASTD: Program at a Glance Start date September 2004. This was the date of the First Plenary meeting at which the Steering Committee recommended proceeding with the assessment. The World Bank and FAO had initiated discussions in August 2002 with a number of stakeholders including NGOs and the private sector to explore their interest in and support for such an assessment of agricultural science and technology. Primary Goals The First Plenary meeting in Nairobi, August 30 to September 3, 2004, adopted four primary goals: (a) Assess the effects of agricultural knowledge, science and technology policy and institutional environments, as well as practices, in the context of sustainable development. (b) Identify where critically important information gaps exist in order to more effectively target research. (c) Make the resulting state of the art, objective, analyses accessible to decision makers at all levels from small producers to those who create international policy. (d) Further the capacity of developing country nationals and institutions to generate, access, and use agricultural knowledge, science and technology that promote sustainable development. Objectives The same First Plenary agreed upon the following three objectives: (1) Bring together the range of stakeholders (consumers, government, NGOs, private sector, producers, scientific community, international agencies) involved in the agricultural sector and rural development to share views, and gain common understanding and vision for the future. (2) Undertake global and sub-global assessments of the role of knowledge, science and technology (KST) as it pertains to agriculture in reducing hunger and poverty, improving rural livelihoods, and health, increasing incomes, and facilitating equitable, environmentally, socially and economically sustainable development. (3) Provide robust information for decision makers on how to ensure that policies, practices and institutional arrangements enable KST to contribute to reducing hunger and poverty, improving rural livelihoods and health, increasing incomes, and facilitating equitable, environmentally, socially and economically sustainable development. The Project Document for the GEF project, which committed US$3.0 million to the program, expressed the objectives somewhat differently in terms of a Development Objective and a Global Environment Objective, respectively: Improve access to agricultural knowledge science and technology that will promote and facilitate sustainable agricultural practices with the aim to improve nutritional security, enhance rural livelihoods while averting environmental degradation, and addressing social and gender inequity and ensuring human health and well-being. Apply the knowledge generated through the IAASTD to provide significant local, national, regional and global environmental benefits including reducing the overall rate of soil, runoff and nutrient losses, diminishing the contamination and eutrophication of freshwaters and soils, reducing the rate of greenhouse gas emissions, and increasing the amount of carbon fixed in agricultural systems. The GEF Project Document then went on to reiterate the above three objectives (1, 2, and 3) as "specific project objectives." vi Activities IAASTD was an assessment undertaken by about 400 world experts selected by the Bureau following nominations by stakeholder groups. This comprised a global assessment and five sub-global assessments. The experts were expected to work in their own capacity and not represent any particular stakeholder group. In addition, individuals, organizations and governments were involved in a peer review process. WBG The World Bank contributed US$1.5 million from the Development Grant Facility and contributions US$1.8 million from the Bank's administrative budget. Other donor 1. Australia, Canada, Crop Life International, the European Commission, France, contributions Ireland, Sweden, Switzerland, United Kingdom, and the United States collectively contributed $3.5 million to Bank-administered trust funds in support of the program. 2. The Global Environment Facility committed US$3.0 million to the program in the form of a Bank-implemented GEF project, as well as $350,000 to the World Bank to prepare and supervise the project. 3. Various agencies, including FAO, Finland, UNDP, UNEP and UNESCO, provided in-kind contributions totaling US$1,372,890. 4. In addition to the above, there was an unquantified amount of support to authors and review editors from their countries and organizations to provide travel, research grants, consultants, and other costs. The Bureau adopted a policy that reserved travel support for non-OECD contributors, so that the figures given here exclude costs for participants from OECD countries. Location The distributed Secretariat was housed at the World Bank, UNEP, and UNESCO, and four Regional Implementing Organizations helped to manage the sub global processes: the African Centre for Technology Studies (ACTS) in Kenya, the Inter- American Institute for Cooperation on Agriculture (IICA) in Costa Rica, the International Centre for Agricultural Research in Dry Areas (ICARDA) in Syria, and WorldFish in Malaysia. Web site www.agassessment.org Governance The assessment was governed by a multi-stakeholder Bureau, comprising two co- chairs, 30 government representatives, 8 representatives from international organizations, and 22 non-government representatives (6 NGOs, 6 commercial private sector, 6 producer, and 4 consumer representatives). Management The assessment was managed by a Director, who also headed a distributed Secretariat which comprised a Coordinator, 2 Senior Technical Specialists, and support staff at the World Bank; 2 staff members at UNESCO; 2 staff members at UNEP; and four Regional Coordinators and support staff at the sub-global host institutes. Latest Howard Elliott et al. Independent Evaluation of IAASTD: Report to the Agriculture program-level and Rural Development Sector Board, The World Bank. June 2009. evaluation vii Key Bank Staff Responsible during Period under Review Position Person Period Global Program Team Leader Pekka Jamsen 2005­2008 Eija Pehu 2008 to present Director, Agriculture and Kevin Cleaver 2002­2007 Rural Development Mark Cackler (Acting) 2007­2008 Department Juergen Voegele 2008 to present Vice President, Sustainable Ian Johnson 1997­2006 Development Network Katherine Sierra 2006 to present Trust Fund Operations Arif Zulfiqar June 1999­2008 Global Programs & Margaret Thalwitz May 2004­2008 Partnerships Global Partnership and Trust Junhui Wu March 2009 to present Fund Operations IAASTD Governance and Management Position Person Period Co-Chairs Hans Herren, President, 2004­2008 Millennium Institute Judi Wakhungu, Executive Director, African Center for Technology Studies Director Robert Watson 2003­2008 viii Principal Ratings* Independent Evaluation** Global Program Review*** Overall Program Outcome Satisfactory Moderately unsatisfactory Sustainability of Program Outcomes Moderately likely Moderately unlikely Design and Implementation of Moderately satisfactory Unsatisfactory Program Monitoring and Evaluation Quality of the Independent ------ Moderately satisfactory Evaluation * These are the project ratings required by Guidelines for GEF Agencies in Conducting Terminal Evaluations, 2008. See immediately below for a summary of the GEF Rating System. ** The Independent Evaluation was commissioned by the Agriculture and Rural Development Sector Board of the World Bank on behalf of the IAASTD governing body (the Bureau), which no longer existed at the time of the evaluation. *** The Global Program Review is an independent review and validation conducted by the Independent Evaluation Group of the World Bank, based upon but not limited to the Independent Evaluation. Summary of the GEF Rating System The achievement of project outcomes and objectives is based on three criteria: a. Relevance. Were the project's outcomes consistent with the focal areas/operational program strategies and country priorities? b. Effectiveness. Are the actual project outcomes commensurate with the original or modified project objectives? c. Efficiency. Was the project cost effective? Was the project the least cost option? Was project implementation delayed, and, if it was, did that affect cost effectiveness? Possible ratings are Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, and Highly Unsatisfactory. The sustainability of project outcomes is the likelihood of continued benefits after the project ends, based on an analysis of four sets of risks that are likely to affect the persistence of project outcomes: (a) financial, (b) sociopolitical, (c) institutional framework and governance risks, and (d) environmental risks. Possible ratings are Likely, Moderately Likely, Moderately Unlikely, and Unlikely. The design and implementation of project monitoring and evaluation is based on the extent to which (a) the project had a sound M&E plan to monitor results and track progress toward achieving project objectives, and (b) the M&E system was in place and facilitated timely tracking of progress toward project objectives. Possible ratings are Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, and Highly Unsatisfactory. The quality of the terminal evaluation report is based on the extent to which (a) the report was consistent, the evidence presented was complete and convincing, and the ratings were well substantiated, and (b) the lessons and recommendations listed in the report are supported by the evidence presented and are relevant to future projects. Possible ratings are Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, and Highly Unsatisfactory. ix Glossary AKST Agricultural Knowledge, Science and Technology is a term encompassing the ways and means used to practice the different types of agricultural activities and including both formal and informal knowledge and technology. (Global Report, p. 560) Assessment An assessment is an evidence-based analysis undertaken for decision makers from a specified authorizing environment. It is problem-driven and identifies gaps in knowledge for implementation of outcomes. It requires judgments that are clearly flagged and provides synthesis to reduce complexity. Its coverage is sufficient to deal with the main range of uncertainty associated with the identified issues. (Drawn from the Global Report, p. 4-5) Efficacy The extent to which the program has achieved, or is expected to achieve, its objectives, taking into account their relative importance. The term is also used as a broader, aggregate measure -- encompassing relevance and efficiency as well -- of the overall outcome of a development intervention or an Assessment. Efficiency The extent to which the program has converted or is expected to convert its resources/inputs (such as funds, expertise, time, etc.) economically into results in order to achieve the maximum possible outputs, outcomes, and impacts with minimum possible inputs. Evaluation The systematic and objective assessment of an ongoing to completed policy, program, or project, its design, implementation, and results. The aim is to determine the relevance and achievement of its objectives, and its developmental effectiveness, efficiency, impact, and sustainability. Genetically A genetically modified organism is an organism whose genetic material has Modified been altered using genetic engineering techniques. These techniques, generally Organism known as recombinant DNA technology, use DNA molecules from different sources, which are combined into one molecule to create a new set of genes. This DNA is then transferred into an organism, giving it modified or novel genes. Governance The structures, functions, processes, and organizational traditions that have been put in place within the context of a program's authorizing environment to ensure that the program is run in such a way that it achieves its objectives in an effective and transparent manner. It is the framework of accountability and responsibility to users, stakeholders and the wider community, within which organizations take decisions, and lead and control their functions, to achieve their objectives. Impacts Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. Independent An evaluation that is carried out by entities and persons free from the control of Evaluation those involved in policy making, management, or implementation of program activities. This entails organizational and behavioral independence, protection from interference, and avoidance of conflicts of interest. Indicator A quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, to reflect the changes connected to an intervention, or to help assess the performance of a development actor. Legitimacy As a criterion for assessing governance and management, the way in which governmental and managerial authority is exercised in relation to those with a legitimate interest in the program -- including shareholders, other stakeholders, implementers, beneficiaries, and the community at large. Management The day-to-day operation of the program within the context of the strategies, policies, processes, and procedures that have been established by the governing body. x Monitoring The continuous assessment of progress achieved during program implementation in order to track compliance with a plan, to identify reasons for noncompliance, and to take necessary actions to improve performance. Monitoring is usually the responsibility of program management and operational staff. Outcomes The achieved or likely short-term and medium-term effects of the outputs of a development intervention. Oversight One of the core functions of the governing body of a program: the performance of the program management unit, appointing key personnel, approving annual budgets and business plans, and overseeing major capital expenditures. Participants For IAASTD, all those outside the Secretariat who participated in the assessment in any way. This included Bureau members, lead authors and authors, reviewers, steering committee members, and government representatives. Partners Stakeholders who are involved in the governance or financing of the program (including the members of the governing, executive, and advisory bodies). In the case of the IAASTD, this included members of the Steering Committee and the Bureau. Public goods Goods which produce benefits that are non-rival (many people can consume, use, or enjoy the good at the same time) and non-excludable (it is difficult to prevent people who do not pay for the good from consuming it). If the benefits of a particular public good accrue across all or many countries, then the good is deemed a global or international public good. Relevance The extent to which the objectives and design of the program are consistent with (a) the current global/regional challenges and concerns in a particular development sector and (b) the needs and priorities of beneficiary countries and groups. Stakeholders The parties who are interested in or affected, either positively or negatively, by the program. Stakeholders are often referred to as "principal" and "other," or "direct" and .indirect." While other or indirect stakeholders -- such as taxpayers in both donor and beneficiary countries, visitors to a beneficiary country, and other indirect beneficiaries -- may have interests as well, these are not ordinarily considered in evaluations unless a principal stakeholder acts as their proxy. Sustainability When the term is applied to the activities of a program, the extent to which the benefits arising from these activities are likely to continue after the activities have been completed. When the term is applied to organizations or programs themselves, the extent to which the organization or program is likely to continue its operational activities over time. Transgenics Transgenic organisms are a subset of GMOs. They are organisms which have inserted DNA that originated in a different species. Some GMOs contain no DNA from other species and are therefore not transgenic but cisgenic. Transparency As a criterion for assessing governance and management, the extent to which a program's decision-making, reporting, and evaluation processes are open and freely available to the general public. This is a metaphorical extension of the meaning used in physical sciences -- a "transparent" objective being one that can be seen through. Source: For evaluation terms, the Sourcebook for Evaluating Global and Regional Partnership Programs: Indicative Principles and Standards. Independent Evaluation Group, World Bank 2007. xi Summary Objectives, Activities, Financial Resources, and Organization 1. In August 2002, the World Bank and the Food and Agriculture Organization (FAO) of the United Nations initiated a global consultative process to determine whether an international assessment of agricultural knowledge, science and technology (AKST) was needed. This was stimulated by earlier discussions at the World Bank with the private sector and non-governmental organizations (NGOs) on the state of scientific understanding of biotechnology and more specifically Genetically Modified Organisms (GMOs). 2. The IAASTD was an assessment by about 400 experts selected by a Bureau following nominations by stakeholder groups and self-nomination. It was composed of a Global Assessment (GA) and five Sub-Global Assessments each producing a separate report: Central and West Asia and North Africa (CWANA); East and South Asia and the Pacific (ESAP); Latin America and the Caribbean (LAC); North America and Europe (NAE); and Sub-Saharan Africa (SSA). There were two main summary documents, the Global Summary for Decision Makers of the Global Report (GSDM) and the Synthesis Report (SR). The process adopted was multidisciplinary and multi-stakeholder calling for the use and integration of information, tools and models from different knowledge paradigms including local and traditional knowledge. The experts were expected to work in their own capacity and not represent any particular stakeholder group. 3. The four primary goals of the IAASTD were (a) to assess the effects of AKST policy and institutional environments, as well as practices, in the context of sustainable development; (b) to identify where critically important information gaps exist in order to more effectively target research; (c) to make the resulting state of the art, objective, analyses accessible to decision makers at all levels from small producers to those who create international policy; and (d) to further the capacity of developing country nationals and institutions to generate, access, and use AKST that promote sustainable development. 4. Below this level, there were three objectives, definable as outputs in evaluation terms: (1) to bring together the range of stakeholders (consumers, government, NGOs, private sector, producers, scientific community, international agencies) involved in the agricultural sector and rural development to share views, and gain common understanding and vision for the future; (2) to undertake global and sub-global assessments of the role of knowledge, science and technology (KST) as it pertains to agriculture in reducing hunger and poverty, improving rural livelihoods, and health, increasing incomes, and facilitating equitable, environmentally, socially and economically sustainable development; and (3) to provide robust information for decision makers on how to ensure that policies, practices and institutional arrangements enable KST to contribute to reducing hunger and poverty, improving rural livelihoods and health, increasing incomes, and facilitating equitable, environmentally, socially and economically sustainable development. 5. In addition to these goals and objectives -- and generally agreed, widely applied and cited in the prefaces to the summary reports -- was the intent to look back 50 years and forward to 2050, often stated in the reports as "looking forward 50 years." xii 6. A governing Bureau included 2 co-chairs, 30 government representatives, 22 non- government representatives, and 8 representatives from international organizations. Within the 22 non-government representatives, there were 6 NGOs, 6 private sector, 6 producer and 4 consumer representatives. A Secretariat, headed by a Director, was responsible for management, operations, coordination and administration. It had three locations, the main office at the World Bank and subsidiary offices at UNEP and UNESCO. A Bank- implemented GEF project committed $3 million to the program, largely for the four Regional Implementing Organizations which helped to manage the sub-global processes: the African Centre for Technology Studies (ACTS) in Kenya, the Inter-American Institute for Cooperation on Agriculture (IICA) in Costa Rica, the International Centre for Agricultural Research in Dry Areas (ICARDA) in Syria, and WorldFish in Malaysia. At each there was a Regional Coordinator and support staff. The Independent Evaluation of IAASTD 7. The quality of the Independent Evaluation (IE) is rated moderately satisfactory. This was commissioned and overseen by the World Bank Agriculture and Rural Development Sector Board. It was independent and had no evaluators on the team with conflicts of interest with respect to IAASTD. The process was generally sound, but there were a number of weaknesses particularly in terms of survey methodology and in terms of which objectives were being evaluated and how each was being rated. However, it contained much valuable material and offered important insights into the process and participants' perceptions. The present review has been somewhat circumspect about the use of the IE survey, particularly the questions on the quality of the report, since the respondents were the authors of the reports they were rating. There were no official comments from IAASTD management on the IE because the IAASTD had closed by the time the evaluation was completed. However, the ex-Director provided comments on the final version saying inter alia that the IE was too negative and too anecdotal, and that it saw the glass as half empty rather than half full. He noted a number of areas of disagreement. The Effectiveness of IAASTD 8. The IAASTD was a useful experience at the nexus of politics and science. However, agricultural technology, with its complexity, diversity and politics, proved to be a bridge too far. The process itself was instructive and there is much useful information in the reports. However, the present review concludes that, for the substantial resources used, the program did not offer sufficient new knowledge or conceptual frameworks for decision makers. It gave conflicting messages, and, for a 50 year time span, it under-estimated the potential of new technologies relative to existing technologies. Attributable impact at the international level has so far been modest at best, and at the national level and below, negligible. This may change but time seems to be running out. RELEVANCE 9. Overall Relevance is rated, on balance, substantial. The issue of agricultural knowledge, science and technology is important for the environmental aspects of global agriculture, notwithstanding the initial much narrower interest of the Bank on GMO policy. xiii Environmental issues in agriculture have been a significant feature of the Bank rural strategy and many rural projects. However, the design of the program exhibited some weaknesses. First, for an assessment that claimed a strong focus on local knowledge, very few farmer or consumer representatives or those closest to them, such as country-level extension specialists, were represented among the authorship participants. Second, there was inadequate design of communication, follow-up, or capacity building. EFFICACY 10. Efficacy was assessed with respect to the three objectives and four goals and, overall, is rated modest. The objectives in the GEF Project Document were not applied in assessing Efficacy since these were formulated one and a half years after the program objectives and not approved at the opening plenary. The present review gives a higher weighting to outcomes -- in terms of the quality of the product and its influence on policy makers -- than it does to process, since the value of process is measured in terms of outcomes or a reasonable expectation of outcomes. However, to the extent that the process added legitimacy to that derived from the quality of the product, process does contribute. 11. At the output level, stakeholders were brought together (objective 1), but there was limited representation of farmers and those closest to them. There was a predominance of northern academics over southern development practitioners, at least in the Bureau, of international NGOs over national and local NGOs, and, out of six representatives of producer groups on the Bureau, only one that had a background to speak for intensive cropping agriculture. While IAASTD had broader representation than major prior studies, it staked a claim to breadth that others did not, particularly in the direction of local knowledge, representation of which was scant. Clearly an assessment was undertaken (objective 2) and reports were issued, but the present review assumes a quality dimension to this objective. The robustness of policy-relevant information for decision makers (objective 3) is questionable. The lack of a conceptual framework, or a country or situational classification for decision makers makes it difficult to apply the largely undifferentiated and variably treated lists of policy options. A focus on a conceptual framework and decision processes as opposed to technological choices -- such as high external input versus organic, or the acceptance or avoidance of GMOs -- could have redirected the conflict away from confrontation towards the means of resolving strategic issues over the longer term. There is an undercurrent against new technology. There are conflicting statements and signals particularly on the more contentious issues. It is difficult to trace evidence since the summary reports only refer very generically to chapters in the global report. Overall, for the expenditures incurred, the findings do not offer enough that is new. 12. At the outcome level, there is a large volume of information in IAASTD (goal a). Chapter 3 particularly offers important findings, well presented within a risk, potential impact, and scale framework. However, the original intent in assessing the effects of "AKST" was to look both to the past and to the future. For two reasons, this was only modestly achieved. Looking back at the past, there was no net assessment of the impact of past intensification on the environment. Looking ahead to the future, the planned modeling scenarios were dropped on the grounds of opacity and locus of control, and no convincing qualitative vision was substituted beyond a general sense of moving away from input xiv intensive agriculture, although some simplified quantitative scenarios survived. The reports lacked a coherent story line from the past into the future. On the main disputed issues of particular interest to the World Bank, GMOs and the potential of organic agriculture, no convergence was achieved. There was limited systematic identification of information gaps (goal b) and wide variation by chapter and section in treatment of that goal. 13. On accessibility (goal c), there was a significant problem with the Web versions being unavailable for six months under an agreement with the publisher -- a crucial period in which to gain brand recognition. The majority of participant respondents in the IE survey found the documents useful for discussion with policy makers although no interviewee for the present review seemed to have had such a discussion. Non-participants surveyed in Africa and a number of other interviewees found other documents such as the 2008 World Development Report (WDR) on agriculture more useful. At the international level there has been some use of IAASTD documents as references, but there have been some notable "cold shoulders" in crucial international forums. No evidence has so far been found to suggest that IAASTD outputs have a reasonable probability of yielding attributable outcomes in due course at the national level. One conclusion might be that time will tell. Another, which seems more plausible at this point, is that time is running out. On capacity development (goal d), this activity was never attempted and the Capacity Development chapter which at least would have pointed a route towards outcomes was dropped. EFFICIENCY 14. The financial cost was approximately US$12 million, but it is estimated that the full resource cost including time and travel costs of all the participants not reimbursed by the program was about US$25 million and possibly more. (About 70 percent of authors and reviewers were not funded by the program and many who were, were not fully reimbursed.) This is high compared to many other information sources accessible to the intended audience. However, these costs might well be justified if the breadth of authorship substantially enhanced the legitimacy of a quality product. However, the limited gains in new messages over other much lower cost sources, the mixed messages and the limited depth in participation down to local practitioner levels weakens that potential legitimacy. Due to the high cost relative to modest incremental benefits in terms of advances in understanding and impact, Efficiency is rated modest. OVERALL PROGRAM OUTCOME 15. Out of the seven objectives and goals, the achievements of five have been modest, one substantial, and one negligible. The highest weighting is given to those objectives and goals closest to impacts -- objective (2) and goals (c) and d), two rated modest and one negligible -- since the value of process is measured in terms of outcomes or reasonable expectation of outcomes. Therefore, overall Efficacy is rated modest, in other words insufficient to warrant a substantial rating. With Relevance rated substantial and Efficacy and Efficiency both rated modest, the overall Outcome rating is moderately unsatisfactory. xv GOVERNANCE 16. With respect to legitimacy, there was broad representation on the Bureau and in authorship from governments, CSOs, academics, international institutions, donors, and other groups, but, as noted, few from farmers, farmer associations, and those in close contact with farmers. 17. On accountability, there was no strong locus of report content management, perhaps almost inevitable in a broad stakeholder, public policy, assessment. In theory, this was the role of the Bureau. However, since there were different views and limited time, the Bureau was not able to firmly manage content, leaving conflicting messages open to different interpretation. Bureau meeting documents show more focus on processes. The substance of the messages was more influenced by authorship profile. Some Review Editors did not sufficiently facilitate resolution of issues or ensure that peer review comments were accommodated. The Director was widely praised for his influence in trying to reach consensus but, by design, he was not controlling content. The overall logistics management of the process appears to have been efficient. 18. Satisfaction with the transparency and fairness of the process varied by participant group. In the Independent Evaluation survey, 46 percent of respondents in the International Agricultural Research Center/Technical Group found the effectiveness of the selection of authors below average compared to only 8 percent of respondents in the NGO group. Governments played a significant role in the nomination of participants. CSOs seem to have strongly influenced the nomination of the authors. Some review respondents also felt this had been quite heavily influenced by World Bank staff consulted by the Secretariat on potential authors. The process was described by one interviewee as "a battle for control of the pen." Some private sector GMO representatives felt strongly that they did not have a fair opportunity. MONITORING AND EVALUATION (M&E) 19. M&E is rated unsatisfactory in the aggregate, and also on both quality of design and quality of implementation. There was no real attempt at M&E although, in a time-bound program, the needs are less and different from an ongoing program. On design, the indicators in the GEF Project Document by the Bank and Secretariat were unsatisfactory, since these reflected a project development objective and a global environment objective that were further down the road beyond outcomes to impacts than was realistic given the timeframe and resources. Coming one and a half years after the First Plenary in Nairobi, these indicators and objectives had not been discussed at that original meeting, and were misleading to those who approved the project. On implementation, no attempt was made to collect data to meet the needs of the more complex of the indicators although some were straightforward, simply calling for completion of a milestone. One donor representative interviewed expressed particular concern about the lack of M&E. xvi SUSTAINABILITY 20. Overall, sustainability is rated moderately unlikely, implying that there are risks for sustainability. Many governments have endorsed the findings, but so far there seems to be no evidence that national research strategies in developing countries are likely to be modified in response to IAASTD findings. Some of those contacted in national public research systems found the material too general. Private companies in the GMO business seem very unlikely to use the findings given their concerns about the process and the findings. However, organic agriculture firms and associations may find areas for support. The CGIAR institutions seem unlikely to significantly redirect their priorities, since there has already been a shift in the directions proposed, in the view of some, taking CGIAR institutions away from their comparative advantage. Some donor projects may be influenced to some degree, but based on discussions with World Bank rural staff there is limited interest partly because the findings are too general and partly because there are other sources of strategic creativity. CSOs are the most likely source of continued investment in the directions proposed by IAASTD, but their funds are limited. 21. An aspect of sustainability relates to environmental counterfactuals. One hypothesis would be that the failure to adopt the IAASTD findings will lead to increased environmental damage such as degradation of land and water through more intensive, high-input agriculture. Another hypothesis would be that widespread adoption of the dominant IAASTD options will lead to reduced productivity gains and more environmental damage through extensification. On balance, the present review finds that the downplaying of the potential for new technology raises questions about the risk to productivity and the potential effects on the environment. World Bank Performance as a Partner 22. The Bank's roles in the IAASTD process were as convener, sponsor and financial contributor, trustee (both fiduciary and executor), host of the Secretariat, program manager, supervisor of the GEF project, and, by default, commissioner of the external evaluation -- the most important roles for this review being convener, host of the Secretariat and supervisor of the GEF project. As a convener, the Bank was effective in getting the interested parties together and it was highly flexible in broadening the range of participating stakeholders. However, it is clear that once the Bank had committed to becoming an IAASTD sponsor and donor, a point of no return was reached at which it was on board for the ride wherever this might lead. Should the Bank have allowed itself to be drawn into a process over which it had no control and into a program for which the objectives had not been set? The present review concludes that it was a high-risk decision and that a more phased and cautious approach would have better managed reputational risk. As host of the Secretariat and supervisor of the GEF project, the Bank's oversight role reveals both challenges and conflicts of interest. First, while, in theory, the Bank supervised the IAASTD Director, the Bank had no control other than through its role in fiduciary management of the funding mechanisms. There was no formal Bank role in the IAASTD process or in monitoring the quality of the product. Second, the task team leader, a Bank designated role, was, for most of the period, a member of the Secretariat. This meant that the Secretariat was, xvii at the Bank task team leader level, supervising itself. However, the Bank ARD Director did sign off on supervision reports. Lessons Learned 23. The issue of public science in a liberal democracy has been an increasing area of study and debate, some of this focused on the divergence between European and North American attitudes to the authority of science. For the increasing role of democratic processes in scientific debate challenges the long-held principles of science of autonomy, impartiality, independence, and objectivity that have traditionally conferred a degree of privilege on science and scientists. As an example of public science, the IAASTD process presented challenges both to science and to the "global citizens" represented at the table. "Is it possible ... for science to be so conceived that it contributes to the give and take of politics? Is it possible to have an educated citizenry sufficiently knowledgeable about science that political discourse can be strengthened?" (Porter and Phillips, 2007, p. 5) The IAASTD experience offers some partial and tentative answers drawn, of course, from one case. 24. In many papers on public science and democracy, there are more questions than answers on how to structure an assessment such as the IAASTD. A few papers, however, offer some principles generally revolving around balance, disclosure and due diligence. These proposed principles offer a framework that partly fits the findings of this review. In particular, the IAASTD experience offers six findings on process that may help to guide future such endeavors: First, it reconfirms that neither physical scientists nor social scientists, either between or within communities, can be expected to come to the same interpretations of peer- reviewed evidence nor can they even be expected to draw from a common literature. Second and related to the first, to ensure balance there is a need for a formal and documented author selection process, perhaps managed to the short-list stage by a disinterested outside party. Third, for balance and due diligence, there is a need for a formal process and professional facilitation for the resolution of disputes and for defining the knowledge gaps. Fourth, for disclosure and due diligence, there is a need for readily traceable reference links between reports and a strong role for reviewers to challenge references and to propose alternatives. Fifth, for disclosure, there is a need for acknowledgment of the political differences encountered, their origins and the implications this may have for any future path towards closure. Sixth, for balance and due diligence, as well as in the interests of efficiency, there needs to be testing of alternative, more phased processes, with potentially challenging issues being parked to one side to be assessed by smaller groups reporting for wider consideration later. In other words, there is a breadth versus depth trade-off evident that warrants more exploration in the direction of efficiency. Not dissimilar findings to the IAASTD have emerged from processes with far fewer participants. xviii 25. The present review concludes that IAASTD was a missed opportunity of considerable cost. The literature and case experience presented in the IAASTD reports, as well as the literature and development experience more widely, point towards a pluralistic strategy for agricultural research and technology development, along with rigorous evaluation to sift success from failure. But, partly due to the way in which the authors were selected and the main reports were translated into the summaries, the overall message which emerged from the IAASTD was a more restrictive, exclusionary message with an undercurrent against new technology, GMOs, and input-intensive agriculture. The IAASTD also failed to provide policy-makers with a conceptual framework that takes into account such factors as the stage of national, sectoral, and zonal development, local physical and social circumstances, and the unfolding of climate change. A more pluralistic strategy would include GMOs and input- intensive approaches alongside organic and low-external-input approaches. It would incorporate greater application of local knowledge alongside continued strengthening of basic and applied research, both public and private, even including the widely condemned linear top-down transfer-of-technology approach for particular purposes. Some participants claim that such pluralism was indeed the gist of the IAASTD messages. While almost any message can be read into the reports, one of pluralism with no options off the table and guided by strong evaluation is clearly not the message that won through. 26. The lessons derived from the IAASTD experience are divided into lessons for future global public assessment programs and lessons for the World Bank. The lessons for assessment programs are as follows: Control of the pen has an enormous influence on the product. Therefore there is a need for a rigorous and balanced formal process in the selection of authors and reviewers as independent as is realistically possible, perhaps monitored for fairness of process by a disinterested institution. In complex assessments with anticipated areas of dispute, a phased process should be considered. One option might have been the following: Stage 1, a rigorous author selection process; Stage 2, small teams of authors would work on the disputed topics following carefully crafted TOR with funds provided to consult or take depositions widely; Stage 3, first stage findings would be tested, and amended where agreed, with the wider participant group; and Stage 4, the analysis and findings in less disputed areas would be brought together with the Stage 3 findings into the final products. With facilitation, these would be required to specify information gaps and to openly characterize the sources of dispute both technically and with respect to political stances. Technology options offered over a 50 year timeframe will inevitably depreciate given the short life of new technology. Such a long timeframe calls for a conceptual framework and decision processes tailored to the changing circumstances of time and place. The documents of a multi-stakeholder assessment of this type need to make explicit the main outstanding disputed assessment issues, the different interpretations of the xix evidence, the evidence gaps that might contribute to closure on those issues, and the political stances contributing to the opposing positions. The number of participants needs to be matched to the available budget so that full payment can be made to all participants. The alternative, with funding for some authors but not others, is a source of potential bias. Sufficient time needs to be provided for sub-reports to be completed to feed into the main report and then into the summary report. To manage this, one option would be some form of Computerized Project Management system to develop a realistically phased schedule, to better project the probable elapsed time and to monitor the critical path. Professional facilitation and support from professional editors could help resolve disputes and improve the impact of documents. The risk of facilitators or editors biasing the outcome seems likely to be lower than the risk of outcomes being biased from less guided and more rancorous dispute or from unclear articulation of, or interpretation of, unclear messages. Closing an assessment immediately on production of the reports gives little chance for dissemination and follow-up. A dissemination strategy should be part of the program design and should be budgeted for and implemented with a defined exit milestone. 27. Lessons for the World Bank are the following: While the Bank should not shy away from controversy, it should not support studies of potentially very controversial issues where neither the process nor objectives can be influenced, when there is no compelling need, or until alternative options have been exhausted. Bank-managed reviews can still draw on wide stakeholder consultation. To answer the main instigating question about GMO policy and organic agriculture, or to address the broader objectives that were ultimately posed, there was no pressing need, in this case, to hand over the assessment to an outside team of uncertain ultimate representation responding to objectives which, at that point, were not even defined. When the Bank supports a global public program assessment in the area of public policy and science, it should focus from the outset on influencing adherence to some principles, in particular: openness, accountability, and procedural fairness. It should also attempt to ensure that the program is budgeted and planned to initiate up front and to then sustain the communication of outputs. Closing an assessment simply with the issuance of reports leaves unfinished business. The higher the reputational risk to the Bank, the greater should be the separation of a program from the Bank. In this case, it was clear that this was a highly political topic with potential for findings that the Bank might disagree with. Yet it was managed and positioned very close to the Bank to the extent that the program management and the xx Bank supervision role were, at one point, almost indistinguishable. Due to the closeness of the program to the Bank, IAASTD reports were perceived by many outsiders as a product endorsed by the Bank. The stated objectives of a project should be realistic, monitorable, and achievable within the budget and within a reasonable timeframe, and with some prospect of demonstrating attribution. In this case, the GEF project had objectives and indicators designed by the Bank that were unrealistic and were arguably misleading to those who approved the project. Implementation Status and Results Reports should accurately reflect both outcome and output level indicators given in the appraisal documents unless there has been a formally agreed revision. 1 1. The Program Program Origin and Objectives 1.1 The World Bank and the Food and Agricultural Organization (FAO) of the United Nations initiated in August 2002 the global consultative process which became the International Assessment of Agricultural Knowledge, Science and Technology for Development (IAASTD),1 with the support of a number of other international organizations, donors and the private sector: GEF, UNEP, UNESCO, Australia, Canada, the European Community, Finland, France, Ireland, Sweden, Switzerland, United States, the European Community, and Crop Life International. 1.2 This followed earlier discussions at the World Bank with executives of a number of biotechnology, pesticide and seed companies in 2000, and a similar meeting with non- governmental organizations (NGOs) in 2001 "on the state of scientific understanding of biotechnology and more specifically transgenics."2 The original intent of the Bank and some of the other major potential players at the initiation of discussions was to improve the scientific understanding of modern biotechnology and to develop a World Bank policy on genetically modified organisms (GMOs), but this was broadened over the early evolution of the program mainly due to input from NGOs, first towards agricultural science and technology more widely and then to include consideration of knowledge, particularly local knowledge. 1.3 The First Plenary of the IAASTD, meeting in Nairobi from August 30 to September 3, 2004, articulated four primary goals for the assessment, as follows: (a) Assess the effects of agricultural knowledge, science, and technology policy and institutional environments, as well as practices, in the context of sustainable development. (b) Identify where critically important information gaps exist in order to more effectively target research. (c) Make the resulting state of the art, objective, analyses accessible to decision makers at all levels from small producers to those who create international policy. (d) Further the capacity of developing country nationals and institutions to generate, access, and use agricultural knowledge, science and technology that promote sustainable development. 1.4 Below these four primary goals, the First Plenary also agreed upon three broad objectives for the assessment: 1. The acronym, IAASTD, was not changed after the addition of the world "Knowledge" into the name of the program. 2. Preface to the IAASTD Global Report. 2 (1) Bring together the range of stakeholders (consumers, government, NGOs, private sector, producers, scientific community, international agencies) involved in the agricultural sector and rural development to share views, and gain common understanding and vision for the future. (2) Undertake global and sub-global assessments of the role of knowledge, science and technology (KST) as it pertains to agriculture in reducing hunger and poverty, improving rural livelihoods, and health, increasing incomes, and facilitating equitable, environmentally, socially and economically sustainable development. (3) Provide robust information for decision makers on how to ensure that policies, practices and institutional arrangements enable KST to contribute to reducing hunger and poverty, improving rural livelihoods and health, increasing incomes, and facilitating equitable, environmentally, socially and economically sustainable development. 1.5 These goals and objectives were not well formulated into a hierarchical structure. While the four primary goals can be largely understood as outcomes in evaluation terms and the objectives can be largely understood as outputs, goal (b) on "identifying gaps" is largely parallel to objective (3) on "providing robust information," yet the two feature at different levels of intent. For the purposes of this GPR, IEG has also reordered both the goals and the objectives into a more logical sequence to facilitate the evaluation of their achievements.3 In addition to these goals and objectives, and generally agreed, widely applied, and cited in the prefaces to the summary reports was the intent to look back 50 years and forward to 2050, often stated in the reports as "looking forward 50 years." 1.6 One and a half years after the First Plenary, the Project Document for the GEF project, which committed US$3.0 million to the IAASTD, reiterated the above three objectives (1, 2, and 3) verbatim as "specific project objectives." But above these, it formulated the higher-order "goals" of the program somewhat differently in terms of a Development Objective and a Global Environment Objective, respectively as follows: (i) Improve access to agricultural knowledge, science and technology that will promote and facilitate sustainable agricultural practices with the aim to improve nutritional security, enhance rural livelihoods while averting environmental degradation, and addressing social and gender inequity and ensuring human health and well-being. (ii) Apply the knowledge generated through the IAASTD to provide significant local, national, regional and global environmental benefits including reducing the overall rate of soil, runoff and nutrient losses, diminishing the contamination and eutrophication of freshwaters and soils, reducing the rate of greenhouse gas emissions, and increasing the amount of carbon fixed in agricultural systems. 3. Primary goal (b) was originally goal (c), and vice versa. Objective (1) was originally objective (3), objective (2) was originally objective (1), and objective (3) was originally objective (2). 3 IEG chose not to base its assessment of the outcome of the program against these latter two objectives, since these were formulated 18 months after the First Plenary, largely unknown to most participants at the Plenary. 1.7 The audience for the assessment was seen as a range of stakeholders including government policymakers, the private sector, NGOs, producer and consumer groups, international organizations and the scientific community. However, the paramount intent seems to have been to influence government policy makers in developing countries such as Ministers of Agriculture and Directors of national agricultural research organizations towards new approaches to the generation and application of technology in agriculture. Activities 1.8 IAASTD was an assessment undertaken by about 400 experts selected by a Bureau following nominations by stakeholder groups. It was composed of a Global Assessment (GA) and five Sub-Global Assessments each producing a separate report: Central and West Asia and North Africa (CWANA); East and South Asia and the Pacific (ESAP); Latin America and the Caribbean (LAC); North America and Europe (NAE); and Sub-Saharan Africa (SSA). In addition, two main summary documents were prepared -- the Global Summary for Decision Makers of the Global Report (GSDM) and the Synthesis Report (SR). The process adopted was multidisciplinary and multi-stakeholder, calling for the use and integration of information, tools and models from different knowledge paradigms including local and traditional knowledge. The experts were expected to work in their own capacity and not represent any particular organization or group. However, there is evidence that some groups did coordinate on their particular positions. 1.9 The formal process began in January 2003 (see Timeline in Annex B). A Steering Committee of 40 representatives from governments, agencies, industry, farmers, other rural producers, consumers, and NGOs produced a Steering Committee report (August 12, 2003). This was followed in August/September 2004 by the first meeting of the parties in Nairobi. At this meeting, the government representatives of 45 countries decided to proceed with the Assessment, and agreed upon the content, scope, timetable and budget. Also, at this meeting the governing Bureau consisting of 30 government representatives, 22 non-government representatives and 8 representatives of international organizations (none of them from the Bank or other donors) was established to oversee the process (see Box 1). The Bureau appointed the two co-chairs (Hans Herren and Judi Wakhungu) and the Director (Robert Watson) whom the Bank had put forward as a candidate.4 From January through May of 2005, there were meetings of all the Global and Sub-global study design teams and towards the end of 2005 the Bureau adopted the outlines of the various global and sub-global assessments. By September 2006 the first draft of the assessments were made available for review and comments. In late 2006 and early 2007 there were two Bureau meetings. By 2007 second drafts of the global and sub-global assessments and the first draft of the GSDM became available for review. The final plenary meeting for the adoption of the reports was in April 2008 held in Johannesburg. At this meeting there were representatives of 61 governments, the World Bank, 4. Robert Watson was previously chair of the Intergovernmental Panel on Climate Change (IPCC) from 1997 to 2002 and co-chair of the Millennium Ecosystem Assessment from 2000 to 2005. 4 a number of UN agencies, and about 50 NGOs. Fifty-eight of the 61 governments approved the GSDM and the Executive Summary of the Synthesis Report.5 The Independent Evaluation (IE), on which the present review draws, was completed in June 2009. Financial Resources 1.10 The total financial costs of the assessment were about US$12 million -- the main sources of funding being a multi-donor Bank-administered trust fund at US$3.1 million, the GEF at US$3.0 million, the World Bank's Development Grant Facility at $1.5 million, and in-kind contributions from a number of donors at US$3.2 million. (See Table 3 in the Efficiency section below.) However, this does not include the resources provided by the many organizations and individuals whose staff costs, travel and other expenses were not reimbursed. The total World Bank contribution (DGF, administrative budget, and other in- kind assistance) was approximately $4 million. 1.11 IAASTD Secretariat staff have estimated the unreimbursed costs to be about the same as the funded costs -- similar to the estimated shares of reimbursed/unreimbursed costs for the Intergovernmental Panel on Climate Change (IPCC) and the Millennium Ecosystem Assessment (MEA). This suggests a round figure of about US$25 million for the total IAASTD costs. However, this may be an underestimate, given the high share of OECD authors (not reimbursed) and the modest level of honoraria provided to non-OECD authors. IAASTD Governance Structure 1.12 Briefly, the Steering Committee which had initiated the process, being no longer needed, ceased to exist after the First Plenary in Nairobi. Thereafter, the governance structure, as shown in Box 1, consisted of a Director, a Bureau, a distributed Secretariat, and Regional Implementing Organizations. The structure was modeled after the IPCC but with modifications. In particular, the Bureau, as a multi-stakeholder and advisory group of participants, was intended to give leadership while retaining the government representatives, whose input and voting voice were considered essential for legitimacy. The government panel was expected to vote on major issues, while the Bureau was to handle inter-sessional issues including author approval. 1.13 The Secretariat in Washington managed the process of nomination of authors and review editors. Washington-based members of the Bureau met with staff at the World Bank to assess the curricula vitae of nominees for their suitability (IE p. 34). These candidates were forwarded to the Bureau for their votes. Rejection of candidates required four Bureau members to register objections. Some Bureau members reported in the IE survey that there was insufficient time to focus on individual author selection. If candidates were unable to serve, alternatives were selected. Lists of potential candidates were drawn from many sources including individual professional contacts, submissions by stakeholder groups such as the NGO group, and self-submission that was open through the Internet. 5. More detail on the process is given in Annex 3 of the IE. The governments of Australia, Canada, and the United States did not fully approve the Global Summary for Decision Makers. They had reservations with respect to balance, GMOs, trade, and the Table GSDM-1. 5 Box 1. IAASTD Structure Director. The Director was head of the Secretariat and central in holding the assessment together. He attended all major meetings and played a more dominant role than the co-chairs. Bureau. This included 2 co-chairs, 30 government representatives, 22 non-government representatives, and 8 representatives from international organizations. Within the 22 non- government representatives, there were 6 NGOs, 6 private sector, 6 producer and 4 consumer representatives. Secretariat. The Secretariat was responsible for management, operations, coordination and administration. It had three locations, the main office at the World Bank (Coordinator, 2 Senior Technical Specialists and support staff) and subsidiary offices at UNEP (2 staff) and UNESCO (2 staff). Regional Implementing Organizations (RIOs). There were four RIOs that helped to manage the sub-global processes: the African Centre for Technology Studies (ACTS) in Kenya, the Inter- American Institute for Cooperation on Agriculture (IICA) in Costa Rica, the International Centre for Agricultural Research in Dry Areas (ICARDA) in Syria, and WorldFish in Malaysia. At each, there was a Regional Coordinator and support staff. The RIOs did not play a significant role in substance. 2. The Independent Evaluation of IAASTD Design and Methodology 2.1 The June 2009 Independent Evaluation (IE) was commissioned by the Agriculture and Rural Development Sector Board of the World Bank on behalf of the IAASTD Bureau which no longer existed by that time. The terms of reference (Annex C) was prepared by the Agriculture and Rural Development Department (ARD) and approved by the Sector Board. The IE was commissioned prior to the termination of the program and therefore before there was public access electronically to the principal reports.6 However, participants (writers, authors, review editors and Bureau members) had drafts and a limited number of printed copies had been distributed to participants and institutions. The evaluation was asked to focus on the three main criteria of evaluation: relevance, effectiveness, and efficiency. It was carried out by a five member evaluation team,7 who issued their final report in June 2009. The draft report could not be formally presented to the program management for comments since the IAASTD Secretariat had closed down by then, but the draft was sent to the IAASTD Director who had taken up another position. He later commented on the final report. Independence and Quality 2.2 The evaluation team operated independently of both the IAASTD Secretariat and the IAASTD process, and there were no evaluators on the team with conflicts of interest with 6. The publisher Island Press had exclusive rights not to have competitive Web-based reports available for six months. 7. The evaluation was led by Howard Elliott with sub-global authors covering each IAASTD sub-global area: Eduardo Trigo (LAC), Ed Rege (SSA), Krishna Alluri (ESAP), and Ayman Abou-Hadid (CWANA). 6 respect to the IAASTD. ARD provided some input on the proposed methodology. The evaluation involved several steps. First, the team interviewed a range of participants and stakeholders, including researchers and academics, government representatives, members of the Global Forum on Agricultural Research, and Civil Society Organization participants. Second, the team administered an on-line survey of writers, authors, review editors, Bureau members and other participants at the final Johannesburg plenary in April 2008. Some 230 usable responses were received. Of these, 106 were contacted by e-mail for further follow up, of which 60 provided additional information. Another 27 were interviewed face-to-face and 46 by telephone. Of 230 respondents, 48 had dropped out of the IAASTD process for some reason. The various reasons are discussed more fully by category in the IE (p. 97), but these included difficulty with the process, the directions the study was taking, and lack of time. 2.3 The present review has been cautious in using the IE survey results as evidence for quality of the IAASTD output (as opposed to the process) since the respondents were obviously interested parties in the preparation of the reports and, as the IE noted, many were predominantly familiar with their own chapters. The IE also carried out a separate telephone survey for Sub-Saharan Africa to assess the level of knowledge about the IAASTD and expectations of impacts. This was addressed to officials based at research institutions and government departments responsible for crop and livestock production in 16 countries. The universe for this was an available list of past conference participants in agriculture. This survey was random in the sense that it did not have any prior information as to whether these people had been involved in IAASTD. 2.4 Overall, the IE is rated moderately satisfactory. While the evaluation was comprehensive, containing much valuable material and offering important insights into the process and the participants' reactions, it exhibited a number of weaknesses: In a significant number of the closed-ended questions, five answers were available: Below Average, Effective, Above Average, Very Effective, and No Answer. This was unbalanced, allowing three positive categories and one negative, potentially biasing answers in a favorable direction -- a methodological issue which makes some of the responses difficult to interpret. However, not all questions were biased in this way. The IE states (p. 91), "For an International Assessment, we assume that the median response would be that it was `effective.'" It is not the place of a survey to make initial assumptions about respondent ratings. In the tabulations of answers by affiliation, the private sector responses were omitted, although only 8 of 230 were in that category. From what is known of private sector views, these would have been the most negative. There are some inaccuracies in the reporting of data. For example, the IE said (p. 96) that 25 percent of respondents responded that multi-functionality of agriculture was new to them, but this was an incorrect reporting of the responses both in terms of the question and the percentage.8 8. The question (IE pp. 59­60) asked about overall "changes in conceptual understanding" from participation in IAASTD of which only 19 percent of respondents overall reported "some changes" in conceptual 7 The evaluation is difficult to follow in tracing what objectives are being evaluated. It is not clear where the main objectives statement in the text (p. 21) comes from. This is stated in what seems to be a highly abbreviated form as: "The IAASTD project objectives were 1) Assess and use information and knowledge, and 2) Capacity building." These are not the same as the objectives stated at the First Plenary and given earlier in the IE document "Program at a Glance" table. The difficulty is exacerbated since the IE also added the GEF objectives assessment in an annex. The main IE text does not give an entirely clear answer to the key question of the extent to which IAASTD achieved its stated objectives.9 The analysis of the GEF objectives in Annex 13 of the IE seems to have been a late add-on, which is not well integrated into the main text. However, the notes to the GEF rating tabulation in Annex 13 are arguably the most insightful parts of the whole report. 2.5 Unfortunately, the sub-global IE documents have not been made publicly available. While these are of variable quality, there is much in them of value including some useful additional survey evidence, some of which did not make it into the main IE text. 2.6 The main Lessons from the IE (p. 37) were the following five, which looked to the future and which the present review largely concurs with (summarized): There is little support in the IE survey responses for the institutionalization of IAASTD. There is opportunity for self-forming communities of practice or networks to address the unfinished business. The design of future assessments should take into account the management of trade- offs between salience, credibility and legitimacy and the tension created. There should be clear rules of conduct and coaching on the performance of participant functions and conflict-resolution procedures. Future assessments should have professional facilitation. There should be urgency in the establishment of the brand name and the purity of the product. In the case of IAASTD, there is a risk of loss of impact associated with the withdrawal of the private sector donor;10 a long hiatus before the availability of the reports electronically; and partial messages not faithful to the reports. understanding. Of those 19 percent of respondents, 25 percent referred to changes in conceptual understanding related to multi-functionality. This implies that 5 percent, not 25 percent gained simply "some changes" in their understanding of multi-functionality. There are other examples. 9. More specifically, the key efficacy question is passed from the Effectiveness 2 section (p. 23) to the Effectiveness 4 section. But then, under the Effectiveness 4 section, it becomes a question in the category of outreach. Yet outreach was only one of several stated objectives. 10. While threatening to withdraw, Crop Life agreed to continue providing its remaining funding. 8 Impact of the Evaluation 2.7 Unlike most other global partnership programs supported by the World Bank,11 the IAASTD was no longer an ongoing program in which the program management had the opportunity to react to and adopt the recommendations. The responsibility for further action now lies with the original sponsors and other stakeholders who care to take up the baton. 2.8 The IAASTD Director has responded to the final June 2009 IE Report, and his response is publicly available on the IAASTD Web site.12 The main points of his response are summarized in Annex D. Briefly, these are: (a) that the IE found the glass half empty when it was actually half full; (b) that the IE contained inaccuracies; (c) that the problem of author turn-over was exaggerated; (d) that dropping the scenarios was the result of a legitimate process; (e) that there was, in fact, a significant outreach effort; (f) that the Secretariat had no control over the 6 month gap to Web publishing and that the summaries were available on the Web from the time they were finalized; (f) that facilitation and conflict resolution would probably not have helped (although the door was left ajar on this issue); and (g) that the IAASTD was timely and effective and provided a vehicle for a wide range of participants to assess the important issues. 3. The Effectiveness of IAASTD 3.1 Among the participants there were three discernible sets of epistemic communities. At risk of substantial simplification these are labeled here as the environmental community, the productivity community, and the bridging community, the latter more intermediate and often with a foot in both camps. The environmental community tended to include the majority of NGOs, organic farming organizations, sociologists, and participants averse to GMOs and intensive agriculture. The productivity community tended to include the majority of CGIAR and national agricultural research system scientists, the majority of staff of international organizations and donors, the private sector, economists, agricultural development practitioners, and natural scientists. The middle group, the bridging community, tended to include a mix of the above players exhibiting aspects of positions of both sides. 3.2 Based on the reports and reviewers' comments there was a range of interpretations of the objectives. While the objectives as written appear to give approximately equal weight to development and sustainability, the approach of many authors suggests that the objective was seen as an opportunity to rebalance the focus away from productivity and growth towards natural resource sustainability. Indeed, for some, this was clearly in the nature of a campaign, although it would be naïve to interpret any epistemic community's approach as entirely neutral. These interpretations of the objectives are portrayed in the interviewee comment, "It was biased but it was a useful bias." However, it is problematic to assess the achievements of the program against this interpretation of the objectives because ex ante, there was no explicit claim for this focus, and ex post, there was no caveat signaled to decision makers that the 11. Two other exceptions are the Millennium Ecosystem Assessment and the World Commission on Dams. 12. www.agassessment.org. 9 report focused mainly on environmental aspects. Therefore, IEG has assessed the achievements of the program against its objectives, as written. Relevance 3.3 Relevance is an aggregate of the relevance of the objectives and the design. The relevance of the objectives, in turn, is assessed in terms of four dimensions: the existence of an international consensus for the program, alignment with beneficiary needs, subsidiarity, and the absence of alternative sources of supply. Relevance is rated, on balance, substantial but with a number of questions about the design of the program/process. 3.4 International Consensus. The IAASTD process began with internal meetings in the World Bank followed by meetings with FAO and other potentially interested parties leading to the Steering Committee meetings and the First Plenary in Nairobi in August-September 2004. (See Annex B Timeline.) During this process, driven mostly by NGOs, the objectives broadened from the original World Bank focus on GMOs to a broader interest in agricultural technology and eventually to encompass local knowledge in addition to science and technology. The early meetings and the strategy statements of donors suggest that there was a fairly broad international consensus on the importance of these issues. Over the years leading up to IAASTD, there had been increasing concern about the environmental impacts of intensive, high-input agriculture. There was also uncertainty among many players including developing country governments about the narrower issue of what policies they should adopt on GMOs since they were seeing potential impacts on their export markets. Shortly after IAASTD was initiated, the global food price crisis gave an additional motivation for the focus on agricultural technology. 3.5 Alignment with the Needs of Beneficiaries. Based on the goals and objectives, the primary intended beneficiaries of the assessment appear to have been decision makers, particularly those in developing countries and, through the decision makers' roles as policy makers, the poorer farmers and consumers in developing countries. The environmental focus and the long timeframe would also imply national public good perspectives of sound environmental stewardship and the welfare of future generations. There was less focus in the objectives on alleviating poverty through agricultural growth. That 20 of the 30 governments represented on the Bureau were from developing countries indicates that they were part of the international consensus underlying the exercise. 3.6 Subsidiarity. This concerns the most appropriate level -- global, regional, national or local -- at which the activities should be carried out in terms of efficiency and responsiveness to the needs of beneficiaries. The assessment was appropriately pitched at the global and sub-global levels. Information and knowledge about the role of agricultural knowledge, science and technology in sustainable development is in theory a global public good, since its reach extends beyond national and regional boundaries. But useful knowledge, especially for decision makers, is also contextual, depending on such factors as the stage of national development and agro-climatic conditions. The five sub-global assessments represented a reasonable disaggregation of countries into five regions based on such criteria without losing the benefits of economies of scale in research. National 10 governments remain free, of course, to disaggregate even further to their own countries, or in the case of large countries, to agro-ecological regions within their countries. 3.7 The more fundamental question concerns the legitimate role of the various stakeholders assembled in the IAASTD process in producing this information and knowledge in the first place. For the increasing role of democratic processes, such as the IAASTD, in scientific debate challenges long-held principles of autonomy, impartiality, independence and objectivity which have traditionally conferred a degree of privilege upon science and scientists (Porter and Philips, 2007, pp. 4­5). As an example of public science, the IAASTD process presented challenges both to science and to the "global citizens" represented at the table. Given that the IAASTD was designed as a democratic process, it appropriately included representatives of governments, CSOs, the private sector, producer and consumer organizations, and international organizations. One could ask, however, why very few farmer or consumer representatives or those closest to them, such as country-level extension specialists, were represented among the authorship participants. Farmers -- among the ultimate beneficiaries of the whole process -- were represented largely by others, rather than representing themselves. 3.8 The Alternatives. There does not appear to have been much consideration of alternative design. The assessment was to follow a somewhat modified IPCC process and structure, which probably did have a demonstrated comparative advantage. However, given the fact that the disputes which surfaced about the role of GMOs, organic agriculture, and trade had been growing for many years, one alternative process would have been to conduct focused studies of these component issues separately and in advance and then, having reached whatever level of resolution and agreement on knowledge gaps was possible, to have moved on to the wider assessment. Another possible alternative would have been to contract the task to an existing organization such as the Global Forum on Agricultural Research (GFAR) or to have it managed by GFAR. But GFAR, although a very important forum, was, and still is, a network of modest capacity. Using an existing organization would have enabled some post-report continuity that would have helped subsequent dissemination and capacity development but all existing organizations would come with "baggage." Another alternative would have been a similar structure but simply much smaller, perhaps with 50 to 80 authors, fully funded and rigorously selected. While numbers in the IAASTD have given legitimacy of representation, they may have reached well into the realm of decreasing returns to scale in terms of management for quality without adding much to legitimacy at the margin. 3.9 The Design of the Process. There was too much focus on the provision of information and not enough on a conceptual framework, on differentiation by country, by stage of development, or by environmental circumstance. As one participant observed, "what an international analysis can provide is not so much information as a conceptual framework within which decision takers can assess the changing facts in their own different policy contexts. ... A good conceptual framework enables a decision taker to recognize relevant issues." As an example, the World Commission on Dams constructed its recommendations around a framework which included seven strategic priorities and related policy principles and then translated these into a set of criteria and guidelines for key decision points.13 The lack of a framework in the case of IAASTD appears to be partly due to the fact that nobody was actively 13. World Commission on Dams, 2000. 11 managing and coordinating the overall emerging storyline. In theory this was the role of the Bureau. In practice, as a review of the minutes of their meetings shows, the majority of recorded Bureau decisions were procedural and logistical rather than guiding substance. 3.10 GMO industry representatives were highly critical about a number of aspects of design14 including the following (summarized): (a) Bureau representation was unbalanced towards NGOs, organic producers and social sciences with mainstream agricultural expertise, the private sector, and natural sciences being underrepresented; (b) author composition was unbalanced; (c) the July 2007 Bureau decision to direct chapter authors to modify their text for a more balanced view was not relayed with any conviction; (d) authors abandoned outlines and developed their own content; (e) the Secretariat gave conflicting messages at authors' meetings; (f) Coordinating Lead Authors did not exercise leadership towards agreement on messages, but acted mainly as coordinators of material; (g) authors with extreme views and personal agendas or active campaigns were able to devote a disproportionate amount of time to the chapters; (h) reviewer comments were disregarded; and (i) there was selective omission of pro-biotechnology references. The present review finds some of these concerns justified, particularly in the area of participant selection. 3.11 There was inadequate provision for any follow-on mechanism. Once its reports were issued, the IAASTD closed shop. At the design stage there was some intent to undertake capacity building, but this was not given much substance and was not taken up for implementation. Box 2. IAASTD's Objectives Were Consistent with the Bank's Rural Sector Strategy IAASTD's objectives were broadly consistent with the Bank's 2003 rural strategy, Reaching the Rural Poor, but the latter targets priorities in somewhat different directions. It stresses the importance of stakeholder involvement and improved linkages to civil society. It outlines land reform, property rights, and administration as priority areas, as well as research, extension and support for producer organizations and user groups and innovation in the pursuit of agricultural productivity and competitiveness. It discusses the need to strengthen the voice of the rural poor in national planning processes. It aims to support social inclusion, particularly to strengthen the voice of women and the landless and to improve women's access to land and finance. However, IASSTD's products in the form of its reports are less consistent with the Bank's rural strategy, since the latter still places substantial importance on productivity, on intensification through new technology, and on marketing and economic growth, albeit with an emphasis also on the environmental impacts. Efficacy 3.12 Efficacy is assessed against the three stated objectives (definable as outputs in evaluation terms) and the four higher order goals (definable as outcomes). 3.13 Overall, efficacy is rated modest 15 on the grounds that, notwithstanding a large body of information, there was no conceptual framework mapped out16 to guide decision makers, no 14. http://www.sfiar.ch/documents/recommend_dubock_iaastd.pdf. IAASTD ­ Process and Structural Issues, 2008. 15. The range of ratings used is high, substantial, modest, and negligible. 12 differentiation by country, economic or agro-ecological circumstance, an undercurrent against new technology particularly in the summary reports, long laundry lists of policy or investment options, mixed messages, and only scattered glimpses of a long-term vision. Some of these weaknesses might have been less significant for a short-term strategic adjustment in favor of greater environmental and social focus, but are significant for a 50 year time frame that clearly will see enormous changes in demand and technologies. Moreover, the many promising initiatives evidenced in the reports, particularly in parts of Chapter 3 of the Global Report, seem to point clearly towards a pluralistic approach that leaves all options on the table while subjecting them to rigorous impact assessment as a decision tool for moving forward. While almost any position can be found in the reports, a pluralistic approach guided by impact assessment is not discernible in the overall messages to decision makers. OUTPUTS Objective (1): The extent to which a range of stakeholders was brought together to share views, gain common understanding and a vision for the future. Rating: Substantial on balance, but with shortcomings. 3.14 Bringing Together a Range of Stakeholders. The substantial range of stakeholders involved conferred considerable legitimacy. For this reason alone this objective is rated, on balance, substantial. However, farmers, their organizations, consumers, national and local level NGOs and extension specialists were largely excluded, in other words many of those who are guardians of local knowledge.17 One interviewee commented: "my main misgiving was that this was a group talking on behalf of farmers ... it was a missed opportunity." The lack of farmer representation seems to have been partly intentional. A major global organization representing producers was given the impression early on that it was not welcome on the Steering Committee. It did not participate thereafter. Of the six Bureau members representing producer groups only one of the six could have been expected to speak partly for intensive cropping agriculture. Based on Web-site material, the six together represented a minute share of the worlds' farming knowledge or production volume. Some 70 percent of the authors and review editors were from OECD countries. There are questions about the selection process which are addressed in the Governance section below. 3.15 Gaining a Common Understanding and Vision. Based on the IE survey, greater "common understanding" was gained by many participants but there was no closure on the contentious issues, if anything positions became polarized. Given the lack of the full scenarios or a qualitative substitute and the mixed messages, the 50 year vision is unclear. 16. There is a section covering conceptual frameworks (Global Report, p. 12) with a diagram (Figure 1-7), but it is difficult to link the diagram to the accompanying text or vice versa, and this appears largely to cover background material leading into the issues rather than setting a framework for decision makers that might have been developed further through the reports. The linkages to the rest of the assessment are not clear. 17. The member of the IE team from SSA noted particularly (in a personal communication) that in SSA there was limited participation by farm organizations, indigenous private sector organizations and local NGOs. 13 Objective (2): The extent to which global and sub-global assessments of the role of AKST were undertaken and the extent to which these can be expected to contribute to reducing hunger and poverty, improving rural livelihoods and health, increasing incomes and facilitating equitable, environmentally, socially and economically sustainable development. Rating: Modest 3.16 This objective is given high weighting because it encompasses the outcome elements including poverty alleviation and sustainability. The output consisted of a Global Assessment, five Sub-Global Assessments, a Global Summary for Decision Makers (GSDM), and a Synthesis Report (SR). The main messages in the GSDM are summarized in Box 3. While clearly an assessment was produced, the present review assumes that there is a Box 3. Summarized Key Findings from IAASTD Global Summary for Decision Makers AKST has contributed to substantial increases in agricultural production but people have benefited unevenly and an emphasis on increasing yield has resulted in negative environmental consequences. These environmental problems associated with poverty create a vicious cycle leading to deforestation and use of marginal land. The future will see changing patterns of food consumption. Agriculture is multi-functional. More attention to agro-ecological sciences will address environmental issues while increasing productivity. Strengthening and redirecting AKST will help address socioeconomic inequities. Greater involvement of women will advance sustainability and development. Many of the challenges will require application of existing knowledge. Some will need new and emerging knowledge. A focus on small-scale agriculture will help realize existing opportunities. Progress in reducing poverty calls for opportunity for innovation and entrepreneurship for resource poor farmers and laborers. Decisions around small-scale sustainability pose difficult choices. Public policy, regulatory frameworks and international agreements are critical for sustainability. Innovative institutional arrangements are essential for ecologically and socially sustainable agriculture. Opening national markets to international competition can offer benefits, but can lead to negative effects on poverty and the environment without basic institutions and infrastructure being in place. Intensive export-oriented agriculture has been accompanied by both benefits and adverse consequences such as unsustainable resource use and exploitative labor conditions, depending on the circumstances. The choice of relevant approaches to adoption and implementation of agricultural innovation is crucial for achieving development and sustainability goals. More and better targeted AKST investments, taking into account the multi-functionality of agriculture by both the public and private sector can help advance development and sustainability. While public-private partnerships are to be encouraged, codes of conduct for universities and research institutes can help avoid conflicts of interest. Achieving sustainability and development will involve creating space for diverse voices and a multiplicity of scientifically well-founded options through, for example, the inclusion of social scientists in the policy and practice of AKST. 14 quality dimension in this objective and questions are raised in this report about the quality of the product. Moreover, whether the assessments can be expected to contribute to the poverty and sustainable development objectives stated is questionable given the limited traction achieved so far. The quality concerns are covered below under Objective (3) and later under Goal (a). Objective (3): The extent to which robust information was provided to decision makers. Rating: Modest 3.17 Quality of Product. The present review finds the overall robustness of the product as a source for developing country policy-makers to be modest, notwithstanding much useful, although often conflicting, material. In the IE survey (IE, p. 110), 56 percent of respondents found the reports useful or better for discussion with policymakers. Table 1 shows a substantial divergence between the CSO respondents (83 percent useful or better) and the respondents from the international agricultural research center/ technical agency (27 percent useful or better).18 A number of interviewees expressed concern with the quality of the reports. Industry representatives, in particular, found the reports biased and felt that they had been unable to influence the assessment. Others found value (reported in Scoones 2008), "Perhaps for the first time, different constituencies had to wrestle with the evidence and experiences that inform our point of view..." Others, particularly in the bridging community, appeared to have a range of somewhat lukewarm views, characterized in comments such as, "It was better than expected," "It provided some useful material," and "It was biased but the bias sent some useful messages." Another said, "A lot was pretty OK but pulling together the findings and messages was weak, it needed good policy briefs." Table 1. Usefulness of IAASTD Reports for Discussion with Policy Makers, by Work Affiliation of Respondents (in descending order of usefulness ratings) Affiliation of Respondent "Above "Below Average" and "Useful" No Answer Average" "Very Useful" (%) (%) (%) (%) NGO/CSO 67 16 13 3 Consultant 46 18 18 18 Other Public Institution 41 13 16 31 National University 38 28 15 20 NARI 16 40 16 28 IARC/Technical Agency 0 27 41 32 Source: Independent evaluation. 3.18 An Undercurrent against New Technology. A number of the main messages, especially in the summaries, were delivered through undercurrents, for example in Findings 10, 11, and 12 of the GSDM. Finding 10 asserts that many of the challenges in the future will 18. Note the earlier concerns expressed about the imbalance in the rating options on the survey questions. Note also that in Table 1 private sector respondents were omitted by the IE. 15 require more innovative applications of existing knowledge (italics added). Finding 11 asserts that some (italics added) challenges will be met through new and emerging (italics added) knowledge, science and technology.19 Then the examples listed of these challenges that could be resolved by new or emerging technologies are heavily circumscribed. These are livestock disease; greenhouse gas mitigation; reducing vulnerability to climate change; reducing reliance on fossil fuels; and addressing socioeconomic issues related to public goods. None of these suggest any attention to yield or productivity increase outside a climate change vulnerability focus.20 The next finding, Finding 12 seems to confirm the undercurrent, asserting that targeting small-scale systems will help realize existing opportunities (italicized in the original presumably to emphasize the differentiation from new technology). Finally, Table GSDM-1 entitled Examples of Policy Approaches to Advance Development and Sustainability Goals -- arguably the key table in the whole report -- strongly reconfirms this undercurrent.21 These findings and the table seem clearly to signal that new technology is unlikely to play as important a role as existing technologies. This is not plausible over 50 years. First, the half-life of new technologies is generally considered to be around 5 to 6 years, suggesting numerous cycles over this time frame.22 Second, simply maintaining productivity is not an option, since food production will have to nearly double by 2050. Third, the expected increase in climatic variability will call for varieties and practices that respond well to this variability and much of the increased productivity will need to come from riskier environments. Yet, through the IAASTD global reports, there are statements supportive of new technology. 3.19 The changed lens in the summaries distorts the overall tone of the findings in the global report. For example, Table 6.2 of the Global Report has some quite useful identification of potential research and policy areas including some regional specificity. It proposes transgenics to incorporate yield enhancing traits and cloning in livestock. Even here, the heading on high productivity systems calls for merely "Maintaining yields in high productivity systems" -- demonstrating the widespread ambivalence towards yield- 19. Since "some" of these are expected to be "emerging" presumably only some of the original "some" would be expected to be new technologies. 20. In the main text of the GSDM (p. 20) the language is somewhat different, "Many" challenges will rely on existing technologies, "other" challenges will require new technologies. In Chapter 6, the language is again different, characterizing "many'" for the challenges to be met by existing technologies but new technologies would be needed only for "intractable and changing" challenges. However, here it notes "transgenic approaches may continue to make significant contributions in the long term," language that did not find its way into the GSDM. Again, later in Chapter 6 is found: "In spite of the limited growth in the development of transgenics, it is possible that these technologies will reemerge as a major contributor to agricultural growth and productivity." Overall, this is a confusing story. 21. The table, although headed "Examples of Policy Approaches to Advance Development and Sustainability Goals," has only three horizontal blocks related respectively to payment for environmental services, the management of germplasm, and the management of water, all focused largely on access rights and environment. Out of 36 bullets under 6 columns the nearest any language comes to a policy aimed directly at development through increasing productivity is a bullet calling for: "sufficient involvement of technology users in science policy and practice." 22. Given the multiplicity of sources, agencies, and interest groups, technological change is always in a state of "continuous disequilibrium" (Biggs 1989 and Tiffen, Mortimore, and Gichuki 1994). 16 increasing technologies -- but even this watered-down message on the potential for transgenics barely survived into the summary for decision makers. 3.20 GMOs are generally placed in the most negative light possible with respect to the selection and interpretation of references. In the key chapter in the Global Report on "Options to Enhance Impact of AKST on Development and Sustainability Goals," the section on Transgenic (GM) Plants, the slant is unmistakable (p. 393). Out of the 23 references used, about half appear to be essentially neutral toward GMOs, generally presenting some finding of fact, for example, the level of variation of Bt toxins between individual GMO plants or some such analysis. Of the remaining half, nearly all but one have clearly negative or strongly cautionary findings about GMOs. There is really only one or perhaps two that tell a positive story. This selection clearly does not reflect the overall profile of the literature. Again, in interpreting these references, the drift becomes noticeably more negative. Nearly all, some 19 of the 23 references, are interpreted in at least a somewhat negative light. The criterion used here was that the interpretation would give a decision maker concern in pursuing the application of GMOs. These 19 not only include some 10 references that are clearly negative about GMOs and therefore more or less accurately interpreted, but another 10 that are largely neutral. With these slants of selection and interpretation overlaid, it is difficult to conclude that this important section of the GR represents balanced and "robust" information. 3.21 Conflicting Statements. On most disputed issues, one can find any position somewhere in the report. For example, on organic farming, one reads, "It is, however, unlikely that organic farming will become a real substitute for industrial agricultural production systems even if organic farming yield were similar to conventional yield ... production cost would likely be higher" (Global Report, p. 273). Yet, later in the report, one reads, "Ecological agricultural systems ... have the potential to improve environmental and social sustainability while maintaining or increasing levels of food production" (GR, p. 385). Much of the Synthesis Report focus on GMOs is on the actual or potential negative aspects. Yet one can find in the Global Report: "Overall, it is likely that the elimination of a powerful tool like transgenesis would slow but not stop the pace of agricultural research and improvement. As a result, humanity would likely be more vulnerable to climatic and other shocks and to increased natural resource scarcity" (GR, p. 353). One interviewee noted that the reports build some cases on false dichotomies, for example, the top-down transfer of technology versus local knowledge or high-input intensification versus organic farming. Table 3-5 in the Global Report (p. 224) offers some puzzling examples. 3.22 Undifferentiated Lists of Policy Options. Particularly in the Executive Summary of the Synthesis Report, but elsewhere also, there are lists of mostly undifferentiated diverse options. Such lists are difficult to characterize as robust information for decision makers. For example, how should a decision maker apply the following list in the Executive Summary of the SR under Poverty and Livelihoods (p. 5): "access ... to land and resources; remunerative local ... and export markets; increasing local value added; empowering farmers to innovatively manage soils, water, biological resources, pests, disease vectors, genetic diversity, and conserve natural resources in a culturally appropriate manner; access to microcredit; legal frameworks ... for resources and land; recourse to conflict resolution; progressive devolution and proactive engagement in intellectual property rights; 17 developments ... that build trust and ... value farmer knowledge; agricultural and natural biodiversity; farmer managed medicinal plants; local seed systems; common pool resource management regimes; fiscal and competition policies; improved access to AKST; novel business approaches; enhanced political power."23 There is nothing particularly wrong with any one of the ideas on such lists but they exhibit little differentiation, prioritization, or situational classification and could be readily produced by a brain-storming session with modest resources. 3.23 The Supporting Evidence. It is difficult to trace the pathways of evidence from the Global and Sub-Global Reports to the Global Summary for Decision Makers because the latter refers simply to Chapter numbers and divergent positions can be found.24 Also, as was pointed out by some interviewees, there is a substantial disconnect between the global reports and the summary. Several interviewees suggested that the summaries were "hijacked" by a small group with a particular agenda, one of them, not unfavorable towards the overall messages, saying that, "my suggestions at being `more balanced' in the report were not really listened to." The lack of more specific references between reports is puzzling because the Fourth Bureau meeting decision in May 2007 said that, "The global SDM will be cross referenced to the relevant sections of the global chapters (our italics)." This never happened. 3.24 References quoted come more from less widely used sources. Assuming that the main chapters of potential overlap on biotechnology were Chapter 7 of the WDR, "Innovating through Science and Technology," and Chapter 5 of the IAASTD, "Looking into the Future for Agriculture and AKST," there were only 6 common references out of 124 in the WDR and 205 in IAASTD. It is clear that these reports were largely drawing on a different segment of the evidence. There was only a 13 percent overlap between the IAASTD Global Report and the 2009 IFPRI Report entitled "Measuring the Economic Impacts of Transgenic Crops in Developing Agriculture during the First Decade."25 There are some references used that have been widely disputed, for example, the widely referenced 2007 article by Badgley et al on the potential for organic agriculture. The present review did not have the resources to review more than a small number of references, but a number of interviewees said that, in their view, there had been selective use of evidence. Two added that comments in that direction had been ignored. A sample analysis of the reviewer comments for two GR chapters found that the number of new references suggested by reviewers was only a few 23. This particular list seems to treat access to new technology as an issue at the same hierarchical level as farmer managed medicinal plants and confusingly is classified under the options to "increase domestic farm gate prices." Yet one of the issues with new technology is that, as productivity increases, farm gate commodity price can fall, particularly for the later adopter. Three paragraphs later, under the heading Food Security, a topic not far removed from Poverty and Livelihoods, AKST investment is given a quite prominent position as an option to increase productivity of major subsistence foods. What is a policy maker supposed to make of these different signals? 24. A similar problem occurs in the Sub-Global Reports. For example, the SSA report has a Key Message in Chapter 2 that, "fertilizers and pesticides have had negative effects on human health and the environment in SSA." However, in the body of the chapter not only is the evidence for this lacking, but the SSA report points out the inadequate use of fertilizer and agricultural inputs in Africa that needs to be addressed. With the Key Messages only teased out at the Chapter level, the contradictory messages were never reconciled. 25. The percentage would have been significantly lower if only the main GMO-focused sections of the IAASTD reports had been taken. 18 percent of the total used. Surprisingly, there were no suggestions in reviewer comments to discard a reference. 3.25 Policy Relevant or Policy Prescriptive? Notwithstanding the aim to be policy relevant as opposed to policy prescriptive, there was clearly policy prescription. What CSOs reported on the findings, as well as the contrary views of industry, confirms the perception of prescription. However, a degree of prescription may have been what was needed, but of a different type -- not prescription of lists of a host of possible policy options, or sets of "information," but a prescription of conceptual frameworks, logical paths and the means of evaluation. These would have needed to be defined by circumstance for decision makers to organize thinking about, strategize, test and evaluate alternative approaches to technology policy and investment. OUTCOMES Primary Goal (a): The extent to which the effects of agricultural knowledge, science and technology policy, institutional environments, and practices in the context of sustainable development, were assessed and subjected to state-of-the-art, objective, analysis (the quality standard of primary goal (c)) Rating: Modest 3.26 This outcome goal draws on the outputs in Objectives (2) and (3), both rated modest. Much of the earlier discussion on the robustness of the evidence applies here. 3.27 Highlights of the Reports. It is widely acknowledged that the IAASTD reports offer a substantial volume of information. There are a number of good sections or chapters in the Global Report. Chapter 2 offers a section with some useful analysis of the literature on the Transfer of Technology model and its drawbacks, and the evolution of farming systems research and participatory breeding. Chapter 3 has a valuable set of findings listing the IAASTD goals26 relevant to each finding along with the degree of certainty, the range of impacts, the scale, and analysis on finding specificity. The whole report would have benefitted from this methodology.27 Chapter 4 is still useful with its limited modeling but was weakened by the decision to abandon the full modeling.28 Many good pieces in the Sub- Global Reports seem to have missed being brought forward to the global summaries. 3.28 At the final intergovernmental plenary, 58 Governments declared that, "all countries see these reports as a valuable contribution to our understanding on agricultural knowledge, science and technology for development recognizing the need to further deepen our understanding of the challenges ahead." The governments of Australia, Canada, and the United States did not fully approve the Global Summary for Decision Makers. They had 26. That is, the objectives list nutrition/hunger, human health, rural livelihoods, environmental sustainability, social sustainability, sustainable economic development. 27. It also addresses arguably the most important technological area for agricultural sustainability with a poverty focus, the enhancement of nitrogen fixation technology, unfortunately not carried forward into the summaries. 28. This is reported to have happened while the Director and Chairperson were out of the room. 19 reservations over balance, GMOs, trade, and the Table GSDM-1. While there are a number of strengths, overall the present review finds substantial weaknesses in the reports, which are outlined in the following paragraphs. 3.29 The Extent of New Findings is Limited. Views on the quality of the reports vary widely by community. One donor representative commented (reported in Scoones, 2008) that, "There is everything and nothing in there. It is too long and does not give us clear directions. Who is going to read all this? Will they learn anything new?" Another participant commented, "It won't change things, people will search for what they want, it won't challenge them." However, some were more positive, for example (also in Scoones, 2008), "The IAASTD provides the evidence to show that locally controlled, biologically-based intensification of farming is the only way forward. In short, it supports food sovereignty." 3.30 To assess how new the findings were, the reports were examined against comparators such as the 2009 World Bank World Development Report on Agriculture, the 2005 Millennium Ecosystem Assessment, and the 1998 World Bank Symposium Proceedings on Agriculture and Environment, particularly the latter. Annex E compares the Key Findings of the IAASTD Global Summary for Decision Makers with some of the main conclusions of this Bank Symposium report.29 The findings are quite similar, although there are a few findings which are evident in one and not the other, as well as some differences in emphasis. Annex E also includes a comparison of the Synthesis Report section "Options for Action" set alongside the main paragraph on options for action in the Symposium Conclusions chapter. Again, the differences are quite modest. The efficiency question therefore becomes whether the incremental gains of IAASTD over these available sources was worth the incremental investment, which is addressed further in the Efficiency section below. 3.31 Multi-functionality itself, one of the most widely stated findings, was not new. This review has found nobody who challenges agriculture's multiple impacts.30 The only quoted reference in the IAASTD reports that is claimed to challenge the concept of multi- functionality (Barnett, 2004) does not appear to do so. The challenge for researchers and policymakers has not been so much the concept as the application and trade-offs. 3.32 The GMO Debate Yielded No Convergence. A review of statements in the GMO sections suggests that the position became more negative on GMOs in the transition from the Sub-Global and Global Reports to the Synthesis Report Executive Summary, and then reverted largely to silence in the GSDM Key Findings. The biotechnology part of the SR Summary is predominantly negative in tone.31, 32 Yet the LAC region Summary for Decision Makers, for example, has quite a balanced discussion on GMOs. There were differing views 29. Note that there were no highlighted findings statements in the Symposium Report so some judgment had to be exercised in identifying the main findings statements. 30. Even as far back as the 1930s the off-site erosion, soil fertility and hydrological elements of agriculture's impacts were widely debated in Africa (Hailey 1938), including the cost implications of ameliorative action. 31. The United States and China argued that this section was not balanced or comprehensive. 32. One point made in the reports in criticism of GMOs is that adoption of transgenic crops has been low. However, adoption of transgenic crops -- now covering about 4 percent of cultivated area after 10 years (1994­ 2004) -- is far ahead of zero tillage which has reached about 7 percent of crop land over about 50 years. 20 among interviewees on how important this lack of resolution on GMOs was. One respondent noted that "the whole study became severely distracted by the GMO and organic issues when these were not the big things." On the contrary, the present review considers this failure of considerable importance given the original World Bank intent in fostering the IAASTD. In discussing the GMO dispute, one interviewee participant found that, while the UN style of negotiating consensus provides political credibility, "from an analytical perspective ... the process of reaching consensus by compromise is almost wholly negative." 3.33 Gaps That Weakened the Analysis. While there was much ground covered, there were several critical issues not tackled. First, there is a missing counterfactual on the impact of past technology, which was fundamental to determining where technology should go in the future. This gap in the analysis of the past 50 years of productivity gains in relation to net environmental impacts33, 34 should have been the starting point for the assessment.35 Second, beyond the usual occasional plea for more resources, there was no significant discussion of the resource constraints and how these impact on strategic choices.36 One interviewee described resource constraints as being an off-limits issue. Third, while the global report has some coverage of the shift of the CGIAR towards environment, gender and equity and less promising crops and areas, the assessment does not come to any findings on where CGIAR research might be directed in future.37 Significantly, the shift in this direction, consistent with the assessment findings, is claimed by some to have reduced CGIAR impact. Fourth, it is argued rightly in a number of places that local knowledge warrants more attention, but the trade-offs and cost implications for poorly funded research systems of a shift in this direction were not tackled.38 Indeed, throughout the reports, issues of cost and country capacity in relation to proposed options were largely by-passed. 33. There is brief reference to a counterfactual in the GR (p. 510, Alston and Pardey, 2001). There is also a reference (p. 153) to the Millennium Ecosystem Assessment 2005 finding which estimated that at 1961 yield levels an extra 1.4 billion hectares of cultivated land would be required to meet current levels of food production. There is also analysis in Chapter 5 (p. 333) on alternative investment scenarios. But none of these findings are carried through into the summary reports. Apparently, the Director had proposed early on that such a counterfactual should be developed but it never was. 34. Several of the more important and widely quoted references in the literature on land use impacts related to productivity were, in fact, quoted in the global report (e.g., Angelsen and Kaimowitz 2001), but these were not used to attempt any analysis of the past net negative or positive impact of technological advances. 35. In addition, an analysis or meta-analysis of the elasticity of poverty to cereal yields might have extended more globally the finding (e.g., on India) that the long-run indirect gains through price and wage effects are far greater than the much more modest short-run direct effects (WDR, 2008). 36. An IE CWANA respondent noted this: "It is not sufficient to assess alternative strategies for science and technology; feasibility of implementation and cost alternatives are necessary if policymakers are to take action" (IE, p.115). 37. For example, the report did not explore IFAD's 2001 finding that this shift has probably helped to reduce the growth in yield of staples and has been ineffective in delivering growth to some areas where the poor are increasingly concentrated, and the conclusion that research should now be refocused on yield. 38. A rarely recognized problem for dedicated researchers in poor countries with budget uncertainty is the risk of making commitments to work with farmer groups that cannot sustained due to budget fluctuations. The 2007 IEG assessment of the Tanzania Agricultural Research Project II reported, "The evidence suggests that, yet again, research activities build up and develop linkages with clients during the peak funding period ... only to sink back down to a power-conserving, almost hibernation mode once the project closes." 21 Primary Goal (b): The extent to which information gaps were identified in order to more effectively target research. Rating: Modest 3.34 This outcome-level goal on information gaps draws somewhat on the outputs assessed under Objectives (2) and (3), both rated modest, but it is largely a separate question. 3.35 Information Gaps to Target Research. Information gaps were not systematically identified, although some sections were better than others. The most obvious place to look for analysis of information gaps is in the biotechnology section of the Synthesis Report since this was where lack of information or different interpretations contributed to the greatest dispute. There are a few generic statements on gaps in this section, for example, "a problem oriented approach to biotechnology R&D would focus investment on local priorities identified through participatory and transparent processes." But there is nothing on the key gaps in the level of knowledge on health and environmental risk. Looking more widely across all the reports, although the term information gaps is not used (surprisingly given the goal), there are many scattered mentions of research needs, research options, and calls for "more research on" or other similar phrases. In the GSDM report, where data gaps seem less of a focus, there are about 10 mentions of research needs.39 There are more mentions in the SR.40 Chapter 3 of the GR is another useful source. However, whether these lists of information needs can be effectively processed by decision makers is questionable.41 Primary Goal (c): The extent to which the resulting state-of-the-art objective analysis was made accessible to decision makers at all levels from small producers to those who create international policy. Rating: Modest 3.36 This outcome-level goal draws on the second part of the output level Objective (2), rated modest. Potentially, accessibility could be through Web sites, presentations, printed reports, international and national forums, and the literature and perhaps through more subtle diffusion of ideas through a number of pathways. There was limited consideration given to a planned dissemination strategy in the initial program planning, but some planning did emerge later, carried out jointly between the Secretariat and Island Press. Some 100 presentational events took place up to June 2009 (Table 2). However, one interviewee not unsympathetic to the findings commented that dissemination became primarily an ad hoc personal agenda of a small number of the leading participants. Another noted, "IAASTD had failed in not 39. These include: research on multi-functionality, on data needs on R&D expenditure, livestock, GMO risks, reduction in costs of digesters, bio-energy, and organic certification. 40. These include: resource conservation technology, low input agriculture, breeding for temperature and pest tolerance, ecosystem services valuation, water efficiency, pest bio-control, bio-substitutes for agro-chemicals, and reduction in fossil fuels. 41. What would have helped would have been research options for a defined range of circumstances with a core set of selection criteria including impacts on productivity, financial returns, risk, equity, gender, employment, and natural resource sustainability, drawing perhaps from the better competitive grant application formats and processes. 22 developing more media participation from the outset, leaving communication mostly to the end -- in fact the transfer of technology model that IAASTD itself criticised!" Two participants interviewed asked the IEG reviewer what was happening on dissemination. Another did not find a regional event which he attended to be very useful. The extent of dissemination and the disseminating individual or organization appears to have been partly determined by how much funding remained to be used and the different needs for reimbursement. 3.37 Web site Accessibility. The Web site has functioned since April 2008 as a main source for the IAASTD output.42 Data on the number of Unique Visitors (i.e., excluding multiple hits) show that from the January 2007 starting date there were an average of 78 visitors per day in 2007, 163 in 2008, and 157 up to October in 2009. The peak number of visitors was in April 2008 when the final plenary took place and many participants were no doubt downloading. The countries with the most visitors that month were the USA (17 percent), Germany (11 percent), and the UK (9 percent). No individual developing countries (DCs) had high enough percentages to be reported. 3.38 Printed Report Access. The publisher Island Press had exclusive rights not to have competitive Web-based reports available for six months -- a damaging agreement. With only one bidder, the Secretariat had limited options, but some alternative should have been found.43 Some 55,000 copies of the reports were distributed free of charge through an IAASTD buy- back from the publisher. From the lists reviewed, the copies appear to have gone to worthy libraries and institutions in developing countries. Given that these went mainly to institutions and given the high price ($95 for the GR and $65 for the Sub-Global Reports and summaries) it is unlikely that many found their way to developing country nationals or small producers. 3.39 Publicizing Events for IAASTD. As shown in Table 2, up to June 2009 there were some 100 publicizing events attended by either the Director and co-chairs and some authors and NGOs with an interest in getting the messages out. Table 2. Presentations of IAASTD Findings by Participant Type and Forum (up to June 2009) Public Forum Scientific Forum Policy Forum Total Advocacy Director 1 10 1 12 Co-Chair(s) 11 15 10 36 Authors 12 15 14 41 Bureau 0 3 7 10 Total 24 43 32 99 Source: Independent evaluation, p. 112. 42. There seems to be an implicit assumption that once material is on a Web site, it is accessible to the world. The author of the present review has recently spent considerable time breaking up the IAASTD GSDM into pieces to send to a practitioner in Africa who can only, on a good day, download a little over 1.0 MB at a time. As one participant noted, good Policy Briefs would have been very useful. 43. In May 2007, the Bureau had recommended that the successful bidder must agree that all reports should be accessible free on the Web shortly after publication. 23 3.40 Access through the Literature. A library citations search using a combination of the Social Science Citation Index and Science Direct found, in October 2009, 22 references citing IAASTD reports. For the 2008 World Bank's WDR on Agriculture, by comparison, 52 citations were found. 3.41 Awareness in Africa. IAASTD did not rate highly in a cell phone survey in Africa that was conducted for the SSA sub-global IE report (E. Rege, personal communication). Some 130 respondents who were on an Africa conference address list, mostly non- participants of IAASTD, agreed to be surveyed. Asked about their familiarity with the IPCC, IAASTD, and the 2008 WDR, 52 percent were most familiar with IPCC, 37 percent were most familiar with the WDR, and only 11 percent were most familiar with IAASTD. Only 19 percent felt that the IAASTD had the highest potential for influencing the African agricultural agenda compared to the IPCC at 45 percent and the WDR at 36 percent. 3.42 Local-Level NGO Awareness. During the November 2009 World Bank Development Marketplace Global Competition on Climate Adaptation, IEG asked 14 country-level NGO representatives more or less randomly if they had heard of the IAASTD. Only 2 of the 14 representatives had heard of it -- both from Europe and North America. 3.43 Reader Accessibility. Application of the tests of ease of assimilation gave a poor rating.44 One donor had urged professional editing, but this was rejected due to concern about the possibility of change in messages and the time element. It is not clear how this decision was made because the May 2007 Bureau meeting called for light editing capacity with editors embedded in teams. The layout and presentation is not in the same class as the WDR or the Millennium Ecosystem Assessment. The messages of IAASTD were made less clear by the widespread use of the acronym "AKST" with many different connotations. Through this acronym many authors avoided the challenge of specifying a means.45 3.44 Evidence of Access in National or International Forums. Traction in international and national forums so far has been modest to negligible. A few interviewees said that they had heard IAASTD referred to in meetings. At the national level, despite asking almost all interviewees, no specific examples were given of impact on national research strategies. One participant social science researcher noted, "I doubt that the documents or the major findings will be directly useful for planning future agricultural programs/projects in [country]. Needed details for such purposes are not available." In a similar vein, another noted that the reports were referred to in discussions but that "the reports could be used to prepare broad outlines of agricultural research strategies to a limited extent as they do not indicate specifics." 44. The Flesch Kincaid Grade Level test suggested the need for the reader to be at about a PhD level of education to be able to assimilate it. 45. That is, specifying a means such as an institutional change, an investment, a strategy, a regulation, or a policy. For example, Global Report (p. 8) found that: "AKST can have substantial roles in the formation of better policies." Who or what is AKST here? Elsewhere the phrase "the AKST model" (GR p. 12) was used as though there was an agreed model. Based on reviewers' comments on drafts, there were widely different interpretations of what constituted "AKST." 24 3.45 At the international level, there are cases where it was not referred to when it would have been highly relevant, for example, according to a participant, at the FAO meeting "How to Feed the World in 2050" held in June 2009. But one informant noted that IAASTD language was sometimes appearing without attribution. The FAO State of Food and Agriculture Report, 2009, refers to the IPCC, the MEA and the WDR, but not to the IAASTD. At the recent seventeenth UN Commission for Sustainable Development (CSD 17), IAASTD findings gained little attention although there was a side event at which IAASTD featured (De Schutter, 2009). At the national level, the Review of International Assessments by the International Institute for Sustainable Development, looked at the implications for Canadian agriculture in UNEP's 2007 Global Environment Outlook, the IPCC reports available up to that point, and the drafts of the then emerging IAASTD. However, in general it found IAASTD drafts to be too generic and to focus more towards Europe than North America. In the United Kingdom, the October 2009 Royal Society study "Reaping the Benefits" quotes the IAASTD findings, but comes to different conclusions,46 advocating a more pluralistic approach. At the 2009 Copenhagen Climate Conference, the background paper for the Agriculture and Rural Development Day has some mention of IAASTD, but there do not appear to be any further references to IAASTD in subsequent documents. At the local level, there is no evidence of accessibility at the level of small producers, which was one of the objectives. Taken overall, the impression is of quite limited traction so far at all three levels. The present review concludes that there is insufficient evidence at this point that substantial future impact would be a reasonable expectation. The failure to achieve primary goal (d), discussed below, is a part of this impact problem. 3.46 The Interpretation of the Reports. The support for almost any position can be found somewhere in the reports, allowing wide interpretation. The interpretation by CSOs, for example, was extreme, that "the report ... is a sobering account of the failure of industrial farming ... the old paradigm on industrial, energy intensive and toxic agriculture is a concept of the past ... the reports lack of support for the further industrialization and globalization of agriculture as well as for genetically engineered plants in particular, was based on a rigorous peer reviewed analysis" (Civil Society Statement from Johannesburg 12 April, 2008).47, 48 More to the center, the IAASTD Director, in testimony to the Financial Services Committee of the US House of Representatives (undated), reported from the same findings that "it is possible that GM crops could offer a range of benefits over the longer term." 46. This very useful study was intended to "identify and assess challenges to food-crop production in the developed and developing world, [and to] ... identify and assess any barriers to the effective introduction and use of biological approaches for enhancing food production." It concludes that there will always be trade-offs and local complexities and argues for an inclusive approach in which no techniques should be ruled out. 47. In the same vein, Greenpeace headed their statement, "Urgent changes needed in global farming practices to avoid environmental destruction. World's leading scientists condemn industrial farming methods and see no role for genetic engineering as a solution to soaring food prices and hunger crisis fears" (Greenpeace, 15 April, 2008). 48. On the GMO side and opposed to the civil society position, Crop Life International, a sponsor, who interpreted the report as anti-GMOs, countered (CropLife Press Release 15 April, 2008), "Modern plant sciences can increase crop quality and productivity in order to meet a growing world demand for food, fibre and fuel. Increasing productivity on currently farmed land is the only way to effectively meet this challenge without ploughing under much more land. For the last 50 years, farmers worldwide have benefited greatly from the use of crop protection products, hybrid crops, and more recently plant biotechnology. We hope that policymakers recognize this and will not be misled by the IAASTD report." 25 Primary Goal (d): The extent to which the capacity of developing country nationals and institutions was furthered towards generating, accessing, and using agricultural knowledge, science and technology that promotes sustainable development. Rating: Negligible 3.47 The planned capacity building chapter that might have been a step towards this goal was dropped. The logframe for the GEF project had called for: "by the end of the project, capacity to: conduct integrated scientific assessments to understand the contribution of agricultural KST in poverty and hunger reduction, improvement of rural livelihoods and health, increasing incomes, equitable and environmentally, socially and economically sustainable development." This was never attempted although IE survey responses indicate general satisfaction with mutual learning within the IAASTD participants themselves. UNINTENDED OUTCOMES 3.48 Arguably, the most significant outcome lies outside the goals. This is the lessons learned from a multi-stakeholder assessment that aimed to reach a consensus and in the attempt, which we argue fell short, left behind a rich seam of experiences about public policy processes. This has already been the subject of valuable analysis (e.g., Scoones, 2008) and there is other ongoing work. The experience represents public good knowledge about processes around the nexus where science, politics, world view, and advocacy meet--or collide. Efficiency 3.49 Efficiency is rated modest. As shown in Table 3, the financial costs were about US$12 million. Total costs, including resources that were not reimbursed, are estimated as being of the order of US$25 million, but this may be an underestimate. Of the authors and reviewers, 71 percent were from OECD countries and were not reimbursed at all. However, the non-OECD participants also were not fully reimbursed for time; honoraria were limited to $2,000 for authors and $1,000 for reviewers although, for most, the actual costs of time provided was almost certainly considerably higher. 3.50 The central efficiency question is whether the IAASTD outputs yielded, or can be expected to yield, sufficient gains towards the objectives and goals to justify the resource cost in comparison with alternative knowledge sources. Efficiency is difficult to measure since it calls for evidence on both costs and outcome expectation into the future or for some cost-effectiveness industry standard. The limited evidence points to a modest rating because overall efficacy is rated as modest, while costs appear relatively high. In terms of comparative costs, assuming $25 million, IAASTD was about five times the cost of the World Bank 2008 WDR on Agriculture. It is comparable to the Millennium Ecosystem Assessment, but the latter seems to have had more global impact. The cost of IAASTD was a different order of magnitude to the cost of either the World Bank 1998 Agriculture and Environment Symposium (World Bank, 1998) which seems to offer many similar findings, or the 2009 Royal Society study Reaping the Benefits (The Royal Society, 2009),49 which 49. The World Bank Symposium cost at most US$200,000, and the Royal Society study was prepared by a limited number of authors who were not reimbursed. 26 Table 3. IAASTD Expenditures IEG Estimates as of End October 2009 Source Amount (US$) 1. Multi-donor Trust Fund (TF054513) (Australia $233,565; Canada 3,132,161 $415,204; Crop Life International $125,000; EC $632,550; France $209,117; Ireland $202,260; Sweden $454,545; UK $516,044; USA $250,000 and some others) /1 2. USAID (TF056453) 235,874 3. French Trust Fund (TF030458) double counted? 88,700 4. GEF for the four regional centers (TF056399 thru TF056402) 2,655,381 5. GEF to prepare and supervise project (TF053915) 350,000 6. World Bank: Development Grant Facility 1,498,025 7. World Bank Administrative Budget 793,866 8. Independent Evaluation estimate of in-kind contributions from the 3,172,890 World Bank ($1.8 million), Finland (US$655,000), UNEP (US$225,000), FAO (US$225,000), UNESCO (US$225,000), UNDP (US$10,000) plus others Total 11,926,897 /1 All these countries and Crop Life International (an industry association) were asked to provide additional support to authors and review editors from their countries or organization to ensure participation. Sweden and Switzerland engaged senior academics as consultants to accompany and contribute to the process. DFID contracted with CAB International to coordinate the consultative process in the UK, and the French government instructed its research institutions to participate through staff time, but with travel support included in an additional grant to the research institutions (INRA, CIRAD, and IRD). Canada increased its contribution to the MDTF to help with travel of authors and review editors when it became apparent that there were few Canadians involved. Mars Corporation allowed an academic from Australia to use funds in a research grant for participation in IAASTD to ensure his input on conservation agriculture. (Source: IE) offers important but different findings. The cost of the World Commission on Dams -- an assessment on a narrower topic but with a wide array of stakeholders, and a number of similarities and disputed issues -- was about $12 million. 3.51 Efficiency of Resource Mobilization and Management. This was generally well handled. Table 3 indicates that sponsors were found from several sources and they were generally well-coordinated. 3.52 The Impact of Budget Constraints on Participant Profile. Funding constraints almost certainly biased author selection. This is regretted by many, including the Director. It resulted in strong potential authors being unable to contribute. The fact that unreimbursed authors and reviewers from the OECD countries reached as much as 71 percent of the total suggests that overall OECD countries achieved more than their share of individual participant influence, which represents another bias overlay. The reimbursement problem is likely to have introduced efficiency biases towards those with less work load, those with more outside funding support, those with "an axe to grind," or those with more of an advocacy remit. The resource allocation issue here relates to the trade-off between depth versus coverage. Overall, comparing the IAASTD reports with other products with fewer authors, it is difficult to see a sufficient gain in quality for the incremental investment. Breadth and numbers give a degree of legitimacy depending on selection processes, but in the end perceptions of positive or negative product quality also influence perceptions of legitimacy for the user. 27 3.53 Process Efficiency. Looking more qualitatively at possible factors of efficiency in the process itself, Scoones (2008) and others have commented on the lack of facilitation and support for conflict resolution. This seems to have been a widespread feeling among participants (IE) and suggests a degree of process inefficiency. Overall Program Rating 3.54 As discussed earlier, IEG has applied the goals and objectives adopted at the First Plenary in Nairobi meeting, the three objective statements also being common to the GEF document. While this excludes consideration of the Global Environment Objective in the GEF Project Document, this was found to be unrealistic. Table 4. Overall Program Outcome Criteria Ratings Relevance Substantial Efficacy Achievement of Objectives: 1. Bring together the range of stakeholders (consumers, government, NGOs, private sector, producers, scientific community, international Substantial agencies) involved in the agricultural sector and rural development to share views, and gain common understanding and vision for the future. 2. Undertake global and sub-global assessments of the role of knowledge, science and technology (KST) as it pertains to agriculture in reducing hunger and poverty, improving rural livelihoods, and Modest health, increasing incomes, and facilitating equitable, environmentally, socially and economically sustainable development. 3. Provide robust information for decision makers on how to ensure that policies, practices and institutional arrangements enable KST to contribute to reducing hunger and poverty, improving rural Modest livelihoods and health, increasing incomes, and facilitating equitable, environmentally, socially and economically sustainable development. Achievement of Primary Goals: (a) Assess the effects of agricultural KST policy and institutional environments, as well as practices, in the context of sustainable Modest development. (b) Identify where critically important information gaps exist in order to Modest more effectively target research. (c) Make the resulting state of the art, objective, analyses accessible to decision makers at all levels from small producers to those who Modest create international policy. (d) Further the capacity of developing country nationals and institutions to generate, access, and use agricultural KST that promote Negligible sustainable development. Overall Efficacy Rating Modest Efficiency Modest Overall Program Outcome Moderately Unsatisfactory 28 3.55 Out of the seven objectives and goals, the achievements of five were rated modest, one substantial, and one negligible (Table 4). The highest weighting is given to those objectives and goals closest to impact -- objective (2) and goals (c) and (d), two rated modest and one negligible -- since the value of process is measured in terms of outcome or reasonable expectation of outcome. Therefore, the overall Efficacy rating is modest, in other words insufficient to warrant a substantial rating. Similarly, Efficiency is rated modest. With Relevance rated substantial and Efficacy and Efficiency both rated Modest, the overall Outcome rating is moderately unsatisfactory. 4. Governance, Monitoring and Evaluation, and Sustainability Governance, Management, and Resources 4.1 Governance and management are assessed here against three criteria: (a) legitimacy, (b) accountability, and (c) transparency and fairness. 4.2 Legitimacy. The evolution of the IAASTD program design involved a shift towards broader representation compared to other studies. It opened to scrutiny "several complex relations that might need to be understood if we were to offer decision makers viable options in the generation of policy" (Feldman, Biggs, and Raina 2010). This review does not give as high a weighting to process as it does to outcome since outcome is measured in terms of achievement of ends rather than means. However, to the extent that process adds legitimacy, it contributes to objectives related to product quality in the same way that decision makers perceptions about the product will gradually come to influence, one way or another, global legitimacy. 4.3 At initiation, there was inevitably narrower legitimacy with fewer players. While this could be interpreted as suggesting that there was little interest in the broader canvas, both the Bank and FAO were persuaded that broadening the assessment was appropriate. For some years there had been significant concern within the two founding players about environmental issues in agriculture. The final stamp of legitimacy from the full range of players came at the Nairobi meeting in August-September 2004, although NGOs and some other players had some misgivings about the process at the time. 4.4 There is an aspect to global legitimacy that is differentiated and related to societal norms. In the late twentieth century there emerged a crisis of confidence in science in Europe, revealing that Europe has a more risk-averse attitude towards science-based policy making, particularly over the issue of genetic engineering. A wide divergence emerged between America and Europe (Skogstad and Hartley 2007) suggesting a deeper societal legitimacy for the process in Europe than in America, where there is a more benevolent view of science. This divergence made the writing of the Sub-Global Report for North America and Europe challenging. 4.5 Accountability. Accountability had some weaknesses. Perhaps almost inevitably in a broad stakeholder, public policy, assessment, there was no strong locus of report content management. In theory, this was the role of the Bureau. However, since there were factions 29 within the Bureau, and limited time, the Bureau was not able to fully perform this role, given the opposing views. Bureau meeting documents show more focus on processes. Substance was more influenced by authorship profile. There were complaints (both in the IE findings and the interviews) that some authors did not adhere to agreed outlines and some did not accommodate changes agreed at meetings. Two interviewees with views divergent from the main emerging thesis complained of becoming persona non grata. Some participants felt that the lead authors did not sufficiently filter or mold material submitted to them. In the perception of some observers, the Review Editors did not play a firm enough role in conflict resolution and in ensuring that peer comments were either accommodated or rejected with a reason -- a potential source of bias. Some interviewees complained that their comments were largely ignored. There is some evidence that these issues may have arisen partly because of pressing time constraints towards the end of the process. The Director was widely praised for his influence in trying to reach consensus, but by design he was not controlling content.50 4.6 The overall logistics management of the process appears to have been efficient,51 although there were some concerns about the power which the Secretariat had in selecting authors. As one participant noted, "The central secretariat did a major job in stimulating and coordinating our work" although they were also overstretched by the scale of the enterprise. Based on the program files, a substantial logistics task was accomplished. 4.7 Transparency and Fairness. Satisfaction with the process was very variable by group. In the IARC/Technical Group, 46 percent found the effectiveness in the selection of authors below average compared to only 8 percent in the NGO group (IE, Question 22g).52 On this issue, there were concerns expressed about the loss of important authors due to lack of travel support. A number of authors dropped out.53 There were complaints about changes in the composition of writing teams and the tight timetable which led to reports that had been supposed to feed later findings being prepared in parallel.54 The IE found that, for some participants, there was confusion about what was expected and about the purpose of "an assessment." 50. For example, as noted, he had wanted both the past net environmental impact analysis and the future scenarios included but was not able to impose either. 51. There were some problems with differences in expense-claiming rules between the Bank and the Sub-global IAASTD institutions. 52. Formally the process of author selection was that the Secretariat received author nominations for the Coordinating Lead Authors, Lead Authors, Contributing Authors, Expert Reviewers, and Review Editors from governments, participating organizations, and other sources, including self-nomination through the Web site. After a process of winnowing, a list of names was submitted to the Bureau for review, a proposed name being rejected if four Bureau members objected. 53. The IE survey found that about 40 percent of those who dropped out did so because they had difficulty with the direction of the study, 33 percent because of difficulty with the process, 31 percent because of other work commitments, 25 percent because they felt that their ideas were not represented in the process, 19 percent because they were no longer contacted by the Secretariat, and 17 percent due to lack of financial incentive or reimbursement of costs. For a number, especially in CWANA region, language was a problem. 54. For example, from the NAE sub global group, a correspondent reported that the summary for decision makers was written without anyone from the chapter being present and before the chapter was finished. (IE, p.115). There were similar complaints from SSA. 30 4.8 CSOs seem to have strongly influenced author nomination. As noted by the IE (p. 102) "tapping a wider pool of potential authors and formally nominating them was a strategic action by a group of CSOs. A second round of contacting potential candidates took place when "not enough heavyweights had come forward."55 One author interviewee said, "The selection of contributing authors was done primarily based on their linkages and not necessarily on the levels of work done in the area." Another felt that the Secretariat held the predominant influence. It appears to have been partly a process of nomination by like-minded people within an epistemic community, described by one as "a battle for control of the pen." Some private sector representatives felt strongly that they had not had a fair opportunity. 4.9 For an assessment aimed at impact on poverty and sustainability at grass roots levels, some groups appear to have been over-represented such as academia in general. In the author and review editor list 41 percent were from universities. IFPRI and the Pesticide Action Network seem to have had a disproportionate share of authors and reviewers. On the Bureau, membership within some categories was shallow. Out of the 9 NGOs, two were either effectively the same organization or very close.56 Of the 9, only one is not an internationally operating NGO. As noted earlier, a greater share of country-level NGOs on the Bureau and in authorship would have offered more local level perspective and legitimized the local knowledge element of the assessment. Monitoring and Evaluation (M&E) 4.10 Both the quality of M&E design and the quality of M&E implementation are rated unsatisfactory, and therefore M&E is rated unsatisfactory in the aggregate as well. There was no real attempt at monitoring and evaluation. The design of the indicators in the GEF Project Document by the Bank and Secretariat was unsatisfactory, since these reflected a set of objectives further down the road beyond outcomes to national-level natural resource impacts than was realistic given the timeframe and resources (Table 5). Moreover, coming one and a half years after the First Plenary in Nairobi, these drew from objectives that had not been discussed at that original meeting. These GEF objectives and indicators were misleading to those who approved the project. The GEF Project Document called for independent mid-term and final evaluations by external evaluators to assess the project's performance in achieving strategic goals and objectives. While no mid-term external evaluation was carried out, this was a sound decision since the time remaining by then was too short. However, an earlier independent mid-term progress paper might have flagged some process issues for correction. On implementation, no attempt was made to collect data to meet the needs of the more complex of the output indicators although some were straightforward, simply calling for completion of a milestone. One donor representative interviewed expressed particular concern about the lack of M&E. The final IE, managed by the Agriculture and Rural Development Department of the World Bank and thus outside the IAASTD program, was appropriately initiated and supervised. 55. Benny Heaerlin, Report on Montpelier Meeting (www.agassessment-watch.org). 56. Practical Action (originally Intermediate Technology Development Group) and another representative from Intermediate Technology Development Group, although listed in the IAASTD documents as "International" Technology Development Group. 31 Table 5. GEF Project Document: Key Performance Indicators Objectives Outcome Indicators Development Objective Appropriate policies, regulations, incentive structures, are To improve access to agricultural KST that developed to support sustainable agricultural management will promote and facilitate sustainable The capacity of institutions is strengthened to design and agricultural practices with the aim to improve implement sustainable management approaches nutritional security, enhance rural livelihoods while reversing environmental degradation, Investments are made to address local, national, and global redressing social and gender inequity, and environmental issues within the context of sustainable ensuring human health and well-being. development Improvement of agricultural productivity under sustainable management, while enhancing the livelihoods of producers Global Environment Objective Appropriate policies, regulations and incentive structures are Apply the knowledge generated through the developed to support integrated natural resource management IAASTD to provide significant local, national, The capacity of institutions strengthened to design and regional and global environmental benefits, implement integrated natural resource management including reducing the overall rate o f natural resource loss and land degradation, Overall rate of natural resource loss and land degradation enhancing landscape biodiversity both in reduced areas of land use and in protected areas, Overall landscape biodiversity enhanced both in areas of land reducing the rate of soil, runoff and nutrient use and in protected areas losses, diminishing the contamination and Rate of soil, runoff and nutrient losses reduced eutrophication of fresh waters and soils, Contamination and eutrophication of fresh waters and reducing the rate of greenhouse gas contamination of soils diminished emissions, and increasing the amount of Rate of greenhouse gas emissions reduced, and more carbon carbon fixed in agricultural systems. fixed from agricultural systems Source: GEF Project Document, March 6, 2006, pp. 5­6. 4.11 The lack of M&E can be somewhat qualified. In a program with a short, predetermined life, a relatively short implementation period and an end product in the form of report delivery, there are somewhat different expectations of M&E from longer-term network-type global programs. On the other hand, a short life span also calls for rapid and effective implementation feedback. Sustainability of Outcomes 4.12 Overall, sustainability is rated moderately unlikely, implying that there are risks for sustainability. In accordance with GEF criteria, this is assessed on financial/economic, sociopolitical, institutional, and environmental criteria. This requires an assessment of the capacity of any potential second-round players to take up the IAASTD findings and to expand or sustain the defined outcomes. 4.13 Financial/Economic Sustainability. The main potential funding agencies to support communication of IAASTD findings seem likely to be among the following: governments (in particular through investment in agricultural research and extension), multilateral and bilateral donors through projects, CSOs, private companies, the CGIAR institutions, and farmer 32 organizations. Many governments endorsed the findings but so far there seems to be no evidence that national research strategies in developing countries are likely to be modified in response to IAASTD findings. As noted, some of those in national public research systems found the material too general. Private companies in the GMO business seem very unlikely to use the findings given their concerns about the process and findings. However, organic agriculture firms and associations may find areas for support. The CGIAR institutions seem unlikely to significantly redirect priorities in response, since there has already been a shift in the directions proposed, in the view of some, away from CGIAR's comparative advantage. Some donor projects may be influenced to some degree, but based on discussions with World Bank rural staff, there is limited interest partly because the findings are so general and partly because there are other sources of strategic creativity. CSOs are the most likely source of continued investment in the directions proposed by IAASTD, but their funds are limited. There are a few initiatives that, if not actually triggered by IAASTD, may complement the findings. For example, the International Institute of Environment and Development (IIED) is undertaking a study on the democratization of food and agricultural research that may take forward some of the themes. Some farmer organizations may be interested, but these are more likely to be the organic farming organizations. The broader national farmer associations and international farmer federations played a very limited role and do not appear to have shown much interest. However, there are many activities at the country level that already focus on many of the issues raised,57 as much among the IAASTD findings is not new. Whether the findings themselves will in some way redirect or trigger extension is difficult to assess at this point. 4.14 Sociopolitical Sustainability. The immediate sociopolitical risk, that the level of stakeholder ownership is insufficient to sustain the outcomes of the program, would be linked to financial, institutional, and environmental sustainability. At the present time, sustainability is questionable on all. 4.15 Institutional Sustainability. The IAASTD mechanism has closed. Other than passing on the Web site to UNEP, nothing has been passed on to any other agency. Whether any particular set of global or national institutions is likely to take up the challenge has been discussed above under the financial heading. 4.16 Environmental Sustainability. An important aspect of sustainability relates to environmental counterfactuals. One hypothesis would be that the failure to adopt the IAASTD findings will lead to increased environmental damage such as degradation of land or water through more intensive high-input agriculture and weak achievement in addressing hunger and poverty. Another hypothesis would be that widespread adoption of the dominant IAASTD options will lead to reduced productivity gains and more environmental damage through extensification. In the absence of either the historical analysis of net environmental impact or of more comprehensive scenario modeling for the future, it is difficult to place relative values on such risks. However, on balance the present review finds that the downplaying of the potential for new technology relative to existing technologies raises questions about the risks to productivity and the potential effects on environment through 57. For example, such activities as the Tanzania Selian Research Station participatory zero tillage work on maize/legume mixtures with a group of farmers near Karatu that warrants replication, or more broadly, the conservation tillage work in Zambia. 33 extensification, and about the risks of perceptions by policy-makers that investments in research and development of new technology may be dispensable. 5. World Bank Performance as a Partner The Multiple Roles of the World Bank 5.1 The Bank's roles in the IAASTD process were as convener, sponsor and financial contributor, trustee (both fiduciary and executor), host of the Secretariat, supervisor of the GEF project, and, by default, commissioner of the external evaluation. We address each in turn. 5.2 As a convener, the Bank was effective in getting the interested parties together and it was highly flexible in broadening the range of stakeholders. Of all the possible conveners, the Bank was the obvious option to initiate and sponsor for such a global assessment along with FAO. Bilateral donors and CSOs are too small or too narrow in roles. Other international institutions do not have as much breadth of experience of agriculture, although they may have more depth in particular areas or a more acceptable image with some stakeholders. Research networks such as the Global Forum on Agricultural Research (GFAR) are too small.58 Whether the Bank had a comparative advantage in some of the softer social areas and in local knowledge issues is more questionable, but the IAASTD was to involve many players with strengths in different areas. 5.3 As a sponsor and contributor, the Bank provided a substantial share of funding and made considerable efforts to raise funds from several different sources. Once the exercise was embarked on, this was an inevitable responsibility, but the Bank performed it well. As a trustee also, the Bank managed the process prudently and appropriately. As a host to the Secretariat, the Bank provided full services to the staff and gave comprehensive and well managed assistance. 5.4 As host of the Secretariat and supervisor of the GEF project, the Bank's oversight roles reveal both challenges and conflicts of interest. First, while, in theory, the Bank supervised the Director, the Bank had no control other than through its role in fiduciary management of the funding mechanisms. There was no Bank role in the IAASTD process or in monitoring product quality. Second, the task team leader, a Bank designated role, was, for most of the period, a member of the Secretariat. This meant that the Secretariat was, at the task team leader level, supervising itself. However, the ARD Director was signing off and did, on two occasions, make comments on supervision reports.59 There were five supervision reports (ISRs) in the system with some issues of accuracy. They list, under indicators, the 58. Such networks were poorly represented among participants although GFAR was asked to be an ex officio Bureau member at a late date. 59. In the next but last ISR, the ARD Director cautions that, "it may be too early to fully conclude that the overall objective of the project, "to improve access to agricultural KST that will promote and facilitate sustainable agricultural practices, with the goal of reducing hunger, poverty ... has been satisfactorily met." He notes that the IE under way will give a clearer view on "how much the assessment is influencing policy." 34 easier ones from the PAD to meet, covering a selection of the output-level indicators and avoiding the overall goal indicators, both environmental and developmental, which were much more challenging. The statement of the GEF Global Environment Objective in the ISRs is not the one in the PAD text and is almost identical to the Project Development Objective. 5.5 The GEF project was placed within the Bank's Europe and Central Asia Region as a host. At the GEF project design stage, in which the ECA Region participated in helping the Secretariat deliver the project to the Board, the Bank should not have designed or accepted the objectives and indicators set. These were unrealistic, unmeasurable, unattributable and lacked legitimacy, since they were not among those discussed at the opening plenary. In effect they were a retrofit. 5.6 As commissioner of the external evaluation, the Bank was thorough in insisting that the evaluation be undertaken, in commissioning the evaluation, and in giving guidance and support. The acting ARD Director played a pivotal role in reaching the agreement between OPCS and the GEF Evaluation Office to eliminate the overlapping evaluation requirements for projects supported by both the DGF and the GEF. (See Annex G.) 5.7 While the Bank made no commitment to take up the IAASTD findings and apply them in its rural development work, some interviewees expressed a hope that the Bank would take up some of the implementation in the directions proposed. There does not seem to be any move in this direction. 5.8 The Bank as well as other sponsors should have pushed more strongly during the early meetings for a communications strategy. This needed more planning and budgeting from the outset. It ended up as largely an ad hoc activity with whatever funds remained. Reputational Risks to the Bank 5.9 It is clear that once the Bank had committed to becoming an IAASTD sponsor and donor, it reached a point of no return after which it was on board for the ride wherever this might lead. This was not a global program which the Bank could influence during the process, since the Bureau oversaw the process and the authors and reviewers were nominated and expected to act as their skills dictated. Should the Bank have allowed itself to be drawn into a process over which it had no control and into a program for which the objectives had not been set? The present review concludes that it was a high-risk decision and that a more phased and cautious approach would have better managed reputational risk. Closeness to a process reduces deniability, distance eases it. While the issues were relevant, there was no pressing need for the Bank which demanded immediate and comprehensive action. 5.10 There was inevitably a high reputational risk for the Bank with a topic as wide as agricultural technology and including GMOs, trade and organic agriculture. Given that the IAASTD was never likely to resolve how to design GMO policy, the Bank seems to have taken too much risk. The fact that an Agriculture WDR came along to overlap with it increased the risk in one direction but decreased it in another. On the one hand, this increased the risk to the messages of the WDR by offering alternative findings that were not consistent. 35 On the other hand, the WDR reduced the risk to the Bank of the IAASTD findings in offering a clear Bank position. It could be argued that IAASTD offered an opportunity to capture a broader view partly outside the traditional areas of Bank focus. It also could be argued that in such a contentious area it was too high a risk given the lack of influence the Bank would have. This review inclines towards the latter argument. There were less risky, more measured, options open to the Bank in order to listen. 5.11 In the event, with the earlier release of the WDR and the pending release of the IAASTD reports, the Bank's Agriculture and Rural Development Sector Board had to examine where these two reports, perceived by many to be coming out of the same institution, diverged in case there was a need for damage control. It has been argued that the two reports are complementary. This is difficult to see. Some findings are similar and could be seen as either duplication or reinforcement; others, particularly the undercurrent of doubts about new technology, largely conflict. 5.12 Once the Bank had agreed to sponsor it, the Bank largely succeeded in keeping the process at arm's length, except in two areas. First, the position of the Director, who was initially a Bank staff member (until mid-2007) with other Bank duties as well, placed the Bank reputationally very close to this controversial topic. This was particularly the case since sponsors were not represented on the Bureau and were not supposed to influence the substance of the process.60 Second, regular Bank staff members seem to have influenced the authorship profile, being involved in the initial short-listing of authors. This brought the Bank closer to the process than was desirable. 5.13 The Bank should have held the IAASTD more at arms' length for three reasons. First, the topics were such that outsiders and participants were likely to exploit the Bank's association.61 Second, it would be politically embarrassing for the Bank to dissociate itself. Third, there was to be a single product not an ongoing process as in other network-type global programs that might have been amenable to some influence over time. Exit Strategy 5.14 From the outset, this was a finite exercise so there was an already planned exit. However, exit should have been linked more closely from the outset to a communications completion milestone since the outreach objectives were expected to drive the intended outcomes. 6. Lessons Learned 6.1 The IAASTD was a useful experience at the nexus of politics and science. However, agricultural technology, with its complexity, diversity and politics, proved to be a bridge too 60. Although one interviewee complained that the Bank had more influence than the others. 61. There is evidence that the Bank has been perceived as closely associated with the products. For example, a number of NGO Web sites flag the Bank sponsorship and imply Bank ownership. From another direction, an open internet letter to the World Bank President criticizing the IAASTD reports, calls on the Bank for a statement either of support or opposition to clarify the Bank's position (DeGregori, 2009). Another example is on the Web site: www.bioscienceresource.org. 36 far. The process itself was instructive and there is much useful information in the reports. However, the present review concludes that, for the substantial resources used, the IAASTD did not offer sufficient new knowledge or conceptual frameworks for decision makers, it gave conflicting messages, and, for a 50 year time span, it under-estimated the potential of new technologies relative to existing technologies. So far, attributable impact at the international level has been modest at best, and at the national level and below, negligible. This may change but time seems to be running out. 6.2 At the output level, stakeholders were brought together (objective 1), but there was limited representation of farmers and those closest to them. There was a predominance of northern academics over southern development practitioners, and at least in the Bureau, of international NGOs over national and local NGOs, and, out of six representatives of producer groups on the Bureau, only one that had a background to speak for intensive cropping agriculture. While IAASTD had broader representation than major prior studies, it staked a claim to breadth that others did not, particularly in the direction of local knowledge, representation of which was scant. Clearly an assessment was undertaken (objective 2) and reports were issued, but the present review assumes a quality dimension to this objective. The robustness of policy-relevant information for decision makers (objective 3) is questionable. The lack of a conceptual framework or country or situational classification for decision makers makes it difficult to apply the largely undifferentiated and variably treated lists of policy options. A focus on a conceptual framework and decision processes as opposed to technological choices -- such as high external input versus organic, or the acceptance or avoidance of GMOs -- could have redirected the conflict away from confrontation towards the means of resolving strategic issues over the longer term. There is an undercurrent against new technology. There are conflicting statements and signals particularly on the more contentious issues. It is difficult to trace evidence since the summary reports only refer very generically to chapters in the global report. Overall, for the costs, the findings do not offer enough that is new. 6.3 At the outcome level, there is a large volume of information in IAASTD (goal a). Chapter 3 particularly offers important findings, well presented within a risk, potential impact, and scale framework. However, the original intent in assessing the effects of "AKST" was to look both to the past and to the future. For two reasons, this was only modestly achieved. Looking back at the past, there was no net assessment of the impact of past intensification on the environment. Looking ahead to the future, the planned modeling scenarios were dropped on the grounds of opacity and locus of control, and no convincing qualitative vision was substituted beyond a general sense of moving away from input intensive agriculture, although some simplified quantitative scenarios survived. The reports lacked a coherent story line from the past into the future. On the main disputed issues of particular interest to the World Bank, GMOs and the potential of organic agriculture, no convergence was achieved. There was limited systematic identification of information gaps (goal b) and wide variation by chapter and section in treatment of that goal. 6.4 On accessibility (goal c), there was a significant problem with the Web versions being unavailable for six months under an agreement with the publisher -- a crucial period in which to gain brand recognition. The majority of participant respondents in the IE survey found the documents useful for discussion with policy makers although no interviewee for 37 this review seemed to have had such a discussion. Non-participants surveyed in Africa and a number of other interviewees found other documents such as the 2008 World Development Report on Agriculture more useful. At the international level there has been some use of IAASTD documents as references, but there have been some notable "cold shoulders" in crucial international forums. No evidence has so far been found to suggest that IAASTD outputs have a reasonable probability of yielding attributable outcomes in due course at the national level. One conclusion might be that time will tell. Another, which seems more plausible at this point, is that time is running out. On capacity development (goal d), this activity was never attempted and the Capacity Development chapter which at least would have pointed a route towards outcomes was dropped. 6.5 The assessment aimed not to be policy prescriptive. To the extent that it advocates, through the range of conflicting signals, a predominantly low-input, organic, small farmer, non-GMO, local knowledge, environmentally focused approach, it did, in fact, prescribe. Whether it was desirable to avoid prescription of any sort is questionable. A degree of prescription would have been helpful to decision makers, but this would have been prescription of a conceptual and decision framework backed by evaluation of alternative approaches rather than disparate lists of possible policy and investment options that by their nature cannot prescribe for any particular country environment. A focus on a conceptual framework and decision processes as opposed to technological choices -- such as high external input versus organic or the acceptance or avoidance of GMOs -- could have redirected the conflict away from confrontation towards the means of resolving strategic issues over the longer term. 6.6 The issue of public science in a liberal democracy has been an increasing area of study and debate, some of this focused on the divergence between Europe and North America in attitudes to the authority of science. For the increasing role of democratic processes in scientific debate challenges the long-held principles of science of autonomy, impartiality, independence, and objectivity that have traditionally conferred a degree of privilege on science and scientists (Porter and Phillips, 2007, pp. 4­5). As an example of public science, the IAASTD process presented challenges both to science and to the "global citizens" represented at the table. Porter and Phillips ask two questions: "Is it possible ... for science to be so conceived that it contributes to the give and take of politics? Is it possible to have an educated citizenry sufficiently knowledgeable about science that political discourse can be strengthened?" The IAASTD experience offers some partial and tentative answers drawn, of course, from one case. 6.7 In many papers on public science and democracy there are more questions than answers on how to structure an assessment such as IAASTD. A few papers, however, offer some principles generally revolving around balance, disclosure, and due diligence. These proposed principles offer a framework that partly fits the findings of this review. In particular, the IAASTD experience offers six findings on process that may help to guide future such endeavors. First, it reconfirms that neither physical scientists nor social scientists, either between or within communities, can be expected to come to the same interpretations of peer reviewed evidence nor can they even be expected to draw from a common literature. 38 Second and related to the first, to ensure balance there is a need for a formal and documented author selection process, perhaps managed to the short list stage by a disinterested outside party. Third, for balance and due diligence, there is a need for a formal process and professional facilitation for the resolution of disputes and for defining the knowledge gaps. Fourth, for disclosure and due diligence, there is a need for readily traceable reference links between reports and a strong role for reviewers to challenge references62 and to propose alternatives. Fifth, for disclosure, there is a need for acknowledgment of the political differences encountered, their origins and the implications this may have for any future path towards closure. Sixth, for balance and due diligence, as well as in the interests of efficiency, there needs to be testing of alternative, more phased processes with potentially challenging issues being parked to one side to be assessed by smaller groups reporting for later wider consideration. In other words, there is a breadth versus depth trade-off evident that warrants more exploration in the direction of efficiency. Not dissimilar findings to the IAASTD have emerged from processes with far fewer participants 6.8 The present review concludes that IAASTD was a missed opportunity of considerable cost. The literature and case experience presented in the IAASTD reports, as well as the literature and development experience more widely, point towards a pluralistic strategy for agricultural research and technology development, along with rigorous evaluation to sift success from failure. But, partly due to the way in which the authors were selected and the main reports were translated into the summaries, the overall message which emerged from the IAASTD was a more restrictive, exclusionary message with an undercurrent against new technology, GMOs, and input-intensive agriculture. The IAASTD also failed to provide policy-makers with a conceptual framework that takes into account such factors as the stage of national, sectoral, and zonal development, local physical and social circumstances, and the unfolding of climate change. A more pluralistic strategy would include GMOs and input- intensive approaches alongside organic and low-external-input approaches. It would incorporate greater application of local knowledge alongside continued strengthening of basic and applied research, both public and private, even including the widely condemned linear top-down transfer-of-technology approach for particular purposes. Some participants claim that such pluralism was indeed the gist of the IAASTD messages. While almost any message can be read into the reports, one of pluralism with no options off the table and guided by strong evaluation is clearly not the message that won through.63 62. The review found somewhat limited reviewer influence on reference material although McKitrick's (2007) call for evidence to be vetted to the point of testing that it can be actually replicated by other scientists would be unrealistic and prohibitively costly. 63. Although somewhat differently framed, such pluralistic inclusiveness and iterative adjustment of ends against means was espoused a quarter of a century ago by Johnston and Clark (1982) and, more recently, pluralism is a feature of the 2009 UK Royal Society report Reaping the Benefits. 39 6.9 The lessons derived from the IAASTD experience are divided into lessons for future global public assessment programs and lessons for the World Bank. The lessons for assessment programs are as follows: Control of the pen has an enormous influence on product. Therefore there is a need for a rigorous and balanced formal process in the selection of authors and reviewers as independent as is realistically possible, perhaps monitored for fairness of the process by a disinterested institution. In complex assessments with anticipated areas of dispute, a phased process should be considered. One option might have been the following: Stage 1, a rigorous author selection process; Stage 2, small teams of authors would work on the disputed topics following carefully crafted TOR with funds provided to consult or take depositions widely; Stage 3, first stage findings would be tested, and amended where agreed, with the wider participant group; and Stage 4, the analysis and findings in less disputed areas would be brought together with the Stage 3 findings into the final products. With facilitation, these would be required to specify information gaps and to openly characterize the sources of dispute both technically and with respect to political stances. Technology options offered over a 50 year timeframe will inevitably depreciate given the short life of new technology. Such a long timeframe calls for a conceptual framework and decision processes tailored to the changing circumstances of time and place. The documents of a multi-stakeholder assessment of this type need to make explicit the main outstanding disputed assessment issues, the different interpretations of the evidence, the evidence gaps that might contribute to closure on those issues, and the political stances contributing to the opposing positions. The number of participants needs to be matched to the available budget so that full payment can be made to all participants. The alternative, with funding for some authors but not others, is a source of potential bias. Sufficient time needs to be provided for sub-reports to be completed to feed into the main report and then into the summary report. To manage this, one option would be some form of Computerized Project Management system to develop a realistically phased schedule, to better project the probable elapsed time and to monitor the critical path. Professional facilitation and support from professional editors could help resolve disputes and improve the impact of documents. The risk of facilitators or editors biasing the outcome seems likely to be lower than the risk of outcomes being biased from less guided and more rancorous dispute or from unclear articulation of, or interpretation of, unclear messages. 40 Closing an assessment immediately on production of the reports gives little chance for dissemination and follow-up. A dissemination strategy should be part of the program design and should be budgeted for and implemented with a defined exit milestone. 6.10 Lessons for the World Bank are the following: While the Bank should not shy away from controversy, it should not support studies of potentially very controversial issues where neither the process nor objectives can be influenced, when there is no compelling need, or until alternative options have been exhausted. Bank-managed reviews can still draw on wide stakeholder consultation. To answer the main instigating question about GMO policy and organic agriculture, or to address the broader objectives that were ultimately posed, there was no pressing need, in this case, to hand over the assessment to an outside team of uncertain ultimate representation responding to objectives which, at that point, were not even defined. When the Bank supports a global public program assessment in the area of public policy and science, it should focus from the outset on influencing adherence to some principles, in particular: openness, accountability, and procedural fairness. It should also attempt to ensure that the program is budgeted and planned to initiate up front and to then sustain, the communication of outputs. Closing an assessment simply with the issuance of reports leaves unfinished business. The higher the reputational risk to the Bank, the greater should be the separation of a program from the Bank. In this case, it was clear that this was a highly political topic with potential for findings that the Bank might disagree with. Yet it was managed and positioned very close to the Bank to the extent that the program management and the Bank supervision role were, at one point, almost indistinguishable. Due to the closeness of the program to the Bank, IAASTD reports were perceived by many outsiders as a product endorsed by the Bank. The stated objectives of a project should be realistic, monitorable, and achievable within the budget and within a reasonable timeframe, and with some prospect of demonstrating attribution. In this case, the GEF project had objectives and indicators designed by the Bank that were unrealistic and were arguably misleading to those who approved the project. Implementation Status and Results Reports should accurately reflect both outcome and output level indicators given in the appraisal documents unless there has been a formally agreed revision. 41 References Angelsen A. and D. Kaimowitz. Agricultural Technologies and Tropical Deforestation. CABI Publishing. Wallingford, UK. Asenso-Okyere et al. 2008. Advancing Agriculture in Developing Countries through Knowledge and Innovation. IFPRI. Badgley, C., J. Moghtader, E. Quintero, E. Zakem, M.J. Chappell, K. Aviles-Vazquez, A.Samulon and I. Perfecto. 2007. Organic agriculture and the global food supply. Renewable Agriculture and Food Systems 22: 86-108. Barnett, A., 2004 From "research" to poverty reducing "innovation." Sussex Research Association. Brighton. 2004. Biggs, S. 1989 A Multiple Source of Innovation Model of Agricultural Research and Technology Promotion. Agricultural Administration (Research and Extension) Network Paper 6, Overseas Development Institute. London. Civil Society Statement from Johannesburg, South Africa. 2008. A New Era of Agriculture Begins Today. 12 April, 2008. CropLife 2008. Science and Technology are Key to Growing More Food. Press Release. Brussels, 15 April 2008. DeGregori, T. 2009. Letter to the President of the World Bank. University of Houston. www.uh.edu. De Schutter, O. 2009, Enabling Policies for Development and Sustainability from the IAASTD. Side Event to the UN CSD Intergovernmental Preparatory Meeting. New York, February 25th, 2009. FAO. 2009. The State of Food and Agriculture. FAO. Rome. Feldman, S., S. Biggs, and R. Raina. A Messy Confrontation of a Crisis in Agricultural Science. Economic and Political Weekly. January 16, 2010. Vol. XLV. No. 3. Greenpeace Statement. Urgent changes needed in global farming practices to avoid environmental destruction. Amsterdam 15 April, 2008. Hailey 1938. An African Survey. A Study of Problems Arising in Africa South of the Sahara. Royal Institute of International Affairs. Oxford University Press. London. IFAD. Rural Poverty Report 2001. The Challenge of Ending Rural Poverty. Oxford University Press. Oxford. UK. IFPRI 2009a. Fifty Years of Progress. Chapter 3, IFPRI, Washington DC. IFPRI, 2009b. Measuring the Economic Impacts of Transgenic Crops in Developing Agriculture during the First Decade. The Policy Review No. 10, IFPRI, Washington DC. Independent Evaluation Group. 2007. Tanzania Agricultural Research II Project. Project Performance Assessment Report. Washington, DC. Johnston, B. and W. Clark. 1982. Redesigning Rural Development -- A Strategic Perspective. Baltimore, MD: Johns Hopkins University Press. McKitrick, R. 2007. Bringing Balance, Disclosure, and Due Diligence into Science-Based Policymaking. In Public Science and Liberal Democracy edited by Porter and Phillips University of Toronto Press. Polyani, M. The Republic of Science: Its Political and Economic Theory. In Knowing and Being, edited by M. Grene. University of Chicago Press. Porter, J. and Peter W.B. Phillips. 2007. Public Science and Liberal Democracy. University of Toronto Press. 42 Scoones, I. 2008. Global Engagements with Global Assessments: The Case of the International Assessment of Agricultural Knowledge, Science and Technology for Development (IAASTD). Working Paper 313. Institute of Development Studies, University of Sussex. Skogstad, G., and S. Hartley. 2007. Science and Policymaking: The Legitimation Conundrum. In Public Science and Liberal Democracy edited by Porter and Phillips University of Toronto Press. The Royal Society. Reaping the Benefits. Science and the sustainable intensification of global agriculture. October 2009. RS policy document. Tiffen, M., M. Mortimore, and F. Gichuki. 1994. More People, Less Erosion. Environmental Recovery in Kenya. ODI. John Wiley and Sons. UK. World Bank. Agriculture and the Environment. Perspectives on Sustainable Development. A World Bank Symposium. 1998. Washington, DC. World Commission on Dams. 2000. Dams and Development: A New Framework for Decision-Making. www.unep.org/dams/WCD/report/WCD_DAMS%20report.pdf. 43 Annex A Annex A. Evaluation Framework for Global Program Reviews Note: This evaluation framework is a general framework that has been designed to cover the wide range of such programs in which the World Bank is involved, encompassing policy and knowledge networks, technical assistance programs, and investment programs. It is not expected that every global program review will cover every question in this table in detail. Table A-1. Assessing the Independence and Quality of the Evaluation Evaluation Questions 1. Evaluation process To what extent was the GRPP evaluation independent of the management of the program, according to the following criteria: Organizational independence? Behavioral independence and protection from interference? Avoidance of conflicts of interest? Factors to take into account in answering these questions include: Who commissioned and managed the evaluation? Who approved the terms of reference and selected the evaluation team? To whom the evaluation team reported, and how the evaluation was reviewed? Any other factors that hindered the independence of the evaluation such as an inadequate budget, or restrictions on access to information, travel, sampling, etc.? 2. Monitoring and evaluation framework of the program To what extent was the evaluation based on an effective M&E framework of the program with: Clear and coherent objectives and strategies that give focus and direction to the program? An expected results chain or logical framework? Measurable indicators that meet the monitoring and reporting needs of the governing body and management of the program? Systematic and regular processes for collecting and managing data? 3. Evaluation approach and scope To what extent was the evaluation objectives-based and evidence-based? To what extent did the evaluation use a results-based framework -- constructed either by the program or by the evaluators? To what extent did the evaluation address: Relevance Governance and management Efficacy Resource mobilization and financial management Efficiency or cost-effectiveness Sustainability, risk, and strategy for devolution or exit 4. Evaluation instruments To what extent did the evaluation utilize the following instruments: Desk and document review Consultations/interviews and with whom Literature review Structured surveys and of whom Site visits and for what purpose: for interviewing implementers/beneficiaries, or for observing activities being implemented or completed Case studies Other Annex A 44 Evaluation Questions 5. Evaluation feedback To what extent have the findings of the evaluation been reflected in: The objectives, strategies, design, or scale of the program? The governance, management, and financing of the program? The monitoring and evaluation framework of the program? Table A-2. Providing an Independent Opinion on the Effectiveness of the Program Every review is expected to cover the first four criteria in the following table: (a) relevance, (b) efficacy, (c) efficiency, and (d) governance and management. A review may also cover (e) resource mobilization and financial management and (f) sustainability, risk, and strategies for devolution or exit if the latter are important issues for the program at the time of GPR, and if there is sufficient information available on which to base an independent opinion. Evaluation Criteria and Questions Relevance: The extent to which the objectives and design of the program are consistent with (a) current global/regional challenges and concerns in a particular development sector and (b) the needs and priorities of beneficiary countries and groups. 1. Supply-side relevance -- the existence of an international consensus that global/regional collective action is required. To what extent does the program reflect an international consensus on the need for action, on the definition of the problem being addressed, on priorities, and on strategies for action? Is the original consensus that led to the creation of the program still present? Is the program still needed to address specific global/regional public concerns? Take into account the origin of the program in answering these questions: Is the program formally responsible for implementing an international convention? Did the program arise out of an international conference? Is the program facilitating the implementation of formal standards and approaches? Is the program primarily donor-driven? Did donors establish the program with little consultation with developing countries? Is the program primarily Bank-driven? Did the World Bank found the program and then seek other partners? 2. Demand-side relevance -- alignment with beneficiary needs, priorities, and strategies. To what extent are the objectives consistent with the needs, priorities, and strategies of beneficiary countries as articulated in the countries' own PRSPs, and in donors' strategies such as the World Bank CASs, and the UN Development Assistance Frameworks? To what extent has the voice of developing and transition countries been expressed in the international consensus underlying the program? 3. Vertical relevance -- consistency with the subsidiarity principle. To what extent are the activities of the program being carried out at the most appropriate level -- global, regional, national, or local -- in terms of efficiency and responsiveness to the needs of beneficiaries? To what extent are the activities of the program competing with or substituting for activities that individual donors or countries could do more efficiently by themselves? Pay particular attention to those programs that, on the face of it, are primarily supporting the provision of national or local public goods. 45 Annex A Evaluation Criteria and Questions 4. Horizontal relevance -- the absence of alternative sources of supply. What is the comparative advantage, value added, or core competency of the program relative to other GRPPs with similar or complementary objectives? To what extent is the program providing additional funding, advocacy, or technical capacity that is otherwise unavailable to meet the program's objectives? To what extent are the good and services being provided by the program in the nature of public goods? Are there alternative ways of providing these goods and services, such as by the private sector under regular market conditions? 5. Relevance of the design of the program To what extent are the strategies and priority activities of the program appropriate for achieving its objectives? What are the major activities of the program: Policy and knowledge networking? Financing country and local-level technical assistance? Financing investments to deliver national, regional, or global public goods? (See Table A- 4.) Has the program articulated an expected results chain or logical framework, along with assumptions that relate the progress of activities with the achievement of the objectives? Does the results chain identify the extent to which the achievement of the objectives depends on the effective functioning of bureaucracies, markets, or collectivities? If so, to what extent are these assumptions valid? For programs providing global or regional public goods, is the design of the program consistent with the way in which the individual efforts of the partners contribute to the collective outcome for the program as a whole -- whether "best shot," "summation," or "weakest link?" Efficacy: The extent to which the program has achieved, or is expected to achieve, its objectives, taking into account their relative importance. 6. Achievement of objectives To what extent have the stated objectives of the program been achieved, or has satisfactory progress been made towards achieving these objectives? To what extent are there implicit objectives that are well understood and agreed upon by the partners and to which the program should also be held accountable? To what extent are there any positive, unintended outcomes of the program that have been convincingly document? To what extent have these assessments by the program or the evaluation been evidence-based? 7. Progress of activities, outputs, and outcomes. To what extent has the program or the evaluation measured the progress of activities, outputs, and outcomes? How did the program or the evaluation aggregate its outputs and outcomes at all levels -- global, regional, national, and local -- to provide an overall summary of its results? To what extent have factors such as changes in the location of the program, its legal structure, or governance processes affected the outputs and outcomes of the program? To what extent have there been outcomes that can be uniquely attributed to the partnership itself -- such as the scale of or joint activities made possible by its organizational setup as a GRPP, or its institutional linkages to a host organization? 8. Linkages to country or local-level activities. To what extent has the program established effective operational linkages with country-level activities, taking into account that: The desired nature of these linkages will vary according to the objectives, design, and implementation of each program? Positive outcomes at the country or local level are generally a joint product of both global/regional and county- level activities? Annex A 46 Evaluation Criteria and Questions Efficiency or cost-effectiveness: Efficiency -- the extent to which the program has converted or is expected to convert its resources/inputs (such as funds, expertise, time, etc.) economically into results. Cost-effectiveness -- the extent to which the program has achieved or is expected to achieve its results at a lower cost compared with alternatives. 9. Efficiency To what extent is it possible to place a monetary value on the benefits arising from the activities of the program? To what extent has the program or the evaluation conducted impact evaluations of representative program activities? To what extent has the program or the evaluation analyzed the program's costs in broad categories (such as overhead vs. activity costs), and categorized the program's activities and associated benefits, even if these cannot be valued in monetary terms? 10. Cost-effectiveness To what extent is the program measuring up against its own business plans: Has the program cost more or less than planned? How did it measure up against its own costing schedule? Have there been any obvious cases of inefficiency or wasted resources? To what extent is the program delivering its activities cost-effectively in comparison with alternatives: How do actual costs compare with benchmarks from similar programs or activities? Are the overhead costs of governing and managing the program reasonable and appropriate in relation to the objectives and activities of the program? How does the program compare with traditional development assistance programs: For beneficiary countries, has receiving the development assistance through the GRPP increased the transactions costs compared with traditional development assistance programs? For donors, has delivering the development assistance through the GRPP reduced donor costs by harmonizing efforts among donors or by reducing overlapping work (such as through joint supervision, monitoring and evaluation)? Governance and management: Governance -- the structures, functions, processes, and organizational traditions that have been put in place within the context of a program's authorizing environment to ensure that the program is run in such a way that it achieves its objectives in an effective and transparent manner. Management -- the day-to-day operation of the program within the context of the strategies, policies, processes, and procedures that have been established by the governing body. Whereas governance is concerned with "doing the right thing," management is concerned with "doing things right." 11. Compliance with generally accepted principles of good governance. To what extent are the governance and management structures and processes well articulated and working well to bring about legitimate and effective governance and management? To what extent do governance and management practices comply with the following seven principles: Legitimacy -- the way in which governmental and managerial authority is exercised in relation to those with a legitimate interest in the program -- including shareholders, other stakeholders, implementers, beneficiaries, and the community at large? Accountability -- the extent to which accountability is defined, accepted, and exercised along the chain of command and control within a program, starting with the annual general meeting of the members or parties at the top and going down to the executive board, the chief executive officer, task team leaders, implementers, and in some cases, to the beneficiaries of the program? Responsibility -- the extent to which the program accepts and exercises responsibility to stakeholders who are not directly involved in the governance of the program and who are not part of the direct chain of accountability in the implementation of the program? 47 Annex A Evaluation Criteria and Questions Fairness -- the extent to which partners and participants, similarly situated, have equal opportunity to influence the program and to receive benefits from the program? Transparency -- the extent to which a program's decision making, reporting, and evaluation processes are open and freely available to the general public? Efficiency -- the extent to which the governance and management structures enhance efficiency or cost- effectiveness in the allocation and use of the program's resources? Probity -- the adherence by all persons in leadership positions to high standards of ethics and professional conduct over and above compliance with the rules and regulations governing the operation of the program? 12. Partnerships and participation To what extent has the program identified a complete list of stakeholders, or "stakeholder map," including the agreed- upon or perceived roles and responsibilities of the categories of stakeholders identified? To what extent is this a routine programmatic function, updated regularly, and transparently available? Has the program adopted primarily a shareholder model of governance (in which membership on the governing body is limited to financial and other contributors), or a stakeholder model (in which membership also includes non- contributors)? To what extent, if any, is the program's legitimacy being sacrificed in order to achieve greater efficiency, or vice-versa? 13. Programs located in host organizations To what extent is the location of the program in the Bank or other partner organization adversely affecting the governance, management, or other aspects of the program, such as compliance with the principles of transparency and fairness? For which functions is the program manager accountable to the host organization and the governing body of the program, respectively? Are conflicts of interest being managed appropriately? To what extent does the host organization play such a dominant role in the program, thereby reducing the incentives of other partners to participate effectively, or reducing the ability of the host organization to look at the weaknesses of the program objectively? Resource mobilization and financial management: Resource mobilization -- the processes by which resources are solicited by a program and provided by donors and partners. Financial management -- the processes that govern the recording and use of funds, including allocation processes, crediting and debiting of accounts, controls that restrict use, accounting, and periodic financial reporting systems. In cases where funds accumulate over time, this would also include the management of the cash and investment portfolio. 14. Resource mobilization To what extent has the program succeeded in raising financial resources commensurate with its objectives? And from what sources -- the Bank, bilateral donors, foundations, etc.? To what extent has the program succeeded in diversifying its funding beyond a small number of donors? To what extent are the sources of funding for the program (including donor restrictions on the use of resources) affecting, positively or negatively: The strategic focus of the program? The outputs and outcomes of the program? The governance and management of the program? The sustainability of the program? 15. Financial management Are there any issues that have emerged during the course of the review in relation to: The quality of financial management and accounting? The methods, criteria, and processes for allocating funds among different activities of the program? Financial management during the early stages of the program? Annex A 48 Evaluation Criteria and Questions Sustainability, risk, and strategy for devolution or exit: Sustainability -- When applied to the activities of a program, the extent to which the benefits arising from these activities are likely to continue after the activities have been completed. When applied to a program itself, the extent to which the organization or program is likely to continue its operational activities over time. Devolution or exit strategy -- a proactive strategy to change the design of a program, to devolve some of its implementation responsibilities, to reduce dependency on external funding, or to phase out the program on the grounds that it has achieved its objectives or that its current design is no longer the best way to sustain the results which the program has achieved. 16. Sustainability of the benefits of the program's activities What is the risk, at the time of evaluation, that the development outcomes (or expected outcomes) of the program will not be maintained (or realized)? This depends on (a) the likelihood that some changes may occur that are detrimental to maintaining or realizing the expected outcomes, and (b) the affect on the expected outcomes if some or all of these changes actually materialize? 17. Sustainability of the program This will depend on a number of factors, such as the continued legitimacy of the program, its financial stability, its continuity of effective management, and its ability to withstand changing market or other conditions. To what extent is there still a sufficient convergence or accommodation of interests among the major partners to sustain the program financially? To what extent has the program developed institutional capacity such as performance- based management, personnel policies, learning programs, and knowledge management that help to sustain a program? In what areas could the program improve in order to enhance its sustainability, such as better marketing of the program's achievements in order to sustain its reputation? 18. Prospects for continuation and strategies for devolution or exit To what extent should the program be sustained? Is the continuation of the program the best way of sustaining the results achieved? Should the design of the program be modified as a result of changed circumstances, either positive or negative? What other alternatives should be considered to sustain the program's results more cost-effectively, in the light of the previous evaluation findings with respect to relevance, efficacy, efficiency, and sustainability: Reinventing the program with the same governance? Phasing out the program? Continuing country or local-level activities with or without devolution of implementation? Seeking alternative financing arrangements, such as revenue-generation, or self-financing to reduce dependency on external sources? "Spinning off" from the host organization? 49 Annex A Table A-3. Assessing the Bank's Performance as a Partner in the Program Evaluation Questions 1. Comparative advantage at the global/regional level. To what extent is the Bank playing up to its comparative advantages at the global/regional level -- its global mandate and reach and convening power? To what extent is the Bank's presence as a partner in the program catalyzing other resources and partners for the program? 2. Comparative advantage at the country level. To what extent is the Bank contributing multi-sector capacity, analytical expertise, and country-level knowledge to the program? To what extent has the Bank's country operations established linkages to the GRPP, where appropriate, to enhance the effectiveness of both? 3. Oversight. To what extent is the Bank exercising effective and independent oversight of its involvement in the program, as appropriate, whether the program is housed in the Bank or externally managed? To what extent is the Bank's oversight independent of the management of the program? To what extent does the Bank's representative on the governing body have a clear terms of reference? 4. Risks and risk management. To what extent have the risks associated with the program been identified and are being effectively managed? For example, IEG identified the following risks in its global review: Bank bears a disproportionate share of responsibility for governing and managing in-house programs? Confusion at the country level between global program activities, Bank activities, and Borrower activities? Representation of NGOs and the commercial private sector on program governing bodies? Unclear role and application of Bank's safeguards? Trust-funded consultants and seconded staff representing the Bank on some program governing bodies? 5. Disengagement strategy. To what extent is the Bank engaged at the appropriate level in relation to the Bank's new strategic framework: Watching brief? Research and knowledge exchange? Policy or advocacy network? Operational platform? To what extent is the Bank facilitating an effective, flexible, and transparent disengagement strategy for the program, in relation to the Bank's objectives for its involvement in the program: The program declares "mission accomplished" and closes? The program continues and the Bank withdraws from all aspects of its participation? The program continues and the Bank remains engaged, but the degree of the Bank's engagement in some or all aspects (such as financing) declines over time? Annex A 50 Table A-4. Common GRPP Activities Policy and knowledge networking 1. Facilitating communica- This includes providing a central point of contact and communication among practitioners tion among practitioners in who are working the sector or area of development to facilitate the sharing of analytical the sector results. It might also include the financing of case studies and comparative studies. 2. Generating and This comprises two related activities. The first is gathering, analyzing and disseminating disseminating information and information, for example, on the evolving HIV/AIDS epidemic and responses to it, including knowledge epidemiological data collection and analysis, needs assessment, resource flows, and country readiness. The second is the systematic assembling and dissemination of knowledge (not merely information) with respect to best practices in a sector on a global/regional basis. 3. Improving donor This should be an active process, not just the side effect of other program activities. This coordination may involve resolving difficult interagency issues in order to improve alignment and efficiency in delivering development assistance. 4. Advocacy This comprises proactive interaction with policymakers and decision makers concerning approaches to development in a sector, commonly in the context of global, regional, or country-level forums. This is intended to create reform conditions in developing countries, as distinct from physical and institutional investments in public goods, and is more proactive than generating and disseminating information and knowledge. 5. Implementing Rules are generally formal. Standards can be formal or informal, and binding or nonbinding, but conventions, rules, or formal implementing standards involves more than simply advocating an approach to development in a and informal standards and sector. In general, there should be some costs associated with noncompliance. Costs can come norms in many forms, including exposure to financial contagion, bad financial ratings by the IMF and other rating agencies, with consequent impacts on access to private finance; lack of access to OECD markets for failing to meet food safety standards, or even the consequences of failing to be seen as progressive in international circles. Financing technical assistance 6. Supporting national-level This is more directed to specific tasks than advocacy. This represents concrete involvement policy, institutional, and in specific and ongoing policy, institutional, and technical reform processes in a sector, from technical reforms deciding on a reform strategy to implementation of new policies and regulations in a sector. It is more than just conducting studies unless the studies are strategic in nature and specific to the reform issue in question. 7. Capacity strengthening This refers to strengthening the capacity of human resources through proactive training (in and training courses or on-the-job), as well as collaborative work with the active involvement of developing country partners. 8. Catalyzing public or This includes improving regulatory frameworks for private investment and implementing private investments in the pilot investments projects. sector Financing investments 9. Financing country-level This refers primarily to physical and institutional investments of the type found in Bank investments to deliver loans and credits (more than the financing of studies), the benefits of which accrue primarily national public goods at the national level. 10. Financing country-level This refers primarily to physical and institutional investments of the type found in Bank investments to deliver loans and credits (more than the financing of studies) to deliver public goods such as global/regional public goods conserving biodiversity of global significance and reducing emissions of ozone-depleting substances and carbon dioxide, the benefits of which accrue globally. 11. Financing global/regional This refers to financing research and development for new products and technologies. investments to deliver These are generally physical products or processes -- the hardware as opposed to the global/regional public goods software of development. 51 Annex B Annex B. Timeline Date Location Type of Meeting Aug 2002 Various Initiating meetings within World Bank and between WB and FAO Nov 6-8, 2002 Dublin, Ireland Steering Committee First half of Consultations 9 Consultative meetings 2003 Jan 31, 2003 Nairobi, Kenya Assessment endorsed Mar 31 ­ April Paris, France European Regional Consultation 1, 2003 August ­ Nairobi, Kenya Opening Plenary September, 2004 March 2006 Washington, DC GEF Project document 2003 to May Various 4 Bureau Meetings 2007 Nov 2005 to Various 4 Global Authors Meetings June 2007 Nov 2005 to Various 20 Sub-Global Authors Meetings June 2007 June 2007 Cape Town Synthesis and SDM meeting Aug 2007 Colombo Synthesis and SDM meeting Oct 2007 Tunis Synthesis and SDM meeting April 2008 Johannesburg Final Inter-governmental Plenary June 2009 Washington DC Independent Evaluation Source: IAASTD, the IAASTD Web site, and the 2009 Independent Evaluation. Note: This covers only the main events. Annex C 52 Annex C. Terms of Reference for Independent Evaluation Independent Evaluation of The International Assessment of Agricultural Science and Technology for Development (IAASTD) TERMS OF REFERENCE 1. Introduction 1.1 Background Between 2001 and mid-2002, the World Bank held meetings with the leaders in the private sector and in civil society to discuss prominent issues in agricultural science and technology. Participants suggested that a comprehensive, multidisciplinary assessment of issues critical to policy formulation would have great value for decision makers confronting conflicting views on agricultural science and technology, practices, policies and institutional effectiveness. Recognizing how important such an international assessment could be to achieving the Millennium Development Goals in August 2002, FAO and the World Bank announced at the World Summit on Sustainable Development a global consultative process on a proposed international assessment of the role of agricultural science and technology in reducing hunger, improving rural livelihoods and stimulating environmentally sustainable economic growth over the coming decades. Representatives of FAO, UNDP, UNEP, UNESCO, the World Bank, WHO, and three Multilateral Environmental Agreements as well as representatives from governments, the private sector, producer and consumer groups, non-governmental organizations, international organizations, foundations, the CGIAR and other scientific organizations from around the world attended the first meeting in Dublin, Ireland, in November 2002. Several regional consultations involving nearly 900 participants representing 110 countries were subsequently held. The Steering Committee met in 2003 to finalize recommendations on the rationale, goal, scope, outputs, outcomes, assessment characteristics, management and governance structure, location of secretariat and the proposed funding philosophy. The Panel of participating governments, taking into account the recommendations of the multi- stakeholder Steering Committee that oversaw the consultative process, and the views of other stakeholders at the IAASTD Intergovernmental Plenary held in 2004, agreed on the objectives, goals, scope, key questions, design, preparation and peer review processes, outputs, timetable, budget and governance structure. The final design of the IAASTD is multi-thematic (agriculture; hunger, nutrition and human health; poverty, livelihoods and economic growth; and environmental and social sustainability) rather than a narrow agricultural productivity focus; multi-spatial (local to national to regional to global) rather than global only as many of the challenges in the agricultural sector are "place-based"; multi-temporal (historical to present to future) so that options for action can take into account both what has happened to-date and what may happen in the future; and utilize both institutional and local knowledge, especially in the sub-global assessments. The IAASTD is co-sponsored by seven international agencies (World Bank, FAO, WHO, UNEP, UNESCO, UNDP and GEF) given that the assessment will address agricultural issues (FAO), development issues (World Bank and UNDP), health issues (WHO), environmental issues (UNEP and GEF) and scientific issues (UNESCO), and between them will manage a distributed secretariat. The IAASTD is intergovernmental, but with a multi-stakeholder Bureau, and involves experts from all stakeholder groups in the design, preparation and peer review of the global and sub-global assessments, to ensure ownership of the process and findings. The assessment is policy relevant, but not policy prescriptive. A well-defined set of principles 53 Annex C and procedures have been established to ensure that the IAASTD process will be open, transparent and legitimate. After numerous author meetings and two rounds of peer review, the IAASTD Global and sub-global Summaries for Decision Makers were presented for rigorous scrutiny to 60 governments and nearly 100 CSO participants in the Final IAASTD Plenary meeting in April 2008. All governments present adopted the IAASTD assessments with the exception of three governments (Australia, Canada and USA) that presented partial reservations. 1.2. Development objective The IAASTD project objective was to improve access to agricultural knowledge, science and technology that will promote and facilitate sustainable agricultural practices with the aim to improve nutritional security, enhance rural livelihoods while reversing environmental degradation, redressing social and gender inequity, and ensuring human health and well-being. Towards this overall objective, the specific objectives were to: a) Undertake one global and five sub-global assessments for Sub-Saharan Africa (SSA); South and East Asia and the Pacific (ESAP); Latin America and the Caribbean (LAC); Central and West Asia and North Africa (CWANA); and North America and Europe (NAE); b) Provide robust information for decision makers on how to ensure that policies, practices and institutional arrangements enable agricultural KST to help reduce hunger and poverty, improve rural livelihoods and health, and facilitate equitable, environmentally, socially and economically sustainable development; and c) Bring together the range of stakeholders (governments, NGOs, private sector, producers, consumers, the CGIAR and others in the scientific community, multilateral environment agencies and international agencies, involved in the agricultural sector and rural development to share experiences, views, and gain common understanding and vision for the future. 1.3 Governance and management The Panel of participating governments agreed in 2004 on the objectives, goals, scope, key questions, design, preparation and peer review processes, outputs, timetable, funding and governance structure. According to the agreed IAASTD Principles and Procedures, the program was governed by the IAASTD Bureau, which included 30 representatives from governments, 22 representatives from civil society (six NGOs, six producer groups, four consumer groups, six private sector entities) and eight from international organizations. The IAASTD Director and two Co-chairs reported to the Bureau on the implementation of the project. The Director was responsible for the management of the Secretariat hosted by the World Bank. In addition, the co-sponsoring organizations provided supporting services. At the regional level, the project has been implemented by the African Center for Technology Studies (ACTS) for the sub-Saharan region (SSA); the World Fish Center for the East and South Asia and the Pacific (ESAP) region; the Instituto Interamericano de Cooperacion para la Agricultura (IICA) for the Latin America and the Caribbean (LAC) region; and the International Centre for Agricultural Research in the Dry Areas (ICARDA) for the Central and West Asia and North Africa (CWANA) region. In addition, the IAASTD Secretariat, hosted by the World Bank, implemented the project activities in the North America and Europe (NAE) region. 1.4 Funding and estimated expenses The initial funding of the project agreed by the co-sponsoring governments was 10.76 million dollars, which included in-kind costs and contributions. These costs have been met by funding of a total of $11 million from a variety of sources. The main financial contributions have been provided by the Global Environment Facility (GEF) $3.35 million, the Development Grant Facility (DGF) $1.5 million and the Multi-Donor Trust Fund partners (Australia, Canada, the European Commission, France, Annex C 54 Ireland, Switzerland, Sweden, United States and the private sector (Crop Life International) $3.3 million. In addition to this, the World Bank has provided support from the Bank budget, Finland has provided a staff contribution to the Secretariat and the co-sponsoring agencies have provided support staff, which in total is valued at $2.9 million. 2. Purpose, Scope and Type of Evaluation A standard project plan requires a terminal evaluation to be carried out. In the case of the IAASTD, the four individual DGF grants ending at different times have this requirement. In addition, the IAASTD GEF global grant requires a mid-term and a terminal evaluation. While there is no formal requirement of an evaluation in the Multi-donor Trust Fund agreements, it is expected that the donors will have an interest in an evaluation. The multiple evaluation requirements by the different grant agreements were later seen to result in multiple evaluations and for cost considerations it was decided to implement only one terminal independent evaluation covering all the different evaluation interests of the project. The principles and standards for evaluating Global and Regional Partnership Programs (GRPPs) developed by IEG under the auspices of the Development Assistance Committee (DAC) of the OECD are used in the evaluation. The evaluation criteria have been discussed with both DGF and GEF to satisfy both their requirements as expressed in the Terms of Reference. In addition, it has been agreed that the evaluation will replace the standard project closing requirement of the Implementation Closing Report (ICR) procedure. This evaluation will therefore contribute to the overall performance ratings for ICR purposes. The evaluation will cover the IAASTD global and regional processes from the initiation until the time of the evaluation in October 2008 as the project is winding down its implementation after publications of the assessment reports and remaining outreach activities in June 2009. The purpose of this evaluation is to enhance the impact of ongoing and future GRPPs by assessing and reporting on the outcomes achieved under the particular grants relative to objectives, the effectiveness of implementation arrangements and other processes designed to deliver results, and to provide feedback to the donors regarding key lessons learned and effectiveness of the grant to inform the design of future partnerships. The audience of the evaluation includes in addition to the financing partners and co-sponsoring UN organizations, more than 400 authors and review editors worldwide that have worked to produce the IAASTD results. The evaluation is also expected to inform all the stakeholders that have participated in the IAASTD process such as governments, the private sector, producer and consumer groups, non-governmental organizations, international organizations, foundations, the CGIAR and other scientific organizations and the general public. 3. Evaluation Questions The Independent Evaluation will assess the IAASTD in relation to the objectives set out in the project document and the logframe; and the following aspects of the IAASTD: (a) Relevance and sustainability 1. To what extent is there an international consensus concerning global challenges and concerns in the sector and to what extent is IAASTD (i) addressing these challenges and concerns; (ii) consistent with client countries' current development activities; and (iii) consistent with the missions and strategies of the other partners/stakeholders; (iv) complementing, duplicating or competing with other international instruments? 2. To what extent is IAASTD: (i) providing global and regional public goods; (ii) supporting international advocacy to improve policies at the national level; (iii) producing and delivering 55 Annex C cross-country, relevant lessons to client countries; and (iv) mobilizing substantial incremental resources. 3. Did the framework provide coherence between and among the global and sub-global assessments? Did the framework prove to be appropriate and utilized by the global and sub- global assessments? 4. To what extent are there effective and complementary linkages, where needed, between global program activities and partner country activities to the mutual benefit of each? 5. Did the global and sub-global assessments contain appropriate information on historical lessons; plausible scenarios or agricultural consumption and production and implications for environmental conditions; agricultural KST policy and institutional arrangements in relationship to environmentally sustainable agriculture? 6. How resilient are the outcomes and impacts of the IAASTD? (b) Effectiveness and Efficiency: 7. To what extent has IAASTD achieved its stated objectives and is adding value to: (i) what partners are doing in the sector; and (ii) what developing countries are doing in the sector in accordance with their own priorities? 8. To what extent did IAASTD have: (i) a clear program and component objectives with verifiable indicators; (ii) a structured set of quantitative and qualitative indicators; (iii) systematic and regular processes for data collection and management; (iv) independence of program-level evaluations; and (v) effective feedback from monitoring and evaluations? 9. Was the project completed on schedule within the approved budget and to what extent did funding positively or negatively affect: (i) the strategic focus of the program; (ii) the governance and management of the program; and (iii) the sustainability of the program? 10. Did the web system provide a user-friendly platform for the authors and peer reviewers? Is the web site user-friendly for the outside community and does it contain the appropriate material? 11. Is there an outreach and communications strategy that is effectively implemented to ensure ownership of the findings: (i) Are the Summaries for Decision Makers effective documents for disseminating the key findings to the full range of decision makers; (ii) Does the outreach and communications strategy ensure dissemination of the findings to key stakeholders, including the cosponsoring agencies, the GEF and its implementing agencies and the MEAs and the Parties to the Conventions? (c) Governance and management 12. To what extent did IAASTD achieve: (i) the required project funding (ii) efficient allocation of resources and sound financial management; (iii) funding from the OECD governments and other institutions for OECD author travel (iv) benefits that are most cost effective than those that could have been achieved by providing the same service on a country-by-country basis; (v) benefits that are more cost effective than those that could be achieved if individual contributors to IAASTD acted alone? 13. To what extent has the Bank's presence as a partner catalyzed non-Bank resources for the program and to what extent are partners maximizing their comparative advantages in support of the IAASTD at the global level (global mandate and reach, convening power, Annex C 56 mobilizing resources) and at the country level (multi-sector capacity, analytical expertise, country level knowledge)? 14. Did the IAASTD Bureau, Secretariat and the sub-global institutions function effectively and to what extent is the governance and management of the program: (i) transparent in providing information about the program; (ii) clear with respect to roles and responsibilities; (iii) fair to clients; and (iv) accountable to donors, clients, scientists, professionals and other stakeholders? 15. To what extent did developing and transition country partners, clients, experts and beneficiaries participate and exercise effective voice in the various aspects of program design, governance, implementation, peer review, monitoring and evaluation? 16. To what extent have the risks associated with the program been identified and effectively managed? (d) Outcome ratings: 17. The evaluation will contribute to the Outcome rating of the project according to GEF Guidelines for Terminal Evaluations. 4. Evaluation Design and Methodology The IAASTD Evaluation will be assigned to an independent Evaluation Coordinator assisted by four Regional Evaluation Coordinators, who are free of conflicts of interests with the Project and report to an independent governing body (ARD Sector Board). The Evaluation Coordinator will submit to the ARD Sector Board for approval an Inception Report proposing the final global and regional TORs including methodology and work plan before the main work commences. The evaluation instruments may include: (i) reviewing program/project documents, including Project Progress Reports; (ii) personal interviews with the program stakeholders (iii) structured surveys or meetings with stakeholders. Due to budget limitations, international travel is not expected to be required but regional contacts will take place through four Regional Evaluation Coordinators to be recruited. 4.1 Work Plan, Responsibilities and Schedule The implementation work plan, main responsibilities of the Sector Board, Evaluation team and timing of activities is envisaged as below. The selected Evaluation Coordinator will prepare a final work plan for the approval of the Sector Board as part of the Inception Report. 1. After selection by the Sector Board, the Evaluation Coordinator will be Sept 08 contracted by the IAASTD Secretariat 2. The Evaluation Coordinator will: 3. Prepare inception report and regional TORs and work plans for approval by the Sept 08 Sector Board. 4. Solicit candidates for proposals for 4 Regional evaluators with the help of the Oct 08 regional institutes for approval by the Sector Board. ACTS-Kenya will contract the 4 Regional evaluators 5. Coordinate the work of the 4 Regional Evaluators. Receive their work plans Oct-Dec 08 and comment on interim reports. 6. Carry out interviews and data collection for global evaluation Oct-Dec 08 7. Receive final regional evaluation reports Dec 08 57 Annex C 8. Consolidate a draft Independent Evaluation report Dec 08 9. Request IAASTD Secretariat comments to the draft report Dec 08 10. Present draft report for comments to the Sector Board, along with IAASTD Dec 08 Secretariat comments on the draft report. 11. Prepare the final Evaluation report for approval by the Sector Board. Feb 09 12. After approval of the final report, the IAASTD task team leader will distribute Feb 09 the report to the donors and stakeholders. The Sector Board will publicly disclose the final evaluation. 13. IEG will conduct a Global Program Review of IAASTD based upon the Evaluation report 4.2 Evaluation Report The main body of the evaluation report should not exceed 50 pages and should include the following sections: 1. Specific description of program objectives, deliverables, activities and units being evaluated (the latter include the Secretariat, the Bureau and the Regional Institutes. Meetings with units may be virtual and information collected should target key questions rather than attempt to be comprehensive) 2. Methodology used to develop findings. 3. Analysis of each of the evaluation dimensions, guided by the questions posed. 4. Summary of Key Findings 5. Conclusions and Lessons Learned 6. Key Recommendations Annexes may be added of people contacted, sources and uses of funds, survey results, and any other relevant information to support the evaluation findings. 4.3 Evaluation Team An Evaluation Coordinator will lead the Evaluation and is responsible for all arrangements for the implementation of the Evaluation (as outlined in the work program and schedule under 4.1) to the governing body. The necessary expertise for the Evaluation Coordinator is assessed against the following criteria Strong experience and background in global and regional policy issues, conventions and assessments on agriculture and sustainable development. Good knowledge of international development organizations; developing country experience; and experience in institutional capacity building. Demonstrated experience in leading and organizing international evaluations and excellent communications and writing skills The Evaluation coordinator will prepare regional TORs, selection criteria and facilitate the selection and contracting of four Regional Evaluation Coordinators to cover the IAASTD implementation in SSA, CWANA, ESAP and LAC regions. Evaluation of the IAASTD implementation for NAE region is expected to be covered by the Evaluation Coordinator. Evaluators should disclose any potential conflicts of interest before accepting this assignment. Any conflicts of interest that arise during the conduct of the evaluation should be referred to the ARD Sector Board for appropriate resolution. Annex C 58 4.4 Budget, logistics and administrative support A total allocation of $50,000 is available for the contract of the Evaluation Coordinator, which is expected to cover all expenses related to the assignment. Contacting will take place under the direction of the ARD Sector Manager. An allocation of $30,000 is available for contracting the four Regional Evaluation Coordinators, which is also expected to cover all expenses related to their assignments. Contracting will take place through ACTS GEF trust fund. The Evaluation Team is expected to organize their work independently without logistics and administrative support by the stakeholders. Requests for such support should be identified in the inception report and after approval by the Sector Board be provided through the ARD Sector Manager. 5. Obligations of Key Participants in the Evaluation 5.1 Governing body The independence of the evaluation (non-interference by project management) is according to accepted evaluation guidelines overseen by the governing body of the project, which is the IAASTD Bureau. The IAASTD project management has confirmed that while the Bureau has not been formally dissolved, it is not functional for the purposes of the Evaluation. Therefore, an alternative governing body independent of the project management was chosen. The ARD Sector Board will oversee the Independent Evaluation as the "de facto" governing body and the ARD Sector Manager will be responsible for administrative functions as the agent of the Sector Board. Tasks of the Sector Board are to: Approve the Evaluation TOR and selection criteria for IAASTD Evaluation Coordinator Recommend selection of the IAASTD Evaluation Coordinator from 3 proposed candidates Approve inception report, regional TORs and work plans as prepared by the Evaluation Coordinator Comment on draft report presented by Evaluation Coordinator Receive final report presented by Evaluation Coordinator 5.2 IAASTD Secretariat The Secretariat will respect the independence of the Evaluation and will refrain from participation in Evaluation activities. However, the IAASTD Secretariat will facilitate the administrative functions of the Evaluation as directed by the ARD Sector Manager, particularly the contracting of the Evaluation Coordinator. 5.3 Regional institutes The regional institutes IICA-Costa Rica, ICARDA-Syria, World Fish Center-Malaysia and ACTS- Kenya will also respect the independence of the Evaluation. They are expected to provide stakeholder contact information to the Regional Evaluation Coordinators and avail staff for interviews. Additional support to the Evaluation will be at the discretion of the management of the Regional Institutes. 59 Annex D Annex D. The External Evaluation Findings and a Summary of the IAASTD Director's Response IE Executive Summary 1. The IAASTD grew out of a convergence of a number of ideas and activities that started to come together at the WSSD in Johannesburg: the reform of international agricultural research, the demands for a clear policy by the World Bank on biotechnology, in particular, transgenics, and special concerns with the case of Sub-Saharan Africa. There was a willingness on the part of both the private sector and civil society organizations to consider a process by which they could come to some understanding around several contentious issues. The recent successes of the Intergovernmental Panel on Climate Change (IPCC) and the Millennium Ecosystem Assessment (MEA) provided support to the proposal that an "assessment" of agricultural knowledge, science and technology would be a good way to bring a wide body of stakeholders together around complex issues. 2. A worldwide consultative process overseen by a high-level Steering Committee, recommended the creation of the International Assessment of Agricultural Knowledge, Science and Technology endorsed by a multistakeholder plenary in Nairobi, August 30- September 3, 2004. IAASTD was established with a combination of attributes that gave rise to a claim of uniqueness: an intergovernmental process with a multistakeholder "Bureau" with principles and procedures that were a "hybrid" of the IPCC and MEA. It would be a multi-sponsor, multi-level, multi-disciplinary, and multi-themed "evidence-based assessment" of "all types of relevant agricultural knowledge, science and technology (AKST) as well as the role of institutions, organizations, governance, markets and trade." 3. This evaluation focuses on process issues but it is useful to mention IAASTD's contributions as a "very significant step in the crucial work of identifying and addressing the structural roots of the global food crisis, as well as in paving the way to design more sustainable food systems for the 21st century." One significant, but understated, achievement of the IAASTD is its contribution to sparking a debate that has been postponed for too many years: the issue of the diversity of agricultural development paradigms.64 4. The IAASTD was a unique ground-breaking assessment in terms of governance (a hybrid between the more classical intergovernmental and non-governmental processes of the Intergovernmental Panel on Climate Change (IPCC) and Millennium Ecosystem Assessment (MA) respectively. It provided a forum in which people from different epistemic communities and different disciplines collaborated over several years. This evaluation is an early attempt to identify some of the many lessons for process and governance that the experience will generate. 5. The evaluation found that IAASTD was similar to other assessments in the sense that design elements coming from the consultative process had implications for later processes and outcomes. These were related to scope, scale, and focus, as well as governance and 64. Oliver de Schutter, UN Special Rapporteur on the Right to Food. CSD preparatory meeting, Feb, 2009. Annex D 60 management. Moreover, with turnover of authors at various stages of the process, it was a continuous effort to keep agreement on key concepts, definitions and practices and focus on a story line within chapters, across chapters and among regional and global reports. 6. Given the scope and scale of IAASTD, it was necessary to develop an evaluation methodology that could answer the questions in the TOR in a way that reflected the ambition of IAASTD itself. To do this, the team took the following steps: 3.1 Consultation with a range of stakeholders (CGIAR, GFAR, NARS leaders, donors, NGOs, Science Council, and individual resources) during the AGM of the CGIAR in Maputo (December 2008). 3.2 A review of lessons and good practices from other assessments and multistakeholder processes that could serve as templates for reviewing the IAASTD experience. 3.3 Designed an on-line Survey of IAASTD authors, reviewers, Bureau members and participants in the Johannesburg plenary based on email addresses provided by the Secretariat. Some questions were stimulated by the interviews in Maputo. The Survey was inclusive and provided structured data, open-ended answers with volunteered information and an entrée to follow up. 3.4 A review of the evolution of IAASTD from proposal through the consultative period to the approval of final procedures in the Bureau meeting in Montpellier. Decisions taken during this period affected the focus and implementation of IAASTD. 3.5 Regional evaluators were encouraged to look at the Sub Global processes and eventually the outcomes in relation to the Global messages of IAASTD. In ESAP the evaluator used a concept model to compare the Global and ESAP messages. In addition to email and telephone follow up of respondents to the Survey, regional evaluators made targeted interviews with key informants and people referred by them. In SSA a short SMS survey of extent of awareness of IAASTD by respondents complemented the other data. 7. The evaluation looks at IAASTD as a component of a knowledge system designed around the goals of IAASTD and answers the sixteen evaluation questions relating to its efficiency, effectiveness and relevance as a system defined around three broad objectives and four specific objectives. (See Program at a Glance). 8. With respect to the "broad objectives," IAASTD successfully completed the Global and five Sub Global Assessments of the role of AKST as it pertains to the multiple goals of a social, economic and environmental nature. It brought together a range of stakeholders, many of them not normally in debates over agricultural KST, and they shared views and their evidence. Given the process of selection and attrition of authors, it was inevitable that "consensus" would be difficult. An "assessment" prefers a clear statement of areas of disagreement to weak "consensus" by dilution and deletion. Divergent stakeholders gained better understanding of each others' views, had a shared commitment to the IAASTD goals but retained different interpretations of both the causes of problems and future pathways to achievement of those goals. 9. With respect to the "specific objectives," they assessed the effects of agricultural KST and institutional environments following a formal process of review and revision using evidence that was acceptable under the principles and procedures. Summaries for Decision Makers at each level, designed to "reduce complexity," were negotiated in Plenary in Johannesburg (April 2008) and adopted by 57 governments with three large agricultural 61 Annex D countries expressing reservations. Both the strong supporters and the critics of the Summaries will argue that they hide the treasures or the errors of the full reports. 10. There remain some gaps in the assessment. The loss of a full analysis of scenarios, which would have brought the institutional drivers and the technical drivers together, made it difficult to assess the probability of alternative scenarios and the feasibility of working towards them. The final specific objective was to further the capacity of developing country nationals and institutions to generate, access and use agricultural KST. 11. An attempt to develop an assessment of the capacity needs to implement the changes in the way business was done was eventually abandoned with information was eventually integrated into other chapters with consequential reduction in treatment of this important subject. 12. In addition to an assessment of the capacity needs to change the way business is done, one end-of-project status of IAASTD was "capacity built to assess agricultural KST at national and regional levels, with decision makers intimately involved with assessment processes." Participants in the sub-global reports gained valuable experience (Annexes 11 and 12) but it is too soon to see countries or regional organizations adopting the "assessment" model rather than other types of review that are continuously being proposed. 13. Through evidence from the Online Survey, follow up exchanges, and interviews with key informants the team recognizes the efficiency and effectiveness of the Secretariat in achieving most of the formal outcomes of the Program. Through interviews it also ascertained that the regional implementing organizations (the hosts of the sub-global coordinators) performed well. 14. IAASTD has been viewed as a unique "social experiment," particularly with respect to its hybrid intergovernmental process with a multi-stakeholder Bureau as well as for is scope and scale: the multiple objectives that AKST is seen to address and its integration across themes, disciplines, scales of analysis and levels of decision making. 15. The evaluation team believes that the IAASTD experiment should have anticipated the potential for conflict where different world views were in discussion and put in place conflict resolution mechanisms and codes of conduct that would have prevented minor disputes from growing into major distractions. Given the general support for multistakeholder processes, the role of the Bureau can be studied for lessons for future actions (see: "Governance and Management" and Annex 1: An accountability framework). 16. The evaluation team used the online survey to analyze how the participants themselves valued the outputs of the Assessment for different purposes. As the producers, first users and potential champions of the Assessment, their views may be considered well informed (with an expected bias to the positive). 17. The 22 key findings of IAASTD (Global Summary for Decision Makers) plus the Executive Summary of the Synthesis report and the Executive Summaries of the 5 Sub- Global reports were painstakingly negotiated in a formal plenary in Johannesburg in April 2008. While noting the critical importance of AKST, Australia, Canada, and the United Annex D 62 States commended IAASTD for an important contribution but did not approve the SDM noting disagreement with a number of assertions, options, and observations that required a more "balanced" treatment. The endorsement by 57 governments is an important part of IAASTD's "brand" image. 18. Reservations by particular governments with respect to individual findings were also noted. Several countries underlined that signing the document would not have any implication for their governments' position in international negotiating fora. A bloc of African countries noted that the report did not deal sufficiently with capacity building needs to meet existing and emerging challenges and to develop and apply new AKST. 19. Through interviews with donors, co-sponsors, research organizations and other partners and users of the outputs, the evaluation team learned that commitment will be demonstrated by actions taken to use the results. Since the outreach campaign will end before the documents in electronic format become widely accessible, the IAASTD "brand" will be used to promote selected parts of the IAASTD message of particular interest to the given champion. 20. The evaluation team found little demand for a strong "institutionalization" of IAASTD. The reasons were different and in some cases contradictory. The now published and soon to be available electronic versions of the Reports are a public good and a resource to which many different actors will add their value. However, useful this might be, we argue that some effort is still required to clarify the message and the IAASTD brand and ensure their use. Suggestions are made for a positive way forward. 21. The team identifies among IAASTD participants and other interviewees two potentially compatible but not necessarily converging pathways. An advocacy pathway takes the main message of enhanced attention to agroecology in general plus the individual messages that various champions promote with donors (such as food sovereignty and multifunctionality). The second approach is more research-oriented: it says that IAASTD needs to fill gaps in coverage such as capacity building, clarify the IAASTD message on contentious issues such as transgenics; quantify the salience of different options and assess the feasibility of implementing them. Only then can benchmarks be set against which progress can be measured and signing the Johannesburg documents becomes a commitment to action. 22. The evaluation team congratulates the Director, Co-Chairs, Bureau, Secretariat, and all the authors and review editors who contributed their time and efforts to bring the Assessment to a successful outcome. 23. The independent evaluation team has included outcome ratings of the IAASTD in compliance with the GEF Terminal Guidelines. Annex 13 provides the detailed ratings and background comments. Summary of the Main Points of the IAASTD Director's Response to the IE The response to the June 2008 IE Assessment came from the Director and has been made available on the IAASTD Web site. This is summarized here: 63 Annex D 1. The tone of the IE portrayed the glass half empty rather than half full. There were a number of inaccuracies in the IE, not all those pointed out in the draft had been addressed. 2. The IE mischaracterized the turn-over of authors. There was not much turnover and a few that did drop out did not affect agreement on, or use of, the key concepts, definitions and practices. The comments also explained the cost reasons why only a limited number of authors were invited to the final plenary. 3. The decision to drop the plausible futures scenarios section was based on substantive comments during a peer reviewed process that questioned the values and assumptions underlying scenario development. However, the comments accept that eliminating the futures construct without allowing time for the authors to respond to the criticisms was not ideal. 4. The IAASTD, both at the global and local/sub-global levels, did focus on outreach and communication and, whereas it certainly could have been more structured and possibly better coordinated and delivered, there have been a number of highly visible national, regional and global fora where IAASTD has received significant attention. Authors and others have participated in about 80 to 100 events since the plenary. 5. On the question of access to the reports, it was explained that the Secretariat had had no control over the publisher's requirement that the full report would not be placed online for six months. However, the Summaries for Decision-Makers and the Executive Summary of the Synthesis Report have been available online since they were finalized. Also, a significant number of reports were distributed free. The summaries are available online in six languages. The entire report will soon be available on a CD. 6. The comments disagree that a more formal conflict resolution process would have prevented the private sector author resignations or the three governments not formally approving the reports. Professional facilitation sometimes derails the process by taking away from the authors the responsibility of ownership. However, this was an aspect to be explored in the future. 7. In summary, the IAASTD was timely and effective in: contributing to the food security and poverty policy debates and action; placing agricultural science and technology within global debates; providing a forum in which people of different disciplines had a voice and collaborated effectively, providing a platform for dialogue and further study, complementing and building on the IPCC and MA reports which addressed only a limited number of aspects of agriculture and food security, and complementing the World Bank's agriculture WDR. 8. The comments suggested that governments and other stakeholders now need to decide whether a follow-up is needed and if so the scope management and governance of that follow-up. They need to recognize the wide range of interest groups and develop suitable democratic principles and procedures Annex E 64 Annex E. A Comparison of IAASTD SDM Findings with the Conclusions Chapter of Agriculture and the Environmenta Key Findings of IAASTD SDM Statements from the Conclusions Chapter of Agriculture and the Environment 1. Agricultural Knowledge, Science and Technology (AKST) has contributed to substantial increases in agricultural production over time, contributing to food security. 2. People have benefited unevenly from these yield increases across regions, in part because of different organizational capacities, sociocultural factors, and institutional and policy environments. 3. Emphasis on increasing yields and productivity Past agricultural growth has been associated with has in some cases had negative consequences a variety of environmental and resource problems on environmental sustainability. including excessive deforestation and forest degradation, water depletion, waterlogging and salinization, fish stock depletion, soil degradation, health effects, and biodiversity losses. 4. The environmental shortcomings of agricultural Adverse policy regimes force impoverished practice associated with poor socioeconomic peasants to mine, rather than augment, the land conditions create a vicious cycle in which poor resources. smallholder farmers have to deforest and use new often marginal lands, thus increasing deforestation and overall degradation. 5. Projections based on a continuation of current policies and practices indicate that global demographic changes and changing patterns of income distribution over the next 50 years will lead to different patterns of the consumption and increased demand for food. 6. Agriculture operates within complex systems and is multifunctional in its nature. 7. An increase and strengthening of AKST The new vision of agriculture research adopts a towards agroecological sciences will contribute to holistic approach that is more sensitive to addressing environmental issues while environmental concerns, while still addressing maintaining and increasing productivity. the need to boost yields and incomes of rural producers and caretakers of the land. 8. Strengthening and redirecting the generation Sustainable rural development must be and delivery of AKST will contribute to technically sound and innovative and must addressing a range of persistent socioeconomic consider environmental and social impacts. inequities. a. Source: World Bank. 1998. Agriculture and the Environment: Perspectives on Sustainable Development. A World Bank Symposium. 65 Annex E Key Findings of IAASTD SDM Statements from the Conclusions Chapter of Agriculture and the Environment 9. Greater and more effective involvement of For several decades, food, agricultural, and women and use of their knowledge, skills and natural resource management policies have been experience will advance progress towards designed without acknowledging that rural men sustainability and development goals and a and women may have different preferences, strengthening and redirection of AKST to address faced different constraints, and respond gender issues will help achieve this. differently to incentives. ... A review of empirical evidence shows that reducing inequalities between men and women in human and physical capital can lead to major efficiency and productivity increases in agriculture 10. Many of the challenges facing agriculture Technology needs to be viewed in a holistic way currently and in the future will require more to determine its fit with the existing social system innovative and integrated applications of existing and the direction of change that will be set in knowledge, science and technology ... as well as motion by its introduction. new approaches for agricultural and natural resource management. 11. Some challenges will be resolved primarily by In order to move closer towards environmentally development and appropriate application of new and socially sustainable rural development, much and emerging AKST. technical innovation is needed. 12. Targeting small-scale agricultural systems by Faulty policies have tended to favor the forging public and private partnerships, increased modernization of large-scale farming at the public research and extension investment helps expense of more efficient and employment realize existing opportunities. intensive family farms. Reforming adverse policies would be a win-win-win situation because the same policy changes could lead to more growth, less poverty, and more sustainable natural resource management .... 13. Significant pro-poor progress requires Case studies show that women, the landless, creating opportunities for innovation and marginal ethnic groups, and the laboring poor are entrepreneurship, which explicitly target resource unlikely to be represented in participatory poor farmers and rural laborers. (approaches). 14. Decisions around small-scale farm Implementing many of the needed changes to sustainability pose difficult policy choices. move toward more sustainable rural development will not be easy. 15. Public policy, and regulatory frameworks and It is essential that an appropriate policy international agreements are critical to framework be in place that is conducive to implementing more sustainable agricultural sustainable rural development. practices. 16. Innovative institutional arrangements are Multiagency approaches - not replacement by the essential to the successful design and adoption private sector or complete abolition - are one way of ecologically and socially sustainable of overcoming the limitations of the public sector. agricultural systems. Innovative multiagency approaches are now being tested in various ways. They seek synergistic interactions in which different types of organizations work together to analyze problems, contribute jointly to their solution, review progress in an iterative fashion, and make course corrections by mutual agreement as necessary. Annex E 66 Key Findings of IAASTD SDM Statements from the Conclusions Chapter of Agriculture and the Environment 17. Opening national agricultural markets to In general, trade liberalization leads to overall net international competition can offer economic gains, but some negative environmental effects benefits that can lead to long-term negative are possible. Where they occur, environmental effects on poverty alleviation, food security and policy regulatory actions may be needed. the environment without basic national institutions and infrastructure being in place. 18. Intensive export-oriented agriculture has (From the component paper on agricultural trade) increased under open market operations but has ... the environmental consequences of farm price been accompanied by both benefits and adverse and trade policy reforms are very complex, with consequences depending on circumstances such lots of offsetting forces at work, of which only a as exportation of soil nutrients and water, subset will ever be easily quantified. ... unsustainable soil or water management or (However) there is much more likelihood that the exploitative labor conditions in some cases. reforms will have a positive net effect on the environment than is often admitted by environmental groups and ... where the environmental effect is negative, the extra damage caused by trade reforms is only a minuscule fraction of that which would occur as a result of normal economic growth, whose adverse welfare effects are likely to be far more than offset by the welfare gains from trade reform. 19. The choice of relevant approaches to (Again) Technology needs to be viewed in a adoption and implementation of agricultural holistic way to determine its fit with the existing innovation is crucial for achieving development social system and the direction of change that will and sustainability goals. be set in motion by its introduction. Outreach mechanisms can be classified broadly into two basic approaches: the extension approach and the empowerment approach. The two often overlap and an effective outreach strategy contains elements of both. 20. More and better targeted AKST investments, The new vision for agricultural research adopts a explicitly taking into account the multifunctionality holistic approach that is more sensitive to of agriculture by both public and private sectors environmental concerns, while still addressing can help advance development and sustainability the need to boost yields and incomes of rural goals. producers and caretakers of the land. 21. While public-private partnerships are to be encouraged the establishment and enforcement of codes of conduct by universities and research institutes can help avoid conflicts of interest and maintain focus on sustainability and development in AKST when private funding complements public sector funds. 67 Annex E Key Findings of IAASTD SDM Statements from the Conclusions Chapter of Agriculture and the Environment 22. Achieving sustainability and development One of the major conclusions of (people-centered goals will involve creating space for diverse agricultural development) is that sustainability voices and perspectives and a multiplicity of does not reside in technologies, which have a scientifically well founded options, through, for half life of perhaps five years. Markets change, example, the inclusion of social scientists in input prices increase, new technological policy and practice of AKST helps direct and opportunities appear, varieties degenerate, pests focused public and private research, extension spread, and competition becomes stiffer. The and education on such goals. (sic) hope for sustainability of agricultural development is in the nature of an ongoing social process of widespread experimentation and sharing of information, of innovation, and group problem- solving. A Comparison of IAASTD Synthesis Options with the Conclusions Chapter of Agriculture and the Environment Main Elements of Options for Action Main Summary Statement from the Synthesis Report (p. 4) Conclusions Chapter of Agriculture and the Environment (p. 345) Successfully meeting development and Sustainable rural development must sustainability goals and responding to new appropriately integrate environmental concerns. priorities and changing circumstances would Such integration includes correcting price require a fundamental shift in AKST, including distortions that encourage excessive use of science, technology, policies, institutions, modern inputs in intensive agriculture, ensuring capacity development and investment. Such a that farmers have secure property rights over shift would recognize and give increased their resources, strengthening community importance to the multifunctionality of agriculture, management systems and empowering local accounting for the complexity of agricultural organizations, improving the performance of systems within diverse social and ecological relevant public institutions that manage and contexts. It would require new institutional and regulate natural resources by devolving organizational arrangements to promote an management decisions to resource users, or integrated approach to the development and groups of users, wherever possible, and giving deployment of AKST. It would also recognize greater attention to sustainability features of farming communities, farm households, and recommended technologies. ... It is evident that farmers as producers and managers of environmentally and socially sustainable rural ecosystems. development depends on appropriate policy frameworks, appropriate institutional approaches, as well as appropriate technologies. Annex F 68 Annex F. Persons Consulted This includes people interviewed or with whom emails were exchanged. Name Designation Date Mark Cackler Sector Manager 09/08/09 Jock Anderson Consultant, World Bank 08/06/09 Eija Pehu World Bank 08/27/09 Derek Byerlee Consultant, WB -- WDR author 08/06/09 Geeta Sethi World Bank 09/26/09 David Nielson World Bank 10/06/09 Robin Mearns World Bank 10/06/09 Dina Umali-Deininger World Bank 09/18/09 Gajanand Pathmanathan World Bank 09/18/09 Louis Coirolo World Bank 09/18/09 Emilia Batagliani World Bank 03/21/10 Robert Watson Then IAASTD Director 11/25/09 Christine Stover Then IAASTD Secretariat 09/26/09 Beverly McIntyre Then IAASTD Secretariat Several occasions 2009 Pekka Jameson Then IAASTD Secretariat 09/24/09 Manny Lantin CGIAR 09/31/09 Thelma Paris IRRI -- Reviewer 10/22/09 Mark Rosegrant IFPRI -- Author 09/12/09 Gershon Feder IFPRI 09/14/09 Dana Dalrymple USAID Reviewer 09/06/09 Prabhu Pingali Then FAO, now Bill and Melinda Gates 10/20/09 Foundation -- Author/Reviewer Rudi Rabbinge Chair, Science Council, CGIAR 09/25/09 Mangala Rai DG, ICAR, India 10/09/09 Jim Ryan Consultant ex CGIAR (not a participant) 09/28/09 Greg Jaffe Center for Science in the Public Interest 08/11/09 -- IAASTD Bureau and Steering Committee Ivar Baste Chief, Environmental Assessment 10/01/09 Branch, UNEP -- Steering Committee Kevin Cleaver Vice President, IFAD 10/08/09 Rodney Cooke IFAD 10/14/09 Howard Eliot IE Coordinator Several interactions Ed Rege IE SSA Author 10/25/09 Benedikt Haerlin Then Greenpeace -- Steering 10/06/09 Committee and Bureau member David Evans Syngenta.-- Steering Committee 10/14/09 69 Annex F Name Designation Date K. Palanasami Then Tamil Nadu University --Author 10/20/09 Michel Petit Institute Agronomique Mediterraneen, 10/06/09 Montpellier -- Lead Author Lim Li Ching Third World Network 10/13/09 Steve Suppan Institute for Agriculture and Trade Policy 10/15/09 -- Author Sir John Marsh Coordinating Lead Author 10/23/09 Michel Pimbert IIED (not a participant) 11/24/09 Jeff Waage LIDC -- Reviewer 11/24/09 David Howlett DFID -- Bureau member 11/24/09 Stephanie Williamson Pesticide Action Network (not a 11/24/09 participant) Howard Shapiro MARS Inc. -- Reviewer 10/28/09 Jules Pretty University of Essex (not a participant) 12/08/09 Sara Scherr Ecoagriculture Partners -- Author 11/05/09 Annex G 70 Annex G. Memorandum on Evaluation of GRPPs Supported by Both the DGF and the GEF THE WORLD BANK/IFC/M.I.G.A. OFFICE MEMORANDUM DATE: March 30, 2008 TO: Robert D. van den Berg, Director, GEF Evaluation Office FROM: Peter C. Harrold, Director, Operations Services CC: Mark Sundberg; Beverly Dianne McIntyre; Cheryl W. Gray; Chiyo Kanda; Christopher D. Gerrard; Lauren Kelly; Emilia Battaglini; James Warren Evans; Juergen Voegele; Kyle Peters; L. Marianne Cabraal; Marea Eleni Hatziolos; Margret C. Thalwitz; Mark E. Cackler; Matthew A. McMahon; Olga B. Jonas; Paul N. Hubbard; Pekka Jamsen; Randall B. Purcell; Claudio R. Volonte; Rohit Khanna; Aaron Zazueta; Neeraj Kumar Negi; Sophia Drewnowski; Robert.Watson; Chiyo Kanda. EXTENSION: 36048 SUBJECT: Problem Regarding Multiple Reviews of the IAASTD Rob: I understand that Chris Gerrard and Lauren Kelly in IEG had a meeting with two of your staff this morning (Aaron Zazueta and Neeraj Negi) to discuss all aspects of this issue. As you know, this issue of overlapping evaluation requirements for global and regional partnership programs (GRPPs) first arose in relation to the Critical Ecosystem Partnership Fund (CEPF), the first phase of which was financed by both the GEF and the DGF. The World Bank was the implementing agency for the GEF contribution to the program. At the end of the first phase, the CEPF Donor Council (the program's governing body) commissioned an external evaluation, after which IEG conducted a Global Program Review (GPR) based on the evaluation. In addition, the World Bank's Environment Department prepared an Implementation Completion Report (ICR), after which IEG conducted an ICR Review. The same issue of overlapping evaluation requirements by the GEF and the DGF is now arising again in relation to the International Assessment of Agricultural Knowledge, Science and Technology for Development (IAASTD) and the Coral Reef Targeted Research and Capacity Building Program (CRTR). Both programs would prefer to conduct an independent external evaluation of the program in the place of an ICR since the former represents a better evaluation instrument than the ICR, among other reasons, because the ICR does not cover or capture the important partnership dimensions of the programs. I now understand that an independent external evaluation of each program would meet the GEF requirement for a terminal evaluation of each "project" -- thereby replacing the need for the ICR -- provided that: 71 Annex G The evaluation was conducted in accordance with generally accepted evaluation principles such as those contained in the IEG/DAC Sourcebook for Evaluating GRPPs. The external evaluation included ratings for (a) project outcome, (b) sustainability of project outcome, and (c) project monitoring and evaluation in accordance with the "GEF Evaluation Office Guidelines for Implementing Agencies to Conduct Terminal Evaluations." IEG prepared a Global Program Review based on the external evaluation which also included ratings of the quality of the external evaluation. I understand from Chris Gerrard that IEG is willing to make a commitment to prepare GPRs of these two global programs in place of ICR Reviews. Therefore, please could you confirm that proceeding in this way is acceptable to the GEF Evaluation Office. Paul, could you also confirm that proceeding in this way is acceptable to the DGF. From the point of view of OPCS, proceeding in this way is acceptable because the only World Bank support for these two programs is presently coming from the Development Grant Facility. If there were also Bank support from IBRD loans or IDA credits, then this would pose some additional issues. Down the road, if the experience with this approach proves acceptable to all parties (OPCS, GPP Group, ENV, ARD, IEG, and GEF), then we might consider further discussions about extending this approach to other global and regional partnership programs financed by the GEF and implemented by the World Bank, even if they are not also financed by the DGF. Best Peter 72 Annex H Annex H. Response of the World Bank's Agricultural and Rural Development Department to IEG's Global Program Review We appreciate IEG's Global Program Review of the Independent Evaluation of the International Assessment of Agricultural Knowledge, Science and Technology for Development (IAASTD), of the effectiveness of the IAASTD and of the Bank's performance. In particular, we believe it offers useful insights on the World Bank's role when considering participation in possible future multi-stakeholder initiatives, especially in cases where a proposed Bank role as a convener, donor and/or sponsor may incur reputational risks, if the Bank has limited or no control over the final outputs. We believe that the IEG Review reflects a good understanding of the IAASTD origin, interagency nature, special multi-sectoral, multi-author and multi-stakeholder features, and the preparation, review and approval processes. We also believe that it provides a sound assessment of the Independent Evaluation of the IAASTD conducted earlier. We note the stricter ratings provided in this IEG Review compared to the Independent Evaluation, caused by somewhat different standards in ratings and methodology, and we recognize the validity of both approaches. With respect to lessons learned and recommendations, we agree with the Review's observation that the issue of agricultural knowledge, science and technology is important for the environmental and social sustainability of global agriculture. We appreciate the conclusion that the Bank was timely in identifying the need for assessing how well agricultural science and technology had performed in the past, and how well it had addressed the trade-offs between productivity and sustainability to lay the foundation for a vision going forward. However, should a similar exercise be proposed in the future, we agree with the IEG Review recommendation that a narrower set of well formulated questions may be likelier to produce policy-relevant assessments, rather than the very wide scope adopted in the IAASTD. As to the role of the Bank in the IAASTD, we appreciate the Review's conclusion that the Bank was effective in bringing the interested parties together, but we also acknowledge the conclusion that, once the Bank had committed to becoming an IAASTD sponsor and donor, significant reputational risks were incurred. In practical terms, it would have been difficult for the Bank to withdraw once the IAASTD was fully under way, which put the Bank in a vulnerable position since the IAASTD decision making process, by design, was outside the Bank's control. So, in hindsight, we agree with the IEG Review's conclusions that (a) while the Bank can have a role in convening such assessments, we put our name and "brand" in a vulnerable position in such negotiated, multi-stakeholder processes, if they are not well designed with a balance of stakeholders, adequate facilitation, and structured dispute resolution mechanisms; and (b) a more phased approach might have better managed potential reputational risks to the Bank. The Global Program Review Series The following reviews are available from IEG. Volume #1, Issue #1: ProVention Consortium Issue #2: Medicines for Malaria Venture Issue #3: Development Gateway Foundation Issue #4: Cities Alliance Volume #2, Issue #1: Critical Ecosystem Partnership Fund Issue #2: Association for the Development of Education in Africa Issue #3: Population and Reproductive Health Capacity Building Program Issue #4: International Land Coalition Volume #3, Issue #1: Consultative Group to Assist the Poor Issue #2: Global Development Network Issue #3: Global Forum for Health Research Issue #4: Global Invasive Species Program Volume #4, Issue #1: Stop Tuberculosis Partnership Issue #2: International Assessment of Agricultural Knowledge, Science, and Technology for Development GPR The International Assessment of Agricultural Knowledge, Science, and Technology for Development (IAASTD) was a multidisciplinary, multi-stakeholder assessment by about 400 experts. It had four primary goals: (a) to assess the effects of agricultural knowledge, sci- ence, and technology policies, institutions, and practices in the context of sustainable devel- opment; (b) to identify information gaps; (c) to make the resulting analyses accessible to decision makers; and (d) to further the capacity of developing countries to generate agricul- tural knowledge, science, and technology for sustainable development. It produced a Global Assessment, five Sub-Global Assessments, a Global Summary for Decision Makers, and a Synthesis Report. The World Bank was a convener, sponsor, financial contributor, trustee, host of the Secretariat, and supervisor of a Global Environment Facility project that con- tributed funds. The present review rates the overall outcome as moderately unsatisfactory. The IAASTD was a useful experience at the nexus of politics and science. However, agricul- tural technology--with its complexity, diversity, and politics--proved to be a bridge too far. For the substantial resources used, it did not offer sufficient new knowledge or conceptual frameworks for decision makers; it gave conflicting messages, and, for a 50-year timeframe, underestimated the potential of new technologies. Although the literature and case experi- ence presented in the IAASTD reports point toward a pluralistic strategy for agricultural research and technology development, along with rigorous evaluation to sift success from failure, the overall message that emerged from the IAASTD was more a restrictive, exclu- sionary one with an undercurrent against new technology, genetically modified organisms, and input-intensive agriculture. Attributable impact from the reports has so far been modest at best at the international level and negligible at the national level and below. This may improve with time, but time may also be running out. This review offers a number of lessons for the design of future assessments in the area of public science and for the World Bank as convener of such assessments.