44799 World Bank Agricultural and Rural Development Department Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects Water for Food Team FOREWORD Demands for accountability and results in development assistance have been at the forefront of the international agenda over the past years. In 2004, the World Bank adopted the results framework as a way to shift the focus from implementing activities to achieving and demonstrating results. It has since brought continued attention to Monitoring and Evaluation (M&E) as a key performance feedback system. The World Bank has long been a major lender to Agricultural Water Management (AWM), and investments in AWM have greatly contributed to meeting escalating food demands and fostering rural development in developing countries. Yet pressures on agricultural water are intensifying, with the need to meet ever-rising food demands while at the same time increasing farmer incomes, reducing poverty, and protecting the environment, all from an increasingly constrained water resource base. The performance of AWM projects has improved in many aspects over the years, but evaluations continue to report major weaknesses in their M&E systems whilst the challenging context would require a much better understanding of what works and does not work. The Toolkit has been prepared to respond to these challenges. It is mainly targeted towards World Bank professionals and their partners in countries who are responsible for designing, implementing, and using a results-based M&E system in AWM projects. There are difficulties to overcome, some of them specific to the sector, but when implemented properly the M&E system can be of great assistance in the design of the project, its implementation, and its assessment. The Toolkit comprises a set of guiding principles and helpful resources. It consists of three main parts: an introduction and overview for project M&E, is followed by guidance notes with explanations and examples on specific components of the M&E system, and by a set of resources for projects. Most of the Toolkit is focused on the specifics of World Bank AWM projects. Many of the principles and techniques covered however are generic and widely applicable. Improving M&E in our projects is crucial for assuring their quality and demonstrating achievements. We hope that this Toolkit will assist project practitioners in successfully implementing the results-agenda. Salah Darghouth Water Adviser for Food and Agriculture The World Bank Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects CONTENTS Acknowledgements Acronyms and abbreviations PART A ­ Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects 1.0 Introduction and overview 1.1 Purpose of the Toolkit 1.2 Users of the Toolkit 1.3 How to use the Toolkit: approach and structure 1.4 Application of the Toolkit 2.0 The nature of agricultural water management projects 2.1 The importance of agricultural water management (AWM) 2.2 Definition of AWM projects ­ what they involve 2.3 The World Bank and agricultural water management 2.4 Recent M&E practice in AWM projects 2.5 Categorising AWM projects 2.6 Implications for M&E design 3.0 Introduction to monitoring and evaluation 3.1 Introduction 3.2 Concepts and definitions 3.3 The purposes of monitoring and evaluation in the project cycle 3.4 The use of logical framework and Results Framework in project design and M&E 3.5 Linking project design to monitoring and evaluation 3.6 The limits of project management 3.7 The challenges of outcome and impact monitoring and evaluation 3.8 The role of leading indicators 3.9 Results-based monitoring and evaluation 3.10 Making results-based monitoring and evaluation work 4.0 Components of monitoring and evaluation systems 4.1 Introduction 4.2 Planning and implementing a project monitoring and evaluation system 4.3 The components of a project monitoring and evaluation system Further reading 4 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects Part B ­ Tools and Guidelines for AWM projects GN1. Building a results framework for M&E of AWM projects Introduction Building a results framework Specifying data collection, reporting and dissemination requirements GN2. Logical Framework Analysis Background The logical framework matrix Strengths and weaknesses of logical framework approach Stages in logical framework analysis GN3. Choosing and specifying indicators What are indicators? Choosing and specifying impact and outcome indicators Choosing and specifying cross-cutting indicators Choosing and specifying exogenous or external indicators Choosing and specifying input, process, and output indicators Example indicators for AWM projects GN4. Linking project and M&E design for AWM projects Introduction Flowchart diagrams illustrating causal chains (project logic) for key interventions Conclusion GN5. Organizational alternatives for M&E of AWM projects Introduction Conducting a `readiness assessment' prior to design and implementation of a project M&E system Organizational arrangement for project M&E Sectoral level monitoring and evaluation Implementing project M&E Building capacity and sustainable M&E systems Incentives and a management perspective GN6. Data collection, processing, and analysis for M&E of AWM projects Introduction Assessing data sources Planning requirements Core data collection methods for AWM projects Data analysis Reporting and using M&E findings GN7. Design and implementation of baseline surveys for AWM projects Introduction Building baseline information What baseline information should be collected? Baselines and targets 5 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects GN8. Remote sensing and GIS applications in M&E of AWM projects Introduction Background Examples of application Conclusions GN9. Impact evaluation of AWM projects Monitoring and evaluation: distinct yet complementary Forms of evaluation in World Bank projects Overarching themes in impact evaluation Key concepts and techniques for impact evaluation Key steps in designing and implementing impact evaluations for AWM projects GN10. AWM projects and M&E of poverty reduction The contribution of irrigation to poverty reduction Implementing the poverty reduction mandate of the World Bank Taking poverty into account in Bank AWM projects Defining poverty-related indicators in AWM projects Monitoring and evaluating AWM project outcomes contributing to poverty reduction Methods for M&E of poverty reduction and sources of data Impact evaluation GN11. Participatory monitoring and evaluation for AWM projects What is participatory M&E? When and how to use Participatory M&E in projects? GN12. Actors in M&E of AWM projects M&E actors in the World Bank project cycle GN13. World Bank procedures for project M&E M&E and the World Bank project cycle Main stages in the project cycle Core World Bank procedures and requirements GN14. M&E for infrastructure rehabilitation and modernization components in AWM projects Introduction Key activities in rehabilitation and modernization of I&D systems Key performance indicators, data needs and analysis Concluding remarks GN15. M&E for management, operation, and maintenance components Introduction Key activities for improving management, operation, and maintenance Key performance indicators, data needs, and analysis Concluding remarks GN16. M&E for Water Users Association formation and support Introduction Key activities for WUA formation and support Key performance indicators, data needs, and analysis 6 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects GN17. Institutional development in AWM projects Introduction Typical institutional development components Monitoring and Evaluation of institutional development components GN18. Benchmarking in Agricultural Water Management Introduction The difference between project M&E and benchmarking Benchmarking stages Examples of benchmarking in agricultural water management Summary 7 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects Part C ­ Resources RN1. Sample terms of reference for M&E of agricultural water management projects RN2. Terms of reference for a baseline survey for an agricultural water management project RN3. Prototype Baseline survey questionnaire for agricultural water management projects RN4. Prototype survey questionnaire for monitoring formation of Water User Associations in agricultural water management projects RN5. Monitoring and evaluation costs RN6. Sample M&E reports RN7. Annotated bibliography related to M&E of AWM Projects RN8. Glossary of terms for monitoring and evaluation of AWM Projects References 8 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects ACKNOWLEDGEMENTS This Toolkit has been prepared by members of the Water for Food Team within the World Bank Energy Transport and Water Department: Martin Burton (consultant), Laurence Smith (consultant), and Julienne Roux, under the guidance of Salah Darghouth, Task Team Leader. Christopher Ward (consultant) supervised the final editing of the document. The team wishes to extend their sincere thanks and appreciation to World Bank staff consulted during the preparation of the Toolkit: Ijsbrand de Jong, Animesh Shrivastava, Qun Li, Andrea Pape-Christiansen, Zhujun Wang, Jonas Mbwangue, Johannes Grijsen, Anne Kuriakose, Luiz Gabriel Azevedo, Isabelle Paris, Hani Abdel-Kader El Sadani Salem, Bulat Utkelov, Kelly Jones, Sandra Ruckstuhl, Manuel Contijoch, Srinivasan Raj Rajagopal, Willem Zijp, Ladisy Komba Chengula, Robert A. Robelus, Ismael Ouedraogo, Rory C. O'Sullivan, Piya Baptista, Caroline Van den Berg, Melissa Williams, Marisa Baldwin, Carla Cathey, Keith Pitman, Dina Umali- DeiningerSatoru Ueda, Joop Stoutjesdijk, and Riikka Rajalahti. The team wishes to thank the peer reviewers of the report: Mei Xie, Doug Olson, Prabir Joardar, Geoffrey Spencer, Gabriel Ionita, Erwin de Nys, Hassan Lamrani, Richard Chisholm, Subramaniam Janakiram, Rita Cestti, Dilshod Khidirov, Rabih Karaky, Abel Mejia, Ahmed Shawky, Abedlkrim Oka, Grant Milne, and Harshadeep Nagaraja Rao. DISCLAIMERS The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and should not be attributed in any manner to the World Bank, to its affiliated organizations, or to members of its Board of Executive Directors or the countries they represent. The material in this work is copyrighted. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or inclusion in any information storage and retrieval system, without the prior written permission of the World Bank. The World Bank encourages dissemination of its work and will normally grant permission promptly. For permission to photocopy or reprint, please send a request with complete information to the Copyright Clearance Center, Inc, 222 Rosewood Drive, Danvers, MA 01923, USA, fax 978-750-4470. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, World Bank, 1818 H Street N.W., Washington DC, 20433, fax 202-522-2422, e-mail:pubrights@worldbank.org. CONTACT DETAILS: To order additional copies, please contact the Water Help Desk at whelpdesk@worldbank.org. This CDROM is available online at http://www.worldbank.org/water. Copyright ® 2008 The International Bank for Reconstruction and Development/The World Bank. All rights reserved. 9 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects ACRONYMS AND ABBREVIATIONS ADB : Asian Development Bank BP : Bank Procedure CAS : Country Assistance Strategy CDD : Community-Driven Development CGE : Computable General Equilibrium CWRAS : Country Water Resources Assistance Strategy DPL : Development Policy Lending FAO : Food and Agriculture Organization GIS : Geographic Information System GPS : Global Positioning System IE : Impact Evaluation IEG : Impact Evaluation Group IFAD : International Fund for Agricultural Development ISR : Implementation Status and Results IWMI : International Water Management Institute LE : Lead Evaluator LFA: Logical framework analysis MDGs: Millennium Development Goals MIS : Management Information System MOM: Management, Operation and Maintenance M&E : Monitoring and Evaluation NGO : Non-Governmental Organization OD : Operational Directive OED : Operations Evaluation Department OP : Operational Procedure OPCS : Operations Policy and Country Services O&M : Operation and Maintenance PA : Poverty Assessment PAD : Project Appraisal Document PCN : Project Concept Note PDO : Project Development Objectives PIP : Project Implementation Plan PIU : Project Implementation Unit PME : Participatory monitoring and evaluation PMU : Project Management Unit PPA : Project Performance Assessment PRA: Participatory Rural Appraisal PSIA : Poverty and Social Impact Assessment PRSP : Poverty Reduction Strategy Paper QEA : Quality at Entry RAP: Rapid Appraisal Process RRA: Rapid Rural Appraisal ToR : Terms of Reference TT : Task Team TTL : Task Team Leader WBI : World Bank Institute WUA : Water Users Association 10 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects 1.0 Introduction and overview Monitoring and evaluation are critical project management tools. Monitoring informs stakeholders of progress and outcomes, and shows where corrective action is needed to adjust implementation plans. Evaluation assesses outcomes and impacts relative to expectations, explains variations, and helps review funding allocations. Taken together, monitoring and evaluation are critical to assessing the relevance and fulfilment of project objectives. The World Bank has long been a major source of assistance for agricultural water development. This has included a range of structural and non-structural measures to harness, control, and manage surface and ground water to improve agricultural production. These measures have typically involved varying combinations of irrigation, drainage and flood control, water conservation and storage, on-farm water management, and institutional reforms to improve sustainability, user operation and management, and cost-recovery. In view of its extensive involvement in agricultural water, and because monitoring and evaluation of progress and outcomes are essential to high quality, poverty reducing investments in the sector, the World Bank has undertaken the preparation of this Toolkit. The objective is to ensure effective monitoring and evaluation in agricultural water projects. The Toolkit is based on the Results Framework, adopted since 2004 by the World Bank as the basic monitoring and evaluation tool for all projects. 1.1 Purpose of the Toolkit The focus of this toolkit is monitoring and evaluation at the project level. However, many of the principles and techniques covered are generic and widely applicable also for programmes at sector level and for policy work. Similarly, although the focus in terms of the provision of detailed guidance and examples is on agricultural water management (AWM) projects, and AWM components within other projects, the concepts and approaches covered are applicable to all agricultural and rural development projects, and to more recently emerging areas such as watershed management and community driven development. In particular, Part A of the Toolkit, Guidance Notes (GNs) 1 to 7, 9, and 11 to 13, and Reference Notes (RNs) 1, 2, and 5 are of broad relevance beyond agricultural water management projects. The Toolkit will help users: · clarify what outcomes and impacts an agricultural water management project will have and how this will be achieved; · decide how progress, outcomes and impact will be monitored and evaluated; · collect and analyse the necessary data for tracking progress, outcomes and impact; · determine the reasons for success or failure, and how to use this understanding to improve future action. 1.2 Users of the Toolkit The Toolkit has been prepared for seven main groups of users (primarily, but not exclusively involved in World Bank projects): · World Bank managers and task teams; · government departments and agencies; · managers and staff responsible for project implementation or management; · farmers and other stakeholders in the project area 11 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects · consultants and other external organisations providing assistance on project design, implementation and management; · wider stakeholders from civil society with a legitimate interest in project outcomes; · other financing agencies involved in agricultural water management. Guidance note GN12 discusses in more detail the range of key actors and their roles in M&E work for World Bank-financed projects. 1.3 How to use the Toolkit: approach and structure The Toolkit comprises a set of guiding principles and helpful resources. It is not intended to be a detailed instructional manual, although it may be used as a support to training in M&E. The Toolkit consists of three main parts: Part A provides an introduction and overview for project monitoring and evaluation. Generic principles are set out that identify and describe all of the key elements of monitoring and evaluation as tools of good project management practice. The focus is on `what to do' and `why'. Frequent references are made to the Guidance Notes in Part B of the Toolkit in which further explanations and examples can be found, and to the resources provided in Part C. It is recommended that readers new to the subject read all of Part A first to gain an overview of the issues and an understanding of the main terminology. They may then refer to Parts B and C (ample cross-references are provided) for detailed explanation and illustration. Within Part A: · Chapter 2.0 provides an overview of the characteristics of Agricultural Water Management projects and their implications for M&E design; · Chapter 3.0 introduces the reader to key concepts for project and M&E design, in particular logical framework analysis and results-based M&E; · Chapter 4.0 sets out the steps needed to plan and implement project M&E, and explains the key components of a project M&E system. The recommended steps also provide a `roadmap' to navigate through the Toolkit. Figure 1 summarizes the `toolkit roadmap' presented in Chapter 4.0, highlighting the links between the nine steps in planning for M&E discussed in Chapter 4.2 and the Guidance Notes in Part B. Part B is made up of notes on tools and techniques that provide specific guidance and examples for agricultural water management projects. The focus of the Guidance Notes (GNs) in this part is on `how to do it'. Within Part B, the Guidance Notes cover eight main topic areas, as follows: · Project logic and results framework (GN1, GN2, GN4) · Indicators (GN3) · Organizational alternatives and data collection and use (GN5, GN6, GN8) · Baselines (GN7) · Evaluation and Benchmarking (GN9, GN18) · Participatory Monitoring and Evaluation (GN11) · Actors and Bank procedures (GN12, GN13) · M&E of key aspects of agricultural water management projects (GN10, GN14, GN15, GN16, GN17). 12 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects Part C contains a range of documentary resources presented as Reference Notes (RNs). They are designed to assist users in the design and implementation of agricultural water management project M&E systems. The Reference Notes in Part C include, for example, templates for drawing up terms of reference for consultants, or for baseline survey design. 13 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects Within Part C, the RNs can be grouped under five main topic areas: · Sample terms of reference (RN1, RN2) · Prototype survey questionnaires (RN3, RN4) · Monitoring and evaluation costs (RN5) · Sample M&E reports (RN6) · Annotated bibliography and glossary of terms (RN7 and RN8) Whilst the guidance notes in Part B can be read sequentially to gain a complete coverage of the briefing and instruction they provide, it is anticipated that most users will wish to read Part A, and then use Parts B and C as reference material to assist with specific tasks to be performed. At the end of Part A, and of each Guidance Note in Part B, a short and selective guide to further reading and reference is provided. An Annotated Bibliography in Part C also provides a comprehensive list of other recent resources for Monitoring and Evaluation. 1.4 Application of the Toolkit The Toolkit should be used throughout the project cycle. It will be particularly useful at the early identification and preparation stages, because good M&E design can contribute to improved project design in three ways. 1) Preparing the rigorous logical framework and Results Framework that underpins M&E requires a clear analysis of the expected developmental impact of a project and of the causal chain linking investments to outcomes and objectives (see GN1 and GN2). Thinking through an impact evaluation framework will oblige project designers to establish and understand the causal model through which the project is expected to achieve the desired aims, and this will enhance the quality of project design (see GN9). 2) Identification of indicators and how they can be used to measure the achievement of objectives will further help to ensure that project objectives are realistic, achievable and measurable, and that they are consistent with the resources provided, activities planned and outputs expected (see Chapter 4.0 below and GN3 for more on this). 3) An effective M&E design requires a good plan for the collection, analysis and reporting of data. In turn this requires assessment of the capacity of the project's management organisation, and of its ability to communicate with its partners and supervising agencies. Thus good M&E design can contribute to better organisational design for project management (see Chapter 4.0 below and GN5 and GN6 for more on this). The Toolkit will also be useful for task teams when preparing the Project Appraisal Document (see GN13), and specifically when preparing its mandatory Results Framework (see GN1 for more on this). The Results Framework summarises the Project Development Objective and the outcomes that contribute to it, together with the related indicators. The Toolkit should also be used by the agency responsible for project implementation when preparing the Project Implementation Document or Plan that will set out the detailed management arrangements and procedures for project implementation (see GN13). During project implementation the Toolkit will remain useful as a source of reference for project managers and M&E staff. It will assist them to run the M&E system, to refine indicators where necessary, and to plan and implement data collection to meet the information needs of management. The Toolkit thus has application at all stages of the World Bank project cycle for agricultural water management projects. It is hoped that other users will also find it a helpful resource for use in their own project cycles. 14 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects 2.0 The Nature of Agricultural Water Management Projects 2.1 The importance of agricultural water management (AWM) Irrigated land is about one fifth of the total arable area in developing countries, but produces 40 percent of all crops and close to 60 percent of cereal production (World Bank, 2006a). It also accounts for about 80 percent of all water use in developing countries. Successful investments in agricultural water management have helped meet rapidly rising demand for food, and have contributed to the growth of farm profitability and poverty reduction, as well as to regional development and environmental protection. In the future, rapidly rising demands for agricultural produce and the limited potential for further rainfed production entail that irrigated agriculture must continue to grow. However, although land and water productivity have much increased, there is significant room for improvement. The productivity potential for "more income per drop" remains sizable. Qualitative improvement in AWM could be achieved too. The environmental impacts of irrigation development have been both positive and negative, and stresses are growing as water availability for irrigated agriculture becomes increasingly constrained. In addition, interventions have often lacked a sufficient focus on poverty (GN10), sometimes excluding from the benefits or even further marginalising, vulnerable households such as the landless, tail-enders within irrigation schemes, or rainfed farmers. Yet the poverty-reducing potential of agricultural water management can be high. Water control in agriculture can boost agricultural growth and can reduce poverty directly and indirectly, benefiting the poor in several ways. Poor farmers can directly benefit from increases in their production that may increase their own consumption and provide a surplus of marketed products for increased farm income. Small farmers and landless laborers can benefit from agricultural employment opportunities and higher wages, and a wide range of rural and urban poor can benefit from related growth in the rural and urban non-farm economy. Crop harvest from irrigated areas leads to strengthened staple or non- staple food output that lowers prices and benefits all consumers, particularly the poor. Thus agricultural growth stimulated by improved water control in farming can generate important income and employment multipliers within the surrounding non-farm economy (Ward, 2007. See GN10). 2.2 Definition of AWM projects ­ what they involve Agricultural water management projects involve the interaction of water, land and people. In developing countries their prime objective is to support economic development and poverty reduction through measures to sustain, increase or improve agricultural production. There is a wide spectrum of AWM projects, ranging from promotion of simple measures to improve water control in rain-fed farming to large scale fully irrigated agriculture. A broad summary of typical components of AWM projects is provided in Box 1. The improvements made by such components lead to the planned outputs from a project and the desired outcomes and impacts in the wider socio-economic environment. It is thus important to understand the contribution that each component can make to the desired outcomes and impacts of a project. For example, the modernization of the physical infrastructure of an irrigation system might be a necessary condition for enhanced agricultural production, but the benefit derived from the modernization effort will depend on a number of other variables. These may include how the water is allocated and distributed by the service provider, and how it is used by the farmer. Such 15 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects an understanding is the basis for project design and for preparation of a Results-Based framework for monitoring and evaluation (see GN1 and GN4). Box 1: Typical components of AWM projects Soil and water conservation measures in watersheds Small scale irrigation Groundwater development and management Rehabilitation or modernization of irrigation and drainage systems Rehabilitation, modernization, or construction for flood protection Rehabilitation, modernization, and upgrading of dams Formation and support of Water Users Associations Measures to modernize and improve the management, operation and maintenance of irrigation and drainage systems Measures to modernize and improve the management of water resources Measures to provide support to water users for enhanced agricultural production Support to the reform of water sector agencies, Ministry of Water Resources, or Ministry of Agriculture Cross cutting components include: Capacity building and training Updating of existing and formulation of new legislation Gender issues Protection and enhancement of the environment Source: Authors 2.3 The World Bank and agricultural water management World population is expected to grow from its present 6.5 billion to 8.2 billion in 2030. Global demand for increased agricultural production will require better management of increasingly scarce water and land resources, in which the World Bank has a key role to play. Water resources are subject to increasing demands, whilst much of the best farm land is also being converted to urban and industrial use. The World Bank has long been a major source of assistance for agricultural and rural development. This has included a range of structural and non-structural measures to harness, control, and manage surface and ground water to improve agricultural production. Lending to both agriculture and agricultural water management remains highly relevant to the challenges of economic development and poverty reduction and has become the focus of renewed attention, as exemplified in the World Development Report 2008 (World Bank, 2007a). World Bank lending for irrigation and drainage averaged US$750 million per year for the period 2004-2007, three times that for the period 2000-2003. Investment in agricultural water management is recognised as a key factor influencing agricultural growth, international trade and poverty reduction. This economic context defines two underlying priorities: an emphasis on productivity of water use and the need for market-driven approaches. Other specific areas of focus of AWM projects in recent years include: · setting AWM within an integrated water resource management framework, seeking efficiency in allocation of water between sectors and integration of the productivity of agricultural water within the broader context of basin-wide water use; · increasing water productivity and farming profitability through markets and an incentive structure; · new institutional arrangements, which give more responsibility to farmers, engage the private sector, and redefine government's role; 16 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects · integrating policies, institutional change, and investments to achieve efficient outcomes in all aspects of AWM from modernization of large-scale systems to enhancing water management in rainfed agriculture, and on the sequencing and prioritization of change processes; · a pragmatic approach to intensification and expansion of AWM, using participatory approaches and new methodologies to make sure that social and environmental concerns enhance the economics and sustainability of investments, and ensuring that the broader benefits of AWM are captured; and · increased attention to the potential for reducing poverty, and the systematic factoring in of gender concerns in AWM programmes. It is recognised that these messages need to be adapted to regional and local situations through a process of dialogue and study that will produce action programmes. At the country level, World Bank Country Water Assistance Strategies (CAS) and Country Water Resources Assistance Strategies (CWRAS), where available, should act as the locus for an integrated approach to AWM within broader sectoral and macroeconomic strategies (World Bank, 2006a). 2.4 Recent M&E practice in AWM projects The World Bank's assistance to agricultural water management (1994-2004) was the subject of an Impact Evaluation Group (IEG) Portfolio Review (World Bank IEG, 2006). Between 1994 and 2004 the Bank lent a total of $13.2 billion for 161 projects across 56 countries that included quantifiable agricultural water management components. These projects directly benefited up to 12 million households and more than 60 million people. Within this total commitment, 42 percent ($5.6 billion) was specifically for agricultural water management components. Almost two-thirds went to South and East Asia and half to China, India, Indonesia, and Pakistan. This IEG review concluded that: · The overall quality of M&E design improved in the late 1990s with the introduction of logical frameworks and their mandatory use in Project Appraisal Documents (PADs), but the Implementation Completion and Results Report (ICR) could benefit from a section on who the beneficiaries are and how they benefit. · Project M&E often did not provide adequate information to inform Bank management of progress toward strategic objectives, particularly poverty alleviation and the Millennium Development Goals (MDGs). · Projects rarely adequately distinguished between the functions of monitoring and evaluation, usually describing monitoring functions only. Thus use of a rigorous evaluation framework was often missing from project planning and implementation, making robust attribution of benefits difficult. · Slightly fewer than half the projects did not have any means of verifying project impacts ­ no surveys or baselines ­ even though more than two-thirds of them included outcome or impact indicators. Only a third of completed projects had a baseline before the project started and less than half attempted to establish a baseline during the project (slightly more than 20 percent never established a baseline). · Overall, only 11 percent of projects were designed to have the tools that would allow rigorous impact assessment, specifically this includes well-defined output and outcome indicators, good baselines, and independent control groups unaffected by project interventions that allow the counterfactual (situation without the project) to be determined. Another 41 percent were able to allow determination of what happened before and after project implementation, but not a robust attribution of observed changes. · AWM projects that had civil works components had quite good monitoring and evaluation systems to track inputs and related outputs, but the quality of the systems declined as the focus moved on to outcomes and impacts. Increased attention to monitoring outcomes and impacts had occurred only in the most recent 2-3 years of the review period. In general most attention had been given to monitoring indicators of project implementation to provide feedback for better management. 17 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects · Even when there was good M&E design, inadequate supervision sometimes reduced effective implementation, and a need for more training was indicated. The review questioned the robustness of the conclusions drawn by projects that asserted improvements in observed production and farmers' incomes and attributed it to the Bank's project-level interventions. It suggested that much greater attention is needed to establish indicators (on this, see GN3) and evaluation frameworks (see GN9) to unambiguously determine and attribute the development impacts of Bank lending. The review concluded that provision of adequate baseline data, whether from project specific surveys or existing data sources, is clearly a key requirement and the two main challenges in this regard are timing and funding: when should baseline date be collected, how, and how will this be resourced? (See GN7). Overall, in responding to all of the challenges highlighted by the IEG review it is necessary to take account of the methodological challenges to be overcome and the resource requirements necessary (see Chapter 3.0 below). 2.5 Categorising AWM projects AWM projects are very diverse, in terms of size, software and hardware components, and supporting complementary investments. The IEG portfolio review (1994-2004) notes that there has been a change in the type of infrastructure financed, and a greater emphasis on non- structural and capacity-building components. An important distinction for the purpose of M&E is that of `dedicated' versus `non-dedicated' projects. Dedicated projects are defined as those for which more than half of the Bank's commitments are for AWM, whilst non-dedicated projects are those that include some AWM components but at less than this level. In 2006, in the portfolio under supervision, the average loan for a dedicated project was $90 million, of which $60 million were for the irrigation and drainage components. The average loan for non-dedicated projects was smaller, at $63 million, of which only $8 million were geared towards irrigation and drainage. While dedicated projects revolve around agricultural water infrastructure and management, non- dedicated projects focus on social concerns and agricultural support services, and generally adopt a community-driven development (CDD) type of approach. The type of infrastructure components financed by dedicated and non-dedicated projects tend to be markedly different even though most projects contain a mix of physical interventions ranging from new-build, redesign and upgrading, to repair of damage caused by deferred maintenance (usually referred to as rehabilitation/ modernization). The infrastructure focus of dedicated projects tends to be construction or modernization of small, medium, and large irrigation systems (see GN14 for specific guidance on monitoring such infrastructure development). Non-dedicated projects support a range of rural infrastructure components, including building new irrigation systems that are small-scale, community-owned, and well integrated in social development programs (see 2.6 for some implications of this type of project for M&E design). Institutional development has also become an important focus of projects. By the 1990s most development agencies were actively advocating reforms in the irrigation sector, emphasizing a reduced role for the government and a larger one for the users, financial autonomy for irrigation agencies, and devolution of management responsibilities to water users' associations, at least at the lower levels of schemes. Projects now include some form of farmers' participation for the design (although still rare) and management of irrigation schemes (now common). Water user associations (WUAs) are expected to be responsible for operation and maintenance (O&M), at least at the lower levels of schemes; typically for command areas varying from a few dozen hectares to around 3000 hectares. Most projects with participatory approaches have some capacity-building components. Most projects also include a cost-recovery system, sometimes for infrastructure construction and most often for O&M financing. Although many countries now have adequate regulations for establishment of WUAs, some projects have supported improvement or 18 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects enactment of the enabling legislation for this. Increasingly, projects are becoming `vehicles' for irrigation agency reforms, including promotion of financial autonomy, strengthening of sub- national administrations, and devolution of responsibilities and capacity building. Improving the performance of water service delivery agencies is now a high priority. Many projects include support to, and reform of, the Ministry of Water Resources and/or Agriculture, or another water sector agency. Reforms focus on improvement in water resources management, agricultural sector management, and bottom-up approaches (see GN16 and GN17). Much of the investment in the 1990s addressed the huge backlog in deferred maintenance and repairs, supported by some related improvements in institutions and management. In the past, few projects tackled the challenge of integrated system modernization, i.e., to change the irrigation delivery system and institutional and incentive structure to provide a sustainable, efficient, and demand-responsive water delivery service, but this is now being addressed (GN15). The challenge is to achieve technically sound and well integrated project design, and disciplined and detailed approaches to project M&E design can assist in this (see Chapter 1.4 above). Many projects also provide for improving the profitability of irrigated farming. Most projects include research and extension services for both crops and for water management at the field level. Some projects also have a crop diversification component. Other components, found less frequently, include: availability of inputs (seeds, fertilizers, pesticides), access to credit, access to markets, support to processing initiatives, support to farmers' cooperatives, and private sector development. For the future a range of interventions is called for in the 2006 report `Directions in Development: Re-engaging in Agricultural Water Management' (World Bank, 2006a). These are: · integrated modernization of existing large-scale irrigation systems; · improving the performance of small-scale and traditional irrigation systems; · on-farm irrigation management improvements; · expansion and improvement of drainage; · enhancing water management in rainfed agriculture; · improving and scaling up watershed management; · waste and drainage water reuse for agriculture; · expanding irrigation and mobilizing new water supplies where feasible with acceptable environmental and social impact. 2.6 Implications for M&E design The mandatory Results Framework (see GN1 and GN13) requires the Project Development Objective (PDO) and the intermediate outcomes ­ also sometimes called results ­ of all project components to be specified in the Project Appraisal Document. Although logical framework analysis (GN2) is not required by the Bank, it remains the best way to establish the causal sequences from inputs to outputs to outcomes, and therefore is generally the mechanism by which the elements for the Results Framework are generated. The logical framework analysis also allows the identification of appropriate indicators and arrangements for their monitoring (GN1 and GN3). Agricultural water projects have specific characteristics that need to be taken into account when designing and managing an M&E system according to World Bank requirements. 1. There is a need for emphasis on results (i.e. on outcomes and impacts) but monitoring these should be complementary to - not at the expense of - monitoring implementation. The past performance of M&E systems for AWM projects is considered to have been relatively poor, particularly with regard to showing achievement against development objectives. This weakness needs to be corrected through the design and implementation of M&E systems capable of assessing project outcomes and impacts, including economic growth and poverty reduction (see GN9 on impact evaluation and GN10 on poverty reduction). The Results 19 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects Framework approach aims to do this, but the methodological challenges in the case of AWM projects, and the M&E resources needed, should not be underestimated, particularly for the evaluation of higher level development impacts (see GN1 for guidance on the Results Framework and GN9 for evaluation). Renewed emphasis on results is desirable but should not detract from the monitoring systems that inform day-to-day management of projects. Thus approaches for impact evaluation must build on project management information systems rather than replace them. 2. Monitoring of infrastructure components is relatively straightforward. Investment in civil works, either for new-build, rehabilitation, or modernization of water management systems, is the major proportion of World Bank commitments to AWM. The planning and implementation of such investment tends to be inherently `blueprint'1 oriented because of the importance of the engineering components and the nature of construction processes. A conventional approach to M&E design using the Results Framework supported by an extended logical framework analysis will generally be applicable (see GN2 for guidance on the logical framework). Inputs, activities, outputs and expected project component outcomes/ results can to a large extent be specified in the project design stage, and comprehensive matching indicators and data collection and reporting systems developed. The achievement of high standards of implementation and operational performance will be facilitated by the development of such comprehensive and effective management information systems. 3. Monitoring and evaluation should also cover the institutional aspects. Typically, major components of dedicated AWM projects are civil works, improvement to the management, operation and maintenance of schemes, establishment and support of WUAs, and institutional development for water management. GN 14, 15, 16, and 17 provide guidance on M&E arrangements for such components. 4. For non-dedicated projects in which AWM is but a small component, M&E of AWM aspects should focus more on `implementation monitoring', i.e. on process and output indicators. In contrast to dedicated projects, the AWM components of non-dedicated projects are only part of a broader and potentially more complex project strategy. For example, in the common case of community-driven development (CDD) projects AWM may be only one item in the `menu' of investments proposed to communities. The AWM aspects of M&E for such projects will need to focus on a few simple key indicators that are a subset of the indicators used for the whole project. These key indicators should focus on process and output indicators (implementation monitoring as described in Chapter 4.0 below) as higher level achievements will be beyond the scope of the AWM components alone. Also, the generic concepts and methods covered in the Toolkit will be applicable to non-AWM components. 5. M&E of a `process' oriented project or programme should particularly focus on the results - achievement of objectives and outcomes - and requires a large degree of flexibility. In many cases, such as a sector-wide programme or CDD project, work plans, expenditure and hence precise inputs, activities and outputs may not be planned in advance but may take shape as the project proceeds. There, a `results-based' orientation to M&E is particularly important, setting clear objectives for project management in terms of project outcomes and impact, but acknowledging greater flexibility during implementation in how these will be achieved. A complete logical framework analysis may still be useful for the planning of project components at commencement or during implementation, but this must allow for flexibility and adaptation. Both the Results Framework and any project component logical frameworks 1A `blueprint' approach can be described as follows. Planning and appraisal teams formulate detailed organizational and work plans for implementation, i.e. the "blueprints". Inputs and associated outputs are set out in advance and it is assumed that the project has only to be implemented as planned to achieve its objectives. In contrast a `process' approach sets clear objectives but allows for much greater flexibility in how the goals are to be reached. 20 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects should be used as management tools and not design blueprints. Where clear sub- components are combined in a wider programme, it may be useful also to develop a Results Framework (and/or a complete logical framework analysis) for each component, including specification of baseline data and its source, and the indicators to be used for M&E. Such analyses and plans for each project sub-component can be "nested" and combined to provide the analysis and planning for the project as a whole. 6. Participatory approaches to monitoring and evaluation are appropriate, and should be widely adopted in CDD based projects. In Participatory M&E, project stakeholders are fully involved in designing the monitoring system and in collecting, analyzing, compiling and sharing the information. The approach can be applied to all types of projects, whether they are `process' or `blueprint' oriented. It is particularly relevant and needed where stakeholders participation is emphasized, and is strongly recommended for CDD based projects (see GN11). 7. Finally, there are other M&E challenges resulting from the new implementation context for World Bank AWM projects. These include: · an increasing number of stakeholders involved in design and implementation, with multiple partners including: farmers, water users, community organizations, and various public and private sector actors; · a focus on demonstrating key results and the increasing need for adopting standard indicators in agricultural projects · multiple project components and timelines for their implementation; · increasingly decentralized decision-making and information flows; and · variation in the capacity of the wide range of actors involved in project implementation and in M&E. Taking account of these issues, this toolkit aims to set out the general principles and methods which can be drawn on selectively to develop M&E systems with the right balance and orientation for the specific project and context of concern. 3.0 Introduction to Monitoring and Evaluation 3.1 Introduction Since the early 2000's the World Bank has been promoting a `Results-Based' approach to development. This chapter sets out the basic concepts and principles that underpin both the Results Framework approach and the logical framework analysis underpinning it. These conceptual principles are relevant to good project design and to the planning and implementation of monitoring and evaluation. For users of this Toolkit an understanding of these concepts and principles is essential, and it is vital to understand the links and complementarities between the Results Framework and logical framework analysis. Readers will note that the terminology and reporting formats are evolving over time but that the conceptual principles stay the same. 3.2 Concepts and definitions of M&E Monitoring and evaluation are distinct but complementary activities. Monitoring and evaluation are tools which managers, government and donors can use to measure and evaluate progress and outcomes, and then feed this information back into processes of decision making, management and governing. The definitions in Box 2 make it clear that these are distinct yet complementary activities. In summary the complementarity between monitoring and evaluation takes three forms: · monitoring can raise questions for evaluation, and evaluation results can indicate that new issues need to be monitored; 21 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects · monitoring and evaluation can use the same data, but frame different analyses for different purposes; and · monitoring and evaluation can be used in tandem as a tool by managers to diagnose and address specific problems. Box 2: Definitions Monitoring is the continuous collection of data on specified indicators to assess for a development intervention (project, programme or policy) its implementation in relation to activity schedules and expenditure of allocated funds, and its progress and achievements in relation to its objectives. Evaluation is the periodic assessment of the design, implementation, outcomes and impact of a development intervention. It should assess the relevance and achievement of objectives, implementation performance in terms of effectiveness and efficiency, and the nature, distribution and sustainability of impacts. Source: OECD, 2002; Casley & Kumar, 1987 Monitoring provides managers and other stakeholders with regular information on progress relative to the whole causal sequence from inputs to outcomes. It is descriptive and by comparing actual progress with target progress, monitoring can alert management of favourable or negative variances. This flow of relevant information during implementation enables managers to keep track of progress, to adjust operations to take account of experience and to formulate budgetary requests and justify any needed increase in expenditure. Indeed, an effective management information system (MIS) that performs these functions is an essential part of good management practice. Frequent evaluation of progress is similarly good management practice. It requires asking why targets are, or are not, being achieved, and thus seeks to establish the reasons for the situations and trends recorded by monitoring. Clearly evaluation should respond when monitoring identifies either problems or opportunities to enhance achievements. The terms `informal' or `ongoing' evaluation can be used to describe evaluation that is conducted primarily by managers themselves as a key part of effective management and project implementation. Periodic formal evaluation involves the recipient government and the World Bank in one or more formal reviews, for example a mid-term review, or an implementation completion review. An ex- post evaluation may be completed a further period after completion, when it is reasonable to expect the full impacts of the intervention to have taken place (see GN13 for more information on World Bank procedures and the project cycle and G9 for more on impact evaluation). Continuous and quality M&E can improve project management considerably. Results-based M&E systems (and a results-based approach to public sector management generally) place emphasis on use of information streams that are more or less continuous, and which can be trusted and used in real time for decision making. When monitoring and evaluation is effective, knowledge should accumulate in the experience and expertise of staff, in the documented institutional memory of the organisation and its partners, and in their planning and management procedures. 3.3 The purposes of monitoring and evaluation in the project cycle Monitoring and evaluation should be integral to the life cycle of a project (Figure 2), and they should continue after completion. M&E provides a flow of information for internal uses by managers, and for external use by stakeholders who expect to see results, want to see demonstrable impacts, and require accountability and trustworthiness on the part of the public 22 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects sector. Governments and organisations are accountable to stakeholders and this requires them to both achieve expected outcomes and be able to provide evidence that demonstrates this success. Monitoring and evaluation of projects can be a powerful means to measure project performance, track progress towards achieving desired goals, and learn and apply lessons. In World Bank-financed projects, M&E should be seen as an integral part of project preparation and regular progress reporting. Without a proper M&E system, it will also be almost impossible to prepare a good Implementation Completion and Results Report. The M&E system and the logical framework underpinning it can also be key to communicating the project strategy to all project implementing partners and stakeholders in a clear way. GN13 outlines the activities specified for monitoring and evaluation at each stage of the World Bank project cycle. Used carefully at all stages of a project cycle, monitoring and evaluation can help to strengthen project design and implementation and stimulate partnerships with project stakeholders. At a sector level monitoring and evaluation can: · improve project and programme design through the feedback provided from mid-term, terminal and ex-post evaluations; · inform and influence sector assistance strategy through analysis of the outcomes and impact of interventions, and the strengths and weaknesses of their implementation, enabling governments and organisations to develop a knowledge base of the types of interventions that are successful (i.e. what works, what does not and why); and · provide the evidential basis for building consensus between stakeholders. At project level monitoring and evaluation can: · provide regular feedback on project performance and show any need for `mid-course' corrections; · identify problems early and propose solutions; · monitor access to project services and outcomes by the target population; · evaluate achievement of project objectives; and · incorporate stakeholder views and promote participation, ownership and accountability. 23 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects The monitoring and evaluation system should be well balanced and should seek to provide for all stakeholders as appropriate to their needs. It is well recognised that participation by project beneficiaries in design and implementation can bring greater "ownership" of project objectives and encourage the sustainability of project benefits. Where possible objectives should be set and indicators for monitoring and evaluation selected in consultation with stakeholders, so that objectives and targets are jointly "owned" (see GN11 on participatory M&E). The early emergence of recorded benefits can then help reinforce ownership, whilst early warning of emerging problems can allow all stakeholders to contribute to corrective action before costs rise. A monitoring and evaluation system run entirely by and for World Bank task team leaders and the implementation agency may not be sufficient to meet all project needs, but neither may one run entirely by and for primary stakeholders. 3.4 The use of logical framework and Results Framework in project design and M&E The Results Framework is based on the concept of project logic and logical framework analysis. Good project design is based on a clear and logical project strategy. As discussed above, a logical framework analysis should be conducted to analyse the causal relations (or "hierarchy") between inputs ­ activities ­ outputs ­ outcomes, leading to the project development objective. The logical framework will identify the elements to be summarized in the Results Framework in the Project Appraisal Document (see GN1). Box 3 defines the hierarchy using the terminology of the Results Framework approach. 24 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects Box 3: Definitions for the levels of a project hierarchy Higher level development objectives: the longer-term widespread improvement in society to which achievement of the project development objective(s) is intended to contribute. Project development objective: the combination of one or more project component outcomes which make up the physical, financial, institutional, social, environmental or other development changes which the project is designed and expected to achieve. Project component outcomes/ results: the effects of project components bringing intermediate effects for beneficiaries in terms of observable change in performance, behaviour or status of resources. Outputs: the products, capital goods and services resulting from a development intervention and which are necessary for the achievement of project component outcomes. Activities: the actions taken by project implementers that deliver the outputs by using the inputs provided (some practitioners do not define activities, relying only on the detailed specification of inputs and outputs). Inputs: the human and material resources financed by the project. Source: Authors Box 4 illustrates the causal relationships that provide the conceptual linkages between the project elements; establishing these helps to design a sound and logical project. Note that a complex project may have more than one objective, and it may be necessary to define the strategy or causal chain of each project component as illustrated by Figure 3 below. This diagram also illustrates how the logical framework analyses of different project components can be "nested" or linked together. A similar approach can be applied to the sub-projects or components of a non-dedicated project or programme. This recognises that the objective and ultimately the impact of one sub-project or project component can be an outcome for the main project or overall programme. Box 4: Logical hierarchy of project design Means-ends equals Logical project design subject to Required conditions chain being in place End Higher level development objectives necessary conditions end (means) Project development objective(s) necessary conditions end (means) Project component outcomes/ results necessary conditions end (means) Outputs necessary conditions end (means) Activities necessary conditions Means Inputs Thus: · IF inputs are provided, THEN activities can take place; · IF activities are successfully completed, THEN planned outputs should result; · IF outputs are used as intended, THEN the project component outcomes/ results should be realised; · IF the outcomes are achieved, THEN the project development objective(s) (PDO) should be achieved; and · IF the PDO is achieved then the expected contribution should be made to higher level developmental objectives. Source: Authors 25 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects Underlying the "if.....then...." logic of the project hierarchy in Box 4 is a basic premise that the achievements and conditions specified for each level in the hierarchy are both necessary and sufficient to result in attainment of the next higher level. Each causal linkage must therefore be planned to ensure that the required conditions not only exist but are sufficient to achieve the next level. These conditions can often, however, be external to the project itself and thus beyond the direct influence of its inputs, activities and outputs; i.e. beyond the direct control of project management. Thus a link in the causal chain may fail because the required conditions fail to materialize or are ineffective. That is, either they are not in place as assumed, or they fail to support achievement of outcomes in the way that was expected. Project design is therefore often based on assumptions that required external conditions will be in place, and expected outcomes are subject to the risk that these conditions may not be in place or may not be sufficiently effective. Such risks can arise from inaccuracies in information, the uncertainty of the project environment, and the unpredictable reactions of participating beneficiaries and other stakeholders and agencies. Typically the uncertainty involved increases the higher the level in the hierarchy. As a result, and particularly when available information cannot be improved, the project designer will have to assume that the necessary external conditions will be in place after making all possible efforts to minimise the risks that they may not be. Using the principles outlined above, logical framework analysis provides a means to identify important assumptions and the risks that these may not be fulfilled. Completion of a full logical framework analysis is not mandatory for World Bank projects as the results-based framework abstracts only a sub-set of the information normally presented in the analysis (see GN1 for the Results Framework and GN2 for the logical framework analysis). However, it is a requirement that important assumptions and risks are identified and justified in the section on "Critical risks and possible controversial aspects" in the Project Appraisal Document (see GN13). Whatever planning and documentation tools are used, the final design of a sound project should specify few assumptions and risks. Risks may have been identified during planning but most, if not all of these, should have been eliminated in the final design. A proposed project that remains subject to several significant and probable risks should not proceed beyond inception. Finally, any project strategy cast as a logical hierarchy simplifies reality and cannot account for all details of the intended plan and its context. Thus the documented strategy is a management tool that needs continual review and adjustment to reflect current contexts and changing needs. The ability to adjust the strategy depends on clarity about what project management is capable of influencing and achieving, and on having the information necessary. Monitoring and evaluation provides the key to the latter. 26 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects 3.5 Linking project design to monitoring and evaluation A logical project strategy (Box 4) provides a structure for the design of project monitoring and evaluation (Figure 4). Information can be extracted from this to complete the Results Framework required for World Bank projects. There are two levels of monitoring: results monitoring and implementation monitoring. Using the terminology in Figure 4, the extent to which the project contributes to its objectives is the impact of the project (there may also be unintended impacts, both positive and negative). The achievement of project outcomes is measured in terms of results, which are the extent to which the observable outcomes are as planned. Monitoring at these two levels is usefully referred to as `results monitoring'. 27 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects The operation and performance of the project can be assessed in terms of the effectiveness and efficiency of the processes through which inputs are utilised in processes to produce the planned outputs. This can be usefully referred to as `implementation monitoring'. Implementation monitoring is essentially done through a management information system, tracking the day-to-day implementation of the project. The MIS will include the procurement system, financial flows etc. as well as records of physical activities and processes, and the key indicators used for implementation monitoring. Results monitoring can also be included in the MIS, or be managed under a separate information system. Evaluation should also be based on the logical framework, using five criteria commonly used in the evaluation of development projects, and of sector and policy level interventions. Box 5 defines these five criteria. Box 5: Evaluation criteria Impact The effect of the project on its wider environment, and its contribution to the wider policy, sector or Country Assistance Strategy development objectives. Relevance The appropriateness of project objectives to the problems intended to be addressed, and to the physical and policy environment within which the project operates. Effectiveness How well the outputs contributed to the achievement of project component outcomes/ results and the overall Project Development Objective(s), and how well assumed external conditions contributed to project achievements. Efficiency Whether project outputs have been achieved at reasonable cost, i.e. how well inputs have been used in activities and converted into outputs. Sustainability The likelihood that benefits produced by the project continue to flow after external funding has ended. Source: Authors Figure 5 shows how these criteria are in turn linked to logical project design and to the types of indicator, completing this overview of how project design and the planning of monitoring and evaluation should be closed linked. Figure 5: Linking project design and evaluation criteria Project logic Types of indicator Evaluation criteria Objectives Impact relevance and PDO and Outcomes impact Outcomes sustainability Outputs Output effectiveness Activities Process and efficiency Inputs Input Source: Authors 3.6 The limits of project management The ability of managers to use the information produced by monitoring and evaluation to adjust a project's strategy during implementation will depend on the flexibility of the project's design and management arrangements. If a project is `process-oriented' (see Section 2.6 above) and designed with an open-ended strategy, then general directions will be indicated but with freedom for project partners to refine the operation of the project as it proceeds. The more flexible the situation, the more a good monitoring and evaluation system is necessary to provide managers with the information needed to be responsive 28 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects and adaptive, and the more the M&E system itself will need to evolve over time as implementation proceeds. If a project is `blue-print' oriented (see Section 2.6 above) and more rigidly designed, the opportunities to adjust the strategy may be restricted to periodic opportunities such as mid-term reviews. For such projects, M&E findings will be critical in informing and providing the justification for change, when change is needed. The design of the M&E system from the commencement of the project will in turn be more `blue-print' oriented, although this should not completely rule out flexibility and the possibility of change. In response to information gained from M&E, project management can be expected to adjust those elements of a project that are within its control, but as illustrated in Figure 6, control over the factors that influence the achievement of objectives diminishes with each higher level of the hierarchy. It is reasonable to hold project management accountable for achievement up to the level of the Project Development Objective, and thus monitoring and evaluation by management up to this level is crucial. This is particularly true during the early stages of a project when change is easiest. Figure 6: The limits of project management Degree of Challenge of Logic control M&E what the project Higher level is expected to objectives contribute to PDO and what the component project can be outcomes/ expected to results achieve and be Outputs what is within accountable lortnoc ytluciffid the direct for Activities control of project Inputs management gnisaerce D gnisaercnI Source: Authors For higher level development objectives to be achieved there may be necessary external conditions that are beyond the direct control of project management. A range of factors may influence the impacts that occur, and managers may be only one of several stakeholder groups and agencies that contribute to achievements. Thus at higher levels in the hierarchy a project's accountability diminishes, although it does not disappear entirely. When monitoring and evaluation reports achievement of the PDO but failure to contribute to wider development objectives as expected, it means that either the project design is faulty, or that the supporting external conditions were not as assumed. In both cases response is needed from all project partners, led by the supervising governmental agency and funding organisation, and informed by the lessons from monitoring and evaluation. In some cases the response necessary to improve impact may be at a sectoral rather than project level. 3.7 The challenges of results monitoring and evaluation Given that `inputs', `activities' and `outputs' are within the direct control of project management, `implementation monitoring' and evaluation is a core management function and it is relatively straightforward. It is achievable largely through internal record-keeping and analysis. Indicators of inputs, processes and outputs are usually generated by project management, and/or by government and funding agency accounting and reporting requirements. Attention to detail and good data management systems are important, but conceptually and methodologically this should be straightforward and a standard aspect of good management practice. 29 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects "Results monitoring" of outcomes and objectives is more challenging. Difficulty increases (as illustrated in Figure 6) at the levels of `outcomes' and `objectives'. These are the subject of `results monitoring'. For `results monitoring' indicators are subject to the twin problems of measurement (or comparability) and attribution (or causality). Problems of measurement. For an indicator to be useful it is necessary to be able to measure whether change has occurred over time compared to a `baseline' (see Chapter 4.0 below, and GN3 and GN7). This is problematic for indicators which are subject to considerable annual or seasonal variability, and thus require a long time series of values for a trend to be determined with statistical validity. Crop yields are a typical example, and one highly relevant as an outcome indicator for AWM projects. Although irrigated crop production will typically be more stable than rainfed agriculture, at least five or more years' data will typically be needed to show that yields have improved. In agriculture such variability in production, compounded by the typical co-variance between producers in a given location, can feed through into volatility in other key `outcome' and `impact' indicators such as food prices, rural employment and rural household incomes. This measurement problem can be compounded by practical problems that are typically most severe in resource poor and remote regions. Recording of crop yields, for example, will require a survey that takes either physical samples or relies on farmer estimates. Both approaches will be subject to sampling and other errors that can only be reduced through intensive training and supervision of enumerators, activities that are costly and time consuming. Data series may already exist for some typical outcome and impact indicators and subject to an assessment of their quality should be used in preference to new data collection. However, where there are gaps, M&E survey designers need to pay particular attention to comparability with the existing data when selecting survey instruments and methods. Even under conditions of close supervision and rigorous design, small changes in the way in which questions are put, the layout of the survey form, and guidance given to enumerators can undermine comparability. This is particularly likely to apply to indicators of household consumption and income, and other measures of poverty. Problems of attribution. Assuming such measurement problems can be solved and that change over time can be observed with statistical validity for an indicator of outcomes or impact, the second of the twin problems is that of attribution. Establishing that the cause of the observed trend is the project and not one or more external factors requires rigour in the evaluation process. A range of approaches is available (see GN9), all essentially requiring the observed change to be tested against a reliable counterfactual (the situation that would have happened had the project not taken place). Whilst not insoluble this problem is often challenging for agricultural water management and other rural development projects, and again will require considerable time and resources. These methodological issues are covered in more detail in GN6 (data collection) and GN9 (evaluation frameworks). The key point to note here is that monitoring and evaluation of outcome and impact indicators will require considerable time and resources. This particularly applies when `formal' methods that can produce results with statistical validity are to be used. Thus adequate human resources and expertise are essential, for what is for all practical purposes an exercise in applied research. These methodological challenges and requirements for staff with applied research skills may be beyond the capacity of the project management organisation, and if so, the services of national agencies and/or external specialists (consultants) will be required. 3.8 The role of leading indicators Whether `results monitoring' and evaluation is carried out by project management or by specialists, information about project outcomes and impact will rarely be available to inform and improve project management during the early or mid stages of the project. In particular, there is usually a lag between agricultural development activities and results. It is important to recognize this lag, which 30 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects can be as much as two years (or longer for some perennial crops), especially as disbursements tend to peak over the last years of the project after a slow start. It will therefore usually be important for project managers to identify and use some `leading indicators' of project performance during the early stages of implementation. Leading indicators, sometimes also called `early outcome indicators', are those that can provide an early indication of whether an expected change will occur, before project implementation is complete, and before the expected change has taken place. For example, market research type data covering whether beneficiaries have access to, are using, and are satisfied with project investments and services can provide leading indicators of anticipated outcomes and impact. The assumption is that if the beneficiaries are both satisfied and actively taking up the services of the project, then it is likely that expected outcomes and impact will be achieved. Similarly, and for agricultural water management projects specifically, the technical performance of water storage and delivery systems can be important in demonstrating that the potential for achievement of impact has been established. Other leading indicators might be identified to provide early warning about the non-fulfilment of necessary external conditions. Such examples could include farm gate price levels compared to those expected, inadequate availability of farm inputs, slow progress in road construction, or a lack of investments by traders in anticipation of receiving increased volumes of produce or higher demand for farm inputs (see GN3 for more examples). Where "benchmarking"2 (GN18) has been carried out for key aspects of irrigation scheme performance, this information may assist the setting of targets for both leading indicators and those finally used for project evaluation. Information from benchmarking may assist assessment of the levels of achievement attained for certain indicators. For both leading and final indicators of outcomes and impact, a degree of pragmatism may often be necessary in the choice of indicators and data collection methods, depending on project characteristics and available resources. Use of case studies and participatory and other informal methods of data collection may be more cost-effective than formal survey methods designed to pass tests of statistical validity, although it may not necessarily be less demanding in terms of the experience and skills required (GN6 covers data collection alternatives in more detail). It is also pragmatic to recognise that the problems of attributing causality (3.7 above) and the complexity of the statistical analysis involved may mean that it is more cost effective to rely on leading indicators such as delivery of services and beneficiary response than to attempt to measure actual impacts for many projects. 3.9 Results-based monitoring and evaluation Governments and international development agencies are increasingly being called upon to demonstrate results i.e. palpable outcomes and impacts. Besides demands for greater accountability and transparency, stakeholders are also demanding greater efficiency and effectiveness of development actions. In consequence, the World Bank has adopted, as discussed above, the Results Framework, to be applied in all aspects of the project cycle. Results-based monitoring and evaluation places particular emphasis on outcomes and impact. It emphasises that it is not sufficient simply to determine that planned outputs have been delivered on time and on budget. The `ends' are more important than the `means' and it is necessary to determine, and show evidence that, planned outcomes and a worthwhile contribution to national goals are being achieved. 2 "Benchmarking" originated in the corporate sector as a means for companies to gauge and improve their performance relative to key competitors. It has been used in the irrigation and drainage sector since the late 1990s to aim at improvement through comparison with relevant and achievable internal or external norms and standards. It seeks to identify gaps in system performance, and feeds into M&E which focuses on how measures taken to close identified gaps are progressing (see GN18). 31 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects GN13 sets out how this is dealt with in the requirements of World Bank procedures during the project cycle, and Chapter 4.0 below and GN1 provide more guidance on how results-based M&E should be designed and implemented. Here it is sufficient to note that a results-based approach is based on the structured approach to project logic and M&E design outlined above. A results-based management approach should enhance public sector performance generally, and is particularly applicable for programme and policy interventions at sector level, such as Development Policy Lending, which adopt a flexible approach to implementation, and for which `inputs', `activities' and `outputs' may not be fully specified in advance. Results-based management for projects needs to build upon monitoring and evaluation systems that are initially focused on the implementation and performance of projects, but which can progress to evaluation of outcomes and impact as implementation proceeds. Where the costs of overcoming the methodological and practical challenges outlined above are prohibitive, or when human resources are inadequate, full and rigorous impact evaluation may need to be applied selectively, choosing only those projects that are most significant because of their scale or innovation, or because they are representative of other similar interventions. Whilst mandatory for all World Bank projects, a results-based approach is particularly important if a project is `process-oriented' and designed with an open-ended strategy, general directions being indicated but detailed work plans and resource provision not specified in advance. This would be the case for CDD based projects and may also apply to many sector level interventions. Clearly it may not be possible to initially develop a full logical framework analysis of inputs, activities and outputs for the purposes of planning, although if useful this can be developed as a management tool during implementation for project or programme components, once these are agreed by project partners and take shape `on the ground'. For all projects a focus on results-based management also puts pressure on the project manager and other members of the task team to change or adapt the project if it is not demonstrating that it can achieve the desired outcomes; looking ahead at achievements, rather than inwardly and narrowly at processes. These observations apply to many agricultural water management projects, especially those involving substantial investments in land and water engineering works and supporting human resource and other management systems. These are complex projects to implement, and whilst often apparently `blueprint' in character, still require flexible and adaptive implementation and operation if they are to achieve their optimal performance, outcomes and impact. 3.10 Making results-based monitoring and evaluation work The selection of indicators needs to be based on the logic of the project design (as outlined above) and the generation of management information must be linked to the phasing of the project. Having the right information at the right time, in the right place and in the right form is the key to successful responsive and adaptive management, and to informed and supportive project partners and wider stakeholders. The Results Framework in the Project Appraisal Document requires specification of how M&E information will be used (see GN1 and GN6), and thus task managers and their counterparts have to address these issues in planning the project and its provisions for M&E. Project managers are responsible for results-based monitoring and for corrective actions. Monitoring progress towards higher level development objectives requires that information be derived from all levels in the logic model of the project, at different time frames, and for different stakeholder needs. This is a management function and project managers must take responsibility for knowing how well the project is being implemented, and whether there are leading indicators showing continued implementation will generate the expected outcomes and impact or whether corrective action is needed. They should also ensure that reliable evidence of impact can 32 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects ultimately be produced to demonstrate accountability and to feedback into the planning of further interventions. Supervision too must make use of the information generated. Supervising organisations must make effective use of the information generated to facilitate flexibility in project management, allowing timely corrective adjustments to be made to implementation, and continually seeking to improve the development effectiveness of programme and policy level interventions. This will require mechanisms to be institutionalised that feedback the lessons from monitoring and evaluation into planning and supervision processes (see GN5 and GN6). 4.0 Components of Monitoring and Evaluation Systems 4.1 Introduction This chapter provides an overview of the main steps involved in planning and implementing a monitoring and evaluation system for agricultural water management projects, and of the main components required. The overall approach is that of results-based management as adopted by the World Bank, and the focus is at the project level, although many of the principles outlined also apply to programmes and policy interventions at sector level. Frequent references are made to the Guidance Notes of this Toolkit in which further explanations, examples and other resources can be found. 4.2 Planning and implementing a project Monitoring and Evaluation System A project M&E system is the set of planning, information gathering, analysis and reporting processes necessary to achieve the purposes within the project cycle set out in Chapter 3.0 above. In summary these are to: · provide regular feedback on project implementation and identify any mid-course corrections and management adjustments needed, · inform project design through the feedback provided from evaluation, · inform, influence and justify sector assistance strategy, · provide the evidence base that helps to build consensus and stakeholder support. Setting up a project M&E system involves nine steps (Figure 7). These need to be considered in the planning stage and the system implemented at project start-up and throughout project implementation. Figure 7: Steps in planning a results-based M&E system Conduct a Agree on Baselines and Plan monitoring, data Necessary readiness outcomes and data collection analysis, communication conditions and assessment objectives and analysis plan and reporting capacities 1 2 3 4 5 6 7 8 9 Establish the Select indicators Select results Plan critical reflection purpose and and evaluation targets and interim evaluations scope of M&E framework Source: Adapted from Kusek and Rist, 2004 33 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects 1. Assess the existing readiness and capacity for monitoring and evaluation · A review of current capacity within the organisation and its partners which will be responsible for project implementation, covering: technical skills, managerial skills, existence and quality of data systems, available technology and existing budgetary provision (see GN5 for more detail on the coverage of such a `readiness assessment'). · Identify any barriers to M&E of the project such as a lack of political will, expertise or experience. · What other organisations such as universities, private consultants or government agencies have the capacity to provide technical assistance and/or training? 2. Establish the purpose and scope. · Why is M&E needed and how comprehensive should the system be? · What are World Bank requirements and national requirements with regard to M&E? (See GN13). · In particular, what should be the scope and degree of rigour of the evaluation of final project impact? (see GN1 and GN9). · Should the M&E process be participatory? In planning and implementing project M&E it is important to recognise the potential benefits of stakeholder participation. There can be benefits from this at all stages of the project cycle including monitoring and evaluation. Concepts and approaches for this are considered in more detail in GN11. 3. Identify and agree with main stakeholders the project's outcomes and development objective(s). · Setting outcomes is essential in building a M&E system. In project design the specification of outputs, activities and inputs follows from this, and the expectation that achievement of outcomes will contribute to the higher level development objective(s) provides the justification for the project (GN1, GN2, and GN4). · In M&E design, indicators, baselines and targets (see below), are similarly derived from the setting of outcomes. · GN14, 15, 16, and 17 make the link between outcomes/ development objectives and typical AWM components (civil works, management, operation, and maintenance, WUAs, and institutional development). · If poverty reduction is to be one of the main objectives of the project, the M&E system has to be designed for it (see GN10). 4. Select key performance indicators and evaluation framework. · Indicators are the qualitative or quantitative variables that measure project performance and achievements (see GN3). · Indicators should be developed for all levels of project logic (see below), i.e. indicators are needed to monitor progress with respect to inputs, activities, outputs, outcomes and development objectives, to feedback on areas of success and where improvement is required (GN1 and GN2). · The evaluation framework sets out the methods to be used to address the question of whether change observed through monitoring indicators can be attributed to the project interventions. A range of approaches are possible and the depth and rigour of impact evaluation required for a specific project given available resources needs to be carefully considered (GN9). 5. Set baselines and plan data collection and analysis · The baseline is the first measurement of an indicator, which sets the pre-project condition against which change can be tracked and evaluated. A single point in time or current value may not be representative and it may be better to use an average, for example, for the three previous years if such data is available. Baseline data must be gathered for the key performance indicators and this may require implementation of a baseline survey unless existing data sources are adequate (see GN7). 34 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects · Subsequent data gathering and repeat surveys for the implementation period of the project and beyond should then be planned. Data collection may be continuous or periodic depending on the nature and purpose of an indicator. A wide range of data collection methods is applicable (see GN6). Use of modern technologies, such as remote sensing, for M&E should be considered (see GN8). · Ideally there should be sufficient capacity and resources to allow ad hoc special studies or investigations to be carried out to address specific problems or issues revealed by the on-going evaluation of monitoring data. These will be one-off and focused investigations of the issue at hand (see GN9). 6. Select results targets · Target setting is the final step in building a results-based approach, following logically from defining outcomes, indicators and baselines. A target is a specification of the quantity, quality, timing and location to be realised for a key indicator by a given date. Starting from the baseline level for an indicator the desired improvement is defined taking account of planned resource provision and activities, to arrive at a performance target for that indicator. Most targets are set annually, but some could be set quarterly or for longer periods. Targets do not have to be single numerical values and sometimes a range of achievement may be more appropriate. Targets should also be kept under review and revised flexibly as necessary to take account of changing resource availability or other factors beyond the control of project management, but not to disguise poor project performance. · It is important to be realistic, taking account of what is feasible and being sensitive to the political issues associated with targets that are publicly announced. As outcomes are typically longer term it is usually necessary to establish targets as short-term outputs on the path to achievement of an outcome. For project management, targets for `leading indicators' (see below and GN3) are particularly useful. Interim targets over shorter time periods for which inputs can be better known or estimated, and set with reference to desired outcomes and impact, are also important for process-orientated interventions for which work plans and resource provision are not fully planned in detail in advance. Information from benchmarking for irrigation scheme performance may assist the setting of targets when relevant (see GN18). 7. Plan monitoring, data analysis, communication and reporting · `Implementation monitoring' tracking the inputs, activities and outputs in annual or multiyear work plans, and `results monitoring' tracking achievement of outcomes and goals, are both needed. The demands for information at each level of management need to be established, responsibilities allocated, and plans made for: · what data to be collected and when; · how data are collected and analysed; · who collects and analyses data; · who reports information, and in what form, to whom and when? · An assessment of the flow of information and degree of detail needed by each level of management will help to clarify the indicators to be measured. The agency managing the project will require different types of information for its own internal management, compared to the reporting requirements of higher levels of government and the World Bank (GN6, GN12, and GN13). · Participation of stakeholders in the M&E system will have important implications on data collection mechanisms, analysis, reporting, and use (see GN11) 8. Plan the form and timing of critical reflection and interim evaluations · For managers evaluation should be a continuously available mode of analysis utilised whenever evaluation results can be useful. Scheduling of events such as management team meetings can, however, be useful to ensure that analysis of progress and critical reflection takes place. Similarly, periodic project review workshops to facilitate analysis 35 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects and discussion with project partners and other stakeholders may be necessary. Supervision requirements of governments and funding agencies may require periodic and formalised evaluations to take place. The data needs and analysis requirements for mid- term, terminal and ex-post evaluations should be considered, and planning for these linked to the planning of monitoring and choice of evaluation framework. A timetable of formal evaluation reports should be set out (GN1, GN9). · An indication also needs to be given at the design stage about feedback mechanisms for evaluation results beyond donor formalities such as mid-term and completion reviews. This is linked both to the development of accountability within the project, sector and higher levels of government, and the need to provide information to support decision making. For example, flows of information may need to be timed to fit into national budget planning activities, and should inform and influence identification and appraisal of any similar future projects or programmes. · Benchmarking can be a useful tool for regular performance assessments and evaluations (GN18) 9. Plan for the necessary conditions and capacities. · It is necessary to plan the organisational structure for M&E, including whether a M&E unit specific to the project is needed. Appropriate organisational structures for M&E should be discussed with partners and other stakeholders. Each partner's responsibilities and information requirements should be considered (GN12). Planning should cover: staffing levels and types, responsibilities and internal linkages, incentives and training needs, relationships with partners and stakeholders, horizontal and vertical lines of communication and authority, physical resource needs and budget. · Monitoring and ongoing evaluation should normally be the responsibility of the project managers. More formal evaluation may often require the expertise and capacity of external specialists. See GN5 for more details. 4.3 The components of a project monitoring and evaluation system Section 4.2 detailed suggested steps in planning a M&E system. Based on this process, a good M&E system has six main components (Box 6). These six components help to ensure that M&E is relevant to the project, within the capacity of the borrower's organisations, and is used to good effect. Each is considered briefly below. Box 6: The main components of a project M&E system 1. Clear statements of measurable objectives for the project and its components. 2. A structured set of indicators covering: inputs, process, outputs, outcomes, impact, exogenous factors and cross-cutting factors. 3. Data collection mechanisms capable of recording progress over time, including baselines and a means to compare progress and achievements against targets. 4. Where applicable, building on data collection with an evaluation framework and methodology capable of establishing causation (attribution). 5. Clear mechanisms for reporting and use of M&E results in decision making. 6. Sustainable organizational arrangements for data collection, management, analysis and reporting. Source: Authors 1. Project objectives Projects are designed to contribute to long-term sectoral development objectives, especially as identified in the CAS, but at the level of project development objective their outcomes should be quite specific and complete. Thus an agricultural water management project may be designed to 36 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects further the sectoral development objectives of increased agricultural productivity, farm incomes and rural employment, but have a project development objective of providing an increased and more reliable irrigation supply through rehabilitation or modernization of an irrigation system (see GN1 and GN2 for guidance on the setting of objectives). Objectives at the level of the PDO should be specific to the project interventions, realistic in the timeframe for their implementation and measurable for evaluation. 2. Structured indicators Indicators provide the qualitative and quantitative detail necessary to monitor and evaluate the objectives at all levels of the project hierarchy. The ability to define an indicator, and agree with partners and stakeholders a target for its achievement, is a demonstration that project objectives are clearly stated, and are understood and supported (see GN3 and GN 14, 15 and 16). The logical framework approach to project design provides an effective structure for planning M&E by defining a hierarchy of objectives for which indicators are required (Figure 8). Classifying project objectives according to their level highlights that management will need to develop systems to provide information (data collection systems) at all levels, from basic accounting through to statistics of project impact. Ultimately constructing good indicators will be an iterative process. Figure 8: A logical structure for project monitoring and evaluation indicators Logic Indicators Source of information on the indicators Objectives Impact Long-term statistical evidence PDO and Outcomes Social and economic surveys of project effects and outcomes. component Plus leading indicators giving management advance warnings outcomes/ from beneficiary perceptions, responses to the project and other Exogenous results measures of performance. and cross- Outputs Output Management observation, records and internal reporting. cutting Activities Process Task management of processes. indicators Financial accounts. Management records of progress. Procurement processes. Inputs Input Financial accounts. Management records of inventories and usage. Source: Authors Input indicators are quantified and time-bound statements of the resources financed by the project, and are usually monitored by routine accounting and management records. They are mainly used by managers closest to implementation, and are consulted frequently (daily or weekly). They are often left out of discussions of project monitoring, though they are part of essential management information. An accounting system is needed to track expenditures and provide data on costs for analysis of the cost effectiveness and efficiency of project processes and the production of outputs. Process indicators monitor the activities completed during implementation, and are often specified as milestones or completion of sub-contracted tasks, as set out in time-scaled work schedules. One of the best process indicators is often to closely monitor the project's procurement processes. Every output depends on the procurement of goods, works or services and the process has well defined steps that can be used to monitor progress by each package of activities. Output indicators monitor the production of goods and services by the project, e.g. kilometres of canal rehabilitated, number of water user associations formed and user fees collected per time period. They are often set with the use of performance measures based on operational ratios or cost, e.g. water conveyance efficiency, cost per kilometre of canal rehabilitation/ modernization, 37 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects cost per hectare of command area, and recurrent operation and maintenance costs assumed by water user associations. The indicators for inputs, activities and outputs, and the systems used for data collection, recording and reporting are sometimes collectively referred to as the project physical and financial monitoring system, or management information system. The core of an M&E system and an essential part of good management practice, it can also be referred to as `implementation monitoring'. They are not included in the PAD Results Framework, but their definition and monitoring is a key aspect of project design and supervision (GN6). Outcome indicators are specific to a project and are the end of the logical chain of cause and effect that underlies its design. Often achievement of outcomes will depend at least in part on the actions of beneficiaries in responding to project outputs, and indicators will depend on data collected from beneficiaries, e.g. change in crop yields or cropping pattern, and investment by farmers in land and water management improvements. It will usually be important for project management to try to gain early indications of project performance in achieving outcomes through the use of leading indicators of outcomes. These may often be obtained by surveying beneficiaries' perceptions of project outputs and services, e.g. farmer perceptions of improved reliability of irrigation supply. Such leading indicators have the twin advantages of consultation with primary stakeholders and advance warning of poor project performance. In agricultural water management projects technical measures of system performance compared to design assumptions can also be a leading indicator of whether the project has established the potential to achieve its higher level objectives. Impact indicators usually refer to medium or long term developmental change to which the project is expected to contribute. Dealing with the effects of project outcomes on beneficiaries, measures of change often involve statistics concerning economic or social welfare, collected either from existing regional or sectoral statistics or through relatively demanding surveys of beneficiaries. The adoption of standardized key outcome and impact indicators for agricultural projects is increasingly under consideration ­ such as the probable inclusion of land and labour productivity indicators in all future World Bank African agricultural projects. These key indicators would show clearly the achievement of broad objectives and allow comparison across interventions. They should still be complemented by project-specific leading indicators and outcome and impact indicators. World Bank staff should check regional and sectoral guidelines and procedures in place in this regard. Exogenous indicators are those that cover factors outside the control of the project but which might affect its outcome, including risks (parameters identified during project design that might compromise project benefits) and the performance of the sector in which the project operates. Use of logical framework analysis for project design will guide the identification of exogenous indicators to match the key assumptions made about necessary external conditions at each level of the logical hierarchy. Cross-cutting indicators may be needed for cross-cutting components of the project which are broader in focus than the specific logical chain of cause and effect that underlies the project design. Examples include capacity building and training or the updating of existing legislation. Alternatively there may be cross-cutting issues which the project may influence at all levels of the project hierarchy. Examples of this are protection of the environment, gender issues, and employment opportunities for poor people. If there is a need to monitor the wider environment, this calls for additional data collection capacity and places an additional burden on a project's M&E programme. This may be best met through use of existing data sources or assignment of the responsibility to another agency (see 38 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects GN5). Pragmatic judgment is required in the careful selection of indicators, and GN3 looks at the necessary characteristics and properties of indicators in detail, with examples relevant to agricultural water management projects. 3. Data collection mechanisms For project monitoring and evaluation, data will be needed both for the baseline situation and for measurement of change over time in the range of indicators selected. In starting to build a management information system, considerations that come first are the sources of information that can potentially supply the relevant data, the reliability of the information, and associated costs and responsibilities. Data collection will incur costs of staff time and other resources, whilst excessive collection of unnecessary data will slow down processes of analysis and reporting, and may lead to a failure to communicate clear messages. Thus it is important to collect only the data that will be used effectively to improve management and decision making. The plan for a project M&E system should be based on a clear and detailed assessment of the following: · What ­ the data to be collected, in what form, with what degree of aggregation or consolidation, and for what purpose; · When ­ the frequency of data collection and reporting; · Who ­ the responsible persons, their responsibilities and capacities; · How - methods and procedures for data collection, checking, validation and storage, and for analysis and reporting; · Where ­ locations for data collection and processing, and the destinations for reported information. The range of data collection methods Project monitoring and evaluation will often make use of a wide range of methods for gathering, analysing, storing and presenting data (for more guidance on methods see GN6). There is no single answer as to which method is best, as this will depend on an organisation's resource availability, access to the sources of data, purpose for the data, and time constraints. Structured and formal methods for data collection (GN6) will tend to be more accurate and reliable, but also more costly and time consuming. For data that is needed frequently and on a routine basis to inform management decision making, it may be preferable to adopt less structured and less costly collection strategies. Rigorous approaches to impact evaluation that address the problem of attribution (Chapter 3.0 above) will generally required a more formal and structured approach (GN6, GN9), and hence may need to be applied selectively. Before decisions are made on the data collection and management strategies to employ, it is important to consult with the users of the information. What are their needs and priorities for the information they require to improve their decision making and the overall process of project implementation? What are their perspectives on the trade-offs that may need to be made? Data collection plans should not be permanently fixed from the commencement of the project. As project management responds to changing circumstances and adopts an adaptive approach to implementation, so will information needs change. There needs to be sufficient adaptability and flexibility in the M&E system to identify new indicators, data sources, collection methods and ways of reporting as required. Core data collection methods for AWM projects Indicators for inputs, processes and outputs will generally come from project management records originating from field sites. The quality of record keeping in the field sets the standard for all other use of data and merits careful planning and attention. It is important that the data collection is systematic and that data are collected on time for all specified periods. The seasonal 39 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects nature of agricultural production imposes particular requirements and constraints upon this. M&E designers should together with managers determine what information will be useful to managers at field, intermediate and senior levels, and how and why it will be useful. Together they should also assess the capacity of existing record-keeping and reporting procedures to generate the information that will be needed. To measure outcomes and impact will typically require the collection of data from formal sample surveys, used in combination where appropriate with methods of participatory rural appraisal (PRA) or rapid rural appraisal (RRA). There must be adequate capacity to do this for baseline data collection and repeat surveys that will compile a continuous or periodic time series of data for key indicators. Where possible, it may be better to add project-specific regular surveys on to existing national or area surveys than to create a new data collection facility (see GN5 for more guidance on organizational arrangements). When M&E reveals problems, there may be need for ad hoc diagnostic studies. Routine monitoring and evaluation, particularly of leading indicators of outcomes, may reveal problems during implementation. An example would be a disappointing response rate among primary beneficiaries such as a low rate of credit uptake to finance improvements in on-farm water management. Such situations may call for ad hoc diagnostic studies to determine the cause of the problem and identify possible solutions. Such studies may call for staff research skills and training beyond those needed for regular collection of data, and thus may need to involve managers themselves or be subcontracted to a university or consultants. Whatever data collection methods are selected, the plans for the monitoring and evaluation of a project should explain and justify the proposed approach and ensure consistency in methods. 4. An evaluation framework and methodology As a continuously available mode of analysis for project managers, ongoing evaluation can be used to address the following key questions for agricultural water management projects. · Is the response of beneficiaries as anticipated and satisfactory? · What are the effects of the project on agricultural production? · Are there any unanticipated effects, positive or negative, for the project or in relation to its wider environment? · Can the causes of all observed changes be discerned and established with evidence? · Does the logic of the intervention model of the project remain valid (i.e. is it the right design)? · Are any ad hoc special or diagnostic studies needed to help answer any of these questions? Such ongoing evaluation feeds into the periodic and more formalised evaluations also typically required (see GN9). For these the focus is whether changes have occurred and what has been the cause. Thus the evaluation tries to determine what portion of the observed and monitored impacts the project caused, and what might have been the result of other events or conditions. The aim is thus attribution of documented change. This type of evaluation is challenging, and it is usually conducted until after the end of the project implementation period, when outcomes and impact will have had time to fully emerge. The longer the time between the project's implementation and the attempt to attribute change, the more likely it is that other factors will also have had a significant influence. The evaluation framework lays out the analytical tools that will be used to address this problem, usually by testing observed change against a counterfactual (i.e. the situation that would have happened had the project not taken place). Identifying the counterfactual is difficult but there are strategies for doing so, using both experimental and quasi-experimental designs (see GN9). Use of random assignment and control or comparison groups are the basic strategies that can be adopted. 40 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects 5. Clear mechanisms for reporting and use of M&E results in decision making There is a range of possible users for the results of monitoring and evaluation of development projects. These include primary stakeholders, the project implementation unit, government agencies, other implementing partners, and donors (see GN12). Clear feedback mechanisms are important if the objectives of M&E itself are to be achieved (see GN6). The key issues requiring attention at the design stage of M&E are set out in Steps 7 and 8 of the design process above. Providing the right information in the right place and right form to be used by the right person in decision making is the ultimate aim. A good flow of information is also closely linked to the development of accountability within the project, sector, government, and donor. In many countries, information on projects and programmes is poor and difficult to access, and the mechanisms for feedback are weak or non existent. The highest payoffs to evaluation arise at the policy and programme level, but project- level evaluation offers an easier and less sensitive starting point in many instances. Information from monitoring and evaluation can be used to demonstrate accountability and to promote knowledge transfers and adaptive learning in government agencies and other organizations. The uses of the information and the feedback mechanisms need to be structured and scheduled according to the needs of managers and other partners and stakeholders. For example: · Project management will need to monitor expenditure and progress against schedules, weekly and at least monthly. · Outputs are unlikely to be measurable at less than three-monthly intervals, and some may need much longer. · Consultations with beneficiaries, or surveys of their satisfaction with project services, should be timed to supply information to use in planning project activities. · The time period for reporting may vary with the level of management: for example, monthly at district level, quarterly at regional or state level. · Some flows of information need to be timed to fit into national budget planning activities. · Annual funding may depend on the results from previous work. · Periodic mid-term and terminal reviews provide milestones by which information has to be ready. · Processes of project identification, preparation and appraisal should show evidence of having made use of the lessons of evaluations of similar projects or programmes. From the start of the project a communication strategy needs to be developed that will address the following questions: · Who will receive what information? · In what format? · When? · Who will prepare the information? · Who will deliver the information? World Bank procedures require that the task team reports at least annually on outcome and impact indicators (see GN13); though reporting to the client country and stakeholders may usefully be more frequent. In relation to World Bank supervision and the utilization of M&E generated information, it is good practice that an M&E report should be produced by the implementation agency at least every six months as part of a bi-annual project progress reporting cycle, and be used by the World Bank on each and every supervision mission. Information should be reported concisely, be relevant to the user and be timed to improve key decision-making events. Four means of communication may be used and will reinforce each other: detailed written information (reports), written executive summaries, and oral and visual presentations. 41 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects Use of the findings to improve performance is the main purpose of building a M&E system. The Results Framework used in World Bank projects thus requires identification of mechanisms for use of M&E information (see GN13). Findings can be used in a variety of ways to improve project performance: · to support adaptive management, `flagging' problems and successes, and provoking revision of strategy; · to identify and plan for additional needs and resource requirements; · to trigger in-depth examinations (diagnostic studies) of any performance problems that exist, and to identify corrections needed; · to improve operational resource allocation decisions; and · to inform periodic formal assessments such as a mid-term review. 6. Sustainable organizational arrangements for data collection, management, analysis and reporting In terms of organisational arrangements there is no one correct way to build a project M&E system. AWM projects vary in their characteristics and requirements, and countries and organisations are at different stages of development with respect to good public management practices in general, and M&E in particular. It is also important to recognise that M&E systems are continuous works in progress that must be flexible and adaptable to changing needs and circumstances. Organization of M&E and project logic The concepts of project logic (Chapter 3.0) draw attention to the fact that project management will need to develop systems to provide information at all levels, from basic operational inventories and accounting through to generation of statistics about outcomes and impact. Building on the structure of indicators, Figure 9 shows the typical nature and location of responsibility for M&E components at each level in the Results Framework. The right hand column of Figure 9 should be regarded only as an illustrative guide to be adapted as necessary. Certainly inputs, activities and their outputs are within the control of project management (Figure 6, Chapter 3.0) and can be monitored and evaluated through internal record-keeping and progress reporting, analysis of this information, and management review. Generally management will want to integrate monitoring with other systems such as financial accounting and computerised project management, and development of such comprehensive management information systems should be supported in the project design. In contrast, the achievement of project outcomes normally depends on how project beneficiaries respond to the goods and services delivered by the project. Compiling evidence for leading indicators of their response and the benefits they derive requires consultation, research and data collection skills that may be beyond the capacity of the project management organisation, but if so, must be carried out in close partnership with it. Then because outcome and impact evaluation will only be measurable towards the end of implementation, or in later years, and because it also requires higher levels of research and analytical skills and objectivity (Chapter 3.0), it may often be better done by a separate agency, independent from implementation. 42 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects Figure 9: A logical structure for organization of project monitoring and evaluation Objectives Indicators M&E components Responsibility for M&E Long term statistical evidence of project impact Higher level National or sectoral agencies and development Impact Exogenous indicators for risk /or independent specialists objectives factors and unanticipated wider environmental and social impacts Socio-economic surveys PDO and Project management and/or component Outcomes M&E of leading indicators independent specialists outcomes/ results Diagnostic studies Outputs Output MIS for physical and financial Activities Process Project staff monitoring Inputs Input Source: Authors However, it must be emphasised that monitoring and evaluation is too important to be left only to independent specialists. Subject to the distinctions outlined above it should generally be an integral part of all project and programme planners' and managers' duties. Monitoring is a tool of good management and the responsible unit should ideally be located within or close to project management. Thus the resources, training and technical assistance for the unit should be specified in the project implementation plan. Where independent external expertise is needed it should be procured through partnership in the case of national or sectoral agencies, or contracted on a consultancy basis supervised by project management in the case of other agencies. An administrative unit for project level M&E that is separate from project management may only be justified in agencies with a weak management history and very limited capacity, or for projects with multiple components implemented by multiple agencies. In the latter case, the M&E unit should still be well integrated into the overall coordinating arrangements for the project or programme. Ultimately the aim should be that project level M&E is well integrated with the planning and management of all policies and interventions by the relevant government departments. Ideally, the use of information generated by project level M&E should become `institutionalised', so that there is no disconnect between project implementation and the overall approaches and strategy of public sector management. This may require capacity building within the government agencies concerned. More detailed consideration of organisational arrangements for project M&E is provided in GN5, and RN5 provides some indications of the costs and resources required for effective M&E of AWM projects. Further reading Baker, J. 2000. Evaluating the impact of development projects on poverty: a handbook for practitioners, Volume 1. Washington D.C.: World Bank. Available at http://go.worldbank.org/8E2ZTGBOI0 Bos, M.G., M. Burton, and D.J. Molden. 2005. Irrigation and drainage performance assessment - practical guidelines. CABI publishing, Wallingford, UK. Casley, D., and K. Kumar. 1987. Project monitoring and evaluation in agriculture, a joint study: The World Bank, International Fund for Agricultural Development, Food and Agricultural Organization of the United Nations. Baltimore, USA; Johns Hopkins University Press. 43 AWM M&E Toolkit Toolkit for Monitoring and Evaluation of Agricultural Water Management Projects Guijt, I., and J. Woodhill. 2002. A Guide for Project M&E. Rome, International Fund for Agricultural Development (IFAD). Available at http://www.ifad.org/evaluation/guide/index.htm Kusek, J.Z., and R.C. Rist. 2004. Ten Steps to a Results-Based Monitoring and Evaluation System. . Washington D.C.: World Bank. Available at: http://go.worldbank.org/7H1YFQS5Z0 OECD (Organisation for Economic Co-operation and Development). 2002. "Glossary of Key Terms in Evaluation and Results-Based Management." Paris: OECD/DAC. Rajalahti, R., J. Woelcke, et al. 2005. Monitoring and Evaluation for World Bank Agricultural Research and Extension Projects: A Good Practice Note. Agricultural and Rural Development Discussion Paper 20. Washington D.C.: World Bank. Available at http://intresources.worldbank.org/INTARD/Resources/M_E_WB_AgExtensionProjects.pdf World Bank Operations Policy and Country Services. 2004. PAD template and guidelines. Washington D.C.: World Bank. Available at http://intresources.worldbank.org/INTINVLENDING/Resources/PADGuide.pdf World Bank. 2006a. Directions in Development: Reengaging in Agricultural Water Management. Challenges and Options. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTARD/Resources/DID_AWM.pdf World Bank. 2007b. Water Supply and Sanitation Monitoring and Evaluation Toolkit. Draft, October 7, 2007. World Bank Independent Evaluation Group. 2006. Water Management in Agriculture: Ten Years of World Bank Assistance, 1994-2004. Washington D.C.: World Bank. Available at: http://go.worldbank.org/ON8M9NGOZ0 Websites World Bank Operations Policy and Country Services: http://go.worldbank.org/68SVPDDGV0 World Bank Sustainable Development Network Results website: http://go.worldbank.org/ZG0PRX9260 44 AWM M&E Toolkit Toolkit for monitoring and evaluation of AWM projects Guidance Note 1 Building a Results Framework for M&E of AWM Projects Guidance Note 1 Building a Results Framework for Monitoring and Evaluation of Agricultural Water Management Projects Introduction Monitoring and evaluation systems are designed to inform project management (World Bank Task Teams and client's project implementation unit) whether implementation is going as planned and whether corrective action is needed to adjust implementation plans. In addition, M&E systems should provide evidence of project outcomes and justify project funding allocations. In 2003 the World Bank moved from use of logical framework analysis as a mandatory requirement in Project Appraisal Documents to a strategy comprising use of a results framework and specification of the arrangements for results monitoring (see GN13). The added emphasis this places on the outcomes of a project, and those of all its components, helps focus project design and management on results. It also forces the M&E system to track these outcomes and impacts rather than simply reporting on production of outputs. As discussed in Part A, Chapter 3, this will often present methodological and resource challenges but appropriate approaches can be selected and tailored to the requirements of specific projects (Guidance Notes GN6, GN7 and GN9 provide more guidance on these approaches). Thus there has been some shift in emphasis from `implementation monitoring' to `results monitoring' (Part A, Section 3.5) based on the desire to ensure that managers, stakeholders, project partners and policy makers gain an understanding of failure or success of the project in reaching the desired outcomes and impact. However, results based M&E systems should still build upon, and add to, implementation focused systems that track input mobilization, activities undertaken and completed, and outputs delivered. These should not be neglected even though increased emphasis is placed on project outcomes. This is particularly the case for `blueprint' oriented projects that make significant investments in construction and other civil works or forms of `hardware'. The results based emphasis is relevant to all projects but is particularly important to the management and evaluation of `process oriented' interventions, particularly those at sector and policy level, for which implementation is less programmed in advance. Building a results framework Overview A results framework is a simplified version of logical framework analysis (see Part A, Chapter 3, and GN2). It focuses on: · the Project Development Objective(s) (PDO); and · intermediate outcomes expected from implementing each individual project component, each of which contribute to the achievement of the PDO. These are sometimes also called intermediate results. For inclusion in the Project Appraisal Document (PAD) the results framework should have the following structure: · statements of the project development objective and intermediate component outcomes; · indicators for the project development objective outcome and intermediate component outcomes/ results; and · an explicit statement of how the outcome information should be used. GN1 Results Framework 1 Toolkit for monitoring and evaluation of AWM projects Guidance Note 1 Building a Results Framework for M&E of AWM Projects It is important to note that the terminology is evolving as the results framework approach is refined, but general principles stay the same. The structure of the results framework as currently presented in the PAD and an example for as agricultural water management project are shown in Table 1. Table 1: The World Bank's results framework for an AWM project PDO Outcome Indicators1 Use of Outcome Information (i) Strengthened state capacity for multi (i) Appropriate institutions created (i) Identification of additional sectoral planning, development and and strengthened activities required sustainable management of water resources, (ii) Continuation of financing of and (ii) Improved irrigation service this project and dialogue with (ii) Improved irrigation service delivery delivery in targeted schemes Government of Maharashtra. on a sustainable basis to increase productivity of irrigated agriculture and (iii) Incremental value from crop contribute to rural poverty reduction. production (rupees) per unit of irrigation water supply (iv) Improved incomes of targeted stakeholders in head, middle and tail reaches. Intermediate Results or Outcomes Results Indicators for Each Use of Results Monitoring (one per component) Component Component One: Component One: Component One: Institutional Restructuring and Capacity (I) Maharashtra Water Resources i) Adaptive management Building Regulatory Authority (MWRRA): ii) Supervision Planning (i) Adoption of MWRRA Act; iii) Outlining additional needs Strengthened capacity for multi sectoral (ii) Framing of the rules of the Act; planning, development and (iii) Formation of the Authority; management of water resources in (iv) Operationalisation of MWRRA Maharashtra's river basins (adequate staffing and budget allocation). (II) Restructuring of MKVDC: (i) Amendment of MKVDC Act; (ii) MKVDC restructured into MKVWRC and functional (adequate staffing, budget allocation and work plan finalized); (iii) Finalisation of the Krishna Valley River Basin Plan. (III) Restructuring the Water Resources Department (WRD): (i) Award and completion of restructuring study (ii) Completion of restructuring of the WRD. Component Two: Component Two: Component One: Improving irrigation service (i) WUAS established in about 286 i) Adaptive management delivery and management targeted schemes; ii) Supervision Planning (ii) Fully functioning WUAs in about iii) Outlining additional needs Increased productivity of irrigated 286 targeted schemes; agriculture (iii) Volumetric charging and bulk supply of water as per entitlement in 6 schemes in the Krishna Basin; (iv) Increased water delivery GN1 Results Framework 2 Toolkit for monitoring and evaluation of AWM projects Guidance Note 1 Building a Results Framework for M&E of AWM Projects efficiency; (v) Improved water use efficiency at the scheme, distributory and minor levels (vi) Incremental income from crop production (Rupees) per unit of irrigation water supply; (vii) Improved fee collection efficiency from agriculture. Source: Adapted from World Bank, 2005f, Maharashtra Water Sector Improvement Project PAD Note: components three and four are not shown here. Note: MKVDC: Maharashtra Krishna Valley Development Corporation MKVWRC: Maharashtra Krishna Valley Water Resources Corporation See GN15 17 for guidance on indicators appropriate to outcomes. Key factors to consider in building the results framework as shown in Table 1 are outlined below. Specifying the project development objective (top row of column one) A clear statement of the project development objective(s) is essential for good project design. For most projects the PDO is expected to describe the main expected outcome of a change in behaviour of the project's primary beneficiaries. In AWM projects this change in behaviour will typically be expressed through change in the technology, productivity, form and/or value of agricultural production, leading to an increase in household income. Common weaknesses in project design include poor articulation and a lack of clarity in the specification of the PDO. Often it may be expressed at too high a level (e.g. relating to Country Assistance Strategy) or too low a level (e.g. relating to project activities). This can be seen as a confusion of the higher level objectives, PDO, and output levels in the logic of project objectives (see Part A, Chapter 3, and GN2). From reading the specification of the PDO the following should be clear: · who are the target beneficiaries and where are they located? · by the end of the project, what problem will have been addressed for this primary target group? · what will this group be doing differently after the project and/or what will be the scale and nature of the change that they experience in terms of well being or other aspects of economic and social development? Key points to remember are that: · specification of the PDO should focus on an outcome that the project can directly achieve given its duration, resources, and approach, and thus it should not encompass longer term outcomes that depend on efforts beyond the project's scope and beyond the control of project management; · the PDO should be kept as short, focused, and measurable as possible; · the PDO should relate to the achievement of the project as a whole and should not simply restate the outputs or outcomes of project components. Specifying the intermediate outcomes/ results of project components (lower rows of column one) The key here is to make clear statements that show how the project components contribute to achieving the project development objective. Thus the expected intermediate outcome or result from each component must be clearly linked to the achievement of the PDO. GN1 Results Framework 3 Toolkit for monitoring and evaluation of AWM projects Guidance Note 1 Building a Results Framework for M&E of AWM Projects For AWM projects intermediate outcomes/ results of project components could take a number of forms, e.g.: · changes in the skills or capacities of a secondary target group (e.g. irrigation agency staff) that must be achieved for them to be able to better serve the primary target group (e.g. irrigating farmers); · measurable improvement in the operating performance of a water delivery system as a result of modernization and/or rehabilitation; · promulgation of new legislation that facilitates turnover of tertiary system management to water user associations. Selecting indicators for the project development objective and each project component outcome/ result (column two) Indicators are quantitative and qualitative variables that provide a means to measure change over time. They can be used to assess the performance of a project compared to planned targets, and to provide evidence that the change observed was the result of the project interventions made. The World Bank PAD requires indicators to be specified for the PDO and the outcome(s) of each project component. If a full logical framework analysis is completed, then indicators will also be defined for the higher level development objectives and for outputs, activities and inputs. GN3 provides more guidance on the selection and specification of indicators for AWM projects (Part A, Chapter 4, also introduces key concepts). Specifying how the outcome/ result information should be used (column three) This column of the results framework is used to describe when and how to take corrective action if the project is at risk of not achieving agreed targets for the selected indicators. Thus clear and time bound targets for the selected indicators that are measured throughout the project lifespan are needed (see Part A, Chapter 4, and GN3 for more guidance on setting and use of targets). The level of detail and flexibility of guidance provided in this column can vary, but in general the more quantification and specification of detail the better. The following statement provides an example: If the rate of tail ender farmers' satisfaction with watercourse improvement is less than 75% one year after work is completed, then the design and implementation protocols for this project component will be reviewed and adjusted. GN6 provides more guidance on using findings. Specifying data collection, reporting and dissemination requirements Accompanying the results framework in the PAD should be a specification of arrangements for data collection, reporting, dissemination and use for decision making. For each of the outcome indicators for the PDO and the intermediate component outcomes/ results, this specification will identify the following: · the source of baseline data for the indicator (see Part A, Chapter 4, and GN7 for more guidance on baseline data and surveys); · target values, usually annual for each year of the project's implementation (Part A, Chapter 4, and GN3); · the frequency of data collection and reporting; · the instruments or methods to be used for data collection; and · who has responsibility for data collection. GN1 Results Framework 4 Toolkit for monitoring and evaluation of AWM projects Guidance Note 1 Building a Results Framework for M&E of AWM Projects Table 2 provides an example of the specification of arrangements for outcome monitoring as required in a PAD for the same project example as Table 1, i.e. the Maharashtra Water Sector Improvement Project. Completion of the `Arrangements for Outcome Monitoring' table requires the prior development for the project of a plan for monitoring and evaluation and for a management information system. Guidance on how to develop such a plan is provided by this toolkit (see Part A, Sections 4.2 and 4.3). Adequate institutional arrangements, and organisational and human capacity, are essential for an effective M&E system. This is relevant to determination of who has responsibility for data collection and is covered further in GN5. GN1 Results Framework 5 Toolkit for monitoring and evaluation of AWM Guidance Note 1 projects Building a Results Framework for M&E of AWM Projects Table 2: The World Bank's specification of arrangements for outcome monitoring for an AWM project Target Values Data collection and reporting Outcome Baseline YR1 YR2 YR3 YR4 YR5 YR6 Frequency Data Responsibility Indicators and Collection for Data Reports Instruments Collection (i) Appropriate 100% Bi annual M&E Project institutions created progress surveys, Management and strengthened report MIS, Unit, scheme independent completion M&E agency reports (ii) Improved Baseline 0% 0% 15% 25% 30% 30% irrigation service for (ii) to delivery in targeted (iv) to be schemes established by MWSIP baseline survey. (iii) Incremental 0% 0% 5% 5% 10% 20% value from crop production (rupees) per unit of irrigation water supply (iv) Improved 0% 0% 15% 25% 30% 40% incomes of targeted stakeholders in head, middle and tail reaches. Outcome Indicators for each component Component One: Quarterly M&E Independent progress surveys, M&E (I) Maharashtra reports MIS, consultants Water Resources None scheme Regulatory Authority completion M&E by (MWRRA): reports, Project (i) Adoption of 100% impact Management MWRRA Act; evaluation at Unit (ii) Framing of the project end rules of the Act; 100% (iii) Formation of the Authority; 100% (iv) Operationalisation 100% of MWRRA GN1 Results Framework 6 Toolkit for monitoring and evaluation of AWM Guidance Note 1 projects Building a Results Framework for M&E of AWM Projects Table 2 (continued) Target Values Data collection and reporting Component Quarterly M&E Independent M&E Two: progress surveys, consultants MWSIP 100% reports MIS, (i) WUAS baseline scheme M&E by Project established in survey completion Management Unit about 286 and reports, targeted scheme 100% impact schemes; reports evaluation (ii) Fully at project functioning 50% 100% end WUAs in about 286 targeted schemes; (iii) Volumetric charging and bulk supply of water as per 20% 60% 100% entitlement in 6 schemes in the 30% 30% 40% Krishna Basin; (iv) Increased water delivery efficiency; (v) Improved 0% 0% 15% 25% 30% 30% water use efficiency at the scheme, Scheme distributory and reports minor levels 0% 0% 5% 5% 10% 205 (vi) Incremental income from crop production (Rupees) per MWSIP unit of irrigation baseline water supply; survey (vii) Improved collection efficiency from agriculture. Source: Adapted from World Bank, 2005f, Maharashtra Water Sector Improvement Project PAD. Note: components three and four are not shown here. Note: MWSIP: Maharashtra Water Sector Improvement Project MWWRRA: Maharashtra Water Resources Regulatory Authority GN1 Results Framework 7 Toolkit for monitoring and evaluation of AWM Guidance Note 1 projects Building a Results Framework for M&E of AWM Projects Coverage of M&E features lacking in the results framework but necessary in the PAD In this guidance note the results framework approach for project M&E planning adopted by the World Bank has been described. It provides a convenient means to summarise M&E requirements within the Project Appraisal Document, and it usefully forces managers to focus on achievement of the planned outcomes of project components and the Project Development Objective(s). However, there are three important aspects of both project and M&E design that are not included in the results framework. 1) The results framework does not explicitly refer to Country Assistance Strategy or sector goals, but the PAD for World Bank supported projects is required to articulate a clear alignment between the project and higher order strategic, programme, or sector outcomes. Thus the "Strategic Context" section of the PAD should describe how the project contributes to these higher level objectives. 2) The results framework does not explicitly set out expected project component outputs, activities and inputs, but specification of these is required in the main text of the PAD. `Implementation monitoring' (see Part A, Chapter 3) should be well planned using the concepts and methods covered in this toolkit. 3) The results framework does not explicitly capture the assumptions that may need to be made about necessary external conditions for project success (i.e. the factors that the project cannot directly control but which are essential for achieving each level of outcomes and objectives, see Part A, Chapter 3). Identification and assessment of these critical assumptions and the risk of their non fulfilment is also required in the main text of the PAD. Thus although it is only mandatory to specify the results framework and the arrangements for results monitoring in the PAD, the project team will often wish to complete a full logical framework analysis during the design process. This can have the following advantages: · it usually helps the project team and the client to reach a shared understanding of the project and to precisely define the different components, their inter linkages and their contributions to outcomes and impact; · it can be a useful tool for facilitating discussions and development of the project design with the national team; · it can be used as a management tool throughout the project cycle; · it assists in establishing the implementation monitoring and evaluation programme set up by the project staff, by detailing performance indicators for project activities, outputs and impacts; and · it can support the development of a sustainable M&E system beyond the lifetime of the project as it is often a better fit with the requirements for M&E in the client countries than the more limited specification of the results framework. GN2 provides an example of logical framework analysis and brief guidelines for its use. Further reading Kusek, J.Z., and R.C. Rist. 2004. Ten Steps to a Results Based Monitoring and Evaluation System. . Washington D.C.: World Bank. Available at http://go.worldbank.org/7H1YFQS5Z0 Rajalahti, R., J. Woelcke, et al. 2005. Monitoring and Evaluation for World Bank Agricultural Research and Extension Projects: A Good Practice Note. Agricultural and Rural Development Discussion Paper 20. Washington D.C.: World Bank. Available at http://intresources.worldbank.org/INTARD/Resources/M_E_WB_AgExtensionProjects.pdf GN1 Results Framework 8 Toolkit for monitoring and evaluation of AWM Guidance Note 2 projects Logical Framework Analysis Guidance Note 2 Logical Framework Analysis Background Logical framework analysis (LFA) was developed in the 1960s to improve project planning, monitoring and evaluation. It was developed to address weaknesses at that time, including: · poor planning, without clear objectives and specification of desired project outcomes; · inadequate specification of M&E processes and indicators · failure to consider external factors and take account of risks. LFA has been adopted by several development agencies as a project planning and management tool. It is not mandatory to complete a LFA for World Bank projects but existing guidelines and procedures acknowledge its utility as a planning and management tool. It is mandatory to prepare a Results Framework for World Bank projects (see GN1), and this is based on the principles and methods of logical framework analysis. The logical framework matrix LFA is used to assist project planning and management, and to develop the project monitoring and evaluation framework. It enables the design of the project to be considered in a systematic and structured way. It is an analytical process based on problem and stakeholder analysis, the setting of objectives and the identification of project content and scope. The results of the analysis are presented in the form of a logical framework matrix (Table 1). (See also the explanation of the concepts of project logic in Part A, Section 3.4). This matrix (commonly termed the logframe) consists of up to four columns and six rows. It summarises the key components of a project's design: · the hierarchy of project objectives or project logic; · necessary external conditions for successful implementation of the project (assumptions that are made and risks that they will not be in place); · how the implementation and results of the project will be monitored and evaluated (indicators and sources of verification at each level). Table 1 uses the terminology of the World Bank Results Framework (GN1), which has also been used throughout Part A of this toolkit. Readers should note that the original formulation of the logical framework matrix, and versions currently used by some development agencies, differ in terminology. This particular applies to the higher levels of objectives and to the column headings. Similarly some versions of LFA may use only the far left and right hand columns, include only two rows for the higher level objectives, or leave out the activity row. Despite these variations the underlying concepts and approach remain the same. Above outputs the number of levels of objectives is not critical, but it is the logical sequence that must be correct. Objectives at each level must be clearly capable of contributing to the achievement of the objectives at the next highest level. Strengths and weaknesses of logical framework analysis The logical framework approach is simply a tool to help structure the planning process. Its use is complementary to that of other planning tools such as financial and economic analyses, gender analysis, and social and environmental impact assessments. Its strengths are that project objectives are clearly defined and planners are forced to identify and test the causal linkages between inputs, activities, outputs and objectives. Planning is also improved by the identification of the important external conditions upon which project success depends. It provides the framework for a project monitoring and evaluation system which includes GN2 Logical Framework Analysis 1 Toolkit for monitoring and evaluation of AWM Guidance Note 2 projects Logical Framework Analysis all stages of a project from its beginning to completion and beyond. Lastly it is an effective tool for communication and can facilitate a common understanding between decision makers, managers, beneficiaries and other project stakeholders. Table 1: Structure of the logical framework matrix Project logic Indicators Sources of Assumptions and risks verification Higher level development How the objective(s) is Data sources that If the PDO(s) is achieved, what objective(s): the longer term to be measured; exist or that can be conditions beyond the project's objective(s), change of state or specified in terms of provided cost direct control need to be in place to improved situation to which quality, quantity and effectively through ensure the expected contribution to achievement of the project timeframe. the completion of the higher level development development objective(s) is surveys or other objectives? intended to contribute. forms of data collection. Project development How the PDO(s) is to be Details of data If the project component outcomes objective(s) (PDO): the measured in terms of its sources, how the are achieved, what conditions combination of one or more quality, quantity and data will be beyond the project's direct control project component outcomes timeframe. collected, by whom need to be in place to achieve the which make up the physical, and when. PDO(s)? financial, institutional, social, environmental or other development changes which the project is designed and expected to achieve. Project component Specification of how Details of data If the outputs are produced, what outcomes: the effects of each project component sources, how the conditions beyond the project's project components in terms of outcome is to be data will be direct control need to be in place to observable change in measured in terms of its collected, by whom achieve the project component performance, behaviour or quality, quantity and and when. outcomes? status of resources. timeframe. Outputs: the products, capital How the outputs are to Details of data If the activities are completed what goods and services resulting be measured in terms of sources, how the conditions beyond the project's from a development intervention their quality, quantity data will be direct control need to be in place to and which are necessary for the and timeframe . collected, by whom produce the outputs? achievement of project and when. component outcomes. Activities: the actions taken by If the inputs are provided in full and project implementers that on time what conditions beyond the deliver the outputs by using the project's direct control need to be in (a summary of the (a summary of the inputs provided. place to ensure completion of the activities and resources costs and budget (this level is not specified in activities? may be included in this may be provided in some versions of LFA) cell) this cell) Inputs: the human, and What preconditions are necessary material resources financed by for input provision and project the project. commencement? Source: Authors The main weakness is that it can promote a `blueprint' approach to development if project design becomes an inflexible and uncreative activity. This can be avoided by regular review and through engagement with all stakeholders. LFA can also neglect environmental or social impacts that do not directly influence the logical hierarchy of objectives. This confirms the importance of paying attention to cross cutting issues and indicators as noted in Part A, Section 4.3, and GN3. In using LFA it is important to ensure that participants in the process understand the process and terminology, and use the logframe as a tool to encourage stakeholder participation, dialogue and agreement on the project components and scope, rather than as a tool to impose external concepts and priorities. GN2 Logical Framework Analysis 2 Toolkit for monitoring and evaluation of AWM Guidance Note 2 projects Logical Framework Analysis Stages in logical framework analysis Box 1: Stages of project planning The stages involved in use of LFA for project planning are shown in Box 1. Analysis stage · Stakeholder analysis ­ identifying and Developing the logical framework matrix itself characterising key stakeholders and assessing involves the following steps: their capacity Problem analysis ­ identifying key problems, · identification of the target group · constraints and opportunities; determining cause · setting the objectives and effect relationships · identifying the outputs · Objective analysis ­ developing solutions from · defining the activities the identified problems; identifying means to · identifying the inputs ends relationships · assessing assumptions and risks · Strategy analysis ­ identifying different strategies to achieve solutions; selecting the most Identification of the Target Group appropriate strategy Formulation of objectives should be preceded by Planning stage the careful identification of the target group, and · Developing logical framework matrix ­ defining its needs. The intended beneficiaries may face the project structure, testing its internal logic and special constraints that require incentives, risks, formulating measurable indicators of technology, extension and organisational achievement methods fitted to their own circumstances. There · Activity scheduling ­ determining the sequence is an increasing awareness of the need to help and dependency of activities; estimating their the poor in rural areas, for example, small scale duration and assigning responsibility farmers, landless labourers, sharecroppers and · Resource scheduling ­ from the activity small tenants, artisans, fishermen and foresters. schedule, developing input schedules and a It is also recognised that women are budget. economically active members of these groups, Source: European Commission, 2004. and there can be a case for projects designed specifically to meet the needs of women. The nature of the target beneficiary group will influence the physical requirements for the project, the choice of technology and the organisational and institutional arrangements that will be most appropriate. Failure to take sufficient account of this is a reason why many projects under achieve. In formulating project objectives the designer must have in mind the following questions. · Who is the project aimed at? · Who is the project trying to reach, influence, persuade, train or help? · Whose behaviour is the project seeking to change? Precise identification of an ultimate beneficiary group requires more than a broad descriptive term such as the "rural poor", the "disadvantaged" or "small farmers". The target group should be identified by reference to the following characteristics, specified for a particular geographical area such as a region or district: 1) Status, for example, in terms of wealth, level of education, land tenure and livestock ownership. 2) Occupation, for example, small landowner, landless labourers, farmers engaged in special crop or livestock activities, artisanal fishers. 3) Access to services, for example, farmers with no access to institutional credit, input supply or extension advice. 4) Gender; 5) Class or caste; 6) Ethnic status. GN2 Logical Framework Analysis 3 Toolkit for monitoring and evaluation of AWM Guidance Note 2 projects Logical Framework Analysis For example: Male and female headed, landless households, who supply labour to irrigated farms in the rice growing zones of Sindh province; not simply, the landless poor in Pakistan. Setting objectives and outcomes An objective describes the desired state which the project is expected to achieve or contribute to. It provides the reason for undertaking the project. A project will usually have three levels of objectives: a higher level development objective, a project development objective (PDO), and intermediate objectives in the form of project component outcomes. The delineation between these can vary from one project to another, i.e. the PDO of one project can be an outcome of another project. The higher level development objective will usually be at the sectoral or sub sectoral level and should provide a clear goal towards which the project is striving. The project will be expected to make a significant contribution to this objective, though not normally to achieve it entirely. The PDO sets out the specific objective that the management of the project will be expected to achieve. In defining a development objective, it is necessary to make sure that: · it provides adequate justification for the project; · its progress can be verified either quantitatively or qualitatively; · it is single purpose, or has multiple purposes which are compatible. The tendency to overstate the development objective in terms of sectoral or national goals (e.g. eradicate rural poverty), or to be vague (e.g. protect the environment), should be avoided. The PDO should specify the changes that can be expected in the target group, organisation or location if the project is completed successfully and on time. It should also state the magnitude of such changes and the time span in which they will be brought about. It is the formulation of the PDO that is of most importance to the project designer and subsequently to the management team. The way in which the PDO contributes to the development objective should be obvious. In defining the PDO, it is necessary to make sure that: · it states clearly the desired change and where this will take place; · it specifies the magnitude of the change to be achieved; · it indicates the timescale for the change; · its progress can be verified quantitatively; · if it conflicts with another objective, priorities are indicated. A PDO should thus be a very specific statement, the achievement of which can be verified. Note that specification of the magnitude of change and the timescale establishes a target for achievement. Expressed as the `Project Development Objective' this statement plays the pivotal role in the World Bank Results Framework (GN1). A target is a quantitative statement of results and a planned performance standard by which actual performance may be subsequently compared and measured (see Part A, Section 4.2.6). The specification of targets in project formulation is useful because it forces the project designer to think in terms of physical quantities, time spans and costs. It enables an assessment to be made of how possible the project is and how realistic the proposals of project achievements are in terms of the resources requested. It also assists in the identification of indicators used to measure progress towards targets in the monitoring and evaluation of a project. The principles outlined above similarly apply to the definition of specific project component outcomes. GN2 Logical Framework Analysis 4 Toolkit for monitoring and evaluation of AWM Guidance Note 2 projects Logical Framework Analysis Identifying project outputs Outputs are the result of activities completed by the project with the use of inputs. They are a pre condition for the subsequent achievement of objectives. Production of outputs should be managed by the project and is less influenced by external conditions which the project management cannot directly control. As most projects have more than one output, their sequential ordering is useful, because the output of one activity is likely to be required for the production of another output. The outputs of a project need to be stated in such a way that: · their realisation can be identified, in terms of quantity, quality, time and place; · as for objectives, a target is specified for the magnitude of output to be produced and the timescale for this; · it is clear if a certain output is a prerequisite for other outputs; · all outputs necessary for achieving the project component outcomes and PDO are listed and all outputs clearly relate to the purpose; · they are feasible within the resources available. By definition outputs are separate from objectives. Unfortunately their confusion is a common design error. Outputs are also commonly confused with activities. Remember that an output is the result of an action or activity. For example, the output of a training activity is trained people; the output of a research activity is the research results; research conducted on inter cropping is an activity; research findings on inter cropping are an output. Defining activities An activity is the action necessary to transform given inputs into planned outputs over a specified period of time. Each activity should have at least one output, which may contribute to a larger output. Activities need to be stated in such a way that: · their implementation can be verified in terms of quantity, time and place; · they are stated in terms of actions being undertaken rather than as completed outputs; · all key activities necessary for achieving the outputs are listed; · there are no activities listed whose outcome cannot be traced upwards to the output level. · it is clear who is responsible for carrying out the activity. Identifying inputs Inputs are the goods, services, personnel and other resources provided for a project for the purpose of undertaking specific activities, producing outputs (results), and achieving objectives. Without inputs, activities cannot take place. A clear and early listing of inputs will help to mitigate some of the commonest problems encountered in implementation, namely the failure to recruit appropriate staff, shortages of funds and deficiencies of equipment or materials. The listing of inputs should specify: · type (of experts, equipment, vehicles, fellowships, training etc.); · quantity, i.e. number or amount; · durations of assignment or use; · cost; · timings for delivery or commencement; · purpose for which provided. Note that money or finance is not an input inputs are the goods and services that money buys. GN2 Logical Framework Analysis 5 Toolkit for monitoring and evaluation of AWM Guidance Note 2 projects Logical Framework Analysis Assessing external conditions: assumptions and risks The logical chain of cause and effect that underlies the left hand column of the logical framework matrix may depend on external events or conditions that must occur or be in place (Part A, Section 3.4). There is a basic premise here that the achievements in the left hand column and the conditions in the right hand column are both necessary and sufficient to cause the next higher level to be attained. Each causal linkage (between inputs, activities, outputs, and objectives) must be planned so as to ensure that at any given level, the necessary conditions not only exist, but are enough to achieve the next level. Usually the linkage is not strong enough (i.e. the sufficiency or completeness does not occur) unless the external environment also influences the project in the desired direction, and risks of the right conditions not being in place are avoided. Development actions in the agricultural sector have a multi disciplinary character with inter actions among many key variables, and linkages with other agricultural projects and with activities in other sectors of the economy. This is particularly true of agricultural water management projects. As it is rarely feasible to tackle all the key variables in a single project, no project can be treated in isolation. For example, Trained irrigation managers may not be able to produce an impact on poverty at farm level unless other essential inputs and services (seeds, fertilizers, pesticides, credit, storage, processing, markets, etc.) are available to farmers to improve farming practices and thus their yields. It is neither possible, nor essential, that all these inputs and services be specifically included in an irrigation management capacity building project, but project success (impact) depends on their being available simultaneously. Some projects fail because the designer did not adequately assess the project's environment and underestimated the risks of a particular development approach. Judgments made about the linkages underlying the logic of a project and the necessary external conditions at each level are subject to a variety of risks. These include the inaccuracy of information, the uncertainty of the project environment, and the unpredictable reactions of target beneficiaries and other stakeholders. The uncertainty involved typically increases the higher the level in the hierarchy. A project designer should make every effort to obtain the information necessary to design a project and be confident of its ability to be implemented and its success. Inevitably, however, the collection of information is constrained by the time and resources available, and in the absence of definitive data the project designer will have to make certain assumptions. Each time an assumption is made a risk that the assumption will not hold is introduced, and hence a risk that the performance of the project will be compromised. In the formulation of a project there will be four main sets of risks and it is important to distinguish these. 1) Inherent risks: it is assumed that inputs and activities will lead to outputs, and that outputs will lead to achievement of outcomes which will contribute to the objectives. These assumptions constitute the project hypothesis or project logic and there should be no need to state them explicitly as the causal relationships between means and ends should be apparent. If there is any risk or uncertainty about any of the links in the chain then further investigation is required to verify the technical relationship involved, probably through consultation with relevant experts. Certain risks will remain, for example: the risk that it is not possible to establish viable Water User Associations within a specified time span given available inputs. This type of risk will always remain in a project design to some extent. We can never be certain that measures taken will always produce the desired results, but with relevant expertise GN2 Logical Framework Analysis 6 Toolkit for monitoring and evaluation of AWM Guidance Note 2 projects Logical Framework Analysis informed judgements can be made. Risks of this type are not normally specified in the right hand column of the logical framework matrix. 2) Universal risks: for example, the risk of war, rare extreme climatic events or other natural disasters. It is unlikely that such risks (or assumptions of non occurrence) need be entered in the matrix. This does not mean that such possibilities should be ignored. The reverse is true as they must be taken account of in project design and may mean the rejection of the project, or at least its delay until the situation improves. 3) Internal risks: there are also risks relating to the inputs and activities under the control of management. Examples include: · delays in the supply of equipment; · delays in the assignment of project staff; · project staff are not suitably qualified, or motivated; · delays in the completion of an activity, for example, completion of survey work because of use of key staff for other tasks. In these cases management is clearly responsible for ensuring that materials are ordered with reasonable allowance for shipment delays, that adequately qualified project staff are recruited on time, that the staff are well motivated and that competing tasks are properly scheduled. Management should be able to exert some influence so as to reduce these risks. Provision should be made in the project design to ensure that management can perform its functions satisfactorily without such risks, and they are thus not normally specified in the matrix. Once work starts then `implementation monitoring' (Part A, Section 3.5) is an essential part of the good management needed to reduce such risks. 4) External risks: conditions upon which the success of the project depends, but which are largely outside the control of project management. This is the important group of risks for which assumptions shown in the logical framework matrix may be necessary. For example, most agricultural water management projects aimed at increasing farm incomes may depend on assumptions about the construction of roads, provision for market access, credit facilities, farm inputs, etc. The project designer could verify that roads and markets do or do not exist, that credit and inputs are available, etc, and no risk would then be involved. These factors become risks when agencies external to project management are responsible for them and it must be assumed that they will satisfactorily fulfil that responsibility, but there is some uncertainty. For example: · when legislation has been promised to decentralise water management responsibilities to local governmental bodies or Water User Associations; · or claims have been made that following market liberalisation and infrastructure investment the private sector will provide marketing services in the project area; · or that a viable rural credit institution has been established. In each case there is a risk of non fulfilment and hence need for an assumption and specification of the conditions required in the right hand column of the logical framework matrix. Introduction of such risk identification in the earliest stages of project identification and formulation will rapidly screen out infeasible project concepts. For example: A project that requires a rural credit institution that the government has no intention of setting up can be rejected out of hand, unless resources are available to make the credit institution an activity of the project itself. GN2 Logical Framework Analysis 7 Toolkit for monitoring and evaluation of AWM Guidance Note 2 projects Logical Framework Analysis Project design should be an iterative process that seeks to determine the optimal form for a particular project given resource constraints. In particular, the identification of risks may force the designer to rethink all or part of an original design or concept in order to remove the risk or reduce it to an acceptable level. Thus the right hand column of the logical framework matrix is of value for two reasons: · to prompt iterative improvement of the project design; · to ensure that any remaining risks are carefully monitored. Thus having identified a risk during preparation of the project design and matrix, the response will be to: 1) reject the project, because the risk is sufficient to call into question the viability of the project and cannot be realistically removed or reduced through re design; 2) or lower or eliminate the risk through re design or by adopting a completely different approach; 3) or for risks that are acceptable at the time of project preparation, to state them as an assumption in the matrix and project documents and to monitor them through the project's life. (For World Bank projects such risks would be highlighted in the relevant section of the PAD). It is important to note that in practice final logical frameworks and project documents should specify few risks or necessary conditions in the right hand column. These may have been listed during the formulation process but most, if not all, should have been eliminated in the final design. A project requiring a large number of necessary external conditions and thus subject to highly probable and significant risks is doomed from inception and should require no further consideration. Note that compared to logical framework analysis a World Bank results framework (GN1) does not explicitly capture the external conditions that may be necessary for project success. Identification and assessment of any critical assumptions and the risk of their non fulfilment is however required in the main text of the PAD. Further Reading European Commission. 2004. Project Cycle Management Guidelines, Vol.1. European Commission, Brussels, March. Available at http://ec.europa.eu/comm/europeaid/qsm/documents/pcm_manual_2004_en.pdf Dearden, P.N. and B. Kowalski, 2003. Programme and Project Cycle Management (PPCM): lessons from South and North, Development in Practice, 13, 5, pp501 514. Wiggins, S. and D. Shields. 1995. Clarifying the `Logical Framework' as a Tool for Planning and Managing Development Projects, Project Appraisal, 10, 1, pp2 12. GN2 Logical Framework Analysis 8 Toolkit for monitoring and evaluation of AWM Guidance Note 3 projects Choosing and Specifying Indicators Guidance Note 3 Choosing and Specifying Indicators What are indicators? Indicators are measures of inputs, processes, outputs, outcomes and impacts for development projects or programmes. Used in monitoring, indicators enable managers to track progress and achievement of stated targets, to demonstrate results and to take corrective action to improve outcomes. In evaluation, indicators are used to assess achievement of outcomes and impacts. Indicators provide the quantitative and qualitative information for these tasks. Choice and specification of indicators is a core activity in building a results based M&E system as it drives all subsequent data collection, analysis and reporting. Progress needs to be monitored and evaluated at all levels in a project, and as explained in Part A, Section 4.0, this requires a structured set of indicators, providing information at each level of the logical hierarchy of objectives (see Boxes 3 and 4 in Part A, Section 3.0). Table 1 provides a categorisation of indicators based on the structure of the results based approach to project design and management. This structured set of indicators needs to be identified during project planning and appraisal and will form the basis of the design of the project M&E system. As with all aspects of the M&E system it must be kept under continual review and updated as the project evolves during implementation. Finally, indicators have to be tailored to the project considered and GN10, GN14, GN15, GN16, GN17 list some possible indicators for typical project components. Table 1: Structured indicators for project monitoring and evaluation Impact indicators: Leading (early measures of medium or long term physical, outcome) financial, institutional, social, environmental indicators: or other developmental change that the advance Cross-cutting project is expected to contribute to. measures of indicators: Outcome indicators: whether an Exogenous measures of cross measures of short term change in expected change or external cutting concerns at all performance, behaviour or status of will occur for indicators: levels. resources for target beneficiaries and other outcomes and measures of affected groups. impacts. necessary For example: Output indicators: external gender disaggregated measures of the goods and services conditions differences; produced and delivered by the project. that support regulatory Process indicators: achievement compliance; measures of the progress and completion of at each level. legislative provision; project activities within planned work capacity building. schedules. Input indicators: measures of the resources used by the project. Source: Authors GN3 Choosing and specifying indicators 1 Toolkit for monitoring and evaluation of AWM Guidance Note 3 projects Choosing and Specifying Indicators Choosing and specifying impact and outcome indicators Impact and outcome indicators are indicators for `results' monitoring and evaluation (Part A, Section 3.5), and their selection requires careful iteration and refinement. This should be done in collaboration and consensus building with the Borrower and other key stakeholders. Participation of all key stakeholders in defining indicators is important because they are then more likely to understand and use the information provided by the indicators for management decision making (see GN11 on participatory M&E). Discussion and agreement of indicators is a process that should start as early as possible in the project cycle and be initially completed during project appraisal. The World Bank Project Appraisal Document requires the specification of indicators for the PDO and the outcomes of each project component, as explained and illustrated in GN1. Increasingly, sectoral and regional guidelines are considered that would make the inclusion of a few standardized, key, outcome and impact indicators compulsory for agricultural projects. This is the case in the Africa Region for example with land and labour productivity indicators under consideration. These key indicators would demonstrate the achievement of broad objectives and allow for inter projects comparison. They should be included in the results framework alongside a set of project specific indicators, both for management and assessment of success. World Bank staff should check regional and sectoral guidelines and procedures in place in this regard. Four main steps are recommended for the formulation of indicators; corresponding to stages 3 and 4 of the M&E design process set out in Part A, Section 4.2. Four steps for formulating impact and outcome indicators 1) Clarify project objectives: the PDO and project component outcomes. 2) Develop a list of possible indicators. 3) Assess each possible indicator. 4) Select the indicators for project monitoring and evaluation. 1) Clarify project objectives: the PDO and project component outcomes. Clear and precise statements of the Project Development Objective(s) and project component outcomes greatly aid the specification of indicators. As discussed in GN1 the objectives and outcomes should be: · clear about where change will occur and for whom; · clear about the problem to be addressed; · clear about the scale and nature of the change to be experienced by the groups affected; · clear about how component outcomes contribute to achievement of the PDO and desired project impacts. The last point is important for the logic and coherence of the project strategy, but also because indicators of achievement at one level may be important to the measurement and evaluation of the achievement at the next higher level. It should also be noted that the project component outcomes of one project can be the PDO of another. Projects can fit together in `nested' programmes of activity, and specification of indicators should match this. 2) Develop a list of possible indicators. Identification of good outcome and impact indicators can be difficult and requires some analysis. The analysis must consider the information that will be needed to be able to make a reliable judgment on whether or not the planned outcome or impact has been achieved, and whether or not the project has been the main cause of this change. Thus indicators are needed that can measure change over time and be used within the context of the chosen framework for evaluation (Part A, Section 4.2.4, and GN9) for the attribution of the observed change. In other words what information is necessary to know whether the project is performing as planned? What questions GN3 Choosing and specifying indicators 2 Toolkit for monitoring and evaluation of AWM Guidance Note 3 projects Choosing and Specifying Indicators need to be answered? Indicators must be chosen that can provide this information in the form that is needed by decision makers. As a result it may be common that one outcome or impact requires more than one indicator to provide all of the information needed for its assessment. Indicators can also take a number of different forms as categorised in Table 2. The analytical process needed can best be illustrated by the examples in Box 1. These illustrate that it is difficult to be prescriptive about the process and analysis that is required to choose outcome and impact indicators. Experience and subject matter expertise relevant to the project are important. In practice a trade off may be necessary between the indicator that would give all information needed and the `practicalities' of data collection and analysis (GN6). Indicators in the real world may be less than perfect, but with careful definition they are nonetheless useful. The process of setting up the M&E system is iterative and not a simple progression, and definition of indicators can not be separated from consideration of data collection, analysis, and use (see GN6). Ultimately the choice of indicators must be well adapted to the characteristics of the project, data availability and accessibility, and the evaluation framework that is to be used. Despite these challenges a number of options can assist the process of defining indicators: · brainstorming with project preparation team members and/or the borrower's personnel; · consulting with sector teams, sector lead specialists and other specialists; · consulting similar project documents; · consulting guidelines such as these; · consulting pre designed indicators (Box 2). GN3 Choosing and specifying indicators 3 Toolkit for monitoring and evaluation of AWM Guidance Note 3 projects Choosing and Specifying Indicators Box 1: The process of getting from statements of objectives and outcomes to indicators. Example 1: Objective/impact: Sustained improvement in the farm income of 5000 farming households in the command area. Analysis: How should farm income be measured (gross or net, per hectare or per household, etc.) and is it changing? What is the source of improved farm income? Increased cropping intensity, change in cropping pattern or livestock enterprises, increased yields, lower input costs or a combination? Are any indicators for these being used for project outputs or project component outcomes? Will this data be available? Is improvement in income occurring over more than one season or year? Are all 5000 target households benefiting? Information needs: Evidence of sustained improvement in farm income compared to a baseline and plausible counterfactual for the target group of households. A measure of the proportion of the target group that are benefiting. Indicators: Average net farm income from crops and livestock per hectare for at least three seasons or years. Proportion of target farm households achieving a sustained improvement in net farm income. Farmers' assessments of why farm incomes have changed. Example 2: Outcome: Improved irrigation service delivery. Analysis: What kinds of improvement are important? What criteria will be used to judge improvement? What information will be required to determine whether an improvement has taken place? What basis for comparison can be used to show that an improvement has occurred? Information needs: Evidence of improvements in the adequacy, equity and reliability of water supply compared to the baseline situation prior to the project. Indicators: Adequacy Relative Water Supply (RWS) Volume of irrigation water supplied As measured at main Volume of irrigation water demand canal intake and each Delivery Performance Ratio Volume of irrigation water supplied tertiary unit intake. (DPR) Target volume of supply Spatial variation of the RWS at tertiary intakes. Equity Spatial variation of the DPR at tertiary intakes. Temporal variation of the RWS at the main canal intake and at tertiary intakes. Reliability Temporal variation of the DPR at the main canal intake and at tertiary intakes. Farmers' and experts' assessments of the cause of measured change in Attribution these indicators. Source: Authors GN3 Choosing and specifying indicators 4 Toolkit for monitoring and evaluation of AWM Guidance Note 3 projects Choosing and Specifying Indicators Table 2: Examples of different forms of indicators Forms of indicator Examples Explanation Simple quantitative · kilometres of canals de silted Direct measurement of the appropriate indicators · person days of training to single quantity or ratio. establish X capability · average yield of crop X per ha Complex quantitative · number of months for which Requires specification of more than one indicators vulnerable households experience data element. Here: months, type of food food shortages shortage, type of household. Compound indicators · number of effectively functioning Requires a standard to be defined, and an WUAs in the project area individual assessment made for each unit · number of scheme modernization of concern. E.g. `effective' functioning for or rehabilitation plans completed each WUA. Similarly, `meeting funding that meet funding criteria criteria' for each modernization or rehabilitation plan. Indices · index of irrigation system Indices combine a number of different performance indicators to enable comparison. Requires · human development index consistency in the selection, measurement and weighting of variables used for the index. Proxy indicators · standard of house construction Needed when it is difficult or too costly to · ownership of consumer durables measure an outcome or impact indicator directly. Provides an indirect or approximate measure based on an assumed relationship. Examples here are assumed to be a proxy for household income. Open ended · stakeholder perceptions of project Open ended enquiry can establish qualitative indicators outcomes stakeholder priorities and reveal · stakeholder perceptions of unexpected changes and outcomes. reasons for change Data collected can be difficult to process and analyse. Focused qualitative · farmer explanations of higher Can collect specific information. Possible indicators yields responses can often be identified by pilot · farmer identification of problems survey and pre coded to aid data with gate operation collection and analysis. Leading indicators · farmer participation rates in Direct or proxy indicators that are voluntary physical works sensitive to change in the early stages of · farmer participation rates in WUA a project. meetings · trader estimates of marketed volumes Source: Adapted from Guijt and Woodhill, 2002. GN3 Choosing and specifying indicators 5 Toolkit for monitoring and evaluation of AWM Guidance Note 3 projects Choosing and Specifying Indicators Box 2: The advantages and disadvantages of using pre-designed indicators Pre designed indicators are those prepared independently of an individual country, organisation, programme or sector context. A number of development institutions have created indicators to track development goals and to develop a common approach to monitoring and evaluating progress both within and across countries. They are increasingly under consideration in the World Bank, and staff should check relevant regional and sectoral procedures in this regard. Advantages: · can facilitate aggregation across similar projects, programmes and policies, · can reduce the costs of building multiple unique monitoring systems, · can facilitate a greater harmonisation of donor requirements. Disadvantages: · may not address project and country specific objectives, · may be viewed as imposed and part of "top down" management, · may not promote key stakeholder participation and ownership, · may lead to the adoption of multiple competing indicators (Kusek and Rist, 2004). For interventions at programme, policy and sector level there may be a strong case for use of pre designed indicators. At project level it may be best to develop or adapt indicators to meet specific management information needs while involving key stakeholders in a participatory process. Source: Authors As part of the development of indicators it may be necessary to identify and make use of leading indicators. As indicated in Table 1 `leading indicators' (sometimes also called `early outcome indicators'), are relevant at the levels of impacts and outcomes. Leading indicators are used to provide an early indication of whether an expected change will occur during the early and mid stages of the project (Part A, Section 3.8). As well as providing information on early outcomes, they can be used where rigorous evaluation of project impacts is likely to be a difficult and/or long term process (Part A, Sections 3.7 and 3.8). The information that leading indicators provide may, however, ultimately be supplemented or replaced by that generated by the final impact and outcome indicators that are chosen as discussed above. The analysis that leads to the choice of leading indicators can focus on the following four questions: · What inputs and services is the project providing to the target group? · Who has access to these inputs and services? · How are they affected and what are their opinions of the inputs and services? · What evidence is there of changes in behaviour and performance? Project managers in particular need to guide the choice of leading indicators by deciding what they need to know during the early stages of the project to improve its performance. The indicators should focus on physical and behavioural changes that can be readily measured. Box 3 suggests some basic leading indicators that may often be useful for agricultural water management projects (see also GN10, GN14, GN15, GN16, and GN17). GN3 Choosing and specifying indicators 6 Toolkit for monitoring and evaluation of AWM Guidance Note 3 projects Choosing and Specifying Indicators Box 3: Some suggested leading indicators for AWM projects · The proportion of the target population that know of the changes (physical works, organisational changes, institutional reforms or services) being implemented by the project? · The proportion of the target population actively participating as expected in the changes being implemented by the project. · The proportion of beneficiaries by location or tertiary unit that were consulted before work started. · Beneficiary perceptions of problems arising from project works and other activities. · Beneficiary perceptions of improvements arising from project works and other activities. · Reasons given by beneficiaries for non participation or for rejection of project services. · Indicators of improved adequacy, equity or reliability of water access and/or delivery. · Early indicators of change in crop or livestock production, e.g.: o change in cropping patterns and/or crop rotations o marketed volume o forecasts of production by farmers o farmer estimates of yields achieved o trader estimates of supply o trends in farm gate and local wholesale prices Source: Authors 3) Assess each possible indicator Each indicator initially selected for inclusion in the M&E programme needs to be carefully scrutinised and tested before acceptance. Table 3 presents criteria against which indicators can be tested to ensure that they are suitable for inclusion. Table 3: Criteria for selection of indicators Criteria Description Relevant Indicators must be representative of the most important aspects of implementation and of the outcomes and impacts intended. Clear Indicators must be unambiguous and clearly defined in the project's context, and in a manner understood and agreed by all stakeholders. Any adjectives used to describe the qualities of an indicator need to be precisely defined. For example: · what is meant by "improved water service delivery?" · an indicator may be "the area of land affected by waterlogging and salinity" but what criteria will be used to classify such land? · for households what is included in "farm income" and what in "non farm income?" Specific Indicators should measure specific changes, and be specific to a timeframe, location and target or other stakeholder group. Measurable There must be practical ways to measure the indicator, either in quantitative or qualitative terms, that are within the capability of the monitoring organisation. It must be possible to collect, process and analyse data in time and within budget. Consistent The values of the indicators should be reliable and comparable over time when collected using the same methods. This is more likely when indicators are measured in a standardised way and with sound sampling procedures. Sensitive Indicators should be sensitive to the expected changes. It is especially important that leading indicators are capable of revealing short term movements. Indicators that require a long time series of values are practically useless for implementation decisions. Attributable Based on an established or probable relationship expected to cause the intended change. In moving from inputs and outputs to outcomes and impacts, attribution must typically rely less on direct observation of cause and effect and more on statistical evidence of change and its probable cause. Source: Adapted from Guijt and Woodhill, 2002, and Kusek and Rist, 2004 4) Select the indicators for monitoring and evaluation The ideal number of indicators for any one outcome or objective is the minimum that answers the questions: "has the objective been satisfactorily achieved and can this achievement be attributed GN3 Choosing and specifying indicators 7 Toolkit for monitoring and evaluation of AWM Guidance Note 3 projects Choosing and Specifying Indicators to the project?" The initial list of indicators generated needs to be narrowed down to a usable and feasible set. It is important to keep data collection within a manageable scope, and hence to reduce the number of indicators, to the minimum necessary to meet key management and reporting needs. Monitoring too many indicators can be self defeating if managers find the time and resources required are excessive and detract from other tasks. Finally, although attempts should be made to limit the number of indicators, projects will vary greatly in their requirements. For the project as a whole, the set of indicators as listed in Table 1 should be the minimum to enable a reliable assessment of the five core evaluation criteria (Part A, Section 3.5): the efficacy and efficiency of project implementation, and its relevance, impact and sustainability. There should be clarity and agreement in the M&E system on the rationale for each indicator amongst the management team and other stakeholders. Frequent changes in indicators should be avoided so as to maintain the continuity and consistency of data collection, but the selection made does need to be kept under review and may need to be updated as a project evolves. If the information being provided by an indicator is not being used then it should be dropped or changed (with details of the change being specified in the Implementation Status and Results report). The exception to this is when a time series of data is being compiled for use in a final impact evaluation. Similarly, if the ability to take management decisions is weakened by gaps in information then additional indicators should be identified to fill the gaps. Use of both quantitative and qualitative indicators Many AWM projects may include components that require qualitative indicators of change and achievement. This may particularly apply to the building of local capacity in the form of community based organisations such as Water User Associations (WUA). Qualitative indicators may typically measure the perceptions of stakeholders, or provide descriptions of their behaviour and their capacity to carry out tasks and management functions. Qualitative information is often needed to explore `why' observed changes are happening. For example, the perceptions of farmers can provide an explanation which can be verified and quantified by a follow up diagnostic survey or study. The `rules' for qualitative indicators are the same as for quantitative indicators; they must be relevant, clear, specific, measurable, consistent, sensitive and attributable. Measurability still applies but refers to the ability to collect data about the indicator rather than to quantify it, for example, collection of expert and stakeholder assessments of the capability of a WUA to carry out a specific irrigation scheme management function. However, achieving consistency over time and across locations in how qualitative data is understood and collected can be more difficult, and will require good training of M&E staff and good documentation of procedures. Monitoring and evaluation of project outcomes and impact can be challenging as discussed in Part A, Section 3.7 and GN9, and for effective and reliable use of quantitative and qualitative indicators, both require experience and relative high level research skills. (See GN6 for more on use of qualitative and quantitative data). Choosing and specifying cross-cutting indicators Cross cutting indicators may be used for components of the project which are broader in focus than the specific logical chain of cause and effect that underlies the project design. Examples include capacity building and training which goes beyond the immediate needs of project implementation, or the reform and updating of existing legislation, again with a wider remit than immediate project needs. Evaluation may wish to consider the wider positive externalities that arise from successful project implementation and may require additional indicators for this. Alternatively there may be cross cutting issues which the project may influence at all levels of the project hierarchy. Examples of this are protection of the environment, gender issues, and other forms of discrimination. Evaluation of a project's performance in terms of gender equality will GN3 Choosing and specifying indicators 8 Toolkit for monitoring and evaluation of AWM Guidance Note 3 projects Choosing and Specifying Indicators require a M&E system that tracks gender disaggregated differences at all levels. Without this it may be difficult to establish the validity of gender sensitive outcomes such as `increased empowerment for women in the use of water". For example, indicators that measure the formation and effectiveness of WUAs may need to be supplemented by indicators that assess the participation of women and their influence on decision making. Similarly, there may be a need to disaggregate indicators by class, caste or ethnicity to determine the extent to which vulnerable groups are participating in a project and benefiting from it. A similar approach of identifying indicators at each level of the project hierarchy might be needed to evaluate the environmental "footprint" of a project, although clearly data collection will need to be limited to a small number of key indicators. Cross cutting indicators may provide the link between project level M&E and objectives specified for sector or national level programmes, again emphasising the need to seek to match data collection efforts at project, sector and national levels (see GN5). In general for AWM projects, use of cross cutting indicators should be kept to a minimum. Their use will mainly be important for particular projects where evaluation of the relevance and sustainability, and in the effect the quality of implementation, is important in addition to effectiveness and efficiency as described above in other words how well was the project implemented, and was it in accord with national goals and aspirations in this regard. In projects where a specific component focuses on these aspects, such indicators are generally integrated in the results framework, both for monitoring early outcomes and project development objectives. Choosing and specifying exogenous or external indicators These indicators provide information on factors beyond the control of project management but which might affect its outcomes. Use of logical framework analysis for project design will guide the identification of these indicators by identifying the key external conditions necessary for the expected achievements at each level of the logical hierarchy (see GN2). Monitoring such indicators will help managers identify the real progress made by a project independent of factors such as unusually favourable weather conditions or fluctuating market prices. Alternatively it may reveal that an essential condition for the success of the project is not in place, perhaps relating to existing or planned government provision or policy reform. When it comes to impact evaluation, a project will eventually be judged by its performance in the context of external circumstances. Choosing and specifying input, process, and output indicators Identification of input, process and output indicators (Table 1) should be relatively straightforward, following directly from the specification of these project elements in a full logical framework analysis or the main text of the project appraisal document. These indicators will be tracked through the project management information system (see GN6). For internal management decision making, the indicators for monitoring progress must be sensitive to change over short periods, and should focus on those aspects that can be influenced by action within managers' operational discretion. The indicators should record the physical and financial data for the direct physical and financial quantities for each activity. Ratios such as percentages may also be useful, for example, allocated funds actually spent compared with budgeted amounts, or for completion of physical works. Operational relevance requires that the data recorded be compared with benchmarks and targets. Deviations from planned physical and financial targets need to be traceable to specific line managers or cost centres if the source of the problem is to be identified and addressed. Priority should thus be given to data on the performance of tasks for which precise targets can be set. Targets for physical performance will be specified in the work program. Financial targeting requires a focus on controllable costs. Data on physical performance also needs to be related to GN3 Choosing and specifying indicators 9 Toolkit for monitoring and evaluation of AWM Guidance Note 3 projects Choosing and Specifying Indicators budgeted costs to ensure that the objectives are reached at acceptable actual costs. Emphasis should be placed on indicators relating either to activities that are critical to the achievement of further tasks or to activities which experience has shown are most prone to delays or cost overruns caused either by internal problems or external conditions. Example indicators for AWM projects Guidance on a wide range of suitable indicators for use in the M&E of World Bank AWM projects can be found in GNs 14 17. Guidance on how to set targets for some indicators for AWM can be derived also from benchmarking of project performance (see GN18). Further reading Guijt, I., and J. Woodhill. 2002. A Guide for Project M&E. Rome, International Fund for Agricultural Development (IFAD). Available at http://www.ifad.org/evaluation/guide/index.htm Kusek, J.Z., and R.C. Rist. 2004. Ten Steps to a Results Based Monitoring and Evaluation System. . Washington D.C.: World Bank. Available at http://go.worldbank.org/7H1YFQS5Z0 Rajalahti, R., J. Woelcke, et al. 2005. Monitoring and Evaluation for World Bank Agricultural Research and Extension Projects: A Good Practice Note. Agricultural and Rural Development Discussion Paper 20. Washington D.C.: World Bank. Available at http://intresources.worldbank.org/INTARD/Resources/M_E_WB_AgExtensionProjects.pdf GN3 Choosing and specifying indicators 10 Toolkit for monitoring and evaluation of AWM Guidance Note 4 projects Linking Project and M&E Design for AWM Projects Guidance Note 4 Linking Project and M&E Design for AWM Projects Introduction Flow charts are a means of presenting project logic as explained and illustrated in graphical and tabular form in Part A, Section 3.4 and developed in the logical framework and results framework approaches (European Commission, 2004; see also GN1 and GN2). Flow charts are used in this note to provide examples of the project logic, causal chain or results chain for typical agricultural water management (AWM) projects or project components. Such depictions promote understanding of the intended project interventions and are good communications tools, and devices to facilitate debate, discussion and team work. Once complete the information contained in a chart should be readily transferable to the results framework as required for World Bank project appraisal documents (see GN1), or for completion of a full project logframe (see GN2). Similarly identification of indicators (see GN3) and a basis for the design of a project M&E system will be facilitated. Flowchart diagrams illustrating causal chains (project logic) for key interventions Problem analysis The main starting point for the design of an AWM project is the identification of problems and constraints affecting water distribution and agricultural production. These problems and constraints are generally identified by the project preparation team, based on observations made during field trips, discussions with stakeholders, reading of project related documents, etc. It is important at the project preparation stage to hold meetings with the team and relevant stakeholders to discuss and summarise the problems and constraints identified, and to discuss the options for addresses these issues. These discussions lead into the structuring of the causal chain/project logic outlined below. An example of a (simplified) unstructured problem analysis is presented in Figure 1, where the main problems affecting the water delivery in an irrigation system have been identified. The identified problems can be grouped, in this case into physical, operational and organisational categories (Figure 2), and a structured problem "tree" formed. The problem tree groups and links the various constraints, identifying cause and effects, and enabling better definition of the problem. Having identified the main problems they can be addressed both individually and collectively (Figure 3), providing a holistic approach to address the underlying root causes of the main problem. The example provided here serves to show the linkages between the physical, operational and organisational aspects often involved in solving constraints in the AWM sector. GN4 Project logic 1 Toolkit for monitoring and evaluation of AWM Guidance Note 4 projects Linking Project and M&E Design for AWM Projects GN4 Project logic 2 Toolkit for monitoring and evaluation of AWM Guidance Note 4 projects Linking Project and M&E Design for AWM Projects Identifying and building project components Using the process of problem analysis and the linkages found through developing the problem tree, the details of each project component can be developed. Figures 4, 5 and 6 show the elements of project components related to system rehabilitation, improved management, operation and maintenance and formation of Water Users Associations, and reform of water institutions (see GN14, GN15, GN16, and GN17 respectively for a more detailed account of each of these components). These components come together as shown in Figure 7 to address the Project Development Objective (PDO). GN4 Project logic 3 Toolkit for monitoring and evaluation of AWM Guidance Note 4 projects Linking Project and M&E Design for AWM Projects GN4 Project logic 4 Toolkit for monitoring and evaluation of AWM Guidance Note 4 projects Linking Project and M&E Design for AWM Projects GN4 Project logic 5 Toolkit for monitoring and evaluation of AWM Guidance Note 4 projects Linking Project and M&E Design for AWM Projects Conclusion The importance of correctly identifying and structuring the problems and constraints which the project is expected to address, or work within, cannot be over emphasised. Due to the nature of AWM projects a holistic view covering technical, social, political, economic, legal and environmental must be adopted such that all the possible influencing factors are identified. The structuring of the identified problems and constraints into causal chains greatly facilitates the identification of the project components and activities, and the development of a robust and achievable results chain. Further reading European Commission. 2004. Project Cycle Management Guidelines, Vol.1. European Commission, Brussels, March. Available at http://ec.europa.eu/comm/europeaid/qsm/documents/pcm_manual_2004_en.pdf GN4 Project logic 6 Toolkit for monitoring and evaluation of AWM Guidance Note 5 projects Organizational alternatives for M&E of AWM projects Guidance Note 5 Organizational Alternatives for Monitoring and Evaluation of Agricultural Water Management Projects Introduction This guidance note discusses the organizational aspects of monitoring and evaluation for agricultural water management projects. There is no single organisational structure for M&E that will match the needs of all agricultural water management projects. It is difficult to specify universal recommendations because AWM projects vary so much in their characteristics and requirements, and because countries and their public sector management systems are at different stages of development with respect to good management practices in general, and M&E in particular. Management approaches and the information systems that inform them also need to be flexible and adaptable to changing needs and circumstances, rather than pre determined and rigid. Conducting a `readiness assessment' prior to design and implementation of a project M&E system Project Monitoring and Evaluation can not be viewed in isolation from wider sectoral and national activities and capacities, and with the progress of a results based management culture within the public sector of the client country. Projects should assess existing capacity and interest, and strive to build systems that can meet the immediate M&E needs of projects, whilst contributing to the longer term building of capacity within partner agencies in government and wider civil society. Kusek and Rist (2004) provide guidance on how to conduct a `Readiness Assessment' for results based monitoring and evaluation at national and sectoral level. Such an assessment should cover: i) incentives and demands for designing and building a results based M&E system; ii) roles and responsibilities and existing structures for assessing performance of government; and iii) capacity building requirements for a results based M&E system. World Bank task teams for AWM projects should draw on the lessons of such an assessment in cases where it has been carried out, but will need to focus more narrowly on the requirements of, and capacities available for the project at hand. Specifically they will need to make an assessment of: · current capacity within the proposed project management organisation and its partners, covering technical skills, managerial skills, existence and quality of data systems, available technology and existing budgetary provision for M&E; · any barriers to M&E for the project such as a lack of political will or relevant expertise and experience; · other organisations such as universities, private consultants or government agencies that have the capacity to provide data collection and analysis services, technical assistance and training; · the linkages that can be made with existing systems for data collection and publication of statistics at a national, regional or sectoral level; and · how the approach to M&E developed for the project can complement and match with existing systems of performance monitoring and assessment within the wider public sector. GN5 Organizational alternatives 1 Toolkit for monitoring and evaluation of AWM Guidance Note 5 projects Organizational alternatives for M&E of AWM projects Organizational arrangements for project M&E The Internal Evaluation Group (IEG) review of World Bank AWM projects (World Bank IEG, 2006) found that responsibility for M&E functions was most frequently shared between the Project Implementation or Management Unit (PIU/PMU) and the involved government line agency. In about 10 percent of projects, community organizations, local government or NGOs were also charged with monitoring responsibilities. It was found that even when there was good M&E design, inadequate supervision sometimes led to ineffective implementation. A relatively common approach for project management is thus to establish a PIU/PMU to manage project implementation. There are then a limited number of alternatives for how the M&E of AWM projects can be organised. There are four basic options: i) create an in house M&E unit within the Project Implementation Unit (PIU/PMU); ii) rely on a government line agency with M&E responsibilities; iii) employ consultants or other independent specialists; and iv) share M&E tasks among the implementing partner organisations and primary stakeholders, including where appropriate community organisations, local government, NGOs and other civil society organisations. In practice, and particularly for large and multiple component projects, a combination of these may be involved. As emphasised in Part A, Section 4.3, ideally the unit with M&E responsibility should be located within or close to project management. This particularly applies to `implementation monitoring' (Part A, Sections 3.5), for which M&E should be an integral part of all project and programme managers' duties. Where independent external expertise is needed, as may be the case for `results monitoring' and impact evaluation (Part A, Section 3.5), it should be procured through partnership in the case of its supply by national or sectoral agencies, or it should be contracted on a consultancy basis supervised by project management (RN1 and RN2 provide examples of Terms of Reference for procurement of such expertise). The use of a separate agency or third party for results monitoring and evaluation can also be important for reasons of independence and credibility. For example in China, Provincial Finance Bureaus are interested in M&E and can contribute credibility and independence to the system. It can often be of great value to employ a competent third party to carry out some or all of the evaluation on behalf of the PMU/PIU. This may particularly be the case when a range of stakeholders are interested in a project, and may have conflicting interests regarding project outcomes and impacts. A competent third party adds a degree of credibility and independence that should satisfy even the most sceptical stakeholder group. However, where possible, the third party should work as an extension of the PMU/PIU and not a substitute. Capacity building within the PMU/PIU or within other government agencies may be necessary for the procurement and supervision of third party project evaluation inputs. An administrative unit for project level M&E that is separate from project management may only be justified in a PIU/PMU with a weak management history and very limited capacity, and for projects with multiple components implemented by multiple agencies. In the latter case the M&E unit should still be well integrated into the overall coordinating arrangements for the project. Similar observations apply for programmes composed of several projects. For example, the Kenya Agricultural Productivity and Sustainable Land Management Project (under preparation in 2007) planned for joint impact evaluation activities with three other World Bank projects in the area. Common data collection will be carried out across the four projects enabling comparison between project outcomes and successes, identification of potential synergies, integrated data sets for geographic up scaling and further research, and economies of scale in the data collection process. GN5 Organizational alternatives 2 Toolkit for monitoring and evaluation of AWM Guidance Note 5 projects Organizational alternatives for M&E of AWM projects For AWM projects the PIU/PMU will typically be comprised of a Project Director and several management units, with each unit being responsible for one section of the work (Figure 1). Thus in a typical project with both rehabilitation/modernization and WUA formation components, there will be a design unit, construction supervision unit and WUA formation and support unit, together with cross cutting units for procurement, financial control and monitoring and evaluation. Under this structure the M&E section is responsible for the M&E activities of the project, collecting data from other sections, project field staff and water users. The M&E unit will be composed of national staff, hired on long term (project) contracts, and will typically comprises 1 or 2 staff. As noted above, the M&E section may be supported by external consultants to assist with the following: · training and provision of technical advice on M&E; · specialist surveys, such as baseline surveys, diagnostic or special studies of leading indicators of project outcomes, and impact evaluation surveys and analysis; and · preparation of biannual/annual reports and future work programmes. The M&E section may also initiate approaches for participatory M&E, involving project stakeholders through community based organisations. This could particularly apply, for example, to the activities of the WUA formation and support section, or to attempts to monitor leading indicators of improved irrigation performance following completion of rehabilitation and modernization. For larger projects, specialist M&E consultants may be employed to carry out all of the M&E activities. They may work with a (small) M&E unit within the PIU/PMU, but they will have a clear remit and responsibility for the M&E work. The advantage of the first approach is that the M&E section is directly responsible to the Project Director, and should be feeding information back to him/her on a regular basis. With a good Project Director this will mean that the M&E is incorporated into the management processes, and that problems can be identified and rectified at an early stage. Under the second approach the Project Director generally has less direct input into the M&E process, and may tend to leave it to the specialists. There is a danger with this approach that the knowledge gained does not feed back into the local project management processes, but only up into the funding agencies and higher levels of government. As a consequence there is little ownership by the local management team and less immediate and direct use of the information. A GN5 Organizational alternatives 3 Toolkit for monitoring and evaluation of AWM Guidance Note 5 projects Organizational alternatives for M&E of AWM projects likely omission in such a scenario is adequate attention to leading indicators of project outcomes and a failure to investigate through special or diagnostic studies the causes of apparent poor performance in the early years of the project. Project management may be well aware that action is needed to modify and improve implementation at an early stage, but may lack both the information and authority needed to take this action effectively. Overall it is clear that getting the right structure for a project's M&E activities and responsibilities can avoid communication failures, conflicts of power and interest, forgotten or duplicated tasks, and wasted efforts. Sectoral level monitoring and evaluation Linking beyond the project to the sectoral and national levels it should be recognised that monitoring and evaluation are essential components of the governance structure, and thus are fundamentally related to the good conduct of public sector management. Thus monitoring and evaluation can be conducted at local, regional and national levels of government. At a national level organisational alternatives include the following: i) a `whole of government' approach entailing broad, comprehensive establishment of monitoring and evaluation across the government, and able to work at the level of sectors and policies. ii) an `enclave' approach, starting at local, state or regional level, or from pilot initiatives for a key ministry or agency; iii) a `blended' approach involving comprehensive monitoring and evaluation of selected areas or interventions, with infrequent or more superficial attention to others. Clearly it is practical and more cost effective if higher level indicators of outcomes and impact that are routinely required for projects, programmes or policy reform are collected at a sector, regional or national level rather than on a project specific basis. Project monitoring should thus be well integrated with existing data collection and reporting of statistics as carried out by the Ministry of Finance, national statistics agency or other government departments. For example, project monitoring should be well matched to national level M&E for PRSP implementation and other national data collection efforts. This will help to develop and reinforce a results based management culture across the public sector and partner agencies at all levels of management. For irrigation schemes and other AWM projects it will also facilitate the necessary transition from a project based mode of implementation and management to ongoing operations, once the investment and implementation period comes to a close. Sectoral level M&E should normally cover all the projects in the sector concerned. As in the case of the project level M&E unit, the sectoral level M&E unit need not be large. The actual size should be commensurate with the volume of work involved, which in turn depends on the nature, size, scope and number of projects in the sector concerned. However, staff of the sectoral level M&E should also have a close working relationship with those responsible for sectoral planning and supervision. The main purpose of sectoral level M&E is to keep track of overall progress in implementation of projects, programmes or policy reform in the sector and to assess results in terms of outcomes and impacts of interventions which have reached the full development stage. While M&E may be carried out at both project and sectoral levels, the relative roles differ. At the project level monitoring plays a more dominant role whereas at the sectoral level evaluation becomes more important. But more significantly, the nature of important issues also differs at the two levels. At the project level most issues are quantitative in nature, relating to inputs, processes, outputs, and the project development objective, whereas at the sectoral level, most issues are policy related and more difficult to quantify, e.g. poverty reduction and employment creation, or of a more qualitative nature, e.g. efficiency and effectiveness of institutions, and empowerment of beneficiaries through participation in decision making. GN5 Organizational alternatives 4 Toolkit for monitoring and evaluation of AWM Guidance Note 5 projects Organizational alternatives for M&E of AWM projects Organizational planning Whatever the organizational set up chosen, it is necessary to budget for M&E activities early in the project life (see RN5 for more guidance on costs of M&E systems and budget plans). World Bank TTLs need to consider organizational alternatives, capacity and training needs of M&E units and technical assistance resources to supplement the M&E unit during project preparation. Unless there is a commitment to adequately resource and fund the M&E activity, including the establishment of baseline data during project implementation, the whole M&E effort can run into trouble. During project design in particular, a TTL should consider whether to have M&E as a separate project component or an easily identifiable sub component of the project. By identifying M&E as a stand alone line item activity in the PAD it can be costed as a separate activity. The project financing plan, which will usually be a mix of counterpart and World Bank financing, can then be agreed on at the negotiations stage, including financing arrangements for M&E activities. In some cases it may be advisable to have a legal covenant to ensure that the M&E unit is properly setup, resourced and maintained through the life of the project. Legal covenants may also need to be written to ensure the timely completion of the baseline survey. Implementing project M&E Ideally, the M&E unit (or units) would be able to take part in design of the system. At the least, the unit should start functioning immediately after the establishment of the PIU/PMU. The tasks that need to be carried out by a PIU/PMU based M&E unit include the following. Thorough study of the project documents: in particular, the unit will need to fully understand the Project Development Objective and project component outcomes, with review of the assumptions in the project design and appraisal and analysis of the linkages, schedules, processes, and activities, inputs required, and targeted outputs, and identification of the critical areas and constraints which are likely to need special attention from project management. The M&E unit should discuss its findings with project and line managers and bring to their attention the critical areas, potential bottlenecks and constraints. Preparation of a complete logical framework analysis extending on the project Results Framework may be a useful aid to this if it has not been done as part of project design. After study of the relevant project documents the unit should review the information needed for M&E work. It should keep information needs to the essentials, and choose only those which are relevant, meaningful and objective, starting for World Bank projects from the Results Framework specified in the Project Appraisal Document. The M&E unit should consult project management and other potential users in determining all information needs. It should identify the indicators which can be used not only for `implementation monitoring', but also for `results monitoring', for on going evaluation by management (see Part A, Section 3.5), and for implementation completion and post project evaluations. When the information needs are identified, the unit should review existing information systems and databases, before undertaking any primary collection of data (see GN6). This can save a considerable amount of time and money. The M&E staff should carefully examine the available data and their sources for their contents, reliability, usefulness, frequency, and timeliness. The M&E staff should also review the pre project data assembled during project formulation and appraisal, and the extent to which this can provide adequate baselines. The unit should design and organise the collection of primary data if existing databases are found to be inadequate (see GN6). It should arrange for a regular flow of data from appropriate sources. Maximum use should be made of existing sources of information. In general, only when these cannot satisfy data needs, should attempts be made to collect new data. There are several methods for collecting additional data, including sample surveys, rapid assessment and in depth case studies (see GN6). The selection of method should be in accordance with the need. GN5 Organizational alternatives 5 Toolkit for monitoring and evaluation of AWM Guidance Note 5 projects Organizational alternatives for M&E of AWM projects The unit should communicate its findings and recommendations in a timely fashion (see GN6). Reports of M&E findings and recommendations should be submitted to the project manager and through him/her to the higher authorities as necessary. It must be stressed that timeliness is of extreme importance because delay in action could be very costly. A good deal of communication can be verbal, but the more important findings and recommendations should be in written reports. It is a good rule that reports should be short, simple, and as straightforward as possible. The most effective channel for communicating findings and recommendations is often staff co ordinating committee meetings, where M&E staff can explain and amplify their recommendations, and can also receive comments and feedback immediately. Building capacity and sustainable M&E systems In terms of capacity building, a good M&E design should develop the capacity of the relevant government or private sector agencies within the borrowing country and build on existing systems. Capacity building is widely acknowledged to be important but is often poorly defined. Capacity building needs will typically include: upgrading conceptual and analytical skills in monitoring and evaluation, selection of indicators, data collection methods, data management, and design of reporting systems. Also, and perhaps most importantly, capacity building will include developing a results oriented management culture that seeks out and effectively uses information in decision making. The M&E capacity requirements of the project should be considered in the context of the capacity and needs of sectoral and national institutions in the country. Virtually all implementing agencies will have existing reporting systems. An M&E design should build on these arrangements but develop further the technical skills required to plan information needs, design data collection, execute studies and surveys, analyse the data, and report results in a format relevant to users. Adequate institutional and human capacity is imperative for effective functioning of M&E systems within implementing organisations. The `readiness assessment' of existing local capacity for M&E at the design stage (referred to above) is thus essential. At a minimum, capacity must include: · the ability to successfully construct indicators, particularly for the Project Development Objective and project component outcomes; · the means to collect, aggregate, analyse, and report on data in relation to implementation, leading and results indicators and their baselines (i.e. to be able to manage the communication of M&E findings); and · the skills and understanding within project management to use the information effectively. Planning of the M&E provisions should then include: · identification of important capacity gaps and how to fill them (e.g. number of staff and the skill levels required); · estimation of the costs of capacity building for M&E; and · development of a capacity building plan to be implemented by the project. Such a capacity building plan for M&E will cover: · the identified skills gaps; · target persons in the central and sub units of the PIU/PMU, higher level government line agencies or project components, other partner implementing agencies or beneficiary organisations that require skill enhancement; · the timeline for the needed skills development; and · appropriate training periods, trainers, and costs. Project M&E typically ends with the closure of the project, yet activities financed by the project are usually on going and design of the M&E system and adequate capacity building can help sustain M&E efforts beyond the project term. This requires support to sectoral level monitoring and evaluation, embedding of monitoring efforts within government departments and national GN5 Organizational alternatives 6 Toolkit for monitoring and evaluation of AWM Guidance Note 5 projects Organizational alternatives for M&E of AWM projects statistical agencies, on going capacity building, and management and incentives for sustainability. Box 1 provides an example of support to establish national monitoring programs by a World Bank project in Vietnam. Box 1: Contributing to a national benchmarking program in Vietnam Water Resources Assistance Project The Water Resources Assistance Project in Vietnam includes three main components: irrigation systems modernization, dam safety management through rehabilitation works, and river flow stabilization works in Thu Bon Basin. The project shall finance a comprehensive benchmarking and training program on irrigation systems modernized. This will be implemented through a Vietnamese institute, which will conduct scheme audits. Preliminary benchmarking and audit of two schemes, Dau Tieng and Cam Son, have taken place during project preparation. The benchmarking program will focus in particular on the technical and financial performance of Irrigation Management Companies, which are public irrigation agencies overseeing management for the entire irrigation system. The Ministry of Agriculture and Rural Development has agreed to expand these benchmarking efforts into a national program on all the subprojects under the Water Resources Assistance Project. Source: World Bank Project Appraisal Document, 2004. Incentives and a management perspective Development of capacity of the required quality must pay attention to the incentives for managers, M&E staff and other people involved. Project managers, M&E officers and other partners should perceive M&E not as a bureaucratic task, but as an opportunity to learn about and to discuss problems and achievements openly, to reflect critically, and criticise constructively. M&E systems are widely approved and accepted in theory as an aid to effective management, but are often found to be poorly operated and ineffective in practice. Linking funds allocation to project performance demonstrated by M&E may be a way to `mainstream' M&E and provide the incentives for its effective implementation and use. Sometimes there is unstated opposition to monitoring in governments or implementing agencies. M&E may reveal weaknesses in management which may be linked with individuals managing a project, and it is human nature to oppose this possibility. To avoid this `human dimension' which can cause resistance to monitoring, management must participate in the design and operation of the M&E system from the earliest stages. This is necessary to avoid resentment of the time required for monitoring activities and data recording as part of a manager's duties, and the investigative nature of the process. It is also essential to ensure that the data collected is actually relevant to the needs of project management, and in the right form to be understood and used by them. As emphasised earlier, management are the prime users of data from monitoring. Following on from this, the involvement of management in monitoring will help avoid the over zealous generation of excessive, unused and unnecessary information. Good incentives for M&E are closely linked to incentives for good management and successful implementation. Examples of common incentives include: · clarity of M&E responsibility in job descriptions and work plans and the job satisfaction that comes from a meaningful and valued role in an organisation; · appropriate salaries and other rewards; · adequate resources to carry out required activities; and · opportunities for professional development and career advancement. GN5 Organizational alternatives 7 Toolkit for monitoring and evaluation of AWM Guidance Note 5 projects Organizational alternatives for M&E of AWM projects Addressing these issues at national and sectoral scale is relevant to the establishment of a successful and sustainable results-based management culture across the public sector and any contracted private sector agencies. Further reading Guijt, I., and J. Woodhill. 2002. A Guide for Project M&E. Rome, International Fund for Agricultural Development (IFAD). Available at http://www.ifad.org/evaluation/guide/index.htm Kusek, J.Z., and R.C. Rist. 2004. Ten Steps to a Results-Based Monitoring and Evaluation System. . Washington D.C.: World Bank. Available at http://go.worldbank.org/7H1YFQS5Z0 Rajalahti, R., J. Woelcke, et al. 2005. Monitoring and Evaluation for World Bank Agricultural Research and Extension Projects: A Good Practice Note. Agricultural and Rural Development Discussion Paper 20. Washington D.C.: World Bank. Available at http://intresources.worldbank.org/INTARD/Resources/M_E_WB_AgExtensionProjects.pdf World Bank Independent Evaluation Group. 2006. Water Management in Agriculture: Ten Years of World Bank Assistance, Capacity building is widely acknowledged to be important but is often poorly defined . Washington D.C.: World Bank. Available at: http://go.worldbank.org/ON8M9NGOZ0 GN5 Organizational alternatives 8 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects Guidance Note 6 Data Collection, Processing, and Analysis for M&E of AWM Projects Introduction This guidance note provides an overview of key concepts (Box 1) and issues relating to data collection and management for project M&E, and some detailed guidance on selection and use of data collection and management methods for agricultural water management projects. Box 1: Concepts Data: raw facts and figures. Information: data given meaning though processes of synthesis, analysis and interpretation. Knowledge: information of repeated use to project management when related to the project situation, and used to establish explanations and the basis for decisions. Primary data: collected directly by the organisation responsible for M&E and may include: · routine internal records such as administrative, budgetary, and personnel data; · work progress reports by managers or supervisors at different levels; and · data from surveys, interviews, direct observation and other data collection methods collected directly from the original source or respondent. Secondary data: collected by other organisations and for a purpose other than M&E of the specific project. E.g. statistics concerning the rural economy compiled by government agencies such as a central statistical office or department, data collected by international organisations such as FAO, UNDP, ILO and UNESCO, and data collected by universities and research institutes or private market research companies. Quantitative data: numerical data for specific variables, e.g. kilometres of canal rehabilitated, hours that women work, percentage of farmers who are WUA members etc. Usually recorded in a numeric form or as pre coded categories. Qualitative data: textual descriptions for indicators and situations, typically gathered from interviews and observation. Source: Authors Assessing data sources Data sources for indicators can be primary or secondary. Primary data are collected directly by the project team or agency concerned, whilst secondary data have been collected by other organizations for purposes not specific to the project concerned (Box 2 provides details of a useful database of major sources of secondary data). GN6 Data collection and analysis 1 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects Box 2: Development Data Platform Useful secondary data sources for project M&E include time series on indicators collected by the international community, national surveys conducted by Ministries and national statistical agencies, and other projects' records and monitoring information. Access to these sources of information is facilitated by the Development Data Platform of the World Bank. The Development Data Platform is an internet based tool made available by the World Bank to development practitioners. It provides the user with: (i) time series data on socio economic, trade, finance and other varaibles for many countries and sub national entities. It provides access to databases from the World Bank, the Food and Agriculture Organization, the International Monetary Fund, the Organisation for Economic Co operation and Development, the United Nations, and other international organizations which gather and disseminate international statistics. (ii) a repository of survey data and documentation from many countries. The catalog is maintained through collaborative efforts by several depositors and holds information on households, investment climate, service delivery and client feedback surveys. Authorized users can access unit record level datasets. This includes for example a list of population and housing census, integrated household surveys, agricultural census and surveys. Source: Adapted from World Bank Development Data Platform website Use of secondary rather than primary data has both advantages and disadvantages. On the positive side its use can be more cost effective, and for many project situations it may simply be too costly to collect detailed primary data when this would require a large and costly household survey, or alternative data collection method of comparable cost. On the negative side secondary data may have limitations if the purpose for which it was collected does not match well with the purpose intended for project M&E. The validity and reliability of the data must be considered, and any sources of bias and inaccuracy that may have arisen during its collection identified. Other potential problems with secondary data can arise in a number of ways. For example: · incomplete coverage of the specific project area; · inability to disaggregate the data to match the boundaries of the project area or sub areas; · inability to disaggregate the data to match the project affected population or sub groups; · inconsistencies in data collection in surveys implemented in different areas, by different teams or in different time periods (e.g. interviewing of household members in one survey and only household heads in another, or use of crop cut measurements for yield in one survey and farmer estimates in another); · inaccuracies arising from inappropriate choice of measurement and collection methods or inadequate training and supervision of data collection staff. Problems such as these may, when severe, invalidate any comparison intended to reveal and measure change in project component outcomes and impact. To address such issues M&E proposals should explain and justify the proposed approach and ensure consistency in methods. The complexity of the statistics and the problems of attributing causality may mean that it is often more cost effective and appropriate to use leading indicators such as delivery of services and beneficiary response as proxies, and at least as a complementary if not sole source of evidence, rather than to attempt to evaluate project impact using only secondary data sources. It is inevitable that in practice a mixture of primary and secondary data sources will be used for the M&E of most AWM projects, particularly those that are of a large scale and/or implemented over a wide area. Data comparability is an important issue to consider when a range of primary and secondary sources may be used. Some desired indicators of impact, such as Water User Association membership and household income attributable to a project, may involve comparisons with the situation before the project, or in areas not covered by the project. Such comparisons may depend on the maintenance of national systems of statistics and national or regional surveys. Before secondary data from a particular source are chosen for indicators of GN6 Data collection and analysis 2 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects project implementation or results it should be confirmed that the data systems are in place and reliable, that the data are valid for the project area or population groups in question (or for any control areas), sufficiently accurate for the purpose intended, and comparable to any other data required for the same aspect of the project's monitoring and evaluation. Overall a data collection system used for project M&E should be assessed in terms of reliability, validity, and timeliness. Reliability is the extent to which the data collection system is stable and consistent across time and space. In other words, measurement of the indicators is conducted in the same way on each occasion. Validity is achieved when indicators measure as directly and accurately as possible the changes of interest and relevance to project management. Timeliness consists of three elements: regularity in the frequency of data collection; currency (how recently data have been collected and how this matches important seasonal events or implementation `milestones'); and availability (provision of information at the right time to support management decisions). Planning requirements The World Bank Project Appraisal Document requires identification of indicators, arrangements for monitoring, and uses of M&E information (see GN1 and GN13). An important consideration is that the M&E system should collect only the data that is intended to be used. Excessive collection of unnecessary data will slow down processes of analysis and reporting, and may lead to a failure to communicate clear messages. Early identification of mechanisms for analysis, reporting, and use of the findings, will help avoid over collection of data. Thus a plan for the project M&E system should be based on a clear and detailed assessment of the following: · What ­ the data to be collected, from which sources, in what form, with what degree of aggregation or consolidation, and for what purpose · When ­ the frequency of data collection and reporting · Who ­ the responsible persons, their responsibilities and capacities · How methods and procedures for data collection, checking, validation and storage, and for analysis, reporting, and use of information · Where ­ locations for data collection and processing, and the destinations for reported information. Core data collection methods for AWM projects Overview Indicators for inputs, processes and outputs will generally come from project management records originating from field sites, and will be part of the project management information system (see below). To measure outcomes and impact will typically require the collection of primary data from formal sample surveys, used in combination with qualitative methods (see below). Existing secondary data may provide an alternative subject to the possible limitations discussed above. There must be adequate capacity for baseline data collection and repeat surveys that will compile a continuous or periodic time series of data for key indicators. Where possible, it may be better to add project specific regular surveys on to existing national or area surveys than to create a new data collection facility (see GN5). Information will be stored and managed either on the management information system or on a separate, but related, results monitoring system. Routine monitoring and evaluation, particularly of leading indicators of outcomes, may reveal problems during implementation. An example would be a disappointing response rate among primary beneficiaries such as a low rate of credit uptake to finance improvements in on farm water management. Such situations may call for ad hoc diagnostic studies to determine the GN6 Data collection and analysis 3 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects cause of the problem and identify possible solutions. Again a range of methods may be used for such studies as considered further below. Project management information system Project management records originating from field sites will provide the raw data for the project management information system (MIS). Box 3, RN5 and RN6 give more information on the MIS. The quality of record keeping in the field sets the standard for all other uses of data and merits careful planning and attention. It is important that the data collection is systematic and that data are collected on time for all specified periods. The seasonal nature of agricultural production imposes particular requirements and constraints upon this. M&E designers and managers should together determine what information will be needed at field, intermediate and senior management levels. They should also assess the capacity of existing record keeping and reporting procedures to generate the information that will be needed. In many cases, the information on outcomes and impacts is stored and managed on a separate results monitoring system, built on similar principles. Coordination between the two systems will be very important. Box 3: Project Management Information System A computerized Management Information System (MIS) is the "window" that captures the quantitative data within the M&E system. It will capture both progress and process data for concurrent monitoring of activities at every level within the project. Starting point: The starting point of the design of an MIS is to identify the inputs, the outputs, and the process to be followed. This should then be followed by a clear and lucid description of the project cycle and the results framework that includes an institutionally defined set of indicators. Indicators will be of physical progress, financial progress, and process (inputs, processes, outputs and outcomes). The designer ­ a locally or internationally hired consultant ­ determines what information is needed and in which form, which computerized modules would best satisfy information needs, and how modules should relate to each other. A developer prepares the computerized aspects of the MIS, where programs and procedures are written for each modules. Softwares are then installed on computers and staff is trained to use it. Critical elements in the design of an operational MIS for AWM projects The MIS should cover financial and accounting management of the projects internal activities and include M&E of sub projects funded by the projects. Census Data or other habitation, income, and poverty level data along with geocode information on the classification by village, district, or other administrative borders need to be available for uploading into the MIS. Storage and access procedures should be organized. A Central Server that is either owned or leased needs to be secured, and all of the data will reside on this central server. Hardware and software and peripheral equipment will be connected to the central server in order to access and process information. Access to the data will be based on predefined protocols and compartmentalized rights of use. Reports will be organized as those being accessible to the general public and those available for project management purposes, only. Summary tables are prepared for easy and quick reading of information, as well as geographic representation. Trained personnel will be needed at each level to manage and operate the equipment. At each level within the project structure a minimum of two persons (coordinator and analyst) will be required to input, process, report, and access information and data for that particular level. Source: Adapted from Ghazali Raheem, in Rajalahti, 2005, and from World Bank Water Supply and Sanitation M&E Toolkit, 2007b, draft. GN6 Data collection and analysis 4 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects The range of relevant data collection methods Project monitoring and evaluation should make use of a wide range of methods for gathering, analysing, storing and presenting data. Figure 1 illustrates some of the possible methods of collecting data. They range from simple record keeping to primary surveys of farm production and indicators of household and individual welfare (the main methods applicable to AWM projects are considered in more detail below). There is no single answer as to which method is best, and selection will depend on the project and project area characteristics, resource and time availability and the needs of the users of the information. Inevitable there are trade offs in terms of time, resource availability and cost in seeking to achieve required levels of reliability, validity and timeliness (as set out above). Before decisions are made on the strategies to employ it is important to consult with the users of the information to gain a clear understanding of their needs and priorities, and their perspectives on the trade offs that may be necessary. Allowing for a number of scenarios and contingencies helps to frame what is possible and what can be afforded. Figure 1: Data collection methods: an illustrative range Method Field experiments Formal and more Census structured methods Panel surveys One off survey Case study Direct observation Focus group interviews Key informant interviews Other PRA/RRA techniques e.g. transect walks, wealth and matrix ranking Community meetings Informal and less Field visits structured methods Ad hoc meetings and interviews Source: Adapted from Kusek and Rist, 2004. When looking at trade offs, more structured and formal data collection methods tend to have greater accuracy and validity, but are also more costly and time consuming. Table 1 provides an illustrative comparison of four major data collection instruments for four important characteristics. For data that is needed frequently and on a routine basis to inform management decision making, it may be preferable to adopt less structured and costly collection strategies. This will often be appropriate for `implementation monitoring' for which timeliness of information provision is paramount. In contrast, rigorous approaches to impact evaluation that address the problem of attribution (Part A, Section 3.7) will generally required a more formal and structured approach (GN9), and hence may need to be applied selectively rather than to all projects. GN6 Data collection and analysis 5 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects Table 1: Comparison of major data collection instruments relevant to project M&E Amount of Characteristic training required Cost for data collectors Completion time Response rate dohte Review of Depends on amount Low Some High program records of data needed m Self-administered Depends on how noitcelloc Moderate None to some Moderate questionnaire distributed Interview Moderate to high Moderate to high Moderate Moderate to good ata Depends on Rating by trained availability of low Moderate to high Short to moderate High D observer cost observers Source: Kusek and Rist, 2004. It is also often possible and necessary to combine data collection methods to achieve the best results and to track particular indicators, and this may involve collection of both quantitative and qualitative data. For example, a participatory or rapid rural appraisal (PRA/RRA) to assess leading indicators of how primary stakeholders are responding to a project might combine several methods of data collection ranging from a transect walk to matrix ranking and focus groups discussions. Alternatively for an ad hoc diagnostic study in response to problems in implementation identified by monitoring, informal key informant interviews and a community meeting might be used to identify the important issues and inform the design of a subsequent formal household survey that would quantify the range and extent of the problems amongst potential beneficiaries. It is not always possible to identify the best method for data collection in advance. Where necessary, the project management team and M&E staff should carry out pilot testing of the M&E system. Such pilot tests will reveal whether a data collection method or instrument such as a survey form can reliably produce the required data, and how best the data collection procedures can be put into operation. A pilot testing approach should particularly be used for major and costly data collection exercises. Data collection plans should not be permanently fixed from the commencement of the project. As project management responds to changing circumstances and adopts an adaptive approach to implementation, so will information needs change. There needs to be sufficient adaptability and flexibility in the M&E system to identify new indicators, data sources, collection methods and ways of reporting as required. Data collection methods most relevant to M&E of AWM Projects Key principles for the most relevant and commonly used data collection methods are briefly reviewed below. It is beyond the scope of this Toolkit to provide more detailed guidance and users are recommended to consult the references listed at the end of this note. 1. Bio physical measurements These are measurements of physical change over time, such as for example, crop yield, flow rate in channels, water table depth and water pollution. Such measures may require recording instruments installed on site, or brought in at the required frequency, operated by project staff or external experts. Selected methods need to be adapted to local conditions, skills and resources, and it may be necessary to seek a compromise between local capacities and technological appropriateness and the achievable level of scientific accuracy. Use of remote sensing techniques (see GN8) may offer an alternative to, or may supplement, on site measurements. Whatever methods are adopted, the data need to be recorded in standard forms to facilitate easy and consistent analysis, and the making of comparisons over time and space. GN6 Data collection and analysis 6 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects 2. Sampling methods For both bio physical and socio economic indicators, sampling will usually be needed given resource constraints and the size of the `system' or `population' to be monitored. The following will be needed to achieve a representative sample and valid and reliable results for the project area, system or target population. Clarification and accurate listing of the sampling frame: a listing of the sampling units that make up the population to be studied. For example, all farming households cultivating land within the scheme command area, channels for flow measurement, and sites for monitoring water table depth or water quality. Selection of an appropriate sample size: based on available budget and resources, the number of sub groups analyzed, the time available, the variation within the population for the key variable(s) to be monitored, and the desired level of accuracy and statistical confidence. Selection of the sampling method. A random sample is often chosen for quantitative data and for particularly for socio economic indicators. In random sampling each individual sampling unit has an equal chance of being selected, there are clearly defined procedures for drawing the sample from the sampling frame, and estimates of the magnitude of sampling error can be made. This enables estimates of population characteristics to be made from the sample estimates. Non random sampling is more often associated with qualitative data collection and analysis, and involves a focused and deliberate sampling from the population, creating the risk of sample selection bias. Purposive sampling might also be used for some bio physical measurements so as to address particular aspects of system performance, or to ensure coverage of key locations or other sources of concern. Stratification of a population before selection of a random sample from each stratum can help to improve the statistical efficiency of sampling, and hence the extent to which the selected sample is representative of the population. In practice, because of the difficulty of compiling a comprehensive sampling frame and/or to improve survey logistics by concentrating the points from which data is collected, it is common to use a multi stage sample procedure. In contrast to stratification this reduces sampling efficiency and thus the potential accuracy of the sample estimates obtained. It may be necessary to compensate for this effect by increasing the sample size. When an accurate and comprehensive sampling frame is available, selection of a small random sample is a straightforward task. However, in many other situations an experienced sampling specialist may be needed to determine the appropriate sampling frame, sample size and sample selection strategy. This is most likely to be the case for socio economic data and for indicators of project component outcomes and project impacts; in other words for `results monitoring' and for project impact evaluation. 3. Formal surveys A formal survey typical involves taking a range of measurements or observations from a relatively large sample. Surveys are commonly used at the start and end of a project, to gather baseline information and compare outcomes to targets. A survey can also be carried out as part of a mid term review, to monitor progress and adapt project implementation as needed. A standardised form may be used for recording physical measurements or estimates based on observation, whilst for socio economic data the survey instrument will usually be a structured questionnaire used to record the data provided by selected respondents who are individually interviewed. RN3 and RN4 provide examples of such survey questionnaires and recording forms. Such socio economic surveys are often an important part of `results monitoring' as they can achieve focused, valid and reliable data collection on topics such as the composition of the target population, attitudes towards the project, and perceptions of change in key variables such as production, incomes, vulnerability, or empowerment (see GN9). However, it will also often be important to supplement the quantitative results of such surveys with in depth qualitative GN6 Data collection and analysis 7 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects information that can be used to help explain the changes that have been recorded and to establish the causes of these changes. Survey questionnaires usually consist of a pre defined and specific sequence of structured closed questions. Numerical or coded responses are recorded and this allows for rapid quantitative and statistical analysis. Open ended questions and/or short checklists for discussion which probe more deeply into respondents' attitudes, knowledge and opinions can also be included, but it will be more difficult to complete a quantitative analysis of the latter type of information unless the responses can be readily categorised and coded. As the coverage of the questionnaire is usually prepared and standardized before the interviews begin, it is important to pre test the interview procedures and questions. The aim is to ensure that they are appropriate and will generate the data that is needed. A survey specialist should specify the sample size and sampling method. The data is then collected by trained enumerators, usually using pre coded forms which can be analyzed in the office once the data has been checked and entered into a computer (see RN3 and RN4 for examples of prototype questionnaires). 4. Semi structured interviews This data collection technique is commonly used in informal surveys and typically involves a relatively small and non random sample. It is used to gain information from an individual or a small group, using a series of broad questions to guide the conversations, but allowing for new questions to arise as a result of the discussion. Such interviews can be used to develop in depth understanding of context, processes and issues, assess unintended impacts, and gather opinions about the relevance and quality of project services (see GN11 for their use in Participatory M&E). Qualitative interviewing of this type can provide understanding of the perspectives, attitudes, and behaviour patterns of the target population. It is thus often appropriate for gathering initial data for leading indicators of project outcomes and impact. It can also be used to generate hypotheses and propositions that are then tested on a wider population using a structured survey. Also, it is flexible enough to allow the interviewer to pursue unanticipated lines of inquiry and to probe into issues in depth. Finally, with this method there is a greater likelihood of getting input from senior officials or other key informants who may hold sensitive information. A M&E specialist should normally select the sample to be interviewed according to the purpose of the investigation. The interview checklist should be pre tested before use and enumerators who conduct the interviews will need to be well trained and experienced. Typically they will need to have good knowledge of the local area and population so as to be able to interact naturally and effectively with the respondents. It is desirable for two trained team members to conduct each interview, the interviewer and an observer who will recording the information provided by the respondent. As this method of data collection is less structured and open ended it can be difficult and time consuming to analyse what is collected and to synthesize clear results. Box 4 provides some suggestions on how to approach this. GN6 Data collection and analysis 8 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects Box 4: Tips to structure open-ended information for analysis Produce a short summary of what each person says, including the main points Begin to look over the responses. Once you have looked at about 25% of them, note the points most frequently mentioned. Then read all the responses and record how many interviewees have responded similarly on each of these main points. Highlight any important quotes to emphasize the main points. Ask other people to also look through the responses to prevent your own biases influencing the way you interpret responses. Number each respondent. Following the list of points you developed, number the main points. Through this numbered coding system, prioritise, summarize, and then analyze the information. Source: Guijt and Woodhill, 2002. 5. Key informant interviews Key informants can be an important source of information for project M&E. Interviews may be face to face or by telephone, and questioning is generally semi structured and open ended but can make use of structured close ended questionnaires. Key informants will be a small and purposively selected sample, chosen because of their particular knowledge and position. The selection of informants should adequately represent possibly diverse viewpoints and concerns, and should be sensitive to gender or other sources of bias. Village chiefs, teachers, WUA representatives, local officials, and higher level officials are examples of key informants relevant to AWM projects. Box 5 shows an example of use of key informant interviews and focus groups to evaluate the impact of a project in Macedonia. Box 5: Social Impact Assessment of the Irrigation Rehabilitation and Restructuring Project, Macedonia The Irrigation Rehabilitation and Restructuring Project (1997 2006) included both institutional reforms for irrigation management and physical rehabilitation of the distributive irrigation infrastructure in three main schemes. A Social Impact Assessment was carried out before completion of the project to identify impact on intended beneficiaries' welfare, and to highlight the level of social impact on the communities. The assessment drew mainly on qualitative data collection, with expert and key informant interviews and focus group discussions. The questionnaires for interviews were pilot tested in project areas, and local researchers were trained over a two day workshop to carry out fieldwork sites interviews. The Social Impact Assessment showed satisfaction of the stakeholders with progress achieved, and also identified areas requiring further attention by the government. According to all irrigation water users respondents, the project significantly improved irrigation infrastructure in all areas, and Irrigation Water Communities, actively supported by water users, improved the quantity and quality of water supply. However, respondents also identified some pending issues with Irrigation Water Communities, for example lack of adequate equipment for cleaning large earth canals. With regard to payment of the irrigation service fee, interviews showed that water users' attitudes underwent a major change during project implementation, from a reluctance to pay to understanding the benefits of timely payments. As to community empowerment, interviews showed that the 2004 reform establishing Irrigation Water Communities and Water Economies was seen as very important by farmers. These organizations enhanced social cohesion in villages and were key in improving the quality of agricultural production. Although turbulent at the beginning, relations between Irrigation Water Communities and Water Economies improved over time. Respondents noted however that members are becoming less active in the Irrigation Water Communities. Source: De Soto et al, 2006. GN6 Data collection and analysis 9 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects 6. Focus group interviews Discussion with selected groups that are familiar with pertinent issues is another technique that can be used to explore issues and processes, and to clarify details and gather opinions, before designing a formal and structured survey. Focus groups are particularly useful for assessing opinions of change and the causes of change, and the quality of project services, and for identifying areas that need improvement. They can also help to identify hierarchical influences within the community if the group is heterogeneous. The M&E specialist will decide on the number of focus groups to be conducted, their composition (more or less homogeneous) and size (usually from six to twelve people). Each group is presented with a broad question that is discussed for the prescribed time period (usually one to two hours). A facilitator makes sure everybody has an opportunity to speak, whilst an observer takes detailed note of the discussion (see also GN11). The main limitations of this method are that it can be expensive and time consuming, and care must be taken in seeking to generalise the findings for the project population or area as a whole. 7. Community meetings Community meetings take the form of public meetings with a larger group. When used for the purpose of gathering project M&E information they should be based on an interview guide or checklist and facilitated by an interdisciplinary team rather than a single interviewer. Attendance at the meeting should be representative of the population of interest, for example, villagers from head, middle and tail reaches of a large irrigation scheme. Meetings should be held at a time of the day and place which is convenient for the community and does not clash with work or other social commitments. On of the most difficult tasks for facilitators is to restrain elite members of the community from dominating the meeting and monopolising the discussion. Facilitators should encourage different people to participate, and opinions on certain subtopics should be verified by polling the attendees when this is possible. 8. Rapid Rural Appraisal (RRA) RRA can be described as a systematic but semi structured research activity carried out by a multidisciplinary team over a relatively short period of time. It can involve a range of informal data collection techniques such as semi structured interviews, transect walks, mapping, and wealth and matrix ranking. It can be used as part of the project M&E system, either at appraisal to gather baseline information and help improve project design, or over the course of the project to assess and evaluate progress. 9. Participatory Rural Appraisal (PRA) PRA uses a similar array of data collection methods to RRA but places greater emphasis on the participation of local people in identifying the issues to be investigated and in the collection and analysis of data. Its application is discussed further in considering participatory approaches to M&E in GN11. 10. Case study In the context of project M&E a case study documents the sequence of events over time related to a person, household, location, or organization, and facilitates in depth understanding of the processes and human and other factors behind observed changes. The need for a case study can arise, for example, from a more general formal survey in which a particular issue emerges as needing more in depth investigation. The M&E specialist should specify the purpose and information needs of the case study, then decide how individuals, households, or organizations will be selected for the study, and how data will be obtained. The case study findings can draw on a variety of evidence from documents, interviews, and direct observations. Questionnaires or checklists may be used to guide the GN6 Data collection and analysis 10 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects information collection. Discussions and observations are typically repeated over time to assess processes of change and achieve an up to date picture as conditions change. Good case studies are difficult to do and can require high level and specialised research skills. They are also time consuming and findings are subject to the limitation that they may not be generalisable for the whole population or project area. 11. Direct observation This method involves structured observation of an activity, behaviour, relationship, phenomenon, network, or process in the field. It can be used to understand the context in which M&E data is collected, and help explain M&E results (see also GN11). Phenomena and processes can be studied in their natural setting and a holistic understanding gained. For example, regular observation of WUA meetings could reveal how priorities are set and decisions made, whilst observation in the field could show the detail of how irrigation operations are performed by farmers. Such activity should be well planned by project management and M&E staff, who should agree a clear conceptual framework, as well as guidelines for what needs to be observed and the information required. It is then necessary to choose and train the appropriate group of observers who may be community members, project staff or knowledgeable outsiders. Information can be recorded in logs or diaries, discussed with stakeholders and used for M&E analysis. In general direct observation should always be used in conjunction with other M&E methods as the quality and usefulness of data is highly dependent on the observer's observational skills and findings can be open to interpretation. 12. Written documents analysis and review of programme records This method involves reviewing project documents and records such as administrative databases, training materials, correspondence and routine progress reports. It can be very useful in identifying issues to investigate further and provide evidence of action, change, and impact, to support respondents' perceptions. For example, a disaggregated analysis of records of financial or labour contributions made to a WUA for scheme maintenance may help to reveal which groups within an irrigation scheme are most actively engaged and prompt investigation of the reasons for lesser involvement of others. In planning the M&E approach, the M&E specialist(s) should prioritise those project records that are most likely to provide useful information in relation to key indicators and to the phases of implementation and results monitoring. The quality of data stored in the project records needs to be assessed, and as far as possible its format and storage managed to facilitate review and analysis in a cost effective and efficient manner. Key procedures in collecting and storing data It is important that all data collection is managed in a systematic fashion and that data are regularly collected for the frequency and period specified. For example, in the case of canal discharges missing or erroneous data at key times can invalidate the whole data collection program for an entire season. Careful monitoring of the data collection process is thus required to avoid this situation. Where there is a significant and diverse amount of data to be collected, then a checklist can be used to monitor which data have been collected (see example in GN15). To avoid laborious processing and reworking of the data it is also important to plan and prepare standard data collection forms and database formats so that the data are recorded and stored in a manner that is systematic and easy to process (see RN4 for an example of survey record sheets). It is important to establish a standard system for labelling data files (see GN15), and to establish a routine for regular safe storage of data backups. Increasingly, data collection is computer assisted. Notwithstanding remote sensing (see GN8), mobile computing tools are increasingly available and can facilitate data collection by M&E staff in GN6 Data collection and analysis 11 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects the field. In particular, personal digital assistants have been used in surveys to record questionnaire answers. The computing device is pre programmed with the list of questions and logical connectors to unfold the question path. Furthermore, the latest generation of computing tools is now looking towards integrating geospatial information together with data recorded by M&E staff. In some other cases, staff in line ministries and provincial governments can respond to online surveys where internet is available. These developments are still at an early stage though in client countries. Field staff must be carefully trained in the use of these technologies, effective support protocols must be in place, and devices have to be adapted to field conditions, in particular handling dirt and rain. Sources of data errors and data validation Box 6 distinguishes between sampling and non sampling errors. The key to high quality M&E work is to reduce non sampling errors. The first step is training and supervision of field staff. The second step is that data checking and validation should be routine in office activities carried out for all data collected from the field prior to final data entry, storage and analysis. Data coming in from the field needs to be checked for coverage, completeness and as far as possible for obvious sources of error, bias and inaccuracy prior to computer entry. Consistency checks can be developed and applied to test the internal validity of the data collected. Once data entry is completed then the computerised records should also be checked against the original survey forms used. This particularly applies to quantitative and coded data for formal surveys, but at least some checks should also be made for records of qualitative data. Handheld terminals used to record data in the field, software routines and scanners can be used to automate some of these functions if resources allow, but for many M&E applications there will be no substitute for careful manual checking of survey forms and computerised records. Box 6: Sampling and non-sampling errors Sampling error is the inevitable result of collecting data from a sample rather than from the whole project population. It is the difference between an estimate derived from a sample survey and the true value for the population as a whole. Statistical theory enables estimation of the magnitude of this error. Sample estimates are thus usually cited within a margin of error (or confidence interval) at a given level of statistical confidence. Typically a confidence level of 95 percent is used, but a lower level may be acceptable for some project M&E applications. Normally a trade off will have to be made between the statistical precision of the estimates derived from samples and the costs and logistics of alternatives for sample size and method of selection. Non sampling errors arise from human mistakes and from observation, measurement and recording inaccuracies, and can occur at all stages of the data collection process. They include: · listing errors · errors of omission · non responses by interviewees · inaccurate responses · measurement errors · recording errors · coding errors · data entry errors. Some of these errors may be systematic and can be allowed for once identified, as long as equipment and conditions remain unchanged. Others occur at random and are difficult to identify and quantify. Appropriate and well designed data collection procedures and adequate supervision and training of staff are the keys to reducing non sampling errors. Where resource constraints apply it may be better to adopt simple and small scale survey methods rather than large and complex formal surveys from which data quality may inevitably be poor. Source: Adapted from Casley and Kumar, 1988 GN6 Data collection and analysis 12 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects Data analysis For quantitative data a mix of approaches will be needed. Some physical quantities recorded in the field may be directly reported. For example, the number of complaints made, or measurements of water table depth for a given location. In other cases raw data will be converted to ratios or other values through calculations or the use of simulation models, for example, calculation of flow rates or poverty ratios. Analysis of socio economic data collected from household or other surveys may typically require the use of statistical and econometric methods, but commercial software packages in widespread use are generally well equipped to perform the most commonly used techniques. Procedures for computing indicators from the raw data should be established in advance and can be presented in a table format (see GN15 for an example). Documentation made available to the analyst could include: · information needed to interpret the data, including codebooks, data dictionaries, guides to constructed variables, and any needed translations; · information needed to conduct the analysis, including a description of the indicator, details on the methodology, and guidelines for using the data. Most learning from qualitative data is obtained by writing descriptive summaries and collating and sorting these summaries into categories of response. Qualitative information drawn from interviews, observations and documents can be processed through content analysis (Box 7). As far as possible all relevant project staff, implementing partners and primary stakeholders should be involved or consulted as open ended discussions about the analysis will help explain the data and develop a collective and iterative learning process. Box 7: Content analysis In reviewing the data, the evaluator develops a classification system for the data, organizing information based on (i) the evaluation questions for which the information was collected, (ii) how the material will be used, and (iii) the need for cross referencing the information. The coding of data can be quite complex and may require many assumptions. Once a classification system has been set up, the analysis phase begins, also a difficult process. This involves looking for patterns in the data and moving beyond description toward developing an understanding of program processes, outcomes, and impacts. This is best carried out with the involvement of team members. New ethnographic and linguistic computer programs are also now available, designed to support the analysis of qualitative data. Source: Baker, 2000. The M&E system should aim at providing timely information of high quality. Timeliness will depend on having as much integration as possible between data collection and processing, and setting clear deadlines in the communication strategy. Reporting and using M&E findings Reporting M&E findings M&E related information has many potential audiences: primary stakeholders, the project implementation unit, government counterparts, implementing partners, and funding agencies. The PAD usually plans the reporting of implementation progress for use by project staff, and specifies arrangements for supervision and reporting to the Bank. For example, the annual Implementation Status and Results report includes indicators of progress toward achieving the project development objective and intermediate outcomes/ results (see GN13). The PAD may also present a schedule for a mid term review mission and its reporting. Beyond World Bank requirements findings can be usefully be communicated more widely and via different channels to the client country and stakeholders (Box 8 provides an example). GN6 Data collection and analysis 13 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects From the outset of the project a communication strategy needs to be developed that will address the following questions: · Who will receive what information? · In what format? · When? · Who will prepare the information? · Who will deliver the information? Box 8: Planned M&E reporting channels in Kecamatan Development Project 3B, Indonesia Internal implementation monitoring: -consultants will report on implementation monthly, via the MIS -government officials will report on project progress and community members will monitor projects within their communities - project staff to carry out six to eight case studies per year to document lessons learned and identify best practices. External monitoring: -28 external NGOs will produce monthly monitoring reports to provide an independent source of qualitative information -32 provincial journalists will be provided with operational expenses so they can visit and report on KDP project sites in regional and national newspapers -World Bank supervision missions will produce aide memoires -consultant firms will carry out participation and local governance impact evaluations, to measure changes related to empowerment, governance and poverty issues using both quantitative and qualitative methodologies. Source: World Bank, 2005g. Data should be presented in a timely and concise manner and be relevant to the target audience. As there are multiple audiences the information needs to be prioritized and presented according to the main interests, preferences, and knowledge level of each user (see below the different channels for reporting). Typically the higher up the management chain, the less need there is for extensive detail and explanation, and aggregated and succinct data relevant to the specific issue will be more appropriate. For this reason personal briefings especially to high level officials can be an effective means of communicating M&E findings. Both informal (phone, e mail, fax, conversations) and formal (briefings, presentations, written reports) communications should be considered. Further down the managerial chain it is likely that more operational data will be desired. When findings are shared with project beneficiaries, visual presentations on a few major themes will be needed, followed by discussion of findings, alternative actions and next steps. Reporting M&E findings will generally entail comparing actual outcomes to targets (see GN3 for examples). The information is usually presented: · as a function of time, showing the indicator trend with regard to its target value. It is necessary to always report against the baseline and intermediate measurements to determine whether progress has been sustained, whether there was only a short spurt of improvement, or whether early improvements have all disappeared. · With an indicator value for different geographical units within the project area as and when appropriate, to show the spatial distribution of the indicator. There are four main components of reporting: written summaries, executive summaries, oral presentations, and visual presentations. · The written summary should contain an introduction (covering the purpose of the report, the key M&E questions, the project background and its objectives) and a description of the methodology. It should present data on findings selectively, organizing information around GN6 Data collection and analysis 14 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects study questions, major themes, or project components, and with effective use of graphical presentations. All recommendations should be supported by evidence. · Executive summaries should be short (one to four pages). Major findings and recommendations should be presented in bullet format. The summary can refer readers to the report or appendices for more details. · Oral presentations should consider who the audience is, what they should remember from the presentation, and what the available delivery resources are. Oral presentations should be simple, clear and tailored to the audience. Complex language and detailed data should be avoided. If possible, an interactive format with the audience should be used. · Visual presentations charts, graphs, and maps are also helpful in highlighting key points and performance findings. They can illustrate directions and trends at a glance. There are a variety of charts (pie, flow, column, time series, scatter plot, bar, range, and so forth) and graphs (line, scatter, bar, pie, surface, pictograph, contour, histogram, area, circle, column) that should be considered in presenting data to the target audience (see GN15 for examples).Tables are best used for presenting data, and highlighting changes, comparisons, and relationships. Charts are better for presenting the message, depicting organizational structures, demonstrating flows, presenting data as symbols, and conveying concepts and ideas. RN6 shows samples of internal M&E reports in the Karnataka Watershed Management Project, India. Using M&E findings Using findings to improve project performance is the main purpose of building a M&E system. The main point of the M&E system is not simply to generate continuous information, but to get that information to the appropriate users in a timely fashion so that the performance feedback can be used to better manage projects and organizations. The Results Framework (GN1) used in World Bank projects thus requires identification of mechanisms for use of M&E information (see also GN13). M&E information should be used for adaptive management. Learning about successes and failures of the project and regular reflection are fundamental to maximise impact. These help refine or revise the project approach and adapt to changing circumstances. To do so, it is necessary to understand the project design, gather and analyze the relevant information to make good decisions, dialogue with key stakeholders, and negotiate required changes. The mid term review is a good opportunity to assess the project approach and, if necessary, revise it with partners. M&E information should be collected, analyzed, and reported in advance to prepare for this review, and necessary diagnostic studies commissioned where there are performance problems. Another important use of M&E information is to improve operational resource allocation decisions and identify and plan for additional needs and resources requirements, especially by monitoring disbursements flows and outputs. To encourage individuals involved in the project, M&E systems are sometimes used also to identify best performers, such as irrigation agency staff, community facilitators, or Water Users Associations. Their achievements are then publicly acknowledged, for example during an annual competition. Such an approach has to be carefully adopted, as project staff and partners should not feel under scrutiny for their performance and loose trust in the monitoring system. Finally, M&E information can be used for awareness raising and advocacy purposes. Box 8 shows some external mechanisms put in place for reporting that help build ownership by the communities involved and awareness at all levels in the country. Internally produced information GN6 Data collection and analysis 15 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects on outputs, outcomes, and impacts can also be broadcast. Box 9 shows examples of planned use of information for two Bank projects. Box 9: Uses of M&E information ­ Results Frameworks of AWM projects Second North-East Irrigated Agriculture Project, Sri Lanka Result Indicator Use of the information Development Objective: Number of sustainable village Year 3: Use the information to streamline Help conflict affected level community organizations the strategies and change/ improve the communities to restore capable of village level program at Mid Term Review to result in livelihoods, and enhance planning, implementation and sustainable village organizations production and incomes monitoring development activities. Outcome: Sustainable Number (and Percent) of Year 1 onward: Poor representation and Community Based Organizations functioning with inclusion flags poor sustainability and Organizations (1,000) are executive committee and need to change/ refine the strategy. formed, strengthened general membership inclusive and empowered. and representative of total village community Number (and Percent) of Year 3 5: Satisfactory levels would help Organizations satisfactorily better targeting villages and beneficiaries engaged in subproject later for relief/ rehabilitation programs. implementation Outcome: Productive and Percent increase of area Year 3 5. Data would help Northeast service capacity of (24,000 ha) farmed due to Provincial Council to plan its agricultural essential village level increased supply of water strategy and allocate resources to social and economic resulting from irrigation enhance agricultural productivity and infrastructure restored scheme rehabilitation production and to establish market and enhanced. technological linkages between farmers and the public/ private sector. Irrigated Agriculture Intensification Loan III, China Result Indicator Use of the information Development Objective: Increase in average per capita -Improve policy, design and increase water and income for typical farm implementation of national agricultural productivity, households, in crop Comprehensive Agriculture raise farmer's competitive production, in water Development (CAD) program; capability, and promote productivity -Provide input by example into national sustainable and Number of counties rural strategy; participatory implementing groundwater - Demonstrate impact of integrated CAD water resource and management plans approach for implementation of real environmental water savings and sustainable irrigated management agriculture under water scarcity; -Help guide project assessment and adjustment at the Mid Term Review. Source: World Bank. Project Appraisal Documents. GN6 Data collection and analysis 16 Toolkit for monitoring and evaluation of AWM Guidance Note 6 projects Data collection and analysis for M&E of AWM projects Further Reading Baker, J. 2000. Evaluating the impact of development projects on poverty: a handbook for practitioners, Volume 1. Washington D.C.: World Bank. Available at http://go.worldbank.org/8E2ZTGBOI0 Casley, D., and K. Kumar. 1988. Collection, Analysis, and Use of Monitoring and Evaluation Data. Washington D.C.: World Bank. Guijt, I., and J. Woodhill. 2002. A Guide for Project M&E. Rome, International Fund for Agricultural Development (IFAD). Available at http://www.ifad.org/evaluation/guide/index.htm Van der Schans, M. L., and P. Lemperiere. 2006. Participatory rapid diagnosis and action planning for irrigated agriculture systems. Improving Irrigation Performance in Africa Project. Rome, Food and Agriculture Organization of the United Nations. World Bank. 1996. The World Bank Participation Sourcebook. Washington D.C.: World Bank. Available at http://www.worldbank.org/wbi/sourcebook/sbhome.htm Website World Bank Development Data Platform website: http://go.worldbank.org/DNBRRS9TB0 GN6 Data collection and analysis 17 Toolkit for monitoring and evaluation of AWM Guidance Note 7 projects Design and implementation of baseline surveys for AWM Projects Guidance Note 7 Design and Implementation of Baseline Surveys for AWM Projects Introduction A performance baseline is information ­ Box 1: AWM projects lack baseline qualitative or quantitative that provides information data at the beginning of, or just prior to, the monitoring period. The baseline is The IEG review of ten year's investment in the sector used as a starting point, or `benchmark', shows that only a third of the projects in a random against which to assess change over time sample had a baseline before the project started, and and thus monitor project performance. less than half attempted to establish a baseline during Baselines are crucial in establishing where the project. When no baseline was specified at the a project stands at a point in time relative project design stage it was more likely to be to the outcome it is trying to achieve. Yet, established very late. Poorly designed and planned M&E almost always led to late attention to establishing many AWM projects either do not a baseline. On the contrary, the major determinants of establish baselines or do this too late (see early baseline creation were identified to be: (a) a Box 1). clear definition of the project outputs and outcomes expected and (b) discussion of the baseline and its Building baseline information use in the PAD. For each indicator in the Results Source: World Bank IEG, 2006. Framework in the Project Appraisal Document (see GN1 and GN13), the project team is required to indicate baseline information, targets, data collection methods and responsibilities, and uses of M&E information (see GN6 for guidance on data collection). The added requirement for IDA projects is that all baseline information should be indicated in the first ISR report. Furthermore, for evaluation purposes the project team often identifies additional data needs for which baseline data collection will be required (RN2 provides an example of the baseline data collection process for the Maharashtra Water Sector Improvement Project, India). Baseline information can be collected as part of project preparation and appraisal. Indeed this may often be desirable to ensure that the baseline information is compiled before the project commences and starts to have an influence on the project area and target population. Baseline data collection can also be a very demanding exercise for newly established project management and M&E teams. This is particularly true if personnel have been recruited for the project and may be lacking in relevant training and experience. Retroactive financing of M&E costs may be needed to cover baseline study costs. Sources of baseline data include the following: · Baseline data may be available from secondary sources. For example, a national statistical agency may have data on indicators commonly used for M&E of AWM projects such as crop yields and market prices. · Some baseline data might be collected by social and environmental impact assessments, and by other feasibility studies carried out during project preparation. · Many indicators will require a survey specifically to collect baseline data. Although this could be carried out as part of project preparation, it will more commonly be implemented soon after the commencement of the project. The PAD should plan and provide for baseline data collection at the start of project implementation. This may require use of an GN7 Baselines 1 Toolkit for monitoring and evaluation of AWM Guidance Note 7 projects Design and implementation of baseline surveys for AWM Projects external agency or consultants if the project management agency initially lacks the expertise and experience. Box 2 provides further practical guidance on what to do in situations where baseline data is missing or difficult to obtain. Box 2: What to do without a baseline? Quite frequently, projects and programmes have to start without baseline data at hand. In this situation, the project team can consider the following alternatives. 1. Track changes with (inside the project area) and without a project presence (outside the project area) if comparable areas can be found. 2. Higher level evaluation techniques can be applied such as random sampling of the project and control area and use of propensity score matching ­ see the example of Vietnam in Box 3 in GN9. 3. Use the initial round of monitoring and first measurements as a starting point (e.g. production during the first implementation season). 4. Look for existing local and national databases of agricultural and other relevant indicators and statistics. In some countries authorities regularly collect data on variables such as farm and off farm income, production and use of inputs. 5. Carry out interviews with local key informants (e.g. village elders and water user association leaders), or focus groups (e.g. women's and savings groups and producer associations), for example, to detect changes in the availability and use of irrigation water, and historical trends and timelines for indicators such as crop yields 6. Use other existing records, such as: · data collected by other donor projects or NGOs; · records compiled by agricultural extension agencies and the Irrigation Department on their activities, such as training events and maintenance programmes; · data on input or output sales from government and private trader records. Source: adapted from Rajalahti and Woelcke, 2005 What baseline information should be collected? Baseline data should be collected for each indicator listed in the Results Framework, and as needed for the purposes of project evaluation. Information overload should be avoided as excess and irrelevant data is unlikely to be used. RN3 presents a prototype baseline survey established by the South Asia Rural Development Unit in the World Bank. It lists the main topics for which a project team may want to collect information, including: · a household and plot level survey which includes questions on irrigation, agricultural activities, income and assets, food security, credit, land tenure, livestock, extension, and marketing. · A scheme level survey which collects information by irrigation and drainage scheme. · A community and water user association survey, intended for leaders of communities and which includes questions on respondent's characteristics, a general community profile, access to education and healthcare, prices of agricultural inputs and outputs including land, Water User Associations, agricultural extension, employment, irrigated crops and access to credit. · A survey of the state level Irrigation Department which oversees service delivery and performance. While the survey contains a number of standard questions meant to evaluate social, economic and agricultural indicators of household and community welfare, it is unique in that it attempts to gather detailed plot level irrigation information, to assess the role of institutions in the successful functioning of irrigation systems and to assess the environmental conditions present in irrigation GN7 Baselines 2 Toolkit for monitoring and evaluation of AWM Guidance Note 7 projects Design and implementation of baseline surveys for AWM Projects districts. As a prototype or model survey, it needs to be customized and adapted to each project's situation. Baselines and targets Because the success of a project will be, in part, measured by comparing target values with achieved or actual values, setting target values is a sensitive issue that needs care (see GN3 on indicators). One method to establish targets is to start with the baseline indicator level and to use historical data or another estimate of the rate of change to set the desired level of improvement needed to arrive at the performance target (taking into account available funding and other resources over the target period). Although it is tempting to set targets relatively low to assure they are reached, it is also important to set the targets high enough to create momentum in project implementation and the desired level of achievement. A comparison of baseline to achieved values can be carried out by: · comparing the situation before the project started with the situation after it started. This requires understanding which factors influenced the outcome. · tracking changes with (inside the project area) and without a project presence (outside the project area). This requires finding comparable areas. · comparing the difference between similar groups (inside the project area), i.e. between a project group and a comparable group within the project area but relatively isolated from the project's influence. This requires finding comparable groups within the same area. (See GN9 for further discussion of impact evaluation frameworks and approaches). Further Reading Rajalahti, R., J. Woelcke, et al. 2005. Monitoring and Evaluation for World Bank Agricultural Research and Extension Projects: A Good Practice Note. Agricultural and Rural Development Discussion Paper 20. Washington D.C.: World Bank. Available at http://intresources.worldbank.org/INTARD/Resources/M_E_WB_AgExtensionProjects.pdf World Bank South Asia Rural Development Unit. 2004. Monitoring and Evaluation of Water and Irrigation Projects ­ Prototype Baseline survey in South Asia. Washington D.C.: World Bank. Available at http://wbln0023.worldbank.org/Internal/SAR/southasiasectors int.nsf/41ByDocName/MEWaterandIrrigation GN7 Baselines 3 Toolkit for monitoring and evaluation of AWM Guidance Note 8 projects Remote Sensing in M&E of AWM Projects Guidance Note 8 Remote Sensing in Monitoring and Evaluation of Agricultural Water Management Projects Introduction With advances in technology and the wider availability of remote sensing data the use of remote sensing for monitoring and evaluation of irrigation and drainage system performance becomes technically and economically feasible. In certain aspects, remote sensing offers considerable advantages over traditional monitoring and evaluation techniques. Remote sensing is usually coupled with a Geographic Information System (GIS) to integrate, process and present data (see below). There are also a number of other spatial technologies that are gaining momentum, in particular Global Positioning System (GPS) and use of free mapping software. GPS enables to geo reference data, and it can be used together with a computer handheld unit to directly record information with its geographical coordinates on the field. Free mapping software such as Google Earth are a simple way to display geo referenced data and can be used where a full GIS is not developed. Finally, more sophisticated systems such as spatial spreadsheets are introduced in environments where suitable expertise is available ­ in particular India. Spatial spreadsheets are a means of organizing large amounts of spatial data, to quickly formulate queries on that data, and to propagate changes in the source data to query results on a large scale. Background The possibilities for using remote sensing for monitoring the performance of I&D systems gained strength in the early 1990s, since which there have been significant developments. Early applications focussed on mapping of irrigated crop lands; this application has developed such that it is now possible to map the irrigated area from different water sources ­ surface, groundwater, from small reservoirs. It is has also been possible for some while to map different crop types and degrees of crop stress; an important more recent development has been the ability to determine actual and potential evapotranspiration from remotely sensed images. There are a number of sources of remotely sensed data, a commonly used source is the National Oceanic and Atmospheric Administration (NOAA) satellite which is equipped with an Advanced Very High Resolution Radiometer (AVHRR) which measures red, near infrared and thermal infrared radiation. In India data have been obtained from Landsat 5 and Indian remote sensing satellites (Thiruvengadachari and Sakthivadivel, 1997). Remote sensing information is usually used in combination with GIS. GIS is defined as a computerized system that permits storage, processing and display of data, with the information linked to a system of geographical co ordinates. It provides a way of assembling any data that GN8 Remote sensing 1 Toolkit for monitoring and evaluation of AWM Guidance Note 8 projects Remote Sensing in M&E of AWM Projects can be referred directly or indirectly to a geographical location (see Figure 1). It allows integration of different types of data from various sources (qualitative and quantitative, socioeconomic, environmental, etc), together with their analysis. It also allows for their presentation in a user friendly format, as the use of maps is a good way to have an overview of project progress and to communicate with stakeholders. In the Karnataka Watershed Project in India for example, GIS combined with remote sensing is extensively used to produce linked maps on soils, ground water potential zones, drainage, transport network and settlement, land use and land cover. These thematic maps are then used at the community level, including individual farmers' plots, to report on activities in the micro watershed and discuss project progress (see RN6).1 Figure 1: Geographic Information System Maps Numerical data Maps GIS Numerical data Satellite photos Source: Asian Development Bank, 1991 Examples of application Three examples of the application of remote sensing and GIS for performance assessment of I&D systems are provided to show the range of applications. More detailed information, particularly on the techniques used to interpret the remotely sensed data, is provided in the references from which this information was obtained. Case Study 1 - Bhadra Project, India (Thiruvengadachari and Sakthivadivel, 1997) The Bhadra Project was one of the first irrigation schemes to be taken up by the Indian Government/World Bank funded National Water Management Project (NWMP), established to raise agricultural productivity and farm income by providing a more equitable, predictable and reliable irrigation service. The Bhadra project is located in the State of Karnataka and comprises a dam with a gross storage capacity of 2,025 million m3, a left bank irrigation and drainage system serving 8,290 ha and a right bank system serving 92,360 ha. The predominant crop is rice. The purpose of the NWMP during the period 1988 94 was to develop and implement an appropriate water management policy on the project. This was assisted by the study carried out by the International Water Management Institute (IWMI) to monitor irrigated area, cropping pattern and rice yield during the rabi season from 1986 87 to 1993 942. Data were collected by 1 See the World Bank GIS website for more information at http://go.worldbank.org/I21YX64I50. See also ESRI support center at http://support.esri.com/ for more information on ArcGIS software. 2 Out of the possible eight seasons six seasons of data were analyzed in full. GN8 Remote sensing 2 Toolkit for monitoring and evaluation of AWM Guidance Note 8 projects Remote Sensing in M&E of AWM Projects Landsat 5 and Indian remote sensing satellites and transmitted to the Indian Earth Station at Shanagar and processed at the National Remote Sensing Agency facilities in Hyderabad. The data analysis involved: · Mapping irrigated crop areas and discriminating rice from other crop areas; · Mapping spatial variability of the rice transplanting period across the command area; · Estimating rice yield using spectral index models; · Measuring yield from satellite derived data on the condition of the rice crop, using selected representative sample areas; · Evaluating the impact of reported waterlogging on rice productivity; · Radiometric normalisation between satellites, sensors, and acquisition periods; · Comparative evaluation of satellite data of different spatial resolution; · Integrating cadastral maps with satellite data. In association, a GIS was developed to evaluate system performance. The study had the following findings: i) It was possible to distinguish between rice and non rice crops, with field checks showing accuracies of remotely sensed data in the range 90 95%. ii) The spatial variability of the rice calendar was relatively easily determined from the normalised difference vegetation index (NDVI) for the peak greenness stage (corresponding to the heading stage of rice). iii) A rice yield model with a statistical significance of 0.76 when compared with field crop cuttings was developed, enabling rice yields and rice production to be estimated for the entire scheme; iv) Accompanied by the GIS the study was able to produce colour coded maps for the entire command area. These maps showed (a) the irrigation intensity, (b) the percentage of rice area cultivated, (c) rice yield (tonnes/ha) and (d) rice production (tonnes) before (1986/87) and after (1992 93) NWMP interventions. These maps were accompanied by tables providing the data for each canal command area, from distributary up to sub division and division level; v) The analysis showed that (Table 1): the irrigation intensity increased in every part of the command area (from 76% to 91%); the cultivated area of rice increased (from 56% to 69%); the yield of rice increased (from 3.8 t/ha to 4.9 t/ha), as did the total production. In addition the average depth of water applied decreased from 1.06 m depth to 0.86 m depth. vi) The cost of applying remote sensing for the 100,000 ha Bhadra project was found to be of the order of US$ 0.10/ha for each irrigation season (1994 95 prices). This estimate includes the cost of satellite data, analysis and statistical generation. When carried out with the crop inventory, unit costs for monitoring waterlogging and salinity worked out at US$0.05/ha and US$0.02/ha respectively. The satellite remote sensing application cost less than 1% of the annual management, operation and maintenance costs. GN8 Remote sensing 3 Toolkit for monitoring and evaluation of AWM Guidance Note 8 projects Remote Sensing in M&E of AWM Projects Table 1: Changes in key indicators over the period 1986-87 to 1993-94 Yeara Irrigated crop Rice area Depth of Area Rice output per unit of area water irrigated per land water applied unit of water (ha) (%) (m) (ha/ha m) (t/ha) (kg/m3) 1986 87 73,529 56 1.06 0.94 3.8 0.3 1989 90 67,366 51 1.04 0.96 5.4 0.4 1992 93 88,424 69 0.80 1.25 4.7 0.5 1993 94 84,412 69 0.86 1.16 4.9 0.5 Source: Thiruvengadachari and Sakthivadivel, 1997 a 1986 87: pre NWMP and no cut off; 1989 90: 25% of command area cut off from irrigation; 1992 93 and 1993 94: without cut off. The overall conclusion from the study was that using remote sensing to monitor and evaluate the performance of the project was feasible and cost effective, and had particular application to large scale systems. In addition the integration of remote sensing with a GIS facility significantly enhanced the performance evaluation and the diagnostic analysis capabilities. Case Study 2 - Sirsa Irrigation Circle, India (W.Bastiaanssen, D.J.Molden, S.Thiruvengadachari, A. Smit, L. Mutuwatte and G. Jayasinghe, 1999) Two studies were carried out by IWMI on the Bhakra Irrigation System in the State of Haryana in northern India. One was for the entire command area of 1.3 million ha, the other for a section of the command area totalling nearly 483,000 ha gross. The study of the entire system is detailed in Sakthivadivel et al, 1999, whilst the more detailed study of the Sirsa Irrigation Circle is summarised here. The key feature of the Sirsa Irrigation Circle study was that it combined information from hydrological modelling, field data and satellite remote sensing in a GIS format that facilitated a view of the irrigation and drainage network that had not been possible before. The remote sensing produced a complete picture of the agricultural productivity for the season under study (rabi season, 1995/96), the modelling allowed detailed study of the hydrological processes in the region and the GIS enabled the integration and analysis of the data collected. The following findings were made: i) Through remote sensing, information on wheat yield and cropping intensity was determined. The wheat yield was found to be relatively uniform throughout the area, but the cropping intensity for wheat was found to be highly variable; ii) Using remote sensing data and modelling the average wheat yield was found to be 3.6 tonnes/ha, with an average water consumption of 428 mm, giving productivity of water of 0.88 kg/m3, equivalent to US$0.14/m3 at the 1996 international wheat price (US$163/tonne). The analysis allowed the segregation of the water supply systems, and identified that tubewell irrigation gave a lower water productivity than surface irrigation; iii) The location of higher wheat intensities and wheat yields was identified; generally these were in the vicinity of main and distributary canals. As one moved away from the main canals and distributary canals, the wheat cropping intensity declined, implying that the warabandi system of water distribution was not functioning as well as it might. At these more distant locations tubewell irrigation was prominent; iv) Overall the depletion fraction of gross inflow was estimated at 82%, that is the crops consumed 82% of the total water supply entering the system (irrigation water, rainfall and groundwater). This is a high percentage, with reuse of seepage water by tubewells being a major factor in the process; v) The analysis enabled the categorisation of the command area into 5 hydrologic classes, each with different patterns of irrigation, inflow, tubewell use and groundwater build up (Table 2). As a consequence agricultural practices and performance were found to be quite different across these hydrological classes. GN8 Remote sensing 4 Toolkit for monitoring and evaluation of AWM Guidance Note 8 projects Remote Sensing in M&E of AWM Projects This knowledge could enable system managers to adopt different management regimes to suit these different classes; vi) The analysis allow a salt balance to be carried out, and identified a potentially serious issue with sustainability of the system, with salt levels estimated to be increasing by 1.81 tonnes/ha annually. The study extended the use of remote sensing from that outlined in the first case study by estimating the productive use of water for the project as a whole. It also allowed quantification of the groundwater and salinity hazards, and highlighted a serious issue with salt build up. Table 2: Performance information for the five hydrological classes of the Sirsa Irrigation Circlea Class Area Irrigation Evapotranspirati Depleted Intensity Wheat Water productivity Annual increases on Fraction yield for wheat Canal Tube- Totalb Wheat Irrigation Wheat Physical Economic Ground Salt well water storage (km3) (mm) (mm) (mm) (mm) (%) (%) (%) (t/ha) (kg/m3) (US$/m3) (mm) (t/ha) 1 86 505 30 541 372 0.74 0.72 0.51 3.45 0.93 0.15 148 2.09 2 76 506 59 651 403 0.90 0.77 0.67 3.91 0.97 0.16 58 1.67 3 99 397 235 753 471 0.90 0.76 0.68 3.92 0.83 0.14 78 1.91 4 118 271 46 385 308 0.78 0.63 0.45 3.49 1.13 0.18 97 1.80 5 109 252 464 64 307 0.91 0.74 0.56 3.63 1.18 0.19 43 1.40 aRabi 1995/96, except noted bAnnual average, 1977-90 Case Study 3 - Nilo Coelho, Brazil (W.G.M. Bastiaanssen, R.A.L Brito, M.G. Bos, R.A. Souza, E.B. Cavalcanti and M.M. Bakker, 2001) This study moves the use of remote sensing on significantly from the previous work by using remotely sensed data to determine a number of performance indicators related to crop water use. Public domain satellite data available over the Internet were used to calculate actual and potential crop evapotranspiration, soil moisture and biomass growth on a monthly basis for the 13,000 ha Nilo Coelho irrigation scheme, in the Pernambuco region, Brazil. Satellite interpreted raster maps3 were merged with vector maps4 of the irrigation water delivery system and monthly values of key irrigation performance indicators for the 31 service units in the pressurized Nilo Coelho scheme were determined. The crops are fruit trees grapes, mangoes, bananas, guava and acerola, grown on soils with a high (70%) sand fraction requiring sophisticated irrigation systems and careful management to maintain the required soil moisture status. The main data used in the study were downloaded free of charge from the National Oceanic and Atmospheric Administration (NOAA) website5. An area 200km by 300km was selected and data downloaded for each month during the course of the study (August 1998 to July 1999). The NOAA satellite data were used to determine the monthly values for actual evapotranspiration using the Surface Energy Balance Algorithm for Land (SEBAL) developed by Bastiaansen et al (1998) and Bastiaanssen (2000). Other indicators (Table 3) were determined from the remote sensed data, and combined with data collected from the extensive flow-measuring network on the scheme to allow comparison of actual scheduled supplies with estimated irrigation needs. 3A raster map is a map based on a data file composed of a (generally) rectangular grid of pixels, with a palette of colours available for each pixel. 4A vector map is a map drawn based on points, lines, curves and polygons. In the computer database these geometrical shapes are represented by mathematical equations. 5http://www.noaa.gov GN8 Remote sensing 5 Toolkit for monitoring and evaluation of AWM Guidance Note 8 projects Remote Sensing in M&E of AWM Projects Table 3: Formulation of irrigation performance indicators applied in the current study in the Nilo Coelho scheme Irrigation performance indicator Mathematical expression Unit Relative water supply (Pgross+ Vc) / ETpot dimensionless Overall consumed ratio (ETpot Pe)/Vc dimensionless Depleted fraction ETact/(Pgross+ Vc) dimensionless Crop Water Deficit ETpot­ ETact mm / month Relative evapotranspiration ETact / ETpot dimensionless Relative soil wetness / FC dimensionless Biomass yield over irrigation supply Bio / Vc kg / m3 Source: W.G.M. Bastiaanssen, R.A.L Brito, M.G. Bos, R.A. Souza, E.B. Cavalcanti and M.M. Bakker, 2001 Note: Pgross = gross precipitation (mm/month), Pe = effective or net precipitation (mm/month), Vc = water delivery from the (river or) reservoir (mm/month), ETact = actual evapotranspiration by irrigated crops (mm/month), ETpot = potential evapotranspiration by irrigated crops (mm/month), =Volumetric soil water content in the rootzone (cm3/cm3) FC = Volumetric soil water content at field capacity (cm3/cm3) Bio = crop growth expressed as above ground dry bio mass growth (kg/ha per month), The analysis of the data identified several areas where management could improve the performance of the system, principally by reducing the amount of water pumped in March/April to make better use of precipitation. It was estimated that savings of 25% in the irrigation water diversions could be made. Through the spatial mapping, areas of excessive and deficit water supply were identified, and further diagnostic studies recommended for specific locations. The cost of the remote sensing exercise amounted to US$ 0.80/ha, with a further US$ 0.20/ha to cover the costs of establishing the associated GIS system. The total cost of US$ 1.0/ha is about 4% of the total scheme management, operation and maintenance costs of US$ 24/ha. The values of the performance indicators defined above were calculated for each month and are presented and briefly discussed below: · The low value of the Depleted Fraction (around 0.60, see Figure 2) shows that not all the precipitation water is consumed. The Depleted Fraction is lowest in April when the drainage rate reaches its peak, this is also the period of high precipitation (Figure 3). Reducing pumping of irrigation water at this time would reduce pumping costs, conserve water and make better use of the available precipitation; GN8 Remote sensing 6 Toolkit for monitoring and evaluation of AWM Guidance Note 8 projects Remote Sensing in M&E of AWM Projects · From February to July the Relative Water Supply (Figure 4) averages around 1.4, with a peak in April of 1.7, indicating an excess of supply over demand, supporting the information provided in Figure 2. At this time the Overall Consumed Ratio, which is the inverse of the Relative Water Supply, is at its lowest value; · The Relative Soil Wetness, a measure of the actual soil moisture content relative to field capacity, ranges between 1 and 1.3, indicating that the soil moisture is maintained at adequate levels throughout the year; · Despite the Relative Soil Wetness being above 1.0 and the Relative Water Supply being around 1.0 in December and January the Relative Evapotranspiration and the Crop Water Deficit indicators (Figure 5) show that the actual evapotranspiration is less than the potential. This is thought to be because of plant physiological effects rather than soil GN8 Remote sensing 7 Toolkit for monitoring and evaluation of AWM Guidance Note 8 projects Remote Sensing in M&E of AWM Projects physical processes. This is the hottest time of the year with maximum air temperatures of 40oC possibly inducing closure of the crop stomata; · The biomass development is presented in Figure 6, showing that the fastest plant growth occurs from February to April when the rains starts, the humidity is high and the air temperature ideal for crop growth. At this time biomass production is reaching 3,500 kg/ha/month, the result of conducive water supply and solar radiation conditions. The figures above present the temporal situation for the scheme as a whole. Figure 7 presents the spatial situation for each of the 31 lateral units served by pumping stations. Determining these figures for each month adds the temporal dimension to the analysis. The variation in the parameters shown can be readily seen from this form of presentation, enabling identification of well and less well performing areas. GN8 Remote sensing 8 Toolkit for monitoring and evaluation of AWM Guidance Note 8 projects Remote Sensing in M&E of AWM Projects Figure 7: Part A: Relative Water Supply for May 1999 (dimensionless); Part B: Crop Water Deficit for January 1999 (mm/month); and Part C: Biomass Yield over Irrigation Supply for February 1998 (kg/m3) GN8 Remote sensing 9 Toolkit for monitoring and evaluation of AWM Guidance Note 8 projects Remote Sensing in M&E of AWM Projects Conclusions The three case studies have illustrated the type of performance data that can be obtained using remote sensing in conjunction with a GIS. The key factor is that the use of remote sensing enables forms and scales of assessment that would otherwise not be possible. A supporting factor is that remote sensing can provide valuable information on system performance at a reasonable cost. The three case studies have shown that remote sensing can accurately quantify leading indicators, including crop type and area, crop yield, crop production, crop water demand and use. A valuable capability is the ability to measure the potential and actual evapotranspiration, and to identify areas where the crop is under stress. With supplementary data a wider set of indicators are available, including those for waterlogging and salinity. Remote sensing provides spatial and temporal coverage on a scale that is impossible to replicate with other means of performance monitoring. The fact that remotely sensed data are collected and stored on a regular basis means that trends can be studied over several years, the data collection is not bounded by the need to take special measures to obtain the data (such as might be the case with seasonal crop cuttings). As such remote sensing offers a valuable tool for analysis of the pre project situation, which could be carried out as part of the post project analysis and assessment. There are, however, some limitations to the use of remote sensing (Bastiaanssen et al, 2001): · It requires specialist expertise to carry out; · It does not explain the causes, it only measures the effects of land surface processes; · High resolution images may not always be available due to cloud cover; · High resolution images are often not instantly available for general purpose use, there can be delays of a month or more. They are also expensive, US$ 600 per scene; · Low resolution images can be obtained on a daily basis but the resolution is only 1.1 km. This is too coarse for interpretation of crop types; · Remote sensing provides a regional scale overview and requires validation of spatially distributed parameters derived from satellite measurements. This is difficult and requires local field studies and consistency checks. References and Further reading Asian Development Bank. 1991. Remote sensing and geographic information system for natural resource management. Environment Paper 9. Manila: Asian Development Bank. Bastiaanssen, W. G. M., T. Van Der Wal, and T.N.M. Visser. 1996. Diagnosis of regional evaporation by remote sensing to support irrigation performance assessment. Irrigation and Drainage Systems, 10(1):1 23. Bastiaanssen, W. G. M., H. Pelgrum, J. Wang, Y. Ma, J.F. Moreno, G.J. Roerink, and T. Van Der Wal. 1998. A remote sensing surface energy balance algorithm for land (SEBAL): 2. Validation. Journal of Hydrology, 212/213:213 229. Bastiaanssen, W. G. M., M. Menenti, R.A. Feddes, and A.A.M Holstag. 1998. A remote sensing surface energy balance algorithm for land (SEBAL): 1. Formulation. Journal of Hydrology, 212/213:198 212. GN8 Remote sensing 10 Toolkit for monitoring and evaluation of AWM Guidance Note 8 projects Remote Sensing in M&E of AWM Projects Bastiaanssen, W. G. M., S. Thiruvengadachari, R. Sakthivadivel, and D.J. Molden. 1999. Satellite remote sensing for estimating productivities of land and water. International Journal of Water Resources Development, 15(1/2):181 196. Bastiaanssen, W. G. M. 1999. Monitoring water resources and agro ecosystem productivity from space. International Archives of Photogrammetry and Remote Sensing, 32(7C2):38 43. Bastiaanssen, W. G. M., D.J. Molden, S. Thiruvengadachari, A.A.M.F.R. Smit, L. Mutuwatte, and G. Jayasinghe. 1999. Remote sensing and hydrologic models for performance assessment in Sirsa Irrigation Circle, India. IWMI research report 27, Colombo, Sri Lanka: IWMI. v, 29p. Bastiaanssen, W. G. M., D.J. Molden, and I.W. Makin. 2000. Remote sensing for irrigated agriculture: Examples from research and possible applications. Agricultural Water Management, 46(2):137 155. Bastiaanssen, W. G. M. 2000. Satellite observations of international river basins for all. International Archives of Photogrammetry and Remote Sensing, 33(B6):319 331. Bastiaanssen, W. G. M. 2000. SEBAL based sensible and latent heat fluxes in the irrigated Gedez Basin, Turkey. Journal of Hydrology, 229(1 2):87 100. Bastiaanssen, W. G. M., R.A.L. Brito, M.G. Bos, R. Souza, E.B. Cavalcanti, and M.M. Bakker. 2001. Low cost satellite data applied to performance monitoring of the Nilo Coelho irrigation scheme, Brazil. Irrigation and Drainage Systems, 15, 53 79, Kluwer Academic Publishers, Dordrecht, The Netherlands. Droogers, P., and W. G. M. Bastiaanssen. 2000. Evaporation estimates using a combined hydrological model and remote sensing approach. Geophysical Research Abstracts, Vol.2: Droogers, P., W. G. M. Bastiaanssen, M. Bayazgul, Y. Kayam, G.W. Kite, and H. Murray Rust. 2000. Distributed agro hydrological modeling of an irrigation system in Western Turkey. Agricultural Water Management, 43(2):183 202. Fabbri, P.J., B.G.H. Gorte, J.W. Van Groenigen, F.D. Van der Meer, and A. Saldana. 1998. Integrating spatial statistics and remote sensing. International Journal of Remote Sensing, 19(9):1793 1814. Hurk, van den B. J. J. M., W. G. M. Bastiaanssen, H. Pelgrum, and E. Van Maijgaard. 1997. A new methodology for assimilation of initial soil moisture fields in weather prediction models using meteostat and NOAA data. Journal of Applied Meteorology, September:1271 1283. Kite, G., and P. Droogers. 2000. Comparing estimates of actual evapotranspiration from satellites, hydrological models, and field data: A case study from Western Turkey. IWMI research report 42, Colombo, Sri Lanka: IWMI. v, 32p. Roerink, G. J., W. G. M. Bastiaanssen, J. Chambouleyron, and M. Menenti. 1997. Relating crop water consumption to irrigation water supply by remote sensing. Water Resources Management, 11(6):445 465. Sakthivadivel, R., U.A. Amerasinghe, and S. Thiruvengadachari. 2001. Using remote sensing techniques to evaluate lining efficacy of watercourses. IWMI research report 46, Colombo, Sri Lanka: IWMI. v, 29p. GN8 Remote sensing 11 Toolkit for monitoring and evaluation of AWM Guidance Note 8 projects Remote Sensing in M&E of AWM Projects Sakthivadivel, R., S. Thiruvengadachari, U.A. Amerasinghe, W. G. M. Bastiaanssen, and D. Molden. 1999. Performance evaluation of the Bhakra Irrigation System, India, using remote sensing and GIS techniques. IWMI research report 28, Colombo, Sri Lanka: IWMI. v, 22p. Schakel, J. K., and W. G. M. Bastiaanssen. 1997. Regional water and salt balances obtained from GIS and hydrological models. ITIS, 4(1):15 17. Thiruvengadachari, S., and R.S. Sakthivadivel. 1997. Satellite remote sensing for assessment of irrigation system performance: A case study in India. IWMI Research Report 9, Colombo, Sri Lanka: IIMI. v, 23p. Wang, J., W. G. M. Bastiaanssen, Y. Ma, and H. Pelgrum. 1998. Aggregation of land surface parameters in the oasis desert systems of north west China. Hydrological Processes, 12:2133 2147. Websites ESRI support center: http://support.esri.com/ World Bank GIS: http://go.worldbank.org/I21YX64I50 GN8 Remote sensing 12 Toolkit for monitoring and evaluation of AWM Guidance Note 9 projects Impact Evaluation of AWM projects Guidance Note 9 Impact Evaluation of AWM Projects The purpose of this guidance note is to explain the key concepts and methods of impact evaluation and how they can be best applied to the monitoring and evaluation of AWM projects. Monitoring and Evaluation: distinct yet complementary Evaluation is the systematic and objective assessment of the design, implementation and results of an on going or completed project, program or policy. The aim is to assess the relevance and fulfilment of the project's objectives, its effectiveness, efficiency, developmental impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision making process of both recipients and donors. An impact evaluation focuses specifically on the developmental changes that have occurred and to what they can be attributed. Monitoring and Evaluation are distinct yet complementary (Table 1). Monitoring provides information on the progress and outcomes of a policy, program, or project at any given time, relative to its respective targets and outcomes, and is largely descriptive. Evaluation assesses outcomes and impacts relative to expectations and to the context in which implementation has occurred, and provides evidence of why targets have or have not been achieved. It other words its seeks to address the issue of causality, and to be able to attribute observed change either to the developmental intervention or to other factors. Table 1: Complementary roles of results-based Monitoring and Evaluation Monitoring Evaluation Clarifies program objectives Analyzes why intended results were or were not achieved Links activities and their resources to Assesses specific causal contributions of objectives activities to results Translate objectives intro performance Examines implementation process indicators and sets targets Routinely collects data on these indicators, Explores unintended results compares actual results with targets Reports progress to managers and alerts Provides lessons, highlights significant them to problems accomplishment or program potential, and offers recommendations for improvement Source: Kusek and Rist, 2004. It has been observed that only a small proportion of World Bank AWM projects have included a rigorous evaluation framework in their provisions for M&E (Box 1). Impact evaluation is, however, crucial in AWM projects to: · Provide an objective basis for assessing the performance of policies, programs, projects and processes. · Demonstrate impacts and the relevance of investment in AWM projects. · Help provide shared accountability for the achievement of the World Bank's objectives. · Assist in `managing by results' by testing the project approach, modifying design and improving effectiveness during implementation. · Improve policies, programs, and projects by identifying and disseminating the lessons learned from experience and by making recommendations drawn from evaluation findings. GN9 Impact evaluation 1 Toolkit for monitoring and evaluation of AWM Guidance Note 9 projects Impact Evaluation of AWM projects Box 1: A lack of a rigorous evaluation framework in AWM projects The IEG review of a ten year period of investment in the AWM sector notes that a rigorous evaluation framework is often missing in AWM projects making robust attribution of benefits difficult. Only 11 percent of projects reviewed in the sample were designed to have the tools that would allow rigorous impact assessment: this included well defined output and outcome indicators, good baselines and independent control groups unaffected by project interventions that would allow the counterfactual to be determined. Another 41 percent would allow determination of what happened before and after project implementation, but not a robust attribution of observed changes. Slightly fewer than half the projects did not have any means of verifying project impacts ­ no surveys or baselines ­ even though more than two thirds of them included outcome or impact indicators. This raises questions about the robustness of the conclusions drawn by most projects that assert improvements in observed production and farmers' incomes and attribute it to the Bank's project level interventions. Source: World Bank IEG, 2006 Forms of evaluation in World Bank projects Procedures for evaluation at the World Bank take threes main forms. Self evaluation, by the unit responsible for the particular program or project: An interim evaluation or mid term evaluation can be undertaken by the project management during implementation as a first review of progress and a prognosis of the likely effects of the project. It is intended to identify project design problems, and is essentially an internal activity undertaken for project management. At the end of the loan disbursement period, the Task Team submits the Implementation Completion and Results (ICR) Report to the Bank Board of Executive Directors. It includes Principal Performance Ratings, Achievement of Development Objective and Outputs (see GN13), and calls, as a minimum, for the comparison of end of project outcome and impact indicators with baseline data. When project M&E has been inadequate, Task Teams experience obvious difficulty in preparing the ICR Report. Independent evaluation: The World Bank's Independent Evaluation Group (IEG) reviews ICR Reports, validates the self rating, and selects projects for auditing (see GN13). Impact evaluation (IE): As the World Bank is implementing the "results agenda", it is encouraging Task Teams and clients to provide for and conduct impact evaluations of projects. A Development Impact Evaluation Taskforce is in charge of the Development Impact Evaluation Initiative, and has made funding available to design evaluation frameworks6. As part of the World Bank's effort to support and scale up the work on impact evaluations, Impact Evaluation was established as a new product line in 2005, under the AAA umbrella (see GN13). Overarching themes in impact evaluation An impact evaluation assesses changes that can be attributed to a particular project, program or policy. Many governments, institutions, and project managers are reluctant to carry out impact evaluations because they are deemed to be expensive, time consuming, and technically complex, and because the findings can be politically sensitive. Yet with proper and early planning, the support of policymakers, and a relatively small investment compared with overall project cost, a rigorous evaluation can be very powerful in assessing the appropriateness and effectiveness of development interventions. 6 The World Bank identified several bottlenecks that limit its ability to conduct impact evaluations at the necessary scale and with the needed continuity: insufficient resources, inadequate incentives, and, in some cases, lack of knowledge and understanding. To address these bottlenecks, the Development IMpact Evaluation (DIME) Initiative is a Bank wide collaborative effort under the leadership of the Bank's Chief Economist that is oriented at: (1) increasing the number of Bank projects with impact evaluation components, particularly in strategic areas and themes; (2) increasing the ability of staff to design and carry out such evaluations, and (3) building a process of systematic learning for effective development interventions based on lessons learned from completed evaluations GN9 Impact evaluation 2 Toolkit for monitoring and evaluation of AWM Guidance Note 9 projects Impact Evaluation of AWM projects Box 2: Some ideas for increasing government buy in to an impact evaluation -Work with the government counterpart from the beginning. Spend a significant amount of time explaining the motivation/usefulness and methods. -Understand who the client(s) are and their needs. Develop an impact evaluation which answers the clients' priority questions. -Understand the other interested parties ­ their position on the issues around the evaluation, how they might affect the evaluation, and how they may react to the results. Spend time with these groups explaining the motivation/usefulness and methods. -Identify a champion within government. This advocate can help with arguing for the evaluation initially, making sure it is well executed, and helping to ensure that the results get used. -Get local researchers involved. A good place to start are those involved in the PRSP, they are likely to have government access and understand the issues and context -Building impact evaluation capacity (especially in terms of analytical work) within government will not only provide benefits in terms of evaluation quality (e.g. a better understanding of the institutional context and implementation) but will also help build a constituency for future evaluations. -If possible, get high profile Bank involvement to support this in dialogue with the government. -Facilitate access to high quality human resources ­ make it as easy as possible for the counterparts to deliver a quality impact evaluation. If you cannot identify the resources on your own, refer to some of the sources in the support sections above. -Do some homework on costs and funding options before talking with government about the evaluation. Think about possible positive spillovers that the evaluation will generate (e.g. data on certain populations of interest). -If applicable, provide interim result to the client(s). In addition to keeping the client up to date on the direction of the evaluation, these can be used for mid term policy corrections if they are warranted. When the results are ready -Make caveats clear up front and repeat ­ this will be important in ensuring that results are interpreted correctly. -Timing of release is important. Understand the domestic political/decisionmaking cycle and time the release of your results in order to maximize the impact on policy. -Don't surprise the key players. It's advisable to give them a preview of the results before broad dissemination. -Provide constructive policy guidance. Identifying the magnitude of the impact and the effects on different groups is only the first step. Discussing the implications for policies that stem directly from the evaluation results will assist government in identifying the best ways to put the evaluation results into action. -Communicate in different forms (e.g. technical, less technical) to different audiences, as much as possible both inside and outside government. Source: World Bank, 2006b. Some overarching themes should be addressed when planning and implementing an impact evaluation: · Integration with the project. A good impact evaluation is not a free standing exercise. The Lead Evaluator (LE) should become intimately familiar with the project, the country and institutional context, the design options that are being considered, and the details of implementation. The TTL and client also need to support the purpose and logic of the evaluation, and to understand the key project design and implementation elements that are critical for the success both of the evaluation and the project. · Relevance. The evaluation has to be structured in such a way as to be able to answer the questions that are important for the clients, and to answer them at the time when feedback can be used effectively in decision making. There needs to be an agreement on the trade offs between rigor, practical feasibility and costs of the IE, and also on a dissemination plan to ensure that the evaluation results are applied. · Government ownership is critical. Government counterparts must be involved in identifying the relevant policy questions, evaluation options and methods, executing the impact evaluation, and incorporating results in future policy choices. The TTL should GN9 Impact evaluation 3 Toolkit for monitoring and evaluation of AWM Guidance Note 9 projects Impact Evaluation of AWM projects build broad based support for the impact evaluation and identify early a champion from the government side (see Box 2). · Flexibility and adaptability. The evaluation must be tailored to the specific project and well adapted to the context, in particular through being planned as early as possible with close attention to the political environment. · Recognition of the issue of timing. The results framework should acknowledge which impacts will be observable within a short time and which others will only be observed after project completion. It is important to avoid creation of unrealistic expectations of achievement. · Baseline. An evaluation will not have adequate data if it is attempted when it is no longer possible to do a baseline survey or in the absence of comparison groups. Without a baseline, controls for comparison must be constructed by using matching methods (see below). Key concepts and techniques for impact evaluation Baker (2000) provides guidance on how to design and conduct an impact evaluation. It is beyond the scope of this Toolkit to provide more than a brief introduction to the alternative approaches and some key recommendations relevant to AWM projects. Users should consult the references provided at the end of this Guidance Note for more detailed explanations. Determining the counterfactual is at the core of evaluation design: that is, what would have happened had the project never taken place. To determine the counterfactual, it is necessary to net out the effect of the interventions from other factors, through the use of control or comparison groups (those who do not participate in a program or receive benefits), which are subsequently compared with the treatment group (individuals who do receive the intervention). Control groups are selected randomly from the same population as the program or project participants, whereas a comparison group is simply another group that does not receive the program under investigation. Because no method is perfect and the project design often constrains choices, the evaluator must carefully explore the methodological options and their combinations in designing the study, with the aim of producing the most robust results possible. Methods to establish and estimate the counterfactual fall into two broad categories: experimental design (randomized), which are the most robust but also difficult to implement, and quasi experimental designs (non randomized) (Table 2). Experimental designs (randomization) These are considered the most robust of the evaluation methodologies and require a randomized selection of the target population as part of the project design. A random selection into treatment and control groups is made within a defined area and population. In this case there should then be no difference between the two groups besides the fact that the treatment group have access to or are influenced by the project or program. Despite its robustness, this method is unlikely to be appropriate for many AWM projects as it will not be possible nor desirable to divide the potential beneficiary population intro treatment and control groups. In cases however where decentralized AWM services are to be provided, eligible sites for a first phase of intervention can be selected randomly, thus providing a sound basis of evaluating the impacts of intervention. For example, where there are limitations in funding, the first phase of a project might cover only half of villages that would be eligible otherwise for assistance, based on socio economic and environmental criteria. These villages with intervention would be selected by a random lottery and would represent the treatment group, and the eligible villages without intervention the control group. GN9 Impact evaluation 4 Toolkit for monitoring and evaluation of AWM Guidance Note 9 projects Impact Evaluation of AWM projects Non experimental or quasi experimental designs These techniques generate comparison groups that resemble the treatment group, at least in observed characteristics, through use of econometric methods. Their main benefits are that these techniques can draw on existing data sources and can be performed after a project has been implemented. Their principal disadvantages are that the reliability of the results is often reduced as the methodology is less robust statistically and the methods required can be statistically complex. · Matching methods or constructed controls: a comparison group that matches the treatment group is assessed using a survey applied to both groups. This technique is usually considered a second best alternative to experimental design. Propensity score matching is the most widely used type of matching (see Box 3). In this method, the comparison group is matched to the treatment group on the basis of a set of observed characteristics or by using the "propensity score" (predicted probability of participation given observed characteristics); the closer the propensity score, the better the match. A good comparison group comes from the same economic environment and is administered the same questionnaire by similarly trained interviewers as the treatment group. Since this method is statistically complex, it requires a team with statistical expertise. A second type of matching that also controls for observable selection bias is a pipeline comparison: in this case the control group is constructed from communities or households that are eligible for the program, but have not yet been selected to receive the intervention. This, in theory, ensures that that the treatment and control groups are comparable in all aspects except that they have not yet received the intervention. In AWM projects, this is a low cost alternative, as it can capitalize on the phasing in of the project, in terms of infrastructure development or institutional changes such as establishment of WUAs. The evaluation will however be limited to the time framework allowed by the last phase of the project and can not look at long term effects. Box 3: Applying the propensity score matching technique to evaluate irrigation investments in Vietnam The study was primarily aimed at generating empirical evidence on how various policy interventions in irrigation (rehabilitation, management improvement, or both) had had an impact at household level in the three selected irrigation schemes in Vietnam. Baseline survey data was not available for the selected irrigation schemes where interventions had taken place. Thus, determining counterfactuals based on neighboring communities became a key concern in the study, and a propensity score matching methodology was adopted. Based on the findings of a qualitative assessment (field visits and focus groups discussions), sample study sites for intensive household surveys were selected such that the socio economic, biophysical and institutional features of intervention areas and their neighboring areas (without intervention) were closely similar except for differences in irrigation infrastructure and/or management. The sample households were selected by stratified random sampling in the study sites. A structured questionnaire was designed, which was tested for the required data from the selected sample households. Trained field investigators administered the pre tested questionnaires through personal interviews with the households under the supervision of the researchers of the study team. The household level data collected covered socioeconomic and biophysical features of agricultural production systems. A first series of regression analyses were carried out on the pool of households to determine the main differences in the level of identified impact indicators between areas, with and without intervention. To obtain the counterfactual and net out the effects caused by factors other than irrigation intervention, comparison groups were formed using a propensity score matching methodology. Households whose propensity scores did not match between the two groups were taken out from the sample. The survey data of these two comparison groups was finally used for the whole analysis. Source: Janaiah and Mekong Economics Ltd, 2004. GN9 Impact evaluation 5 Toolkit for monitoring and evaluation of AWM Guidance Note 9 projects Impact Evaluation of AWM projects · Double difference or difference in differences methods: one compares a treatment and comparison group (first difference) before and after a program (second difference). Comparators should be dropped when propensity scores are used and if they have scores outside the range observed for the treatment group. · Instrumental variables or statistical control methods: one uses one or more variables that matter to participation but not to outcomes given participation. This identifies the exogenous variation in outcomes attributable to the program, recognizing that its placement is not random but purposive. The "instrumental variables" are first used to predict program participation; then one sees how the outcome indicator varies with the predicted values. · Reflexive comparisons: a baseline survey of participants is done before the intervention and a follow up survey is done after. The baseline provides the comparison group, and impact is measured by the change in outcome indicators before and after the intervention. This approach is limited in its ability to isolate the exogenous factors that may have caused change for the participants independent of the influence of the project. Qualitative methods Qualitative and participatory techniques can also be used to assess impact. These techniques often provide critical insights into beneficiaries' perspectives, the value of programs to beneficiaries, the processes that may have affected outcomes and a deeper interpretation of results observed using quantitative analysis. Because measuring the counterfactual is at the core of impact analysis techniques, qualitative designs have generally been used in conjunction with other evaluation techniques. The benefits of qualitative assessments are that they are flexible, can be specifically tailored to the needs of the evaluation, can be carried out quickly, and can greatly enhance the findings of an impact evaluation through providing a better understanding of stakeholders' perceptions and priorities, and of the conditions and processes that may have affected program impact. Among the main drawbacks are the subjectivity involved in data collection, the lack of a comparison group, and the lack of statistical robustness, given typically small sample sizes. The validity and reliability of qualitative data are also highly dependent on the methodological skill, sensitivity, and training of the evaluator. Integrating Quantitative and Qualitative Methods Integrating quantitative and qualitative evaluations can often be the best approach to meet a project's information needs. In combining the two approaches, qualitative methods can be used to inform the key impact evaluation questions, improve the questionnaire or the stratification of the quantitative sample, and analyze the social, economic, and political context within which a project takes place. Quantitative methods can be used to inform qualitative data collection strategies, to inform on the extent to which the results observed in the qualitative work can be generalized to a larger population and statistical analysis can be used to control for household characteristics and the socio economic conditions of different study areas, thereby eliminating alternative explanations of the observed outcomes. GN9 Impact evaluation 6 Toolkit for monitoring and evaluation of AWM Guidance Note 9 projects Impact Evaluation of AWM projects Table 2: Evaluation Methods and Corresponding Data Requirements Data requirement Use of qualitative Method Minimal Ideal approach Experimental or Single project cross Baseline and follow up Inform design of survey randomized controls section with and without surveys on both instrument, sampling. beneficiaries beneficiaries and non Identify indicators. beneficiaries. Allows for Data collection and control of contemporaneous recording using: events, in addition to -Textual data providing control for -Informal or semi measuring impact (allowing structured interviews for a difference in difference -Focus groups or estimation) community meetings Nonexperimental -Direct observation designs -Participatory a. Constructed Large survey, census, Large survey, and smaller methods controls or national budget, or project based household -Photographs matching LSMS type of survey survey, both with two points -Triangulation over sample in time to control for -Data analysis beneficiaries contemporaneous events b. Reflexive Baseline and follow up Time series or panel on comparisons and on beneficiaries beneficiaries and double difference comparable non beneficaries c. Statistical control or Cross section data Cross section and time instrumental representative of series representative of both variable beneficiary population the beneficiary and non with corresponding beneficary population with instrumental variables corresponding instrumental variable Source: Baker, 2000. Evaluating sector reforms AWM projects are often used as a basis for launching sector wide, national reforms. · Approaches in the absence of a counterfactual include: o Qualitative studies that assess conditions of the population (often identifying vulnerable subgroups) before, during, and after sectoral reforms are implemented through focus groups, interviews, and other qualitative techniques. o "Before and After," which compares the performance of key variables during and after a program with those prior to the program. The approach uses statistical methods to evaluate whether there is a significant change in some essential variables over time. This approach often gives biased results because it assumes that had it not been for the program, the performance indicators would have kept the same values. · Approaches that generate a counterfactual include in particular Computable General Equilibrium (CGE) models that attempt to contrast outcomes in treatment and comparison groups through simulations. These models seek to trace the operation of the real economy and are generally based on detailed social accounting matrices collected from data on national accounts, household expenditure surveys, and other survey data. Computable General Equilibrium models do produce outcomes for the counterfactual, though the strength of the model is entirely dependent on the validity of assumptions made in constructing the model. This can be problematic as databases are often incomplete and many of the parameters have not been estimated by formal econometric methods. These models are also very time consuming, cumbersome, and expensive to generate. GN9 Impact evaluation 7 Toolkit for monitoring and evaluation of AWM Guidance Note 9 projects Impact Evaluation of AWM projects · A useful World Bank tool with regard to sector reform assessment is Poverty and Social Impact Assessment (PSIA). The PSIA7 analyses, ex-ante or ex-post, the distributional impact of policy reforms on the well being of different stakeholder groups with a particular focus on the poor and vulnerable. The PSIA can be used to assess the poverty and social impacts of water sector reforms; for example, it was being used for this purpose in Yemen in 2007. Key steps in designing and implementing impact evaluations for AWM projects There is no standard approach to conducting an impact evaluation. Each evaluation has to be tailored to the specific project, country and institutional context and the actors involved. However, regardless of the size, program, type or methodology used for the evaluation, there are some steps to follow at each stage of the project cycle (see Box 4). 7See World Bank, 2003. http://siteresources.worldbank.org/INTTOPPSISOU/Resources/PSIA_Guide .pdf GN9 Impact evaluation 8 Toolkit for monitoring and evaluation of AWM Guidance Note 9 projects Impact Evaluation of AWM projects Box 4: Steps in designing and implementing impact evaluations in the World Bank project cycle Project identification to the Project Concept Note i. Determine whether or not to carry out an evaluation ii. Form the evaluation team iii. The TTL, Lead Evaluator (LE), and government counterparts clarify objectives of the evaluation and the policy questions that the evaluation might address iv. As the scope of the evaluation will depend in part on the available budget envelope, identify likely sources of funding, including World Bank, government, and external donors Project Preparation through Appraisal v. Identify other interventions in the project area, for ensuring the validity of attribution of the observed changes or testing the synergies between interventions and considering combining evaluations or at least evaluation data collection efforts. vi. Build the Results Framework, by clarifying the outcomes of interest and the indicators best suited to measure changes in those outcomes, and the expected time horizon for changes in those outcomes. vii. Design the evaluation: - the LE shall develop the identification strategy i.e. how to identify the impact of the project separately from changes due to other causes; - the LE will select the sample of units (households, communities, water user groups depending on the project) that will best serve as a comparison against which to measure results in the sample selected into the project. viii. Explore data availability and develop a data collection strategy: sample design and selection, data collection instrument development, pilot testing, data collection, data management and access. ix. The TTL and LE identify field work staff and plan the necessary capacity building activities in collaboration with government officials and partnerships with local academics. x. The stakeholders agree on how to integrate the impact evaluation into the project as a mechanism to increase the effectiveness of project implementation. xi. The TTL decides whether or not to formally initiate an IE activity by putting an IE code in the Bank internal management information system (SAP) and writing up a concept note for the impact evaluation. During Project Implementation xii. Monitor project implementation to identify threats to the validity of the evaluation design (such as spillover effects on the comparison group) or opportunities to redesign the evaluation, increase its scope and effectiveness, and/or reduce data costs. xiii. Ongoing data collection. xiv. Analyze the data as soon as data becomes available. xv. Write up the findings and discuss them with policymakers and other stakeholders. xvi. Incorporate the findings in project design, inform changes in the project, for example at mid term, and inform project assessment at completion. xvii. Promote the dissemination of results to inform the next generation of projects as well as government policy development. xviii.Continue collaboration with government and local researchers to build capacity for impact evaluation. Source: Adapted from Baker, 2000, and World Bank, 2006b. 1. Deciding whether or not to carry out an impact evaluation Impact evaluations require significant resources (see RN5). For AWM projects they are best mobilised when the project is innovative, targets difficult to reach groups or persistent problems, or is expected to have gender differentiated impacts. The evaluation might also be crucial in filling in knowledge gaps of what works and what does not in organizational and institutional reform. Finally, gaining support from policymakers and financiers for an impact evaluation can be challenging but is a prerequisite for proceeding (Box 2). GN9 Impact evaluation 9 Toolkit for monitoring and evaluation of AWM Guidance Note 9 projects Impact Evaluation of AWM projects In the past, impact evaluations were constrained by the lack of data and the technical challenges of developing a counterfactual. Over the past few years significant improvements in both these areas have made IEs easier to implement on a systematic basis: household level data gathered through household surveys or demographic and health surveys are more available, and a range of evaluation techniques have been developed to construct the counterfactual (as considered above). 2. Clarify evaluation objectives Clear objectives are essential to identify information needs, set output and impact indicators and construct a solid evaluation strategy to provide answers to the questions posed. The Results Framework provides a good basis for this. AWM projects can have far reaching impacts on economic growth, poverty, the environment and empowerment. Narrowing the questions the impact evaluation intends to assess is a difficult yet crucial exercise and it must be done in partnership with government counterparts. Although they may be significant (as discussed in GN10), capturing the full multiplier effects of AWM investments on the local economy would be methodologically and practically difficult, and project level impact evaluations should focus on first round impacts; in particular: agricultural productivity, returns to farming, household incomes, and on farm employment. 3. Explore data availability Many types of data, from cross sectional or panel surveys to qualitative open ended interviews, can be used to carry out impact evaluation studies. In many cases, the impact evaluation will take advantage of some kind of existing data or `piggyback' on an ongoing survey, which saves considerably on costs. 4. Design the evaluation The scale and complexity of the evaluation design, the choice of methodologies and the balance between quantitative and qualitative techniques will depend on: - the key evaluation questions to be answered; - timing and budget constraints8; - and the capacity of the evaluation team. The design will also be influenced by the way in which the impact evaluation will fit into the broader M&E strategy applied to a project. Finally, even after the design has been built into the project, evaluators should be prepared to be flexible and make modifications to the design as the project is implemented. 5. Form the evaluation team A range of skills is needed in evaluation work. Among the core team is the Lead Evaluator (LE), analysts (both economist and other social scientists), and, when new data collection is necessary, a sampling expert, survey designer, fieldwork manager and fieldwork team, and data managers and processors. It is important to identify team members as early as possible and establish mechanisms for communication with the Task Team and Government Counterparts. In building up the evaluation team, the TTL and LE should assess local capacity and identify the appropriate institutional arrangements to ensure impartiality and quality in the evaluation results. Impact evaluation can be undertaken in collaboration with a private firm, research institute, or public agency. The degree of institutional separation to put in place between the evaluation providers and users has to be clarified. 6. Data development GN6 briefly describes the main data collection methods. Household surveys are commonly used, but a range of techniques can be called upon. In particular, most of the methodologies can incorporate qualitative and participatory techniques in the design of the survey instrument, in the 8 GN9 Impact evaluation 10 Toolkit for monitoring and evaluation of AWM Guidance Note 9 projects Impact Evaluation of AWM projects identification of indicators, and in the identification of controls, variables used for matching, or instrumental variables. For evaluations that generate their own data it is necessary to decide on what to measure, develop data collection instruments and approaches, define the sample strategy and framework, design questionnaires, and design the data management system. 7. Analysis, Reporting, and Dissemination The analysis of the evaluation data, whether quantitative or qualitative, requires collaboration between the analysts, data producers, and policymakers to clarify questions and ensure timely, high quality results. Whether analyzing quantitative or qualitative information, a few general recommendations can be made. · Analysis commonly takes longer than anticipated, particularly if the data are not clean or accessible at the beginning of the analysis, if the analysts are not experienced with the type of evaluation work, or if there is an emphasis on capacity building through collaborative work. Often the analysis takes more than a year after producing the data. · The TTL and LE should plan to produce several products as outputs from the analytical work, trying to ensure the timing of outputs around key events when decisions regarding the future of the project will be made (such as mid term reviews, elections, or closings of a pilot phase), and keeping in mind the importance of differentiating products according to the audience. · The products will have the most policy relevance if they include clear and practical recommendations stemming from the impact analysis Reporting and dissemination of the findings must be an integral part of the impact evaluation plan. It is necessary to consider from the start the types of outputs that will be produced, their timing, and their audience. For World Bank projects, a first use of evaluation information is to enhance the analysis of impacts and lessons learned in the midterm review report during project life and in the ICR Report at project completion (see GN13). Other outputs include reports to policy makers, academic research, donors, stakeholders in the project, and the general public in the country. These different audiences will have different information needs. The results of the evaluation will have most policy relevance and effect if they include clear and practical recommendations stemming from the analysis. (GN6 explores in more detail the uses of M&E information). Further reading Baker, J. 2000. Evaluating the impact of development projects on poverty: a handbook for practitioners, Volume 1. Washington D.C.: World Bank. Available at http://go.worldbank.org/8E2ZTGBOI0 Kusek, J.Z., and R.C. Rist. 2004. Ten Steps to a Results Based Monitoring and Evaluation System. . Washington D.C.: World Bank. Available at http://go.worldbank.org/7H1YFQS5Z0 Ravallion, M. 2000. The Mystery of the Vanishing Benefits: An Introduction to Impact Evaluation. World Bank Economic Review, Vol.15, No. 1, pp: 115 140. Washington D.C.: World Bank. Available at: http://povlibrary.worldbank.org/library/view/4642/ World Bank. 2006b. Impact Evaluation and the Project Cycle. Doing Impact Evaluation Series No. 1. Thematic group on poverty analysis, monitoring and impact evaluation. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTISPMA/Resources/383704 1146752240884/doing_ie_series_01.pdf GN9 Impact evaluation 11 Toolkit for monitoring and evaluation of AWM Guidance Note 9 projects Impact Evaluation of AWM projects World Bank Operations Policy and Country Services. 2005a. Implementing Impact Evaluations at the World Bank: Guidance Note. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTISPMA/Resources/Training-Events-and- Materials/IEGuidanceNoteFINALApril06.pdf Websites World Bank Thematic group on poverty analysis, monitoring and impact evaluation. Impact Evaluation: http://go.worldbank.org/5FDCUE6PC0 World Bank Development Impact Evaluation Initiative: http://go.worldbank.org/1F1W42VYV0 GN9 Impact evaluation 12 Toolkit for monitoring and evaluation of AWM Guidance Note 10 projects AWM projects and M&E of poverty reduction Guidance Note 10 AWM Projects and M&E of Poverty Reduction The poverty reduction impacts of agricultural water management projects can be increased by integrating poverty related concerns in project design and appraisal. A review of poverty analysis in World Bank AWM operations was commissioned in 2006 (Ward, 2007, draft), and will lead to guidance on approaches for project preparation and project M&E. This guidance note provides an interim summary of established key concepts and best practice. The contribution of irrigation to poverty reduction Recent studies have strengthened the evidence that investment in irrigation can contribute to poverty reduction (for example, Hussain, 2005; Smith, 2004). The incidence of poverty is generally lower in irrigated areas than in rainfed regions, and access to irrigation water usually reduces the incidence and severity of poverty. Irrigation enables households to improve crop productivity, grow higher value crops, and generate higher incomes and employment. Income inequality and poverty rates are consistently lower for irrigated areas, and households with access to irrigation and complementary inputs are less likely to be poor. Irrigation reduces poverty through three direct first round effects: increased food output, higher demand for employment, and higher real incomes. Irrigation also has longer run effects on the poor through a multiplier effect that will typically drive an increase in non farm rural output and employment as the level of rural spending rises. Security of water supply, reduced variability of output, employment, and income, and the better opportunities for crop diversification reduce the vulnerability to risk of the poor. Multiple uses of irrigation water, including water supply, livestock watering, fishing, artisanal industries and home gardens, can also be particularly important for the poor, and not least for women and children within households. Social benefits may also accrue. Improved agricultural water use has been linked, for example, to such diverse effects as reduced seasonal rural out migration, and increased girls' attendance at school. However, there are also potentially negative impacts on the poor that may need to be mitigated and can limit the effectiveness of irrigation as a poverty reducing investment. Poor irrigation performance can lead to poverty for those not able to receive adequate supplies of irrigation water. There is more poverty in some irrigated areas and social groups than others: poverty tends to afflict particularly the agriculture dependent landless and female headed households. Irrigation can even have direct negative impacts on the poor in situations where adverse social, health and environmental costs of irrigation outweigh the benefits the poor receive from irrigation. In addition, poor people do not necessarily benefit most from irrigation. Position within a scheme and land distribution affect the incidence of poverty. The positive impacts of irrigation on poverty have been shown to be highest where landholdings and irrigation water supplies are equitably distributed (Hussain, 2005). Implementing the poverty reduction mandate of the World Bank The World Bank's mission is sustainable poverty reduction. Within this broad framework, a critical priority for the Bank is promoting broad based economic growth, given its importance in reducing poverty. The Operational Procedure 1.009 on poverty reduction states that the World Bank's support for poverty reduction is focused on three pillars: increasing opportunity, enhancing empowerment, and strengthening security. To define and implement effective national poverty reduction strategies, efforts focus on country level analytical and programmatic work: 9World Bank OPCS, Operational Manual. GN10 Poverty 1 Toolkit for monitoring and evaluation of AWM Guidance Note 10 projects AWM projects and M&E of poverty reduction The Poverty Assessment10 (PA) is a key instrument designed to assess the extent and causes of poverty in a given country and to propose a strategy to ameliorate its effects. It reviews levels and changes over time and across regions in poverty indicators, assesses the impact of growth and public actions on poverty and inequality, and reviews the adequacy of a country's poverty monitoring and evaluation arrangements. PAs generally feed into country owned processes to develop strategies to reduce poverty, help build in country capacity, and support joint work and partnerships. The business plan of the World Bank for a country is stated in the Country Assistance Strategy (CAS). The CAS sets out a four year selective program of Bank Group support linked to the country's development strategy, and is based on the national Poverty Reduction Strategy Paper (PRSP), where available. The CAS includes a comprehensive diagnosis drawing on analytic work by the World Bank, the government, and/or other partners of the development challenges facing the country, including the incidence, trends, and causes of poverty. The CAS identifies the key areas where the Bank Group's assistance can have the biggest impact on poverty reduction. In some countries, the World Bank develops Country Water Resources Assistance Strategies (CWRASs), in support to the implementation of the World Bank's Water Resources Sector Strategy. CWRASs are individual country strategies that define 3 year programs of Bank lending and non lending support. They are consistent with the overarching CAS and PRSP. Where a country is proposing reform programs that will have significant distributional impacts, a Poverty and Social Impact Assessment (see GN9) should be carried out, particularly where the World Bank is to support the reforms through a Development Policy Lending programme. Investment projects have to be defined and implemented within the framework of the CAS (and if applicable CWRAS), which should guarantee their consistency with the national poverty reduction strategy. As noted above, AWM investment projects also target economic growth deemed essential for poverty alleviation, through stimulating growth in agricultural production and incomes. There is no formal World Bank requirement for the specific analysis of poverty issues in project appraisal, and there are no official guidelines for poverty analysis or poverty monitoring and evaluation in projects. However, a number of instruments are used that typically involve some aspects of poverty analysis. These include social impact assessment, economic and financial analysis, triggering of safeguards, and monitoring of key socio economic factors. Taking poverty into account in Bank AWM projects A review of a sample of recent World Bank financed AWM projects shows that these could further improve in terms of attention to poverty reduction (Ward, 2007, draft). In particular, the poverty reduction process supported by the project was rarely made explicit. The "project logic" in the Results Framework was sometimes not clear and there was generally a loose conception conveyed of what constitutes poverty and poverty reduction. Social assessment and social analysis carried out under project preparation addressed some aspects of poverty but often without linking stated social development objectives to poverty reduction. Moreover, the social assessment often failed to identify who are the poor. The analysis of institutional arrangements often did not explicitly consider the poor, despite the strong impact of institutional arrangements on the poor and on mitigation of negative impacts. Finally, the Results Framework rarely defined poverty targets or intermediate results, and monitoring systems typically were not designed to 10See World Bank, 2004a. GN10 Poverty 2 Toolkit for monitoring and evaluation of AWM Guidance Note 10 projects AWM projects and M&E of poverty reduction provide a picture of progress against poverty related targets. Employment aspects, important for the landless poor, were often not considered (see Box 1 for an example of good practice). Box 1: Good practice on poverty in the India Madhya Pradesh Project The project demonstrates a good fit with national and state strategy for poverty reduction, showing how project actions will implement poverty reduction strategy through infrastructure investment, decentralization and empowerment, and improved natural resource management. Water resource management issues that affect poverty outcomes are analysed. The project is clear about the beneficiaries and their incomes. The key performance indicator is a productivity measure "net benefit per unit of water delivered" but there is also a quantified target for moving 75,000 farmers out of poverty. The poverty reduction model is clear. Technical improvements and decentralized management will deliver reliable water services at reasonable cost by financially self sustaining entities. This will increase the area irrigated, improve water productivity, and allow intensification and diversification. The poor will benefit the most, with their net income going up by four times (compared to three times for larger farmers). The project also considers employment impacts for the landless (an extra 122,000 jobs). Social analysis drove the pro poor design. Social analysis started early and was integrated with the financial and technical analysis. The analysis captured social issues and impacts, and the resulting social and institutional design promoted poverty reduction by emphasizing decentralized and participatory approaches. This pro poor design was helped by a preparation process that considered the poor, including: · use of good data on a sample of 450 farmers available through the National Sample Survey at the village level allowed the project to characterize beneficiaries by income level. · The project takes place in five basins and with different scales of scheme (major, minor, village level etc.). Thirteen models were prepared and analysed in an integrated fashion from the technical, economic and financial, social and environmental standpoints. · The stakeholder and environmental analyses were done by a unified team using participatory approaches. · The farm models allowed the team to forecast expected benefits by farm size, and the poverty line was correlated to the farm size. Source: Ward, 2007, Draft Defining poverty-related indicators in AWM projects The World Bank defines poverty as "pronounced deprivation in well being"11. It recognizes that poverty has many dimensions, including not only material deprivation, as measured by an appropriate concept of income or consumption, but also low achievements in education and health, vulnerability, exposure to risk, and finally voicelessness and powerlessness (see Box 2). This understanding brings to the fore contributions of different areas of action for the poverty reduction agenda and suggests different measures of poverty reduction. 11World Bank, 2000. GN10 Poverty 3 Toolkit for monitoring and evaluation of AWM Guidance Note 10 projects AWM projects and M&E of poverty reduction Box 2: Measuring poverty in its multiple dimensions · Income poverty This approach, based on household income and expenditure surveys, has become the workhorse of quantitative poverty analysis and policy discourse. It has several strengths: in particular, by collecting information beyond monetary income or consumption in the household surveys, it is possible to obtain a broader picture of well being and poverty, investigate the relationships among different dimensions of poverty, and test hypotheses on the likely impact of policy interventions. On the other hand however, variation of survey designs over time or among countries often make comparisons difficult, and collection of information at the household level hides inequality within the household. A key building block in developing income and consumption measures of poverty is the poverty line the critical level of income or consumption below which an individual or household is determined to be poor. The poverty line is generally country specific, and can be adjusted for different areas within the country if prices or access to goods and services differs. The most common way to measure poverty is the `headcount' measure, which calculates the percentage of the population with income or consumption levels below the poverty line. Other poverty measures, which take into account the distance of poor people from the poverty line (the poverty gap) and the degree of income inequality among poor people (the squared poverty gap), can also be readily calculated given the necessary data. · Measuring deprivation in health and education Non income indicators include in particular, infant and under five mortality rates, and gross primary school enrolment rate. · Vulnerability Vulnerability is a dynamic concept, and its measurement centres on the variability of income or consumption, or on the variability of other dimensions of well being, such as health or housing. Assessing vulnerability is more complex than measuring poverty at a point in time: it requires household panel data, that is, household surveys that follow the same households over several years with data on assets (physical, human, and social capital), in combination with data on formal safety nets, the functioning of markets, and overall economic policies. · Voicelessness and powerlessness Voicelessness and powerlessness can be measured using a combination of participatory methods, polls, and national surveys on qualitative variables such as the extent of civil and political liberties. A participatory method for example, is that people discuss the range of institutions important in the daily lives of the poor, identify criteria, and rate those. Characteristics usually include trust, participation, accountability, ability to build unity, responsiveness, respect, fairness and caring, and listening. Source: World Bank, 2000. Monitoring and evaluating AWM project outcomes contributing to poverty reduction Most AWM projects do not list poverty reduction as a Project Development Objective ­ although it may be cited as a higher level objective ­ as it typically involves a large number of variables beyond direct project control. Projects often, however, identify outcomes or impacts related to income increases, improved distribution of benefits, reduced exposure to risk, and empowerment. Some of these could clearly be put in the context of a causal chain or intervention model leading to poverty reduction (GN4); in particular, increased reliability of water, equitable distribution of water, inclusion of the vulnerable in water users associations, increases in agricultural output and farm incomes, improvements in the distribution of income, and increases in employment opportunities (Box 3 gives a range of poverty related intermediate outcomes and results as cited in new AWM projects from the period 2003 2005). These indicators could also explicitly serve as warning signals for identifying potential negative impacts harming the poor. GN10 Poverty 4 Toolkit for monitoring and evaluation of AWM Guidance Note 10 projects AWM projects and M&E of poverty reduction Box 3: Example of outcomes or Project Development Objectives for AWM projects · Reliability: actual water supplied to demand planned (Drainage, Irrigation and Wetlands Improvement Project, Uzbekistan) · Equity: farmers on tail end watercourses receive due water share 80% of the time (Sindh On Farm Water Management Project, Pakistan), satisfaction with equity of water distribution (Irrigation Distribution System & Management Improvement Project, Azerbaijan) · Increase in farm output value and on farm income (Jiangxi Integrated Agricultural Modernization Project, China) · Number of employment opportunities generated (Second North East Irrigated Agriculture Project, Sri Lanka) · Inclusion of the vulnerable : Recognition and enhancement of role of women in sector governance institutions; stakeholders, especially poorer and traditional water users feel better served (Water Resources & Irrigation Sector Management Program, Indonesia) · Distributional benefits : Improved incomes of targeted stakeholders in head, middle and tail reaches (Maharashtra Water Sector Improvement Project, India) Source: Ward, 2007, Draft The poverty reduction outcomes of AWM projects can be linked to all of the three strategic dimensions identified by the World Bank, namely increasing opportunity, enhancing empowerment, and strengthening security. However, in most cases poverty reduction will be monitored using an income poverty measure. Such is the case, for example, in the India Madhya Pradesh Project (see Box 1) or in the evaluation of poverty reduction impacts of irrigation investments in Vietnam (see Box 4). The analysis can capture: the distribution of beneficiaries between different income groups, and in particular the proportion of poor people amongst the beneficiaries; and the distribution of quantified net benefits amongst income groups. The analysis can be based on quantified or subjective estimates of wealth. Empowerment is an important topic for projects that are specifically based on a Community Driven Development (CDD) approach. Such projects are in most cases multi sectoral, with AWM components that may be small compared to the total programme of project activity. An example where AWM is a key component of the project is the World Bank Irrigation Based Community Development Project in Morocco. It will monitor the extent of beneficiary participation in sectoral programs. Indicators will include the number of Participatory Programming Team meetings and Core Rural Development Committee meetings, and the share of investments financed by the beneficiaries and rural communities. A participatory monitoring and evaluation system has been recommended for this project (World Bank, 2006. Project Appraisal Document).12 Finally, given the potentially important trickle down and multiplier effects of AWM investments on the rural economy, an impact evaluation (see GN9) could also consider induced indirect benefits at the community level. However, given the difficulties of monitoring and isolating impacts of the project from other drivers of local economic development, it is recommended that M&E efforts focus on the first round benefits of on farm employment and income, agricultural wage rates, and benefits from multiple uses of water, rather than attempting to assess wider impacts on the non agricultural sector and on the local economy. Methods for M&E of poverty reduction and sources of data The project should evaluate relevant existing secondary sources of information (such as the agricultural census, or national surveys), before generating its own set of data. Also, analyzing poverty requires an understanding of national and sectoral macro factors, and the project could usefully list analytical work carried out in the sector (see Table 1). 12For more guidance on M&E of empowerment, see: Alsop and Heinsohn, 2005, and World Bank, 2005c. GN10 Poverty 5 Toolkit for monitoring and evaluation of AWM Guidance Note 10 projects AWM projects and M&E of poverty reduction Table 1: Instruments for poverty analysis and measurement in AWM projects What should be analyzed and measured Some sources of data At the national and sectoral level, macro factors PRSP, CAS, CWRAS, sector studies. such as investment policy, land policy, tax and Specific policy reforms can be analyzed through a atad price policy Poverty and Social Impact Assessment. elbaliavA Poverty and socio economic data at the national, Country statistics, National surveys, Poverty regional and local levels Assessments, Labor market surveys Records of cooperatives, credit unions, and other financial institutions Other local development interventions studies At the scheme level, institutional and Social assessment as part of project preparation. management factors Administrative records Qualitative studies, Participatory M&E atad Structural factors such as land tenure, water Farm level surveys distribution, cropping patterns, household Administrative records detarenegtcejorP attributes Poverty and socio economic data for the Household budget, income and expenditure population in the scheme area and surrounding surveys villages. Multi topic household surveys: Living Standards Measurement Surveys, Demographic and Health Surveys Subjective wealth ranking Intra household factors, particularly gender Household level surveys focused Qualitative studies, Participatory M&E Source: Adapted from Ward, 2007, draft. Most measures of poverty rely on household level survey data (see GN6). The M&E system should also consider making use of qualitative methods, such as case studies or interviews with key informants. In particular, an important tool with regard to poverty reduction and distributional issues is the use of participatory M&E methods (GN11 examines participatory M&E). This approach yields a number of advantages: · A relevant local definition and classification of poverty (or, equivalently, ranking of well being). · Measures of welfare impact (i.e. marginal improvement in welfare of the less well off) according to notions more in line with people's perceptions and local circumstances, than according to some exogenously determined indicator. · The basis for a deeper understanding of local constraints and factors that determine poverty/distributional impact. This, in turn, can feed into improved design for pro poor projects. · Early warning if the project is having unintended negative impacts on the poor. Impact evaluation GN9 reviews concepts and methods for project impact evaluation. Measuring the counterfactual, which is the situation that would have happened had the project not taken place, is at the core of impact analysis techniques. Both qualitative and quantitative methods can be used, separately and combined. Given the variation in project types, evaluation questions, data availability, cost, time constraints, and country circumstances, each impact evaluation study will be different and will require some combination of appropriate methodologies (see Box 4 for a good practice example). GN10 Poverty 6 Toolkit for monitoring and evaluation of AWM Guidance Note 10 projects AWM projects and M&E of poverty reduction Box 4: Evaluating poverty reduction impacts of irrigation investments in Vietnam The study was aimed at generating empirical evidence on how various policy interventions in irrigation (rehabilitation, management improvement, or both) had had an impact in Vietnam at a micro level. The primary research question of the study was: How effective are public irrigation expenditures in increasing rural incomes, particularly for the poor? Further, is it more effective in terms of poverty reduction to invest in rehabilitating existing infrastructure, or in improving management? The study covered three irrigation schemes. Both qualitative and quantitative assessments were carried out. Based on the qualitative assessment, the hypotheses tested by the quantitative analysis of survey data were that the interventions that took place under the selected schemes have significantly improved the availability of irrigation water, which in turn increased rice yield, farm profits, reduced unit cost of production, reduced production uncertainties, improved household income, and reduced rural poverty. For a quantitative assessment, in depth household surveys were conducted over a period of March to May 2003, and farm level data on a wide range of variables related to crop year 2002 was collected in the selected study sites. The total number of households surveyed was 1253, including households with and without interventions. The counterfactual was constructed from neighboring areas using a propensity score matching technique (see GN9). The following indicators were identified and quantified for the study: Cropping intensity Crop income Production risks, estimated by the coefficient of variation in yield of paddy Household income, with the share of agriculture income Food consumption expenditure, as per capita food consumption expenditure is widely considered an ideal indicator of poverty Incidence of poverty: estimated poverty based on food poverty line, and overall poverty based on wealth ranking. Furthermore, the statistical analysis was conducted separately for the head ender and tail ender farmers in the schemes. Some findings In Song Chu scheme, there was a gain of about 18% in rice yield, a 22% higher income, and 14% higher food expenditure with intervention. Under the An Tranch scheme, own labor costs reduced by 22%, and rice yields increased by 13%. In the Dau Tieng scheme, paddy yields increased by about 22%, and the intervention also enhanced the productivity of non rice crops, and amplified agricultural diversification. Results show that farmers at the tail end of the canal system have substantially benefited from increases in paddy yields, rice income, total farm income, and per capita food expenditures in all schemes. Regarding poverty reduction, as estimated on a food poverty line, the intervention was much more successful in Dau Tieng irrigation scheme than in the other schemes. However, the study recommends caution in the interpretation of the findings, as increased non farm income could have been an important additional factor to the increase in rice income, contributing to the rapid reduction in poverty. Source: Janaiah and Mekong Economics Ltd, 2004. Further reading Alsop, R and N. Heinsohn. 2005. Measuring Empowerment in Practice: Structuring Analysis and Framing Indicators. World Bank Policy Research Working Paper 3510, February 2005. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTEMPOWERMENT/Resources/41307_wps3510.pdf Baker, J. 2000. Evaluating the impact of development projects on poverty: a handbook for practitioners, Volume 1. Washington D.C.: World Bank. Available at http://go.worldbank.org/8E2ZTGBOI0 Hussain, H. 2005. Pro Poor Intervention Strategies in Irrigated Agriculture in Asia. Executive summary. Colombo, Sri Lanka: International Water Management Institute. Available at http://www.iwmi.cgiar.org/propoor/files/ADB_Project/IWMI ADB%20Final%20 %20sum.pdf GN10 Poverty 7 Toolkit for monitoring and evaluation of AWM Guidance Note 10 projects AWM projects and M&E of poverty reduction Smith, L.E.D. 2004, Assessment of the Contribution of Irrigation to Poverty Reduction and Sustainable Livelihoods, International Journal of Water Resources Development, 20,2, pp243- 257. Sur, M. 2006. Monitoring and evaluating the poverty impacts of agricultural water investments. Agricultural and Rural Development Notes: Issue 15, June 2006. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTARD/Resources/Note15_ME_poverty.pdf Ward, C. 2007. Poverty Analysis in Agricultural Water Operations. Phase 1: Review of World Bank financed projects. Draft, May 19, 2007. World Bank. 1993. Poverty reduction handbook, Volume 1. Washington D.C.: World Bank. World Bank. 2000. Attacking poverty. World Development Report, ISSN 0163-5085 ; 2000/2001. New York : Published for the World Bank, Oxford University Press. Available at http://go.worldbank.org/L8RGH3WLI0 World Bank. 2003. A User's Guide to Poverty and Social Impact Analysis. Washington D.C.: World Bank. Available at http://go.worldbank.org/IR9SLBWTQ0 World Bank. 2004a. Guidance Note on Poverty Assessments. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTPOVERTY/Resources/335642- 1098192957114/op1_pa_guidance.pdf World Bank. 2005c. Community Driven Development and Social Capital: Designing a baseline survey in the Philippines. Social Development Department Working Paper. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/EXTSOCIALDEVELOPMENT/Resources/244362- 1164107274725/3182370-1164201144397/CDD-SoCap-Summary.pdf World Bank Operations Policy and Country Services. The World Bank Operational Manual. Washington D.C.: World Bank. Available at http://wbln0018.worldbank.org/institutional/manuals/opmanual.nsf/05TOCpages/The%20World% 20Bank%20Operational%20Manual Websites ADB/ IWMI. 2005a. Pro-Poor Intervention Strategies in Irrigated Agriculture in Asia. Available at http://www.iwmi.cgiar.org/propoor/index.asp?nc=3899&id=1061&msid=235 World Bank Social Development Department. Community Driven Development: http://go.worldbank.org/24K8IHVVS0 World Bank Social Development Department. Monitoring and Evaluation of Community Driven Development: http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPMENT/EXTCDD/ 0,,contentMDK:20598866~menuPK:608227~pagePK:148956~piPK:216618~theSitePK:430161,0 0.html World Bank Thematic group on poverty analysis, monitoring and impact evaluation. Poverty Monitoring: http://go.worldbank.org/UN7VBV2VW0 GN10 Poverty 8 Toolkit for monitoring and evaluation of AWM Guidance Note 11 projects Participatory M&E for AWM Projects Guidance Note 11 Participatory M&E for AWM Projects In recent years it has been recognised that there is an increased need to engage in direct dialogue with stakeholders involved in development projects. There has been a significant increase in use of participatory appraisal and planning approaches, and in establishment of participatory processes for management of infrastructure and natural resources. One critical dimension of this engagement is to involve stakeholders in the monitoring and evaluation of project activities, outputs and outcomes. This guidance note summarises the relevant key concepts and best practice. What is participatory M&E? The World Bank defines Participatory Monitoring and Evaluation (PME) as "a process of collaborative problem solving through the generation and use of knowledge. It is a process that leads to corrective action by involving all levels of stakeholders in shared decision making"13. Its key principles are: · local people are active participants--not just sources of information. · Stakeholders evaluate, outsiders facilitate. · A focus on building stakeholder capacity for analysis and problem solving. · A process that builds commitment to implementing any recommended corrective actions. Participation in M&E is often incorrectly understood to mean that local people simply have the role of collecting information, and that outsiders still determine the selection of indicators, analytical frameworks and reporting methods. In contrast, PME should be considered a different process to traditional M&E (see Table 1). In this process, project stakeholders are fully involved in designing the monitoring system and in collecting, analyzing, compiling and sharing the information. Table 1: Major differences between conventional and participatory M&E Conventional M&E Participatory M&E Primary stakeholders, project staff, Who plans and manages the Senior managers, outside managers, and other stakeholders, often process: experts helped by a facilitator Design and adapt the methodology, Role of primary stakeholders collect and analyze data, share findings, Provide information only (the intended beneficiaries): identify lessons learned and link them to action Internally defined indicators, including (but Externally defined, mainly How success is measured: not exclusively) more qualitative quantitative indicators judgments and stories of personal change Approach: Predetermined and fixed Indicative and adaptive Source: Adapted by Rajalahti and Woelcke, 2005 from Guijt and Woodhill, 2002 It can also be the case that PME is viewed as using only qualitative methods and that it produces unreliable data. In contrast, true participation requires that project stakeholders are involved in negotiating what needs to be assessed and measured, and with what level of rigor, validity and reliability, and then in selection of the appropriate methods. As with traditional or non participatory 13 World Bank, 1998 GN11 Participatory M&E 1 Toolkit for monitoring and evaluation of AWM Guidance Note 11 projects Participatory M&E for AWM Projects M&E, analytical rigor and the best quality of information will often be achieved through the use of a combination of data collection and analytical methods (GN6). When and how to use participatory M&E in projects? PME can have multiple purposes and benefits. For the World Bank the most commonly identified are: · ensuring that AWM projects and programs are responsive to the genuine needs of intended clients. · Empowering stakeholders to take action. · Strengthening I&D and other AWM institutions through better programs, accountability and transparency. · Determining the impact (also unintended impacts) of projects on the intended beneficiaries and stakeholders from their respective perspectives. · Improving information provision for strategic planning at different levels. Though each PME process is unique, there are basic steps to undertake in planning. It needs to be decided: when to use PME, who to involve in PME, what to do and how best to do it? 1. Deciding when to use PME PME can be used at any stage of the AWM project and the degree of participation can vary from stage to stage. However, it is best that PME activities are initiated at the very beginning of the project as this will increase the likelihood of mainstreaming PME in the project cycle. It is necessary to clarify what it is hoped the process can achieve that would not be possible with an externally driven and implemented approach. If resources are limited time, staff and finances it is better to prioritize when to use PME rather than to sacrifice the quality of the activities. During the planning process, or later when questions arise and a participatory M&E approach is likely to be more useful, the project team together with the beneficiaries and implementers decides the timing of specific PME activities (see Table 2). GN11 Participatory M&E 2 Toolkit for monitoring and evaluation of AWM Guidance Note 11 projects Participatory M&E for AWM Projects Table 2: Degree of participation at different stages of a PME process Level of participation When? Who to involve? Potential challenges in PME Define what is meant Design stage and The key stakeholders ­ farmers, Availability of resources by "impact" during evaluation water users, project staff, local Capacity building needs as needed and national authorities, World Facilitation Bank staff. Different stakeholders are likely to have different perceptions on impact. Design purpose, Design Essentially farmers and local Availability of resources process and methods Implementation authorities with M&E staff Capacity building needs for M&E Facilitation Define themes to monitor and evaluate Select and define Design The key stakeholders ­ farmers, Easily non functional if indicators water users, project staff, local the objectives and and national authorities, Bank available indicators are staff. not clear Different stakeholders are likely Capacity requirements to have different preferences for high indicators Potentially time consuming Harmonization of indicators accross groups challenging Give their opinion of Evaluation Primarily farmers and local Time and capacity project history and Implementation authorities requirements changes in the context Help collecting data for Baseline Depends on the purpose ­ Beneficiaries' opportunity M&E Monitoring farmers, local authorities are in a costs can become high Evaluation key role in providing and May pose a risk of collecting information at the local crowding out the true level. effect of project Project staff, managers and outcomes and make it borrower are in a key role in difficult to determine providing information. trends, patterns, outcomes or intermediate results Help analyzing and Monitoring All stakeholders ­ specific Time and capacity drawing conclusions Evaluation involvement depends on the requirements from the data and theme results Share feedback with Monitoring All stakeholders should receive Requires high level of the primary Evaluation information on the findings and commitment and stakeholders Baseline as have a chance to provide objectivity of the wider Present and needed feedback on issues that directly stakeholder community communicate the concern them findings Give views on the Mid term All stakeholders Need to involve the degree to which project evaluation stakeholders also in the objectives have been Final evaluation design process met Source: Rajalahti and Woelcke, 2005 2. Deciding who to involve in PME PME in AWM projects can involve all stakeholders in the various facets of farm and non farm activities. There is a need to assess and continuously reassess the need for and affordable degree of participation by the possible stakeholder groups (see Table 2). GN11 Participatory M&E 3 Toolkit for monitoring and evaluation of AWM Guidance Note 11 projects Participatory M&E for AWM Projects The following questions can guide the decision making (Guijt, 1999): · Who has a perspective or knowledge that is essential? What does each of the participating groups expect from the monitoring process? It will help to clarify to what extent each group is willing and able to participate in different tasks. · Is the process of collating and calculating the information important, or only the final information? · Who is going to use the final evaluation? Those who are to use it should understand what the data is based upon and how it was collected and analysed. · What skills does the analysis require? Whose capacity should be strengthened? The more difficult the analysis, the more caution should be used in encouraging broad participation unless it is clear who it will benefit and how. 3. Deciding on the scope of the work: what to do and how to do it for PME After deciding when to use PME, who to involve and the degree of participation by different stakeholders, it is essential to clarify the aim of the PME process, objectives to monitor, methodologies, roles and responsibilities, logistical arrangements and funding, work schedules, potential capacity issues, and finally the data collection and analysis. The process usually begins during project planning, or later during implementation when the need arises, with a workshop involving a facilitator and representatives of stakeholders. As an example, Table 3 sets out guidelines for introducing participatory impact monitoring to rural women's groups in Zambia. Table 3: The design of introductory workshops of Participatory Impact Monitoring Day 1 Day 2 Opening. Welcome of group members Recap Objectives of the workshop Overview of PME procedure Assessment of monitoring and evaluation Fears and expectations (impact) experience Reduction of number of impacts to be Clarification of basic terms monitored Examples of monitoring Development of indicators Energizer Energizer Review of project objectives Monitoring and documentation of data Vision of impacts and changes (role play) Responsibilities and Code of Conduct of the Monitoring Team Election of Monitoring Team Feedback by participants Evaluation of the workshop Source: Engelhardt Wendt, 2003 In the process, the following issues are important to keep in mind. · Systematic dialogue and facilitation during PME: the degree and timing of involvement may vary, but a systematic dialogue between farmers, water users, project staff, government officials, the business community, and their designate institutions and agencies is essential throughout the process. It may be necessary to choose an appropriate and non biased facilitator to guide the process (in a gender sensitive manner). · Ensuring local participation and external validity and reliability: in PME, there is often a tension between involving different stakeholders and their views on what constitutes `trustworthy' data, and meeting conventional norms of rigor. The question of ensuring both local participation and external validity largely depends on the level at which monitoring information is needed and by whom it is used. In a participatory process, negotiation about what each stakeholder group considers `rigor' will be required. It also GN11 Participatory M&E 4 Toolkit for monitoring and evaluation of AWM Guidance Note 11 projects Participatory M&E for AWM Projects calls for greater acceptance of different information sources and the use of alternative methods for assessing reliability, other than through conventional scientific measurement. · Foster use of local knowledge: local knowledge includes individual and collective perceptions of stakeholders. Too often little time is spent on understanding local uses of indicators and existing feedback systems. All these elements may be translated into criteria and methods for evaluating results. · Avoid over ambitious plans: allow sufficient time to build local skills for indicator selection and collection and other assessments. It is better to start simply and monitor only some aspects of the project. As experience grows and capacities build, the system can be expanded. For example, in Brazil, farmer based research on agro forestry systems started by monitoring plant diversity, labor input and ground cover. After the first year, farmers and scientists decided to include soil nutrients and the research process itself. · Create incentives for stakeholders to participate in PME and sustain the system: a common pitfall is to assume local interest and full participation, but local people do not have necessarily the resources needed for it and are not interested in the same kind of information as the implementation agency and government departments (see Box 3 below). · Baselines are even more complex than usual in a participatory context (see GN7 on baselines). The first issue lies with the context of participatory projects, which commonly start tentatively with small interventions. The second issue relates to the complexity of combining different perceptions of project components and bringing them together to define the measurement that represents the base case. Guijt (1999) promotes the use of a `rolling baseline' to develop a sequence of approximations that start with participatory appraisals to determine the outline of the collaborative research venture, and move towards clearer objectives that then form the basis for gathering baseline data. 4. Objectives to monitor and indicators to use It is necessary to clarify what the objectives of the work being monitored are for each group involved. When more than one group is involved, project objectives might differ and not always be shared sufficiently to allow for joint monitoring. It is crucial that indicators are negotiated with stakeholders rather than being "pre defined" or "objective" (Box 1). Clarity about the end users and end uses of the information is paramount. It is also necessary to build indicator collection systems on what exists locally, collect indicators at optimal moments (based on agreed timing, frequency and responsibilities) and avoid collecting too much data. GN11 Participatory M&E 5 Toolkit for monitoring and evaluation of AWM Guidance Note 11 projects Participatory M&E for AWM Projects Box 1: Participatory indicator identification in an extension project in Mexico In a farmer to farmer extension program in Mexico, the project team followed these steps to develop indicators. Define broad indicator areas (based on the Project Development Objective) Select currently used indicators for these areas Define stakeholder groups Select stakeholder groups to be consulted Develop indicators with different stakeholder groups Test the developed indicators across different stakeholder groups to assess their significance to others and effectiveness at indicating change Agree on a priority list among indicator options Carry out fieldwork to gather data for indicators Create lists of indicators for full evaluation use with a focus on indicators with specific importance for different actors with a limit, (e.g., three key indicators for each stakeholder group). The program team identified the range of different institutional and individual actors who affect and are affected by the project. They then prioritized three stakeholder groups to be consulted for indicator development in this trial phase: (a) farmers (participating and non participating), (b) farmer extension agents and (c) funding agencies. The research team initially proposed seven indicator areas. These were eventually narrowed down to four, based on the groups' objectives: (1) changes to local, regional, political, and sector practice and policy (e.g., level of dependence on external resources, involvement of local people, growth of local institutions, and changes in policy and practice); (2) dissemination impacts: extension to other localities or regions (e.g., horizontal and vertical linkages with other projects, agencies and NGOs beyond the region); (3) changes to the roles of the individuals in the project (primarily the coordinator, outside advisors, immediate project participants, and NGO staff); and (4) changes in the institutional structure (within and beyond the actual project). Source: Baluert and Quintanar (2000) in Kusek and Rist, 2002. 5. Methods and tools Participatory methods are varied, borrow from many disciplines, and are adapted to meet the specific jobs at hand. Many of the methods and tools useful for traditional M&E can actually be used in Box 2: Some participatory M&E tools either a participatory or non participatory way. Different data collection methods are described in · biophysical measurements GN6, and more specific participatory tools are · forms diaries described by Guijt, 1998, and also in Box 2. · · photographs or videos · maps Combinations of quantitative and qualitative · transects methods, and of those that are more and less · well being or social mapping participatory or natural science oriented, are likely to · impact flow diagrams emerge from the discussion between stakeholders. · systems diagrams It is important to test methods and tools as they are · matrix scoring introduced, provide necessary training, and ensure · relative scales and ladders incentives are in place for sustained participation in · ranking and pocket charts the process (see Box 3). · calendars The leading methods applicable for AWM projects · daily routines are considered further below. · institutional diagrams · network diagrams Participatory rural appraisal · dreams realized Participatory rural appraisal (PRA) evolved from · critical event analysis rapid rural appraisal; a set of informal techniques · case studies used by development practitioners in rural areas to · participatory theatre quickly collect and analyze data. PRA is a Source: Guijt, 1998 community based method in which data collection GN11 Participatory M&E 6 Toolkit for monitoring and evaluation of AWM Guidance Note 11 projects Participatory M&E for AWM Projects and analysis are undertaken by local people, with a multidisciplinary team of outsiders facilitating the process (World Bank, 1996). Tools commonly used in PRA are community meetings, semi structured interviewing, focus group discussions, preference ranking, mapping and modeling, and seasonal and historical diagramming. Van der Schans and Lemperiere (2006) set up guidelines for carrying out Participatory Rapid Diagnosis and Action Planning (PRDA) in irrigated agriculture. This is a research method for analyzing and improving the performance of an irrigation scheme together with farmers, based on the principles of Rapid Rural Appraisal and Participatory Rural Appraisal. The PRDA quickly identifies the main constraints to irrigation scheme performance, and generates a plan for improvement. PRA or PRDA can particularly be used during a project's preparation, to understand the context and help design the project. Beneficiary and stakeholder assessment Beneficiary assessment is a technique that focuses on listening and consultation among a range of stakeholder groups. The overall objective of this method is to help beneficiaries and other local level stakeholders identify and design development initiatives, signal constraints to their participation, and give feedback on these activities to those designing and managing a project or formulating policy. Beneficiary assessment commonly makes use of a set of tools that include conversational interviews, focus group discussions, and participant observation14. Such an assessment is useful at different stages in the project: · During the project design stage, a broad stakeholder group assessment can clarify the vision and perspective within which AWM initiatives will be undertaken. The perspectives of various stakeholders can contribute to determining the directions for change and their respective roles toward supporting and strengthening such a change process. This assessment helps to define the Project Development Objectives. · The assessment can be used for evaluation (mid term or final) to review and assess the assumptions behind the project activities, after a specific timeframe of implementation. It also serves to assess the relevance of the project in a changing context, the implementation of project activities, and the extent to which there are deterring factors to reaching desired outcomes. Finally, it is used to assess project impact and outcomes in terms of changes among target beneficiaries and their environments as a result of project interventions, and to draw lessons for future activities. The target beneficiaries as well as the main implementers are at the centre of this evaluation. Conversational or qualitative interviews Conversational (or semi structured) interviews constitute the basic tool of inquiry for the PME practitioner. Conversational interviews often take place in the homes of the interviewees, who are apt to be most comfortable there. Interviews should be conducted in the local dialect in such a way that open ended questions revolve around a number of themes or topics that project management has selected. The objective is to gain in depth information on beneficiary views in relation to a planned or ongoing activity by encouraging beneficiaries to speak freely and bring to light issues of concern to project management. Interviews can be conducted on a one to one basis or in focus groups. The advantages of individual interviews are that people are likely to speak more freely, without worrying what peers or other community members may think. Lower status or introverted members of communities may not feel comfortable speaking out in groups. Unguided discussion is apt to be vague and, therefore, of little use for decision making; probing for specificity is often required (e.g., if the farmer stated that he is not satisfied with water service, the interviewer should probe to find reasons for this dissatisfaction, preferably prioritizing them). 14For details on the necessary steps to follow in projects, see Salmen, L. 2000. Beneficiary Assessment for Agricultural Extension: A Manual of Good Practice. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTARD/825826 1111055015956/20424580/Beneficiary_Assess.pdf GN11 Participatory M&E 7 Toolkit for monitoring and evaluation of AWM Guidance Note 11 projects Participatory M&E for AWM Projects Focus group discussions For PME, in addition to enabling a wider coverage of the beneficiary population in a given time, interviews carried out in focus groups can serve as a cross check to individual interviews. The groups should normally comprise six to twelve people with common characteristics (e.g., groups of intended beneficiaries may be comprised of married women, male heads of households, youth from 15 to 35, and so on). However, there are times when it may be of use to purposefully mix the constituents of a focus group ­ for example, with members and non members of WUAs in order to better appreciate the nature of conflict and communication between them, and provide the opportunity for indigenous solutions. A pre prepared interview guide should be used in conducting these interviews. The interviewer takes on a facilitative role, guiding the discussion to cover topics from the thematic guide or outline of key points, and ensuring that everyone has an opportunity to participate. This will generally entail encouraging the more reticent, introverted persons to speak up while providing less encouragement to those most apt to dominate the discussion. A second researcher should also be present to take notes. Although the difficulty of quantifying focus group discussions may be considered a liability, their use as a cross check and as a fairly rapid and easy to read `barometer' of the mood of a community on many topics makes focus groups a useful component of the PME approach. Participant observation This technique generally involves protracted residence in a targeted community. During this stay, it is expected that the participant observer will establish enough rapport and involvement to help him or her accurately represent the conditions within the community as they relate to project objectives. The participant observer normally spends from one to three weeks in a given community. The researcher will focus on the areas of concern identified in an interview guide. During this stay in the community, the participant observer should prepare case studies of five to ten households based on repeated visits and observation. Participant observation, being costly and time consuming, should be used selectively on topics of particular interest that are of a sensitive nature and lend themselves to this form of intensive personal interaction. Community scorecard Community Score Card is a community based monitoring tool and a hybrid of the techniques of social audit, community monitoring, and citizen report cards. It has a strong focus on empowerment and accountability as it includes an interface meeting between service providers and the community that allows for immediate feedback on quality and adequacy of services provided in the community. This tool can be used for tracking inputs or expenditures, generation of benchmark performance criteria used by communities and service providers to assess their services, monitoring the quality of services over time, comparison of performance across facilities and districts, generating feedback mechanisms between providers and users, and for strengthening citizen voice and community empowerment15. It requires at least one, and more often several, community meetings. 6. Analyze and use the data and findings After analyzing data collected with stakeholders as much as is possible and useful, the information generated needs to be disseminated, discussed and used for corrective or adaptive management of the project as needed (see GN6). Findings can also help to strengthen the PME process itself. One of the factors that will motivate those involved in PME is the clear and direct usefulness of collecting and analyzing information (see Box 3). Results should be disseminated in a variety of ways adapted to user preferences, and to serve as a basis for discussion. Methods can include use of slides, videos, pictures, stories, role plays, discussion groups, workshops, and written 15 For more details, see http://povlibrary.worldbank.org/files/14548_32_CommScore.pdf in World Bank, 2003. GN11 Participatory M&E 8 Toolkit for monitoring and evaluation of AWM Guidance Note 11 projects Participatory M&E for AWM Projects reports of different lengths and different formats (see GN6). The focus on problem solving means that PME is oriented toward developing an understanding of a problem or situation in a way that can lead to timely action and resolution. Box 3: Factors influencing people's sustained participation in M&E · Perceived benefits (and partial or short-term costs) of PME · Relevance of PME to the priorities of participating groups · Flexibility of the PME process to deal with diverse and changing information needs · Quick and relevant feedback of findings · Capacity to act on recommendations that might arise from PME findings · Degree of maturity, capabilities, leadership and identity of the groups involved, including their openness to sharing power · Local political history, as this influences society's openness to stakeholders' initiatives · Dealing with short-term survival needs of participants, while pursuing longer-term information needs (especially in natural resource management) · Material support to make the PME possible (e.g. pens, books, training, etc.) Source: Guijt, 1999 Further reading Engelhardt-Wendt, E. 2003. Guideline for Introductory Workshops of Participatory Impact Monitoring. Internationale Weiterbilding und Entwicklung and Deutscher Entwicklungdiesnt. Available at http://zambia.ded.de/cipp/ded/lib/pub/object/downloadfile,lang,2/oid,2749/ticket,g_u_e_s_t/~/Parti cpatory_Impact_Monitoring.pdf Guijt, I. 1998. Participatory Monitoring and impact assessment of sustainable agriculture initiatives: an introduction to the keys. Bernents. SARL discussion paper 1. London IIED. Guijt, I. 1999. Participatory monitoring and evaluation for natural resource management and research. Socio-economic Methodologies for Natural Resources Research. Chatham, UK: Natural Resources Institute. Available at http://www.nri.org/publications/bpg/bpg04.pdf Narayan, D., 1993. Participatory Evaluation: Tools for Managing Change in Water and Sanitation. World Bank Technical Paper no. 207. Washington D.C.: World Bank. Available at http://poverty2.forumone.com/files/14789_Participatory_Evaluation.pdf Rajalahti, R., J. Woelcke, et al. 2005. Monitoring and Evaluation for World Bank Agricultural Research and Extension Projects: A Good Practice Note. Agricultural and Rural Development Discussion Paper 20. Washington D.C.: World Bank. Available at http://intresources.worldbank.org/INTARD/Resources/M_E_WB_AgExtensionProjects.pdf Sirker, K., and T. Shah. 2003. Community Based Monitoring & Evaluation System Methodology Design Workshop (June 11­14, 2002) and Follow-up (January ­ March 2003) Report. Draft, August 23, 2004. World Bank Institute's Community Empowerment and Social Inclusion Learning Program. Available at http://info.worldbank.org/etools/library/latestversion.asp?49429 Van der Schans, M. L., and P. Lemperiere. 2006. Participatory rapid diagnosis and action planning for irrigated agriculture systems. Improving Irrigation Performance in Africa Project. Rome, Food and Agriculture Organization of the United Nations. Available at ftp://ftp.fao.org/agl/iptrid/appia_manual_en.pdf GN11 Participatory M&E 9 Toolkit for monitoring and evaluation of AWM Guidance Note 11 projects Participatory M&E for AWM Projects World Bank. 1996. The World Bank Participation Sourcebook. Washington D.C.: World Bank. Available at http://www.worldbank.org/wbi/sourcebook/sbhome.htm World Bank. 1998. Participation and Social Assessment Tools and Techniques. Washington D.C.: World Bank. Available at http://go.worldbank.org/1OW4UEBZ91 Website World Bank Social Development Department. Participation and Civic Engagement: http://go.worldbank.org/FMRAMWVYV0 GN11 Participatory M&E 10 Toolkit for monitoring and evaluation of AWM Guidance Note 12 projects Actors in M&E of AWM Projects Guidance Note 12 Actors in Monitoring and Evaluation of Agricultural Water Management Projects M&E actors in the World Bank project cycle The World Bank and the borrowing country work closely throughout the project cycle although they have different roles and responsibilities. Table 1 summarises the roles of different entities for M&E along the project cycle (see GN13 for details on the project cycle and M&E requirements). Table 1: Actors and their tasks Who Responsibility at different stages Main Bank outputs The Client The World Bank and borrowing countries jointly identify Government projects that support their development goals. The borrowing country is responsible for project preparation: it A Project Implementation Plan may conducts studies and impact assessments that refine the be prepared by the client country. objectives, components, schedule, institutional responsibility and implementation plan of the project. Borrowers are encouraged to prepare Project Implementation Plans. The implementation of the project, including monitoring and evaluation and the Management Information System, is the responsibility of the borrowing country. Government Ministries and agencies collaborate on project design and implementation, and they can be both users and providers of M&E information. Typically, an AWM project might involve: · Ministry of Finance · Ministry of Planning · Ministry of Agriculture · Ministry of Water Resources · Ministry of Environment · Water or Basin Agencies Local governments also dialogue with Bank Task Team and national Ministries on project design and implementation, and M&E. Project World Bank financed investment operations are Implementatio traditionally implemented through entities or structures n Unit / which are generically referred to as project Project implementation units (PIU) or project management units Management (PMU). PIUs/ PMUs vary in size, function, physical Unit location, legal status, and degree of integration into existing country structures16. PIUs/ PMUs facilitate project implementation by performing one or both of two general functions: project coordination and project implementation. World Bank The Task Team Leader has overall responsibility for the Task Team project from inception to completion Leader 16 For a discussion of consequences of PIUs and possible alternatives, see World Bank, 2005. Guidance Note for Project Management, Strengthening Institutional Capacity during Project Implementation. GN12 Actors 1 Toolkit for monitoring and evaluation of AWM Guidance Note 12 projects Actors in M&E of AWM Projects Who Responsibility at different stages Main Bank outputs World Bank The Bank team creates the Project Concept Note at the The Project Concept Note is Task Team end of the identification phase. During preparation, the approved at the internal PCN review Task Team plays a supporting role, offering analysis and meeting. advice where requested. At appraisal, Bank staff reviews the work done during identification and preparation. The The Project Appraisal Document or Task Team prepares for Bank management either Project Program Document are submitted Appraisal Documents (investment projects) or Program to the Bank's Board of Executive Documents (for programmatic development policy Directors for approval at the lending). negotiation and approval phase. Appropriate documents are also submitted for final clearance by the borrowing government. The Bank is responsible for project implementation support, which covers broad monitoring activities, evaluative review, reporting, and technical assistance activities. At least once a year, Task Teams should visit sites of projects or representative samples of subprojects under implementation. The Implementation Status and Results (ISR) report is a The ISR report is prepared at least concise summary of key project information. Ratings annually and reports on the Status report on indicators of progress of project implementation of Agreed Outcome Indicators. It is and achievement of project development objectives. approved by the Sector Manager, These indicators should be the same as those identified in and copied to the Country Director. the Project Appraisal Document. The aide mémoire is prepared for An aide mémoire to the ISR report details findings, the client country. decisions, and recommendations, and remedies agreed to jointly with the borrower. No later than six months after loan closing, the Task The Implementation Completion Team prepares the Implementation Completion and and Results Report is submitted to Results (ICR) Report. It includes Principal Performance the Bank Board of Executive Ratings, Achievement of Development Objective and Directors for information purposes. Outputs. Operations Operations Policy and Country Services (OPCS) has Policy and responsibility for the policies and procedures that govern Country Bank operations. Services It provides advice and support to World Bank (OPCS) Management and staff on preparing and implementing of lending and non lending operations and on portfolio management. Quality The Quality Assurance Group's (QAG) primary objective At the end of each yearly exercise, Assurance is to promote excellence in Bank performance by QAG produces a Quality at Entry Group (QAG) increasing accountability and enhancing learning. synthesis report and a Quality of QAG carries out annual reviews of the quality of Supervision synthesis report that operational products from a sample of projects, for Quality are discussed with the Regions at Entry, and for Quality of Supervision. Each operation is before being submitted to the rated on a four point scale Highly Satisfactory, Board's Committee on Development Satisfactory, Marginal and Unsatisfactory. Effectiveness. Independent Independent Evaluation Group's (IEG) objective is to Project Performance Assessment Evaluation enhance development effectiveness through excellence reports are produced for a sample Group (IEG) and independence in evaluation. One of its tools is Project of projects. These are used as Reviews: one in four completed projects is subject to a building blocks for Sector and Project Performance Assessment. An audit takes about Thematic Reviews, for the Annual six weeks to complete. Project Performance Assessment Review of Operations Evaluation, reports rate projects in terms of their outcome (taking into and for the Annual Review of account relevance, efficacy, and efficiency), sustainability Development Effectiveness. of results, and institutional development impact. IEG reports to the Bank's Board of Executive Directors. GN12 Actors 2 Toolkit for monitoring and evaluation of AWM Guidance Note 12 projects Actors in M&E of AWM Projects Who Responsibility at different stages Main Bank outputs World Bank The World Bank Institute (WBI) supports the World Bank's Institute (WBI) learning and knowledge agenda. Its evaluation unit provides training and technical assistance in M&E, both for Bank staff and client countries counterparts. Training events are funded by the Bank Learning Board and Trust Funds. Stakeholders The World Bank actively promotes stakeholder participation in the project cycle in order to improve project quality. Operations Policy and Country Services guidelines recommend use of Participatory Monitoring and Evaluation. Consultants, Design or implementation of M&E components can be research contracted out to national or foreign actors. institutes, universities, NGOs Source: Authors Websites World Bank Institute: www.worldbank.org/wbi World Bank Operations Policy and Country Services: http://go.worldbank.org/68SVPDDGV0 World Bank Quality Assurance Group: http://go.worldbank.org/J4OO0PFYM0 World Bank Social Development Department. Participation and Civic Engagement: http://go.worldbank.org/FMRAMWVYV0 GN12 Actors 3 Toolkit for monitoring and evaluation of AWM Guidance Note 13 projects World Bank Procedures for Project M&E Guidance Note 13 World Bank Procedures for Project Monitoring and Evaluation Monitoring and evaluation and the World Bank project cycle The projects the World Bank finances are conceived and supervised according to a well documented project cycle, the main stages of which are outlined in Figure 1 (see GN12 for the role of different actors). As these stages must now be applied in the context of the results management framework of the Bank, the focus of monitoring and evaluation (M&E) has shifted from monitoring implementation to tracking results (see GN1). M&E information is key at many of these stages, in particular previous project results should inform project identification, and supervision and completion reporting should be based on M&E information. Main stages in the project cycle (i) Country Assistance Strategy The World Bank's Country Assistance Strategy (CAS) forms the blueprint for its assistance to a country. The CAS sets out a four year selective program of Bank Group support linked to the GN13 WB procedures 1 Toolkit for monitoring and evaluation of AWM Guidance Note 13 projects World Bank Procedures for Project M&E country's development strategy. Since 2002 the CAS is based on the national Poverty Reduction Strategy Paper (PRSP), where available (see GN10). (ii) Identification During the identification phase, World Bank teams work with the government to identify projects that can be funded. Once a project has been identified, the Bank team creates a Project Concept Note (PCN) which is an internal document of four to five pages that outlines the basic elements of the project, its proposed objective, likely risks, alternative scenarios to setting up the project, and a likely timetable for the project approval process. The review meeting for the PCN should take place shortly after project identification, and will constitute the first, and often the most important, formal review stage for most operations. The proposed processing schedule for the operation should be reviewed and a decision taken as to whether processing (PCN review to Board approval) can be completed in the required time (on average, 12 months). (iii) Preparation The preparation phase, carried out by the borrowing country, can take anything from a few months to several years, depending on the complexity of the project being proposed. The major design, fiduciary, and safeguard issues are approved at the informal Quality Enhancement Review meeting, just prior to the pre appraisal mission or earlier. (iv) Appraisal During the appraisal phase, World Bank staff review the work done during identification and preparation, often spending three to four weeks in the client country. They prepare for World Bank management either Project Appraisal Documents (PAD) for investment projects or Program Documents for adjustment operations, and the Financial Management team assesses the financial aspects of the project. In 2004, the Bank moved from logical framework analysis as a mandatory requirement for inclusion in the PAD to use of the Results Framework and specification of arrangements for results monitoring (data sources, responsibilities, capacities, and costs; see GN1 for details). Box 1 details relevant sections of the PAD with regard to requirements for M&E. Whilst the terminology and the approach might be evolving, the results framework stays a simplified version of logical framework analysis (see GN1 and GN2). It focuses on the Project Development Objective (PDO), and intermediate outcomes/ results expected from implementation of each project component that contributes to overall achievement of the PDO. The added emphasis on outcomes helps focus management and design on results. It also forces the M&E system to track the outcomes and impacts. The results framework in the PAD (an example of which is given in GN1) requires that attention is paid to defining: · the PDO and project component statements, · indicators for the PDO and intermediate component outcomes, and · how the outcome information will be used. GN13 WB procedures 2 Toolkit for monitoring and evaluation of AWM Guidance Note 13 projects World Bank Procedures for Project M&E Box 1: Table of contents for the PAD ­ sections relevant to M&E A. STRATEGIC CONTEXT AND RATIONALE 3. Higher level objectives to which the project contributes How would the project contribute to borrower's higher level objectives? Relevant CAS objective(s)? Higher level outcomes are beyond the responsibility of the project and do not require M&E arrangements within the project. B. PROJECT DESCRIPTION 1. Project development objective and key indicators If the project is successful, what will be its principal outcome for the primary target group? How will progress toward achieving this principal project outcome be measured? 2. Project components For each component, what is the principal target group and the main project related outcome for that group? For each component, what are the key inputs and outputs? C. IMPLEMENTATION 3. Monitoring and evaluation of outcomes/results Where will the data for the project's outcome and results indicators come from? Where will the capacity and responsibility for collection of indicator data and analysis of results be located? Do capacities have to be strengthened? If so, how? What additional costs are required, if any? What mechanisms will allow the indicators to be used by managers and policy makers to assess the project's effectiveness during implementation and after the project is completed? D. APPRAISAL SUMMARY 4. Social How will the main social impacts of the project be monitored? 5. Environment Any important environmental issues? How will the main environmental benefits be monitored? TECHNICAL ANNEX 3. Results framework and monitoring (Recommended length 2 4 pages) The results framework focuses on the PDO to be achieved and the intermediate outcomes expected. The results framework matrix indicates objectives and intermediate outcomes/ results, indicators, and use of monitoring information. While PDO indicators normally cannot be observed or measured before the end of the project, intermediate outcomes capture performance that can be observed or measured while the project is still under implementation, and focus on the performance of key actors, and the value they add toward achieving the PDO. The results framework presents one or more indicators to measure success in achieving the PDO and one or more indicators to measure achievement of intermediate outcomes. The indicators should be presented with baseline values and target values. The section on arrangements for project monitoring should discuss the institutional and data collection arrangements for integrating monitoring and evaluation at the outcome/results level (both intermediate and end of project) into project management. This includes: Institutional issues: How will monitoring and evaluation complement project management? How will participatory M&E be integrated into management and capacity building for the communities involved (if applicable)? Data collection: If the project is drawing on data collected by Government statistical offices or line agencies, which statistics would be used and what is the reliability of this information? Where information is to be derived specifically for measurement of project results and outcomes, what are the associated costs and institutional responsibilities? Capacity: Where there is limited capacity in the country to derive the necessary information, how will local capacity be supplemented through the project, and what will be the costs of doing this? Source: World Bank OPCS, 2004 (v) Negotiation and Board Approval At the negotiation and approval phase, the Project Appraisal Document or the Program Document, along with the Memorandum of the President and legal documents, are submitted to the Bank's Board of Executive Directors for approval. The appropriate documents are also submitted for final clearance by the borrowing government. GN13 WB procedures 3 Toolkit for monitoring and evaluation of AWM Guidance Note 13 projects World Bank Procedures for Project M&E (vi) Implementation and Supervision The implementation of the project is the responsibility of the borrowing country, while the World Bank is responsible for supervision, which covers monitoring, evaluative review, reporting, and technical assistance activities. To assist in project implementation the borrowing country usually prepares a Project Implementation Plan (PIP) or Manual, which contains implementation arrangements and the implementation plan, and details key indicators for monitoring and evaluation. At least once a year, Task Teams should visit the sites of projects or representative samples of subprojects under implementation. Supervision missions should pay adequate attention to M&E. In particular, a core team member should have the responsibility to review M&E activities and ensure that project indicators are being monitored adequately. Specific areas of focus will be: M&E systems adjustment, review of M&E capacity and identification of training needs, assessment of response capacity from high level partners on M&E, and progress achieved on sensitive or key indicators. The Implementation Status and Results (ISR) report prepared annually is a concise summary of key project information (see Box 2). It includes a Status of Agreed Outcome Indicators table, which reports on indicators of progress towards achieving project development objectives, and intermediate outcomes and results (see example in Table 1). This information is normally provided by the M&E system put in place by the project. An aide mémoire to the ISR report details findings, decisions, and recommendations, and proposed remedies agreed jointly with the borrower. In addition to regular supervision activities and periodic site visits, it is useful to carry out an early review, after the first year of project implementation, and a comprehensive mid term review. Box 2: ISR report ­ reporting on performance indicators The ISR report requires reporting on indicators (as detailed in the PAD or specified subsequently) of: · Progress toward achieving project development objectives, and · Intermediate outcomes. Indicators can be adjusted but the ISR should adequately report on key information to assess project progress toward achievement of project development objectives: · The indicators should be the same as identified in the PAD. Projects are not required to be "retrofitted" with new indicators and measurement systems, but if such upgrading is important and can be done cost effectively, it should be pursued with the client. Changes should be specified in the ISR report, and approved by the manager. Substantial changes, such as changes of the project development objective, would require Board approval. · Some baseline data, on at least one indicator, should be presented in the first ISR report (see GN7). Any delay in data collection should be justified. The second ISR report should include baseline data for all indicators in the Status of Agreed Outcome Indicators table. · Where many intermediate outcome indicators were identified in the PAD, the Task Team should select a limited number of key ones which best track progress towards achieving project development objectives. These indicators are included in the Status of Agreed Outcome Indicators table. Indicators of minor importance can go in an annex of the aide mémoire. Source: World Bank OPCS. 2005b. GN13 WB procedures 4 Toolkit for monitoring and evaluation of AWM Guidance Note 13 projects World Bank Procedures for Project M&E Table 1: Example of ISR reporting, Alborz Integrated Land and Water Management Project, Iran Status of agreed outcome indicators Measurement Insert the measured value, or a qualitative indicator, or a brief explanation of why Indicators indicators are not available, together with the date of the information End-of-project target Baseline value Progress to date number Number or text Date Number or text Date Number or text Date PDO Indicators Cropping intensity 20 09/30/04 20 12/12/06 34 10/31/12 (%) Intermediate outcome indicators Area covered by 0 09/30/04 0 12/12/06 52000 10/31/12 rehabilitated or new irrigation and drainage systems (ha) Number of 9 09/30/04 21 12/12/06 25 10/31/12 competent operational staff and consultants Quality of the information on outcomes [If poor, mention in Issues and Actions for Management Attention] Good Fair X Poor Last ISR This ISR Progress toward achievement of PDO XX XX Ratings:HS=Highly Satisfactory; S=Satisfactory; MS=Moderately Satisfactory; MU=Moderately Unsatisfactory; U=Unsatisfactory;HU=Highly Unsatisfactory; NA=Not Applicable; NR=Not Rated PDO rating explanation: (Explain the basis of the DO rating (likelihood of achieving project development objectives) based on implementation ratings and outcomes so far. If the DO rating is MU, U, or HU, what actions are to be taken and what are the target dates?) Source: Project ISR report, 12/27/2006 The Quality Assurance Group (see GN12) conducts annual reviews of the quality of operational products from a sample of projects. Quality at Entry is assessed shortly after Board approval. Quality at Entry is assessed: as to whether project objectives are worthwhile and the risks commensurate with potential rewards; whether the project is likely to achieve its objectives; and whether the underlying logic is clearly articulated. The Quality of Supervision review looks in particular at the degree to which project performance is assessed realistically and reported candidly, with particular emphasis on achieving objectives. It also assessed whether emerging problems are being addressed promptly and proactively, incorporating global best practices adapted to country circumstances, and whether adjustments are made to project design to suit changing circumstances. GN13 WB procedures 5 Toolkit for monitoring and evaluation of AWM Guidance Note 13 projects World Bank Procedures for Project M&E (vii) Completion At the end of the loan disbursement period (anywhere from 1 10 years), the Task Team prepares the Implementation Completion and Results (ICR) report, which identifies accomplishments and problems. It is submitted to the Bank Board of Executive Directors for information purposes, no later than six months after closing of the loan. The ICR reports on achievement of outputs, intermediate outcomes/ results, and project development objectives, for which M&E information is key. It also evaluates factors that influenced these results and identifies lessons learned. The ICR can be relevant for both internal ­ Board members, World Bank managers and staff ­ and external ­ partners, stakeholders, the general public ­ audiences. It intends to provide a complete account of the performance and results of the project, to capture and disseminate experience, provide accountability and transparency, and provide a realistic self evaluation of performance. Poor M&E information and disruption of data collection efforts make writing a thorough ICR report very difficult. In the Macedonia Irrigation Rehabilitation and Restructuring Project for example, Crop Budget Surveys were identified as the main tool to monitor project impacts but were carried out only during the first three years of project implementation. The lack of surveys after the third year, when the major reforms supported by the project took place, was very limiting for the preparation of the ICR. Annex 1 of the ICR report specifically shows the progress on key performance indicators for impact and outcome, as identified in the PAD (see Table 2 for an example). Annex 4 of the ICR details delivery of outputs, component by component, under the project, and can also display physical and financial targets achieved for each activity17. Table 2: Some key performance indicators at completion for Rural Poverty Alleviation and Natural Resources Management Project, Brazil Indicator Baseline Optimal target values Formally Actual value achieved at value (from appraisal revised target completion or target year documents) values (a) PDO Indicator Indicator 1: Number of beneficiary families in poor communities supported Value (quantitative or 0 255,000 482,576 qualitative) Date achieved 02/11/1998 03/31/2006 Comments (including Over 482,576 families without repetition were supported in a total area of 9.02 million ha % achievement) which is 1.11 times larger than the PAD's target figure of 8.1 million ha. Achievement 189% Indicator 5: Kilometers of improved rural feeder roads Value (quantitative or 0 8,000 6,000 2,168 qualitative) Date achieved 02/11/1998 07/11/2002 03/31/2006 Comments (including The original target was reduced due to difficulties in carrying out rehabilitation. % achievement) Achievement of original target 27% Achievement of revised target 36% (b) Intermediate outcome indicator(s) Indicator 1: Training events for beneficiaries Value (quantitative or 0 4,683 8,529 qualitative) Date achieved 02/11/1998 03/31/2006 Comments (including 680,726 beneficiaries were trained in 8,529 events (up from original estimate of 165,360). % achievement) Due to the extension of closing dates a larger amounts of events took place. Achievement 182% Source: Project ICR, 2006 17See World Bank OPCS, 2007 GN13 WB procedures 6 Toolkit for monitoring and evaluation of AWM Guidance Note 13 projects World Bank Procedures for Project M&E The Bank's Independent Evaluation Group (IEG) reviews ICR reports, validates the self rating, and selects about one in four completed projects which might have potential for further learning. For such projects, IEG conducts an audit to measure its outcome against the original objectives. The Project Performance Assessment takes about six weeks to produce and normally includes a field mission. These reports rate projects in terms of their outcome (taking into account relevance, efficacy, and efficiency), sustainability of results, and institutional development impact. Box 3: Impact Evaluation, a new Bank product (viii) Evaluation Task Teams are encouraged to make IE is an AAA Bank product since 2005. The IE has the provisions for impact evaluation of objective of establishing through a rigorous analysis the their projects. A Development Impact changes in outcomes that can be attributed to a specific Evaluation Taskforce is in charge of intervention. The IE is led by an evaluation specialist working in close liaison with the Task Team, and should the Development Impact Evaluation feed in (if applicable) to the mid term review and the ICR. Initiative, and has made funding The IE is different to IEG evaluations in that their available to design evaluation responsibility is to evaluate Bank's operations ex post frameworks18. As part of the Bank's along many dimensions ­ e.g. the relevance of each effort to support and scale up the work operation and its alignment within a country's development on impact evaluations, Operations strategy, and the extent to which the objectives of the Policy and Country Services (OPCS) operations have been achieved and are sustainable. An IE established Impact Evaluation as a seeks to evaluate the impact of an intervention on the basis new product line, under the AAA of a sound counterfactual. umbrella (see Box 3).19 GN9 reviews Source: World Bank OPCS, 2005a Impact Evaluation in detail. The impact of some projects may also be reviewed briefly in the context of broader evaluations, such as CAS Completion Reviews, IEG Country Assistance Evaluation, or the Global Monitoring Report. Core World Bank procedures and requirements Core World Bank procedures and requirements are established in Operational Directives (OD), Operational Procedures (OP) and Bank Procedures (BP) of the Operational Manual. They highlight stages of identification to Board presentation, arrangements for supervision, implementation completion reporting, as well as basic guidelines on Monitoring and Evaluation. These procedures and directives have been updated by Operations Policy and Country Services processing guidelines and policies, as an initiative to modernize and simplify investment lending documentation. OPCS introduced in particular the Project Concept Note, Project Appraisal Document, and Project Supervision Reporting guidelines. A number of documents have been modified recently to reflect the introduction of the results management framework. The various elements associated with each of these components are presented in more detail in Table 3, at the end of this guidance note. Finally, these corporate requirements and procedures are also complemented by an array of guidance material made available on different topics. 18 The World Bank identified several bottlenecks that limit its ability to conduct impact evaluations at the necessary scale and with the needed continuity: insufficient resources, inadequate incentives, and, in some cases, lack of knowledge and understanding. To address these bottlenecks, the Development IMpact Evaluation (DIME) Initiative is a Bank wide collaborative effort under the leadership of the Bank's Chief Economist that is oriented at: (1) increasing the number of Bank projects with impact evaluation components, particularly in strategic areas and themes; (2) increasing the ability of staff to design and carry out such evaluations, and (3) building a process of systematic learning on effective development interventions based on lessons learned from completed evaluations. See GN9. 19See World Bank OPCS, 2005a. GN13 WB procedures 7 Toolkit for monitoring and evaluation of AWM Guidance Note 13 projects World Bank Procedures for Project M&E Table 3: Main corporate requirements and procedures with regard to M&E Title Type/ source Date Content in link with M&E OP/BP 10.00, Operational 1994 The different stages in the process are: Investment Lending: Manual Identification Identification to Board Preparation and Preappraisal Presentation Appraisal Negotiations Preparation of Final Documents Board Presentation Project Concept Note OPCS, policies 2003 The Project Concept Note is very short and focuses on and guidance project design. It includes proposed project development objectives. Project Appraisal OPCS, policies 2004 Project Appraisal Document guidelines were revised in 2004 Document and guidance to focus on results oriented design and monitoring. The main text of the PAD focuses on key points and includes sections on project development objectives and key indicators, and on implementation arrangements for monitoring and evaluation of outcomes/results. The technical annex 3 on the results framework and monitoring details objectives and intermediate outcomes, indicators, and use of monitoring information. The indicators should be presented with baseline values and target values. BP 10.00, Annex B Operational 1994 The Project Implementation Plan contains implementation Elements of a Project Manual arrangements and the implementation plan, and details key Implementation Plan indicators for monitoring and evaluation. The Project Implementation Plan is not a requirement, but borrowers are encouraged to prepare one. OD 10.70 Project Operational 1989 Plans for monitoring and evaluation are to be included in all Monitoring and Manual Bank funded projects. Monitoring and evaluation are the Evaluation responsibility of the borrower's project management team. Design of the information system and its use for monitoring progress requires identification of the primary users, quantification of project objectives, and selection of a minimum core of quantitative and qualitative indicators for monitoring progress towards these objectives OP/BP 13.05, Project Operational 2001 Project supervision covers monitoring, evaluative review, Supervision Manual reporting, and technical assistance activities to ascertain whether the borrower is carrying out the project with due diligence identify problems promptly recommend changes identify the key risks prepare the Bank's Implementation Completion and Results Report The Task Team develops, and agrees with the borrower on, an overall plan for Bank supervision of the project. During project implementation, the Task Team regularly monitors progress, monitors procurement, reviews Project Monitoring Reports, ascertains the extent of compliance, assesses risks. Performance Problems. If the Task Team notes any significant deviations from the provisions of the Project Appraisal Document and the Project Implementation Plan, it discusses with the borrower the steps required to get the project back on track. In addition to regular supervision activities and periodic site visits, it may be useful to carry out an early review after the first year of project implementation; and a comprehensive midterm review. GN13 WB procedures 8 Toolkit for monitoring and evaluation of AWM Guidance Note 13 projects World Bank Procedures for Project M&E Supervision OPCS, policies 2005 Planning for supervision begins during project preparation Guidelines, and guidance and must include the design of appropriate monitoring and Implementation Status evaluation systems. and Results Report The Implementation Status and Results report is a concise summary of key project information. Ratings should reflect the project's status in terms of implementation progress, and achievement of stated project objectives. It is recommended that the ratings for implementation progress and development objectives should be supported by data on key performance indicators. ISR Supplementary OPCS, policies 2005 The Key Performance Indicator Annex in the old Project Guidance: Reporting and guidance Status Report has been replaced with a table called Status of on Progress Towards Agreed Outcome Indicators, in line with the results Outcomes framework approach in the new Project Appraisal Document format. The new section in the main Implementation Status and Results report requires reporting on indicators (as detailed in the Project Appraisal Document or specified subsequently) of: -Progress toward achieving project development objective(s), -and Intermediate outcomes. OP/BP 13.55 Operational 1999 In no case is the Implementation Completion and Results Implementation Manual Report circulated to the Board later than six months after Completion Reporting loan closing. The report includes Principal Performance Ratings, Achievement of Development Objective and Outputs. Source: Authors Further reading Kusek, J.Z., and R.C. Rist. 2004. Ten Steps to a Results-Based Monitoring and Evaluation System. . Washington D.C.: World Bank. Available at http://go.worldbank.org/7H1YFQS5Z0 World Bank Independent Evaluation Group. 1995. Performance indicators in Bank-financed agricultural products: a first edition note, Volume 1. Working paper. Independent Evaluation Group. Washington D.C.: World Bank. World Bank Operations Policy and Country Services. Designing M&E for Investment Operations. Washington D.C.: World Bank. Available at http://go.worldbank.org/RLQKPSE1F0 World Bank Operations Policy and Country Services. The World Bank Operational Manual. Washington D.C.: World Bank. Available at http://wbln0018.worldbank.org/institutional/manuals/opmanual.nsf/05TOCpages/The%20World% 20Bank%20Operational%20Manual World Bank Operations Policy and Country Services. 2005a. Implementing Impact Evaluations at the World Bank: Guidance Note. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTISPMA/Resources/Training-Events-and- Materials/IEGuidanceNoteFINALApril06.pdf World Bank Operations Policy and Country Services. 2005b. ISR guidelines. Washington D.C.: World Bank. Available at http://go.worldbank.org/GTP7W9DOK0 World Bank Operations Policy and Country Services. 2007. Implementation Completion and Results Report Guidelines. Washington D.C.: World Bank. Available at GN13 WB procedures 9 Toolkit for monitoring and evaluation of AWM Guidance Note 13 projects World Bank Procedures for Project M&E World Bank. 2006b. Impact Evaluation and the Project Cycle. Doing Impact Evaluation Series No. 1. Thematic group on poverty analysis, monitoring and impact evaluation. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTISPMA/Resources/383704- 1146752240884/doing_ie_series_01.pdf Websites World Bank. Project cycle: http://go.worldbank.org/GI967K75D0 World Bank Operations Policy and Country Services. Policies and guidance for Investment Lending: http://go.worldbank.org/PPQEM510J2 World Bank Development Impact Evaluation Initiative: http://go.worldbank.org/1F1W42VYV0 World Bank Sustainable Development Network Results website: http://go.worldbank.org/ZG0PRX9260 GN13 WB procedures 10 Toolkit for monitoring and evaluation of AWM Guidance Note 14 projects M&E for Rehabilitation and Modernization Components of AWM Projects Guidance Note 14 M&E for rehabilitation and modernization components of agricultural water management projects Introduction Rehabilitating the physical infrastructure is often one of the main pathways to improving the performance of irrigation and drainage (I&D) systems. The needs for rehabilitation may have arisen as a result of a numbers of root causes, all of which have resulted in a failure to invest sufficiently in maintenance of the I&D system. Modernization is also often a part of projects, with new works being constructed or changes made to upgrade and modernize the I&D systems in order to improve system operation and maintenance. The returns to rehabilitation and modernization can be immediate and significant, though it is important to note that these benefits can often be short lived if associated institutional measures, such as improved management, operation and maintenance procedures, including service fee recovery, are not also put in place to complement the physical improvements (see GN15, GN16, and GN17 for more details). The physical improvements are expensive, in many projects the largest single component is for physical works. It is important therefore that the rehabilitation and modernization process is properly monitored and evaluated. Key activities in rehabilitation and physical modernization of I&D systems Table 1 lists some of the typical components of irrigation and drainage system rehabilitation and modernization, flood protection and dam safety works. The rehabilitation and physical modernization may be for one system, or for a number of systems, and will follow the general process of: · Survey · Outline design · Preliminary cost estimate and benefit · Selection/prioritisation · Detailed survey · Detailed design · Detailed cost estimate · Tendering · Construction · Commissioning The initial survey, outline design, cost estimate and preliminary selection (if more than one scheme is involved) may have been carried out during the project feasibility study, in which case the project will commence with the detailed survey followed by detailed design and the letting of tenders. The construction work may be let to one contractor, or to several, depending on the scale and locations of the works. GN14 Rehabilitation & Modernization 11 Toolkit for monitoring and evaluation of AWM Guidance Note 14 projects M&E for Rehabilitation and Modernization Components of AWM Projects Rehabilitation and modernization interventions are generally a compromise between priority improvements and complete upgrading, dictated by economics and affordability. The technical integrity of proposals is always a fundamental prerequisite. All procedures must be open and transparent. Table 1: Typical rehabilitation components of agricultural water management projects Component Project activities Rehabilitation and Main systems (primary and secondary canals and drains): modernization of Remove sediment from canal sections irrigation and drainage Reform canal sections systems Line canals Rehabilitate canal embankments Rehabilitate headworks Rehabilitate pump stations Rehabilitate structures Rehabilitate/install new control structures Rehabilitate/install new measuring structures Remove sediment and vegetation from drains Reform drain sections Construct new drain sections Rehabilitate/install drain structures Rehabilitate/install drainage pumping stations On farm: Rehabilitate/construct new farm irrigation channels (lined or unlined) Rehabilitate/construct new on farm control structures Rehabilitate/construct new farm open drainage channels Rehabilitate/install new buried drainage system Install piezometers (for groundwater monitoring) Land levelling Land consolidation Rehabilitation or Repair/construct new river training works construction for flood Repair/construct new flood protection embankments protection Repair/construct new structures associated with flood protection works (intakes, drain outfalls, etc.) Rehabilitation and Rehabilitate/construct new spillway upgrading of dams Rehabilitate gates and control structures Repair dam embankments Repair dam core Rehabilitate/upgrade/install new monitoring equipment Rehabilitate/upgrade/install new flood early warning system Reafforest upstream catchment Soil erosion works in upstream catchment Source: Authors Key performance indicators, data needs, and analysis Overview Typical objectives and related performance indicators for rehabilitation and modernization work are summarised in Table 2. Implementation monitoring of the planning, design and construction phase is particularly important and is discussed in more detail in later sections. GN14 Rehabilitation & Modernization 12 Toolkit for monitoring and evaluation of AWM Guidance Note 14 projects M&E for Rehabilitation and Modernization Components of AWM Projects Table 2: Typical implementation and results framework for rehabilitation and modernization works Assessment level Examples Project development Improved and sustainable increase in irrigated agricultural productivity objective Project outcome(s) · More reliable, timely and adequate irrigation water supplies to all parts of the irrigation network · More reliable, timely and adequate drainage of the irrigated area · Increased security from flooding · Reduced levels of danger and risk (particularly from dam failure) Project outputs · Fully functioning irrigation system · Fully functioning drainage system · Secure and functioning flood protection works · Secure and functioning dam and reservoir storage Project activities · See Table 1 Project inputs · Survey, design, construction and construction supervision personnel · Beneficiary participation · Construction materials Source: Authors For I&D systems the output of the rehabilitation and modernization work is to return the system(s) to a fully functioning state in which irrigation water can be conveyed, controlled and delivered to the tertiary unit outlet, or to the field. Figure 1(a) outlines the situation that often occurs within a run down irrigation, where water supplies are inadequate, inequitable and unreliable as a result of damaged conveyance systems and control structures. In this case the top enders have excess water, with some of the water being returned to the drainage system. The tail enders have insufficient or no water with the result that their level of production is low. Often in such cases the tail enders grow lower value crops which are not so dependent on irrigation water. Following rehabilitation (Figure 1(b)) the conveyance systems and control structures are repaired, whilst the system has been modernized (in this example) by the addition of measuring structures at key locations. Irrigation water supplies are the same for all parts of the system, with tail enders now having the opportunity to crop higher value crops and to obtain production levels comparable with that at the top end of the system. Again it must be emphasised that the improvement in the water delivery depends also on improvements in the management, operation and maintenance (MOM) of the system; this is largely an institutional/organisation issue relying on the capabilities of the MOM personnel (see GN15, GN16, and GN17). GN14 Rehabilitation & Modernization 13 Toolkit for monitoring and evaluation of AWM Guidance Note 14 projects M&E for Rehabilitation and Modernization Components of AWM Projects Implementation monitoring Due to the importance of the planning, design and construction phase relatively detailed information is provided below on implementation monitoring and evaluation. There are well established procedures for the monitoring and evaluation of the construction of physical components of rehabilitation projects. These primarily focus on the quantity, quality, cost and timing of the works carried out, with a relatively simple set of performance indicators required. Key tools in the M&E process are project management packages, such as Microsoft Project, or spreadsheets in which the planned and actual situation can be recorded, plotted and compared. Examples are provided below of the monitoring carried out for the World Bank funded On Farm Irrigation Project in Kyrgyzstan, with data taken from the 6 monthly review reports (World Bank, 2006c). These data track the progress of the design and construction components of the project, and serve to establish if the project is on track, and to identify shortfalls and problems. A key aspect of this particular project, at this time, is the careful monitoring of the expenditure as the project nears its end. The project combines physical rehabilitation of the on farm works with formation of Water Users Associations (WUAs), with some 450 WUAs formed throughout the country, of which 63 have reached a stage that they are eligible for on farm rehabilitation works. The total area to be rehabilitated is 120,400 ha at a total cost of US$18.9 million. The average rehabilitation cost is US$157/ha, including both on farm and off farm20 costs. Though the main costs are associated with construction, the costs of the design work need to be monitored where possible. In this case some of the designs were sub contracted to two design companies, one in the north and one in the south of the country21. The progress of the designs was 20 On farm relates to works within the tertiary unit, off farm relates to works in the main system (primary and secondary canals and drains) 21 The designs were sub contracted out to these companies in 2003 due to delays and design quality problems with in house project design teams. The subsequent improvement in design quality and speed of completion of GN14 Rehabilitation & Modernization 14 Toolkit for monitoring and evaluation of AWM Guidance Note 14 projects M&E for Rehabilitation and Modernization Components of AWM Projects monitored and the final evaluation of the completed designs enables analysis of the cost of each design (Table 3), and identification of the reason for higher, or lower, design costs. The higher costs of the designs in Naryn Oblast, for example, are due to the mountainous nature of the region and the difficulties with access for surveys and discussions with farmers. The quality of the design work was monitored by the World Bank mission, who engaged an experienced design engineer to monitor the design work. The gap between the number of designs done (76 No.) and the actual contracts let (63 No.) is due to the time lag between design, tendering of contracts and calculation of total contract costs, which were found to be higher than expected due to increased construction costs. Close monitoring of this situation identified the need to stop design work proceeding any further. Table 3: Example of summary table of current status of design contracts Companies a/ Cost/design No b/ US$ US$ Osh 13 82,710 6,362 Jalalabad 13 65,347 5,027 Batken 8 38,733 4,842 Issk Kul 5 49,539 9,908 Naryn 7 78,062 11,152 Talas 3 17,734 5,911 Total 49 332,125 6,778 Note: a/ Dolboor in south, R&R in north/center b/ Number of completed designs; some design work by others Source: World Bank, 2006c The project data are summarised on Table 4; the data are for sub projects in one of the six Oblasts (regions) in the project area. The columns show all the data required to monitor these sub projects, with the area irrigated, the contracted cost, current actual expenditure, value of the works completed to date, amount remaining, remaining cost allowing for possible price increases, start date, planned end date, extension dates (if applicable), delay and name of the contractor. Table 5 presents a summary of the project status at the time of the review (in this case June 2006). The data show that 6 contracts have had problems and have been cancelled, leaving 63 sub projects. with an overall average rehabilitation cost of US$157/ha. However, the average price of the last six contracts let was US$375,000, or US$197/ha reflecting the increased cost of construction materials. Figures 2 and 3 provide an example of the monitoring carried out on the completion of sub projects during the project period. They show a slow build up in 2003 2004, a peak in 2005 and 2006 and a gradual decline as 11 sub projects are completed in 2008, making a total of 63 sub projects completed in all. The financial disbursements required to match these contracts need to be carefully monitored. Figures 4 shows the actual disbursement for the rehabilitation works by month for years 2004 2006, whilst Figure 5 shows the planned and actual disbursements by quarter for 2004 2006. Figure 4 shows a relatively regular expenditure (shown by the relatively regular gradient of the lines), whilst Figure 5 shows that initially the actual expenditure lagged behind the planned expenditure (showing delay in the works), whilst from the last quarter of 2005 to the second quarter of 2006 actual expenditure is well ahead of the plan, showing an increase and acceleration of the works. The steeper gradient of the curves in Figure 4 show that the monthly disbursements are increasing year on year. In 2004 the average disbursement was US$220,000, rising to US$300,000 in 2005 and US$431,000 in the first half of 2006. Overall, however, in the example presented, the total disbursement is US$11.1 million, 56% of the planned total, whilst 88% of the planned project period has elapsed. Even though the rate of disbursement is increasing, the works are behind schedule at this stage and will require an extension of an additional 14 months for completion (based on the current rates of progress). the designs has justified this approach to solving the problem. The monitoring and early identification of this problem was central to project completion. GN14 Rehabilitation & Modernization 15 Toolkit for monitoring and evaluation of AWM Guidance Note 14 projects M&E for Rehabilitation and Modernization Components of AWM Projects Table 4: Project implementation monitoring spreadsheet sutatS delecnac etelpmoc etelpmoc gniogno delecnac gniogno gniogno gniogno etelpmoc gniogno etelpmoc gniogno gniogno gniogno gniogno etelpmoc gniogno ete 60 %48 %19 %68 %07 %12 %55 %73 %69 %16 %49 %32 %0 %56 %07 % pl omC unJ03 %811 %611 %011 ete 50 %48 %28 %08 %63 %54 %82 %78 %14 %79 %96 %14 %29 %62 % pl omC ovN42 mr Fa-nO ah/$t 42 05 411 993 25 16 112 032 511 001 59 212 16 65 152 97 58 081 341 osC t 000' 1.97 1.261 9.212 7.184 6.39 3.79 3.563 9.783 0.081 0.181 6.161 4.434 7.302 3.881 7.263 4.121 7.032 5.772 osC DSU 122,4 otsk b/ 60 055,3 205,6 801,8 242,31 989,3 699 109,7 132,2 091,7 803,4 208,6 394 477,8 706,1 182,3 825,11 714,7 919,79 or W unJ03 '000 SGK,t a/ osC desiver 055,3 867,6 180,9 745,91 989,3 088,3 857,41 455,51 115,7 433,7 148,6 502,31 477,8 584,7 583,01 669,4 866,9 242,11 735,461 l na 320,7 867,6 180,9 745,91 164,8 494,31 465,51 675,6 158,6 148,6 502,31 889,31 311,01 718,4 866,9 242,11 giiro 732,361 )t gniseD oscesab( 146,7 467,5 268,9 444,31 618,8 921,51 380,51 252,7 562,7 867,7 243,41 023,51 041,21 883,5 528,9 375,11 116,661 tcar oN ontC 10 40/10 50 24 81 )2(81 33 85 90 23 61 06 51 )2(51 16 62 42 74 PIO PIO PIO PIO PIO PIO PIO PIO PIO PIO PIO PIO PIO PIO PIO PIO PIO PIO ) AU ha(aerA 162,3 768,1 730,1 395,1 437,1 096,1 075,1 308,1 807,1 740,2 563,3 644,1 835,1 831,2 245,1 W 933,82 tcartnoc detagirrI tsa blO 354,24 353,22 667,62 143,12 393,431 )etad on/yaR ehtfosetadnoitelp la ott TCEJORP ub-s NOITAGIRRI stce tcartnoc2 gnitroperehtfosa(0057,93 ) GSK=1DSU moclanigiroehthtiwnosirap ojrP 60 onyaR delecnactcartnoc tcartnocdn2 ub-S onyaR iraDy e tayaH tako lamhkaB egnahcxEfo egnahcxefoetartnerrucnodesab)egnahcxefoetar MR nuJ 03 AU TS ubO uuSakhsaK ordyHtygnaM onyaRta oNsolooT takooNsolooT risA yegnuKkryK onyaR loKny n tlA kutsoDloaraK :setoN etadotsnoisiverlasedulcni %51(ecnavdasedulcni etaR dna(tsoc gningistcartnocfoetad eciovnidevorppatsalfoetad mocni)skeewni(syaled FA-NO onitagirrI W LABO HSO uuS-araK tamhkaR tamhkaR kalapaJ kirAynaJ omohC tcartnoc2omohC vonidlolaJ lakiAzaM navarA haS ukaN gezU /a /b /c /d /e /f /g 1 1 2 3 4 5 6 7 8 9 01 11 21 31 41 51 GN14 Rehabilitation & Modernization 16 Toolkit for monitoring and evaluation of AWM Guidance Note 14 projects M&E for Rehabilitation and Modernization Components of AWM Projects Table 4: Project implementation monitoring spreadsheet (cont.) Implementation Dates Delay in weeksg/ actual on on No WUA start e/ end f/ extension completion original revised Bidders Contractor OSH OBLAST Kara-Suu Rayon 1 Rakhmat contract cancelled 5 Apr 02 31 Mar 03 31 Dec 03 cancelled 5 Uzgenvodstroi 1 Rakhmat 2nd contract 26 Feb 04 30 Apr 05 1 May 06 29 Jun 06 61 9 2 Karasuuayilkurulush 2 Japalak 26 Nov 02 7 Dec 03 5 Aug 05 6 OSH Suu 3 Jany Arik 25 Mar 05 25 Mar 07 4 Consortium Hydrostroitel 4 Chomo 20 Dec 03 1 Jun 05 1 Apr 06 56 13 4 Karasuusuukurulush Chomo 2 contract 20 Apr 06 31 Dec 06 Brik 5 Jaloldinov 27 Nov 04 22 Nov 06 4 Dostuk 6 Maz Aikal 20 Dec 05 20 Dec 07 5 Consortium Vodstroi Aravan Rayon 7 Sahy Darie 21 Jan 03 1 Feb 04 15 May 05 7 Apr 05 4 KVT 8 Obu Hayat 27 Nov 04 22 Nov 06 3 Selen 9 Kashka Suu 18 Dec 03 31 May 05 6 Apr 05 3 Inzhenernaya Zachita 10 Mangyt Hydro 22 Mar 06 22 Mar 08 5 Yugstroiservice Naukat Rayon 11 Toolos Nookat 3 Dec 03 1 Dec 05 30 3 Shakhtostroi Toolos Nookat 2 contract 30 Jun 06 1 May 07 Besh Batir 12 Asir 22 Mar 06 22 Mar 08 4 Jalal Abad SPMK 13 Kyrk Kungey 26 Apr 04 10 Nov 05 5 Jun 06 33 4 3 Temir Tash 20 Oct 06 Uzgen Rayon 14 Altyn Kol Bakhmal 26 Mar 04 1 Apr 06 19 May 06 13 3 Consortium Vodnik 15 Karaol Dostuk 18 May 05 18 Mar 07 4 Brik Notes: a/ includes all revisions to date b/ includes advance (15%) c/ Rate of Exchange USD1=KGS 39,7500 (as of the reporting date) d/ cost (and rate of exchange) based on current rate of exchange e/ date of contract signing f/ date of last approved invoice g/ delays (in weeks) in comparison with the original completion dates of the contract GN14 Rehabilitation & Modernization 17 Toolkit for monitoring and evaluation of AWM Guidance Note 14 projects M&E for Rehabilitation and Modernization Components of AWM Projects Table 5: Example of summary table of current status of rehabilitation contracts No Area '000 ha Cost No Area '000 ha Cost Contr Incl Cancel $ mn a/ $/ha b/ Contr Incl Cancel $ mn a/$/hab/ South North/Center Osh Talas cancelled 3 5.0 0.4 76 cancelled complete 5 10.5 0.9 90 complete 5 9.6 0.8 85 ongoing 10 17.8 2.9 163 ongoing 3 5.1 0.7 137 Osh 18 28.3 5.0 4.2 149 Talas 8 14.7 1.5 103 Jalal-Abad Issyk-Kul cancelled 2 4.1 0.4 88 cancelled 1 2.2 0.2 78 complete 4 7.4 0.7 92 complete 5 10.3 1.5 142 ongoing 9 20.3 3.4 168 ongoing 4 11.2 1.2 107 J-abad 15 27.7 4.1 4.5 161 Issyk-Kul 10 21.5 2.2 2.8 132 Batken Naryn cancelled cancelled complete 1 3.1 0.5 159 complete 2 2.5 0.2 71 ongoing 6 10.2 2.6 251 ongoing 9 12.5 2.7 213 Batken 7 13.2 3.0 230 Naryn 11 15.0 2.8 189 South North/Center cancelled 5 9.0 0.7 82 cancelled 1 2.2 0.2 78 complete 10 21.0 2.1 101 complete 12 22.4 2.5 109 ongoing 25 48.2 8.9 184 ongoing 16 28.7 4.6 158 South 40 69.3 9.0 11.7 169 North/Center 29 51.1 2.2 7.2 140 Project TOTAL cancelled 6 11.3 0.9 81 complete 22 43.4 4.6 105 ongoing 41 77.0 13.4 174 Project 69 120.4 11.3 18.9 157 Notes a/ actual costs for completed sub projects, current (working) estimate for ongoing sub projects b/ overall cost/ha including off farm works Source: PIU/OIP, Bishkek, July 2006 GN14 Rehabilitation & Modernization 18 Toolkit for monitoring and evaluation of AWM Guidance Note 14 projects M&E for Rehabilitation and Modernization Components of AWM Projects GN14 Rehabilitation & Modernization 19 Toolkit for monitoring and evaluation of AWM Guidance Note 14 projects M&E for Rehabilitation and Modernization Components of AWM Projects GN14 Rehabilitation & Modernization 20 Toolkit for monitoring and evaluation of AWM Guidance Note 14 projects M&E for Rehabilitation and Modernization Components of AWM Projects Plotting the percent of the financial progress against the time expended on a given contract is a very useful way of monitoring project progress. Figure 6 provides an example for one of the six Oblasts in the project. It shows that three of the 9 sub projects in this Oblast (OIP 25(1), OIP 21(2) and OIP 22(1)), are falling behind in the work programme as the percentage time elapsed on the sub project exceeds the percentage of work done (as shown by the financial payments to the contractor). Whilst in construction projects the financial expenditure and the time expended are not always aligned22, this form of monitoring can provide early warning of potential problems and delays in project execution. Results monitoring As discussed above the output of the rehabilitation and modernization of I&D systems are fully functioning systems, able to deliver irrigation water, and remove drainage water, in a reliable, timely and adequate manner. The outcomes of this work, if other requirements are satisfied, include: · Reduced seepage from canals and leakage from structures, resulting in increased conveyance efficiencies · Increased canal and drain capacity, able to take design flows · Increased security of supply (lowered risk of structure collapse or canal breach) · Improved control, enabling better distribution of available supplies · Improved measurement, enabling better distribution and monitoring of available supplies · Increased irrigated area · Increased crop production 22Some components of a construction contract may be of low financial value but take time, such as the construction of a diversion channel prior to the main work on a river weir, or the repair of a small aqueduct in an inaccessible location. In such cases the expenditure is low relative to the time taken. An experienced contractor will take account of such situations and factor them into the construction works programming and cash flow forecasting. GN14 Rehabilitation & Modernization 21 Toolkit for monitoring and evaluation of AWM Guidance Note 14 projects M&E for Rehabilitation and Modernization Components of AWM Projects Relevant performance indicators for results monitoring of I&D systems are provided in Table 6. The main indicators are those related to crop production, and value of crop production, either in total or per unit area or per unit water supplied. These indicators should improve following project completion; crop areas and intensities may increase, total abstractions may reduce due to higher conveyance efficiencies and better management, production per unit of water abstracted may be higher. The indicators can be determined for the whole system, or for parts of the system, such as tertiary units, in order to assess the reliability, adequacy and equity of water supply to all parts of the system. Environmental impacts of the project will need to be monitored, an important area for such monitoring is the quality of the drainage water. Salinity, chemical and biological monitoring should be carried out on a regular basis. For the indicators related to water delivery the total seasonal volume of water abstracted should change, either to be more if the main canal capacity has been increased (e.g. desilted) to take additional flow, or to be less if the conveyance efficiency and operation have improved to enable the same or more production with less water. As with the agricultural production indicators the water delivery indicators can be determined for the scheme as a whole, or for parts, such as tertiary units. The Seasonal23 Irrigation Supply per Unit Command Area (m3/ha) at the tertiary unit intake is a useful indicator of reliability, adequacy and equity, as is the Seasonal Relative Irrigation Water Supply which measures the water supplied to that required at a given location. The sustainability of the system is determined from the financial indicators, which measure the irrigation service fees collected and the expenditure on management, operation and maintenance. In some countries, such as Egypt, there are separate government organisations for irrigation and drainage, in which case the irrigation and drainage expenses may be kept separate. Expenditure on maintenance is a key indicator, as this is one of the key drivers of system sustainability. The values of these indicators (fee recovery, maintenance expenditure) should, in general, increase following rehabilitation and modernization, though in some cases the maintenance costs may reduce following rehabilitation as the system is cheaper to maintain. Whether the indicator values should increase or decrease will often depend on the pre rehabilitation situation, if the low maintenance expenditure prior to rehabilitation was the cause of the deterioration then the expenditure on maintenance post rehabilitation should increase. Ground water levels are dependent on drainage water removal and efficient irrigation application. The groundwater level is an important performance indicator; it may be worthwhile installing piezometers24 as part of the project in order to be able to monitor groundwater levels over time. 23 Could be seasonal or annual, depending on the cropping pattern. If there is more than one season and there are marked differences between the seasons' cropping patterns and water availability it is preferable to consider each season separately. 24 If it is a drainage project installing or repairing piezometers will normally be one of the project activities. GN14 Rehabilitation & Modernization 22 Toolkit for monitoring and evaluation of AWM Guidance Note 14 projects M&E for Rehabilitation and Modernization Components of AWM Projects Table 6: Key indicators for outcome monitoring and evaluation of I&D rehabilitation and modernization projects Indicators Definition Notes1 Agricultural production Total seasonal2 area cropped per unit Total seasonal area cropped a command area (Cropping intensity) Total command area of system Total seasonal crop production (Tonnes) Total seasonal crop production by crop type within command area a Total seasonal crop production per unit Total seasonal crop production a command area (crop yield, kg/ha) Total command area of system Total seasonal value of crop production Total seasonal value of agricultural crop production received by a ($) producers Total seasonal value of crop production Total seasonal value of crop production a per unit command area ($/ha) Total command area of system Total seasonal crop production per unit Total seasonal crop production a water supply (kg/m3) Total seasonal volume of irrigation water supply Total seasonal value of crop production Total seasonal value of crop production a per unit water consumed ($/m3) Total seasonal volume of crop water demand (Etc) Total seasonal value of crop production Total seasonal value of crop production a per unit water supplied ($/m3) Total seasonal volume of irrigation water supply Irrigation water delivery Total seasonal volume of irrigation water Total seasonal volume of water diverted or pumped for irrigation (not a supply (MCM) including diversion of internal drainage) Seasonal irrigation water supply per unit Total seasonal volume of irrigation water supply a command area (m3/ha) Total command area of system Main system water delivery efficiency Total seasonal volume of irrigation water delivery b Total seasonal volume of irrigation water supply Seasonal relative irrigation water supply Total seasonal volume of irrigation water supply a Total seasonal volume of crop water demand Water delivery capacity Canal capacity at head of system Peak irrigation water demand at head of system Financial Total seasonal MOM expenditure3 per unit Total seasonal MOM expenditure c command area ($/ha) Total command area of system Total seasonal MOM expenditure per unit Total seasonal MOM expenditure c irrigation water supply ($/m3) Total seasonal volume of irrigation water supply Total seasonal maintenance expenditure Total seasonal maintenance expenditure c per unit command area ($/ha) Total command area of system Total seasonal maintenance expenditure Total seasonal maintenance expenditure c fraction Total seasonal MOM expenditure MOM funding ratio Actual annual income d Budget required for sustainable MOM Fee collection ratio Irrigation (and drainage) service fees collected d Irrigation (and drainage) service fees due Farm profit Total farm income ­ total farm expenditure e Drainage water removal Average depth to groundwater (m) Average seasonal depth to groundwater calculated from water table f observations over the irrigation area Environmental protection Salinity of soil water (mmhos/cm) Electrical conductivity of soil water f Soil salinity (mmhos/cm) Electrical conductivity of soil f Salinity of water in open drain Electrical conductivity of water in open drains f (mmhos/cm) Drainage water quality: Biological Biological load of drainage water expressed as Biological Oxygen f (mg/litre) Demand (BOD) Drainage water quality: Chemical Chemical load of drainage water expressed as Chemical Oxygen f (mg/litre) Demand (COD) Notes: 1. Location and sampling interval: a. Determine for total command area and individual tertiary units b. Discharges measured at the main canal intake and tertiary unit intakes c. Determine for total command area, main system only and individual Water Users Associations d. Determine for individual service providers (government agency or Water Users Associations) e. For individual water users f. Periodic sampling at selected locations GN14 Rehabilitation & Modernization 23 Toolkit for monitoring and evaluation of AWM Guidance Note 14 projects M&E for Rehabilitation and Modernization Components of AWM Projects 2. May be seasonal or annual, depending on the circumstances. If there is more than one season and there are marked differences between the seasons' cropping patterns and water availability it is preferable to consider each season separately 3. Costs for irrigation water delivery and drainage water removal may be kept separate or combined; it depends if there is a separate drainage authority. Concluding remarks Some additional points that should be considered in the monitoring and evaluation of physical works for AWM projects are outlined below: 1. It will be important to monitor changes in the cost of construction material and input costs over the project period. In most situations a materials cost escalation clause and index will be built into the contract to allow for additional payments to the contractor to allow for escalation in the cost of basic materials; 2. It is important to use the monitoring data collected to plan early for any extension of the project. Projects often start slowly and speed up as initial problems and difficulties are overcome; this should be factored into the calculations when estimating any additional time required; 3. A key part of the monitoring process is discussion with project beneficiaries to establish if they are satisfied with the planning, design and construction works. Care should be taken with the selection of those interviewed in this respect to obtain an unbiased and representative sample; 4. A good sign, and an important indicator of success, are requests by other WUAs to be included in the project. These requests increase over time as WUAs see the benefits arising to neighbouring WUAs who are benefiting from project activities; 5. In projects similar to that described above, it is preferable to have a slow, yet steady start and to build confidence in the work being carried out. Hasty starts, with poor initial results, may deter WUAs from participating in the project, especially where they have to make a financial contribution to the work; 6. Failing contracts should be closed early rather than be allowed to drag on over time. Clear procedures for terminating failing contracts should be set out in the contract documents; 7. An important role for the supervising mission staff is the monitoring and evaluation of the quality and performance of the project's construction supervision team. Further Reading Lock, D., 1993. Project management. Fifth edition. Gower Publishing, Aldershot, UK. World Bank. 2006c. Mission Report: Irrigation Engineer. IDA 12th Review Mission, On Farm Irrigation Project, Kyrgyz Republic, July. GN14 Rehabilitation & Modernization 24 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components Guidance Note 15 M&E for Management, Operation, and Maintenance Components Introduction Improving the management, operation and maintenance of the irrigation and drainage (I&D) systems is often included in project design in order to support and make best use of physical rehabilitation/ modernization of the system. This component of a project can cover a number of activities ranging from changing the culture of the government I&D agency, to improved processes and procedures for water delivery and use. The terms management, operation and maintenance (MOM) are used to cover the following processes and procedures. The terms apply equally to systems, or parts of systems, managed by government agencies or Water Users Associations: Management covers the overall management and administration of the I&D system. It includes: day to day administration of the organisation; hiring of staff; management of staff, liaison and communication with water users, government agencies, etc.; annual planning and budgeting; accounting; performance appraisal (of the organisation's activities and the personnel). Operation covers the operation of the irrigation and drainage systems, and includes: seasonal water supply and demand planning; irrigation scheduling; water allocation; regulation of Box 1: Typical project activities for improving I&D gates and control structures; system management, operation and maintenance measurement of discharges; control and operation of the water source (intake Study options for reorganising/restructuring of structures or pump stations); liaison and government I&D agencies communication with water users; Establish a service delivery culture within the I&D operational performance assessment. agency Establish or update procedures for management and Maintenance covers the maintenance of administration of I&D systems, incorporating the use of the irrigation and drainage system, and modern technology (computers, communication systems, etc) includes: planning, budgeting, inspection, Study and develop, or update, norms for financial designing, costing, implementing, requirements for sustainable management, operation supervising and recording of maintenance and maintenance (MOM) of I&D systems work; liaison and communication with Establish systems for setting and recovery of Irrigation water users on maintenance needs and Service Fees (ISFs) activities; maintenance performance Establish or update procedures for operation of I&D assessment. systems, both at the main system and on farm level Establish, or update, processes and procedures for maintenance of I&D systems Key activities for improving Establish asset management procedures for long term management, operation, and sustainability of I&D infrastructure maintenance Establish costs for sustainable maintenance of I&D Typical project interventions to improve systems Preparation of MOM manuals management, operation and maintenance Training and capacity building of I&D staff, and water (MOM) of I&D systems are summarised in users Box 1. Source: World Bank Project Appraisal Documents GN 15 MOM Components 1 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components Key performance indicators, data needs and analysis Overview Typical objectives and related performance indicators for interventions to improve management, operation and maintenance of irrigation and drainage systems are summarised in Table 1. Table 1: Typical implementation and results framework for interventions to improve management, operation and maintenance Assessment Examples level Project (a) Improved and sustainable increase in irrigated agricultural productivity; or development (b) Increased productivity of water; or objective (c) Sustainable management of water resources for irrigation Project · Improved level of service delivery component · Reduction in total volume of water diverted for irrigation outcomes · Increase in agricultural productivity per unit of water diverted · Reduction in area waterlogged or salinised · Reduction of, or reversing, the decline in groundwater levels · More reliable, timely and adequate irrigation water supplies to all parts of the irrigation network · More reliable, timely and adequate drainage of the irrigated area · Improved setting and recovery of irrigation service fees (ISFs) · Income and expenditure on O&M matches requirements Project outputs · Restructured government I&D agency · Knowledgeable and skilled personnel · Improved O&M and fee recovery processes and procedures · Trained O&M personnel · O&M manuals produced and in use · Asset management processes established and in use · Study options for reorganising/restructuring of government I&D agencies Project activities · Establish a service delivery culture within the I&D agency · Establish or update procedures for management and administration of I&D systems, incorporating the use of modern technology (computers, communication systems, etc) · Study and develop, or update, norms for financial requirements for sustainable management, operation and maintenance (MOM) of I&D systems · Establish systems for setting and recovery of Irrigation Service Fees (ISFs) · Establish or update procedures for operation of I&D systems, both at the main system and on farm level · Establish, or update, processes and procedures for maintenance of I&D systems · Establish asset management procedures for long term sustainability of I&D infrastructure · Establish costs for sustainable maintenance of I&D systems · Preparation of MOM manuals · Training and capacity building of I&D staff, and water users Project inputs · Specialist inputs (in management, operation and maintenance, institutional development, and training) · Beneficiary participation · Equipment, vehicles and materials Source: Authors GN 15 MOM Components 2 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components The overall aim is to improve the distribution and use of irrigation water, reduce the amount of wastage and improve the drainage through interventions that focus on improving the "human capital". Whilst some support may be given in terms of vehicles, office equipment, materials, etc. the focus is on enabling government and WUA staff and water users to make better use of the water resource. Improvements to the physical infrastructure of the I&D system as described in GN14 can provide an enabling environment for improved system operation. It is, however, the day to day activities of those involved in the management, operation and maintenance of the system that are needed to provide adequate, reliable and timely irrigation water supply and drainage water removal. Improving the level of service should lead to improvements in the fee collection and therefore greater sustainability of the I&D system and the livelihoods dependent on it. Table 2 outlines typical outputs and outcomes from interventions to improve the management, operation and maintenance of I&D systems. It will be important to establish a results monitoring framework during implementation in order to see if the project activities are having the desired effect. For example, the results of capacity building and training of I&D operations staff should be visible in improvements in the level of service delivery, which can be measured by the changes in reliability, adequacy, timeliness and equity of water distribution. Though the water service delivery may improve relatively quickly, changes in agricultural productivity may take longer as farmers adjust to the new water supply situation. Implementation monitoring The distinction between results monitoring and implementation monitoring is less clear cut than is the case with the rehabilitation and modernization aspects of a project as described in GN14. For implementation monitoring the focus will be on progress made towards achieving the outputs as defined in Table 2, such as: · Completion of the training needs assessment and preparation of the Training Plan; · Commencement of training and numbers trained · Preparation of the guidelines for preparing asset management plans; · Progress with carrying out asset surveys of the system, systems or parts of system(s); · Surveys to identify maintenance requirements and costs; · Preparation of guidelines for preparation of ISF tariffs; · Progress with awareness campaigns on ISF tariff and ISF collection. In order to monitor these project activities it is useful to break them down into their component parts, such as shown above with the training component comprising the carrying out of a training needs assessment and the preparation of the Training Plan, followed by the training. An additional intermediate stage might be "Agreement on Training Plan and training budget", giving a clear cut milestone on the progress of the training programme. GN 15 MOM Components 3 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components Table 2: Typical activities, outputs and outcomes for improving I&D system management, operation and maintenance No. Activity Possible outputs Possible outcomes 1 Study options for Restructuring plan prepared Agency restructured ­ organisational reorganisation/restructuring Vision, strategy and Action structure, staffing, duties and of government irrigation Plan prepared responsibilities, etc. and drainage agencies Revised financial plan Changed staff attitudes and behaviour Restructuring plan Changed attitude and behaviour by implemented water users towards I&D agency 2 Establish a service delivery Organisational culture Changed organisational culture (e.g. culture within the I&D analysis carried out, reported as shown by attention and agency on and recommendations responsiveness by I&D agency staff to made water users needs) Strategy and Action Plan Improved level of water user prepared for changing satisfaction with service provision organisational culture Reduced number of complaints Strategy and Action Plan recorded implemented Increased number of compliments recorded Improved relations between water users and I&D agency Increased ISF recovery rates Improvement in level of service provided, particularly in water delivery 3 Establish or update Increased number of Changed staffing levels (may be procedures for computers in active use for reduced in some areas, increased in management and management and others, such as information systems) administration of I&D administration Improved level of service provision systems, incorporating the Improved quality of records Improved quality and timeliness of use of modern technology and data availability reporting to government and other (computers, Improvement in quality of stakeholders communication systems, reports prepared Improved level of water user etc) Improved availability and use satisfaction with service provision of communication systems (CB radios, mobile phones, etc.) 4 Study and develop, or Up to date estimates available Closer matching of government funds update, norms for financial for costs of management, to actual needs for financing requirements for operation and maintenance of sustainable MOM sustainable management, individual I&D systems Improved targeting of available funds operation and maintenance Asset management plans to priority and/or cost effective areas (MOM) of I&D systems available for individual I&D Improved condition and performance systems of I&D infrastructure Annual budgets prepared based on updated information 5 Establish systems for Detailed outline prepared of ISF recovery levels high and sustained setting and recovery of processes and procedures for over time Irrigation Service Fees setting and collecting the ISF Annual income from ISF collection and (ISFs) ISF fee tariff in place for each other sources sufficient for sustainable I&D system. Tariff based on MOM of the I&D system detailed cost calculations Improved expenditure on maintenance Detailed and accurate records (as a consequence of high ISF kept of ISF fee due and recovery) amount paid Annual expenditure on maintenance ISF collection levels match adequate to sustain the system in the actual MOM costs medium to long term 6 Establish or update Operation guidelines prepared Improved level of water delivery procedures for operation of and in use (reliable, adequate, timely, efficient and I&D systems, both at the Seasonal water plan equitable) main system and on farm (availability and supply) Improved level of water user level available satisfaction with service provision Irrigation schedules made on Increased agricultural production regular basis Reduced number of complaints by Flow measurement records water users related to water availability GN 15 MOM Components 4 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components available and complete and delivery Seasonal report on water delivery to, and demand by, water users 7 Establish, or update, Maintenance guidelines Improved level of water delivery processes and procedures prepared and in use (reliable, adequate, timely, efficient and for maintenance of I&D Maintenance needs register equitable) systems prepared Reduced number of incidences of Maintenance work identified, failure of I&D infrastructure prioritised and cost effective Increased level of water user Annual expenditure on satisfaction with service provision maintenance matches Increased agricultural production assessed needs (i.e. adequate Reduced number of complaints by to maintain the system in the water users related to water availability medium to long term) and delivery 8 Establish asset Asset management guidelines Level of investment in capital works management procedures prepared and in use (replacement) for long term sustainability Asset surveys carried out Improved level of service provision, of I&D infrastructure Asset management plan particularly water delivery prepared and being followed Reduced number of incidences of Expenditure on annual failure of I&D infrastructure maintenance and capital Improved level of water user replacement in line with asset satisfaction with service provision management plan Increased agricultural production Reduced number of complaints by water users related to water availability and delivery 9 Establish costs for Up to date estimates available Increased understanding by Ministry of sustainable maintenance of of maintenance needs and Finance and water users of the real I&D systems costs for each I&D system costs for maintenance and the consequences on agricultural production of inadequate maintenance funding Closer matching of funds from government and water users to actual needs for financing system maintenance Improved targeting of available funds to priority and/or cost effective areas of maintenance work 10 Preparation of MOM MOM manuals prepared, Improved level of service provision manuals reviewed, printed, distributed Increased level of understanding, and in use knowledge and skills of O&M staff Scheme specific handbooks prepared, checked and in use for each I&D system 11 Training and capacity Human resources Increased level of understanding, development plan prepared knowledge, skills and motivation of building of I&D staff, and Up to date training strategy staff and plan prepared Increased level of understanding, water users Annual training programme knowledge, skills and motivation of prepared water users Increased expenditure on Improved understanding by individuals training of their role and contribution to the I&D Number of training days agency Number and type of trainees Improved relations between I&D trained agency and water users Source: Authors GN 15 MOM Components 5 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components Results monitoring As discussed above and outlined in Tables 1 and 2 the outputs of interventions to improve the management, operation and maintenance of I&D systems seek, in the main, to provide a more reliable, adequate and timely delivery of irrigation water and removal of drainage water (see the section below on water delivery indicators for a more comprehensive definition of the criteria for irrigation service delivery). The benefits arising from this work include more efficient use of irrigation water, less water diverted, reduction in waterlogging and salinisation, increased productivity of water diverted, improved fee recovery and greater sustainability of the I&D system. Relevant indicators to measure these outcomes are provided in Table 4. These indicators are similar to those used in GN14 as the activities of rehabilitation and modernization and improvements in MOM are complementary and seek the same outcomes. Whilst improvements in MOM often accompany physical works related to rehabilitation and modernization, it is possible to have a project focussed solely on improving MOM without a major infrastructure component. Table 4 provides more detailed definition of indicators than the terms used in Table 1. One or more of the indicators provided in Table 3 can be used to quantify the indicator statement made in Table 1. In addition, for the detailed indicators, a target value can be set against which the performance can be assessed: Table 3: definition of indicators Indicator statement Performance Definition Indicator target indicator(s) used value Improved expenditure on Fee collection ratio Total irrigation service fees collected > 0.90 maintenance Total irrigation service fees due Annual income from ISF MOM funding ratio Actual annual income >0.90 collection sufficient for Total annual budget required for sustainable MOM sustainable MOM of the I&D system Total annual expenditure on MOM >0.90 O&M fraction Total annual budget required for sustainable MOM Annual expenditure on >0.70 maintenance adequate to Total annual expenditure on maintenance Maintenance (for gravity sustain the system in the fraction Total annual budget required for sustainable MOM systems) medium to long term Source: Adapted from Bos et al, 2005 Water delivery indicators Water delivery is a central component of system management, operation and maintenance. If required, more detailed analysis of system operation and water delivery can be carried out. The main criteria used in assessing the performance of irrigation water delivery are: · Reliability · Adequacy (of supply) · Timeliness · Equity · Efficiency · Productivity · Cost (and cost effectiveness) For these criteria a number of indicators can be used, as listed in Table 5. As can be seen some indicators provide information on more than one criterion. GN 15 MOM Components 6 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components Table 4: Key indicators for outcome monitoring and evaluation of irrigation and drainage system management, operation and maintenance Indicators Definition Notes1 Agricultural production Total seasonal2 area cropped per unit Total seasonal area cropped a command area (Cropping intensity) Total command area of system Total seasonal crop production (Tonnes) Total seasonal crop production by crop type within command area a Total seasonal crop production per unit Total seasonal crop production a command area (crop yield, kg/ha) Total command area of system Total seasonal value of crop production Total seasonal value of agricultural crop production received by a ($) producers Total seasonal value of crop production Total seasonal value of crop production a per unit command area ($/ha) Total command area of system Total seasonal crop production per unit Total seasonal crop production a water supply (kg/m3) Total seasonal volume of irrigation water supply Total seasonal value of crop production Total seasonal value of crop production a per unit water consumed ($/m3) Total seasonal volume of crop water demand (Etc) Total seasonal value of crop production Total seasonal value of crop production a per unit water supplied ($/m3) Total seasonal volume of irrigation water supply Irrigation water delivery Total seasonal volume of irrigation water Total seasonal volume of water diverted or pumped for irrigation (not a supply (MCM) including diversion of internal drainage) Seasonal irrigation water supply per unit Total seasonal volume of irrigation water supply a command area (m3/ha) Total command area of system Main system water delivery efficiency Total seasonal volume of irrigation water delivery b Total seasonal volume of irrigation water supply Seasonal relative irrigation water supply Total seasonal volume of irrigation water supply a Total seasonal volume of crop water demand Water delivery capacity Canal capacity at head of system Peak irrigation water demand at head of system Financial Total seasonal MOM expenditure3 per unit Total seasonal MOM expenditure c command area ($/ha) Total command area of system Total seasonal MOM expenditure per unit Total seasonal MOM expenditure c irrigation water supply ($/m3) Total seasonal volume of irrigation water supply Total seasonal maintenance expenditure Total seasonal maintenance expenditure c per unit command area ($/ha) Total command area of system Total seasonal maintenance expenditure Total seasonal maintenance expenditure c fraction Total seasonal MOM expenditure MOM funding ratio Actual annual income d Budget required for sustainable MOM Fee collection ratio Irrigation (and drainage) service fees collected d Irrigation (and drainage) service fees due Farm profit Total farm income ­ total farm expenditure e Drainage water removal Average depth to groundwater (m) Average seasonal depth to groundwater calculated from water table f observations over the irrigation area Environmental protection Salinity of soil water (mmhos/cm) Electrical conductivity of soil water f Soil salinity (mmhos/cm) Electrical conductivity of soil f Salinity of water in open drain Electrical conductivity of water in open drains f (mmhos/cm) Drainage water quality: Biological Biological load of drainage water expressed as Biological Oxygen f (mg/litre) Demand (BOD) Drainage water quality: Chemical Chemical load of drainage water expressed as Chemical Oxygen f (mg/litre) Demand (COD) Source: Adapted from Bos et al, 2005 and Malano and Burton, 2001 Notes: 4. Location and sampling interval: a. Determine for total command area and individual tertiary units b. Discharges measured at the main canal intake and tertiary unit intakes c. Determine for total command area, main system only and individual Water Users Associations d. Determine for individual service providers (government agency or Water Users Associations) e. For individual water users f. Periodic sampling at selected locations GN 15 MOM Components 7 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components 5. May be seasonal or annual, depending on the circumstances. If there is more than one season and there are marked differences between the seasons' cropping patterns and water availability it is preferable to consider each season separately 6. Costs for irrigation water delivery and drainage water removal may be kept separate or combined; it depends if there is a separate drainage authority. Table 5: Indicators used for assessing different performance criteria related to water delivery Performance Notes Criteria Definition indicator Variation of the RWS at the main canal intake Volume of irrigation water supply and at tertiary intakes during the season Relative Water Supply Volume of irrigation water demand indicates the level of reliability of water supply Reliability and delivery Variation of the DPR at tertiary unit intakes Delivery Performance Volume of irrigation water supplied during the season indicates the level of Ratio Target volume of irrigation water supply reliability water delivery Measured at main canal intake and each Relative Water Supply Volume of irrigation water supplied tertiary unit intake. Target value = 1.0, less (RWS) Volume of irrigation water demand than 1.0 indicates water shortage Adequacy Measured at main canal intake and each Delivery Performance Volume of irrigation water supplied tertiary unit. Target value = 1.0. If there is a Ratio (DPR) Target volume of irrigation water supply water shortage the target supply may be less than the actual irrigation water demand. The planned/required interval between Dependability of Actual irrigation interval irrigations is either that planned (such as in a Irrigation Interval Planned/Required irrigation interval planned irrigation rotation regime) or that dictated by the crop's soil moisture status. Timeliness Compares the actual date and time of delivery Actual date/time of irrigation water delivery Timeliness of Irrigation (planned in the rotation or requested by the Planned/Required date/time of irrigation Water Delivery farmer) compared to the actual delivery date water delivery and time. Volume of irrigation water supply Variation of the RWS at tertiary intakes Relative Water Supply Volume of irrigation water demand indicates degree of equity or inequity Equity Delivery Performance Volume of irrigation water supplied Variation of the RWS at tertiary intakes Ratio Target volume of irrigation water supply indicates degree of equity or inequity Comparison of the RWS at the main canal Volume of irrigation water supply Relative Water Supply intake and the tertiary unit intakes indicates the Volume of irrigation water demand level of losses Useful indicator. Relatively easy to obtain a Volume of water needed by crop meaningful value. Estimate crop irrigation Overall scheme Volume of water diverted/pumped from water demand at the field (using FAO efficiency source CROPWAT programme, or similar) and Efficiency measure actual discharge at main canal intake. Measure discharges at main canal intake and Volume of water delivered (to tertiary unit) Main system water offtakes to tertiary units. Value may change Volume of water diverted/pumped from delivery efficiency due to the seasons (wet/dry), with drainage source inflow possible in wet season. Total crop production As measure of efficiency use to determine Crop production per unit Volume of water diverted/pumped from change in production per unit of water diverted water supply source at source. Useful for monoculture schemes. Total crop production Increasingly important indicator. Need to be Crop production per unit Volume of water delivered (to tertiary unit or careful where there is mixed cropping. water delivered field) Productivity Total value of crop production Increasingly important indicator. Use the value Value of crop production Volume of water delivered (to tertiary unit or of crop production where there is mixed per unit water delivered field) cropping. Assesses the cost of the ISF compared to the ISF collected to GVP Total irrigation service fee (ISF) collected total gross value of production. A broad ratio Total gross value of production (GVP) indicator only as other costs are involved. Cost Assesses the costs of the ISF as a fraction (or effective percentage) of the total input costs for planting, ness ISF to total crop input Irrigation service fee (ISF) due for the crop harvesting and marketing the crop. Often costs ratio Total input costs for the crop found to be in the range of 4 10% of total input costs where the ISF is set at adequate levels to recover sustainable MOM costs. Source: Adapted from Bos et al, 2005 and Malano and Burton, 2001 GN 15 MOM Components 8 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components Practical examples Examples are provided below of the performance indicators used for monitoring the impact of the World Bank funded Irrigation Improvement Project (IIP) in Egypt. The IIP rehabilitated and modernized the secondary canals and mesqas (tertiary units) in selected command areas, modernized the operation procedures and formed Water Users Associations to manage the tertiary (on farm) systems. A key feature of the project was the substantial change in the management and operation procedures on the secondary and tertiary canals. For the evaluation the performance of selected improved secondary and tertiary canals was compared to unimproved secondary and tertiary canals. For this exercise a table was drawn up (Table 6, Figure 1) to show: · where the data were to be collected · the measurement units · who was to collect the data · how it was to be collected · the frequency of collection · the period over which the data were to be collected It is important that the data collection is systematic and that data are collected for all of the specified period. This is particularly the case for canal discharges. Missing or erroneous data at key times can invalidate the whole data collection programme for an entire season; careful monitoring of the data collection is thus required to avoid this situation. Figure 2 presents the cropping pattern for the six branch canals that are being monitored. This form of presentation allows a quick visual understanding of the similarities and differences in the cropping pattern in each command area. Such figures can be used to compare the pre and post project cropping patterns to evaluate the changes that have taken place as a result of project interventions. As mentioned previously, in some schemes one would be looking for farmers in the tail end locations changing their cropping pattern to match the improved water delivery situation brought about by the project. Table 725 shows how the irrigation water demand can be calculated for each secondary (branch) canal and information on the value of crop production per unit of water delivered at the secondary (branch) canal intake. Note that due to the mixed cropping and different crop yields, the production per unit water consumed is determined for each crop but not for the Branch Canals as a whole. The key indicator is the value of the crop production per unit of water consumed which is determined for each crop and the Branch Canal as a whole. Table 8 shows the data needs and calculations to determine the key indicators for secondary (branch) canal performance, showing the total seasonal volume of demand, supply, and delivery per unit area, the main system water delivery efficiency and the seasonal relative irrigation water supply. These figures allow comparison between each secondary canal each year, and when collected for several years allow trend analysis of the performance of each Branch Canal. Note that, in this case, the seasonal relative irrigation water supply is calculated from the irrigation demand in the field and the water delivered at the mesqa intake. The calculation does not include the losses in delivering the water from the mesqa intake to the crop root zone as these losses were not measured during the survey and are thus unquantified. This factor should be taken into account when assessing the values of the relative irrigation water supply (RIWS); for example if the tertiary system losses are 50% a target value of the RIWS (actual supply/demand) would be 2.0. In the case shown here the (surface) irrigation supply may be supplemented by contributions from groundwater. Table 9 provides an example of the data collected and processed to determine the performance indicators related to the MOM cost and number of personnel. In the example shown there are only two sets of data 25The terminology used for the indicators in these tables and figures may vary from those used in Table 2 GN 15 MOM Components 9 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components for the six secondary (branch) canals as the canals are located in two Directorates and data for each secondary canal has been derived from the data for each of these Directorates. These costs can be compared with the value of the crops produced, either on a per unit area or per unit water supply basis. Table 10 provides an example of a summary table for presenting drainage information in secondary (branch) canal command areas. Data were collected for tertiary units (mesqas) in each secondary canal command area and summarised in this table. The figures presented are compared with the standards and colour coding used to show areas of concern, or critical areas. Concluding remarks Some additional points that should be considered in the monitoring and evaluation of management, operation and maintenance interventions are outlined below: 1. Measuring discharges is a central part of the M&E process but it is often difficult to obtain accurate and reliable data, either because measuring structures are not in place or because staff do not measure the discharges on a regular basis. Discharge data should be carefully checked and field visits made to ascertain the condition of measuring structures and the capability and motivation of field staff in taking daily measurements; 2. It is preferable that systems are put in place during the project to collect data as part of the regular management, operation and maintenance processes by the main system and on farm service providers (the I&D agency and the WUAs); 3. Due to the wide coverage of data collection it will not always be possible to collect M&E data from all points in the I&D system. It is a common practice to select samples from the head, middle and tail end sections of the I&D system in order to assess such factors as the equity of water distribution; 4. Consideration should be given to incorporating automatic flow or water level measuring equipment in the project and establishing measurement stations early on in the project in order to assess changes in the flow patterns arising from project interventions; References and Further Reading Bos, M.G., M. Burton, and D.J. Molden. 2005. Irrigation and drainage performance assessment practical guidelines. CABI publishing, Wallingford, UK. Hector, M., and M. Burton. 2001. Guidelines for benchmarking performance in the irrigation and drainage sector. IPTRID Secretariat, Food and Agriculture Organisation of the UN, Rome. World Bank. 2005b. Benchmarking Initiative in the Irrigation and Drainage Sector, Egypt: Final report. Washington D.C.: World Bank, December 2005. World Bank. 2005d. Guidelines for Benchmarking Performance in the Irrigation and Drainage Sector in Egypt. Washington D.C.: World Bank, December 2005. GN 15 MOM Components 10 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components Figure 1: Example of locations for data collection (to be read in conjunction with Table 6) 4 4 4 5 6 3 3 3 6 6 2 2 2 2 2 2 1 2 2 2 2 2 6 3 3 3 6 6 5 4 4 4 Source: World Bank, 2005a; 2005b GN 15 MOM Components 11 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components Table 6: Example specification for data collection Map Location Data collected Units By whom How Frequency Period Remarks collected collected of collected collection 1 Branch canal Discharge Irrigation Measure Daily Season Level data recorded daily by intake entering branch m Service ment Irrigation Service staff. On canal: m District staff two systems water level and · Flow depth m3//s IIS staff gate opening data collected · Gate hrs, mins by WMRI under contract to opening IIP, using automatic water · Discharge level recorders. · Duration of flow 1 Branch canal · Water mmhos/cm Irrigation Measure Once per Season Data regularly collected for intake quality Improveme ment month two canals by Water nt Service Management Research (IIS) staff Institute (WMRI) 1a Branch canal Discharge IIS staff Measure Daily Season Data regularly collected for tail escape leaving branch m ment two canals by WMRI canal: m3//s · Flow depth hrs, mins · Discharge · Duration of flow 2 Mesqa1 intake Discharge Pump Measure Hourly Season Data collected by WUA for all delivered to hrs operator ment mesqas for charging and cost mesqa: m calculation purposes · Pumping litres hours · Pumping head (intake, delivery) · Fuel consumed 3 Selected Groundwater EPADP 10 12 times Season 12 piezometers installed in mesqas in and soil data: m staff per season each branch canal command. head, middle, · Depth to mmhos/cm and tail) groundwater mmhos/cm Once/seaso · Salinity of n groundwater (EC) · Soil salinity at 40 cm depth 4 Selected Drainage water Drainage Measure Periodically Season EPADP field staff will monitor mesqas levels: m service field ment selected collector drain (outfalls to · Number of staff outfalls during the season selected days and record the number of mesqas in the collector days they are submerged head, middle, outlet and tail) submerged during season 5 Secondary Drainage water Drainage Measure Daily (water Season WMRI are monitoring drain outfall level and flow: m service field ment level) drainage water quality on a · Drainage m3/s staff Monthly regular basis for two of the water level mmhos/cm EPADP (water systems · Discharge staff quality) · Water quality (EC) GN 15 MOM Components 12 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components 6 Selected · Command ha WUA Interview Once per Season Simple crop data collection mesqas along area IIS staff s with season procedures need to be tested branch canal For a typical 10 farmers. with WUAs to ascertain (head, middle, ha sample area: ha whether reliable crop data tail) · Crop type days From can be obtained for · Crop area bags agricultur comparison between WUAs. · Crop kgs al These can be cross checked duration LE2 cooperati with data collected from other LE ves and sources (crop cuttings by · Crop production Ministry Ministry of Agriculture, data (bags) of collected by agricultural Agricultur cooperatives, etc.) · Weight of bags (by e crop type) · Crop market price · Cost of production 6 Selected Water user IIS staff Survey 2 times per Season mesqas along satisfaction season branch canal survey: (mid (10 head, 10 · Satisfaction season and middle, 10 tail) with water just after delivery harvest) · Satisfaction with drainage removal · Problems/c onstraints 7 District irrigation Irrigation District Office Seasonally Season These data are available at system Directorate MOM Irrigation records the Directorate level. If expenditure and ha Engineer possible they should be staffing: broken down to Branch Canal · Total LE command areas. If not the command Directorate level data can be area LE used as they are · Total representative of the average annual LE annual MOM expenditure and MOM No. maintenance expenditure. expenditure LE Historic data can also be (salaries, analysed for comparative office costs, purposes and trend analysis. operation, maintenanc e, etc.) · Total annual planned maintenanc e expenditure on canal systems · Total annual actual maintenanc e expenditure on canal systems · Total number of staff GN 15 MOM Components 13 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components · Total cost of staff 8 District Drainage District Office Annually Season These data are available at drainage Directorate MOM Drainage records the Directorate level. If system expenditure and ha Engineer possible they should be staffing: broken down to Branch Canal · Total LE command areas. If not the command Directorate level data can be area LE used as they are · Total representative of the average annual No. annual MOM expenditure and MOM LE maintenance expenditure. expenditure Historic data can also be (salaries, analysed for comparative office costs, purposes and trend analysis. operation, maintenanc e, etc.) · Total annual planned maintenanc e expenditure on drainage systems · Total annual actual maintenanc e expenditure on drainage systems · Total number of staff · Total cost of staff 9 Branch canal Complaints: District Office Each Season and mesqas · Number of No. Irrigation records season complaints Engineer · Nature of complaint · Action taken 9 Branch canal Complaints: District Office Each Season collector drain · Number of No. Drainage records season and secondary complaints Engineer drains · Nature of complaint · Action taken Source: World Bank, 2005b; 2005d Notes: 1. Mesqa ­ Tertiary unit 2. LE ­ Egyptian pounds GN 15 MOM Components 14 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components GN 15 MOM Components 15 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components Table 7: Example of irrigation output performance latoT lanosaes fo d 3 latoT lanosaes eulav rpporc-cudo -cudo tinurep retaw eilppus m/EL 88.0 28.0 82.1 86.0 32.3 26.2 fo de 3 srotacidni latoT lanosaes udorpporc -c eulav rpporctinurep noittinurep m 00. 05. 81. 00. 00. 00.1 05.1 81.1 00.0 00.0 00.1 05.1 81.1 00.0 27.2 00.1 05.1 81.1 83.1 00.0 00.1 05.1 81.1 00.0 27.2 00.1 05.1 81.1 00.0 27.2 noit retaw musnoc E/L 1 1 1 0 0 retaw de musnoc 3 m/gk 11.1 55.1 39.0 00.0 00.0 11.1 55.1 39.0 00.0 00.0 11.1 55.1 39.0 00.0 87.7 11.1 55.1 39.0 75.21 00.0 11.1 55.1 39.0 00.0 87.7 11.1 55.1 39.0 00.0 87.7 )5002 gu (A EL e moctuO latoT lanosaes fo 87.5 eulav rpporc noit-cudo tinurep dna m .def 0 0 0 0 0 0 0 0 = noit moc aera E/L 007,2 477,2 517,1 676,2 007,2 477,2 517,1 486,2 007,2 477,2 517,1 009,4 539,2 007,2 477,2 517,1 024,2 914,2 007,2 477,2 517,1 009,4 037,2 007,2 477,2 517,1 009,4 688,2 $SU 1 porC dleiy .def/gk 0003 0682 0531 00022 00041 0003 0682 0531 00022 00041 0003 0682 0531 00022 00041 0003 0682 0531 00022 00041 0003 0682 0531 00022 00041 0003 0682 0531 00022 00041 latoT lanosaes fo eulav porc udorp -c noit EL 000' 1,7 0,7 35 30 955 0 0 617,41 992,3 329,1 541 0 0 863,5 225,7 070,4 549 0 998,3 634,61 946,1 756,4 768,1 198,4 0 560,31 907,6 500,5 085,1 0 301,2 893,51 357,5 937,2 282 0 207,1 674,01 latoT lanosaes porc udorp -c noit sennoT 849,7 022,7 044 0 0 666,3 389,1 411 0 0 853,8 691,4 447 0 141,11 238,1 208,4 074,1 764,44 0 554,7 061,5 442,1 0 900,6 293,6 428,2 222 0 368,4 fo e it no y 3 tniop latoT lanosaes mulov agirri retaw lppus m 827,336,61 954,775,6 061,008,21 115,613,91 848,867,4 804,399,3 yrtne 3 ,3 ,0 9, 542 523 81 0 0 0 0 863, 0 0 0 ataD porclatoT noitagirri retaw eriuqer tnem - m 51,7 76,4 274 884,692,21 004,992,3 024,282,1 069,221 087,407,4 002,225,7 023,417,2 800,997 0 234,1 698,764,21 899,846,1 198,501,3 113,975,1 531,735,3 533,178,9 754,907,6 738,733,3 582,633,1 765,277 641,651,21 781,357,5 616,628,1 734,832 403,526 345,344,8 noit-agirrI retaw dna med .def/3 0072 0581 0541 0571 0081 0072 0581 0541 0571 0081 0072 0581 0541 0571 0081 0072 0581 0541 0571 0081 0072 0581 0541 0571 0081 0072 0581 0541 0571 0081 seratceh m 24.0 atad porC ecirp gk E/L 09.0 79.0 72.1 11.0 53.0 09.0 79.0 72.1 11.0 53.0 09.0 79.0 72.1 11.0 53.0 09.0 79.0 72.1 11.0 53.0 09.0 79.0 72.1 11.0 53.0 09.0 79.0 72.1 11.0 53.0 =sercA esaB porC dleiy .def/gk 0003 0682 0531 00022 00041 0003 0682 0531 00022 00041 0003 0682 0531 00022 00041 0003 0682 0531 00022 00041 0003 0682 0531 00022 00041 0003 0682 0531 00022 00041 40.1 = aera 623 0 0 2221 396 58 0 0 155 0 97 6 116 0 229 0 24 9 1312 789 461 0 43 7 naddeF depporC .deF 9462 5252 0055 0002 6872 7641 0065 9761 9801 1202 0045 5842 4081 0465 0363 1 % 71.84 9.54 39.5 0 0 001 1.16 66.43 42.4 0 0 001 57.94 2.62 48.9 0 .41 12 001 13.11 90.13 71.02 34.73 0 001 60.44 99.13 43.61 0 16.7 001 7.85 2.72 35.4 0 75.9 001 50-4002retni epyt snae ragu tege teeb selba snae ragu tege teeb selba snae ragu tege teeb selba taeh snae ragu tege teeb selba snae ragu tege teeb selba snae ragu tegeteeb selba porC meesreBtaeh taeh taeh W B S V meesreB taeh W B S V meesreB taeh W B S V meesreB W B S V meesreB W B S V meesreB W B S V - d W:nosaeS moC nam aera .deF 0055 0002 0065 0045 0465 0363 fo a e seulavlanaC ma arb ac hcn lan lE N yawtneseB seulavlanaC hcnarB miaNtiewaZ seulavlanaC hcnarB addiaBlE seulavlanaC hcnarB tlaqaD seulavlanaC hcnarB ruohnaS meedaK seulavlanaC hcnarB leehseN hcnarB Source:World Bank 2005b; 2005d GN 15 MOM Components 16 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components Table 8: Example of irrigation water delivery performance lanosaeS evitaler itagirri retawno ylppus 06.0 78.0 06.0 78.0 72.0 a/n metsysniaM eviledretaw yr ycneiciffe % %3.44 %9.16 %6.85 %4.44 %9.76 a/n srotacidnI lanosaeslatoT noitagirri ylppusretaw tinurep dnammoc aera naddef/3m 4203 9823 6822 7753 648 0011 elbaliavatonataD a/n latoT osaes lan noitagirri yreviledretaw tinurep dnanmmoc aera naddef/3m 0431 7302 9331 7851 475 a/n lan foe latoT osaes mulov retawporc ta(dnamed )dleif naddef/3m 6322 2532 6222 8281 5512 6232 kaeP noitagirri dnamedretaw fodaehta metsys s/3m a/n a/n a/n a/n a/n a/n s/3m a/n a/n a/n a/n a/n a/n tniop atadnoitagirrI lanaC tayticapac fodaeh metsys lan foe latoT osaes mulov retawporc ta(dnamed )dleif 3m 884,692,21 087,407,4 698,764,21 533,178,9 641,651,21 345,344,8 rytneataD lan foe latoT osaes mulov noitagirri retaw eviled yr 3m 934,273,7 751,370,4 072,994,7 999,075,8 483,932,3 a/n lan foe latoT osaes mulov noitagirri ylppusretaw 3m 827,336,61 954,775,6 061,008,21 115,613,91 848,867,4 804,399,3 dna latoT mmoc foaera itagirri no metsys snaddef 005,5 000,2 006,5 004,5 046,5 036,3 50-4002retni atadlareneG sutatS /devorpmI( )devorpminU I I IU I I IU etarotceriD W areheB areheB areheB kiehSlErfaK kiehSlErfaK aibrahG :nosaeS hcnarBfoemaN lanaC yawtneseB miaNtiewaZ adieBlE tlaqaD amidaKlEruohnaS amidaKlElihseN Source: World Bank, 2005b; 2005d GN 15 MOM Components 17 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components Table 9: Example of assessment of MOM costs Season: Winter 2004-05 Data entry point General data Area Water supply Directorate annual costs and personnel Branch Canal annual costs and personnel Command Total command Total seasonal Total annual Total annual Total number of Total annual Total annual Total annual Total annual Total Branch Canal area of area of volume of Directorate Directorate Directorate cost of Branch Canal Branch Canal Branch Canal personnel Branch Canal Directorate irrigation water MOM O&M costs personnel Directorate MOM costs O&M costs personnel cost Name of Branch Status (Improved/ supply expenditure personnel Canal Directorate Unimproved) - - feddans feddans m3 LE LE No. LE LE LE LE No. Besentway Behera I 5,500 360,000 16,633,728 13,270,000 9,700,000 915 3,570,000 202,736 148,194 54,542 13.98 Zaweit Naim Behera I 2,000 360,000 6,577,459 13,270,000 9,700,000 915 3,570,000 73,722 53,889 19,833 5.08 El Baidda Behera UI 5,600 360,000 12,800,160 13,270,000 9,700,000 915 3,570,000 206,422 150,889 55,533 14.23 Daqalt Kafr El Sheik I 5,400 300,000 19,316,511 12,000,000 6,000,000 800 3,000,000 216,000 108,000 54,000 14.40 Sanhour El Kadeema Kafr El Sheik I 5,640 300,000 4,768,848 12,000,000 6,000,000 800 3,000,000 225,600 112,800 56,400 15.04 Nesheel Gharbia UI 3,630 300,000 3,993,408 12,000,000 6,000,000 800 3,000,000 145,200 72,600 36,300 9.68 Performance Indicators Branch Canal seasonal costs and personnel per unit area Total seasonal Total seasonal Total seasonal Total annual Total cost per Irrigation MOM costs for MOM costs for maintenance maintenance person command area per irrigation water irrigation water expenditure for expenditure employed on unit staff delivery per unit delivery per unit irrigation water fraction for water delivery command area irrigation water delivery per unit irrigation water supply command area delivery LE/feddan LE/m3 LE/feddan - LE/person Feddan/person 18.43 0.012 13.47 0.73 3,902 393 18.43 0.011 13.47 0.73 3,902 393 18.43 0.016 13.47 0.73 3,902 393 20.00 0.011 10.00 0.50 3,750 375 20.00 0.047 10.00 0.50 3,750 375 20.00 0.036 10.00 0.50 3,750 375 Note: O&M costs taken as maintenance costs as operation cost element is low Source: World Bank 2005b; 2005d GN 15 MOM Components 18 Toolkit for monitoring and evaluation of AWM Guidance Note 15 projects M&E for Management, Operation, and Maintenance Components Table 10: Example of a summary of drainage performance at tertiary unit level Season: Winter 2004-05 Groundwater salinity Soil salinity Crop yield (tonnes/feddan) Grounwater level (m) Branch Canal Mesqa (Mmhos/cm) (Mmhos/cm) Wheat Beans Top Tail Top Tail Top Tail Top Tail Top Tail Sharaf Elden 1 90 93 1 1.2 1.36 0.46 3 2.4 Sharaf Elden 2 65 68 1.5 1.6 0.85 0.45 1.6 3.2 1.34 1.42 Eldeb and Abdallah 60 75 1.2 0.9 2.2 2.68 1.28 1.2 Besentway El Tlaten 65 85 2.7 2.6 1.19 0.67 3.16 2.72 El Tabakh 82 2.1 0.43 2.8 1.27 1.15 El Shnawy 50 93 2.8 2 0.46 0.8 3.52 1.8 Average values 69 83 2.02 1.72 0.86 0.66 2.70 2.60 1.30 1.26 Omr Darwesh 58 61 1.5 1.9 0.69 1.16 2.8 1.52 1.22 Eslah Naaym 32 60 1.5 3 0.2 0.26 2.6 3.68 1.57 1.28 Elmostahdasa 64 58 1.9 2.1 0.74 0.89 1.4 3.2 Zaweit Naim Mohamed Ramadan 55 54 5.7 4.8 0.68 0.98 3.72 1.8 Saleh Elbana 55 54 3.2 1.8 0.36 0.66 Elbarada 54 50 2 3 0.52 0.93 2.2 2.4 1.1 Average values 53 56 2.63 2.77 0.53 0.81 2.12 2.10 0.45 0.42 Mesqa 1 62 56 1.4 1.5 0.91 0,68 1.84 1.48 1.55 1.22 Mesqa 2 92 101 1.9 2.7 1.01 1.13 2.12 3.44 1.34 1.42 Mesqa 3 93 98 3 3.1 1.9 1.3 1.95 2.05 1.57 1.34 El Baidda Mesqa 4 72 96 1.9 3.8 0.66 0.71 1.87 1.75 1.28 1.2 Mesqa 5 74 81 1.4 1.5 1.92 0.75 Mesqa 6 55 60 1.6 1.7 0.92 2.13 Average values 75 82 1.87 2.38 1.22 1.00 1.95 2.18 1.44 1.30 El Beda 70 82 1 0.9 1.5 1.65 1.1 El Hohoda 72 70 1.2 1 2.65 2.69 2.56 Om Hnesh 65 98 0.9 3.2 2.1 2.3 1.3 Daqalt El Raha 78 45 2.1 0.9 2.5 2.6 2.12 El Kom 45 55 0.8 2.2 3.2 2.3 1.5 Shams Elden 75 60 2.1 2 2.3 2.5 2.25 Average values 68 68 1.35 1.70 2.38 2.34 2.25 2.34 1.30 1.30 El Oydat 50 50 3.5 2.2 2.5 2.7 5.06 1.7 El Sant 45 45 3.1 3.8 3.8 5.4 5.8 5.8 1.4 1.6 El Nahal 70 96 3.7 3.2 3.1 2.9 2.45 2.45 1.3 Sanhour El Mobasher 9 40 43 2 3.7 3.4 3.3 5.4 5 1.5 1.2 Kadeema Mobasher 15 58 50 1.8 1.6 2.5 3.3 5.9 1.5 Mobasher 16 50 63 1.2 1.5 2.6 3.5 5.4 5.2 Average values 52 58 2.55 2.67 2.98 3.52 4.76 4.90 1.53 1.37 El Hoyyd 75 92 7.5 9.3 0.69 1.16 2.8 2.16 El Barary 82 108 1.8 3.2 0.2 0.26 2.4 1.3 1.5 Gobran 77 75 1.5 6.2 0.74 0.69 3 2.96 Nesheel Andria 75 92 2.4 10.3 0.66 0.48 1.4 Naser 1 93 90 1.8 8.1 0.36 0.66 2.33 1.4 Naser 2 93 92 2.2 4.7 0.52 0.73 2.2 Average values 83 92 2.87 6.97 0.53 0.66 2.73 2.41 1.35 1.45 Area of concern Critical value Source: World Bank, 2005b; 2005d GN 15 MOM Components 19 Toolkit for monitoring and evaluation of AWM Guidance Note 16 projects M&E for WUA Formation and Support Guidance Note 16 M&E for Water Users Association Formation and Support Introduction It has long been recognised that interventions in rehabilitating or modernising the physical infrastructure of I&D systems are often not sufficient for improving their performance. Other interventions relating to institutional and organisational issues, which might at one time have been covered in the Assumptions column of the Logical Framework, are now being considered within the project design. Institutional development seeks to build the social and human capital of communities and organisations, such that they are better able to manage the physical infrastructure (Box 1). A useful conceptual framework is to consider the objectives of institutional development at the three levels of: - Institutions: developing the incentives, norms, laws, rules, or policies, that enable organizations and individuals to function effectively - Organizations: developing the processes and systems enabling organizations to manage resources, perform their function, and achieve and sustain desired outputs and outcomes - Individuals: developing the skills, knowledge, understanding and experience required by individuals to perform their functions Projects focussing on participatory irrigation Box 1: Definitions management have been ongoing since the early 1990s: almost all AWM dedicated projects in the Social capital is the structure of relationships Bank now include components on WUA between individuals and organisations. Changes in formation and support. In addition, non- these relationship structures enable individuals and dedicated AWM projects, in which irrigation and organisations to act and relate in new ways. drainage is only a small component of the rural development agenda, often feature Community Physical capital is embodied in the tools, machines Driven Development1. Finally, to complement and physical works that enable individuals and the institutional changes brought about by WUA organisations to produce goods and services. formation, a majority of projects now include irrigation and water resources agency restructuring components (see GN17 for Human capital is created by facilitating changes in guidance on M&E of such components). an individual's understanding, knowledge, skills and capabilities that enable them to act in new ways. The main objectives of participatory irrigation Within an organisation these new capabilities enable management are three-fold: to involve and the organisation to function in new ways. empower stakeholders in the management of their water resources; to increase efficiency and Institutional design is "the process of developing a cost effectiveness in service delivery; and to put set of rules that participants in a process in place a sustainable management framework. understand, agree upon, and are willing to follow" This guidance note focuses on the first Source: Adapted from Ostrom, 1992 objective, whilst GN15 provides more detailed information on the last two objectives. 1 For more information, see World Bank Social Development Department. Community Driven Development: http://go.worldbank.org/24K8IHVVS0 GN16 M&E WUAs 1 Toolkit for monitoring and evaluation of AWM Guidance Note 16 projects M&E for WUA Formation and Support The underlying rationale for participation in irrigation is that users have a direct interest in the water delivery function because of its influence on the profitability of their agricultural operations. WUAs have proven, in the best cases, to be efficient, accountable and responsive. Associations have often been more successful than government agencies in recovering costs through higher charges and higher collection rates. Maintenance activities by the associations have helped stop the deterioration of infrastructure, but the impact on efficiency and productivity is mixed (World Bank, 2006a). In some cases there is a risk that influential members of the community capture the WUA development process, which can reduce opportunities for the vulnerable. It is therefore important to monitor the level of involvement and degree of democracy exhibited by the WUA in the management of its affairs. Key activities for WUA formation and support The key activities of project interventions to establish and support WUAs are summarised in Box 2. Table 1 lists these key activities together with their possible outputs and outcomes. A starting point for establishing WUAs is Box 2: Main project activities related to WUA suitable enabling legislation. In some formation and support instances pilot projects have been set up to establish WUAs without sufficiently · Enact new or upgrade existing legal framework for strong legislation, with the result that the establishing WUAs and Federations process has not worked. · Formation of WUA Support Units · Formation and establishment of WUAs An important part of the process of · Publicity, communication and awareness campaigns establishing WUAs is the establishment of · Training and capacity building programmes a WUA Support Unit to work with water · Development of management capability, including record keeping and performance monitoring users in the formation and establishment Development of financial management capability of WUAs. The initial stages for · · Development of technical management capability establishment of WUAs is often the (system operation and maintenance) hardest as it may be a new concept and · Support for the purchase of maintenance machinery farmers may be suspicious of the motives and equipment and benefits. Significant effort is required · Development of processes and procedures for WUA for awareness campaigns to inform the Regulatory Authority farming community and gain their · Formation and establishment of Federations of WUAs acceptance and support for the process. · Formation and establishment of National Association Often a majority of water users is required of WUAs (under the law and statutes of the WUA) to establish and legally register an Source: Authors association. Once legally established work commences on strengthening the administrative and management framework of the association, with training programmes on procedures for organising and managing General Meetings, day-to-day management, conflict resolution, etc. Allied to this is training for the WUA Accountant on financial management, fee setting and fee collection, and training for the operation and maintenance staff of the association on planning and managing water allocation and distribution, measuring and recording water deliveries, planning, costing and managing maintenance work, etc. In some cases the project may provide credit to enable the WUA to purchase maintenance machinery and equipment. It is important that this is only done if the WUA has first shown itself capable of managing the association and is able to recover from the water users the fees required to cover the cost of operating, maintaining and, in time, replacing this machinery and equipment. An associated activity is the establishment of a WUA Regulatory Authority with responsibilities, under the law, for monitoring and regulating WUAs. The Authority should monitor the financial, institutional and technical functioning of the WUAs, ensuring that they manage their finances properly, are properly transparent and accountable to their members, and provide adequate levels of service to water users in terms of the operation and maintenance of the irrigation and drainage systems. GN16 M&E WUAs 2 Toolkit for monitoring and evaluation of AWM Guidance Note 16 projects M&E for WUA Formation and Support When WUAs have become established there may be a need for WUAs to federate and to take on management responsibilities for the secondary and possibly primary canal and drainage systems. Further support will be required for this purpose, potentially involving changes to the legislation, awareness creation, training and capacity building. In addition a National Association of WUAs may be formed, bringing together all WUAs in a single body to represent the interests of water users nationally. When established the National Association can be a powerful lobbying agency to politicians and government, and a provider to its member organisations for services such as advisory services, training and capacity building. Table 1: Key activities, outputs and outcomes for WUA formation and support No. Activity Possible outputs Possible outcomes 1 Enact new, or upgrade · Existing water law revised · WUAs legally registered under new WUA existing, legislation for · New WUA law enacted law establishing WUAs · Model WUA statutes drafted and Federations · Model WUA by-laws drafted 2 Formation of WUA · WUA Support Units (SUs) formed · Formed and functioning WUAs, ably Support Units and functioning with offices, vehicles supported by the WUA Support Unit and equipment · Trained Support Unit personnel 3 Formation and · WUAs formed by agreement of water · WUAs functioning according to the law, establishment of users WUA statutes and by-laws WUAs · Statutes and by-laws discussed and · Representative system functioning, agreed by water users representing water users views and · WUA Council elected and functioning concerns and providing feedback · WUA management executive · Irrigation service fee (ISF) level set and appointed and functioning fees collected to match sustainable · Representative system discussed management, operation and maintenance and agreed by water users (MOM) needs · Representatives elected and · I&D system being operated and functioning maintained by WUA Management · General Assembly held and minutes Executive taken · Performance being monitored and reported by WUA Management Executive and WUA Council 4 Publicity, · Communication and awareness · Water users fully aware of WUA, its duties communication and needs assessment carried out and and responsibilities. awareness campaigns plan prepared · Water users fully aware of their own duties · Communication and awareness and responsibilities in respect of the WUA campaign implemented ­ brochures, · Water users supportive of the WUA, using leaflets, posters prepared and their services and paying the ISF distributed, meetings held, etc. 5 Training and capacity · Training needs assessment carried · WUA Council know roles and building programmes out and training plan produced responsibilities and function effectively · Training implemented ­ training · WUA Management Executive know roles material prepared, courses organised and responsibilities and possess relevant and run skills to function effectively · Training evaluated · WUA accounts kept properly, as verified by annual audits · Operation improved ­ water allocation planned and delivery is reliable, adequate, timely and equitable · Maintenance improved ­ maintenance work identified, planned and carried out in good time. Maintenance expenditure adequate to sustain the system 6 Development of · Knowledgeable and capable · Well managed WUA ­ democratic, management management staff functioning representative and reporting to members capability, including effectively effectively record keeping and · Knowledge of what records are · Well kept records GN16 M&E WUAs 3 Toolkit for monitoring and evaluation of AWM Guidance Note 16 projects M&E for WUA Formation and Support performance required and how to keep them · WUA performance monitored and monitoring · Knowledge of performance evaluated as part of the management monitoring procedures process 7 Development of · Knowledgeable and capable financial · Financially well managed WUA financial management management staff functioning · Well kept financial records capability effectively · ISF level set and fees collected to match · Knowledge of financial procedures MOM financial needs and tax codes · Effective ISF setting and collection procedures in place 8 Development of · Knowledgeable and capable O&M · I&D system well operated - water technical management staff functioning effectively distribution reliable, adequate, timely and capability (system · Knowledge and application of adequate operation and objectives and procedures for · I&D system well maintained ­ maintenance) operation and maintenance of I&D maintenance funds adequate and systems maintenance needs identified and acted upon in good time · Water users satisfied with water delivery, and willing to pay ISF 9 Support for the · Need for maintenance machinery and · Maintenance work being carried out purchase of equipment identified and costed effectively and efficiently maintenance · Water users accept repayment plan · System operation not hindered by machinery and for maintenance machinery and maintenance issues equipment equipment · Water users satisfied with maintenance · Machinery and equipment purchased performance and repaying loans for and in operation machinery and equipment 10 Formation and · Federations of WUAs formed by · Federations functioning according to the establishment of agreement of WUAs and water users law, statutes and by-laws Federations of WUAs · Statutes and by-laws discussed and · Irrigation service fees (ISFs) being set and agreed by water users collected · Federation Council elected and · I&D system being operated and functioning maintained by Federation Management · Federation management executive Executive appointed and functioning · Performance being monitored and · Fee setting and collection procedures reported by Federation Management in place and functioning Executive and Council 11 Development of · Establishment of WUA Regulatory · WUA RA functioning effectively and processes and Authority (RA) incorporated in WUA efficiently procedures for WUA law · WUAs reporting to WUA RA Regulatory Authority · WUA RA established with offices, · WUA RA taking action where WUAs are staff and resources not conforming to regulations · Processes and procedures for WUA · WUA performance monitored RA identified, established and functioning 12 Formation and · National Association of WUAs formed · National Association functioning according establishment of by agreement of WUAs and water to the law, statutes and by-laws National Association users · Membership fees set and collected of WUAs · Statutes and by-laws discussed and · Performance being monitored and agreed reported by National Association · National Association Council elected Management Executive and Council and functioning · National Assembly activities being · National Association management implemented effectively and efficiently executive appointed and functioning · National Assembly held and minutes taken · National Assembly activities identified, costed and agreed by members, and implemented Source: Authors GN16 M&E WUAs 4 Toolkit for monitoring and evaluation of AWM Guidance Note 16 projects M&E for WUA Formation and Support Key performance indicators, data needs, and analysis Overview Typical objectives and related performance indicators for interventions to establish and support Water Users Associations are summarised in Table 2. Table 2: Typical implementation and results framework for interventions to establish and support Water Users Associations Assessment level Examples Project development Effective and sustainable water users' institutions and organisations established objective Project outcomes · Responsibility for management, operation and maintenance and financing of I&D systems effectively transferred from government to water users · Government effectively regulating WUAs and Federations of WUAs · Irrigation water delivery is reliable, adequate, timely and equitable · Systems are adequately and sustainably maintained · Water users are satisfied with water service provision · Agricultural production is not constrained by (lack of) irrigation and drainage service provision · Adequate fees are recovered from water users to cover MOM costs Project outputs · Legal framework for WUAs formulated or revised and in use · Effective and functioning WUA Support Units · WUAs legally formed and functioning effectively - democratic, representative, efficient and effective in work functions · WUA Federations legally formed and functioning effectively · National WUA Association formed and functioning effectively · WUA Regulatory Unit formed, staffed and functioning effectively · WUA offices established, equipped and functioning effectively · WUA personnel trained and effective in their job functions · Water users contacted and made aware of roles and responsibilities · Relevant government agency staff identified and made aware of roles and responsibilities for WUAs and themselves Project activities · Enact new or upgrade existing legal framework for establishing WUAs and Federations · Formation of WUA Support Units · Formation and establishment of WUAs · Publicity, communication and awareness campaigns · Training and capacity building programmes · Development of management capability, including record keeping and performance monitoring · Development of financial management capability · Development of technical management capability (system operation and maintenance) · Support for the purchase of maintenance machinery and equipment · Development of processes and procedures for WUA Regulatory Authority · Formation and establishment of Federations of WUAs · Formation and establishment of National Association of WUAs Project inputs · Specialist inputs ­ legal specialists, WUA specialists, institutional development specialists, training specialists · Beneficiary participation · Offices, machinery, equipment, vehicles and materials Source: Authors The indicators used for monitoring and evaluation of the WUA formation and support are relatively wide ranging, covering institutional, financial and technical issues related to the establishment of the WUAs, their management and the operation and maintenance of the irrigation and drainage (I&D) systems. During implementation the focus is on monitoring the progress of the various activities against the plan, GN16 M&E WUAs 5 Toolkit for monitoring and evaluation of AWM Guidance Note 16 projects M&E for WUA Formation and Support project time expended and budget. Measures include how many WUAs have been formed, how many training course run, how many and what category of people have been trained, etc. Because the project interventions take place in a staged programme, impact assessment can be carried out before the end of the project as project interventions are completed in individual WUAs. Information is collected before the interventions are started and then again following completion, though care has to be taken that sufficient time is allowed to enable the full benefits of the interventions to be realised. Implementation monitoring Implementation measures and indicators for the project activities listed in Table 1 are provided in Table 2. These measures and indicators will generally be reported on in the project's Quarterly Reports. A useful measure of WUA progress can be through the use of milestones. For the World Bank funded On-Farm Irrigation Project in the Kyrgyz Republic the reporting system for milestones is shown in Table 3. When a WUA has achieved Milestone 7 it becomes eligible for rehabilitation works under the project. Table 3: Framework for WUA development monitoring using milestones Milestone Number of WUAs by Region A B . N Total 1. Formerly established Last reporting period Current reporting period Changes 2. Staff hired and training started Last reporting period. Current reporting period, changes 3. O&M plan prepared Last reporting period. Current reporting period, changes 4. Irrigation Service Fee paid Last reporting period. Current reporting period, changes 5. Rehabilitation alternatives developed Last reporting period. Current reporting period, changes 6. Rehabilitation alternative selected Last reporting period. Current reporting period, changes 7. WUA is ready for cooperation Last reporting period. Current reporting period, changes Source: Annex J, Project Status Report, Quarter I, 2006. On-Farm Irrigation Project, Kyrgyz Republic. Once established the project will need to monitor the status of each WUA. This is needed to monitor the progress of WUAs towards the project's objectives and also helps to identify WUAs which are not performing well and where additional support is required from the project. Table 4 provides examples of key performance indicators used for annual monitoring and evaluation of WUA performance following the sequence of activities set out in Table 1. These indicators can be broadly grouped (Table 5) into three categories: · Institutional · Financial · Technical The institutional indicators focus on the membership, level of representation and level of accountability within the WUA. The financial indicators focus on the area irrigated and the level of fee collection from the irrigated area. The financial management processes are also considered by checking if the WUA has an accountant, and that the association's books have been audited and found satisfactory. The technical aspects focus on water management and system maintenance, with a check that sufficient funds are being invested in the maintenance of the infrastructure. GN16 M&E WUAs 6 Toolkit for monitoring and evaluation of AWM Guidance Note 16 projects M&E for WUA Formation and Support A scoring system2 is given in Table 5 in order that WUAs can be ranked and good and bad performers identified. Where performance is good the reasons for the good performance can be investigated and the lessons learnt transferred to other WUAs. Where performance is poor further investigation is required to identify the causes in order that remedial action can be taken by the project. Table 4: List of possible activities and indicators for implementation monitoring No. Activity Measures and Indicators 1 Enact new, or upgrade · Status of legislation (drafted, enacted, in use) existing, legislation for establishing WUAs and Federations 2 Formation of WUA Support · Number of Support Units formed (each quarter, year) Units · Number and types of staff · Training events carried out (for Support Unit staff) 3 Formation and · Number of WUAs formed (each quarter, year) establishment of WUAs · Milestone achieved (formed, staff hired, O&M plan prepared, etc.) · Area covered by WUAs (area and as a percentage of the total irrigable area in the country) · Number of WUAs formed in each Region · Assets transferred from government to WUA account 4 Publicity, communication · Status of campaigns (needs identified, material produced, campaign and awareness campaigns started, activities done, etc.) · Number and types of people, communities, agencies, etc. contacted through the campaigns · Impact evaluation (pre- and post campaign awareness assessment) 5 Training and capacity · Status of programmes (needs identified, training plan produced, training building programmes material produced, trainees identified, training course run, etc.) · Number and types of training courses carried out · Number and types of people trained · Training evaluation (pre- and post-training knowledge tests, pre- and post-training assessment of understanding, knowledge and skills) 6 Development of · Status (identification of needs, development plan, management systems management capability, functioning, etc.) including record keeping · Implementation of plan ­ training, preparation of maps, records, filing and performance system, etc.) monitoring · Performance monitoring ­ meetings held, level of attendance, complaints, issues arising, etc. 7 Development of financial · Status (identification of needs, development plan, financial systems management capability functioning, etc.) · Implementation of plan ­ training, preparation of recording systems, bills and receipts, etc. · Performance monitoring ­ fee level set, budget, expenditure, fee collection, results of annual audit, etc.) 8 Development of technical · Status (identification of needs, development plan, O&M systems management capability functioning, etc.) (system operation and · Implementation of plan ­ training, preparation of scheduling systems, maintenance) water delivery records, etc. · Performance monitoring ­ water abstracted and used, amount of water invoiced and paid for, crops grown, yields, maintenance work carried out, complaints, issues arising, etc.) 9 Support for the purchase of · Status (identification of needs, plan, purchased, etc.) maintenance machinery · Implementation of plan ­ training, purchase, cost, etc. and equipment · Performance monitoring ­ work completed each year, expenditure on fuel, maintenance, etc 10 Formation and · Number of Federations formed (each quarter, year) 2 Note that this scoring system has been developed for this particular situation (Albania), it will need to be adapted for other situations. Where there are monetary units used in the scoring system it is important that these are updated annually in line with inflation GN16 M&E WUAs 7 Toolkit for monitoring and evaluation of AWM Guidance Note 16 projects M&E for WUA Formation and Support establishment of · Milestone achieved (formed, staff hired, O&M plan prepared, etc.) Federations of WUAs · Area covered by Federations (area and as percentage of total area) · Number of Federations formed in each Region 11 Development of processes · Status (formed, staff trained, etc.) and procedures for WUA · Activities being carried out (reporting forms prepared, reports received, Regulatory Authority database designed and operational, etc.) · Number of complete sets of records for WUAs · Performance monitoring ­ Reported on in Annual Report: WUAs status, budget, expenditure, fees set and recovered, maintenance expenditure, etc. 12 Formation and · Status (discussions, National Association formed, meetings held, etc.) establishment of National · Number of members and total area covered Association of WUAs Source: Authors Table 5: Example of key indicators used to monitor the performance of Water Users Associations Water Users Association Performance Indicators Indicator Definition Value Scoring Score Formation Area transferred to WUA Area transferred to WUA 2 = 100% Total gross area serviced by the system 1 = 50-99% 0 = <50% Membership, Representation and Accountability WUA membership Total number of WUA members 2 = >50% ratio Total number of irrigators in service area 1 = 25-50% 0 = <25% Annual General Annual General Meeting held 2 = Yes Meetings 0 = No Annual General Number of WUA members attending AGM 2 = >50% Meeting attendance Total number of WUA members 1 = 30-50% 0 = <30% Administrative Council Number of meetings held during the year 2 = >5 meetings held (January-December) 1 = 1-5 0 = 0 Administrative Council Number of elections for members of 2 = Yes elections Administrative Council held in last 2 years 0 = No Women members of Number of women members of 2 = 1 or more Administrative Council Administrative Council 0 = None Area irrigated First irrigation crop Total annual recorded (first) irrigation crop 2 = >50% area ratio (of total area 1 = 30-50% service area) Total gross area serviced by the system 0 = <30% Crop audit correction Reported area of first irrigation 2 = >90% factor Crop area measured from crop area audit 1 = 75-90% survey 0 = <75% Financial Employment of Accountant employed and duration of 2 = Yes, >4 months Accountant employment 1 = Yes, <4 months 0 = None ISF collection per Total ISF collected 2 = >1800* Lek/ha hectare of service Total gross area serviced by the system 1 = 1000-1800 Lek/ha area * Adjusted to current values 0 = <1000 Lek/ha ISF collection as Total ISF collected 2 = >90% percent of target Target total annual Irrigation Service Fees 1 = 60-90% 0 = <60% ISF collection per Total ISF collected 2 = >2500* Lek/ha hectare irrigated Total annual irrigated crop area 1 = 1000-2500 Lek/ha * Adjusted to current values 0 = <1000 Lek/ha Financial Audit of Level of approval of WUA financial affairs by 2 = Accounts approved WUA independent auditors 1 = No audit undertaken 0 = Accounts qualified/rejected Operation Area managed by Total gross area serviced by the system 2 = < 250 ha GN16 M&E WUAs 8 Toolkit for monitoring and evaluation of AWM Guidance Note 16 projects M&E for WUA Formation and Support Water Masters Number of Water Masters employed by WUA 1 = > 250 ha 0 = No Water Masters Degree of flow Level of flow measurement at the head of the 2 = Full water measurement measurement system (either primary canal or secondary record canals) 1 = Some water measurement 0 = No measurement Maintenance Annual maintenance Extent of annual maintenance planning, 2 = Inspection undertaken planning costing and implementation and detailed plan produced Note: The inspection plan must be reviewed 1 = Maintenance plan and scored by the PMU staff. produced without proper inspection 0 = No plan produced. Maintenance Maintenance cost 2 = >1000* Lek/ha expenditure per unit Total gross area serviced by the system 1 = 500-1000 Lek/ha of total service area *Adjusted to current values 0 = <500 Lek/ha Maintenance Maintenance expenditure 2 = >70% expenditure to Gross revenue collected 1 = 40-70% revenue ratio 0 = <40% Total Score Sum of scores for performance indicators. 2 = >32 Top scores indicate Water Users Association 1 = 20-32 than need no further support. 0 = <20 Source: Halcrow, 2003 Note: 1 US$ = 140 Lek (2002) Results monitoring Most methods for M&E can be adapted to assess WUA development. A more rigorous evaluative framework, to conform to the principles of comparability (or measurement), and causality (or attribution), poses great challenges to the practitioner, but obstacles are not insurmountable. Of particular use here are household surveys and participatory M&E (see GN6 for broad guidance on data collection, analysis, and use, and GN11 on participatory M&E). Household surveys are useful in producing both objective data ­ frequency and types of specific, concrete actions respondents have taken ­ and perceptual information, when respondents apply a score to their own evaluation of the qualitative dimensions of those interactions (Alsop and Heinsohn, 2005). Such surveys are reliable sources of valid and quantitative information. They are, however, expensive to implement, and complex to design and analyse. GN9 provides more information on household surveys, and sample household surveys are provided in the Resource Notes section, RN3 and RN4. Participatory M&E has great advantages in assessing institutional changes at the local level (GN11). Participatory M&E provides one means to collect and systematically capture data that reflects local people's views and perceptions, and is also paramount in (Sirker and Shah, 2003): - setting realistic targets or range for targets - promoting transparency and accountability - deepening the participation process by providing further opportunities for people to improve their voice and empowerment during the process of project implementation GN11 on Participatory M&E describes in greater detail the strengths and challenges of this method. Interim impact studies can be carried out by the project through a programme of pre-and post intervention data collection, as set out in Table 6. This assessment measures the performance before and after WUA formation and system rehabilitation, producing data for the following indicators: · Cropping intensity (%) · Water supply per unit command area (m3/ha) · Water supply per unit irrigated area (m3/ha) · Total gross value of production per unit command area ($/ha) · Total gross value of production per unit water supply ($/m3) · Total ISF collected per unit command area ($/ha) GN16 M&E WUAs 9 Toolkit for monitoring and evaluation of AWM Guidance Note 16 projects M&E for WUA Formation and Support · Total ISF collected per unit water supply ($/m3) · Percentage payment to Service Provider (%) · ISF collection rate (%) · ISF collected as a percentage of gross value of production (%) · O&M expenditure per unit command area ($/ha) · O&M expenditure as percentage of total ISF collected (%) · Maintenance expenditure per unit command area ($/ha) From these data graphical plots can be made to compare the performance of the WUAs in the sample set (Figure 1). In the case shown (Table 5) the performance following WUA formation and rehabilitation has improved, the Total Gross Value of Production (GVP) per unit area and per unit water supply has increased, O&M expenditure has increased, fee collection per unit water supply has increased, and the ISF collection rate has also increased. The amount of water used has decreased significantly, indicating more efficient on-farm water use, and significantly increasing GVP per unit of water. GN16 M&E WUAs 10 Toolkit for monitoring and evaluation of AWM Guidance Note 16 projects M&E for WUA Formation and Support Table 6: Data requirements and indicators for interim impact studies 2006 Impact Study WUA Summary Sheet Name of WUA: Sarcha Rayon: Ala-Buka Oblast: Jalal-Abad Year established: 06-Jun-05 Registered: Ayil Okmotu: Ala-Buka Population: 5,550 No. WUA members: 750 Farmers' farms: 13 No. Individual farmers: 6 No. Water source: Stary Lazvan off-farm canal Situation prior to irrigation: On-farm irrigation system was partially broken; rubber seals on concrete-lined canals were worn out; sections of canalette were broken or missing; tail end sections of earthen canals were eroded; water conveyance efficiency was low. Rehabilitation work done: 16.6.km of canals repaired; 3 hydroposts constructed; 70 structures constructed Resolution for rehabilitation passed by: 81% of WUA membership Feasibility study: Dec-02 Rehab contact signed: 28-Aug-03 Completed: 31-Dec-04 Construction period: 16 months Total contract value (Som):6,721,400 per unit area: 4623 Som/ha Before rehab. After rehab. %change after PERFORMANCE ASSESSMENT Units 2000 2001 2005 2006 and before Command and irrigated area Command area: ha 1,461 1,461 1,454 1,454 -0.5% Irrigated area: ha 1,397 1,461 1,454 1,453 4.0% Water supplied to WUA: Total: m3 8,900,000 8,900,000 4,900,000 4,859,000 -45.4% Total per unit command area: m3/ha 6,092 6,092 3,370 3,342 -45.1% Total per unit irrigated area: m3/ha 6,372 6,092 3,370 3,345 -47.5% Crop yields: Winter wheat ton/ha 2.6 2.9 3.5 3.5 34.6% Grain maize ton/ha 4.1 4.3 5.2 4.8 17.1% Tobacco ton/ha 1.8 1.8 1.9 1.9 5.6% Sunflower ton/ha 1.4 1.4 2.2 2 42.9% Potatoes ton/ha 14 15 19 19 35.7% Vegetables ton/ha 15 16 22 22 46.7% Orchards ton/ha 0 8 10.5 - Perennial grass ton/ha 1.8 2.2 -100.0% Cropping pattern % area % area Winter wheat % 29.4 33.6 17.2 30.9 5.1% Grain maize % 12.1 23.6 18.9 17.9 47.9% Tobacco % 1.8 2.4 0.1 0.7 -61.1% Sunflower % 12.1 16.5 24.1 21.2 75.2% Potatoes % 2.9 10 4.8 1.7 -41.4% Vegetables % 3.2 8.8 0.7 2.4 -25.0% Orchards % 9.4 9.4 -100.0% Perennial grass % 0 5.1 0.3 - Homestead lands 24.7 24.8 24.8 0.4% Total (Cropping intensity) % 95.6 100.0 100.0 99.9 4.5% Value of production: Total gross value of production Som 27,538,097 64,419,581 42,157,276 36,251,928 31.6% Total gross value of production/unit area Som/ha 18,849 44,093 28,994 24,933 32.3% Total gross value of production/unit water Som/m3 3.094 7.238 8.604 7.461 141.1% ISF collection: ISF rate tyin/m3 3.2 4.2 5 4.8 50.0% ISF due Som 284,800 373,800 245,000 233,232 -18.1% Total ISF collected Som 213,640 210,274 217,820 184,549 -13.6% Paid to RID Som 158,058 158,080 123,170 103,057 -34.8% Total ISF collected per unit area Som/ha 146 144 150 127 -13.2% Total ISF collected per unit water supply Som/m3 0.024 0.024 0.044 0.038 58.2% ISF collection rate % 75.0% 56.3% 88.9% 79.1% 5.5% ISF collected as % of total gross prod. value % 0.8% 0.3% 0.5% 0.5% -34.4% O&M expenditure: Total O&M expenditure Som 55,582 52,194 94,650 81,492 46.6% Total O&M expenditure/unit area Som/ha 38.0 35.7 65.1 56.0 47.3% Expenditure on maintenance Som 0 0 0 20,036 - Expenditure on maintenance/unit area Som/ha 0.00 0.00 0.00 13.78 - Source: On-Farm Irrigation Project, Kyrgyz Republic, January 2007 GN16 M&E WUAs 11 Toolkit for monitoring and evaluation of AWM Guidance Note 16 projects M&E for WUA Formation and Support Figure 1: Plots of post WUA formation and system rehabilitation performance and change from pre WUA formation and rehabilitation situation )%( egnahc egatnecreP )%( egnahc egatnecreP %0.053 %0.003 %0.052 %0.002 %0.051 %0.001 %0.05 %0.0 .02- .04- .06- .08- %0 %0 %0 %0 %0.0 %0.05- %0.001- %0.041 %0.021 %0.001 %0.08 %0.06 %0.04 %0.02 noimlaH-uuS-kA noimlaH-uuS-kA noitatilibaher-tsop uuS-akhsaK 6002- aera uuS-akhsaK ot-erp eiraD-yhaS dna eiraD-yhaS ylppus kalapaJ m kalapaJ egnahc % paruM-taruM paruM-taruM reta ahcraS ylppusretawtinurep moctinu ahcraS 6002- wtinu iaM-ukrogoJ noitcudo rep iaM-ukrogoJ aera ukuuJ-nahkraD rep ukuuJ-nahkraD rpfo noitatilibaher-tsop dna ytkypryhC ytkypryhC m P V noitcelloc G leB eulav ot-erp leB F nohC-rutogO egnahc nohC-rutogO moctinurep oboD-nooJ ssorglatoT oboD-nooJ % SIlatoT rumO-uuS rumO-uuS detcelloc 00.52 00.02 00.51 00.01 00.5 00.0 0.004 0.053 0.003 0.052 0.002 0.051 0.001 0.05 0.0 FSIlatoT )3m/moS(retaw tinu rep PVG )ah/moS( aera tinu rep FSI latoT )%( egnahc egatnecreP )%( egnahc egatnecreP %0.06 %0.04 %0.02 %0.0 %0.02- %0.04- %0.06- noimlaH-uuS-kA noimlaH-uuS-kA uuS-akhsaK uuS-akhsaK aera noitatilibaher-tsop %0.08 %0.06 %0.04 %0.02 %0.0 %0.02- %0.04- %0.06- eiraD-yhaS eiraD-yhaS dna kalapaJ ot-erp kalapaJ noitatilibaher-tsop m paruM-taruM paruM-taruM moctinu egnahc ot-erp ahcraS etar % ahcraS iaM-ukrogoJ iaM-ukrogoJ egnahc rep % ukuuJ-nahkraD 6002- noitcelloc ukuuJ-nahkraD ylppus ytkypryhC aera F ytkypryhC dna SI 6002- leB leB m reta etar nohC-rutogO nohC-rutogO W oboD-nooJ moctinurep oboD-nooJ noitcelloc rumO-uuS 000,8 000,7 000,6 000,5 000,4 000,3 000,2 000,1 0 ylppusreta rumO-uuS FSI %0.0 )ah/3m( W %0.041 %0.021 %0.001 %0.08 %0.06 %0.04 %0.02 )%( etar noitcelloc FSI aera tinu rep ylppus retaW Source: On-Farm Irrigation Project , Kyrgyz Republic, January 2007 GN16 M&E WUAs 12 Toolkit for monitoring and evaluation of AWM Guidance Note 16 projects M&E for WUA Formation and Support Further Reading Alsop, R and N. Heinsohn. 2005. Measuring Empowerment in Practice: Structuring Analysis and Framing Indicators. World Bank Policy Research Working Paper 3510, February 2005. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTEMPOWERMENT/Resources/41307_wps3510.pdf Halcrow. 2003. Water Users Association Survey 2003. Second Irrigation and Drainage Rehabilitation Project, Albania. Halcrow Water, Burderop Park, Swindon, United Kingdom. Ostrom, E. 1992. Crafting institutions for self-governing irrigation systems. ICS Press, Institute of Contemporary Studies, San Francisco, California Sirker, K., and T. Shah. 2003. Community Based Monitoring & Evaluation System Methodology Design Workshop (June 11­14, 2002) and Follow-up (January ­ March 2003) Report. Draft, August 23, 2004. World Bank Institute's Community Empowerment and Social Inclusion Learning Program. Available at http://info.worldbank.org/etools/library/latestversion.asp?49429 World Bank. 2006a. Directions in Development: Reengaging in Agricultural Water Management. Challenges and Options. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTARD/Resources/DID_AWM.pdf Website World Bank Social Development Department. Community Driven Development: http://go.worldbank.org/24K8IHVVS0 GN16 M&E WUA formation 13 Toolkit for monitoring and evaluation of AWM Guidance Note 17 projects Institutional Development in AWM Projects Guidance Note 17 Institutional Development in AWM Projects Introduction Institutional development is an intrinsic part of most AWM projects, ranging from interventions to change a country's water code to giving more rights and responsibilities to water users (World Bank, 2006). The focus is on water management, making more efficient and productive use of the available water resources, thus increasing incomes and leaving more water in the river available for other uses and users, including the natural environment. As outlined in GN16 institutional development seeks to build the social and human capital of communities and organisations, such that they are better able to manage the physical infrastructure. Institutional development requires changes: - In institutions - the laws, rules, policies or norms governing the interactions between society, individuals and organisations; - In organisations in the way they go about their business, their processes and procedures; and - By individuals in the manner they interact with each other, and with water-related organisations. These changes are occurring rapidly at present, brought about by a number of drivers for change such as decentralization, accountability, stakeholders' participation, recognition of the holistic nature of water, and the need for integrated water resources management. In the relationship between service providers and water users, increasing levels of education and economic development are driving individual expectations and narrowing the knowledge and skills gap between the formerly powerful government irrigation and drainage agencies and water users. Recognizing limited capacity and funding by government and the need to build ownership by water users, the management of irrigation and drainage systems is being transferred to water users. With increased responsibility and experience, water users are challenging the norms that have held for the last 50-60 years and are demanding better levels of service from government irrigation and drainage agencies. These agencies in turn have to change and adapt to the new reality, becoming more focused on service delivery and satisfying their customers' demands. A key factor in this process is the increasing requirement for users to pay the full cost for the service delivered; for this payment, water users are requiring not only better service, but also more transparency and accountability in the use of the service fees. Gradually the relationship between service provider and water user is changing to a more equal partnership. Despite increased importance in projects, little guidance is yet available on the monitoring and evaluation of institutional development impact. This GN lays out some principles for such M&E, from the identification of typical institutional development components to indicative related output, outcomes, and indicators. Box 1 shows an example of application in a World Bank project. GN17 Institutional Development 14 Toolkit for monitoring and evaluation of AWM Guidance Note 17 projects Institutional Development in AWM Projects Box 1: Institutional Development in Maharashtra Water Sector Improvement Project, India The World Bank funded Maharashtra Water Sector Improvement Project, India seeks to strengthen the state's capacity for multi-sectoral planning, development and sustainable management of the water resources, and improve irrigation service delivery and productivity of irrigated agriculture. The major project component, improving irrigation service delivery and management worth US$321 million, supports rehabilitation and modernization of selected irrigation schemes alongside with formation and capacity building of Water Users' Associations in the schemes. The US$ 21 million component on water sector institutional restructuring and capacity building supports (i) the establishment and operationalization of Maharashtra Water Resources Regulatory Authority, (ii) the restructuring of the existing Maharashtra Krishna Valley Development Corporation into the Maharashtra Krishna Valley Water Resources Corporation as a river basin agency, and its capacity building; (iii) the restructuring and capacity building of the Water Resources Department, (iv) capacity building of the Water and Land Management Institute, and (v) the establishment of an integrated computerized information system. A number of institutional indicators have been identified as key performance indicators during appraisal: - Establishment and operationalization of Maharashtra Water Resources Regulatory Authority - Initiation of and progress made on restructuring of Maharashtra Krishna Valley Development Corporation, and its capacity building - Restructuring and capacity building of Water Resources Department - Formation, operationalization, and fostering of Water Users Associations in the selected irrigation schemes Source: World Bank PAD, Maharashtra Water Sector Improvement Project Typical institutional development components Typical project interventions related to institutional development are summarised below and in Box 2. Table 1 lists these key activities together with their possible, indicative, outputs and outcomes. Table 2 outlines possible output and outcome indicators related to these activities, which are discussed in turn in the sections below. (i) Enact new or upgrade existing water resources legislation. A precursor to many Box 2: Typical institutional development institutional development interventions in the components of AWM projects water sector is upgrading and modernisation of Enact new or upgrade existing water resources the water resources legislation. This may legislation include, inter alia: legislation related to the Reorganisation/restructuring of water resources formation of river basin councils; the formation departments of water users associations (see below); Reorganisation/restructuring of government irrigation restructuring the water resources agency and and drainage agencies its roles and responsibilities; restructuring the Transfer of irrigation and drainage agency roles and irrigation and drainage agency and its roles and functions from government to parastatal or private responsibilities; strengthening the entities environmental control on water abstraction and Enact new or upgrade existing legislation for water users associations wastewater disposal. Without satisfactory Formation and support of water users associations, legislation many project interventions cannot be federations of water users associations and national successfully implemented. A key factor to unions of water users associations consider when embarking on changes in the Publicity, communication and awareness campaigns water resources legislation is the time taken to Establishment of regulatory authorities for water formulate, discuss, revise and eventually pass resources management, irrigation and drainage, and the legislation. Sana'a Basin Water water users associations Management Project in Yemen for example Formation of river basin councils and associated aims at assisting the National Water Resources organisations and institutions Formation of Water Councils Authority with the drafting and finalization of Training and capacity building bylaws and regulations for water rights management. Source: Authors GN17 Institutional Development 15 Toolkit for monitoring and evaluation of AWM Guidance Note 17 projects Institutional Development in AWM Projects (ii) Reorganisation/restructuring of water resources departments. Government water resources departments are being reformed in order to address issues arising from the increasing pressure on water resources quantity and quality. In some situations the water resources department has been responsible for both the water resource allocation and the management of the irrigation systems. As the pressures on the available water supplies increase, these two functions are being separated, with a water resources department being formed to allocate, manage and regulate the water allocation to water users, one component of which is irrigation. Irrigation, though often the major water user, is therefore increasingly seen as a separate bulk water supply function, similar to potable water supply. In these cases, the reorganisation/restructuring of the water resources department entails separating the two functions and creating two separate legal entities, one for water resources allocation and regulation, often on a river basin framework, the other for irrigation water allocation and service fee recovery. (iii) Reorganisation/restructuring of government irrigation and drainage agencies. Government run irrigation and drainage agencies are being reorganised/restructured as part of the process of irrigation management transfer to water users. With the changes taking place with greater participation and involvement by water users in irrigation and drainage system management, government I&D agencies have to change and evolve. In some cases the government I&D agency is being reorganised, in other cases it is being transformed into a parastatal or private entity (see below). (iv) Transfer of irrigation and drainage agency roles and functions from government to parastatal or private entities. In some cases the role and functions of the government irrigation and drainage department is being transferred to parastatal or private entities. This process has been implemented since 1963 in the Philippines with the formation of the National Irrigation Administration (NIA), and more recently in December 2006 in Georgia where the former Department for Amelioration Scheme Management (DASM) was dissolved and four state-owned limited companies formed to take over DASM's roles and responsibilities. The purpose of these transfers is to increase the level of autonomy of the I&D service provider, reduce the dependency on government funding and increase the level of fee recovery from water users. (v) Enact new or upgrade existing legislation for water users associations. The establishment of water users associations, and their roles and responsibilities, need to be detailed in the water resources legislation, and in most cases, in separate specific legislation. In some countries existing legislation on cooperatives or legal entities has been used to form water users associations; this is in general not as successful as specific legislation directed at water users associations. (vi) Formation and support of water users associations, federations of water users associations and national unions of water users associations. A major change in the irrigation and drainage sector in recent years has been the formation and support of water users associations. As water users have gained experience and confidence in the management of their irrigation and drainage systems they have often moved to form Federations of WUAs and National Unions of WUAs. Federations of WUAs have taken over responsibility for the management of the main, or off-farm, irrigation and drainage systems, whilst National Unions of WUAs have been formed to represent irrigation water users interests to government and other water users. GN16 provides more details on monitoring and evaluation of water users associations, federations and national unions. (vii) Publicity, communication and awareness campaigns. Communication and awareness campaigns may be required at a number of levels. For some projects they may be directed at a the wider audience, such as politicians, government agency personnel and the general public, while for other projects they will be focused on a narrow target group, such as water users within the command area of a recently formed water users association. (viii) Establishment of regulatory authorities for water resources management, irrigation and drainage, and water users associations. Associated with the organisational changes in the water resources sector is the need for greater regulation and control. In management of water resources, GN17 Institutional Development 16 Toolkit for monitoring and evaluation of AWM Guidance Note 17 projects Institutional Development in AWM Projects regulation is required for abstraction of water for irrigation and other users, and disposal of wastewater. Where irrigation and drainage agencies are turned into parastatal organisations or privatised, regulation is required to ensure that they provide the required services at reasonable cost to water users. Similarly at the water user association level, regulation is required to ensure that the water users associations are providing an adequate level of service to water users at a reasonable cost, and are representing their members interests. (ix) Formation of river basin councils and associated organisations and institutions. In some cases, such as the Water Management Improvement Project in Kyrgyzstan, efforts to improve water management are extending from the field level up to the basin level. In this case the Water Code has been revised to allow for the formation of river basin councils and the management of water resources based on these river basins. Associated with this transformation is the formation of the National Water Council and its executive body, the State Water Administration, with regional executive offices in each of the river basins. The State Water Administration will have authority, under the National Water Council, for licensing, monitoring and regulation of all surface and groundwater resources in the country. (x) Formation of Water Councils. In some locations local Water Councils have been formed as a forum for irrigation and other water users and concerned parties to meet to discuss and resolve issues arising from water resource allocation and use. These Water Councils are generally based on local hydraulic systems where there are conflicts over water abstraction and use, and generally comprise WUAs, local government, other water users (domestic and industrial) and the irrigation and drainage service provider. (xi) Training and capacity building. Training and capacity building are key parts of all of the change processes mentioned above. Training has been central to the successful formation of water users associations where there has been little or no previous experience of voluntary users associations. Similarly where government agencies are being reorganised or restructured, training and capacity building are essential in bringing about the required changes in understanding, knowledge and skills. Of particular importance for irrigation and drainage agencies is a change in thinking, moving from a top-down government agency to a customer focussed service provider. Monitoring and Evaluation of institutional development components Implementation monitoring of institutional development components in AWM projects typically focuses on the status of revisions of regulatory texts, formation or restructuring of organizations, and delivery of training programs and communication campaigns. Such aspects should be included in the Management Information System (see GN6). Results monitoring is more challenging but should be attempted. Early outcome or leading indicators (see GN3) such as formal adoption of regulatory texts or effective functioning of newly formed or restructured organizations are particularly useful. As the focus moves to higher level objectives, for example improved water resources management and allocation, the measurement and attribution of change to project components becomes challenging. Data collection methods for results monitoring vary greatly and are linked to the indicator identified. They can include: - Review of administrative records, for example to compute the number of water rights-related disputes solved, or to assess the number of decision meetings held over the year in the newly created water councils - Surveys or key informant/ group interviews to evaluate perceptions on changes brought about by the project. Score cards for example can be used to estimate users' satisfaction with the service provided (see GN11). - Case studies of specific organizations or training programs, such as an in-depth audit of the restructured irrigation agency, or a self-assessment workshop carried out with its staff. GN6 provides more details on the principles for data collection, analysis, and use in M&E systems. GN17 Institutional Development 17 Toolkit for monitoring and evaluation of AWM Guidance Note 17 projects Institutional Development in AWM Projects Table 1 and Table 2 list possible output, outcome, and related indicators for typical institutional development components in AWM projects. Table 1: Key activities, outputs and outcomes for institutional development components of AWM projects No. Activity Possible outputs Possible outcomes 1 Enact new or upgrade · Proposed draft water legislation · Legislation passed by Parliament existing water resources produced · New regulations introduced, regulated and legislation · Draft legislation discussed, enforced reviewed, and revised 2 Reorganisation/restructur · Restructuring study produced · Water resources department ing of water resources · Restructuring study discussed, restructured/reorganised department reviewed and revised · Revised budget agreed with Ministry of Finance · Restructuring study, or components of, adopted 3 Reorganisation/restructur · Restructuring study produced · Water resources department ing of government · Restructuring study discussed, restructured/reorganised irrigation and drainage reviewed and revised · Revised budget agreed with Ministry of Finance agency · Restructuring study, or · I&D agency focussed on service delivery and components of, adopted more dependent on water users for income 4 Transfer of irrigation and · Consultation document produced · New entity created drainage agency roles and discussed · Transfer legislation approved by Ministry and/or and functions from · Asset survey and inventory Parliament government to parastatal carried out and assets transferred · Transfer enacted or private entities · New management structure, processes and procedures · Reduced government involvement in management of irrigation and drainage systems · Increased involvement and participation by water users in system management, operation and maintenance · Reduced government funding to I&D sector, increased funding from water users · Shake-out of viable and non-viable I&D systems 5 Enact new, or upgrade · Existing water law revised · WUAs, Federations and National Union legally existing, legislation for · Model statutes drafted for each registered establishing WUAs, organisation · Law enacted covering the formation of WUAs, Federations and National · Model by-laws drafted for each Federations and National Union Union organisation 6 Formation and support of · WUAs formed by agreement of · WUAs, Federations and National Union WUAs, Federations of water users; Federations and functioning according to the law, statutes and by- WUAs and National National Union by agreement of laws Union of WUAs WUAs · WUAs, Federations and National Union fully · Statutes and by-laws discussed representative of their members, representing and agreed water users views and concerns and providing · Representative Council or Board feedback elected · Irrigation and drainage service fee level set and · Management executive fees collected to match sustainable management, appointed operation and maintenance (MOM) needs, at all levels under WUA or Federation control · I&D system being effectively operated and maintained by WUA and Federation Management Executive · Performance being monitored and reported by WUA and Federation Management Executive and WUA and Federation Council 7 Publicity, communication · Communication and awareness · Organisation executives and staff fully aware of and awareness needs assessment carried out their duties and responsibilities. campaigns and plan prepared · Water users, members and stakeholders fully GN17 Institutional Development 18 Toolkit for monitoring and evaluation of AWM Guidance Note 17 projects Institutional Development in AWM Projects · Communication and awareness aware of their own duties and responsibilities in campaign implemented ­ respect of the organisations of which they are brochures, leaflets, posters members prepared and distributed, · Water users, members and stakeholders meetings held, etc. supportive of the organisations of which they are members, participating in their activities and paying their membership/service fees 8 Establishment of · Establishment of Regulatory · RA functioning effectively and efficiently regulatory authorities for Authority (RA) incorporated in law · Relevant persons/parties reporting to RA water resources · RA established with offices, staff · RA taking action where persons/parties are not management, irrigation and resources conforming to regulations and drainage and water · Processes and procedures for · Relevant persons/parties performance monitored users associations RA identified and established 9 Formation of river basin · National Water Council (or · River basin development coordinated councils and associated similar) formed · Key stakeholders engaged in management of the organisations and · National Water Council executive river basin institutions (or similar) formed · Surface and groundwater abstractions licensed, · River Basin Councils formed monitored, regulated and enforced · Draft river basin plans produced · Waste water disposal to surface and groundwater licensed, monitored, regulated and enforced · River basin ecology protected and enhanced 10 Formation of local Water · Local Water Councils of key · Local water resources issues discussed and water Councils water resources stakeholders allocation and use agreed formed · Local disputes over water resources allocation and use reduced 11 Training and capacity · Training needs assessment · Organisation executive and staff know roles and building programmes carried out and training plan responsibilities and function effectively produced · Organisation executive and staff possess relevant · Training implemented ­ training understanding, knowledge and skills to function material prepared, courses effectively organised and run · Organisation functions effectively - institutionally, · Training evaluated financially and technically · Management, operation and maintenance of irrigation and drainage systems improved ­ water allocation planned and delivery is reliable, adequate, timely and equitable; maintenance work identified, planned and carried out in good time. Maintenance expenditure adequate to sustain the system Source: Authors GN17 Institutional Development 19 Toolkit for monitoring and evaluation of AWM Guidance Note 17 projects Institutional Development in AWM Projects Table 2: Possible implementation and results monitoring indicators for institutional development components of AWM projects No. Component Possible measures and indicators for Possible indicators for results monitoring implementation monitoring 1 Enact new or · Status of draft water legislation (under · Legislation enacted upgrade existing preparation; draft completed; draft · Level of activity in key areas (for the project) water resources reviewed; draft accepted; final draft; covered by the legislation (organisations legislation new/upgraded legislation enacted) formed; actions taken; legislation enforced, etc.) 2 Reorganisation/restr · Status of restructuring study (under · Activity of water resources department since ucturing of water preparation; report completed; report and restructuring (levels of performance - service resources recommendations reviewed; report and provision; liaison with water users; information department recommendations accepted; systems; knowledge on water resource implementation started) availability; allocation and use; licenses issued; knowledge on wastewater discharges and pollution levels; action on pollution; etc.) 3 Reorganisation/restr · Status of restructuring study (under · Activity of irrigation and drainage department ucturing of preparation; report completed; report and since restructuring (levels of performance - government recommendations reviewed; report and service provision; liaison with water users; irrigation and recommendations accepted; service fee setting and fee recovery; drainage agency implementation started) information systems; knowledge on irrigation demand and water resource availability; water allocation and measured use; knowledge on asset inventory; knowledge on management, operation and maintenance costs required to sustain individual systems; environmental protection measures; etc.) 4 Transfer of irrigation · Status of consultation process · Level of performance of new entity (mission and drainage · Status with asset inventory (asset survey; statement; organisational structure and staffing agency roles and assessment; allocation; disposal) levels; level of training and capacity building; functions from · Status with transfer (preparation of legal fee setting and fee recovery; degree of liaison government to documents; enactment of legal transfer; and cooperation with water users; level of parastatal or private transfer completed; closure /restructuring government subsidies; command area serviced entities of former entity) and actually irrigated) 5 Enact new, or · Status of legislation (under preparation; · Number, total command area covered and upgrade existing, draft completed; draft reviewed; draft dates when WUAs and Federations legally legislation for accepted; final draft; new/upgraded registered; date when National Union establishing WUAs, legislation enacted) registered, number of member organisations Federations and · Status of model statutes and by-laws and area covered National Union (see ((under preparation; draft completed; draft GN-16 for more reviewed; draft accepted; final draft detail) passed by Council/Board) 6 Formation and · Status of WUA, Federation and National · Number, total command area covered and support of WUAs, Union formation (initiated; formed; legally dates when WUAs and Federations legally Federations of constituted and registered; functioning) registered; date when National Union WUAs and National · Status of institutional development registered, number of member organisations Union of WUAs (see (number of meetings held per year; and area covered GN-16 for more attendance at Annual General Meeting; · Activity and sustainability of WUAs, detail) level of representation) Federations and National Union over time (age · Status of financial development (level of since legal registration; meetings held; meeting service fee; fee collection rate; level of attendance; level of representation; creditors and debtors) membership/service fee setting and fee · Status of technical development (quantity recovery; level of interaction/influence on of water delivered relative to demand; external organisations and government; level level of complaints over water of service provided to members; for WUAs and supply/drainage; number of water masters Federations - expenditure on maintenance employed; condition of system; against requirements; trends in area cultivated, expenditure on maintenance) cropping patterns, yields and farmer income; trends in water supply and water allocation; level of waterlogging and salinity, etc.) GN17 Institutional Development 20 Toolkit for monitoring and evaluation of AWM Guidance Note 17 projects Institutional Development in AWM Projects 7 Publicity, · Status of communication and awareness · Awareness of identified issues (roles and communication and needs assessment (assessment responsibilities of organisations; rights and awareness completed and plan prepared; plan responsibilities of membership of organisation; campaigns discussed and agreed; plan implemented) costs and benefits of membership of · Status of plan implementation (meetings organisations) held; brochures/flyers/posters produced · Awareness of organisation's activities (dates of and disseminated;) meetings; submissions to meetings; outcomes · Status of evaluation (awareness surveys of meetings) carried out and reported; amendments to · Members support of the organisation (payment plan) of membership/service fees; levels of appreciation of service provided; participation in meetings and organisation's affairs) 8 Establishment of · Status of establishment of Regulatory · Functioning of RA (level of performance of regulatory Authority (draft charter prepared; charter functions ­ regular functions; numbers of authorities for water discussed and agreed; charter and actions taken for non-compliance situations; resources formation legislation passed; RA reduction in numbers of non-compliant management, established) situations; reduction in number of complaints) irrigation and · Status of RA establishment (level of drainage and water staffing; level of activity; number of users associations functions completed; information systems functioning) 9 Formation of river · Status of establishment of National Water · Activity of National Water Council (meetings basin councils and Council (structure and composition held, number and type of decisions made; associated proposed, discussed and agreed; Council decisions implemented; level of participation by organisations and formed) Council members) institutions · Status with establishment of River Basin · Activity of River Basin Councils (meetings held; Councils, (structure and composition number and type of decisions made; decisions proposed, discussed and agreed; Council implemented; level of stakeholder formed) engagement; level of submissions made to · Status with river basin plans (draft RBC) proposals prepared; consultation meetings · Activity of River Basin Council executive held; discussion and agreement; (number of surface and groundwater enactment; implementation) abstraction licences issued; level of monitoring · Status with implementation of river basin of abstractions, discharges, groundwater table plans (planned actions implemented; levels, wastewater disposal; consultation meetings held; level of · Quality of river ecology (water quality and awareness; performance of key indicators quantity; degree of aquatic activity; level of ­ river flows, pollution levels, abstractions, public enjoyment of riverine environment) wastewater discharge levels, etc.) 10 Formation of local · Status with formation of local Water · Activity of Water Councils (meetings held; Water Councils Councils (concept developed; proposal number and type of topics discussed and made; meetings held; proposal discussed agreements reached; level of implementation and agreed; Council formed) and compliance with agreements; level of · Status with Water Council activity (number participation of stakeholders; level of of meetings held; actions planned and satisfaction of stakeholders with Council) taken; agreements reached and results; · Level of local disputes over water resources level of complaints; level of conflict) allocation and use 11 Training and · Status of training needs assessment (TNA · Satisfaction of trainees with the training and capacity building carried out; training plan prepared; plan capacity programs programmes discussed and agreed; plan implemented) · Degree of application of training (evidence of · Status of training implementation (training application of understanding, knowledge and material prepared; courses organised and skills gained - organisations run effectively; run; numbers trained accounts properly kept; systems operated and · Status of evaluation (post-course maintained, etc.) evaluation; evaluation report prepared; amendments to training plan) Source: Authors GN17 Institutional Development 21 Toolkit for monitoring and evaluation of AWM Guidance Note 17 projects Institutional Development in AWM Projects Further Reading World Bank. 2006a. Directions in Development: Reengaging in Agricultural Water Management. Challenges and Options. World Bank, Washington D.C. Available at: http:siteresources.worldbank.org/INTARD/Resources/DID_AWM.pdf GN17 Institutional Development 22 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM Guidance Note 18 Benchmarking in Agricultural Water Management Introduction Benchmarking originated in the corporate business sector as a means for companies to gauge, and subsequently improve, their performance relative to key competitors. By studying key competitors' outputs, and the processes used to achieve those outputs, many organisations have been able to adopt best management practices and enhance their own performance (Box 1). In some cases organisations have done so well that they have, in turn, become the organisation that others use as a benchmark. Benchmarking has been used in the irrigation and drainage sector since the late 1990s. Benchmarking programmes have been initiated in a number of countries, including Australia, China, Egypt, India and Sri Lanka. At the international level, a collaborative programme by the International Water Management Institute (IWMI), the World Bank, the International Commission for Irrigation and Drainage, and the International Program for Technology and Research in Irrigation and Drainage (IPTRID) has led to the development of guidelines for benchmarking in the irrigation and drainage sector (Malano and Burton, 2001), and the development of an Online Irrigation Benchmarking Service, managed by IWMI3. In the water supply and sanitation sector the World Bank has been supporting the International Benchmarking Network for Water and Sanitation Utilities since the 1990s4. There are many reasons why organisations may be interested in the benchmarking activity. The private Box 1: Definition sector is primarily driven by a desire to improve return Benchmarking is "a systematic process for on investment or return to shareholders; in the public securing continual improvement through sector the aim is to improve the effectiveness and comparison with relevant and achievable efficiency of the organisation and the level of service internal or external norms and standards". provision. In the irrigation and drainage sector service providers are responding to a variety of drivers, Source: Malano and Burton, 2001 including: · Increasing competition for water, both within the irrigated agriculture sector, and from other sectors; · Increasing demand on the irrigation sector to produce more food for growing populations. Coupled with the pressure on available water resources, this results in the "more crop per drop" initiative promoted by the International Water Management Institute and the Food and Agriculture Organisation; · Growing pressure to effect cost savings whilst increasing the productivity and efficiency of resource use; 3Available at http://www.lk.iwmi.org:82/oibs/LoadBench.htm 4 The International Benchmarking Network for Water and Sanitation Utilities provides access to the world's largest database for Water Supply and Sanitation utilities with performance indicators of more than 2,000 utilities in 84 countries (for more details, see www.ib-net.org). GN18 Benchmarking 1 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM · Turnover and privatisation of irrigation and drainage schemes to water users, leading to more transparent and accountable (to users) management practices; · Increasing interest by the wider community in productive and efficient water resource use and the protection of aquatic environments; · Increasing need for accountability to both government and water users in respect of water resource use and price paid for water. Different drivers will apply in different situations, and it is important at the outset of a benchmarking programme to identify the key drivers that are forcing change within the irrigation and drainage sector. Benchmarking is about moving from one level of performance to another (Figure 1). It is about changing the way in which systems are managed and about raising the expectations of all parties as to the level of achievable performance. It is a change management process that requires identification of shortcomings, and then acceptance by key stakeholders of the need, and pathways for achieving the identified goals. Benchmarking is part of a strategic planning process which asks and answers such questions as: "Where are we now?", "Where do we want to be?", and "How do we get there?" The difference between project M&E and benchmarking Benchmarking is an altogether different process to project monitoring and evaluation. Benchmarking can be used as a process to identify components of a project to be improved, and standard project M&E processes then used as Stage 6 of the benchmarking process, as outlined in the section below. Benchmarking seeks to identify gaps in system performance, M&E is part of the process of ascertaining how measures taken to close identified gaps are progressing. The results of benchmarking can provide information which can be used to assist the setting of targets for improvement during project design and for use with selected in indicators in M&E (see Part A, Section 3.8 and Chapter 4, and also GN1). GN18 Benchmarking 2 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM Benchmarking stages There are six key stages to benchmarking (Figure 2; Malano et al, 2004; Burton et al, 2005): Stage 1 Identification and planning This stage identifies: · The objectives and boundaries of the benchmarking programme; · Who the benchmarking is for; · The key processes; · The related performance indicators; and · The data requirements. It is important at the outset to identify the objectives and boundaries of the benchmarking exercise. Is the objective to improve the efficiency and productivity of water alone, or irrigated farming as a whole? Is the benchmarking for the individual farmer, the service provider, the regulator or government? Having decided on these key issues, it is necessary to identify the processes involved within the identified boundaries and the related performance indicators and data needs. A key part of the process is to identify successful organisations or irrigation and drainage systems with similar processes. Use of key descriptors (Box 2) enables similar systems and processes to be identified and enables meaningful comparison to take place. For example the water use on a rice scheme will be significantly different from that on a cotton scheme. In identifying the key processes (Figure 3) the following questions can be asked: · What are the objectives of the enterprise? · How is success measured? What are the outputs and desired outcomes? GN18 Benchmarking 3 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM · What are the processes that contribute to the attainment of these outputs and Box 2 Descriptors for irrigation and outcomes? drainage schemes · How can these processes be Irrigable area measured? Drained area Annual irrigated area It is also important to consider the impact of Climate the key processes; the consequences of water Water resources availability abstraction from rivers and pollution from Water source agricultural drainage water are key Average annual rainfall considerations in this respect. Average annual reference crop potential evapotranspiration (ETo) Method of water abstraction Possible key processes and indicators include: Water delivery infrastructure · Irrigation water abstraction, conveyance Type of water distribution and application Type of drainage Volume of water abstracted for Predominant on-farm irrigation method irrigation Major crops (with percentages of total Irrigation water abstraction per unit irrigated area) area Average farm size Relative irrigation water supply Type of irrigation system management Type of drainage system management (abstraction/demand) · Crop production Source: Malano and Burton, 2001 Irrigated area Cropping intensity Crop yield Value of crop production per unit area Value of crop production per unit water abstracted · Business processes Cash flow (investment vs returns) Total annual income Annual profit · Environmental impact Waster water quality (biological/chemical content) Minimum flow levels in river GN18 Benchmarking 4 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM Stage 2 Data collection Data are collected and the value of performance indicators determined. The data collection programme will identify what data are to be collected, by whom, how frequently, where, and how accurate the data need to be. These data are for the system under review and the benchmark system(s), and will include input, process, output, outcome and impact performance indicators. Additional data may have to be collected for the benchmarking exercise beyond those already collected for day-to-day system management, operation, and maintenance. Stage 3 Analysis Data are analysed and the performance gap(s) identified in the key processes (Figure 4). The analysis also identifies the cause of the performance gap, and the action(s) to close the gap. Recommendations are formulated from the options available, and then reviewed, and refined. Further data collection may be required for diagnostic analysis where additional information and understanding are required to identify root causes of the performance gap. This can be either the beneficial causes of the better performing system(s) or the constraining causes of the less well performing systems. GN18 Benchmarking 5 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM Stage 4 Integration To achieve change, the action plan has to be integrated into the operational processes and procedures of the scheme, requiring acceptance by key stakeholders. Benchmarking programmes may fail at this stage due to insufficient attention being paid to gaining acceptance or concurrence with the action plan. Agreement between the water user and the licensing or regulatory authority can play a key role in setting performance targets. Information on realistic and achievable targets can be obtained through the benchmarking process of identifying best practices for key processes on different irrigation systems. Stage 5 Action This stage requires implementation of the proposed actions. Leadership by senior management plays a key role in ensuring that the action plan is implemented successfully. Stage 6 Monitoring and evaluation An important part of the change management programme is monitoring the implementation of the action plan and its impact on the key processes. The performance indicators identified in Stage 1 are central to this process. Figure 2 shows a cyclical programme of activities, though there may be a break of some years between one benchmarking exercise and another. During this period the lessons learnt from the benchmarking programme are implemented, monitored and evaluated, with refinements being made as experience is gained with implementing the new processes and procedures. As mentioned previously it has been the case with some organisations that they have so improved their performance that they have become the benchmark. Examples of benchmarking in agricultural water management Australia The Australian National Committee of Irrigation and Drainage (ANCID) was one of the first organisations to implement a benchmarking programme in the irrigation and drainage sector. It began in 1998 with 33 schemes managed by irrigation service providers and now has over 40 schemes in the programme, covering some 75% of the irrigation water provider business in GN18 Benchmarking 6 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM Australia. The total business distributes 18,000 GL of water annually, providing water for some 2 million ha and generating a water provider gross revenue of US$ 162 million and agricultural production of value US$ 5.7 billion (Alexander and Potter, 2004). The crops grown include rice, maize, grape vines, cotton, sugar cane, pasture, citrus and vegetables. The benchmarking programme uses 65 performance indicators: · System operation (12 indicators) · Business processes (25 indicators) · Financial management (14indicators) · Environmental management (14indicators) These indicators have been formulated to fit with the "triple bottom line" approach adopted by the industry, measuring performance in economic, environmental and social dimensions. A key feature of the Australian benchmarking programme is the "three tier" reporting of data to protect commercial confidentiality. Tier 1 collects data on general irrigation water provision ("Who we are"), Tier 2 collects data on performance ("How we interact") and Tier 3 collects data on confidential internal business performance benchmarking ("How we improve"). The data are collected each year using a standard questionnaire, each contributor indicating what data can and cannot be released. The data are analysed and the report made available to all contributors, with anonymous data presented for others to compare their performance with. If a contributor wishes to obtain more information on the confidential data they write to ANCID who forward their request on to the relevant contributor. Figure 5 presents examples of the performance indicators used. As can be seen there is a wide range in the values of each of the indicators, this is due to individual differences between the systems (the crop types, method of irrigation, lined/unlined canals, etc.). This highlights the importance of using the system descriptors (Box 2) to categorise systems to enable comparison of like with like. GN18 Benchmarking 7 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM GN18 Benchmarking 8 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM The achievements of the benchmarking programme are summarised as (Alexander and Potter, 2004): · Allowing comparison of the performance of irrigation water providers relative to each other, both at the domestic and international level; · Providing a more progressive and accountable image of the irrigation sector; · Monitoring the uptake and impact of modern technology; · Improvement in record keeping and performance analysis by service providers; · Availability of objective and reliable data across a substantial part of the irrigation industry; · Adoption by businesses of the ANCID benchmarking approach and formulation of their own inter-business benchmarking systems; · More confident setting by business managers of targets for water delivery efficiency, operation, health and safety and resource use. Egypt Between May 2004 and September 2005 the World Bank carried out a pilot benchmarking project in Egypt in association with the Ministry of Irrigation and Water Resources. The purpose of the project was to review previous work on benchmarking in Egypt, update and develop benchmarking indicators for use across the I&D sector, pilot the application of benchmarking, train staff from two government departments (Department of Irrigation and the Egyptian Public Authority for Drainage Projects), and disseminate the findings of the exercise. The benchmarking exercise was carried out over two irrigation seasons, the 2004 summer season and the 2004/5 winter season, and covered 6 secondary (branch) canals totalling some 27,770 feddans (11,670 ha). Four of the systems have been improved under the Irrigation Improvement Project, and provided with sub-surface drainage systems by the Egyptian Public Authority for Drainage Projects. Two of the systems are planned for inclusion in the forthcoming Integrated Irrigation Improvement Project. The World Bank has supported/is supporting these two projects, in addition to the pilot benchmarking project. Workshops were held with key stakeholders to identify the purpose and boundaries of the benchmarking exercise. While the overall goal of the benchmarking programme was to sustain and increase agricultural production a specific objective of the benchmarking programme was to make more productive use of available water supplies through better water management and system maintenance. To achieve this objective benchmarking of four key processes was carried out: · Irrigation water delivery · Drainage water removal · Maintenance of infrastructure · Environmental protection (through management of water quality) The indicators used for performance measurement are summarised for the winter season in Table 1. In this table the "best" values have been highlighted in gold, whilst critical values are highlighted in red and areas for concern in yellow. Some of the indicators have not been given highlights as these are indicative indicators and it is not possible to judge them one against another. This is the case for example with the Total Seasonal Crop Water Demand (at field), and the Total Seasonal Irrigation Water Supply per Unit Command Area, where the value depends on the cropping pattern within the Branch Canal ­ there is no one "best" figure here but the value does serve to show the relative scale of supply to, and demand by, each Branch Canal. The Seasonal Relative Irrigation Water Supply is then the prime indicator linking the supply and demand. GN18 Benchmarking 9 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM Table 1: Summary performance table for irrigation and drainage ­ Egypt, Winter 2004 5 Branch Canal y mia l Description stin watnese Nte addia E a U B wiaZ B l tlaqa ruohna meeda leehse E D S K N Irrigation Total seasonal value of crop LE/feddan 2,676 2,684 2,935 2,419 2,730 2,886 production per unit command area Total seasonal value of crop LE/m3 0.88 0.82 1.28 0.68 3.23 2.62 production per unit water supply Total seasonal volume of crop m3/feddan 2,236 2,352 2,226 1,828 2,155 2,326 water demand (at field) Total seasonal irrigation water m3/feddan 3,024 3,289 2,286 3,577 846 1,110 supply per unit command area Total seasonal irrigation water m3/feddan 1,340 2,037 1,339 1,587 574 N/a delivery per unit command area Main system water delivery % 44% 62% 59% 44% 68% N/a efficiency Seasonal relative irrigation water - 0.60 0.87 0.60 0.87 0.41 N/a supply Total seasonal MOM costs for LE/feddan 18.43 18.43 18.43 20.00 20.00 20.00 irrigation water delivery per unit command area Total seasonal MOM costs for LE/m3 0.012 0.011 0.016 0.011 0.031 0.036 irrigation water delivery per unit irrigation water supply Total seasonal maintenance LE/feddan 13.47 13.47 13.47 10.00 10.00 10.00 expenditure for irrigation water delivery per unit command area Total annual maintenance - 0.73 0.73 0.73 0.50 0.50 0.50 expenditure fraction for irrigation water delivery Total cost per person employed LE/person 3,902 3,902 3,902 3,750 3,750 3,750 on water delivery Irrigation command area per unit Feddan/ 393 393 393 375 375 375 staff person Head:Tail mesqa pumping hours - 1.06 1.03 N/a 0.66 0.88 N/a ratio Drainage Groundwater level (depth to) m 0.80 0.58 0.82 0.75 0.58 0.95 Groundwater salinity Mmhos/cm 2.2 3.0 2.1 2.0 3.2 6.1 Soil salinity Mmhos/cm 0.8 0.7 1.1 2.8 3.9 3.5 Farmer questionnaire Irrigation problems: - Very severe Counts - - - - - - - Severe Counts - - - - 1 3 - Mild Counts 2 7 0 0 18 2 GN18 Benchmarking 10 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM Drainage problems: - Severe Counts - - - - - 1 - Mild Counts - - - 5 - 3 - Little Counts - 1 - 14 1 2 Legend Best value Critical Area of value concern Source: World Bank, 2005a; 2005b Note : LE ­ Egyptian pounds 1 US$ = 5.78 LE From the pilot study it was concluded that: · The process was valuable in identifying the performance in key management units (the Branch Canal). Comparing the performance of similar management units enabled best practice and suitable performance targets to be identified, identified gaps in performance and provided (some) answers to the root causes of these performance gaps; · Diagnostic analysis is a fundamental part of benchmarking. Analysis of the initial set of performance indicators led on to further data collection and interviews with water users to identify the root causes of poor levels of performance; · The value of comparative performance assessment and establishing benchmarks for selected performance indicators cannot be over-emphasised; it provides real targets against which less well performing systems can be judged; · Involvement of the water users in the process through discussions and questionnaires is an essential part of the benchmarking process; · Due to the varying levels of performance across a range of indicators it is not always possible to identify one "best practice" system. In some cases the irrigation water delivery performance was good, but the drainage performance was poor, and vice versa. Nevertheless, individual, achievable targets are obtained to use as benchmarks; · If benchmarking is to be adopted on a wider scale as a management tool there should be greater involvement with the system managers, the District Irrigation and Drainage Engineers in the process, and the water users. These key stakeholders must be engaged in the process at the outset, and the analysis and findings shared with them at all stages; · In future developments a GIS would be a useful tool used to process, analyse and present the data. Albania A participatory benchmarking process was used in 2001-2002 on the World Bank funded Second Irrigation and Drainage Rehabilitation Project in Albania to monitor the performance of 9 Federations of Water Users Associations formed under the project. Regular 2-weekly meetings were held with the Presidents of the Federations (Figure 6). During the meeting data were collected from each Federation President and a number of performance indicators calculated (Table 2). These data were displayed on the meeting room wall for all to see. Figure 6: Meeting of Federation Presidents GN18 Benchmarking 11 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM During the final meeting of the season the latest figures were collected and used to determine two key performance indicators ­ percentage of command area irrigated, and irrigation service fees collected per unit area (Table 3). This simplified approach gave a very clear indication of relative performance, clearly identifying Federations where the project needed to provide further support to strengthen the Federations. In one year (2001) the winning Federation was presented with a motorbike in recognition of their performance. Table 2: Data collected and key performance indicators calculated at each 2 weekly Federation meeting Date:29 June 2001 Federation Data Units Peqin Kavaje Erseni Cukas Krutje Lushnje Myzeqeja Albulena Gjanci aumPanxhi Lumi Ndroq- STP Bitaj/ Lumi Rezervaar Lumi Water resource Shkumbini Callik Thana Thana Thana Stp Mujalli May Gjancil Shkumbini Totalcommand area ha 10000 4000 9189 10000 8000 5500 12000 6000 1509 Lengthofmaincanal km 42 15 29 32 30 16 22 50 29.7 Number ofWUAs No. 12 4 12 10 8 5 16 18 4 Area irrigated last year (2000) ha 3100 1370 300 600 960 0 2000 360 870 Fee collected last year (2000) 000 Lek n/a 492 n/a 366 60 0 2.5 1000 1560 Planned irrigationarea 2001 ha 3000 1170 1400 2000 1600 650 4500 2000 950 Cumulative area irrigated to date ha 900 240 460 700 480 220 1440 660 317 Budget for this year (2001) 000 Lek 2900 890 1000 1250 550 980 1010 7400 2400 Fee collected to date 000 Lek 1511 422 554 1173 427 33 473 3793 613 Analysis Percentage area irrigated - 2000 % 31 34 3 6 12 0 17 6 58 Fee collected per unit area ­ 2000 Lek/ha n/a 123 n/a 37 8 0 0 167 1034 Planned irrigationarea ­ 2001 % 30 29 15 20 20 12 38 33 63 Budget per unit area ­ 2001 Lek/ha 290 223 109 125 1 178 1 1233 1590 Percentage oftotalcommand area irrigated to date % 9 6 5 7 6 4 12 11 21 Percentage ofbudget collected to date % 52 47 55 94 78 3 47 51 26 Source: Halcrow, 2001 Note: 1 US$ = 144 Lek (2001) Table 3: Simplified table for benchmarking performance of WUA Federations Federation Total Area irrigated IrrigationService Irrigation Service Rank command during the Fees collected Fee collected per area (ha) season (Lek) unit area (Lek/ha) Albulena 5313 27% 948,000 178 7 Erzeni 4450 24% 847,000 190 5 Peqin-Kavaje 7872 54% 3,028,000 384 2 NaumPanxhi 2128 61% 1,228,000 577 1 Cukas 6022 34% 1,110,000 184 6 Krutje 6577 54% 2,353,000 357 3 Lushnje 3588 55% 855,000 238 4 Myzeqeja 1980 0% 66,000 34 9 Gjanci 55377 44% 757,000 137 8 Source: Halcrow, 2001 Note: 1 US$ = 144 Lek (2001) China In China the World Bank has initiated a benchmarking programme based on the Rapid Appraisal Process (RAP) developed by Burt and Styles (1999) for investigating options for modernisation of irrigation and drainage systems. This process entails visiting an irrigation scheme for a period of about 2 weeks, collecting an extensive set of data and then analysing the scheme performance (Burt and Styles, 2004). The data collection is based on a set of spreadsheets which contain GN18 Benchmarking 12 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM several hundred questions that evaluators must answer in a standardised format. The questions cover water supply, personnel management, types of canal structures, level of water delivery service and a variety of water management related topics. From the data entered into the spreadsheets a large set of external and internal indicators are computed. The external indicators computed include: The total annual volume of irrigation water available at the user level Total annual volume of irrigation water supply into the command area (from all sources) Total annual volume of water managed by the service provider Total annual volume of water supply Total annual volume of irrigation water delivery to users by the service provider Total annual volume of pumped groundwater Total annual volume of field evapotranspiration requirements Peak irrigation water requirements Annual relative irrigation supply Annual relative water supply Command area irrigation efficiency Water delivery capacity Various financial indicators, agricultural productivity, economic and environmental impact indicators are also determined from the process. There are an extensive number of internal indicators which are used to: Identify the key factors related to water control throughout a project; Define the level of water delivery service provided to users; Examine specific hardware and management techniques and processes used in the control and distribution of water. The external indicators inform on the outputs of the system (productivity and efficiency of use of inputs), the internal indicators inform on how the system is actually operated and the water delivery service provided at all levels. Analysis of the results and consideration of the possible options for modernisation requires an experienced irrigation engineer who fully understands the options available and the likely consequences of each option on the irrigation system under consideration. He/she must be able to evaluate whether it is possible to conserve and make better use of the available water and enhance the environment through improved water management. To this end training of personnel is a key part of the RAP approach, providing rapid, systematic education in modernisation of irrigation projects. Sri Lanka In Sri Lanka, Jayatillake (2004) reports work by the Irrigation Management Division of the Irrigation Department to benchmark scheme performance across the country. Cropping intensity across 52 schemes was a starting point for identification of relative performance, and a range of between 0.35 and 2.0 was found. In 5 schemes in one water resources system, benchmarking was used to reduce the period of water issues made from reservoirs during the Maha and Yala seasons in order to conserve limited water supplies (Figure 7). Further international comparative assessments were then made with other rice-based schemes using the On-line Irrigation Benchmarking Service (Malano et al, 2004) benchmarking tool (Figure 8). GN18 Benchmarking 13 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM Maharashtra State, India In Maharashtra State in India, the Union Ministry of Water Resources, the Indian Government, the state Irrigation Department and the Indian National Committee on Irrigation and Drainage initiated a benchmarking programme in 2002. The initiative was part of a process promoted by the International Commission on Irrigation and Drainage based on the procedures outlined in the IPTRID/World Bank guidelines for benchmarking performance in the irrigation and drainage sector (Malano and Burton, 2001). GN18 Benchmarking 14 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM The state of Maharashtra had, for over 25 years, been collecting and analysing data for five main parameters: i) Irrigation potential created and utilized; ii) Seasonal and total annual irrigated area; iii) Water use efficiency; iv) Recovery of irrigation charges; v) Crop yields. In addition a socio-economic survey was carried out once every 5 years. The requirements for collecting and analysing these data had been set out in the Public Works Department Handbook, the Irrigation Act and government resolutions. The information was presented, in the main, for the state as a whole rather than for individual irrigation and drainage schemes. The 2002 benchmarking initiative started with eight irrigation schemes covering a total of 0.46 million ha. Out of the 20 indicators proposed in the IPTRID/World Bank benchmarking guidelines (Malano and Burton, 2001), 15 were selected. By 2003 the number of schemes had risen to 84, including 30 major schemes (command area > 10,000 ha), 26 medium schemes (command area 2,000-10,000 ha) and 28 minor schemes (command area <2,000 ha), and the number of benchmarking indicators reduced to 10. The ten indicators used are listed in Table 4. Table 4: Indicators used in Maharashtra State, India Category Indicator System performance · Annual irrigation water supply per unit irrigated area Agricultural productivity · Output per unit irrigated area · Output per unit irrigation supply Financial aspects · Cost recovery ratio · Total O&M cost per unit area · Revenue per unit volume of water supplied · Maintenance cost to revenue ratio · Mandays for O&M per unit area · Total O&M cost per unit volume of water supplied Environmental aspects · Land damage index Source: GOM, 2004 The annual benchmarking report (GOM, 2004) provides details of the values of the indicators for each of the schemes (Figure 9), and shows a wide range of variation in the performance of each scheme. The presentation of the data in this way "has resulted in healthy competition among field officers" (Sodal, 2004), achieving one of the key objectives of benchmarking - the knowledge for scheme managers of where they are relative to other systems and the desire to improve the performance of the schemes for which they are responsible. GN18 Benchmarking 15 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM Source: GOM, 2004 GN18 Benchmarking 16 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM Summary The examples provided show a range of benchmarking programmes, each with the overall aim of improving system performance, but with quite different approaches. In the Maharashtra and Sri Lankan case the programmes were initiated and implemented by government agencies, in China and Egypt the programmes were initiated by the World Bank working in collaboration with government agencies, whilst in Albania the benchmarking programme was part of an ongoing project. The Australian case is useful in that it is a service carried out by an independent body (ANCID) and provides information back to managers of irrigation and drainage systems to enable them to make decisions on performance improvement. This is a key point in the process; where senior management of an irrigation department or agency are using benchmarking as a tool to quantify the performance of individual schemes and identify measures for improvement of less well performing schemes, it is important that they inform, involve and empower scheme managers and water users. It will be the scheme managers and water users who understand their systems that will bring about the changes required to improve performance; they should be part of the process from the beginning. References and Further Reading Alexander, P.J., and M.O. Potter. 2004. Benchmarking of Australian irrigation water provider businesses. In Special Issue: Benchmarking in the Irrigation and Drainage Sector. Irrigation and Drainage 53 (2). New York: Wiley, June 2004. ANCID. 2000. 1998/99. Australian Irrigation Water Provider ­ Benchmarking Report. Victoria, Australia. 68 pages. Bos, M.G, M.A. Burton, and D.Molden. 2005. Irrigation and Drainage Performance Assessment: Practical Guidelines. CABI Publishing, Wallingford, UK. Burt, C.M. and S.W. Styles. 1999. Modern water control and management practices in Irrigation: Impact on performance. Water Report No. 19, Food and Agriculture Organisation of the UN, Rome. Burt, C.M. and S.W.Styles. 2004. Conceptualizing irrigation project modernization through benchmarking and the rapid appraisal process. In Special Issue: Benchmarking in the Irrigation and Drainage Sector. Irrigation and Drainage 53 (2). New York: Wiley, June 2004. Burton, M., H. Malano, and I. Makin. 2005. Benchmarking for improved performance in irrigation and drainage. In Shaping the Future of Water for Agriculture: A Sourcebook for Investment in Agricultural Water Management. Washington D.C.: World Bank. GOM, 2003. Benchmarking of irrigation projects in Maharashtra, 2001-2002. Irrigation Department, Government of Maharashtra, India. Halcrow, 2001. Report on the 2001 Improved Levels of Service Programme, Albania: Second Irrigation and Drainage Rehabilitation Project, Project Management Unit of the Ministry of Agriculture and Food, Tirana. Jayatillake, H.M. 2004. Application of performance assessment and benchmarking tool to help improve irrigation system performance in Sri Lanka. In Special Issue: Benchmarking in the Irrigation and Drainage Sector. Irrigation and Drainage 53 (2). New York: Wiley, June 2004. GN18 Benchmarking 17 Toolkit for monitoring and evaluation of AWM Guidance Note 18 projects Benchmarking in AWM Malano, H., and M. Burton. 2001. Guidelines for benchmarking performance in the irrigation and drainage sector. International Programme for Technology and Research in Irrigation and Drainage Secretariat, Food and Agriculture Organisation of the UN, Rome. Malano, H., M. Burton, and I. Makin. 2004. Benchmarking performance in the irrigation and drainage sector: A tool for change. In Special Issue: Benchmarking in the Irrigation and Drainage Sector. Irrigation and Drainage 53 (2). New York: Wiley, June 2004. Sodal, S.V. 2004. Present practices and opportunities for performance improvement of irrigation schemes in Maharashtra State of India (POPIIS). In Special Issue: Benchmarking in the Irrigation and Drainage Sector. Irrigation and Drainage 53 (2). New York: Wiley, June 2004. World Bank. 2005b. Benchmarking Initiative in the Irrigation and Drainage Sector, Egypt: Final report. Washington D.C.: World Bank, December 2005. World Bank. 2005d. Guidelines for Benchmarking Performance in the Irrigation and Drainage Sector in Egypt. Washington D.C.: World Bank, December 2005. Website On-Line Irrigation Benchmarking Service: http://www.lk.iwmi.org:82/oibs/LoadBench.htm GN18 Benchmarking 18 Toolkit for monitoring and evaluation of AWM Reference Note 1 projects ToR for M&E of project Reference Note 1 Sample terms of reference for M&E of agricultural water management projects Each project is unique in terms of objectives pursued, indicators monitored, desired scope of assessment, budget limitations, hierarchies of control, or timing of staff appointments. Terms of reference have to be tailored for each circumstance. Box 1 presents a list of recommended elements to include in the Terms of Reference (ToR). Box 1: Recommended elements to include when constructing the ToR Background: · Description of the project (goal, purpose, outcomes). · Contribution of the job contract to the project Purpose of the task being contracted · Main purpose, key audience, and expected outputs · Formal decisions that the task supports and planned use of outputs from the task Scope and method · Overall scope of work · Desired type of analysis, approach and methods, particularly what is expected in terms of participatory approaches Issues to be covered · Delimitation of themes in relation to the purpose of the task · Extent to which cross cutting issues (gender, poverty, empowerment) are to be dealt with Personnel requirements · Number of people to be involved in the task and the time allotted for each · Necessary professional qualifications and experience Schedule · Start date, timing of interim analysis, deadline Stakeholders to be involved · Who should be involved: authorities, institutions, groups, individualsm funding agency, cooperating institution, steering committee · How people/ groups will be involved Remuneration · Rates · Costs covered and not covered Documentation · Ownership of work, and extent to which documentation will be distributed Source: Guijt and Woodhill, 2002. A single ToR can be prepared for hiring a team or agency for the whole M&E system, or separate ToRs can be issued for different positions, such Consultant to set up M&E system, M&E coordinator, M&E staff, M&E responsibilities for the Task Team Leader, and M&E responsibilities for the implementing partner. The following section gives an example of ToR for M&E in the case of the Maharashtra Water Sector Improvement Project, India1. GR2 provides a specific sample ToR for conducting a baseline survey. 1Adapted from World Bank. 2005e. Maharashtra Water Sector Improvement Project, India. Draft Terms of Reference for Monitoring and Evaluation of project. Draft, December 27, 2005. 1 Toolkit for monitoring and evaluation of AWM Reference Note 1 projects ToR for M&E of project 1. Background The Government of is implementing the Project. The project, with a total cost amounting to . became effective on . and will have a duration of years. It is scheduled to close on . 2. Project objectives The primary objectives of the proposed project are: (i) to strengthen the state's capacity for multi sectoral planning, development, and sustainable management of the water resources, and (ii) to improve irrigation service delivery on a sustainable basis to increase productivity of irrigated agriculture and contribute to rural poverty reduction. 3. Project components The project proposes to (i) undertake institutional restructuring and capacity building of the water management agencies; (ii) improve irrigation water service delivery and management, (iii) improve the knowledge base of the water sector; and (iv) support project management and monitoring activities. Details of each proposed activity are provided below. 4. Project Monitoring and Evaluation (a) Objectives M&E is of critical importance for tracking the progress in the implementation of various project components, identifying problems as they arise and guiding remedial actions to help ensure that the project achieves its intended objectives. (b) M&E activities The project M&E will focus on three priority areas: M&E the progress in the provision of critical project inputs, including financial and physical inputs (for example rehabilitation works) and implementation of institutional reforms (such as formation and operationalization of the WUAs), and M&E progress in achieving the project outputs and outcomes. The results framework for assessing progress in project outputs and outcomes is presented in the Annex 2a. The year wise implementation target values as per the Project Agreement are listed in Annex 2b. The definition of key project performance indicators are presented in Annex 3. The key performance indicators selected for measuring and monitoring progress towards the project development objectives will include institutional indicators, improved service delivery indicators, improved crop productivity indicators, poverty impact indicators, sustainability indicators, and innovative pilot indicators. 5. Main tasks of the agency / consultant (a) Monitoring To monitor and evaluate progress in the provision of project inputs, the agency/ consultant will: Review progress in the implementation of the components and subcomponents of the project every six months over the project period by keeping track of selected key performance indicators and project outputs. The performance indicators are listed in Annex 2. Annex 4 lists Project outputs that will also be monitored. The proposed data collection sheets for assembling the information required for these performance indicators and component outputs are presented in Annex 5 and Annex 6 and will cover all schemes under the project. There data collection sheets will be completed by the relevant State authorities/ Project Management Unit staff for each scheme and other departments as relevant. At the outset of the M&E consultancy, the consultants will review the formats of these data sheets and in consultation with the Project Management Unit will suggest and make necessary 2 Toolkit for monitoring and evaluation of AWM Reference Note 1 projects ToR for M&E of project changes to these forms to ensure that they adequately capture the required data. Once these forms have been agreed to, they shall be used across schemes and components of the project as relevant. At the time of each 6 monthly review, the scheme level data on the performance indicators as well as the data from the monitoring tables will be provided to the consultants by the Project Management Unit, in electronic form. The consultants will be responsible for analyzing these data across schemes and components and reporting progress towards achieving the project results and outputs (Annex 4). Prepare a report every six months ( reports in total), which will: describe the progress in project implementation relative to the preidentified project targets, which are listed in Annex 2a, identify the factors which delay or undermine effective implementation of the components and subcomponents, propose remedial actions where applicable, and highlight success stories. For each 6 monthly monitoring phase, the agency/ consultant will have to visit one major, one medium and 3 minor irrigation schemes which are randomly selected, for the purpose of undertaking field level verification of implementation progress. In selecting the medium and minor irrigation schemes for random visits, the agency/ consultant should exclude schemes that have already been visited. For the major irrigation schemes, repeat visits will be undertaken only after all schemes have been visited. During the 6 monthly visits, the consultants will undertake an independent assessment of the scheme level performance indicators (Annex 6) to verify the accuracy of the data collection and make necessary suggestions for improvements/ adjustments as needed. During there monitoring visits, the consultants shall also visit randomly selected WUAs in the major scheme, randomly selected WUAs in the medium scheme and ... WUAs in each of the minor schemes. During the visits to the WUAs, the consultants will carry out an independent assessment of the functioning of the WUA and gather beneficiary feedback on the overall functioning of the WUA, the benefits received by the members of the WUA, and areas which require greater attention in order for the project to achieve its development objectives. Discussion with the WUA should cover issues relating to the adequacy (quantity and timing) of water delivery, assessment and collection of fees, changes in cropping patterns and other relevant subjects. The findings of the field assessment shall then be discussed with the Water Resources Department / Agriculture Department and other Departments as relevant. A summary of the findings and discussions shall be included in the 6 monthly monitoring reports. (b) Evaluation In addition to the regular 6 monthly monitoring, there are 3 junctures during the project period at which project impact assessment exercises will be undertaken by the consultants to evaluate the project's performance and progress towards achieving the project's development objectives. The first evaluation exercise will be undertaken prior to the First Mid Term Review (around ..), the second exercise will be undertaken prior to the Second Mid Term Review (around .), and the final impact evaluation exercise will be undertaken prior to project closure (around .). As part of the project evaluations the consultants shall: Evaluate the projects' performance on the basis of selected key output and outcome indicators given in Annex 2 and 4. Prepare three impact assessment evaluation reports which will (i) describe the status in achieving the project's expected outputs and outcomes, (ii) identify the factors which prevent the achievement of these outputs and outcomes, and (iii) identify corrective actions and summarize lessons learnt. These evaluation reports are expected to serve as background reports for the First and Second Mid Term Reviews of the project and project implementation completion report. The dates for the preparation and delivery of these reports will therefore be finalized in consultation with the Project Management Unit. 3 Toolkit for monitoring and evaluation of AWM Reference Note 1 projects ToR for M&E of project As part of each of the evaluation studies, the consultants will be expected to carry out household surveys to analyze the impact of the project. The impact assessment survey shall consist of five components (I) a household survey, (ii) a village level survey, (iii) a scheme level survey, and (iv) a Water Resources Department level questionnaires, and (v) river basin level questionnaire. The sample for the impact assessment surveys for the Mid Term Reviews shall include: 1 major, 1 medium, and . Minor schemes which have already been covered under the baseline survey. The Project Management Unit shall provide the consultant with a list of the schemes that were included in the baseline study and where work on the project has progressed sufficiently to enable monitoring of project outputs and impacts. In the major scheme, as per the baseline, the same villages will be selected for the survey. In each of these villages, the households surveyed during the baseline survey will be revisited. In the rare occasion that one of these households cannot be traced, then a replacement household will be randomly selected. However, the consultants shall ensure that only a minimum number of replacement households are needed. In the medium scheme, as per the baseline, the same villages will be selected for the impact assessment survey. In each of these villages, the households surveyed during the baseline survey will be revisited. In the rare occasion that one of these households cannot be traced, then a replacement household will be randomly selected. However, the consultants shall ensure that only a minimum number of replacement households are needed. In the .. minor schemes, as per the baseline, the same villages will be selected for the impact assessment survey. In each of these villages, the households surveyed during the baseline survey will be revisited. In the rare occasion that one of these households cannot be traced, then a replacement household will be randomly selected. However, the consultants shall ensure that only a minimum number of replacement households are needed. In total it is envisaged that the sample for the First and second Mid Term Reviews impact evaluations will include . households, spread across villages and schemes. The sample for the Final Impact Assessment Survey shall include: major, medium, and minor schemes The sample will include the major, medium and minor schemes covered by the project that were included as part of the baseline survey. The sample shall also include the control group from the baseline survey that consisted of major schemes, medium schemes, and . minor schemes randomly selected from all non project schemes in the state. The sample for the remaining . major, . medium and minor schemes included in the project shall be randomly selected using the same methodology employed to select the sample for the baseline study (see Annex 8). The Project Management Unit shall provide the consultants with the list of the schemes included in the project for the purpose of sampling. For the project schemes and control group included in the baseline survey, the consultants will visit the same villages and households selected in each scheme. For the additional schemes included in the final impact assessment survey, villages and households will be selected as per the methodology of the baseline. In total it is envisaged that the sample for the final impact evaluation will include households, spread across villages and schemes. The final impact evaluation should also assess the impact of the agricultural support services component as well as activities under the . pilots. Annex 7 provides a list of the schemes covered under the baseline survey and formats of questionnaires to be used for baseline survey. Annex 8 provides the guidelines with respect to the survey implementation for the impact assessment surveys. The final impact assessment 4 Toolkit for monitoring and evaluation of AWM Reference Note 1 projects ToR for M&E of project survey must be completed within 6 months prior to completion of the Final Evaluation Report. Ideally, the Final Evaluation Report will be completed 3 months before project closing. The content of the First and Second Mid Term Reviews and Final Impact Assessment Surveys will include all questions from the baseline survey as a "core" module. In addition to this "core" module, the consultants should add additional questions and content as needed to ensure that the impact assessment surveys adequately capture all results indicators being monitored as part of the project. The content of the impact evaluation questionnaires will be finalized in consultation with the Project Management Unit. The Agency/ Consultant will be expected to pool the baseline survey data (to be provided by the Project Management Unit) and Final Impact Assessment survey data to create a panel data set to analyze and measure changes with respect to the performance of the project. Indicative list of indicators/ subjects to be included in the impact assessments: o Engineering parameters, o Environmental, o Maintenance, performance, and cost recovery o Agricultural production and household socio economic indicators o Village level indicators o Institutional The Agency/ Consultant will be expected to produce a report which will describe the results of the repeat survey, particularly the changes between the baseline and repeat survey results. 6. Outputs of the consultancy (a) Project monitoring outputs (i) Inception report The Agency/ Consultant shall submit an inception report within 30 days from the date of start of work. In the inception report, the consultant should review the contents of monitoring reports and the data sheets for calculating the performance indicators to ensure nothing is left out. Inception report will also address the contents of the first six monthly monitoring report. The inception report should also include details of final sample selection and the methodology proposed to select the sample and the revised version of the data sheets that will be provided in both languages. (ii) Monitoring report The Agency/ Consultant will produce 12 reports, which are to be submitted to the Project Management Unit in the months of . and .. of each year, until the closing of the project. The proposed schedule for report delivery is presented below. The First monitoring report will be due on the month of , year .. Report Number Delivery Date First Monitoring Report In month of ., year Second Monitoring Report In month of ., year . Twelfth Monitoring Report In month of ., year (b) Project evaluation outputs The Agency/ Consultant will produce and submit to the Project Management Unit: Report presenting the findings of the Mid Term Reviews (First and Second), and Final Impact Assessment survey as per the schedule below: First evaluation report on the First Mid Term Review (around the month of , year .) Second evaluation report on the Second Mid Term Review (around the month of , year .) Final Evaluation Report in the month of , year .. 5 Toolkit for monitoring and evaluation of AWM Reference Note 1 projects ToR for M&E of project 7. Data services and facilities to be provided by the Government of . The Government of . shall provide all the relevant data and any other literature available with Government to the consultant. However, the consultant will be responsible for collecting the required field data with respect to engineering, economic, and social aspects in selected schemes mentioned above (section 5.) The Project Management Unit shall provide the following documents to the consultant free of cost Project Appraisal Document Project Implementation Plan Baseline survey questionnaires, data and reports Copies of guidelines/ direction issued by Government of . from time to time. Reports related to commissioned consultancy studies, as and when available Other information relating to agriculture, irrigation, groundwater, etc, available with the Government of . Any other general relevant literature available with the Government The consultant shall have to make his own arrangement for office/ residential accommodation, transportation, communication, office equipment, office and field staff, etc. 8. Format of Monitoring and Evaluation Reports Monitoring and Evaluation provide useful information to stakeholders that lead to project improvements, funding decisions, accountability, and learning. Therefore, it is essential that the findings of monitoring and evaluation are communicated clearly, accurately and appropriately. (a) Monitoring Report The following headings and sub headings are suggested for the Monitoring Report. However, the contents of the monitoring reports will be agreed upon with Government of . after the submission of M&E consultancy inception report. Data use policy: the data will be owned by Government of I. Executive summary II. Introduction and Context III. Implementation Progress in the terms of achieving Components and Subcomponents For each component and subcomponent, the discussion will (i) describe the progress in project implementation relative the pre identified project targets, which are listed in Annex 2a, (ii) identify the factors which delay or undermine effective implementation of the component/ subcomponents, and (iii) propose remedial actions where applicable. IV. Summary and Conclusions Annexes (b) Evaluation Report Within 15 days of the submission of the Second Monitoring Report, the consultant will submit the format and the contents to be covered in the First Mid Term Review impact evaluation report. The contents of this report will be reviewed and agreed with the Project Management Unit, for use of the preparation of the first evaluation report after the submission of the Second Monitoring Report. Within 15 days of the submission of Number . monitoring Report, the consultant will submit the format and contents to be covered in the Second Mid Term Review impact evaluation report. The contents of this report will be reviewed and agreed with the Project Management Unit, for use of the preparation of the second evaluation report after the submission of the Number .. Monitoring Report. Within one month of the submission of the Number . Monitoring Report, the consultant will submit the format and contents to be covered in the final evaluation report. The 6 Toolkit for monitoring and evaluation of AWM Reference Note 1 projects ToR for M&E of project contents of this report will be reviewed and agreed with the Project Management Unit, after the Number . Monitoring Report. For each component and subcomponent, the impact assessment reports will (i) describe the status in achieving the component's/ subcomponent's expected outputs and outcomes, (ii) identify the factors which prevent the achievement of these outputs and outcomes, and (iii) identify corrective actions, success stories and summarize lessons learnt. 9. Review Committee A review Committee comprising the following members (Chairman, Secretary, Members) will review and approve the reports of the consultants: , Water Resources Department, Government of... , Agriculture Department 10. Procedure for Review of Outputs of the Consultant After submission of the half yearly monitoring report by the consultant, a Review Committee meeting will be held to review the substance and quality of the report. The consultant shall make a presentation, including feedback, at this meeting. The committee shall review the progress with reference to the agreed ToR and advice necessary corrections and modifications, if any, within 3 weeks from the date of submission of the monitoring report. The consultant will incorporate these suggestions of review committee in the subsequent report. The Review Committee shall review the evaluation reports submitted by the consultant within one month. As the Review Committee meeting, the consultant will make a presentation of the report. The consultant will modify the reports based on the comments. 11. Key professional position whose Curriculum Vitae and experience would be evaluated Curriculum Vitae and experience of the following expert associated with aspiring consultant/ agency would be evaluated for the purpose of awarding of consultancy. Team Leader, having extensive experience in project coordination and management, and good report writing skills. Any previous experience in project monitoring and evaluation will be a plus. M&E Specialist/ Statistician Agricultural Economist preferably with experience in econometrics and project economic analysis Irrigation Specialist Groundwater Specialist (Hydrogeologist) Community Management Specialist, with experience in community mobilization (experience with water users organizations a plus) Sociologist Agriculture Specialist Environmental Specialist 12. Submission of reports and final data sets The consultant shall provide fifteen copies of each of the reports. The Consultants shall provide thirty copies of the final report. The final data sets from the impact evaluation surveys should be provided in ASCII format as well as in a common statistical package such as SPSS or Stata. After conclusion of the study, documents and other data procured and analyzed by the consultant for this study shall be handed over to the Superintending Engineer in the Project Management Unit of the project. Upon completion of the study and submission of final report, the consultant 7 Toolkit for monitoring and evaluation of AWM Reference Note 1 projects ToR for M&E of project should also hand over Hardware and Software, including all equipments solely procured for this study, to the Superintending Engineer. 13. Budget The consultant shall provide detailed estimates of budget including staff charges, travel equipments, hardware, software, ground surveys, technical meeting, report preparation and other related activities in the attached formats (Annex 9). The cost estimates shall be based on the assessment of the resources needed to carry out the assignment: staff time, logistical support, and physical input (for example vehicles, laboratory equipment). Cost shall be divided intro two broad categories: a) fee or remuneration, and b) reimbursable (for example travel expenses, hotel charges, etc), and further divided into foreign and local costs. The cost of staff time shall be estimated on a realistic basis for foreign and national personnel. 14. Payment schedule The payment schedule will be as under: Payment schedule as percentage of contract Amount Submission of inception report 2.5% First Monitoring Report (In .) 5 % .. Twelfth and final Monitoring Report (In ) 5% Final Evaluation Report (In ) 17.5% ANNEX Annex 1: List of irrigation schemes under the project Annex 2: (a) Results framework, as presented in the Project Appraisal Document (b) Monitoring Framework, as presented in the Project Appraisal Document Annex 3: Definition of indicators for evaluating performance Annex 4: Monitoring Indicators and Progress Tracking: targets at Mid Term Reviews and Project end. Annex 5: Data recording sheets for reporting on (a) Physical implementation progress (b) Contracts and financial progress (c) Consultancy activities (d) Water Resources Regulatory Authority establishment (e) Water Resources Regulatory Authority operationalization (f) Water Resources Regulatory Authority activities and capacity building (g) Integrated computerized information system (h) Water Resources Department strengthening (i) WUA establishment (j) WUA operations (k) WUA physical works implementation progress (l) WUA capacity building (m) Water charge collection efficiency (n) Agriculture support service component (o) Dam safety works (p) Irrigation Department works (q) Pilots Annex 6: Data recording forms for calculating performance indicators Annex 7: (a) List of irrigation schemes covered under the baseline survey 8 Toolkit for monitoring and evaluation of AWM Reference Note 1 projects ToR for M&E of project (b) List of baseline survey questionnaires Annex 8: Guidelines for implementing the Impact Assessment Surveys (sample, refinement of the questionnaires, survey implementation, cleaning and analyzing data, staffing). Annex 9: Budget estimate for consultancy for M&E 9 Toolkit for monitoring and evaluation of AWM Reference Note 2 projects ToR for baseline survey Reference Note 2 Terms of reference for a baseline survey for an agricultural water management project Adapted from World Bank. 2004b. Maharashtra Water Sector Improvement Project, India. Draft Terms of Reference for Base Line Survey. Draft, July 2, 2004. I. Introduction 1.1 Background The water sector in is faced with critical challenges. Recognizing the urgent need to begin addressing these challenges, the Government of . has taken a number of actions, including: (i) raising canal water charges to meet full O&M cost; (ii) finalizing the State Water Policy; (iii) adopting a policy of mandatory water audit; (iv) finalizing the "Farmers Management of Irrigation Systems Act" to provide for the transfer of management of irrigation system to water user associations (WUAs); (v) finalizing the draft for the establishment of the Water Resources Regulatory Authority, and; (vi) initiated restructuring of irrigation sector institutions. The effective implementation of these reforms is expected to greatly improve the irrigation water sector performance in the state and generate benefits for society and he economy as a whole, particularly for many poor agricultural farmers. 1.2 Project objectives The Government of . , through the Irrigation Department, is preparing the Project. Through this project, the Government of . will implement the above mentioned reforms as a means of improving the performance of the water sector and enhancing the livelihoods of agricultural households, the majority of whom are poor. The primary objectives of the proposed project are: (i) to strengthen the state's capacity for multi sectoral planning, development, and sustainable management of the water resources, and (ii) to improve irrigation service delivery on a sustainable basis to increase productivity of irrigated agriculture and contribute to rural poverty reduction. 1.3 Project components The project proposes to (i) undertake institutional restructuring and capacity building of the water management agencies; (ii) improve irrigation water service delivery and management, (iii) improve the knowledge base of the water sector; and (iv) support project management and monitoring activities. Details of each proposed activity are provided below. 1.4 Essential studies As part of project preparation for the project, a baseline survey will be undertaken during .. This survey will collect information that will aid in project planning as well as provide a basis for monitoring and evaluating the impact of the project. II. The present study 2.1 Base line Survey In order to conduct the baseline survey, the Government of .. will hire a consulting firm to: (i) develop the sample frame for the baseline survey; (ii) revise and finalize the draft baseline survey questionnaires; (iii) arrange for the questionnaires to be translated into relevant languages; (iv) field test the questionnaires; (v) incorporate revisions to the questionnaire after field testing the 10 Toolkit for monitoring and evaluation of AWM Reference Note 2 projects ToR for baseline survey questionnaires; (vi) hire and train the field supervisors and enumerators; (vii) plan the field work logistics; (viii) conduct a pilot survey and revise the questionnaire based on the findings of the pilot survey; (ix) prepare survey implementation and questionnaire documentation; (x) supervise survey implementation and ensure quality control; (xi) develop the data entry program, supervise the project database and arrange for data cleaning and entry; and (xii) analyze and report the findings of the survey and provide datasets and final documentation. 2.2 Survey design An appropriate sample frame for the project baseline survey will be developed by the selected firm in consultation with the Government of .. A multi stage sample will be drawn and the selected firm will recommend the optimal sample size. Proposed sample design (a) The total sample for the project areas should include 5 major schemes, 5 medium schemes and 15 minor schemes. These schemes will be selected from the main river basins in the state. One major scheme, one medium schemes and 3 minor schemes should be selected from the following river basins : . Two major schemes, two medium schemes, and 6 minor schemes should be selected from the basin. In the basin, 1 major scheme, one medium scheme and 3 minor schemes should be selected from moderate and high rainfall zones in the western part of the state and 1 major scheme, one medium scheme and 3 minor schemes should be randomly selected from the remaining agro climatic zones (i.e. excluding the moderate and high rainfall zone). When there are multiple major/medium/minor schemes in a basin, a random selection of the required number of schemes should be made. It is suggested that the schemes are selected with probability proportionate to their command area. (b) The control group should include 1 major schemes, 2 medium schemes, 6 minor schemes randomly selected from all non project schemes in the state. Farmers in the control group will not directly be exposed to project interventions. In selecting the control groups, care should be taken to ensure that the control groups have similar socio economic characteristics to the treatment group (sample from the project areas). (c) In each major scheme (both for the "treatment" sample as well as for the control) 18 villages should be randomly selected to reflect the conditions in different locations (reaches) within the scheme. The following methodology is suggested for selecting the 18 villages in each scheme. First, within each major scheme, a main canal should be randomly selected and divided into three reaches head, middle and tail. Within each of the reaches, the two distributaries should be randomly selected. Each of these distributaries should be further divided into three reaches, head, middle and tail. Within each of these reaches a minor will be randomly selected and one village will be randomly selected from all the villages in the selected minor. In each village, 15 farm households will be randomly selected. A listing of all household in the village will be required to allow random selection of households. (d) In each medium scheme (both for the treatment sample as well as for the control), 6 villages should be randomly selected to reflect the conditions in different locations reaches within the scheme. The following methodology is suggested for selecting the 6 villages in each scheme. First, within each medium scheme, a main canal should be randomly selected from all the villages .. (e) It is estimated that the sample will include approximately . Farm households spread over . villages. Survey instrument The baseline survey consists of four components: (i) a household survey; (ii) a village level survey; (iii) a scheme level survey; and (iv) an Irrigation Department survey. The household survey will collect information on household demographics, income, asset ownership, land use patterns, agricultural production, access to and use of irrigation, extension and marketing, participation in irrigation institutions, experience with irrigation services and payment of water charges from the selected farm households. The village questionnaire will be used to develop village profiles. The information collected in this module will include major cropping patterns in the 11 Toolkit for monitoring and evaluation of AWM Reference Note 2 projects ToR for baseline survey village, sources of irrigation and patterns of irrigation use, the prevailing social and economic infrastructure in the village, prevailing prices of selected commodities and inputs, and details on any existing water user associations. This questionnaire will be completed by interviewing various village leaders such as the village head, local government officials, the principal of a school etc. A village questionnaire will need to be completed in each selected village. The third module collects information on scheme level parameters including the volume of water delivered to users, irrigated area, water use efficiency, agricultural production, and assessment and collection of water charges. The fourth module collects information on the staffing and delivery of services of the Irrigation Department. This module will provide a baseline to gauge the Effectiveness and efficiency of service delivery of the Irrigation Department and help assess the performance against international benchmarks. The draft survey questionnaires will be provided to the selected firm by Government of ... Based on the proposed sample design, it is estimated that approximately 2600 household questionnaires, 171 village questionnaires, 29 34 scheme level questionnaires and 1irrigation department questionnaire will need to be completed. III. Specific tasks The selected consulting firm will lead the following activities in collaboration with Government of: (i) Developing the sample frame and sampling methodology a) Recommend the methodology for sampling, develop the sample frame, calculate the optimal sample size and select the sample. The final sample and details of the statistical methodology used to select the sample need to be cleared by Government of ... b) A listing of all households in the village will be required to allow random selection of households. During the listing exercise, for each household in the village, data should be collected on the area of land owned and whether or not the household uses surface irrigation from the selected scheme. c) The selected firm will construct the sample weights and provide documentation on the methodology used to construct the weights. (ii) Develop the survey questionnaires a) Refinement and adaptation of the data collection instruments questionnaires. In particular, the selected firm will be responsible for tailoring the draft questionnaires to the local situation in . And assuring that they are adequate to collect baseline data on the selected indicators (a list of indicative indicators is included in the annex). Revisions of the questionnaire should be made in consultation with Government of and the final version of the questionnaire should be cleared by Government of . b) The firm will be responsible for the layout of the questionnaires, making sure all skip patterns are clear and coding all questions. c) The firm will also be responsible for preparing all support documentation including coding guides, interviewer and supervisor manuals and the data entry manual. Hard copies and electronic versions of all versions of all documentation need to be provided to Government of .. (iii) Implementing the survey a) Based on the sample frame, the firm will create plans for implementing the sample design and train staff to implement the designed sample. b) Selection and training of field workers. This activity consists of all the work necessary to develop training materials and manuals for all persons involved in fieldwork and to train field workers so that they understand the content of the questions, the layout and coding strategy of the questionnaires. Training will be required for interviewers, supervisors of interviewers and data entry personnel. Training to the field staff should be provided for a minimum of one week. c) The firm will have primary responsibility for the field test and piloting of the questionnaire. The field test should consists of informally completing 4 5 households questionnaires, 1 12 Toolkit for monitoring and evaluation of AWM Reference Note 2 projects ToR for baseline survey scheme questionnaire and 2 village questionnaires to ensure the relevance of the questions and check the wording of the questions prior to the pilot survey. The pilot survey will include at least 30 households in 2 different villages in schemes not selected in the sample. After the field test and pilot survey the questionnaire will be revised for errors detected in the field test (in consultation with Government of . staff). Results of the pilot should be entered in the data entry program developed for the project to debug any errors in the program, data entry, and/or coding. (This will be subject to quality control by the Government of .). Once the questionnaire is revised after the pilot, the data entry software will also need to be revised to reflect the changes in the questionnaire. d) The firm will be responsible for all field operations, including logistical arrangements for data collection arid obtaining household consent. The firm will contact local officials and village leaders to explain the project and obtain community consent for the baseline survey. They will also obtain maps, lists, and other community records as required. e) Household surveys should be conducted during a single visit to the household. In exceptional cases (if respondents do not have time/are not at home/coding errors need to be verified) it may be necessary to return to the household on more than one occasion, but this should not be the norm. f) GPS coordinates need to be collected for each survey village. The cost of purchasing basic handheld GPS readers should be included in the proposal. After completion of this study, the GPS Reader will become a property of Government of .. g) The firm will prepare a brief fieldwork progress reports for Government of . at the end of each month. The report will include the number of surveys competed, problems encountered and how they were resolved (for example, the number of replacement households and why they were necessary and the number of questionnaires entered in the data entry software). (iv) Cleaning and analyzing the baseline data a) The firm will develop a data entry program using software that can check for ranges and consistency of the data and generate reports indicating missing data, data outside of the accepted ranges, and inconsistent answers. The firm will be responsible for all data entry, data cleaning, database management, as well as coding open ended questions, and verification of the data. An ASCII version of the data as well as a complete data set including variable names, descriptions and labels prepared in a commonly used software package such as STATA or SPSS will be provided to Government of . with the draft final report. b) The firm will prepare a report on the findings of the baseline study which will include a statistical abstract of the data collected. The contents of the final report will be agreed upon with Government of .. after the submission of the mid term report. c) Data use policy: the data will be owned by Government of ... IV. Staffing In order to undertake this consultancy, the selected firm will require a survey manager (team leader) who should have extensive experience in designing and conducting household level socio economic surveys and analyzing survey data. The consultant will also have key personnel with the following qualifications: a. Sampling Expert/Statistician b. Irrigation Engineer c. Agricultural Scientist d. Economist e. Sociologist. While evaluating the proposal the qualifications and experience of the personnel will be considered. 13 Toolkit for monitoring and evaluation of AWM Reference Note 2 projects ToR for baseline survey The survey staff should be constituted as follows: * Core survey staff: composed of the survey manager, the field manager, the data manager, and the data entry staff who will be responsible for overall field supervision, coordination, and monitoring of data collection and data entry and data management activities. * Field survey staff: the field operations will be conducted by teams composed of a supervisor and 4/6 interviewers. If the firm does not have expertise in any of the above mentioned specializations, they are free to hire the services of specialists with above mentioned skills. Inputs in terms of man months of each of these staff should be included in the technical and financial proposals. V. Duration and time schedule The Consultancy shall be completed within 9 months of signing the consulting contract. A mobilization period of 30 days shall be allowed after the date of signing the contract. The date of starting the work shall be the 30th day following the date of signing the contract. A detailed time schedule of various activities to be undertaken during the consultancy must be provided by the Consultant. The following is the target period fixed for submission of various reports: (i) Inception Report Within 30 days from the date of start of work. The inception report should include details of the final sample selection and the methodology proposed to select the sample and construct sampling weights and the revised versions of the questionnaires that have been field tested. The questionnaires should be provided in both languages. (ii) Progress report ­ At the end of each month from the date of start of work (iii) Mid term Report At the end of 5 months from the date of start of work. (iv) Draft Final Report At the end of 8 months from the date of start of work. (v) Final Report At the end of 9 months from the date of start of work. VI. Submission of reports and final data sets The consultant shall provide fifteen copies of each of the inception report, mid term report, and draft final report. The Consultants shall provide thirty copies of the final report. The final data sets should be provided in ASCII format as well as in a common statistical package such as SPSS or Stata. After conclusion of the study documents and other data procured and analyzed by the consultants for this study shall be handed over to the Superintending Engineer in the Project Management Preparation Unit of the project. Upon completion of the study and submission of final report, the Consultants shall also hand over, Hardware and Software including all equipments solely procured for this study, to the Superintending Engineer. VII. Support to the firm by the Government of . Based on the specific requests of the consulting firm, any published data required for the specified tasks that is available in the offices of the Irrigation Department will be supplied to the consultant. If required, the Irrigation Department will also facilitate the collection of published data required for specified tasks from other departments in government including the . Remote Sensing Application Centre, the . Research Institute, and the department of Agriculture. The consultants will make their own arrangements for office space, conveyance for staff, telephone and communication facilities and the client will not be responsible for these. The Project Management Preparation Unit will be the Nodal Department for this study. The Superintending Engineer will act as Chief Coordinator and be responsible for responding to queries from the selected consulting firm. 14 Toolkit for monitoring and evaluation of AWM Reference Note 2 projects ToR for baseline survey VIII. Constitution of review committee To ensure effective implementation of the baseline survey, a Review Committee been constituted by Irrigation Department. The Committee will provide necessary advice and mid course correction where necessary. The composition of the committee is as under (chairman, members, secretary): Irrigation Department Agriculture Department .. The consultant will interact with the Committee: (i) At the time of inception of study (ii) Before finalisation of mid term report. (iii) Before finalisation of draft Final report. (iv) During the preparation of Final report. (v) If needed, after review of the monthly progress report. The Chairman can co opt any other member if he finds it necessary. IX. Budget The consultant shall provide, detailed estimates of budget including staff charges, travel, equipments, hardware, software, ground surveys, technical meetings, report preparation and other related activities. ANNEX: Indicative list of indicators Draft questionnaires 15 Toolkit for monitoring and evaluation of AWM Reference Note 3 projects Model Survey Questionnaires Reference Note 3 Prototype Baseline survey questionnaire for agricultural water management projects Adapted from World Bank South Asia Rural Development Unit. 2004. Monitoring and Evaluation of Water and Irrigation Projects ­ Prototype Baseline survey in South Asia. Washington D.C.: World Bank. Available at http://wbln0023.worldbank.org/Internal/SAR/southasiasectors- int.nsf/41ByDocName/MEWaterandIrrigation. Introduction The prototype irrigation survey is designed to assess the climate for new irrigation projects and irrigation improvement projects in South Asia. It is intended to be a baseline assessment of performance indicators that determine likelihood of success of World Bank investment in irrigation. It is divided into four sections: - A household and plot level survey intended to be completed by individual farmers (see Table 1), - A scheme level survey for each scheme present in a project (see Table 2). - A community and water user association survey intended for leaders of communities of which the farmers are members, and - A survey for a state level irrigation department which oversees service delivery and performance for several communities (see Table 4) The community level survey lists questions on: -Respondent's characteristics -General community profile -Access to education and healthcare -Prices of agricultural inputs and outputs, of land -Water User Association (see Table 3) -Agricultural Extension -Employment -Irrigated crops -Access to credit While the survey contains a number of customary questions meant to evaluate social, economic and agricultural conditions of communities, it is unique in that it attempts to gather detailed plot level irrigation information, assess the role of institutions in the successful functioning of irrigation systems and gathers parameters intended to assess the environmental conditions present in irrigation districts. The survey questions give information on a list of indicators commonly useful in irrigation projects: irrigation and engineer parameters, agricultural production and household socio economic indicators, village level indicators, and institutional aspects2. As a prototype survey, it needs to be customized and adapted to each project's situation. The prototype has been applied to the Maharashtra Water Sector Improvement Project, India, as a pilot. 2See http://wbln0023.worldbank.org/Internal/SAR/southasiasectors int.nsf/bc43e62e0130676085256da2005d8441/7c4b96288404e62a85257004006e1e06/$FILE/Indicative%20List%20of%2 0Indicators Final.xls for the matrix linking commonly useful indicators to survey questions. 16 Toolkit for monitoring and evaluation of AWM Reference Note 3 projects Model Survey Questionnaires Table 1: Household Survey Question Subquestion Answer 1. For the canal irrigated portion Adequacy of the Water 1= Satisfied of your land, please rate the 2= Partially Satisfied following 3= Dissatisfied Regularity/Reliability of supply 1= Satisfied 2= Partially Satisfied 3= Dissatisfied 2. Did you receive your share of water 1= Yes 2= No 3. Does a Water User Association exist that oversees management of and delivery of 1= Yes 2= No irrigation water? 3= Don't Know 4. Was this WUA organized by the government/irrigation department? 1= Yes 2= No 5. Are you a member of this WUA? 1= Yes 2= No 6. Do you attend WUA meetings? 1= Yes 2= No 7. What role does the WUA To maintain canals 1= Yes 2= No play? (choose all that apply) To maintain drains 1= Yes 2= No To collect fees 1= Yes 2= No To sort out problems 1= Yes 2= No To decide on the timing of irrigation delivery 1= Yes 2= No 8. How would you rate the overall performance of the WUA? 1= Satisfied 2= Partially Satisfied 3= Dissatisfied 9. Do you pay for water? 1= Yes 2= No 10. Whom do you pay? 1= Irrigation Dept. 2= WUA 3=Other (specify) 11. Do you pay fees regularly, as per schedule? 1= Always 2= Often 3= Sometimes 4= Rarely 5= Never 12. Does failure to pay fees result in punishment? 1= Yes 2= No 13. If yes, what is the punishment? Use codes 14. Have you paid a bribe for service in the last two years? 1= Yes 2= No 15. Are you willing to pay/pay -Not paying now but willing to pay Use codes more for better irrigation -Not paying now and not willing to pay service? -Paying and willing to pay more -Paying and not willing to pay more 16. What improvements would Improved quality of service 1= Yes 2= No you expect after paying/paying Improved quantity of water 1= Yes 2= No more? Improved reliability of water 1= Yes 2= No Improved timeliness of water delivery 1= Yes 2= No Other (specify) 1= Yes 2= No 17. In the future will you change the crops you grow if water supply is more reliable? 1= Yes 2= No 18. What crops will you grow? Crop Codes 1= Yes 2= No 19. Do you feel that you can engage in other farming activites because your water 1= Yes 2= No supply is more reliable? 20. Please specify the farming Purchase farm equipment 1= Yes 2= No activities that you would engage Rent farm equipment 1= Yes 2= No in as a result of the increased Hire labor/more labor 1= Yes 2= No reliability of your water supply? Grow higher value crops 1= Yes 2= No (include all that apply) Improve on farm irrigation 1= Yes 2= No Other (specify) 1= Yes 2= No 21. In the past 5 years, have you adopted any water saving techniques? 1= Yes 2= No 22. Does your WUA provide services other than water delivery? 1= Yes 2= No 23. Indicate the services Extension 1= Yes 2= No provided by the WUA which you Storage 1= Yes 2= No have used in the last 12 months: Marketing assistance 1= Yes 2= No (include all that apply) if Input assistance 1= Yes 2= No 17 Toolkit for monitoring and evaluation of AWM Reference Note 3 projects Model Survey Questionnaires Question Subquestion Answer applicable Technical Advice 1= Yes 2= No Other (specify) 1= Yes 2= No For each Household Member In the past 12 months, what was your primary Use codes (using an Identifier Number) occupation? In the past 12 months, what was your secondary Use codes occupation? Do you reside in this household year round? 1= Yes 2= No What Ethnic/ Religious groups do you belong to? Use codes How many months per year do you reside in this Months household? Agricultural Income: For each During the past 12 months, was employed by 1= Yes 2= No Household Member (using an others as a wage worker Identifier Number) and at each Type of wage employment Use codes season Number of days employed during the past 12 Days months in wage employment Method of payment 1=Piece rate 2=Time rate Daily Wage rate Amount What the average value of daily payments Amount received What was the average daily value of all in kind Amount payments (meals etc) that you received in addition to cash. Other Income: For each Did participate in salaried work 1= Yes 2= No Household Member (using an Number of months worked during the past 12 Months Identifier Number) months in salaried work How much was paid for this work Amount Did operate a business 1= Yes 2= No How much spent on the following items per month Specify amount for in operating this business: Equipment/building, each item Raw materials, Wages/salaries, Electricity/Water, Transportation, Other (specify) What was the total income for this business during Amount the past 12 months Other Household Income Pensions, Government Payments and Benefits, For each item: Sources Family/relatives/friends, NGO/private -monthly or annually organizations, Other income (interest for bank -amount deposits, shares etc.), Equipment (tractors, plows, etc.) leased out, Animals leased out, Land leased out, Buildings leased out, Other sources (specify) Assets: Furniture, Clocks or Does any member of the household own any of the 1= Yes 2= No watches, Kerosene, Gas or following items Electric Cooker, Iron or heaters, If you sold one of these how much money could Amount Refrigerator or freezer, Fans, you get for it today Sewing machine, Radio, cassette players, Television sets, Bicycle, Motorcycle and scooters, Tractor hand tractor, Other vehicles, Phone, Other Housing Is this dwelling owned by a member of your household Do you have legal title to the dwelling or any document that shows ownership If you sold this dwelling today, how much money would you receive for it Amount Do you rent this dwelling for goods, services or cash How much does this household pay to rent this dwelling cash and cash value of in kind services What type of dwelling is this: detached house, Multi family house, Use codes Separate Apartment What is the major construction material of the external walls Use codes 18 Toolkit for monitoring and evaluation of AWM Reference Note 3 projects Model Survey Questionnaires Question Subquestion Answer What is the major material of the roof Use codes What is the major material of the floor Use codes How many rooms do the members of your household occupy, including Number bedrooms, living rooms and rooms used for household enterprises Does your household have an electricity connection 1= Yes 2= No In the rainy/wet season, what is the main source of water for drinking and Use codes cooking in your household In the dry season, is your main source of water for drinking the same as in 1= Yes 2= No the rainy season In the dry season, what is the main source of water for drinking and Use codes cooking in your household What is the type of toilet that is used in your household Use codes What disposal system is this toilet connected to Use codes Food In the last seven days, has the household consumed less preferred foods Never security Rarely (once) From time to time (2 or 3 times Often (5 or more times) In the last seven days, have you reduced the quantity of food served to Never men in this household Rarely From time to time Often In the last seven days, have you reduced your own consumption of food Never Rarely From time to time Often Have you reduced the quantity of food served to children in your Never household in the last seven days Rarely From time to time Often Have members of this household skipped meals in the last seven days Never because of a shortage of food Rarely From time to time Often Have members of this household skipped meals for a whole day because 1= Yes 2= No of a shortage of food Credit Have you or any members of your household wanted to apply for a loan 1= Yes 2= No for any purpose during the last five years Have you or any members of your household applied for a loan for any 1= Yes 2= No purpose during the last five years Why haven't you applied for a loan in the last 5 years Use codes List all loans for which you have applied in the last 2 years and your Use codes existing loans, with: When applied, Purpose, Source, Approved/ Rejected, If rejected, what were the reasons given, How much received, How much requested, Loan Term, Interest Rate, Amount of loan outstanding, difficulties experienced in repaying the loan Plots Please specify each plot of land in your household that was owned, List sharecropped, rented in or rented out during the past 12 months What is the area of the plot Local Unit, Ha Do you own, sharecrop, or lease this plot 1= Yes 2= No Was this plot irrigated during each season in the past 12 months For each season, 1= Yes 2= No What is the area of the plot that was irrigated during each seasons in the For each season, 1= past 12 months Yes 2= No Where is this plot located: near the main canal, branch, distributary, Use codes minor. Where is this plot located in the scheme: which canal is it connected to Use codes In general, what is the slope of this plot Use codes 19 Toolkit for monitoring and evaluation of AWM Reference Note 3 projects Model Survey Questionnaires Question Subquestion Answer What kind of improvements does this plot have Use codes From today, how many years has it been since this plot was fallow, or Years cleared from the bush In the last 5 years, how many times have you experienced crop losses on Number this plot What was the reason for this crop loss Use codes What was the amount of money that you lost (in both crop and asset Amount damage) as a result of this damage Land tenure: How did your household acquire this land Use codes Plots Owned What legal title or ownership rights do you have for this plot of land Use codes by the Year this plot was first acquired. Year Household If you were to sell this plot of land today, how much could you sell it Amount for Land tenure: During the last 12 months, in which seasons have you farmed this Use codes Plots Rented land in/Sharecropp From whom is this plot rented or borrowed Use codes ed Cash / Value of in What is the total amount of money and in kind payments that you kind paid to the owner for this land during the past 12 months Did the owner of this land pay for any of the farming inputs used 1= Yes 2= No What was the value of these inputs Amount Land tenure: How did your household acquire this land Use codes Plots Rented What legal title or ownership rights do you have for this plot of land Use codes Out Year this plot was first acquired Year If you were to sell this plot of land today, how much could you sell it Amount for To whom is this plot rented or lent out Use codes What share of the output is given to your household by the tenants Percent What is the total amount of money and in kind payments that you Cash / Value of in received for renting this land during the past 12 months kind Plot What method of irrigation do you use in each growing season on your farm Use codes irrigatio Water Source for the plot Use codes n data Please describe the method by which water is delivered to this plot. Use codes If groundwater: Is this plot irrigated with groundwater from your own well, Is Use codes an electric pumpset or diesel pumpset used to pump water from the well If you do not own a well, but receive water from someone else's well, how Amount at each much did you pay for the water per season for this plot (cash equivalent) season during the past 12 months What soil type/land quality is the plot Use codes What is the approximate area of your land that is serviced by surface drains Use codes Describe the operation and maintenance conditions for the drainage Excellent to Unusable facilities on this plot What is the length of the surface drains serving this plot Length Is flooding a problem on this plot of land 1= Yes 2= No In the last 5 years, how many times has flooding caused a loss of crops on Number this plot of land What was the amount of money that you lost (in both crop and asset Amount damage) as a result of this flooding Has a high water table reduced yields on this plot 1= Yes 2= No In the last 5 years, how many times has a high water table caused reduced Number yields on this plot of land Wells How many wells do you own ? 1= Yes 2= No Record the depth of each well in each growing season Depth For each well, how does the groundwater level compare to five years ago Risen, Fallen, Stayed the same What is the horsepower rating for each electric/ diesel pumpset that you Power for each pump own Crop What crop(s) did you grow on each of your plots in Year 1 at each season Use codes 20 Toolkit for monitoring and evaluation of AWM Reference Note 3 projects Model Survey Questionnaires Question Subquestion Answer timelin What crop(s) did you grow on each of your plots in Year 2 at each season Use codes e What crop(s) did you grow on each of your plots in Year 3 at each season Use codes Plot Of the crops grown on each of your plots how much did you harvest during Quantity at each yields each growing season during the past 12 months season for each plot and Of the crops harvested on each of your plots how much did you sell during Quantity output the past 12 months prices Where was the crop sold Use codes What was the price you received for each crop that you harvested and sold Amount in each season Of the total quantity harvested, how much was consumed Quantity Of the total quantity harvested, how much was set aside for Quantity seed/feed/stored What was the total value of crop losses during the past 12 months for each Quantity plot What were the reasons for crop losses Use codes Plot Seeds Quantity input Fertilizer Quantity quantiti Manure Quantity es at Pesticide, Herbicide, Fungicide Quantity each Hired Labor Quantity season Land preparation Quantity Agricultural Equipment Rental Quantity Irrigation charges Quantity Transportation costs Quantity Post harvest costs Quantity Other production costs Quantity Livesto Does any member of your household currently own any livestock or poultry 1= Yes 2= No ck and Does any member of your household currently own any of the following 1= Yes 2= No Fisheri animals: Beef cattle, Milk cows, Work Animals, Pigs for Breeding, Sheep . es How many of each of these does your household currently own Number If you sold one of these animals today, how much money could you get for it Amount What is the total amount of income that your household has received from Amount the sale of the following animal groups in the last 12 months How much money have you earned in total in the past 12 months on food Amount products derived from the following animal groups? How much money have you spent in total in the past 12 months on feed and Amount veterinary care for each of the following animal groups How many of each of the animals did your household comsume in the last Number 12 months Extensi Did you consult with an agricultural extension agent or an agricultural 1= Yes 2= No on and extension center during the last 12 months to seek advice or assistance on Marketi growing crops ng How many times during the last 12 months did members of your household Number consult an agricultural extension agent or an agricultural extension center to discuss growing crops For what crops were these consultations made List three most important What kinds of assistance or information was requested Use codes How much did you pay for this assistance during the last 12 months Amount How many times during the last 12 months did members of your household Number consult an agricultural extension agent or an agricultural extension center to discuss raising animals For what animals were these consultations made? List three most important What kinds of assistance or information were requested Use codes How much did you pay for this assistance during the last 12 months Amount Was this extension information useful 1= Yes 2= No In what ways was this extension information useful Use codes 21 Toolkit for monitoring and evaluation of AWM Reference Note 3 projects Model Survey Questionnaires Question Subquestion Answer Of the production that you sold at market in the last 12 months, what Percent for each case percentage: Was sold directly from the farmer, Was sold at the nearest periodic market, Was sold at the nearest daily market What is the distance to the nearest periodic market Distance How often does this periodic market meet Frequency Source: South Asia Rural Development Unit, World Bank, 2004 Table 2: Scheme Survey Question Subquestion Answer Name of irrigation scheme Type of scheme 1=Major 2=Medium 3=Minor Location of Name of basin Scheme What irrigation districts are covered by the scheme Cultural command area Ha Irrigable command area Ha Gross irrigated area Ha Number of villages served by the scheme Number Number of households in the scheme command area Number How many WUAs have already been established in this scheme Number What was the total live storage of this scheme during the past year Million Cubic Meter In addition to Rural domestic purposes, Urban domestic purposes, Industries, 1= Yes 2= No water for Hydroelectric power generation, Fisheries, Other irrigation, does If the answer is "yes" to any of the above, what is the live storage Million Cubic Meter this scheme reservation for each of these uses supply any Million Cubic Meter water for any of the following What was the total volume of water released for each of these purposes uses Cropping In each seasons, what was the net cultivated area for each of the Ha patterns for the crops in this scheme last 12 months In each seasons, what percentage of the net cultivated area for Percent each of the crops was irrigated What percentage of the irrigated area in this scheme is irrigated Percent by canals, by lift irrigation, by wells, by tanks, by other sources What percentage of the irrigated area in this scheme is irrigated Percent conjuntively (i.e.surface and groundwater sources)? What was the yield per hectare for the both the irrigated and Yield rainfed land in this scheme, for each crop Performance Water delivery efficiency Percent indicators Reliability of water delivery Rate Adequacy of water charges for full O&M Rate Water fee collection performance (from all water users, from Rate farmers, from industry, from domestic users) Annual change in groundwater table Depth Water use efficiency at scheme level M3/Ha Source: South Asia Rural Development Unit, World Bank, 2004 22 Toolkit for monitoring and evaluation of AWM Reference Note 3 projects Model Survey Questionnaires Table 3: Community Survey on Water User Association Question Answer Name of the Irrigation Scheme Please describe the structure of Water User Associations in your area: Use codes None, no WUAs formally exist , WUAs exist on paper but conduct no meaningful functions, WUAs exist but possess limited powers, WUAs exist and have authority and influence in the community How do farmers in your area determine how to allocate water: Each Use codes Farmer decides on their own without consultation, Farmers informally coordinate their water usage activity but this is largely unorganized, An informal, organized body allocates water usage within the community Name of the WUA How old is this WUA? Years What is the actual irrigated land area that is served by this WUA Ha What is the average size of land holdings served by this WUA? Ha How many farms are there in this WUA? Number How many farms are between 0 1 Ha, 1 2 Ha, more than 2 Ha Number What percentage of the irrigation water supply comes from each of the Percent following sources: Canal, Lift, Tank, Well, River Does the irrigation system under control of the WUA contain 1= Yes 2= No constructed drainage ditches What is the land area that is served by surface/ buried drains in this Ha WUA? What is the total land area that contains natural drainage Ha What percentage of the farmers' irrigation needs come from the WUA Percent How much water is allocated to your WUA m3 How many members are in your WUA Who is in charge of collecting water charges WUA, Irrigation Department, Other Who determines the amount that is charged for water WUA, Irrigation Department, Other What is the amount and the basis of the water charge. Basis: By area Amount, Basis per year, By crop and area per year, Per irrigation, Volumetric, Timing of irrigation What percentage of these charges are collected Percent Please provide an estimate of the total annual water charges collected Amount, including in kind from all farmers in the community throughout the year How frequently are these charges revised Frequency What is the total annual cash value of the following in kind services that Amount are provided to the WUA by water users: Labor, Crop, Construction, Other Is there a special charge for private well usage 1= Yes 2= No Are your water rights clearly defined by the water authority 1= Yes 2= No In your area, are WUAs empowered in the following respects: 1= Yes 2= No recognized under the law, have the power to levy a tax, can dismiss employees Does the state and/or federal government support the operation of your 1= Yes 2= No WUA Indicate the areas in which the government supports the WUA: training, 1= Yes 2= No investment, financing, co shared programs, assistance, other In your area, -The government has passed laws that have 1= Yes 2= No please indicate the empowered the WUA following regarding -The WUA rely on local authorities to enforce the WUAs these laws relationship local -Local authorities effectively prosecute wrong authorities doers -Corruption is a problem with the local authorities Are you satisfied with the relationship between the WUA and the 1= Yes 2= No irrigation department 23 Toolkit for monitoring and evaluation of AWM Reference Note 3 projects Model Survey Questionnaires Question Answer To receive water, do the farmers have to belong to the WUA 1= Yes 2= No How frequently does the WUA meet Frequency What percentage of the members attend the meetings on a regular Percent basis Do you have women participating in the management of the WUA 1= Yes 2= No How often do you change the executive board of your association Frequency Select the answer -Elected by all farmers with one vote per farmer Use codes that most closely -Elected by all farmers with votes weighted by matches the farm size manner in which Appointed the governing board of the WUA is chosen Does a government employee sit on the board of this WUA 1= Yes 2= No Do women sit on the board of this WUA 1= Yes 2= No Do small scale farmers sit on the board of this WUA 1= Yes 2= No How many people are employed by your WUA Number Does your association employ professionals 1= Yes 2= No Number of WUA -Professional, permanent employees (college Number employees degrees and well trained technicians) employed in each -Professional employees that are temporary or of the following contract functions: -Non professional, permanent employees -Non professional employees that are temporary or contract Specify as a Operating costs for water delivery Percent percentage of total Maintenance costs WUA operating Replacement of Infrastructure cost, the following Replacement of Machinery and Equipment items Taxes and Water Rights Other (specify) Do you conduct formal audits of your accounts 1= Yes 2= No Do you publish your annual accounts and present them to your users 1= Yes 2= No In your area, The WUA is sustainable financially 1= Yes 2= No please indicate the The WUA requires government support to following regarding function the financial WUA equipment is well maintained strength of the Essential equipment functions regularly WUA WUA has the power to charge for water WUA has the power to obtain loans Is your WUA involved in providing services other than irrigation water 1= Yes 2= No delivery Indicate the Distribution of water in the area, Canal Include all that apply additional services maintenance, Facility construction, Collection of provided by the fees other than water fees, Crop diversification, WUA: Farmer Cooperative, Technical advice, Sideline businesses such as transportation, Supply production inputs for farmers, Provide marketing services to farmers, Other (specify) How frequently is water provided to farmers Sometimes, On demand, Weekly Is the WUA evenhanded in its water allocation 1= Yes 2= No What are the Tail end farmers receive less water, Delivery Include all that apply reasons for this? problems, Not enough water for all farmers, All farmers want water at the same time, Interest group(s) prevent equality When farmers complain about water service, does your association 1= Yes 2= No take corrective action What is the nature of the corrective action taken No action taken, Warning, Monetary Fine, Suspension of 24 Toolkit for monitoring and evaluation of AWM Reference Note 3 projects Model Survey Questionnaires Question Answer water delivery, Other Will farmers in your association be willing to switch to high value crops 1= Yes 2= No if water service is improved Are salinity and water table monitored in your district 1= Yes 2= No What percentage of farms in your association have salinity problems Percent What percentage of farms in your association have water table level of Percent less than xx meters Which of the High use of fertilizer, High use of pesticides, Include all that apply following Unsafe disposal of agrochemicals, Soil Erosion, environmental Loss of biodiversity, Loss of wetlands, Other problems are present on farms in your area? Is the irrigation water supply used for human consumption in your area 1= Yes 2= No Have there been episodes of waterborne disease in you area 1= Yes 2= No Is aquaculture present in your area's irrigation system 1= Yes 2= No Source: South Asia Rural Development Unit, World Bank, 2004 Table 4: Irrigation Department Survey Question Subquestion Answer Name of Irrigation Department Irrigation District Position of the Respondent at the Irrigation Department Who manages the irrigation Government, Private company, Joint Use codes department? organization, NGO In your irrigation district, do individual WUAs also belong to a larger, 1= Yes 2= No project level organization Does the larger, project level organization operate the main canals What was the average annual Total Staff Salaries, Improvement and Amount amount of money paid for the modernization of existing irrigation following functions of the structures, Operation and irrigation department in the Maintenance, Rehabilitation of last 5 years rundown irrigation structures, Other (specify) Who manages the irrigation Irrigation department, WUAs, NGO, Use codes supply infrastructure Private firm, Other (specify) What was the average annual -Country or State Government Amount amount received from the -Foreign Sources different sources of the -Water user fees irrigation department budget -NGOs over the last 5 years -Other (specify) Please list the number of -Professional, permanent employees Number irrigation department (college degrees and well trained employees employed in the technicians) following functions -Professional employees that are temporary or contract -Non professional, permanent employees -Non professional employees that are temporary or contract What is the average number of years that a typical professional Years employee works for the irrigation department How many of the irrigation department staff actually work in the field Number indicate the average -Office work Percent for each type of staff percentage of time per month -Travel that each type of irrigation -Interacting with farmers staff member spends in the -Operating irrigation infrastructure following functions -Other (specify) 25 Toolkit for monitoring and evaluation of AWM Reference Note 3 projects Model Survey Questionnaires Question Subquestion Answer Which of the following most The department is overstaffed, The Use codes closely describes the staffing number of staff is sufficient, The situation in the irrigation department is understaffed department With the existing staff, are Ahead of schedule, On schedule, Use codes how quickly are department Behind schedule tasks completed How would you describe the Training is excellent, average, below Use codes level of training given to average managers at all levels within the irrigation department How would you describe the Strong service orientation, Adequate Use codes service orientation of service orientation, Inadequate department managers service orientation How often are department Once every 6 12 months, Once every Use codes employees given a written 13 24 months, Less than once every performance evaluation 24 months, No written evaluations are conducted in the department How easy is it for the irrigation Easy to dismiss employees, Difficult Use codes department to dismiss but possible to dismiss employees, employees with cause Virtually impossible to dismiss employees How often are employees Frequently, Occasionally, Never Use codes dismissed from the department Are individual irrigation department employees able to make significant 1= Yes 2= No departmental decisions without prior authorization from their superiors How often do employees take Frequently, Occasionally, Never Use codes the initiative to make these decisions Which of the following most Promotions occur regularly to a Use codes closely describes the significant number of individuals, irrigation department's Promotion is based on time in service, promotion system Promotions occur infrequently, No promotion system exists Is the staff of the irrigation -Collecting water charges 1= Yes 2= No department given strong -Delivering water on schedule incentives (i.e. bonuses or penalties) for the following River Name of Basin Basin Position of the Respondent at the Riverbasin Authority Authority What is the area of the basin square km What is the amount of water resources in the basin Million cubic meter per year What year was the River Basin Organization created Year What is the name of the governing body of the River Basin Organization Please select the answer that most closely matches the Nominated, Appointed, manner in which the governing board of the river basin is Designated chosen Which group was involved in the choosing the River Basin Federal Government, State Organization using this process Government, Local Government, Users Was the creation of the River Basin Organization a Bottom up, Top down bottom up or top down process Does the River Basin Organization hold forums to hear 1= Yes 2= No disputes 26 Toolkit for monitoring and evaluation of AWM Reference Note 3 projects Model Survey Questionnaires Question Subquestion Answer Indicate the percentage of each of the following Percent at each level responsibilities that are performed at the local, basin, state, and national levels: Water Administration, Infrastructure Financing, Water Quality Enforcement, Setting Water Quality Standards, Water Allocation, Other (specify) What is the current water charge for the following user Amount groups: Irrigation Users, Industrial Users, Domestic Users, Other (specify) What percentage of the water charges stay in the basin Percent What percentage of the water charges stay in the project Percent What is the annual budget for the River Basin Amount Organization What percentage of the budget was used for the following Percent activities: Investment in the basin, Other development activities, Operation and Maintenance, Water quality activities, Other (specify) Source: South Asia Rural Development Unit, World Bank, 2004 27 Toolkit for monitoring and evaluation of AWM Reference Note 4 projects Model WUA Survey Questionnaires Reference Note 4 Prototype survey questionnaire for monitoring formation of Water User Associations in agricultural water management projects Introduction This survey questionnaire was used on the Second Irrigation and Drainage Project in Albania as part of the programme to monitor and evaluate the WUA formation and support component of the project. The survey sought to ascertain the strength of each WUA in terms of its institutional, financial and technical functions. Points arising from the analysis of the data collected using this survey form may include: · The climatic conditions during the surveyed year (wet, dry, average), and the subsequent demand for irrigation; · Assessment of how many WUAs are functioning adequately, and how many are non functional; · Level of WUA membership relative to total number of landowners; · Level of irrigation demand during the season; · Extent of the reported irrigated area; · Extent of the actual irrigated area; · Level of actual fee recovery to potential fee recovery; · Collection rates for the Irrigation Service Fee; · Financial performance of WUAs and relative levels of income and expenditure; · Levels of expenditure relative to assessed minimum expenditure levels, especially in regard to maintenance; · Level of build up of operational funds (for capital expenditure); · Quality of the discharge measurement carried out by the WUA; · Number of unauthorised offtakes; · Amount and cost of training undertaken; · Number and types of disputes recorded by the WUA; · Levels of employment of WUA staff; · Condition of infrastructure and incidence of emergency repairs; · The number of associations with a reasonable set of maps and records; Table 1 summarises the data collected, whilst Table 2 presents the contents list of the report that was based on these data. 28 Toolkit for monitoring and evaluation of AWM Reference Note 4 projects Model WUA Survey Questionnaires Table 1: Data collected through WUA survey questionnaire Category Data collected A1. Background · Location (Region, District, etc.) information (from WUA · Scheme name supervisory/ regulatory · WUA total service area office) ­ Governance · Number of landholders in service area and membership · Type of water source (river, tubewell, etc.) · Name of water source · Type of scheme (gravity, lift) · Type of headworks (dam, weir, free intake, pump, other) · Distance to headworks (from WUA service area) · Year primary canal, secondary canal, main drains rehabilitated · Date and area transferred · Registered address · Court registration/legal certificate · Bank account details · Current situation with annual audits A2. Details from WUA · Water source manager (WUA, Federation, Government agency, etc.) Chairman and staff ­ · Headworks manager (WUA, Federation, Government agency, etc.) Governance and · Primary canal manager (WUA, Federation, Government agency, etc.) membership · WUA current total service area · Number of landholders in service area · Cultivable command area · Reasons for low CCA, if less than 80% total service area · Total number of WUAs on primary canal, and numbers upstream and downstream · Number of WUA members · Annual membership fee · Representative system for General Assembly (Yes/No) · Numbers attending last General Assembly · Number of Administrative Council members · Number of Administrative Council meetings last year · Minutes of meeting available (Seen/Not seen, Quality of minutes) B. Details from WUA · WUA management: Number and categories of staff, period worked and salaries Chairman and staff ­ · Water supply and demand: Water supply information for previous year ­ area planned; Management and irrigation water demand; number of irrigation days in season; total volume of water supplied; operation days of rainfall each month; days irrigation canals closed each month; reasons for closure; estimated percentage of total crop water requirement from (i) rain, (ii) iriigation; in season irrigation scheduling (Yes/No); basis for water allocation; overall WUA assessment of water availability · Drainage: Area serviced by drains; extent and nature of drainage problems; area flooded · Cultivable command area: by gravity; by pumping; area not irrigated; reasons for not irrigating; action needed to make water available; area permanently not cultivable · Actual irrigated area: area of first irrigation; total area irrigated in season · Income and expenditure: Budget prepared for this year (Y/N); accounts prepared (Y/N); WUA balance sheet ­ income items, expenditure items; irrigation service fee (gravity/pumped); number of irrigators paying ISF; number of irrigators remaining to pay ISF · WUA Records (existence and quality): Irrigation system map; irrigation system inventory; crop record book; up to date membership list · WUA assets (existence and quality): offices; furniture; computers; transport; other C. Details from WUA · Infrastructure inventory: Details (number, size, capacity, lengths, lined/unlined, year Chairman and staff ­ rehabilitated, etc.) of infrastructure managed by the WUA ­ reservoirs; headworks; pumps; Infrastructure primary canal; secondary canals · Infrastructure condition (functional, non-functional): Canals and drains · Operational problems (in previous year): infrastructure damaged; number of unauthorised offtakes; list of operational problems occurring; details of emergency repairs done · Maintenance: Form of maintenance inspections; method used for determination of maintenance budget; amount of identified maintenance completed; value of maintenance work done; percentage of WUA income spent on maintenance; description of maintenance work done 29 Toolkit for monitoring and evaluation of AWM Reference Note 4 projects Model WUA Survey Questionnaires · Disputes: Number and form of dispute · Training: Who trained, in what, for how long · Awareness creation activities: Posters/brochures seen, village meetings held · Enumerators assessment of WUA personnel: Scored assessment, excellent to very poor Source: 2002 WUA Survey, Second Irrigation and Drainage Rehabilitation Project, Albania, World Bank 30 Toolkit for monitoring and evaluation of AWM Reference Note 4 projects Model WUA Survey Questionnaires Box 1: Example of contents list for WUA survey report Water Users Associations Survey 2002 Second Irrigation and Drainage Rehabilitation Project 1 EXECUTIVE SUMMARY AND RECOMMENDATIONS ........................................................ 1.1 Key Findings.................................................................................................................. 1.2 Other Findings............................................................................................................... 2 INTRODUCTION AND BACKGROUND............................................................................... 2.1 Purpose......................................................................................................................... 2.2 Project Brief................................................................................................................... 2.3 Description of a Water Users Association....................................................................... 2.4 Sources of Information................................................................................................ 2.5 Data Quality................................................................................................................... 3 WATER USERS ASSOCIATION DETAILS.......................................................................... 3.1 Types of Water Users Association ................................................................................. 3.1.1 Number and Types of Water Users Associations..................................................... 3.1.2 Reasons for Non Functioning.................................................................................. 3.2 Water Users Association Administration......................................................................... 3.2.1 Assets .................................................................................................................... 3.2.2 Records................................................................................................................ 3.3 Area ............................................................................................................................ 3.4 Drainage Problems...................................................................................................... 3.5 Water Users Association Membership.......................................................................... 3.5.1 Villages................................................................................................................. 3.5.2 Landholders.......................................................................................................... 3.5.3 Membership.......................................................................................................... 3.6 Representation............................................................................................................ 3.6.1 General Assembly ................................................................................................ 3.6.2 Representation Systems....................................................................................... 3.6.3 Administrative Council .......................................................................................... 3.7 Water Users Association Staffing................................................................................. 3.8 System Management................................................................................................... 3.9 Condition of Infrastructure............................................................................................ 3.9.1 Irrigation............................................................................................................... 3.9.2 Infrastructure Condition......................................................................................... 3.9.3 Maintenance Activities.......................................................................................... 3.9.4 Damage and Theft................................................................................................ 3.9.5 Operational Issues................................................................................................ 3.10 Water Supply in 2003 .................................................................................................. 3.10.1 Irrigation Area Supplied ........................................................................................ 3.10.2 Irrigated Area Planning......................................................................................... 3.10.3 Water Measurement ............................................................................................. 3.10.4 Irrigation Scheduling............................................................................................. 3.10.5 Irrigation Supply Timing ........................................................................................ 3.10.6 System Closure .................................................................................................... 3.10.7 Source of Crop Water Requirements..................................................................... 3.11 Water Users Association Financial Management.......................................................... 3.11.1 Accounting Records.............................................................................................. 3.11.2 Budgets and Income............................................................................................. 3.11.3 Trends in Irrigated Area and Income..................................................................... 3.11.4 Financial Performance.......................................................................................... 3.12 Training and Awareness .............................................................................................. 3.13 Disputes...................................................................................................................... 3.13.1 Disputes with other Water Users Associations or Federations............................... 3.13.2 Disputes with Members......................................................................................... 4 WATER USERS ASSOCIATION PERFORMANCE ASSESSMENT................................... 4.1 Key Indicators.............................................................................................................. 4.2 Performance Indicator Scores...................................................................................... 4.3 Characteristics of Water Users Association by Score Level.......................................... Source: 2002 WUA Survey, Second Irrigation and Drainage Rehabilitation Project, Albania, World Bank 31 Toolkit for monitoring and evaluation of AWM Reference Note 4 projects Model WUA Survey Questionnaires WATER USERS ASSOCIATION SURVEY PROFORMA Year: . Enumerator: Survey Date: WUA Name: WUA Code: WUA type Project 1 Do parts A, B, C (tick ONE) Project 2 rehabilitated June 02 Do parts A, B, C Project 2 contract let Dec 02 Do parts A, B Project 2 identified for future rehabilitation Do parts A, B Independent Do part A PART A: ALL WATER USERS ASSOCIATIONS PART A 1 DATA PROVIDED BY PROJECT MANAGEMENT UNIT Listing Data From Field Staff District(s): Scheme Name: Scheme Code (if any): WUA Gross Service Area (ha) Supporting Note Number of Landholders in WUA command area: Was WUA Functional in 2002 Yes (Tick ONE) No Did WUA Irrigate in 2002 Yes (Tick ONE) No Supporting Note Data From Engineering Unit Project Phase: Project 1 (Tick ONE) Project 2 Cycle I Project 2 Cycle II Spontaneous Water Source Type(s): River Reservoir/Lake Groundwater (Shallow wells or Tubewells) Water Source Name(s): Type of Scheme: Gravity Pump/Lift Type(s) of Headworks: Dam Diversion Weir Free Intake from River Pump(s) Other (Specify): 32 Toolkit for monitoring and evaluation of AWM Reference Note 4 projects Model WUA Survey Questionnaires WATER USERS ASSOCIATION SURVEY 2002 WUA Code: PART A 1 (continued) Scheme Code Primary Canal ID: Distance to Headworks (km): Year Primary Canal Rehabilitated: Year Secondary Canal(s) Rehabilitated: Year Drains Rehabilitated: Data From Support Unit Area Transferred by December 2002 (ha): WUA Registered Address: WUA Court Registration: Number: Date: WUA Bank Account(s) Name of Bank: Account Number(s): Membership of Federation of WUAs: No FWUA Name: Yes Crop Area Survey in 2002 Area surveyed in WUA (ha): Crop Area Correction Factor (%): Financial Audit of 2001 Financial Year undertaken in No 2002 Yes Outcome: Account unqualified Accounts qualified Accounts rejected Legal Action started Rehabilitation Contribution Paid by December 2002: Amount: Lek % of Total Amount Due: 33 Toolkit for monitoring and evaluation of AWM Reference Note 4 projects Model WUA Survey Questionnaires WATER USERS ASSOCIATION SURVEY 2002 WUA Code: PART A 2 WATER USERS ASSOCIATION DETAILS FROM CHAIRMAN Irrigation Scheme Water Source Manager(s): WUA Federation of WUAs Water Enterprise Other (Specify): Headworks Manager(s): WUA Federation of WUAs Water Enterprise Other (Specify): Primary Canal Manager(s): WUA Federation of WUAs Water Enterprise Other (Specify): Gross Service Area (GSA) in 2002 (ha): Supporting Note Number of Villages having Land in GSA: Number of Landholders in GSA: Number of Landholders in GSA has changed since No Transfer: Yes Reason(s): (Tick ONE) Cultivable Command Area (CCA) in 2002 (ha): % of GSA: Reason(s) if CCA is less than 80% of GSA: Total Number of WUAs on Primary Canal: Number Upstream/Above: Number Downstream/Below: Total Number: Was Water Supplied by the WUA in 2002? Yes (Tick ONE, If No give Reasons) No Reasons: Did Farmers Cultivated Irrigated Crops in 2002? Yes (Tick ONE, If No give Reasons) No Reasons: 34 Toolkit for monitoring and evaluation of AWM Reference Note 4 projects Model WUA Survey Questionnaires WATER USERS ASSOCIATION SURVEY 2002 WUA Code: PART A 2 (continued) Water Users' Association Membership Number of WUA members in 2002: Annual Membership Fee (Lek): General Assembly Representative System for General Assembly: No Yes Total Number of Representatives: Number of Female Representatives: Number of Meetings of General Assembly in 2002: Number: Date of Last Meeting: Number of WUA Members/Representatives at Last Meeting of General Assembly: Minutes of Meetings of General Assembly: None (Tick ONE) Reported as complete (but not seen) Incomplete Complete/Detailed Administrative Council Number of Administrative Council Members in Total Number: 2002: Number of Female Members: Date of Last Elections of Administrative Council Members: Number of Administrative Council Meetings in Number: 2002: Date of Last Meeting: Minutes of Meetings of: Administrative Council: None (Tick One) Reported as complete (but not seen) Incomplete Complete/Detailed 35 Toolkit for monitoring and evaluation of AWM Reference Note 4 projects Model WUA Survey Questionnaires WATER USERS ASSOCIATION SURVEY 2002 WUA Code: PART B: MANAGEMENT AND OPERATIONAL INFORMATION PART B 1 WATER USERS ASSOCIATION MANAGEMENT Total Number of Months Worked/Paid Year First Male or Year Last in 2002 Post Employed Female Elected or Elected Total period Amount Paid (Lek) worked Chairman Deputy Chairman General Secretary Accountant Water Master No.1 Water Master No.2 Water Master No.3 Water Master No.4 Water Master No.5 Water Master No.6 Labourers (No.) Others: PART B 2 WATER USERS ASSOCIATION WATER SUPPLY IN 2002 Month May June July Aug Sept Total Total days in month (31) (30) (31) (31) (30) Planned irrigated area (hectare)1 Planned irrigation water demand (000 m3)1 Irrigating days during month Average hours supply per day (out of 24) Do they measure irrigation water? No Yes Where?: Total volume of irrigation water supplied Gravity (000 m3)1 Pumped Days of rainfall each month (days) Days irrigation system closed each month Reason(s) for Closure2: Estimated percentage of From rain (%) total crop water From irrigation (%) requirement Shortfall in supply (%) In season irrigation scheduling No How often? Yes Water allocations based on: Demands from farmers Fixed schedule Other (Specify): Overall WUA assessment of More than adequate Adequate Less than adequate None irrigation water availability 1 Codes: NC ­ Not Calculated NM ­ Not Measured NA ­ Not Available 2 1 ­ Rain 2 ­ No demand 3 ­ Damaged canal 4 ­ Disputes 5 ­ Other (specify) 36 Toolkit for monitoring and evaluation of AWM Reference Note 4 projects Model WUA Survey Questionnaires WATER USERS ASSOCIATION SURVEY 2002 WUA Code: PART B 3 WATER USERS ASSOCIATION DRAINAGE IN 2002 Summer Winter (April Sept) (Oct March) Gross Service Area (ha) (from page 1) Is there a drainage problem in the GSA ? No No Yes Yes Area with no drainage problem (ha) Area with slight drainage problem (ha) Area with serious drainage problem (ha) Area flooded (ha) Note: No problem = No damage to crops (no standing water after 24 hours of rain/irrigation) Slight = Some damage to crops (standing water for 2 to 3 days) Serious = Serious damage to crops (standing water for 4 or more days) Flooded = No crops (standing water for most of season) PART B 4 WATER USERS ASSOCIATION CULTIVABLE COMMAND AREA IN 2002 TOTAL Gross Service Area (ha) (from page 1) 1 Cultivable area in command by gravity in 2002 (ha) 2 Cultivable area in command by pumping in 2002 (ha) 3 Cultivable area not irrigated in 2002 (ha) Reasons for area not irrigated in 2002 Action required to make water available 4. Area within Gross Service Area PERMANENTLY not cultivable (ha): Total Note: The areas reported in 1 to 4 should sum to the Total GSA at transfer 37 Toolkit for monitoring and evaluation of AWM Reference Note 4 projects Model WUA Survey Questionnaires WATER USERS ASSOCIATION SURVEY 2002 WUA Code: PART B 5 WATER USERS ASSOCIATION REPORTED IRRIGATED AREA IN 2002 Area of first irrigation (ha) Total area irrigated (ha) PART B 6 WATER USERS ASSOCIATION BUDGET AND ACTUAL INCOME/EXPENDITURE IN 2002 Budget for 2002: Not Prepared Copy Available (Tick ONE) Copy Not Available Accounts for 2002: Not Prepared Copy Available (Tick ONE) Copy Not Available ITEM BUDGET ACTUAL Balance from 2001 INCOME Irrigation fees Annual Contribution Membership fee Fines Special Contributions Credits EXPENDITURE Due to Water Enterprise and/or Federation Salaries Equipment materials Maintenance irrigation Maintenance of drainage Electrical Power Administrative Expenses Office Expenses Paying off of Credits Reserve Fund Balance in bank Irrigation Service Fee for 2002 Gravity Lek/ha1 Pumped Lek/ha1 Number Total amount paid/due Number of irrigators fully paying irrigation fees Number of irrigators remaining to pay irrigation fees 1 ISF may be charged in terms of Lek/hour ­ amend as necessary 38 Toolkit for monitoring and evaluation of AWM Reference Note 4 projects Model WUA Survey Questionnaires WATER USERS ASSOCIATION SURVEY 2002 WUA Code PART B 7 WATER USERS ASSOCIATION RECORDS Irrigation System Map: None Sketch Map Scale: 1: Detailed Irrigation System Inventory: None Summary Detailed Crop Book: None Not Used Incomplete Complete Membership List: None Out of Date Up to date PART B 8 WATER USERS ASSOCIATION ASSETS Office Space No Yes Owned Rented Free Use or Loaned Office Furniture: No Yes, specify: Communication Equipment: No Yes, specify: Computer No Yes, specify: Transport No Yes, specify: Other: No Yes, specify: 39 Toolkit for monitoring and evaluation of AWM Reference Note 4 projects Model WUA Survey Questionnaires WATER USERS ASSOCIATION SURVEY 2002 WUA Code PART C: WATER USERS ASSOCIATIONS IN PROJECT AND REHABILITATION COMPLETED BY JUNE 2002 PART C 1 WATER USERS ASSOCIATION INFRASTRUCTURE Main Condition Capa city Measure Total Length Year ITEM ID Name/ better or ment Length Lined tenance Number (l/s or Gated3 Rehab done in worse than MCM) facility3 (km) (km) ilitated 20023 20014 Reservoir1 Headworks2 Pump(s)2 1Capacity measured in millions of cubic metres (MCM) 2Capacity measured in litres per second (l/s) 3None, Partly, Full 4Worse, No change, Improved Condition Measure Total Length Main Year ITEM ID Name/ Capa city better or ment Length Lined tenance Number Gated3 Rehab (l/s) facility3 worse than (km) (km) done in ilitated 20023 20014 Primary Canal How Area Numbe Number Length Length Year Main Condition many range r with range lined Rehab tenance better or (No.) (ha) gated measur (km) (%) ilitated done in worse than Secondary (No.) ing 20023 20014 Canals structure (No.) Direct How Total How offtakes from many area fed many. Primary (No.) (ha) gated Canal (within (No.) WUA command area) 40 Toolkit for monitoring and evaluation of AWM Reference Note 4 projects Model WUA Survey Questionnaires WATER USERS ASSOCIATION SURVEY 2002 WUA Code: PART C 2 CANAL AND DRAIN CONDITION SUMMARY IN 2002 IRRIGATION SEASON Percentage (%) in Condition 3 1 2 4 Moderate 5 Fully Minor loss Serious loss loss of Functional function of function Non functional function (%) (%) (%) (%) (%) Canals Primary Canal Secondary Canals Tertiary Canals Drains Tertiary Drains Secondary Drains Main Drain PART C 3 PROBLEMS WITH OPERATION OF IRRIGATION SYSTEM IN 2002 List infrastructure damaged by vandalism in 2002: List any infrastructure stolen in 2002: Approximate number of illegal Number: offtakes and area served in Area served: system in 2002: Do they pay for water used? Yes No Number of unauthorised pumps Number: being operated and area served Area Served: in 2002: Do they pay for water used? Yes No Any other operational problems: 1. 2. 3. 4. Emergency repairs in 2002 41 Toolkit for monitoring and evaluation of AWM Reference Note 4 projects Model WUA Survey Questionnaires WATER USERS ASSOCIATION SURVEY 2002 WUA Code PART C 4 WATER USERS ASSOCIATION MAINTENANCE PROCEDURE IN 2002 Method used to identify maintenance requirement: None Water Master's Report Chairman's Estimate Maintenance Inspection Other (Specify) Method for determination of maintenance budget: None (Tick ONE) Fixed % of Budget Estimates Detailed Costing Other (Specify) Amount of total identified maintenance completed for 2002 None season: Partly: % (Tick ONE) Full Value of executed maintenance works for 2002 season (Lek): Percentage of WUA income spent on maintenance (%) Short Description of Executed Maintenance Activities for 2002 season Item Activities1/Method2 Notes Headworks: Pump(s): Primary Canal: Secondary Canals: Tertiary Canals: Main drain: Secondary drain: Inspection Roads: 1 Activities 1 = Removal vegetation and debris 2 = Desilting 3 = Repair of earthen embankments 4 = Repair to lined canal sections 5 = Maintenance of gates and other control structures 6 = Repair or replacement of gates and other control structures 7 = Repair or overhaul of pump(s) 8 = Replacement of pump(s) 9 = Grading or repair to inspection roads 10 = Repair to ancillary structures, such as culverts and bridges 2 Method 1 = Contractor 2 = Water Enterprise 3 = Federation of WUAs 4 = Hired Labour 5 = Water Masters 6 = Farmers 42 Toolkit for monitoring and evaluation of AWM Reference Note 4 projects Model WUA Survey Questionnaires WATER USERS ASSOCIATION SURVEY 2002 WUA Code: PART C 5 DISPUTES IN 2002 Number in Method of Resolution Issues 2002 Disputes with Federation Disputes with other WUA Disputes with members/ irrigators PART C 6 TRAINING COURSES IN 2002 Chairman's Assessment of Number Trained WUA Training Course Course Title personnel Up to 2002 In 2002 1 Excellent 3 Adequate 2 Good 4 Poor Chairman Accountant Water Masters Administrative Council PART C 7 AWARENESS ACTIVITIES IN 2002 Seen/Used/Held in Chairman's Assessment of Usefulness WUA 1 Excellent 3 ­ Adequate 2 Good 4 Poor Poster Seen Displayed Brochure Seen Distributed Village Meeting Held Number: .. Part C8 ENUMERATOR'S ASSESSMENT OF WUA PERSONNEL Chairman Accountant Water Masters Other: .. Score (1 5) 1 ­ Excellent 2 ­ Good 3 ­ Adequate 4 Poor 5 ­ Very Poor 43 Toolkit for monitoring and evaluation of AWM Reference Note 5 projects M&E Costs Reference Note 5 Monitoring and evaluation costs Project M&E costs need to follow realistic budget estimates. The most important factors to consider when estimating an accurate budget for M&E activities are scope, methodology, and stakeholder participation. Given the various types of interventions and the development circumstances in each project, it is not practical to prescribe the percentage of the total budget of every project that should be allocated for monitoring and evaluation actions. Few projects detail at appraisal stage the costs for the M&E system, as they include monitoring and evaluation activities within the project management component in most cases. Table 1 gives examples from AWM projects in Fiscal Year 03 and 04, in which the M&E component was specifically budgeted for. These projects, with the exception of the Algeria project, specifically budget for an impact evaluation of their activities. Box 1 provides guidance on costs estimates for impact evaluations, and Box 2 on Management Information System design and development. Table 1: Examples of budgets for M&E in AWM projects, FY03 05 I&D Cost of M&E Project cost component Country Project component in US$ million as % of in US$ million total Uzbekistan Drainage, Irrigation, and Wetlands 1.3 60 96 Improvement Project India Maharashtra Water Sector 1.46 325 84 Improvement Project Pakistan Sindh On Farm Water Management 1.73 61.14 80 Project Indonesia Third Kecamatan Development 3.1 249.8 20 Project Algeria Second Rural Employment Project 0.5 95 10 Source: World Bank. Project Appraisal Documents. 44 Toolkit for monitoring and evaluation of AWM Reference Note 5 projects M&E Costs Box 1: Illustrative costs of an impact evaluation These figures are meant to be purely illustrative: all costs, and especially those for data collection, are going to depend on the impact evaluation method used, and, in turn, the available funds will affect the choice of method. Furthermore, data collection and consultant costs can vary significantly across countries. - The Lead Evaluator will be responsible for evaluation design, survey instrument design and data analysis. If the LE is external to the bank: For a senior researcher around $50 100,000 for professional fees plus travel (travel estimates of at least 3 trips, one of which will be during preparation). If the researcher is internal to the Bank: estimate about 5 15 staff weeks a year. The lead evaluator will need to be involved in all stages of the project. - Sample design expert, about $3000 professional fees plus travel (one trip). This cost will most likely occur during preparation. - Data collection: contract either the national statistical agency or a survey firm. The choice will depend on firm capacity and experience relative to the desired sample size. If the evaluation requires a separate survey, the cost will depend on the sample size, questionnaire length, and the geographical dispersion of the sample. Per interview costs can range from $50 to $150 per respondent per interview. For example, a sample size of 1000 households for two rounds of data collection will cost from $100,000 to $300,000. If it is possible to piggy-back on an existing survey, the cost will be lower than this. - Supervision through the length of the evaluation. This should be a locally based consultant who covers all aspects of the implementation of the evaluation. Costs here can range from $12,000 to $20,000 per year. - Supervision costs from Bank Budget funds will need to be budgeted for. This will likely include one trip per year, and may be combined with other supervision duties, depending on who in the Bank team is supervising the evaluation. - Dissemination of the interim and final results. This will entail two trips of either the counterparts to headquarters or of the evaluation team to the country. Sources of financing: Items 1 and 2 will require project preparation funds for all work that takes place before approval. Grant financing (or the use of the IE code within the Bank Budget) is necessary to fund the later stages of the lead evaluator's work (which are likely to extend beyond the life of the project). Items 3 and 4 can be financed through project funds for activities within the project cycle. For data collection outside of the project cycle other government funds or grants should be used. Item 6 will require grant financing or Bank Budget (possibly through the IE code). Source: World Bank, 2006b. In preparing the M&E workplan, it is necessary to allow adequate provision at the outset of the project for all activities related to M&E3. These will include: Costs for collecting baseline data Concurrent monitoring of implementation activities Independent evaluations and studies Training and capacity building workshops on Management Information System and Monitoring and Evaluation System design, development, and implementation Table 2 gives an example of a typical M&E budget plan, including a breakdown of key expense categories over time. 3Rajalahti, R., J. Woelcke, et al. 2005. Monitoring and Evaluation for World Bank Agricultural Research and Extension Projects: A Good Practice Note. Agricultural and Rural Development Discussion Paper 20. Washington D.C.: World Bank. Available at http://intresources.worldbank.org/INTARD/Resources/M_E_WB_AgExtensionProjects.pdf 45 Toolkit for monitoring and evaluation of AWM Reference Note 5 projects M&E Costs Table 2: Spreadsheet for an indicative M&E budget plan Unit Year 1 .. Final Year Price Quant Cost Quant Cost ity ity Staffing M&E coordinator MIS Specialist Statistician/ Analyst Office equipment Computers Printers UPS Furniture MIS Applications Applications Training Introduction to Computing Computing applications Statistical analysis Report preparation Database and MIS Network principles Beneficiary training Field visits Workshops Assessment, Studies, and Surveys Baseline Surveys Annual M&E review and reports M&E workshop Safeguards reports Social and Environmental Impact Assessments Mid term review Annual Audit Operations and Maintenance Supplies and Maintenance Installation Total Source: Rajalahti and Woelcke, 2005 46 Toolkit for monitoring and evaluation of AWM Reference Note 5 projects M&E Costs Box 3 provides information on the timing and costing of MIS design, development, and implementation. Box 2: Timing and costing the Management Information System Developing a MIS is a long process which length is often underestimated. It is an iterative process which requires adaptation after testing and in the process of use because of changing conditions and priorities. The main activities and timing to set up a MIS are given in below table: Activities Description Timing Responsible person Preparation Review PAD, PIP 1 week Designer Appraisal · Design · 2 3 months · Designer · Start developing · 2 4 months · Developer (programming) · Monitoring · 2 6 weeks · Designer development Implementation · Before 1st · Installation · Main office 2 5 weeks, · Developer application · Training regional offices 15 · Developer and staff weeks MIS specialist · Once a year and · Monitoring · 1 week in 4 month · Designer or MIS mid term review period specialist · Supervision · Checklist · 1 week · Mission members mission monitoring (including Designer) The cost of a MIS are determined by the complexity of the project (size, components, number of donors, level of decentralization) and the complexity of the MIS. They can vary between 75,000$ and 500,000$. The percentage on different activities can be roughly estimated as: 30 % for design, 60 % for development and implementation and 10 % for training. Source: World Bank Water Supply and Sanitation M&E Toolkit, 2007b, draft. 47 Toolkit for monitoring and evaluation of AWM Reference Note 6 projects Sample M&E reports Reference Note 6 Sample M&E reports Karnataka Watershed Management Project, India This Reference Note shows some of the key components and outputs of the M&E system used in Karnataka Watershed Management Project (KWDP). The objective of the World Bank Karnataka Watershed Management Project (KWDP) is to improve the productive potential of selected watersheds and their associated natural resources base in Karnataka State, India, and strengthen community and institutional arrangements for natural resources management. The main mechanism for watershed development and protection is participatory planning and project implementation by communities, with support from government and non government organization (NGO) partners. The approach in KWDP is to blend technically sound soil and water conservation with investments in capacity and institution building. Monitoring and Evaluation, and learning for adaptive management are important components of the project. These build on a number of mechanisms: · independent M&E by an external agency (Antrix Corporation under Indian Space Research Organization); · supportive monitoring activities by the state Watershed Development Department and communities; · Self assessment and third party review of Community Based Organizations. The US$ 100 million project started in 2002 and demonstrates good results two years from completion in 2007, such as an increase of 30% in household incomes, an increase in cropping intensity from 106% prior to the project to 124%, an increase in crop yields by 24%, increased employment opportunities, and reclamation of wastelands and common property resources (Dave, 2007). The M&E system emphasizes the use of modern technologies including in particular, remote sensing and geographic information systems (GIS). Figure 1 shows the communication window of the Management Information System (MIS) used under the project, Figure 2 shows the integrated use of GIS and MIS, and Figure 3 displays an application of remote sensing technology to monitor field bunding. 48 Toolkit for monitoring and evaluation of AWM Reference Note 6 projects Sample M&E reports Figure 1: Management Information System in Karnataka Watershed Management Project Source: Dave, 2007 Figure 2: Integration of MIS and GIS in Karnataka Watershed Management Project Source: Dave, 2007 49 Toolkit for monitoring and evaluation of AWM Reference Note 6 projects Sample M&E reports Figure 3: Field bunding monitoring via remote sensing Before After Source: Dave, 2007 Under the M&E system, Antrix Corporation has worked closely with the Watershed Development Department, World Bank and other stakeholders to provide objective information on project inputs and outputs, processes, impacts and outcomes. The following section presents outlines and excerpts from Antrix M&E reports and presentations. Midterm review of the Karnataka Watershed Management Project Antrix Corporation, 2005. Impact of Sujala. First season visible impacts. Power Point Presentation for the midterm review of the Karnataka Watershed Management Project. Monitoring, Evaluation & Learning Unit. Antrix Corporation. Indian Space Research Organization, Bangalore. June 2005. The objectives of the study are to record and document the short term visible impacts on socio economic conditions and natural resources regeneration. The methodology of study includes household surveys on sample sites, case studies, remote sensing, and the information available from the MIS. The study reviews the activities carried out under the different project components, namely: soil and moisture conservation, major drainage line treatments, and common property resources development. By comparing baseline data to midterm data, the presentation shows the progress in crop diversification (see Figure 4), crop yields, agro horticulture, and groundwater level. 50 Toolkit for monitoring and evaluation of AWM Reference Note 6 projects Sample M&E reports With regard to social mobilization, the study reviews the activities carried out in forming communities based organizations and building capacity, and their impacts (see Table 1). Table 1: Impact of capacity building activities at midterm Impact Self Help Groups Area Groups Increased awareness and skills Moderate Moderate Low Improved Participation Good Moderate Increased self confidence Good Good Increased decision making capability Moderate Moderate Increased environmental awareness Low Moderate Capable to undertake income generating Good Moderate activities Source: Antrix Corporation, 2005. Assessment of Common Property Resources Development Antrix Corporation, 2006. Common Property Resources Development under Sujala ­ an assessment. Monitoring, Evaluation & Learning Unit. Antrix Corporation. Indian Space Research Organization, Bangalore. The table of content for the assessment of common property resources development is as follows: Executive Summary Major Observations Suggestions for Improvement 1.0 Introduction 2.0 Objectives 3.0 Methodology 4.0 Results and Discussion 4.1 Investment Pattern 4.2 Soil and Water Conservation 4.3 Species planted and survival rate 4.4 Survival of species by type of common property resource 4.5 Survival by variety of species 4.6 Reasons for poor survival 4.7 Employment under common property resources 5.0 A few best practices observed Appendix Field photos To enhance livelihoods and reduce poverty of the vulnerable groups, such as marginal farmers and landless, the project emphasizes the development of common property resources by physical treatment of non arable and degraded lands through community participation. The report studies the developments of the common property resources with respect to investment pattern, survival rate of the species planted, community participation, employment generation, and Operation & Maintenance aspects. A random sample was selected for study from the pool of subwatersheds where interventions in common lands have taken place. Structured questionnaires, focused group discussions, ground verifications, and satellite data were the main data sources used. Various treatments like soil and moisture conservation, reforestation, afforestation, have been carried out in grazing lands, wastelands, degraded forests, tank foreshores, drainage lines, schools, graveyards, roads, etc. The study reveals that the overall survival of the plant species recorded is low (57% to 40%) and is a matter to be monitored more closely in future operations. Low community participation, lack of funds for critical irrigation as well as watch and ward, non 51 Toolkit for monitoring and evaluation of AWM Reference Note 6 projects Sample M&E reports finalization of benefit sharing mechanism are identified as the main bottlenecks for survival. Social fencing, which is the prime indicator of community participation is found to be very low. The investment pattern analysis indicates that both forestry and horticulture activities have increased over the project life, where most of the investment at the beginning had gone into soil and moisture conservation. On an average, about 3000 man days of employment have been generated per micro watershed. Substantial opportunities have been created for manual labor and involvement of Self Help Groups in development and maintenance of assets in common property resources. A few best practices have been observed in some of the micro watersheds. In summary, the project is rightfully putting greater emphasis on common property resources development. Implementation and follow up process adopted need refinement and streamlining however. Some suggestions are made for the improvement and sustainability of assets created in common property resources under the project, including in particular increased capacity building and awareness raising, increased involvement of local institutions, quality of saplings, and performance based payments. Assessment of horticultural plants survival rate Antrix Corporation, 2007. An assessment of horticultural plants survival rate. Monitoring, Evaluation & Learning Unit. Antrix Corporation. Indian Space Research Organization, Bangalore. The table of content for the assessment of horticultural plants survival rate is as follows: Executive Summary Major Observations Suggestions for Improvement 1.0 Introduction 2.0 Objectives 3.0 Methodology 4.0 Analysis 4.1 Caste and class wise distribution of beneficiaries 4.2 Size of holding and survival of crops 4.3 Planting and survival by caste 4.4 Survival by species/ varieties 4.5 Irrigation options and survival 4.6 Other field inputs Horticulture is crucial in improving the productivity of land, generating employment, and providing substantial income to the farmers even during adverse climatic conditions. Fruit crops, vegetables, floriculture, and coconut have been taken up across various districts. The report attempts to examine and document the survival rate of the horticultural plantations and their sustainability. Beneficiary assessment using structured schedules, field visits, sample satellite image analysis have been carried out for the study. The overall survival rate is about 64%. The class wise analysis indicates higher survival in the plantations of the marginal class. The overall survival rate of saplings supported by farm ponds was about 79%, a positive result of the watershed concept linking farm ponds with horticulture under the project. The survival rate in the absence of any irrigation option is about 60% among large and small farmers, and about 52% among the marginal class. The major concern however is the relatively low survival recorded among the small land holding class (59%) who constitute substantial portion (46%) of the beneficiaries under this sector. 52 Toolkit for monitoring and evaluation of AWM Reference Note 6 projects Sample M&E reports The report recommends that greater emphasis should be on the species selection, quality of saplings, season of planting, protective measures, etc, in order to achieve better survival rate. Village Based Training with emphasis on Integrated Pest Management and Integrated Nutrient Management, effective modulation of water needs through organic manuring, pruning, and improved dry land farming practices, etc needs to be imparted. References Antrix Corporation, 2005. Impact of Sujala. First season visible impacts. Power Point Presentation for the midterm review of the Karnataka Watershed Management Project. Monitoring, Evaluation & Learning Unit. Antrix Corporation. ISRO, Bangalore. June 2005. Antrix Corporation, 2006. Common Property Resources Development under Sujala. Monitoring, Evaluation & Learning Unit. Antrix Corporation. Indian Space Research Organization, Bangalore. Dave, S. 2007. Watershed policies and implementation: Experience from Karnataka, India. Karnataka Watershed Development Project. Power Point Presentation, Feb 08, 2007. 53 Toolkit for monitoring and evaluation of AWM Reference Note 7 projects Annotated Bibliography related to M&E of AWM Projects Reference Note 7 Annotated Bibliography related to Monitoring and Evaluation of AWM Projects General Guidance on M&E Bos, M.G., M. Burton, and D.J. Molden. 2005. Irrigation and drainage performance assessment - practical guidelines. CABI publishing, Wallingford, UK This book guides for practitioners in designing and carrying out performance assessment and implement performance-based management. It proposes four categories of performance indicators (water balance, service and maintenance, environment, economics, and emerging indicators, using remote sensing), with detailed description of suggested indicators. Casley, D., and K. Kumar. 1987. Project monitoring and evaluation in agriculture, a joint study: The World Bank, International Fund for Agricultural Development, Food and Agricultural Organization of the United Nations. Baltimore, USA; Johns Hopkins University Press. Monitoring must be integrated within the project management structure. The book describes: - How to design and set up a monitoring system - The monitoring of financial and management progress - Beneficiary contact monitoring - Follow-up diagnostic studies for monitoring - Communicating information - Internal ongoing evaluation and Formal evaluation Guijt, I., and J. Woodhill. 2002. A Guide for Project M&E. Rome, International Fund for Agricultural Development (IFAD). Available at http://www.ifad.org/evaluation/guide/index.htm This extensive guide has been written to help project managers, monitoring and evaluation staff, consultants, and government counterparts improve the quality of M&E in IFAD-supported projects. The guide features a `managing for impact' approach to M&E. It emphasizes use of participatory M&E techniques. Main chapters focus on: - Linking project design, annual planning and M&E - Setting up the M&E system - Deciding what to monitor and evaluate - Gathering, managing and communicating information - Putting in place the necessary capacities and conditions - Reflecting critically to improve action The appendix provide guidance on the M&E matrix, which is used for listing information needs and data collection and analysis methods, and on methods for data collection. The guide also include comprehensive sample terms of reference for key M&E actors. Kusek, J.Z., and R.C. Rist. 2004. Ten Steps to a Results-Based Monitoring and Evaluation System. Washington D.C.: World Bank. Available at http://intresources.worldbank.org/INTINFRES/Resources/2780429- 1173484479651/10StepsToRBM&E_2004.pdf The comprehensive guide for practitioners is based on the `management for results' new paradigm. It describes the essential actions involved in building an M&E system: - Formulate outcomes and goals - Select outcome indicators to monitor - Gather baseline information on the current condition 54 Toolkit for monitoring and evaluation of AWM Reference Note 7 projects Annotated Bibliography related to M&E of AWM Projects Set specific targets to reach and dates for reaching them Regularly collect data to assess whether the targets are being met Analyze and report the results The model emphasizes in particular conducting a readiness assessment and sustaining the M&E system. World Bank Operations Policy and Country Services. Designing M&E for Investment Operations. Washington D.C.: World Bank. Available at http://go.worldbank.org/RLQKPSE1F0 OPCS briefly reviews the steps to build a M&E system: Start with a good M&E design and monitoring system Use the Logical framework approach to develop a good M&E design Emphasize the importance of building base line data Encourage the use participatory M&E Rajalahti, R., J. Woelcke, et al. 2005. Monitoring and Evaluation for World Bank Agricultural Research and Extension Projects: A Good Practice Note. Agricultural and Rural Development Discussion Paper 20. Washington D.C.: World Bank. Available at http://intresources.worldbank.org/INTARD/Resources/M_E_WB_AgExtensionProjects.pdf The note provides a step by step guide for developing a M&E system in Agricultural Research and Extension projects: Challenges for M&E in the sector Results framework requirements M&E arrangements, including the Management Information System, institutional and human capacity requirements, data sources and methodologies for data collection and analysis, and costs. The Appendix lists indicators and data sources. Participatory M&E Self evaluation and independent evaluation of outcomes and impacts Indicators World Bank Independent Evaluation Group. 1995. Performance indicators in Bank financed agricultural products: a first edition note, Volume 1. Working paper. Independent Evaluation Group. Washington D.C.: World Bank. The report briefly explains requirements for project M&E, and then presents seven sets of performance indicators, including: Sector wide Agriculture indicators Irrigation and Drainage indicators, grouped into six performance categories including hydraulic, agricultural, economic and financial, sustainability and social impact. Methods and tools for M&E Casley, D., and K. Kumar. 1988. Collection, Analysis, and Use of Monitoring and Evaluation Data. Washington D.C.: World Bank. The book explains the data collection and analysis techniques referred to in the preceding volume, for agriculture and rural development projects. Each chapter deals extensively with a specific area of data collection, analysis, strengths and limitations, and use: Qualitative interviewing of key informants Group interviews Participant observation Structured surveys Sampling Measurement of crop production and yields Exploratory analysis of data Statistical analysis of data Presenting data to the user 55 Toolkit for monitoring and evaluation of AWM Reference Note 7 projects Annotated Bibliography related to M&E of AWM Projects Participatory Monitoring and Evaluation Guijt, I. 1998. Participatory Monitoring and impact assessment of sustainable agriculture initiatives: an introduction to the keys. Bernents. SARL discussion paper 1. London IIED. The booklet describes an indicator based approach to participatory monitoring. It lists the steps in design: identify participants, objectives, select indicators, methods, and then implement and use the findings. Twenty participatory methods are described. Guijt, I. 1999. Participatory monitoring and evaluation for natural resource management and research. Socio economic Methodologies for Natural Resources Research. Chatham, UK: Natural Resources Institute. Available at http://www.nri.org/publications/bpg/bpg04.pdf The author summarises the trends stimulating interest in more participatory forms of monitoring and evaluation. The main issues with participatory monitoring and evaluation are described, including actual participation, baselines, and the question of rigorous analysis. Van der Schans, M. L., and P. Lemperiere. 2006. Participatory rapid diagnosis and action planning for irrigated agriculture systems. Improving Irrigation Performance in Africa Project. Rome, Food and Agriculture Organization of the United Nations. Available at ftp://ftp.fao.org/agl/iptrid/appia_manual_en.pdf This is a research method for analyzing and improving the performance of an irrigation scheme together with farmers, based on the principles of Rapid Rural Appraisal and Participatory Rural Appraisal. The PRDA method quickly identifies the main constraints of an irrigation scheme managed with farmers, and generates a plan for improvement. The manual sets up guidelines on the application of the method, briefly describes useful participatory tools, and provides prototype reporting sheets. World Bank. 1998. Participation and Social Assessment Tools and Techniques. Washington D.C.: World Bank. Available at http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPMENT/0,,content MDK:20502497~pagePK:148956~piPK:216618~theSitePK:244363,00.html The kit provides information for structuring training sessions and materials that can be directly transferred to transparencies for presentations. It first provides the conceptual overview and establishes the link between social analysis and participation. It then focuses on how to do stakeholder analysis, Participatory Rural Appraisal, and Beneficiary Assessment. It also examines how these methods can be applied to Participatory Monitoring and Evaluation. M&E of poverty reduction impacts ADB/ IWMI. 2005b. Pro Poor Intervention Strategies in Irrigated Agriculture in Asia. Executive summary. Available at http://www.iwmi.cgiar.org/propoor/files/ADB_Project/IWMI ADB%20Final%20 %20sum.pdf The study, based on extensive primary data and a review of global literature from 2001 to 2005, examined the evidence regarding the effects of irrigation in six Asian countries. The report provides a generic framework for understanding and designing pro poor interventions in irrigated agriculture. It shows that irrigation reduces poverty, and that indirect benefits of irrigation at the local and broader economy level can be much larger than the direct crop productivity benefits of irrigation. It recommends that: Interventions in irrigated agriculture are re oriented with a pro poor focus Irrigation agriculture investment packages are made more comprehensive, including addressing issues related to resource distribution and access to support services 56 Toolkit for monitoring and evaluation of AWM Reference Note 7 projects Annotated Bibliography related to M&E of AWM Projects - In designing irrigation interventions, it is important to incorporate poverty alleviation criteria, a generic typology of direct and indirect benefits and disbenefits, and of beneficiaries and affected, and a micro, meso and macro level framework for identifying constraints and opportunities for enhancing benefits of interventions to the poor. Alsop, R and N. Heinsohn. 2005. Measuring Empowerment in Practice: Structuring Analysis and Framing Indicators. World Bank Policy Research Working Paper 3510, February 2005. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTEMPOWERMENT/Resources/41307_wps3510.pdf Empowerment is defined as "the process of enhancing an individual's or group's capacity to make purposive choices and to transform those choices into desired actions and outcomes." The authors propose two frameworks for understanding and assessing empowerment: - indirect measures, based on assets and institutions - direct measures, by looking at existence, use, and achievement of choice Ward, C. 2007. Poverty Analysis in Agricultural Water Operations. Phase 1: Review of World Bank financed projects. Preliminary Results of a Review. Draft, May 19, 2007. A review of poverty analysis in Bank-financed AWM projects was carried out on a sample of projects over April to June 2006. It shows that projects could further streamline poverty reduction concerns. In particular: - The poverty reduction process supported by the project was rarely explicit. Social assessment and social analysis carried out under project preparation addressed some aspects of poverty but often without linking stated social development objectives to poverty reduction. - The analysis of institutional arrangements often did not explicitly consider the poor, despite the strong impact of institutional arrangements on the poor and on mitigation of negative impacts. - Finally, the Results Framework rarely defined poverty targets or intermediate results, and monitoring systems typically were not designed to provide a picture of progress against poverty related targets. World Bank. 2000. Attacking poverty. World Development Report, ISSN 0163-5085 ; 2000/2001. New York : Published for the World Bank, Oxford University Press. Available at http://go.worldbank.org/L8RGH3WLI0 The World Development Report defines poverty as "pronounced deprivation in well- being". It recognizes that poverty has many dimensions, including not only material deprivation, as measured by an appropriate concept of income or consumption, but also low achievements in education and health, vulnerability, exposure to risk, and finally voicelessness and powerlessness. This understanding brings to the fore contributions of different areas of action for the poverty reduction agenda and suggests different measures of poverty reduction. World Bank. 2002. Empowerment and poverty reduction sourcebook. Washington D.C.: World Bank. Available at http://go.worldbank.org/FD9HH8DH11 The Sourcebook identifies empowerment as "the expansion of assets and capabilities of poor people to participate in, negotiate with, influence, control and hold accountable institutions that affect their lives." The four key elements of empowerment that must underlie institutional reform are: - 57 Toolkit for monitoring and evaluation of AWM Reference Note 7 projects Annotated Bibliography related to M&E of AWM Projects - Access to information, - Inclusion/participation, - Accountability, and - Local organizational capacity. World Bank. 2003. A User's Guide to Poverty and Social Impact Analysis. Washington D.C.: World Bank. Available at http://go.worldbank.org/IR9SLBWTQ0 The PSIA analyses ex-ante or ex-post the distributional impact of policy reforms on the well being of different stakeholder groups (both income and non-income dimensions) with a particular focus on the poor and vulnerable. It examines how impacts on the poor are transmitted through five "channels": employment, prices, access to goods and services, assets, and transfers and taxes. The guide lists a number of methods and tools for carrying out PSIA. Impact Evaluations Baker, J. 2000. Evaluating the impact of development projects on poverty: a handbook for practitioners, Volume 1. Washington D.C.: World Bank. Available at http://go.worldbank.org/8E2ZTGBOI0 Chapter 1 presents an overview of concepts and methods for impact evaluation. Chapter 2 discusses key steps and related issues to consider in implementation. Chapter 3 illustrates statistical analytical techniques for an household survey-based set of data. Chapter 4 includes a discussion of lessons learned from a rich set of "good practice" evaluations of poverty projects that have been reviewed. The Annex describes fifteen different case studies of impact evaluation, by detailing research questions addressed and evaluation design, data collection procedures, econometric techniques utilised, responsibilities for the evaluation, and lessons learned. The Annex also gives sample terms of reference and sample budgets. Janaiah, A., and Mekong Economics Ltd. 2004. The Poverty Impacts of Public Irrigation Expenditures in Vietnam. Report under an Asian Development Bank and World Bank- funded Research Project. Hanoi. May 2004. The impact evaluation was primarily aimed at generating empirical evidence on how various policy interventions in irrigation (rehabilitation, management improvement, or both) had had an impact in the three selected irrigation schemes in Vietnam at a micro, household, level. It made use of both qualitative and quantitative methods of data collection. Ravallion, M. 2000. The Mystery of the Vanishing Benefits: An Introduction to Impact Evaluation. World Bank Economic Review, Vol.15, No. 1, pp: 115-140. Washington D.C.: World Bank. Available at http://povlibrary.worldbank.org/library/view/4642/ The article provides an introduction to the concepts and methods of impact evaluation. The author provides an explanation in the context of a fictional and concrete application. In the process of figuring out how to evaluate a human resource program targeted to the poor, an analyst from the Ministry of Finance in the fictional country learns the strengths and weaknesses of the main methods of ex post impact evaluation. World Bank. 2006b. Impact Evaluation and the Project Cycle. Doing Impact Evaluation Series No. 1. Thematic group on poverty analysis, monitoring and impact evaluation. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTISPMA/Resources/383704- 1146752240884/doing_ie_series_01.pdf 58 Toolkit for monitoring and evaluation of AWM Reference Note 7 projects Annotated Bibliography related to M&E of AWM Projects An Impact Evaluation Working Paper Series has been established in recognition of the importance of impact evaluation studies for World Bank operations and for development in general. The first issue describes impact evaluation at the three main stages of the project cycle ­ preparation, appraisal and implementation. World Bank Operations Policy and Country Services. 2005a. Implementing Impact Evaluations at the World Bank: Guidance Note. Washington D.C.: World Bank. Available at http://intresources.worldbank.org/INTRESULTS/PoliciesandGuidance/20899741/IEGuidanceNote FINALApril06.pdf The brief guidelines emphasize use of a logical framework, performance indicators, and Participatory M&E. AWM operations in the Bank World Bank Independent Evaluation Group. 2006. Water Management in Agriculture: Ten Years of World Bank Assistance, 1994 2004. Washington D.C.: World Bank. Available at http://lnweb18.worldbank.org/oed/oeddoclib.nsf/DocUNIDViewForJavaSearch/116A7B237B9C05 C285257280006133D7/$file/water_manangement_agriculture.pdf The report looks at ten year investment in the sector. It assessed in particular M&E systems in AWM projects, based on a sample review of 80 projects, out of 161 with quantifiable irrigation and drainage component over the period. Overall quality of M&E design improved in the late 1990s with the introduction of logical frameworks and their mandatory use in Project Appraisal Documents. Even so, M&E of outcomes and impacts is not adequately dealt with. Less than half of the projects established a baseline, sometimes late during project implementation, and only 10% had a rigorous evaluative framework. Much greater attention is needed to establish indicators and evaluative frameworks to unambiguously determine and attribute the development impacts of Bank lending in the sector. World Bank. 2006a. Directions in Development: Reengaging in Agricultural Water Management. Challenges and Options. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTARD/Resources/DID_AWM.pdf The report gives strategic focus to the implementation of various AWM interventions. It sets up the changing context of demand and supply for agricultural water, identifies the policy, institutional, and incentive reform options that will accelerate productivity improvements and pro poor growth, and articulates priorities for investment in AWM. Useful websites World Bank Social Development Department. Participation and Civic Engagement: http://go.worldbank.org/FMRAMWVYV0 Information is organized in the following categories: Social Accountability Enabling Environment for Civic Engagement Participatory Monitoring and Evaluation Participation at the Project, Program, and Policy Level World Bank Thematic group on poverty analysis, monitoring and impact evaluation. Impact Evaluation: http://go.worldbank.org/5FDCUE6PC0 Information is organized in the following categories: Overview Methods and Techniques Collection of selected impact evaluations organized by sector and by country. Range of datasets and guide to data instruments and surveys. Training Events and Materials Key Readings 59 Toolkit for monitoring and evaluation of AWM Reference Note 7 projects Annotated Bibliography related to M&E of AWM Projects World Bank Thematic group on poverty analysis, monitoring and impact evaluation. Poverty Monitoring: http://go.worldbank.org/UN7VBV2VW0 The website provides a list of key readings on: Selecting Indicators and Setting Targets Poverty Monitoring Systems Statistical Systems 60 Toolkit for monitoring and evaluation of AWM Reference Note 8 projects Glossary Reference Note 8 Glossary of terms for monitoring and evaluation of AWM Projects Source: OECD Glossary of Key Terms in Evaluation and Results-Based Management (2002) Activity: Actions taken or work performed through which inputs, such as funds, technical assistance, and other type of resources, are mobilized to produce specific outputs. Appraisal: An overall assessment of the relevance, feasibility and potential sustainability of a development intervention prior to a decision of funding. Note: In development agencies, banks, etc., the purpose of appraisal is to enable decisionmakers to decide whether the activity represents an appropriate use of corporate resources. Related term: ex ante evaluation. Attribution: The ascription of a causal link between observed (or expected to be observed) changes and a specific intervention. Note: Attribution refers to that which is to be credited for the observed changes or results achieved. It represents the extent to which observed development effects can be attributed to a specific intervention or to the performance of one or more partner taking account of other interventions, (anticipated or unanticipated) confounding factors, or external shocks. Base line study: An analysis describing the situation prior to a development intervention, against which progress can be assessed or comparisons made. Benchmark: Reference point or standard against which performance or achievements can be assessed. Note: A benchmark refers to the performance that has been achieved in the recent past by other comparable organizations, or what can be reasonably inferred to have been achieved in the circumstances. Beneficiaries: The individuals, groups, or organizations, whether targeted or not, that benefit, directly or indirectly, from the development intervention. Related terms: target groups. Cluster evaluation: An evaluation of a set of related activities, projects and/or programs. Counterfactual: The situation or condition which hypothetically may prevail for individuals, organizations, or groups where there is no development intervention. Country Program Evaluation/ Country Assistance Evaluation: Evaluation of one or more donor's or agency's portfolio of development interventions, and the assistance strategy behind them, in a partner country. 61 Toolkit for monitoring and evaluation of AWM Reference Note 8 projects Glossary Data collection tools: Methodologies used to identify information sources and collect information during an evaluation. Note: Examples are informal and formal surveys, direct and participatory observation, community interviews, focus groups, expert opinion, case studies, literature search. Development objective: Intended impact contributing to physical, financial, institutional, social, environmental, or other benefits to a society, community, or group of people via one or more development interventions. Effectiveness: The extent to which the development intervention's objectives were achieved, or are expected to be achieved, taking into account their relative importance. Note: Also used as an aggregate measure of (or judgment about) the merit or worth of an activity, i.e., the extent to which an intervention has attained, or is expected to attain, its major relevant objectives efficiently in a sustainable fashion and with a positive institutional development impact. Efficiency: A measure of how economically resources/ inputs (funds, expertise, time, etc.) are converted to results. Evaluation: The systematic and objective assessment of an on going or completed project, program or policy, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision making process of both recipients and donors. Evaluation also refers to the process of determining the worth or significance of an activity, policy or program. An assessment, as systematic and objective as possible, of a planned, on going, or completed development intervention. Note: Evaluation in some instances involves the definition of appropriate standards, the examination of performance against those standards, an assessment of actual and expected results and the identification of relevant lessons. Related term: review. Ex ante evaluation: An evaluation that is performed before implementation of a development intervention. Related terms: appraisal. Ex post evaluation: Evaluation of a development intervention after it has been completed. Note: It may be undertaken directly after or long after completion. The intention is to identify the factors of success or failure, to assess the sustainability of results and impacts, and to draw conclusions that may inform other interventions. External evaluation: The evaluation of a development intervention conducted by entities and/or individuals outside the donor and implementing organizations. Feedback: The transmission of findings generated through the evaluation process to parties for whom it is relevant and useful so as to facilitate learning. This may involve the collection and dissemination of findings, conclusions, recommendations and lessons from experience. 62 Toolkit for monitoring and evaluation of AWM Reference Note 8 projects Glossary Impacts: Positive and negative, primary and secondary, long term effects produced by a development intervention, directly or indirectly, intended or unintended. Independent evaluation: An evaluation carried out by entities and persons free of the control of those responsible for design and implementation of the development intervention. Note: The credibility of an evaluation depends in part on how independently it has been carried out. Independence implies freedom from political influence and organizational pressure. It is characterized by full access to information and by full autonomy in carrying out investigations and reporting findings. Indicator: Quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, to reflect the changes connected to an intervention, or to help assess the performance of a development actor. Inputs: The financial, human, and material resources used for the development intervention. Institutional Development Impact: The extent to which an intervention improves or weakens the ability of a country or region to make more efficient, equitable, and sustainable use of its human, financial, and natural resources, for example through: (a) better definition, stability, transparency, enforceability and predictability of institutional arrangements and/or (b) better alignment of the mission and capacity of an organization with its mandate, which derives from these institutional arrangements. Such impacts can include intended and unintended effects of an action. Joint evaluation: An evaluation to which different donor agencies and/or partners participate. Note: There are various degrees of "jointness" depending on the extent to which individual partners cooperate in the evaluation process, merge their evaluation resources and combine their evaluation reporting. Joint evaluations can help overcome attribution problems in assessing the effectiveness of programs and strategies, the complementarity of efforts supported by different partners, the quality of aid coordination, etc. Logical framework (Logframe): Management tool used to improve the design of interventions, most often at the project level. It involves identifying strategic elements (inputs, outputs, outcomes, impact) and their causal relationships, indicators, and the assumptions or risks that may influence success and failure. It thus facilitates planning, execution and evaluation of a development intervention. Related term: results based management. Meta evaluation: The term is used for evaluations designed to aggregate findings from a series of evaluations. It can also be used to denote the evaluation of an evaluation to judge its quality and/or assess the performance of the evaluators. Mid term evaluation: Evaluation performed toward the middle of the period of implementation of the intervention. Monitoring: A continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications 63 Toolkit for monitoring and evaluation of AWM Reference Note 8 projects Glossary of the extent of progress and achievement of objectives and progress in the use of allocated funds. Related term: indicator. Outcome: The likely or achieved short term and medium term effects of an intervention's outputs. Related terms: outputs, impacts. Outputs: The products, capital goods and services that result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of outcomes. Participatory evaluation: Evaluation method in which representatives of agencies and stakeholders (including beneficiaries) work together in designing, carrying out and interpreting an evaluation. Performance: The degree to which a development intervention or a development partner operates according to specific criteria/standards/ guidelines or achieves results in accordance with stated goals or plans. Program evaluation: Evaluation of a set of interventions, marshaled to attain specific global, regional, country, or sector development objectives. Note: a development program is a time bound intervention involving multiple activities that may cut across sectors, themes and/or geographic areas. Related term: Country program/strategy evaluation. Project evaluation: Evaluation of an individual development intervention designed to achieve specific objectives within specified resources and implementation schedules, often within the framework of a broader program. Note: Cost benefit analysis is a major instrument of project evaluation for projects with measurable benefits. When benefits cannot be quantified, cost effectiveness is a suitable approach. Project or program objective: The intended physical, financial, institutional, social, environmental, or other development results to which a project or program is expected to contribute. Results Based Management: A management strategy focusing on performance and achievement of outputs, outcomes and impacts. Related term: logical framework. Reliability Consistency or dependability of data and evaluation judgements, with reference to the quality of the instruments, procedures and analyses used to collect and interpret evaluation data. Results The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention. Related terms : outcome, impacts. 64 Toolkit for monitoring and evaluation of AWM Reference Note 8 projects Glossary Results Chain The causal sequence for a development intervention that stipulates the necessary sequence to achieve desired objectivesbeginning with inputs, moving through activities and outputs, and culminating in outcomes, impacts, and feedback. In some agencies, reach is part of the results chain. Review: An assessment of the performance of an intervention, periodically or on an ad hoc basis. Note: Frequently "evaluation" is used for a more comprehensive and/or more in depth assessment than "review." Reviews tend to emphasize operational aspects. Sometimes the terms "review" and "evaluation" are used as synonyms. Related term: evaluation. Self evaluation: An evaluation by those who are entrusted with the design and delivery of a development intervention. Stakeholders: Agencies, organizations, groups or individuals who have a direct or indirect interest in the development intervention or its evaluation. Sustainability: The continuation of benefits from a development intervention after major development assistance has been completed. The probability of continued long term benefits. The resilience to risk of the net benefit flows over time. Target group: The specific individuals or organizations for whose benefit the development intervention is undertaken. Terms of reference: Written document presenting the purpose and scope of the evaluation, the methods to be used, the standard against which performance is to be assessed or analyses are to be conducted, the resources and time allocated, and reporting requirements. Two other expressions sometimes used with the same meaning are "scope of work" and "evaluation mandate." Triangulation: The use of three or more theories, sources or types of information, or types of analysis to verify and substantiate an assessment. Note: by combining multiple data sources, methods, analyses or theories, evaluators seek to overcome the bias that comes from single informants, single methods, single observer or single theory studies. Validity: The extent to which the data collection strategies and instruments measure what they purport to measure. 65 Toolkit for monitoring and evaluation of AWM References projects REFERE CES Alexander, P.J., and M.O. Potter. 2004. Benchmarking of Australian irrigation water provider businesses. In Special Issue: Benchmarking in the Irrigation and Drainage Sector. Irrigation and Drainage 53 (2). New York: Wiley, June 2004. Alsop, R and N. Heinsohn. 2005. Measuring Empowerment in Practice: Structuring Analysis and Framing Indicators. World Bank Policy Research Working Paper 3510, February 2005. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTEMPOWERMENT/Resources/41307_wps3510.pdf ANCID. 2000. 1998/99. Australian Irrigation Water Provider ­ Benchmarking Report. Victoria, Australia. 68 pages. Asian Development Bank. 1991. Remote sensing and geographic information system for natural resource management. Environment Paper 9. Manila: Asian Development Bank. Baker, J. 2000. Evaluating the impact of development projects on poverty: a handbook for practitioners, Volume 1. Washington D.C.: World Bank. Available at http://go.worldbank.org/8E2ZTGBOI0 Bastiaanssen, W. G. M., T. Van Der Wal, and T.N.M. Visser. 1996. Diagnosis of regional evaporation by remote sensing to support irrigation performance assessment. Irrigation and Drainage Systems, 10(1):1 23. Bastiaanssen, W. G. M., H. Pelgrum, J. Wang, Y. Ma, J.F. Moreno, G.J. Roerink, and T. Van Der Wal. 1998. A remote sensing surface energy balance algorithm for land (SEBAL): 2. Validation. Journal of Hydrology, 212/213:213 229. Bastiaanssen, W. G. M., M. Menenti, R.A. Feddes, and A.A.M Holstag. 1998. A remote sensing surface energy balance algorithm for land (SEBAL): 1. Formulation. Journal of Hydrology, 212/213:198 212. Bastiaanssen, W. G. M., S. Thiruvengadachari, R. Sakthivadivel, and D.J. Molden. 1999. Satellite remote sensing for estimating productivities of land and water. International Journal of Water Resources Development, 15(1/2):181 196. Bastiaanssen, W. G. M. 1999. Monitoring water resources and agro ecosystem productivity from space. International Archives of Photogrammetry and Remote Sensing, 32(7C2):38 43. Bastiaanssen, W. G. M., D.J. Molden, S. Thiruvengadachari, A.A.M.F.R. Smit, L. Mutuwatte, and G. Jayasinghe. 1999. Remote sensing and hydrologic models for performance assessment in Sirsa Irrigation Circle, India. IWMI research report 27, Colombo, Sri Lanka: IWMI. v, 29p. Bastiaanssen, W. G. M., D.J. Molden, and I.W. Makin. 2000. Remote sensing for irrigated agriculture: Examples from research and possible applications. Agricultural Water Management, 46(2):137 155. Bastiaanssen, W. G. M. 2000. Satellite observations of international river basins for all. International Archives of Photogrammetry and Remote Sensing, 33(B6):319 331. Bastiaanssen, W. G. M. 2000. SEBAL based sensible and latent heat fluxes in the irrigated Gedez Basin, Turkey. Journal of Hydrology, 229(1 2):87 100. 66 Toolkit for monitoring and evaluation of AWM References projects Bastiaanssen, W. G. M., R.A.L. Brito, M.G. Bos, R. Souza, E.B. Cavalcanti, and M.M. Bakker. 2001. Low cost satellite data applied to performance monitoring of the Nilo Coelho irrigation scheme, Brazil. Irrigation and Drainage Systems, 15, 53 79, Kluwer Academic Publishers, Dordrecht, The Netherlands. Bos, M.G., M. Burton, and D.J. Molden. 2005. Irrigation and drainage performance assessment practical guidelines. CABI publishing, Wallingford, UK. Burt, C.M. and S.W. Styles. 1999. Modern water control and management practices in Irrigation: Impact on performance. Water Report No. 19, Food and Agriculture Organisation of the UN, Rome. Burt, C.M. and S.W.Styles. 2004. Conceptualizing irrigation project modernization through benchmarking and the rapid appraisal process. In Special Issue: Benchmarking in the Irrigation and Drainage Sector. Irrigation and Drainage 53 (2). New York: Wiley, June 2004. Burton, M., H. Malano, and I. Makin. 2005. Benchmarking for improved performance in irrigation and drainage. In Shaping the Future of Water for Agriculture: A Sourcebook for Investment in Agricultural Water Management. Washington D.C.: World Bank. Casley, D., and K. Kumar. 1987. Project monitoring and evaluation in agriculture, a joint study: The World Bank, International Fund for Agricultural Development, Food and Agricultural Organization of the United Nations. Baltimore, USA; Johns Hopkins University Press. Casley, D., and K. Kumar. 1988. Collection, Analysis, and Use of Monitoring and Evaluation Data. Washington D.C.: World Bank. De Soto, H., L. Simonovska, M. Mladenovski, G. Iliev, P. Saiti, and P. Nikolov. 2006. Social Impact Assessment, The Irrigation Rehabilitation and Restructuring Project, Macedonia. Droogers, P., and W. G. M. Bastiaanssen. 2000. Evaporation estimates using a combined hydrological model and remote sensing approach. Geophysical Research Abstracts, Vol.2: Droogers, P., W. G. M. Bastiaanssen, M. Bayazgul, Y. Kayam, G.W. Kite, and H. Murray Rust. 2000. Distributed agro hydrological modeling of an irrigation system in Western Turkey. Agricultural Water Management, 43(2):183 202. Engelhardt Wendt, E. 2003. Guideline for Introductory Workshops of Participatory Impact Monitoring. Internationale Weiterbilding und Entwicklung and Deutscher Entwicklungdiesnt. Available at http://zambia.ded.de/cipp/ded/lib/pub/object/downloadfile,lang,2/oid,2749/ticket,g_u_e_s_t/~/Parti cpatory_Impact_Monitoring.pdf Estrella, M. 2000. Learning from change Issues and Experiences in Participatory Monitoring and Evaluation. Edited by Marisol Estrella, with Jutta Blauert, Dindo Campilan, John Gaventa, Julian Gonsalves, Irene Guijt, Deb Johnson, and Roger Ricafort. IDRC/ITDG Publishing 2000. Available at http://siteresources.worldbank.org/INTPCENG/1143331 1116505657479/20509240/learnfromchange.pdf European Commission. 2004. Project Cycle Management Guidelines, Vol.1. European Commission, Brussels, March. Available at http://ec.europa.eu/comm/europeaid/qsm/documents/pcm_manual_2004_en.pdf 67 Toolkit for monitoring and evaluation of AWM References projects Fabbri, P.J., B.G.H. Gorte, J.W. Van Groenigen, F.D. Van der Meer, and A. Saldana. 1998. Integrating spatial statistics and remote sensing. International Journal of Remote Sensing, 19(9):1793 1814. GOM, 2003. Benchmarking of irrigation projects in Maharashtra, 2001 2002. Irrigation Department, Government of Maharashtra, India. Guijt, I. 1998. Participatory Monitoring and impact assessment of sustainable agriculture initiatives: an introduction to the keys. Bernents. SARL discussion paper 1. London IIED. Guijt, I. 1999. Participatory monitoring and evaluation for natural resource management and research. Socio economic Methodologies for Natural Resources Research. Chatham, UK: Natural Resources Institute. Available at http://www.nri.org/publications/bpg/bpg04.pdf Guijt, I., and J. Woodhill. 2002. A Guide for Project M&E. Rome, International Fund for Agricultural Development (IFAD). Available at http://www.ifad.org/evaluation/guide/index.htm Halcrow, 2001. Report on the 2001 Improved Levels of Service Programme, Albania: Second Irrigation and Drainage Rehabilitation Project, Project Management Unit of the Ministry of Agriculture and Food, Tirana. Halcrow. 2003. Water Users Association Survey 2003. Second Irrigation and Drainage Rehabilitation Project, Albania. Halcrow Water, Burderop Park, Swindon, United Kingdom. Hector, M., and M. Burton. 2001. Guidelines for benchmarking performance in the irrigation and drainage sector. IPTRID Secretariat, Food and Agriculture Organisation of the UN, Rome. Horton, D., et al. 2003. Evaluating capacity development: experiences from research and development organizations around the world. The Netherlands: International Service for National Agricultural Research (ISNAR); Canada: International Development Research Centre (IDRC), the Netherlands: ACP EU Technical Centre for Agricultural and Rural Cooperation (CTA). Available at http://www.isnar.cgiar.org/publications/ecd book.htm Hurk, van den B. J. J. M., W. G. M. Bastiaanssen, H. Pelgrum, and E. Van Maijgaard. 1997. A new methodology for assimilation of initial soil moisture fields in weather prediction models using meteostat and NOAA data. Journal of Applied Meteorology, September:1271 1283. Hussain, H. 2005. Pro Poor Intervention Strategies in Irrigated Agriculture in Asia. Executive summary. Colombo, Sri Lanka: International Water Management Institute. Available at http://www.iwmi.cgiar.org/propoor/files/ADB_Project/IWMI ADB%20Final%20 %20sum.pdf Janaiah, A., and Mekong Economics Ltd. 2004. The Poverty Impacts of Public Irrigation Expenditures in Vietnam. Report under an Asian Development Bank and World Bank funded Research Project. Hanoi. May 2004. Jayatillake, H.M. 2004. Application of performance assessment and benchmarking tool to help improve irrigation system performance in Sri Lanka. In Special Issue: Benchmarking in the Irrigation and Drainage Sector. Irrigation and Drainage 53 (2). New York: Wiley, June 2004. Kite, G., and P. Droogers. 2000. Comparing estimates of actual evapotranspiration from satellites, hydrological models, and field data: A case study from Western Turkey. IWMI research report 42, Colombo, Sri Lanka: IWMI. v, 32p. Kusek, J.Z. and R.C. Rist. 2004. Ten Steps to a Results Based Monitoring and Evaluation System. . Washington D.C.: World Bank. Available at http://go.worldbank.org/7H1YFQS5Z0 68 Toolkit for monitoring and evaluation of AWM References projects Lock, D., 1993. Project management. Fifth edition. Gower Publishing, Aldershot, UK. Malano, H., and M. Burton. 2001. Guidelines for benchmarking performance in the irrigation and drainage sector. International Programme for Technology and Research in Irrigation and Drainage Secretariat, Food and Agriculture Organisation of the UN, Rome. Malano, H., M. Burton, and I. Makin. 2004. Benchmarking performance in the irrigation and drainage sector: A tool for change. In Special Issue: Benchmarking in the Irrigation and Drainage Sector. Irrigation and Drainage 53 (2). New York: Wiley, June 2004. Narayan, D., 1993. Participatory Evaluation: Tools for Managing Change in Water and Sanitation. World Bank Technical Paper no. 207. Washington D.C.: World Bank. Available at http://poverty2.forumone.com/files/14789_Participatory_Evaluation.pdf OECD (Organisation for Economic Co operation and Development). 2002. Glossary of Key Terms in Evaluation and Results Based Management. Available at http://www.oecd.org/dataoecd/29/21/2754804.pdf Osborne, D., and T. Gaebler. 1992. Reinventing Government: How the Entrepreneurial Spirit Is Transforming the Public Sector. Reading, Mass.: Addison Wesley. Ostrom, E. 1992. Crafting institutions for self governing irrigation systems. ICS Press, Institute of Contemporary Studies, San Francisco, California Rajalahti, R., J. Woelcke, et al. 2005. Monitoring and Evaluation for World Bank Agricultural Research and Extension Projects: A Good Practice Note. Agricultural and Rural Development Discussion Paper 20. Washington D.C.: World Bank. Available at http://intresources.worldbank.org/INTARD/Resources/M_E_WB_AgExtensionProjects.pdf Ravallion, M. 2000. The Mystery of the Vanishing Benefits: An Introduction to Impact Evaluation. World Bank Economic Review, Vol.15, No. 1, pp: 115 140. Washington D.C.: World Bank. Available at http://povlibrary.worldbank.org/library/view/4642/ Roerink, G. J., W. G. M. Bastiaanssen, J. Chambouleyron, and M. Menenti. 1997. Relating crop water consumption to irrigation water supply by remote sensing. Water Resources Management, 11(6):445 465. Sakthivadivel, R., U.A. Amerasinghe, and S. Thiruvengadachari. 2001. Using remote sensing techniques to evaluate lining efficacy of watercourses. IWMI research report 46, Colombo, Sri Lanka: IWMI. v, 29p. Sakthivadivel, R., S. Thiruvengadachari, U.A. Amerasinghe, W. G. M. Bastiaanssen, and D. Molden. 1999. Performance evaluation of the Bhakra Irrigation System, India, using remote sensing and GIS techniques. IWMI research report 28, Colombo, Sri Lanka: IWMI. v, 22p. Schakel, J. K., and W. G. M. Bastiaanssen. 1997. Regional water and salt balances obtained from GIS and hydrological models. ITIS, 4(1):15 17. Sirker, K., and T. Shah. 2003. Community Based Monitoring & Evaluation System Methodology Design Workshop (June 11­14, 2002) and Follow up (January ­ March 2003) Report. Draft, August 23, 2004. World Bank Institute's Community Empowerment and Social Inclusion Learning Program. Available at http://info.worldbank.org/etools/library/latestversion.asp?49429 69 Toolkit for monitoring and evaluation of AWM References projects Small, L.E. and M. Svendsen. 1992. A framework for assessing irrigation performance. IFPRI Working Papers on Irrigation Performance No.1. Washington D.C.: International Food Policy Research Institute, August. Smith, L.E.D. 2004, Assessment of the Contribution of Irrigation to Poverty Reduction and Sustainable Livelihoods, International Journal of Water Resources Development, 20,2, pp243 257. Sodal, S.V. 2004. Present practices and opportunities for performance improvement of irrigation schemes in Maharashtra State of India (POPIIS). In Special Issue: Benchmarking in the Irrigation and Drainage Sector. Irrigation and Drainage 53 (2). New York: Wiley, June 2004 Sur, M. 2006. Monitoring and evaluating the poverty impacts of agricultural water investments. Agricultural and Rural Development Notes: Issue 15, June 2006. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTARD/Resources/Note15_ME_poverty.pdf Thiruvengadachari, S., and R.S. Sakthivadivel. 1997. Satellite remote sensing for assessment of irrigation system performance: A case study in India. IWMI Research Report 9, Colombo, Sri Lanka: IIMI. v, 23p. Van der Schans, M. L., and P. Lemperiere. 2006. Participatory rapid diagnosis and action planning for irrigated agriculture systems. Improving Irrigation Performance in Africa Project. Rome, Food and Agriculture Organization of the United Nations. Available at ftp://ftp.fao.org/agl/iptrid/appia_manual_en.pdf Wang, J., W. G. M. Bastiaanssen, Y. Ma, and H. Pelgrum. 1998. Aggregation of land surface parameters in the oasis desert systems of north west China. Hydrological Processes, 12:2133 2147. Ward, C. 2007. Poverty Analysis in Agricultural Water Operations. Phase 1: Review of World Bank financed projects. Preliminary Results of a Review. Draft, May 19, 2007. Wiggins, S. and D. Shields. 1995. Clarifying the `Logical Framework' as a Tool for Planning and Managing Development Projects, Project Appraisal, 10, 1. World Bank Business Warehouse. Internal management information system of the World Bank. Washington D.C.: World Bank. World Bank. 1993. Poverty reduction handbook, Volume 1. Washington D.C.: World Bank. World Bank. 1996. The World Bank Participation Sourcebook. Washington D.C.: World Bank. Available at http://www.worldbank.org/wbi/sourcebook/sbhome.htm World Bank. 1998. Participation and Social Assessment Tools and Techniques. Washington D.C.: World Bank. Available at http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPMENT/0,,content MDK:20502497~pagePK:148956~piPK:216618~theSitePK:244363,00.html World Bank. 2000. Attacking poverty. World Development Report 2000/2001, ISSN 0163 5085 ; 2000/2001. New York : Published for the World Bank, Oxford University Press. Available at http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTPOVERTY/0,,contentMDK:20195989 ~pagePK:148956~piPK:216618~theSitePK:336992,00.html World Bank. 2002. Empowerment and poverty reduction sourcebook. Washington D.C.: World Bank. Available at 70 Toolkit for monitoring and evaluation of AWM References projects http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTPOVERTY/EXTEMPOWERMENT/0, ,contentMDK:20260036~menuPK:543261~pagePK:148956~piPK:216618~theSitePK:486411,00. html World Bank. 2003. A User's Guide to Poverty and Social Impact Analysis. Washington D.C.: World Bank. Available at http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTPOVERTY/EXTPSIA/0,,contentMDK :20454976~menuPK:1107972~pagePK:148956~piPK:216618~theSitePK:490130,00.html World Bank. 2004a. Guidance Note on Poverty Assessments. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTPOVERTY/Resources/335642 1098192957114/op1_pa_guidance.pdf World Bank. 2004b. Maharashtra Water Sector Improvement Project, India. Draft Terms of Reference for Base Line Survey. Draft, July 2, 2004. World Bank. 2005a. Applying a capacity results framework in LAO PDR and other pilot countries. Capacity Development Briefs, Number 14. World Bank. 2005b. Benchmarking Initiative in the Irrigation and Drainage Sector, Egypt: Final report. Washington D.C.: World Bank, December 2005. World Bank. 2005c. Community Driven Development and Social Capital: Designing a baseline survey in the Philippines. Social Development Department Working Paper. Washington D.C.: World Bank. Available at http://www wds.worldbank.org/servlet/WDSContentServer/WDSP/IB/2005/07/28/000160016_200507281547 34/Rendered/PDF/32405a10PH0whi10may20final01public1.pdf World Bank. 2005d. Guidelines for Benchmarking Performance in the Irrigation and Drainage Sector in Egypt. Washington D.C.: World Bank, December 2005. World Bank. 2005e. Maharashtra Water Sector Improvement Project, India. Draft Terms of Reference for Monitoring and Evaluation of project. Draft, December 27, 2005. World Bank. 2005f. Maharashtra Water Sector Improvement Project, India. Project Appraisal Document. World Bank. 2005g. Kecamatan Development Project 3B, Indonesia. Project Appraisal Document. World Bank. 2006a. Directions in Development: Reengaging in Agricultural Water Management. Challenges and Options. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTARD/Resources/DID_AWM.pdf World Bank. 2006b. Impact Evaluation and the Project Cycle. Doing Impact Evaluation Series No. 1. Thematic group on poverty analysis, monitoring and impact evaluation. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTISPMA/Resources/383704 1146752240884/doing_ie_series_01.pdf World Bank. 2006c. Mission Report: Irrigation Engineer. IDA 12th Review Mission, On Farm Irrigation Project, Kyrgyz Republic, July World Bank. 2007a. Agriculture for Development. World Development Report 2008. Washington D.C.: World Bank. Available at 71 Toolkit for monitoring and evaluation of AWM References projects http://econ.worldbank.org/WBSITE/EXTERNAL/EXTDEC/EXTRESEARCH/EXTWDRS/EXTWDR 2008/0,,menuPK:2795178~pagePK:64167702~piPK:64167676~theSitePK:2795143,00.html World Bank. 2007b. Water Supply and Sanitation Monitoring and Evaluation Toolkit. Draft, October 7, 2007. World Bank Independent Evaluation Group. 1995. Performance indicators in Bank financed agricultural products: a first edition note, Volume 1. Working paper. Independent Evaluation Group. Washington D.C.: World Bank. World Bank Independent Evaluation Group. 2006. Water Management in Agriculture: Ten Years of World Bank Assistance, 1994 2004. Washington D.C.: World Bank. Available at http://lnweb18.worldbank.org/oed/oeddoclib.nsf/DocUNIDViewForJavaSearch/116A7B237B9C05 C285257280006133D7/$file/water_manangement_agriculture.pdf World Bank Operations Policy and Country Services. Designing M&E for Investment Operations. Washington D.C.: World Bank. Available at http://go.worldbank.org/RLQKPSE1F0 World Bank Operations Policy and Country Services. The World Bank Operational Manual. Washington D.C.: World Bank. Available at http://wbln0018.worldbank.org/institutional/manuals/opmanual.nsf/05TOCpages/The%20World% 20Bank%20Operational%20Manual?OpenDocument World Bank Operations Policy and Country Services. 2004. PAD template and guidelines. Washington D.C.: World Bank. Available at http://siteresources.worldbank.org/INTINVLENDING/Resources/PADGuide.pdf World Bank Operations Policy and Country Services. 2005a. Implementing Impact Evaluations at the World Bank: Guidance Note. Washington D.C.: World Bank. Available at http://intresources.worldbank.org/INTRESULTS/PoliciesandGuidance/20899741/IEGuidanceNote FINALApril06.pdf World Bank Operations Policy and Country Services. 2005b. ISR guidelines. Washington D.C.: World Bank. Available at http://go.worldbank.org/GTP7W9DOK0 World Bank Operations Policy and Country Services. 2007. Implementation Completion and Results Report Guidelines. Washington D.C.: World Bank. Available at http://intresources.worldbank.org/INTINVLENDING/Resources/ICR/ICRGuideFINAL_RevFeb9_2 007.pdf World Bank South Asia Rural Development Unit. 2004. Monitoring and Evaluation of Water and Irrigation Projects ­ Prototype Baseline survey in South Asia. Washington D.C.: World Bank. Available at http://wbln0023.worldbank.org/Internal/SAR/southasiasectors int.nsf/41ByDocName/MEWaterandIrrigation Useful websites ADB/ IWMI. 2005a. Pro Poor Intervention Strategies in Irrigated Agriculture in Asia. Available at http://www.iwmi.cgiar.org/propoor/index.asp?nc=3899&id=1061&msid=235 ESRI support center: http://support.esri.com/ World Bank. Development Data Platform: http://go.worldbank.org/DNBRRS9TB0 World Bank Development Impact Evaluation Initiative: http://go.worldbank.org/HIYKB2QV00 72 Toolkit for monitoring and evaluation of AWM References projects World Bank GIS: http://go.worldbank.org/I21YX64I50 World Bank. Project cycle: http://go.worldbank.org/GI967K75D0 World Bank Institute: www.worldbank.org/wbi World Bank Operations Policy and Country Services: http://go.worldbank.org/68SVPDDGV0 World Bank Operations Policy and Country Services. Policies and guidance for Investment Lending: http://go.worldbank.org/PPQEM510J2 World Bank Poverty Reduction Group. Tools and methods for Poverty and Social Impact Assessment: http://lnweb18.worldbank.org/ESSD/sdvext.nsf/81ByDocName/ToolsandMethods World Bank Quality Assurance Group: http://go.worldbank.org/J4OO0PFYM0 World Bank Social Development Department. Community Driven Development: http://go.worldbank.org/24K8IHVVS0 World Bank Social Development Department. Monitoring and Evaluation of Community Driven Development: http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPMENT/EXTCDD/ 0,,contentMDK:20598866~menuPK:608227~pagePK:148956~piPK:216618~theSitePK:430161,0 0.html World Bank Social Development Department. Participation and Civic Engagement: http://go.worldbank.org/FMRAMWVYV0 World Bank Sustainable Development Network Results Website: http://go.worldbank.org/ZG0PRX9260 World Bank Thematic group on poverty analysis, monitoring and impact evaluation. Impact Evaluation: http://go.worldbank.org/5FDCUE6PC0 World Bank Thematic group on poverty analysis, monitoring and impact evaluation. Poverty Monitoring: http://go.worldbank.org/UN7VBV2VW0 73