80704 Reviewing Project Results Retrospectively Using a Results-Focused Approach to Capacity Development World Bank Institute Capacity Development and Results June 2011 [1 Copyright © June 2011 The World Bank 1818 H Street, N.W. Washington, D.C. 20433, USA All rights reserved This paper was prepared by Dawn Roberts, Cristina Ling and Natalia Agapitova. Samuel Otoo provided overall guidance. Design: Sharon Fisher, World Bank Institute About the World Bank Institute (WBI) WBI’s mission is to be a global facilitator of capacity development for poverty reduction, helping leaders, institutions, and coalitions address their capacity constraints to achieving development results. For more information, visit www.worldbank.org/wbi. About the WBI Capacity Development and Results Practice (WBICR) WBICR provides advisory services, knowledge products, action research and knowledge sharing that inform and support the design and management of results-focused capacity development strategies, policies and programs. For more information, email capacity4change@worldbank.org. 2] reviewing project results retrospectively Contents Abbreviations . ........................................................................................................................... 5 Introduction . .............................................................................................................................. 7 Overview of the Capacity Development and Results Framework ......................................... 8 Understanding the Project Context ....................................................................................... 10 Mapping the Program Logic .................................................................................................. 12 Examining Opportunities for Adaptive Project Management . ........................................... 17 Lessons Learned from Test Cases .......................................................................................... 19 Annex A—Institutional Capacities and Their Contributing Characteristics ................................... 23 B—Intermediate Capacity Outcomes and Their Attributes ................................................ 26 C—Institutional Capacity Indicators Database ..................................................................... 28 D—References ......................................................................................................................... 29 [3 4] reviewing project results retrospectively Abbreviations CDRF Capacity Development and Results Framework CVGT Commission Villageoise de Gestion des Terroirs (Village Land Management Committee) ICO Intermediate Capacity Outcome ICR Implementation Completion Report IEG Independent Evaluation Group ISR Implementation Status Report M&E Monitoring and Evaluation OECD Organisation for Economic Cooperation and Development PAD Project Appraisal Document PDO Project Development Objective WBI World Bank Institute abbreviations [5 6] reviewing project results retrospectively Introduction T he standard quality assurance and capacity development projects within the World evaluation processes for World Bank Bank’s lending portfolio. The key concepts are investment lending operation projects have broadly applicable to development projects. often been found inadequate for interventions with training, technical assistance, South-South The CDRF can help task teams to determine exchange and other capacity development which interventions work, which monitoring and components. The World Bank Institute (WBI) evaluation (M&E) practices need refining, and how developed the Capacity Development and the design and implementation of future projects Results Framework (CDRF) to be a resource for can be improved. Sections lead practitioners addressing this need and to help promote more through a review process by explaining the stages results-focused capacity development practices. of an iterative evaluation cycle: A 2005 evaluation conducted by the World • Gain an orientation to the project context Bank Independent Evaluation Group (IEG), Understanding which capacity constraints a Capacity Building in Africa, found that “even project was designed to address though capacity building is a stated corporate and Regional priority [for the Bank], there are • Identify each change story no standard quality assurance processes for the Mapping interventions to associated underlying diagnosis and design of measures, and intermediate capacity outcomes and capacity capacity building interventions are not routinely development results tracked, monitored, and evaluated” (p. 44, World Bank 2005). IEG’s 2008 evaluation of project- • Assess monitoring arrangements based and WBI training reiterated this concern, Determining whether the right indicators were noting that “The Bank does not adequately tracked to promote the intended results monitor or evaluate training results” (p.55, World Bank 2008). Other studies have presented similar • Draw conclusions findings and determined that the results chains of Reviewing change processes and data gaps to capacity development interventions are frequently inform good practice for strengthening future poorly articulated (OECD 2005, 2006; World Bank M&E activities 2006; Taylor and Clark 2008). Ultimately, the CDRF can guide task teams The CDRF provides a systematic approach through an in-depth exploration of the outcomes for development practitioners to review their of capacity development interventions, to gain a program logic, assess the results of capacity practical orientation for designing more effective development interventions and strengthen future results frameworks and monitoring arrangements projects. The steps outlined by the CDRF for during project development. monitoring and evaluating capacity development outcomes are compatible with the Bank’s current project cycle and can help to assure quality during project appraisal, implementation status and completion reporting, and impact evaluation. This guide describes how to apply the CDRF for reviewing the results of existing or completed introduction [7 Overview of the Capacity Development and Results Framework T he CDRF is a strategic and country-led • A change process logic to facilitate the approach to capacity development that assignment of measurable results indicators emphasizes the empowerment of local • Standard sets of intermediate and final agents through learning, knowledge and innova- outcome indicators that can be flexibly applied tion. The CDRF serves as a valuable tool for evalu- across sectors and countries ators by articulating the results chain of capacity development activities—the progression from The CDRF specifies three institutional capacity needs assessments and interventions to outcomes areas that can be enhanced through capacity and impact toward development goals. Figure 1 development interventions, and any assessment presents an overview of the capacity development of results is contingent on understanding process as defined by the CDRF. these three areas in the project context. Development
objectives are defined in terms Evaluators and other development practitioners of improvements
in demand side institutions can use the CDRF to assess results in existing or (stakeholder ownership) and supply side completed capacity development projects. The institutions (policy efficiency and
organizational following features of the CDRF support a guided effectiveness):
 project review, from understanding the initial institutional capacity constraints and needed • Strength of stakeholder ownership—the change process(es) to identifying the indicators formal and informal political, economic and and measures assigned to track this change: social forces that determine the priority that • A structured framework to guide and define a government, civil society, and the private theory of change for capacity development sector give to a development goal. Shifts in this Figure 1. CDRF Capacity Development Process Development Goal Development Actions s Sup ion ply s t itutholder cre Inst • In In ake po ase e ituti and hen st • Im licy in fficie ons Dem S t re ngt hip r s p st nc org rove e rumen y of • e aniz Resources own f t atio fective s nal nes arra s • Financial nge of men • Human ts • Technology • Infrastructure Agents of Change • Raised awareness Intermediate Capacity Outcomes • Enhanced skills • Improved consensus and teamwork • Strengthened coalitions • Enhanced networks • New implementation Knowledge Services know-how 8] reviewing project results retrospectively Table 1. Institutional Capacity Areas and Their Contributing Characteristics Characteristics Contributing to Characteristics Contributing to Characteristics Contributing to the Strength of Stakeholder the Efficiency of Policy the Effectiveness of Ownership Instruments Organizational Arrangements • Commitment of social and • Clarity in defining rights and • Clarity of mission political leaders responsibilities • Achievement of outcomes • Compatibility of social norms • Consistency • Operational efficiency and values • Legitimacy • Financial viability and probity • Stakeholder participation in • Incentives for compliance setting priorities • Communications and • Ease of administration stakeholder relations • Stakeholder demand for accountability • Reduced negative externalities • Adaptability • Transparency of information to • Suitable flexibility stakeholders • Resistance to corruption area are reflected by widespread changes in Any World Bank project with a capacity develop- attitudes or behaviors that cut across society. ment component explicitly or implicitly targets a change in one or more of the CDRF’s capaity • Efficiency of policy instruments—the admin- characteristics. The following sections draw on istrative rules, laws, regulations, standards, and examples from Bank operations to illustrate how other formal incentives that a society uses to the CDRF can serve as a vehicle for identifying guide stakeholder actions to achieve develop- targeted change processes and assessing results. ment goals. Policy-related capacity changes extend beyond the design or implementation of a new policy, rule, or procedure to focus on the effects on stakeholder actions that have occurred as a result. • Effectiveness of organizational arrange- ments—the systems, rules of action, processes, personnel and other resources that state or non-state stakeholders bring together within a specific organization to achieve a development goal. These capacity changes extend beyond the development of individuals’ knowledge and skills to focus on the results that occur due to shifts in organizational behavior. For each of these institutional capacity areas the CDRF specifies contributing characteristics that can be enhanced through capacity development interventions (Table 1). This set of 19 capacity characteristics provides a comprehensive and standardized approach for the measurement of capacity development results. Annex A presents a more in-depth description of the characteristics. overview of the Capacity development and results framework [9 Understanding the Project Context T he first step in identifying the effectiveness development objective(s). The process of defining of capacity development interventions capacity challenges can be complicated given is to understand the capacity constraints that capacity development objectives might be or challenges that the learning or knowledge targeting multiple institutional capacities and exchange activities were designed to address. there might be multiple change paths designated The project appraisal document (PAD) typically for a single institutional capacity area. Each outlines such challenges in narrative descriptions challenge should be interpreted in specific terms of critical country and sector issues, both in the that extend beyond such vague constructs as opening section (Strategic Context and Rationale) “limited institutional capacity” to spell out which and the first annex (Country and Sector or characteristics need to be improved or enhanced. Program Background).1 In cases where a PAD has not provided sufficient According to the conceptual framework provided contextual information in the background sec- by the CDRF, capacity development projects are tions, descriptions of project components (the either explicitly or implicitly designed to address planned capacity development interventions) one or more of three types of challenges, directly can also help to elucidate the relevant capacity aligned to the CDRF’s areas of institutional constraints. In addition, the reviewer could consult capacity: subsequent project documents or speak directly • Low priority attached to the development goal with the task team leader or other knowledgeable by key stakeholders (strength of stakeholder stakeholders to gain clarification. ownership) • Deficiencies in the policy framework guiding pursuit of the development goal by different stakeholders (efficiency of policy instruments) • Weak performance of the organization(s) charged with achievement of the goal (effectiveness of organizational arrangements) Descriptions of challenges provided in PADs appear in plain language rather than in termi- nology specific to the CDRF. Hence, a reviewer should first highlight the main issues identified as part of the rationale for the project in general and for Bank involvement in particular. Once these key issues have been isolated, the reviewer can map them to the institutional capacity areas and identify the contributing characteristics for which change is desired. Table 2 presents examples from three projects to illustrate this process. Identifying and understanding the capacity 1To find standard project documents, visit www.worldbank. challenges that a project has been designed org and search under Projects Portfolio using the appropriate to address is critical for assessing the project’s keywords. The Projects Portfolio contains documentation for effectiveness in contributing to the capacity World Bank lending projects from 1947 to the present. 10 ] reviewing project results retrospectively Table 2. Reviewing Project Background Information to Identify Capacity Challenges Project Narrative Description of Capacity Corresponding Capacity Area Challenges (Excerpted from PAD) and Targeted Characteristic Multi-Sectoral AIDS “General awareness of the disease is fairly Capacity Area Project (Malawi) high, but so are misconceptions about how Strength of Stakeholder to avoid the disease. As a result, high risk Ownership—Compatibility of behavior among sexually active youth and social norms and values with the adults continues… The immediate impacts are development goal staggering: 70% of all admissions to hospital medical wards are AIDS related, and HIV/AIDS Targeted Characteristic is now the leading cause of death in the most Widespread changes are needed productive age group (20-49 years).” in the attitudes and behavior of local stakeholders to support the achievement of the development goal (reduce the transmission of HIV/AIDS). Community and Basic “There is also lack of transparency and Capacity Area Health Project accountability in the flow of funds for primary Efficiency of Policy Instruments— (Tajikistan) care, since PHC [primary health care] funds Resistance to corruption normally flow through hospitals, polyclinics or jamoats (local village councils) and there is Targeted Characteristic plenty of scope for diversion, especially for any The government’s health financing non-salary allocations… Informal payments are and budgeting process needs rampant.” refinement to reduce opportunities for rent seeking behavior by public officials. Community-Based “Past efforts to tackle the policy, institutional Capacity Area Rural Development and technological constraints facing the rural Effectiveness of Organizational Project (Burkina Faso) poor have been centrally driven and sectorally Arrangements—Operational focused. As a result, their effectiveness has efficiency been low. Decentralized decision-making and economic empowerment of beneficiary Targeted Characteristic communities is expected to improve the choice, Village land management relevance, cost effectiveness, and maintenance committees (the new structure for of rural infrastructure… In addition, to meet village-level governance) need to the demands of the local population, the plan, execute and manage local deconcentration of sectoral ministries… will projects to optimize the delivery of improve the delivery mechanisms of public public services and infrastructure goods and services and make them more relative to cost. demand-responsive.” understanding the project context [ 11 Mapping the Program Logic O nce the project reviewer understands prohibitively high because demands for infor- the capacity challenges an intervention mal payments were rampant. Public officials has been designed to address, the next easily diverted healthcare financing. The health- step for identifying capacity development results care financing process needed to be more is to trace the change process logic intended resistant to corruption, with reduced opportuni- to strengthen the targeted institutional capacity ties for rent-seeking behavior by public health- characteristics. The CDRF capacity development care officials. A relevant capacity development process (Figure 1) serves as a guide, leading the result in this context could simply be a mea- practitioner to first focus on identifying changes sured change in the share of household expen- in capacity development outcomes through the ditures allocated for health, to gauge whether a following types of questions: new and supposedly more corruption-resistant • What were the institutional and/or policy reforms process was having the desired effect. This needed to achieve the development goal? result would reflect increased efficiency of • What indicators and measures were used to policy instruments—in this case, a transparent assess the results of these reforms? Were these healthcare budgeting and financing process reforms evaluated? that discouraged (or prevented) the inappropri- ate diversion of funding. Using the results frameworks available in a project’s PAD, midterm review, implementation • Community-Based Rural Development status report (ISR) and/or implementation Project. Local services and infrastructure completion report (ICR), a reviewer should be have been provided via a centralized able to identify existing measures that have administration that does not draw effectively been assigned to gauge progress in addressing on local resources. A village land management a targeted capacity challenge. The evaluator or committee has been established in each village task team can rely on a common sense approach to allow for decentralized decisionmaking in looking for evidence of capacity development and development, but these entities need the results that convey the capacity development capacity to plan, execute and manage local change stories for the three projects from Table 2: projects effectively and efficiently. A relevant capacity development result would be a • Multi-Sectoral AIDS Project. Misconceptions measurable change in the share of projects about how to avoid HIV/AIDS transmission implemented by the village land management were common, and sexually active youth and committees that are assessed to be technically adults frequently engaged in high-risk behavior. sound and cost-effectively implemented. Such Widespread changes in the behavior of youth an assessment would be based on a cost- were needed to achieve the development efficiency analysis. goal (reducing the transmission of HIV/AIDS). A relevant capacity development result would Table 3 lists the measures, data sources, and be a measured change in behavior among a values (documented results) for these examples. targeted segment of the population. Such a change would reflect shifting social norms and Reviewers should flag cases in projects where relate to the strength of stakeholder ownership. no changes appear to have been tracked for specific institutional capacity characteristics. In • Community and Basic Health Project. that situation, proceeding to examine the other Household expenditures for healthcare were components of the change story (understanding 12 ] reviewing project results retrospectively Table 3. Assessing Capacity Development Results Project Targeted Capacity Measure Data Source Measured Values Development Outcome (Evidence of Results) Multi-Sectoral Increased compatibility Median age at first Demographic 2005—Males: 17 years; AIDS Project of the development sex among 15–24 year Health Females: 16 years (Malawi) goal (reducing the olds Surveys and transmission of HIV/AIDS) Multiple 2007—Males: 16 years; with social norms and Indicator Females: 16 years values Cluster Surveys No targets set Community and Increased resistance of Percentage of house- Household Baseline (2003) for Basic Health the healthcare budgeting hold expenditure surveys by Tajikistan: 5% Project process to corruption allocated to health Ministry (Tajikistan) care in project- of Health 2007—Sughd: 4.8% supported areas Department (measure was for Reform 2007—Khatlon: 3.1% established to confirm that funding diversions No targets set: and demands for Household survey informal payments data collection was have been minimized/ scheduled for after eliminated) project completion. Community- Increased operational Percentage of micro Project Baseline (2001): 0 Based Rural efficiency of the Com- projects that are reporting; Development mission Villageoise de technically sound cost-efficiency At Completion (2007): Project (Burkina Gestion des Terroirs and cost-effectively analysis 90% Faso) (CVGT—Village Land implemented Management Committee) Target (2007): 75% the results chain) could be useful for exploring Project reviewers should trace the complete pro- how capacity development interventions were gram logic for each capacity development change intended to address identified challenges and process to understand how and why interven- what arrangements were established for defining tions contributed to the desired results. Again, and tracking desired outcomes. the CDRF capacity development process (Figure 1) guides the practitioner to focus on a series of A project evaluator or task team leader might questions to explore the theory of change: have the opportunity to conduct a follow-up • Who were the agents of change who initiated review or data collection if warranted to clarify or or managed the desired change process(es)? confirm results. If the complete examination of • What intermediate capacity outcomes (ICOs) project documents fails to explain what resulted led to the measurable changes in institutional from capacity development interventions, then capacity characteristics? the reviewer should consider requesting more • What capacity development interventions detailed information from the task team leader. If contributed to the targeted ICOs? the project outcomes continue to remain unclear, the reviewer could potentially collaborate with The CDRF provides guidance for identifying the task team leader or other key stakeholders to ICOs These outcomes reflect the initial change collect data from administrative records or from results of capacity development interventions and intended beneficiaries as appropriate to substan- are determined by the needs of the agents of tiate the capacity development change story. change (what they need to know or accomplish to mapping the program logic [ 13 advance the institutional change process). ICOs of Health personnel and primary healthcare therefore serve two critical purposes in the overall department staff to calculate and administer a assessment of capacity development results: new per capita healthcare financing strategy in • Scoping out and defining each change agent’s each rayon of two targeted oblasts. The new role in the envisioned change knowledge was applied for the implementation • Setting milestones for monitoring progress of this standard transparent financing formula, toward achieving capacity development which was expected to limit opportunities for objectives diverting funds to non-designated uses such as discretionary payments to healthcare officials. The CDRF defines six types of ICOs: (1) Raised awareness • Community-Based Rural Development. The (2) Enhanced knowledge and skills project supported training for each village (3) Improved consensus and teamwork land management committee for prioritizing, (4) Strengthened coalitions planning, and executing local development (5) Enhanced networks programs. The knowledge and skills gained (6) New implementation know-how contributed to a strengthened coalition for Measures for these outcomes can be customized local governance in which members shared a for each project to capture changes in behaviors, common agenda for action. In addition, relationships, processes and actions of individuals local community organizations received and groups. Annex B lists the ICOs with their technical assistance to apply new knowledge operational attributes. in helping to formulate a local development plan. Finally, local service providers and field Project reviewers should consider in practical teams received project training to promote terms which individuals or groups needed to their collaboration with the village land change which behaviors or relationships, or de- management committee in implementing the velop which products and services, to achieve the new local development plan. designed capacity changes. For this retrospective approach, where projects have already been de- In each of these projects, ICOs were defined signed and implemented, a useful step can be to and measured to track progress toward targeted examine project activities to discern which capac- capacity development results. Table 4 shows these ity development interventions were delivered for outcomes with their specific measures. achieving targeted capacity development results. The three projects provide examples for this step. The standard project results frameworks in PADs and other Bank documents frequently do not • Multi-Sectoral AIDS Project. The project include clear intermediate outcomes for capacity supported a broad awareness campaign to development interventions. If such measures are address common misconceptions about how absent, a project reviewer should investigate to avoid HIV/AIDS transmission. Information further to substantiate the capacity development on HIV and AIDS was disseminated via the change story. Without measures for ICOs, it is radio, schools and health facilities to educate difficult to claim that project activities contributed youth in particular and the general population to changes in institutional capacity characteristics. more broadly about how to prevent the sexual transmission of HIV. The rationale was Methods for locating evaluation data related to that increased knowledge for preventing HIV specific interventions include the following: transmission would result in motivation among • Search for a project (or project evaluation) the targeted population to achieve widespread website. In many cases, project implementers changes in behavior. or other major stakeholders establish a public website to promote transparency, and • Community and Basic Health Project. information from surveys and other evaluation Training was provided for designated Ministry activities are posted. 14 ] reviewing project results retrospectively Table 4. Identifying Intermediate Capacity Outcomes Project Intermediate Measure Data Measured Values Capacity Outcome Source (Evidence of Results) Multi- Enhanced Percentage of young Demo- 2005—Males: 37%; Females: 25% Sectoral knowledge for people aged 15–24 graphic AIDS Project preventing HIV who correctly identify Health 2007—Males: 41.9; Females: 42.1% (Malawi) transmission ways of preventing the Surveys sexual transmission of and Targets (2010) HIV and reject major Multiple Males: 75%; Females: 75% misconceptions about Indicator HIV transmission (by Cluster gender and residence) Surveys Community Implementation of Percentage of total Annual Baseline (2006)—0% in two target and Basic new healthcare per primary health care reports oblasts Health capita financing expenditure paid of rayon/ Project strategy by capitation in oblast 2009—Sughd Average: 2.45%; (Tajikistan) project-supported health Range: 0.24% to 12.81% per rayon oblasts (this measure depart- 2009—Khatlon Average: 6.0% reflects the extent ments Range: 1.7% to 18.3% per rayon to which the new standard transparent 2010—Sughd Average: 2.05%; financing formula is Range: 0.36% to 12.31% being implemented 2009—Khatlon Average: 6.18% versus reliance on the Range: 2.72% to 14.09% previous pattern of discretionary spending) Target (2012) 100% in Spitamen rayon and 20% in each of the other 43 rayons in two target oblasts Strengthened Percentage of villages CVGT Baseline (2001): 0% coalition for with representative and Annual governing local participatory bodies Reports At Completion (2007): 149% [2,986 development (CVGT or village CVGTs established, compared to land management 2,000 targeted at appraisal] committees) assuming their role in local Target (2007): 60% development Community- Formulation of a Percentage of villages CVGT Baseline (2001): 0% Based local development covered by the project Annual Rural plan that have adopted a Reports At Completion (2007): 148% [2961 Develop- local development villages adopted plans, compared ment plan to 2,000 targeted at appraisal] Project (Burkina Target (2007): 75% Faso) Implementation Percentage of CVGTs CVGT Baseline (2001): 0% of the local that have substantially Annual At Completion (2007): 98% development plan completed sub- Reports projects identified Target (2007): 60% in their local development plan mapping the program logic [ 15 • Examine the websites and/or publications of • Community and Basic Health Project. Ministry other donors who are partners in the capacity of Health personnel and primary healthcare development initiative. Other organizations department staff have proceeded to calculate that collaborate with the Bank in supporting and administer a new per capita healthcare project activities or funding a closely related financing strategy at the rayon level in two initiative sometimes post survey results from targeted oblasts. Measurable progress was the evaluation of Bank-supported activities or achieved in rolling out this implementation conduct their own evaluations of specific proj- from the baseline year (2006) to 2009 to move ect components and disseminate the findings. away from a discretionary spending model, but • Contact the project task team leader with ques- this progress appeared to stall in 2010. The tions regarding the project. If the project docu- degree to which the implementation of the mentation describes the capacity development new financing strategy was contributing to the interventions that were designed to enhance an capacity development objective could not be institutional capacity characteristic, the reviewer effectively gauged using the assigned measure can explore with the task team whether those for this capacity development outcome (reduc- interventions were evaluated and/or how their ing the share of the household expenditure success was determined. In some cases, team allocated to health) given that other factors members or outside consultants can provide could influence this outcome (e.g., changes in reports or data that confirm an ICO not report- household income, disease outbreak, etc.). ed in the project’s results framework. • Community-Based Rural Development. The Identifying the various components and steps of village population and local organizations a results chain allows a reviewer to piece together developed skills and knowledge to how a project was designed to address a capacity work together in forming a village land challenge and what actually happened as a result management committee and then adopting of project interventions. For the three example a local development plan and executing projects (tables 2–4), the basic change stories are: and managing local development projects according to this plan. These projects related • Multi-Sectoral AIDS Project. Information on to the water supply, social infrastructure, and HIV and AIDS disseminated via project activi- environmental management and other less ties appeared to be educating some of the common themes. A cost-efficiency analysis targeted population (young people aged based on unit cost comparisons was conducted 15–24). Data from 2005 and 2007 showed that at the completion of the project. This analysis males were very slowly gaining knowledge for concluded that the sub-projects unit costs preventing HIV transmission whereas the infor- fell between 34 and 94 percent of sectoral mation seemed to be reaching females slightly comparators, with the largest efficiency gains in more effectively. The project was not on track social infrastructure. for achieving ICO targets for 2010. In addition, the minor gains in knowledge about how to Most capacity development interventions prevent HIV transmission did not appear to delivered via Bank-supported projects have have yet had an influence on the sexual behav- multiple and sometimes overlapping change ior (age at first sex) for this targeted population. stories that address a combination of institutional This review suggested that this project, with its capacity constraints. Isolating and tracing each current mix of capacity development interven- change process, as illustrated through these tions, would not achieve the desired changes three examples, is critical to understand the for the targeted capacity characteristic (making overall effectiveness of capacity development the widespread behavior of this group more interventions and the results achieved. compatible with the development goal of reducing HIV transmission). 16 ] reviewing project results retrospectively Examining Opportunities for Adaptive Project Management T he project cycle defined by the CDRF A reviewer assessing the results of a Bank project prescribes two levels for monitoring should examine the monitoring arrangements and progress toward targeted outcomes investigate any discrepancies between the actual and adjusting project activities if needed to and targeted values for outcome measures. In better ensure the achievement of capacity the three example projects, important opportuni- development objectives. First, project teams ties for adaptive project management could have should monitor progress toward ICOs on a been missed: regular, predetermined schedule. Second, where appropriate for longer projects, teams should • Multi-Sectoral AIDS Project. Survey data assess changes in the targeted institutional (demographic health surveys and multiple capacity characteristic. Both of these processes indicator cluster surveys) were used to monitor should be iterative, with project implementers whether youth (ages 15–24) were increasing monitoring the current values of targeted their understanding of how to prevent HIV outcome measures, adjusting project activities transmission. The data, provided at two-year where warranted and feasible and circling back intervals, indicated that project targets for ICOs to check the values of the measures again. The were not being achieved. The same household timing and frequency of project monitoring will surveys assessed changes in the sexual vary widely by capacity development intervention behavior of this population. Despite some depending on the time frame in which outcomes measurable increase in the understanding of are expected to occur, the availability of data and youth, this population continued to engage the relative burden of data collection and analysis. in their first sexual encounter at a young age. Various issues make these monitoring Given the cycle described above, project arrangements suboptimal: monitoring arrangements should ideally specify the following as part of the original project oo The reliance on these household surveys to design: measure knowledge gains does not provide • Target milestones to be achieved, for both the prompt feedback to project implementers ICO and the capacity development outcome for assessing the effectiveness of awareness (change in institutional capacity characteristic) campaigns. Furthermore, these surveys do • Frequency of measurement for both levels of not specifically target the audiences reached outcomes by the campaigns, so the data might not • Responsibility for measurement provide accurate feedback for shaping future • Prescribed actions or processes for determining activities. alternative actions that should be taken in cases of discrepancies (where monitoring shows that oo Changes in widespread behavior are likely milestones are not being achieved as planned) to happen slowly, particularly in this case where many of the targeted respondents will Capacity development change processes are have already had their first sexual encounter influenced by a variety of factors, and building in before possibly learning new information opportunities for adaptive project management through the awareness campaign. through effective monitoring helps to ensure the achievement of capacity development objectives. examining opportunities for adaptive project management [ 17 Focus groups or brief surveys administered to development plans). These activities included the targeted groups directly after awareness a beneficiaries’ perception study and the campaign activities could have potentially pro- evaluation of a stakeholder workshop attended vided more actionable feedback for the project by representatives of village land management implementers. committees. The project also demonstrated an adaptive management approach via some • Community and Basic Health Project. The midstream changes, most notably the addition new per capita financing for healthcare is of 1,013 target villages following a field mission expected to provide fewer opportunities for assessing project implementation. officials to divert funding for unintended uses. The hoped-for result was that households To identify the results of capacity development would reduce costs for healthcare that included interventions, project reviewers should first trace regular demands for informal payments to the program logic and associated measures healthcare providers. Project monitoring to explore the cause-and-effect relationships showed that the financing strategy was being between project activities and outcomes. Once implemented in most rayons slower than this basic understanding is established, however, planned, but project documents did not it is also important to explore how the project describe any adjustments that the team was was implemented and what discrepancies exist undertaking to investigate or address this between the expected and actual outcomes. issue. Household surveys to assess changes in Developing this understanding could require expenditures were not scheduled until after going beyond the project documentation project completion. Therefore, the measure available in Bank systems and interviewing the was not useful in providing insight to the task team leader or another key stakeholder to project team in implementing related changes. better understand issues encountered during Moreover, the project team noted in project implementation. Focus group discussions documents that the costs of healthcare as a with stakeholders and beneficiaries could also share of the overall household expenditures provide valuable information related to project could be influenced by many other factors, implementation and capacity development most notably the rising incomes simultaneously outcomes. occurring in some target areas. These details indicate that the project did not establish The guidelines for adaptive project manage- optimal measures or monitoring arrangements ment set forth by the CDRF are based on the for ensuring achievement of the capacity understanding that the evaluation of ICOs and development objective. longer term capacity development results could be introduced and/or refined at any stage in the • Community-Based Rural Development. The project cycle. Ideally, M&E plans are established formation of and reliance on a local entity during the project design phase. However, in cases (the village land management committee) for where measurement opportunities have been planning and executing local development missed earlier in the project cycle, retrospective projects was expected to result in the efficient methods for reconstructing baseline data can be delivery of public services and infrastructure applied to review or assess the results achieved. A to address local needs. While the change key consideration in this case will be how best to in the institutional capacity characteristic establish baseline values against which the desired (operational efficiency) was not assessed until change(s) can be measured. Common approaches after project completion via the cost-efficiency for reconstructing baseline data include using analysis, the project did incorporate evaluation administrative records (e.g., ministry or agency activities to monitor progress in achieving databases or reports, community organization the designated ICOs (i.e., establishment of records, university studies, or other existing the village land management committee and sources) or conducting surveys or focus groups the adoption and implementation of local to draw on the recall of key stakeholders. 18 ] reviewing project results retrospectively Lessons Learned from Test Cases U sing the CDRF to review projects and 6. Assemble each individual capacity assess results helps practitioners to under- development change story to understand stand what works in capacity development whether and how interventions contributed interventions and compile lessons learned for in- to the expected results and to explore any forming future project design. The basic steps for instances when project implementation this approach, as demonstrated through the three was or should have been adjusted to better example projects, include the following: achieve those results. 1. Assemble the available project documenta- 7. Fill information gaps as needed by contact- tion, particularly the PAD, midterm review, ing the task team leader with questions or implementation status reports (ISRs), imple- by reviewing data or publications provided mentation completion report (ICR) and aide by other donors who collaborated with memoires from supervision visits. the Bank on this initiative. In some cases, reviewers might work with the task team 2. Review narrative sections of the PAD to leader to identify opportunities to collect understand the project context, looking at additional data on project outcomes from the sector and country background sections beneficiaries or other key stakeholders. and the description of project components to understand which institutional capacity This process helps to ensure that capacity devel- characteristics the project was designed to opment projects are accountable for achieving address. targeted results. Compiling information gained through such project reviews over time could yield 3. Review the project results framework in all an important body of knowledge about which available documents (PAD, ISRs and ICR) to capacity development interventions are effective identify indicators and measures related to in which contexts. changes in the targeted institutional capac- ity characteristics. Note the data sources, The exercise of reviewing projects to illustrate targets, current values and arrangements this approach also provided useful guidance for for monitoring. how the CDRF could be used to strengthen M&E practice during the project design phase. Exam- 4. Examine what capacity development ining project documentation highlighted areas interventions were designed to contribute where effective M&E input could play a critical to targeted changes in the institutional role in supporting, promoting and confirming the capacity characteristics, noting the specific achievement of a capacity development objective: activities and the targeted participants. • Thinking beyond the outputs of capacity 5. Identify measures of ICOs by reviewing the development interventions to define project results framework—the outcomes measurable ICOs. The CDRF provides guidance expected to occur as a direct result of the to practitioners for defining the outcomes capacity development interventions. Note expected to directly result from capacity the data sources, targets, current values and development activities for individuals or arrangements for monitoring. groups. These relate to changes in knowledge, behaviors, relationships, processes and actions. Identifying which stakeholders (agents of lessons learned from test cases [ 19 change) will initiate or manage the needed (at http://wbicdrf.worldbank.org) to provide change and what these stakeholders need examples of measures for each capacity to achieve (ICO) to facilitate the envisioned characteristic in a variety of sectors. Annex C change process is necessary for designing further describes the database. the right interventions for contributing to the overall capacity development objective. • Applying the CDRF to refine the use of the Bank’s standard project reporting documents. • Prescribing monitoring arrangements to The CDRF is compatible with the Bank’s support adaptive project management. The current quality assurance and reporting process of assigning effective indicators and processes and can add value to the PAD, ISRs, measures for intermediate and final outcomes and ICR going forward. By identifying and should be accompanied by the consideration communicating the targeted change processes of how best to monitor progress along the and capacity development outcomes, project prescribed results chain. The arrangements for teams can promote and implement results- monitoring should specify at which intervals focused capacity development. This CDRF- each outcome will be assessed and who will driven approach would include the following fulfill this responsibility. In addition, the project dimensions in standard Bank documents: team should outline key decision points in cases where there are gaps between current oo Sectorissues presented as part of a project’s and expected values in terms of what actions strategic context would be described in the team will take to investigate and handle terms of the institutional capacity challenges these discrepancies. that they represent. This approach sets the stage for identifying specific measurable • Assigning effective measures for tracking capacity development objectives. changes in institutional capacity characteristics. A key lesson was the importance and potential oo The project development objective(s) (PDO) difficulty of measuring changes for a specific of a project with a substantial capacity capacity development outcome. In cases where development focus would be expressed changes in targeted institutional capacity in terms of a clear capacity development characteristics can be expected to occur objective or would be directly linked to the during the project term, practitioners need capacity development objectives. The PDO- to think through what types of measures are level indicators would then be designed to feasible for a given project context, what data measure progress in achieving the capacity sources the measures will rely on, and how the development objective(s). responsibility for monitoring and assessment will be distributed. In cases where institutional oo Projectcomponents would be linked changes are observable only after project to capacity development objectives so completion, the indicators are important for that a targeted capacity development impact evaluation. change process is the focus of one unified component or of a group of closely Ideally, a measure for an institutional capacity associated components. characteristic should be one that can be assessed at some point(s) during project oo Given the grouping of project components implementation rather than just at baseline linked to capacity development objectives and after project completion. This will allow noted above, the project results framework implementers to adjust the design of capacity would assess each capacity development development activities if change is not being change process rather than simply present- facilitated as planned. Through an ongoing ing outcome indicators for a somewhat arbi- review of Bank projects, WBI has assembled trary array of components. the Institutional Capacity Indicators Database 20 ] reviewing project results retrospectively oo Two levels of measures would be assigned in the project results framework and monitoring table for each capacity development objective to track both the intermediate and institutional outcomes. Where possible, arrangements would be specified to support an adaptive project management approach. Reviewing project documents for capacity development interventions provides an important lesson for practitioners about the importance of building in an intact results chain for project design, implementation and evaluation. Whereas many current project documents require reviewers to piece together a capacity development change story by searching for potential intermediate outcomes measures and associated indicators of institutional capacity characteristics, newly developed or revised results frameworks could present clearer sequences of outcome measures for each capacity development objective. Conducting a project review using the CDRF is therefore useful not only for understanding the results of previous Bank operations but also for developing a stronger results orientation for capacity development initiatives going forward. lessons learned from test cases [ 21 22 ] reviewing project results retrospectively Annex A. Institutional Capacities and Their Contributing Characteristics Three broad areas of institutional capacity and their contributing characteristics help assess institutional capacity needs, inform design, and measure progress toward results for a capacity development program. Table 5. Strength of Stakeholder Ownership How strong is stakeholder ownership of the government, civil society, and the private sector development goal? Stakeholder ownership give to a development goal. The institutional comprises formal and informal political, economic characteristics listed below contribute to the and social forces that determine the priority that strength of stakeholder ownership. Institutional Characteristics Areas Covered Commitment of social and Social and political leaders consistently and regularly support the development political leaders goal through public statements, decisions, votes, and other actions demonstrating their commitment and accountability for achievement of the goal. Leaders demonstrate sound ethics and integrity related to the goal, faithfully honoring their pledges of support and not engaging in activities or missions that would undermine progress toward the goal. Compatibility of social norms The social norms and widespread beliefs that shape the behavior of local and values stakeholders are compatible with the development goal. Stakeholder participation in Priority-setting processes related to the development goal include opportunities setting priorities for all stakeholders to express opinions freely and without fear of repression, and these opportunities are communicated with adequate time for stakeholder engagement and response. Stakeholders actively participate through these established mechanisms, and government and other public service entities are responsive to the expressed views of civil society and other relevant parties. Stakeholder demand for Stakeholders know their rights related to the development goal, claim those accountability rights, and communicate their grievances and proposals for change to the government and legislature. If public officials and other public service providers fail to meet expectations about the achievement of the goal or are using allocated resources for unintended purposes, stakeholders freely hold them accountable for their conduct and performance. Stakeholders organize and collectively appeal unfavorable decisions concerning the goal. Stakeholders demand and use instruments of accountability that are available to the public, such as government scorecard information. Transparency of information Government and other public service entities provide accurate, relevant, to stakeholders verifiable, and timely information about the development goal to all stakeholders. They also explain actions concerning the goal in terms that stakeholders can use to participate in setting priorities, monitoring progress, and evaluating actions of public officials responsible for the goal. Any information influencing decisions related to the goal is fully disclosed to stakeholders. institutional CAPACITIES and THEIR COntributing characteristics [ 23 Table 6. Efficiency of Policy Instruments How efficient are policy instruments in guiding tions, standards, and other formal incentives that stakeholder behavior toward achieving the devel- a society uses to guide stakeholder actions to opment goal? The efficiency of policy instruments achieve its goals. The characteristics listed con- comprises the administrative rules, laws, regula- tribute to the efficiency of policy instruments Institutional Characteristics Areas Covered Clarity in defining rights and The rights and responsibilities of stakeholders related to the development responsibilities goal are clearly defined and communicated. Stakeholders share a common understanding of the policy goal and the targets of any specified regulations. The authorities and processes concerning the policy instrument are clearly identified and communicated to stakeholders. Stakeholders understand the objectives, rights and responsibilities prescribed by the policy instrument. Consistency Policy instrument related to the development goal is consistent with policy instru- ments for other goals. The objectives, rights and responsibilities established by the policy instrument related to the goal do not conflict with those for other goals. Legitimacy Policy instrument is formulated based on an informed, transparent, participatory, and deliberate process. The instrument conforms to recognized principles and standards and is perceived as desirable and appropriate within the local system of norms, values, beliefs, and definitions. Stakeholders and unaffiliated expert reviewers perceive the actions and sanctions prescribed by the policy as fair. Stakeholders’ rights to appeal are assured by the policy instrument. Incentives for compliance Policy instrument facilitates desired economic and social exchange activities related to the development goal, minimizes uncertainty, time, information and other costs to participants in these transactions, and provides sanctions for non-compliance. Possible measures include: share of targeted stakeholders who elect self-policing, self-auditing, or self-disclosure options provided by the policy instrument; share of targeted stakeholders who comply with provisions of the policy instrument. Ease of administration Policy instrument is administratively easy for the regulatory or governing body to implement. Implementation incurs only those administrative costs required to achieve the intended policy outcomes related to the goal. Administrative tasks are defined and allocated with consideration of duty bearers’ competencies, existing responsibilities, and available resources. Risk of negative externalities Policy instrument minimizes unintended negative impacts. Targeted beneficiaries, non-beneficiaries, and other non-regulated stakeholders do not experience foreseeable unintended costs or ill effects as a result of the policy instrument. Suitable flexibility Policy instrument is predictably flexible in addressing varying situations. The policy instrument allows for timely revision when the underlying social and political circumstances have changed. The policy instrument is explicit in defining the conditions under which its terms should be revised. Resistance to corruption Policy instrument minimizes opportunities for corruption, rent seeking, and regulatory capture; includes mechanisms to monitor and report corruption; and provides credible and enforceable penalties for corrupt behavior. The policy instrument does not reflect the efforts of vested interests to manipulate the economic and/or legal environment to secure undue privileges or compensation at the expense of the greater public good. 24 ] reviewing project results retrospectively Table 7. Effectiveness of Organizational Arrangements How effective are organizational arrangements in and other resources that state or non-state stake- achieving the development goal? The effective- holders bring together to achieve a development ness of organizational arrangements comprises goal. The characteristics listed contribute to the the systems, rules of action, processes, personnel, effectiveness of organizational arrangements. Institutional Characteristics Areas Covered Clarity of mission The vision and mission of the organization/group are strongly aligned with the goal and articulated, providing its members with points of reference for formulating strategy, making decisions, and gaining commitment from management, staff, and other stakeholders to work toward the goal. Relevant stakeholders recognize the mandate of the organization. Achievement of outcomes The organization/group consistently seeks and achieves outcomes that lead directly to the goal. Operational efficiency The strategies, inputs, processes, and technology of the organization/group are managed to optimize the quantity and quality of output, relative to the cost of accomplishing its support for the goal. Financial viability and probity The organization/group secures the funds needed to cover its operating costs sustainably and allocates funds according to business needs. Financial management includes the tracking of expenditures on a functional basis to inform future budget forecasting, decision-making, and accountability reporting. Organizational systems and practices support and enforce transparent financial management, including the reporting of externally verified accounts, to ensure that the organization’s resources are allocated effectively to achieve its stated goal. Communications and The organization/group seeks and obtains the support of stakeholders for its stakeholder relations work related to the development goal. Organizational decision-making and operational processes involve consultations and proactive communications with appropriate stakeholders. Adaptability The organization/group regularly monitors its internal and external environment for information relevant to the development goal and is proactive in adapting its strategy accordingly. The organization encourages innovation in strategies, processes and products; manages knowledge; rewards performance; and creates and/or adapts to new technologies. institutional CAPACITIES and THEIR COntributing characteristics [ 25 Annex B. Intermediate Capacity Outcomes and Their Attributes An ICO is defined as a change in the ability or change process. Once practitioners identify these disposition of stakeholders (or agents of change) outcomes, they can plan capacity development to take actions. This change is considered an ICO, activities in a more concrete way by: because the expectation is that the stakeholders— thanks to the changed ability or disposition—will • Empowering local agents of change act to affect institutional changes toward the devel- Who are the local agents that will drive change? opment goal of a capacity development program. For example, agents may be individuals or groups of individuals in an organization, such as Six standard ICOs provide a typology to guide the senior-level officials from the ministry of finance formulation of a capacity development program and planning commission who hold lead (Figure 2). Development practitioners can use the responsibility for budget planning. typology to develop project-specific objectives for capacity development as well as identify mile- • Defining learning objectives stones to observe, monitor and measure change What differences are expected after the capac- in the capacity areas in need of improvement. By ity development intervention is successfully monitoring these milestones, practitioners can completed? Are changes expected in behav- track progress toward the achievement of measur- iors, relationships, processes or development able results. If sufficient progress has not been actions? For example, improved consensus made, they can revise project activities as neces- might be an agreement reached within the sary in a timely manner. ministry of finance among the departments of budget, finance and accounting on the pro- Identifying the ICOs to be achieved requires cess for implementing a medium term budget assessing what local agents need to advance a framework. Figure 2. Intermediate Capacity Outcomes and Their Operational Attributes (1) Raised (2) Enhanced Knowledge Awareness and Skills • Understanding • Acquisition of new • Attitude knowledge or skills • Confidence • Application of new • Motivation knowledge or skills (3) Improved Consensus (4) Strengthened (5) Enhanced and Teamwork Coalitions Networks • Communication • Common agenda for • Common interest • Coordination action • Processes for collaboration • Contributions • Commitment to act • Incentives for participation • Cohesion • Trust • Generating traffic • Leveraging diversities (6) New Implementation Know-how • Formulation of policies, strategies, or plans • Implementation of policies, strategies, or plans 26 ] reviewing project results retrospectively Table 8. Examples of Objectives and Indicators for Intermediate Capacity Outcomes Intermediate Examples of Objectives Examples of Indicators Capacity Outcomes Raised awareness Improve participant confidence and Number and percentage of staff in line ministries motivation to implement budget targeted by a program who report improved reform confidence and motivation to implement budget reform Enhanced knowledge Enhance staff ability to apply new Number and percentage of finance division or skills skills in medium term budget staff trained in macro-economic forecasting planning who apply the tools in planning a medium term budget Improved consensus Improve communication and Improved communication and coordination and teamwork coordination among departments within the ministry of finance among the of the ministry of finance on an departments of budget, finance, and accounting action plan for implementing a contributed to move forward the action plan for medium term budget framework implementing a medium term budget framework Strengthened Advance multiple stakeholders' Joint plan of action to implement a medium coalitions common agenda for action to term budget framework moved forward by implement a medium term budget ministry of finance, planning commission, and framework line ministries Enhanced networks Strengthen the ability of Application of common technical standards stakeholders with varying agendas among community of practice of local budget but connected by common directors interests to apply technical standards New implementation Formulate the ministry of finance Strategy for implementation of medium term Know-how and line ministries’ implementation budget framework developed by ministry of strategy for the medium term finance and line ministries, based on a learning- budget framework through an by-doing approach experiential participatory process • Developing an implementation approach What are the most appropriate activities to achieve the targeted ICOs? For example, which activities would be appropriate for enhancing skills in government to implement a medium term budget framework? A conventional approach would be to conduct a training course on budgeting for local officials. However, to achieve ICOs, semi-structured learning activities—such as peer-to-peer knowledge exchanges or an action-learning workshop—might be more appropriate. Thus, considering the outcomes to achieve helps practitioners move beyond conventional training activities toward transformational learning activities that advance results. intermediate capacity outcomes and their attributes [ 27 Annex C. Institutional Capacity Indicators Database The Institutional Capacity Indicators Database • Expand understanding of results management (at http://wbicdrf.worldbank.org) is a searchable for capacity development catalogue of real-world capacity characteristics and their indicators. In this way, project teams can Real-World Example break institutional capacities down into observable Doreen is a task team leader within an urban and measurable units to inform program design, development sector who focuses on public sector monitor and measure changes and report tangible governance. She is working on a capacity devel- results for their capacity development programs. opment program with the goal of improving pub- lic services in targeted urban areas. She initially Background planned to focus primarily on organizational ca- Task teams often need concrete results for capac- pacity development. However, after working with ity development initiatives, to show the viability stakeholders and using the database to inform the of their efforts and accountability to stakehold- needs assessment, she adopted a more compre- ers and donors. However, evaluation of capacity hensive approach to institutional capacities that development activities often focuses on measuring also targeted the area of stakeholder ownership. outputs rather than outcomes. Also, the absence of appropriate indicators and failure to effectively Through the database, Doreen understood how track them limit the possibility for teams to make constraints to stakeholder ownership could thwart needed, timely adjustments to their programs. the program’s ability to achieve locally owned results. She searched the database for examples WBI developed the database as a convenient tool of capacity characteristic outcome indicators and for task teams to find examples of indicators and their measures to help inform program design. measures for institutional capacity challenges their Together with stakeholders, Doreen also priori- projects face. The database features examples of tized the characteristics according to the most indicators from a review of development databases needed results. and approximately 200 existing and closed World Bank projects across sectors and regions. Doreen was able to better allocate limited funds for the capacity development program and to Uses and Benefits support needed results in a previously neglected • Identify characteristics of institutional capacity to capacity area. Program planning became a more inform program design transparent and collaborative process with stake- • Identify indicators of those characteristics for holders in identifying capacity characteristics, indi- periodic data collection to monitor and evaluate cators and data measures for local M&E practice. institutional change Table 9 presents an example of an institutional • Prioritize characteristics according to the most capacity characteristic, its indicator and measure needed results to better target interventions that Doreen will use in the program. Table 9. Example of Data from the Insitutional Capacity Indicators Database Sector Urban development Development Goal Improve public services in targeted urban areas Institutional Capacity Area Stakeholder ownership Characteristic Stakeholder participation in setting priorities Indicator Participation of community stakeholders in decisions regarding local public services Measure Participation rate of poorest and vulnerable community members in planning meetings Data Source Management Information System of Oversight Consultant, World Bank 28 ] reviewing project results retrospectively Annex D. References Organisation for Economic Co-operation and Development (OECD). 2005. The Paris Declaration on Aid Effectiveness. Paris. Organisation for Economic Co-operation and Development (OECD). 2006. Survey on Monitoring the Paris Declaration: Overview of the Results. Paris. Otoo, S., Agapitova, N., and Behrens J. 2009. The Capacity Development Results Framework: A Strategic and Results-Oriented Approach to Learning for Capacity Development. World Bank Institute, Washington D.C. Taylor, P. and P. Clarke. 2008. Capacity for a Change. Institute of Development Studies. Sussex. World Bank Independent Evaluation Group (IEG). 2005. Capacity Building in Africa: An IEG Evaluation of World Bank Support. Washington, D.C. World Bank Independent Evaluation Group (IEG). 2006. Annual Review of Development Effectiveness. Washington, D.C. World Bank Independent Evaluation Group (IEG). 2008. Using Training to Build Capacity For Development: An IEG Evaluation of the World Bank’s Project-Based and WBI Training. Washington, D.C. World Bank Institute. 2011. A Review of Capacity Development Results Measurement in World Bank Projects: The Need for a Systematic Approach. Washington, D.C. references [ 29 30 ] reviewing project results retrospectively The original had problem with text extraction. pdftotext Unable to extract text.