Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) Report Number: ICRR0021814 1. Project Data Project ID Project Name P105860 NP: PAF II Country Practice Area(Lead) Nepal Agriculture and Food L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) COFN-C1200,COFN-C1380,IDA- 30-Sep-2012 244,355,012.92 49220,IDA-H3370,IDA-H6720,IDA- H8570,TF-99274 Bank Approval Date Closing Date (Actual) 06-Dec-2007 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 100,000,000.00 0.00 Revised Commitment 100,000,000.00 0.00 Actual 246,263,385.11 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hassan Wally John R. Eriksson Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives This project was the second phase of the Poverty Alleviation Fund (PAF) which was launched in 2002 as a community-driven development (CDD) program for addressing the related problems of rural poverty and social exclusion in Nepal. Page 1 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) The Project Development Objective (PDO) stated in the Project Appraisal Document (para 15) was identical to the one stated in the Financing Agreement (FA, p. 5) and aimed to: "improve living conditions, livelihoods and empowerment among the rural poor, with particular attention to groups that have traditionally been excluded by reasons of gender, ethnicity, caste and location." b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? No d. Components The project was supported by the following five components: 1. Small-Scale Village and Community Infrastructure (appraisal cost: US$34.6 million, first additional financing (AF): US$28.5 million, second AF: US$15.7 million, actual cost: US$71.25 million). This component aimed to finance small investments in infrastructure and services demanded by community organizations (COs) formed by PAF under Component 4. 2. Income Generating Sub-Projects (appraisal cost: US$40.3 million, first AF: US$21.66 million, second AF: US$29.1 million, actual cost: US$100.31 million). Matching grants would be provided to self-selected groups of poor and excluded people for income-generation activities, based on objective criteria including ethnicity, caste, gender and poverty levels. Beneficiaries would contribute about 10% of the sub-project cost in cash. 3. Innovations and Special Programs (appraisal cost: US$9.6 million, first AF: US$3.07 million, second AF: US$9.5 million, actual cost: US$19.78 million). An innovations window would support proposals meriting special consideration owing to exceptional needs in a given context, or demonstrating innovative ways to improve livelihoods development and reach targeted groups. 4. Capacity Building and Implementation Support (appraisal cost: US$19 million, first AF: US$8.7 million, second AF: US$14.5 million, actual cost: US$42.04 million). This component aimed to improve capacity to implement PAF projects through partner organizations (POs), and to support implementing capacity of PAF. It included the following six sub-components: (i) social mobilization of community groups; (ii) capacity building for local bodies; (iii) capacity building for target groups engaged in income generating Page 2 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) activities; (iv) support to rural and community finance; and (v) information, monitoring and evaluation (M&E). 5. Project Management, Planning and M&E (appraisal cost: US$5.04 million, first AF: US$3.07 million, second AF: US$11.2 million, actual cost: US$42.04 million). To finance PAF staff and agency operating expenses, including financial, procurement and environmental management and audits. Revised Components. 1. At the time of the second AF in 2013, the name of Component 3 was changed from “Innovation and special programs” to “Product development, market linkages and pilots”, to better describe the innovative activities proposed under the AF. This component under the revised name aimed to support COs that were more advanced through the pocket area approach and to pilot-test the PAF model in a peri-urban setting. The name of Component 5 “Administration of PAF II” was changed to “Project management, planning and M&E”. 2. In response to a severe earthquake in 2015, funds were reallocated among the components at the 2016 restructuring. Under Component 1 and 2, a sub-component was added to each, to specifically target earthquake-hit areas. Under Component 4, two sub-components were added for social mobilization of affected COs and skills development for earthquake recovery needs. These additional sub-components were co-financed by IDA (75%) and IFAD (25%). Under Component 5, a sub-component was added to strengthen technical support for after-earthquake related needs. A sub-component on knowledge management financed by IFAD was also added. 3. During the 2017 restructuring, funds were reallocated among components to adjust the outstanding needs for the remaining period. Similarly, the proportions of co-financing for the earthquake related sub- components were adjusted to address the financing needs of the remaining period (IFAD 75% and IDA 25%). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. The original total project cost was expected to be US$109 million. The actual cost reported by the ICR (p. 2) was US$259.59 million. The difference was mainly due to receiving two IDA additional financing as well as funding through the Food Crisis Response Trust Fund (more details below). Financing. The project was originally financed through an IDA Grant worth US$100 million. Two AFs were approved during implementation, the first was for US$65 million and the second was for US$80 million. The project also received US$10 million from above-mentioned Trust Fund at the first AF. The International Fund for Agriculture Development (IFAD) contributed a total of US$9.0 million in co-financing funds. The total cumulative approval amount was US$264 million. According to the ICR (Annex 3 and p. 2) the total cumulative disbursed amount at completion was US$253.09 million, of these US$99.47 were from the original Grant, US$63.43 million from the first AF, US$73.37 million from the second AF, US$10 million from Page 3 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) the Trust Fund, and US$6.82 million from IFAD. In a further communication, the project team explained that "the final amount of total disbursement at the end of the closing of Grace Period was 259.91 million. Only US$2.89 million of IDA (99 % disbursed) and US$1.2 million of IFAD (87 % disbursed) were undisbursed, and refunded to the IDA and IFAD." Borrower Contribution. The Government of Nepal was expected to contribute 1% of the original Grant amount and beneficiaries were expected to contribute 8% of the original Grant. These contributions were expected to be in cash for the Government amount and in-kind for beneficiaries. The ICR (p. 2) reported that that the borrower was expected to contribute US$21 million. At completion the total amount contributed was US$6.5 million. Dates. The project was approved on December 6, 2007. It became effective on March 19, 2008 and was expected to close on September 30, 2012. The actual closing date was six years later on December 30, 2018. The later than expected closing was due to receiving additional financing and scaling up project activities. The Mid-term Review was conducted on May 13, 2010 compared to a planned date on September 2010. The project was restructured five times all Level 2 restructuring. The first restructuring was on June 2, 2011, when the cumulative amount disbursed was US$98.49 million, in order to approve an Additional Financing (AF) worth US$65 million of IDA grant and additional US$10 million from Food Price Crisis Response Trust Fund. The objective was to expand the project area, scale up targets, scale up income generation activities under Component 2 to address Food Price Crisis, and revise the Results Framework (RF). The second restructuring was on June 5, 2013, when the cumulative amount disbursed was US$138.80 million, in order to approve an Additional Financing worth US$80 million of IDA grant. The objective was to expand the project area, scale up targets, and change the name of Component 2 to pilot pocket area and peri-urban activities. The third restructuring was on June 25, 2014, when the cumulative amount disbursed was US$162.80 million, in order to change in closing date of first AF and reallocate funds between disbursement categories. The objective of these changes was to fully utilize the balance of the first AF. The fourth restructuring was on February 12, 2016, when the cumulative amount disbursed was US$208.91 million, in order to revise the RF, change in components and cost, change in closing date of second AF, and reallocate funds between disbursement categories. The objective of these changes was to add activities to address the needs of the earthquake impacted areas. The fifth restructuring was on November 30, 2017, when the cumulative amount disbursed was US$236.26 million, in order to change components and cost reallocation between disbursement categories. The objective of these changes was to fully utilize the balance of the second AF. Page 4 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) 3. Relevance of Objectives Rationale Context at Appraisal. With an average GDP per capita of US$270 in 2005, Nepal was the poorest country in South Asia and the twelfth poorest in the world. The poverty rate in rural areas stood at 35% compared to 10% in urban centers, and evidence suggested that inequality and regional disparities were increasing. Inequality was partly due to a bias in public expenditures in the past which privileged ruling elites and urban centers; in addition to longstanding patterns of gender, ethnic and caste-based exclusion. The country also experienced a decade marked by violent conflict that ended in 2006. Everywhere in the country, ethnic minorities, lower caste communities and women (especially female headed households) lag seriously behind in terms of incomes, assets and most human development indicators. In 2002, the Poverty Alleviation Fund (PAF) was launched as a community-driven development (CDD) instrument for addressing the related problems of rural poverty and social exclusion. Building on the lessons and success of the first phase, the second phase was launched with scaled up coverage to reach 75 districts or 30% of the rural population. Relevance of Objectives at Appraisal. Objectives were in line with the Bank's November 2003 Country Assistance Strategy for Nepal (CAS), which was in turn was aligned with Nepal's PRSP and the Government's Tenth Five-Year Plan (2002-2007). The CAS identified PAF as a promising instrument for targeted support to achieve rural poverty reduction and social inclusion objectives. Objectives were also in line with the Bank's January 2007 Interim Strategy Note (ISN) which prioritized the inter-related goals of inclusion, state building and growth. Objectives were also in line with the development priorities of the Government of Nepal especially the need to better target interventions towards poor, marginalized and excluded groups, and to engage these beneficiaries directly both in priority-setting and implementation of investments intended to serve them. Relevance of Objectives at Completion. Objectives became less relevant at completion due to a shift in the Government priorities. The Government became focused on investments in economic, physical and social infrastructures to promote economic progress and help Nepal become a middle-income country by 2030. The Bank’s Nepal Country Partnership Framework (CPF) (2019–2023) highlighted three focus areas for engagement: (i) public institutions; (ii) private-sector led jobs and growth; and (iii) inclusion and resilience. However, PAF and CDD projects were removed from the latest CPF reflecting a decrease in relevance. Furthermore, towards completion Nepal moved towards a relatively decentralized federal governing model which was not compatible with the more centralized mode of operation of the project (ICR, para 19). The statement of objectives was focused and included a clear connection to higher level objectives, namely rural poverty reduction and social inclusion. Page 5 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) While the project remained consistent with the Bank’s strategy throughout implementation, objectives became less relevant at completion as Government priorities changed. Objectives also were not compatible with the federal governing system adopted by the Government. Therefore, the relevance of Objectives is rated as Substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective PDO: to improve living conditions, livelihoods and empowerment among the rural poor, with particular attention to groups that have traditionally been excluded by reasons of gender, ethnicity, caste and location. The PDO as stated includes the following four sub-objectives: (i) improve living conditions; (ii) improve livelihoods; (iii) improve levels of empowerment; and (iv) reach the marginalized population. Rationale Theory of Change. The theory of change is the relationship between activities, outputs from those activities and the final outcome (to improve living conditions, livelihoods and empowerment among the rural poor). In this case change is facilitated by a number of critical elements including: the possibility of providing direct funding to community level institutions under the current political setting, ability and willingness of marginalized population to take advantage of the opportunities created by the project, Budget allocation and staffing of local governments allows for improving rural infrastructure services to compliment project activities; Private companies and service providers recognize economic opportunities aimed at market linkages for income generation activities promoted by the project. To improve living conditions and livelihoods the project would finance small-scale village and community infrastructure, income generating sub-projects, and support development of products and market linkages. These activities were expected to create and improve community infrastructure, support income-generating activities, develop local business and market linkages. This would result in improved access and use of services and improved market linkages. As a result, beneficiaries would have better access to infrastructure and their income would improve. To improve levels of participation and empowerment, the project would support community level institution building, capacity building and participatory decision making, and establish effective targeting to reach out to the marginalized population. These activities were expected to improve local participation in decision making, Page 6 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) local prioritization of demands through participatory decision making at the community level, and participation of marginalized population. This would result in community and individual empowerment and improved levels of enhanced institutional capacity for integrated resources management. As a result, beneficiaries would have better participation and empowerment, and marginalized populations would be reached. The long-term impact of the project would be alleviation of rural poverty, and sustainable and inclusive economic growth. Overall, the stated activities were logical and had clear causal links to the PDO. Objective 1 To improve living conditions of the targeted beneficiaries in project areas. Outputs The following outputs were reported in the ICR (Annex 1) unless referenced otherwise.  317,404 households benefited from increased access to community infrastructure compared to an original target of 15,000 and formally revised target of 164,000 (target significantly exceeded).  4,449 infrastructure sub-projects completed (target: 4000) with target community participation, according to agreed design and quality standards (target exceeded). Sub-projects mostly consisted of water supply and sanitation, irrigation, access to rural roads, rural energy and community building.  69% of infrastructure sub-projects completed with functional O&M system (target 70%, almost fully achieved).  219 PAF supported Infrastructures were rehabilitated in earth quake-impacted areas (target: 256). Outcome The PDO Outcome indicator 1, showing that 317,404 households benefited from increased access to community infrastructure by project closure, almost doubled the revised target of 164,000 households (ICR, p.34). The RF and the team explain this result by reporting that more COs than anticipated chose to support small community infrastructure activities and that there was also a tendency to support relatively large infrastructure sub-projects-that benefited a larger number of households. PDO outcome indicator 1 did not measure outcome in terms of the extent to which living conditions were improved. This was more adequately measured by PDO Outcome Indicator 2 (see objective 2). Achievements also included the completion of 4,449 sub-projects (against a target of 4,000 infrastructure sub-projects). According to a 2014 sustainability study conducted by PAF, about 69% of the infrastructure was sustainable with adequate O&M funds. A comprehensive impact evaluation found that the improved infrastructure had a positive impact of beneficiaries as they could spend more time on income generating activities rather than household chores such as water fetching. Also, improved roads and bridges facilitated easier travel to markets and to seek medical attention. Page 7 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) The project also supported irrigation infrastructure. However, the impact of these investments on improving food security and agricultural productivity were not documented by the project. Based on the above-mentioned information, it is evident that the project contributed to improving the living conditions of the targeted beneficiaries in project areas. The project also met or exceeded its outcome targets. Therefore, efficacy of this outcome is rated Substantial Rating Substantial OBJECTIVE 2 Objective To improve livelihoods among the rural poor. Rationale Outputs The following outputs were reported in the ICR (Annex 1) unless referenced otherwise.  By project completion 71.43 % of beneficiary households had increased their incomes by at least 15% against base year (original target: 70%, target achieved)  57% of income generating activities community organizations members belonged to targeted households (original target: 54%, target exceeded).  60% of community organizations members accessed funds from the revolving fund; more than one time for Income Generating Activities (IGAs) during the lifetime of the project (baseline: none, target not specified).  41.5% of community organizations members showed improved levels of food security in terms of months of food sufficiency (original target: 68%, no baseline, target not achieved).  30 pocket area development pilots were established (original target: 60, revised target: 30, target achieved). These benefited 363 community organizations and 8,759 CO members (ICR, para 29). Due to the shortage of the remaining funds and also to spend adequate time to monitor the results of the first 30 PA pilots, the target was revised to 30 at 2nd restructuring (CR, p.40)  40 peri-urban development pilots were established (original target: 90). The ICR (para 30) explained that the costs of peri-urban pilots were high, and the target was reduced to 40 to ensure adequate funding. These pilots covered 1,014 households involved in 174 sub-projects (111 for income generation activities and 63 for skill enhancement training). Project data showed that there was a 27.3% income increase in real terms among the participating beneficiaries (ICR. para 31).  Earthquake relief. 11,578 community organizations members benefited from asset/cash transfer for revolving fund revitalization (original target: 40,000, revised target: 14,000, baseline: none, target not achieved). The ICR (para 32) explained that original target was based on a preliminary assessment of Households impacted by the earthquake. A detailed analysis found those to be around 14,000 Households. Page 8 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) Outcome The extent to which living conditions improved was captured by PDO Outcome Indicator 2 "Percentage of beneficiary households who have increased their incomes by at least 15% against base year (2007)". The ICR reported that that 71.4% of households met this threshold, compared to an original target of 70.0% (ICR, p.35). The comprehensive impact evaluation (CIE) found that food security in terms of months of food sufficiency reached 41.5% among beneficiaries, which was below the target of 68%. The CIE also reported that households that benefited from the project's revolving fund increased their total expenditure by 22% compared to control households, and spent 43–53% more on health and education compared to the control group. While the project achieved only 60% of its intermediate target on food security, the main outcome (increasing income by 15%) was achieved. It is also plausible to assume that beneficiary households who experienced an increase in their income would be in a better position to improve their food security-since they have more money to spend on purchasing food supplies. Based on the above-mentioned information, it is evident that the project contributed to improving livelihoods of the rural poor in project areas. Therefore, efficacy of this outcome is rated Substantial despite concerns on falling short of achieving the food security target. Rating Substantial OBJECTIVE 3 Objective To improve empowerment among the rural poor. Rationale Outputs  63% of key positions in project community organizations are from targeted households (target; 60%, target exceeded).  Women represented 78% of key positions (ICR, para 33). Outcome The project succeeded in exceeding its target on filling key positions in project community organizations from targeted households. Attribution of women empowerment solely to project activities is hard because according to the ICR (para 39) members of community organizations "frequently participated in several groups and development activities" that might have not been supported by the project. In a further communication, the project team explained that women empowerment was assessed as part of the Comprehensive Impact Evaluation (CIE). Empowerment was assessed using indicators such as “the Page 9 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) women keep income”, and “women asked when property sold”. According to the ICR (Annex 7, and per the team’s comments), the CIE found that there was a positive difference when comparing women participating in COs activities with control groups (those women who did not participate COs). While most reported results were qualitative in nature, it is plausible to assume that the project positively contributed to women empowerment. However, more could have been done to strengthen the assessment. For example, the size and representativeness of focus groups could have been reported and metrics could have been constructed to measure the degree and extent of responses. Overall, and based on the above-mentioned assessment, the efficacy of this outcome is rated substantial despite attribution concerns, and relatively limited criteria used to assess empowerment. Rating Substantial OBJECTIVE 4 Objective To reach the marginalized population that has traditionally been excluded by reasons of gender, ethnicity, caste and location. Rationale Outputs  80% of the members of community organizations are women (target: 40%, exceeded).  57% of the members of income generating activities community organizations belong to targeted households (no target provided). Outcome The project data revealed that of the direct project beneficiaries 80% were women and 66% were ultra-poor (defined by lacking sufficient food for more than three months in a year). Evidence points to the success of the project in reaching women and marginalized groups including Dalit, Janajati, and Muslim, where 32% of the project-supported community organizations members were Janajati exceeding the target of 20%, 22% were Dalit exceeding the target of 20% and 3% were Muslim, no target provided for Muslims. The remaining members (43%) were from other castes/ethnic backgrounds including Brahmin, Chhetri and Thakuri (higher castes but received project support due to poverty criteria). According to the ICR (para 35), "fieldwork did not find any evidence of elite capture." Based on the above-mentioned information, it is evident that the project succeeded in reaching the marginalized population. Therefore, efficacy of this outcome is rated Substantial. Page 10 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Overall Efficacy is rated Substantial. The first sub-objective (to improve living conditions) is rated Substantial since the evidence provided points improvement in the living conditions of the targeted beneficiaries in project areas. The second sub-objective (to improve livelihoods) is rated Substantial. Project beneficiaries experienced income increases that exceeded the target, Food security for project beneficiaries improved compared to non-beneficiaries, and proxy indicators (health and education expenditure) were significantly higher for beneficiary households compared to control households. The third sub-objective (to improve levels of empowerment) is rated Substantial based on the results of the CIE and evidence in the ICR which pointed to improved participation of women in decision making. The fourth sub-objective (to reach the marginalized populations that have traditionally been excluded is rated substantial. Overall Efficacy Rating Substantial 5. Efficiency Economic and Financial Efficiency ex ante  Due to the project's demand-driven character, carrying out a detailed ex-ante cost-benefit analysis, making rate of return calculations was impossible at appraisal.  Instead, results from the first phase (PAF I) were used to guide the analysis, where livestock purchases were by far the most common income generation sub-project type selected by communities, representing about one quarter of all PAF I sub-projects.  The results showed that an estimated-projected and sustained increase in income to beneficiary households ranged from 10% to 12%, which translates to about a 20% to 25% rate of return.  The PAD (p. 17) emphasized that sub-projects chosen by the communities often reflect local priorities compared to nationally designed projects. This represents a dimension of efficiency that is seldom captured in standard analysis. Page 11 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) ex post  The ex-post economic and financial analysis (EFA) at completion found that the financial and economic returns to project investments in income generating activities (poultry, cow/buffalo, goat, piggery, retail business, vegetable farming, irrigation, micro hydro facility, water supply, and rice mills) and small infrastructures were 18% and 17%, respectively. The financial net present value (FNPV) and the expected net present value (ENPV) were about US$ 145 million and US$ 117 million, respectively.  A sensitivity analysis showed that both economic and financial returns declined due to the prolonged implementation period, but according to the ICR (Annex 4). the ERR and FRR, at 11% and 10%, respectively, remained "economically viable" despite the decline.  The EFA assessed the economic and financial viability of the project based on economic and financial data from a sample of 400 randomly drawn beneficiary households. It was carried out on the basis of individual analysis of selected income generating activities and infrastructure schemes for which reliable and sufficient data were available to measure project benefits.  Key assumptions. The project benefits were assessed over a 15 year period at 2018 financial prices, using the opportunity cost of capital at 10%. Main source of benefits from the project were returns from income generating sub-projects and small-scale village and community infrastructure. Financial prices of locally traded outputs and inputs were converted into economic prices by deducting direct subsidies and taxes.  Actual benefits would have been higher if the returns on employment had been considered. Also, the returns on investments from capacity development for earthquake rehabilitation and program management were excluded from the analysis due to lack of reliable data (ICR, para 38). Administrative and Institutional Efficiency The project closed on December 30, 2018, six years beyond the expected closing date on September 30, 2012. In total the project was implemented in about eleven years. This delay was to accommodate scaling up the project and to allow enough time to disburse funds from additional financing. The project management arrangements did not rectify an overlap in administrative power between the Vice Chair and Executive Director of PAF. According to the ICR (para 66) "this overlap of responsibilities became a serious constraint to efficient implementation and decision-making." Also, weak financial management capacity resulted in delays and quality issues related to implementation progress reports, including interim unaudited financial reports, and delays in settlement of advances, and a lack of proper registration of expenses (ICR, para 92). Hiring skilled technical staff was difficult because the project lacked a strong human resources system. this hindered the project's fiduciary and M&E systems (ICR, para 127). Overall, efficiency is rated substantial because at 17% the ex post ERR exceeded the opportunity cost of capital at 10%. The ex post EFA included logical assumptions and was robust enough to support the assigned rating. That said, the project suffered from weaknesses at the administrative and institutional levels as noted above. Efficiency Rating Substantial Page 12 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 68.00 ICR Estimate  17.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of Objectives is rated Substantial. Overall Efficacy is rated Substantial. The efficacy of the first and second sub-objectives are rated Substantial based on evidence the provided. The efficacy of the third sub- objective, to improve levels of empowerment, is rated Substantial based on the evidence provided in the Comprehensive Impact Evaluation (CIE) and the ICR which pointed to increased participation of women in decision making. Evidence justifies a Substantial rating for efficacy of the fourth sub-objective, to reach marginalized populations. Efficiency is rated Substantial, owing to a relatively strong EFA, despite institutional and administrative weaknesses. A split evaluation of outcome was not carried out in this review because the PDO remained the same, and only the outcome targets were expanded after the project received two AFs. Based on a Substantial rating for all three criteria (Relevance of Objectives, Overall Efficacy, and Efficiency), Outcome is rated Satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome The ICR (para 117) identifies two main risks that could potentially impact the development outcome of the project:  First, there are concerns related to the sustainability of the community organizations established by the project. By project completion the future of the whole PAF program was uncertain. PAF staff who were funded by the project had their contracts terminated and the PAF Secretariat remained with a small number of core staff under the Ministry of Land Management, Cooperatives and Poverty Alleviation. According to the ICR (para 120) the community organizations might be federated into cooperatives. Also, community organizations that were established later under the project might not Page 13 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) be sustainable as they only received support from the project for a short period of time (two to three years).  Second, transfer to new local level governments under the coordination of the Ministry of Land Management, Cooperatives and Poverty Alleviation. Local Governments became responsible for poverty related programs and were expected to continue to support the selected community organizations under the overall coordination of the Ministry of Land Management, Cooperatives and Poverty Alleviation. It is yet to be seen how Local Governments will support the established community organizations. This transition is risky due to the low capacity in some Local Governments. 8. Assessment of Bank Performance a. Quality-at-Entry The project was the second phase of the Poverty Alleviation Program (PAF) which was initially launched in 2002. The project was a scale up of the first phase and expected to cover 75 districts compared to 25 districts under the first phase. The project objectives were in line with the Bank and Country priorities at appraisal. Design featured a community driven development (CDD) approach and promoted the involvement and participation of ethnic and caste minorities. The project used the same targeting mechanisms and beneficiary profiles under phase I, but with extending scope nationally in a phased manner, increasing by 15 districts each year. Notable lessons from phase I and other CDD projects in South East Asia included: streamlined and transparent procurement and financial management rules, user-friendly operational manuals, and clear service standards for partners and project staff were all found to be essential for effective implementation, Community control of funds and investment decisions supported efficiency and ownership, and ensured accountability and sustainability. To encourage community contributions, there needs to be a measurable improvement in service delivery (PAD, 9). Three risks were identified at appraisal. However, the ICR did not discuss whether these materialized during implementation. The risk that the governing model of the country might change was not foreseen at appraisal. This had a negative effect on the project because the project's design and approach was not compatible with the federal governing model adopted by the Government. A notable design shortcoming was the lack of a clear consolidation/graduation strategy for community organizations (COs) as the project was designed that it would continue to expand and create new community organizations (ICR, para 103), without leaving enough time for these COs to mature and independently operate. Finally, M&E design was adequate, but some indicators were redundant (see section 9 for more details). Overall, there were moderate shortcomings in ensuring Quality at Entry, therefore Quality at Entry is rated Moderately Satisfactory. Quality-at-Entry Rating Moderately Satisfactory Page 14 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) b. Quality of supervision The project was implemented under a challenging political environment with changes at the political leadership and uncertainty around the project (ICR, para 108). The project benefited from a stable task team leadership which helped establishing confidence and enabled timely identification of the critical needs of the project. The ICR (para 172) stated that "missions were regularly conducted", but did not report on the number of missions conducted. The Bank team attempted to modernize the project based on the results of the Mid-Term Review. The team also raised the need to reform the PAF By-law (which regulates the PAF institutional and management framework) to improve its institutional efficiency. The project benefited from the Bank's technical support to enhance M&E capacity and strengthen impact evaluation analysis (ICR, para 105). The Bank team could have used the two additional financings (AFs) to reformulate the project, adapt its components and modality to post-conflict country needs and priorities(ICR, para 107), and to reform the PAF By-law. However, the Bank missed the opportunity to align the project design so that it would remain relevant in the post-conflict context. The project also could have benefited from stronger Bank support to strengthen fiduciary capacity. The weak fiduciary capacity contributed to an investigation by the Commission for Investigation of Abuse and Authority (CIAA – an apex constitutional body of the Government of Nepal (GoN) to investigate and probe cases against persons holding any public office, see section 10-b for more detail). A notable design shortcoming that should have been addressed by the team during AFs was the lack of a graduation/consolidation strategy for community organizations. According to the ICR (para 110) "the project kept expanding and creating new community organizations even towards the end without adequate resources for consolidation of results." The project team wrongly assumed that PAF as an institution of the government would continue beyond the project closure. However, the core function of PAF was transferred to the Ministry of Land Management, Cooperatives and Poverty Alleviation and the respective local governments. This new institutional arrangement disrupted the transition plans put in place by the Bank team and raised concern regarding the sustainability of some community organizations, particularly those established later in the project. Quality of supervision suffered from moderate shortcomings. Notably, the Bank could have used the AFs to better align the project to the shifting government priorities. Therefore, Quality of Supervision is rated Moderately Satisfactory. Overall, Bank performance is rated moderately satisfactory due to moderate shortcoming for both Quality at Entry and Quality of Supervision. Page 15 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The PAD did not include an explicit theory of change (ToC) as it was not required at the time of appraisal. Nevertheless, the ICR included a ToC that reflected the intended activities, expected outputs/intermediate outcomes, outcomes and long-term outcomes. The monitoring and evaluation of the project builds on the existing M&E system established under phase I. The existing system would further be optimized through a two-way integration of financial and operational data and incorporation of intermediate project performance indicators under the MIS. The PDO was to be assessed through four outcome indicators: (1) number of households to benefit from increased access to community infrastructure, (2) by project completion 25% of beneficiary households were to have increased their incomes by at least 15% against the base year, (3) 30% of beneficiary groups members were to have come from target groups, and (4) 30% of decision-making positions in community groups were to be occupied by target groups. These indicators were directly linked the PDO and measurable, but the RF lacked baseline data for any of them. The RF included twenty intermediate outcome indicators to assess the different activities supported by the project. Most of these indicators were measurable and linked to the project activities. However, some were mis-classified. For example: the following are properly output or at most intermediate outcome indicators: population benefiting from new rural infrastructure and 30% of beneficiary group members coming from targeted female-headed, hard core poor, Dalit or Janajati households. On the other hand, the indicator measuring the percentage increase in household income over baseline value" is an outcome level indicator rather than an intermediate level one. b. M&E Implementation M&E implementation was challenging due to the expansion of project activities, but with limited M&E staff capacity. According to the ICR (para 77) inadequate human resources hindered the proper implementation of the M&E system, particularly after the CIAA investigation caused a high level of M&E staff turnover." M&E data suffered from lack of accuracy and consistency throughout implementation (ICR, para 77). There were also challenges at the field level as community organizations were required to monitor their revolving funds, but at times to struggled to track certain indicators such as repayment rate (ICR, para 78). At a later stage of implementation community organizations received training to strengthen their bookkeeping capacity. Revisions to RF. The project had two AFs in 2011 and 2013, and three restructurings, where five indicator targets were revised upwards to reflect expansion of project coverage. Only one indicator saw a downward revision "project beneficiaries-nonmember households" was revised from 180,000 to 55,000. Page 16 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) The project team explained that non-member households declined because these households became members of COs. The ICR did not explain the reason this downward revision of 44%. The PDO indicators were unchanged throughout the project. PDO Indicator 3 (percentage of beneficiary group members coming from targeted groups) was dropped at the first AF and was integrated with the indicator for the representation of targeted groups for decision making. These changes were logical and necessary to accommodate the expansion in the project coverage. Overall, implementation was challenging due to limited capacity, and data suffered from accuracy and consistency issues. c. M&E Utilization According to the ICR (para 79) "the data on relevant indicators were collected systematically and the results framework was populated and reported to the Government of Nepal and the World Bank team." At the Mid-term Review, the decision to scale-up the project was based on the project's M&E data. The ICR (para 79) also reported that data was used "effectively" for the ICR itself and for the comprehensive impact evaluation. While the M&E system generated a sizable amount of data, it was not effectively processed and analyzed due to weak M&E capacity and high staff turnover. This limited the project management capacity to effectively engage the Government on policy and decision-making (ICR, para 79). Overall, the Quality of M&E is rated Modest. While design was adequate, implementation was challenging due to limited capacity, and data had accuracy and consistency issues. Utilization was limited due to limited processing and analysis of data. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as a category B. It triggered three safeguard policies at appraisal: Environmental Assessment (OP/BP 4.01), Indigenous Peoples (OP/BP 4.10), and Natural Habitat (OP/BP 4.04).The project was expected to deliver environmental benefits through enhanced capacity for environmental management, community investment in natural resources management, and investments that protect the health of the poor from environmental health risks. An Environmental Management Framework (EMF) prepared under phase I was revised to simplify the screening of community projects, provide more directly relevant advice for the mitigation of environmental impacts, and allow the project to expand operations into protected areas (PAD, para 52). The environmental screening procedure and guidelines were also revised during implementation to institutionalize environmental and social concerns in the 30 most common types of sub-projects. Also, a Vulnerable Community Development Plan (VCDP) and an Operational Manual were developed to improve targeting of vulnerable groups so as to increase their participation in decision-making processes (ICR, para 85). Page 17 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) OP/BP 4.36 on Forests was triggered at the second AF because there was a possibility of some infrastructure sub-projects being in or near community forests; similarly, some livestock sub-projects were expected to increase pressure on community or government forests. The EMF was revised again in 2017 to help community groups identify environmental and social risks, and link mitigation measures during the selection and preparation of plans, and construction of sub-projects (ICR, para 86). Also, a grievance redressal mechanism (GRM) was institutionalized and GRM committees were established in 43 of the 55 project districts. Safeguard compliance improved when the project hired an environmental and social/gender specialist to oversee safeguard management and mainstream gender. However, documentation of land donations to the project remained weak despite establishing a database for land acquisition records by the project. Finally, the ICR (para 83) reported that "compliance with Environmental and Social Safeguards was rated as “Moderately Satisfactory” in most implementation and supervision reports (ISRs)." However, details on specific mitigation activities were not reported in the ICR. b. Fiduciary Compliance Financial Management (FM). FM suffered from internal control weaknesses, delays in settlement of advances, and a lack of proper registration of expenses, among others (ICR, para 92). The weak capacity of the financing department staff and their frequent turnover resulted in quality issues and delayed submission of periodic trimester implementation progress reports, including interim unaudited financial reports. The appointment of an internal auditor was delayed. This delay contributed to systematic internal control weaknesses and lack of timely corrective measures (ICR, para 92). The Bank provided intense support and regular follow up to address FM weaknesses. According to the ICR (para 94) FM weaknesses improved over time. In a further communication the team explained that "all the projects in Nepal should go through annual audit by Independent audit through the Auditor General’s office." Investigation by the Commission for Investigation of Abuse and Authority (CIAA). According to the ICR (para 68 and 69) the project's weak fiduciary capacity triggered an investigation by CIAA-an apex constitutional body of the Government to investigate and probe cases against persons holding any public office. In late August 2014, CIAA started investigating field visit advances for PAF portfolio managers during the 2010, 2011 and 2012 financial years. CIAA alleged that some of the field expense receipts were forged, while others did not comply with government standards. PAF responded by explaining that in remote areas official receipts were non-existent and thus in the field several receipts had to be produced in the locally available format. In total, CIAA charged 33 PAF staff including the former Executive Director, the former Chief of Finance and Administrative Division, account officers and several portfolio managers with misuse of field travel funds in 2010, 2011 and 2012. In 2018, a special court convicted nine former PAF staff including the former Executive Director. Each former staff was charged for irregularities in field trips ranging from US$ 700 to US$ 3500. The Bank was informed that many of invoices and receipts associated with the field trips had been verified during the court process and the original charged amount was significantly reduced. New PAF management came in August 2015 and started working with the Bank on an action plan to improve fiduciary capacity. The ICR (para 67) reported that neither the Bank team nor the ICR team had access to the files and information of the CIAA investigation. Page 18 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) Procurement. Procurement management benefited from a dedicated procurement officer among the PAF staff. According to the ICR (para 98) "PAF performed satisfactorily in the implementation of the procurement plan agreed between the Borrower and the Bank." However, community organizations lacked capacity to maintain procurement registers and files (ICR, par 99). An ex-post review concluded that some contracts at the initial phase of the project were declared mis-procured. According to the ICR (para 100): "all these contracts were of small value" and "the Borrower reimbursed the amount that was disbursed by the Bank against the mis-procured contracts." c. Unintended impacts (Positive or Negative) According to the ICR (para 59): "The institutional building and empowerment process resulted in higher negotiation power. Beyond PAF activities, COs and their members perceived themselves as more capable to negotiate with non-government organizations, donors and local level government institutions. Similarly, by having access to loans at a reasonable rate, poor households started to use their bargaining power to negotiate better conditions with informal moneylenders. Through this economic empowerment process, several beneficiaries gained access to formal banking services that offered regulated interest rates and thus avoided predatory lending. As a result, according to the CIE report, the annual interest rate that the beneficiaries faced had declined significantly compared to that of the baseline and control groups." d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Overall Efficacy is rated Substantial and with the Moderately Outcome Satisfactory Substantial rating for Relevance Satisfactory and Efficiency, an overall Satisfactory is deemed justified. Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The ICR included 8 lessons. The following three are emphasized with some adaptation of language:  In changing socio-political environments, project design and implementation need to adapt over time in order to maintain consistency and relevance to the evolving Page 19 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) development challenges. Projects need to be designed and implemented with more flexibility in mind, which will allow the project to better adapt to the evolving environment. Having more frequent high-level meetings to discuss the relevance of the design in light of socio-political changes could help projects progress in line with the new institutional demands.  The post completion sustainability of community level institutions needs a clear plan to be in place at entry. The project experience showed that funding from development partners may not continue over the long-term. This calls for a consolidation strategy to be built into the design, and a future vision addressing how beneficiaries could graduate from the project and maintain sustainability.  A demand driven approach combined with direct funding is an effective tool in keeping target communities at the forefront and reaching out to the intended beneficiaries. The project encouraged communities to take initiatives to improve their livelihoods, particularly in organizing themselves into community organizations. The project also provided resources directly to the poor, thereby bringing in efficiency and transparency to the activities and ensuring community ownership. Using local knowledge and expertise enhanced the project's targeting strategy- an important issue given Nepal’s diverse ethnicity and languages. 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain This is the second and last phase of the Poverty Alleviation Fund (PAF) program. Assessing achievements on the ground would be useful to verify results in light of M&E weaknesses, and would provide an opportunity to generate relevant lessons that would be helpful in guiding similar programs in the future. 14. Comments on Quality of ICR Quality of evidence. The ICR acknowledged that M&E implementation was weak and that there were concerns on the accuracy and consistency of data. The ICR alternatively relied on the Comprehensive Impact Evaluation (CIE) to triangulate evidence. Quality of Analysis. The ICR provided clear linking between evidence and findings. Extent to which lessons are based on evidence and analysis. Lessons were based on the project experience although they could be tied more explicitly to project experience. Page 20 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NP: PAF II (P105860) Results Orientation. The ICR included a good discussion on outcomes despite concerns on the accuracy of the M&E data. Internal Consistency. Various parts of the ICR were logically linked and integrated. Consistency with guidelines. The assigned ratings in the ICR were justified and backed by sound arguments. Conciseness. The ICR provided thorough coverage of the implementation experience and candidly reported on shortcomings. There was enough clarity in the report’s messaging and the performance story was direct, well informed and tightly presented. The ICR could have reported on the number of supervision missions and the risks identified at appraisal whether any materialized and efficacy of mitigation measures. Overall, the ICR Quality is rated Substantial, despite minor shortcomings. a. Quality of ICR Rating Substantial Page 21 of 21