ICRR 12408 Report Number : ICRR12408 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 06/15/2006 PROJ ID :P035770 Appraisal Actual Project Name :Pilot Reconstruction Project Project Costs 54.2 83.7 US$M ) (US$M) Country :Azerbaijan Loan/ US$M ) Loan /Credit (US$M) 20.0 18.7 Sector (s):Board: ): UD - Power (36%), US$M ) Cofinancing (US$M) General agriculture fishing and forestry sector (32%), Central government administration (14%), General education sector (10%), Other social services (8%) L/C Number :C3109 FY ) Board Approval (FY) 99 Partners involved : Closing Date 12/31/2001 09/30/2005 Evaluator : Panel Reviewer : Division Manager : Division : Soniya Carvalho Ridley Nelson Alain A. Barbu IEGSG 2. Project Objectives and Components a. Objectives The objectives of the Azerbaijan Pilot Reconstruction Project (APRP) were to: (i) Facilitate the repatriation of people that have been 'internally displaced' (IDPs) as a result of the conflict over Nagorno-Karabakh; (ii) Raise the standard of living for IDPs . This was to be achieved through : the development of an effective approach to the voluntary resettlement and rehabilitation of IDPs; the establishment and strengthening of agencies in Azerbaijan involved in the voluntary resettlement of IDPs, demining, and reconstruction of areas affected by the conflict over Upper Garabagh; and assistance in raising and coordinating support from international donors and Non-Governmental Organizations (NGOs). The APRP project objectives were not revised . However, APRP was followed by a Supplemental Credit which aimed at fixing the weaknesses in the design of the APRP . The objective of the Supplemental Credit as stated in the Memorandum and Recommendation of the President for the Supplemental Credit, page 4 was to "improve the living conditions and living standards of the most vulnerable groups of IDPs ", but the Agreement Amending Development Credit Agreement does not mention vulnerable groups (something the ICR should have flagged as noted in Section 11 below). The APRP was to finance medium-scale infrastructure while the Supplemental Credit was to finance community-driven microprojects (including microcredit projects). While the ICR notes that the project "intended to further a broader range of policy and sectoral objectives " including "privatization of housing and land, development of rural micro-finance institutions, reform of health and education, improvement in power distribution management and cost recovery for electricity from IDPs " (ICR page 6), these were not identified as objectives in the PADs of either the APRP or the Supplemental Credit. This ICR Review does not, therefore, assess the project's achievements against these broader objectives. Nevertheless, greater clarity in the ICR about objectives was warranted (and is noted in Section 11 below). b. Components (or Key Conditions in the case of Adjustment Loans ): The project had six components (figures do not add up to totals because the recurrent costs are not included - ICR, page 23): Relocation: (at appraisal US$1.8 million; actual US$1.8 million) to facilitate the return of about 20,000 IDPs to their former homes by financing a program to provide cash support to returning IDPs . Shelter: (at appraisal US$14.0 million; actual US$18.1 million) to reconstruct or repair about 3,500 houses destroyed or damaged during the war. Employment and Income Generation: (at appraisal US$10.9 million; actual US$15.5 million) (a) agriculture – to help IDPs in three areas start, develop, and sustain agricultural based activities by providing inputs, rehabilitating irrigation infrastructure, and providing support services; (b) micro-credits and small grants – to assist returning IDPs in the starting up of small businesses; (c) public works, environmental mitigation – to finance labor-intensive public works projects in small towns and villages . Health and Education: (at appraisal US$5.1 million; actual US$12.1 million) (a) health – to strengthen the delivery of primary and preventive care services by repairing and rehabilitating health care facilities, providing essential drugs, medical supplies and equipment, and financing training, capacity building, and Monitoring and Evaluation (M&E); (b) education – to increase school attendance through the structural repair and rehabilitation of educational facilities, supply of learning materials, and finance costs of training, capacity building, and M&E . Infrastructure and Energy: (at appraisal US$14.5 million; actual US$25.7 million) (a) infrastructure – to repair and rehabilitate water supply systems, transport and telecommunication infrastructure, and finance the costs of capacity building and M&E; (b) energy – to repair and rehabilitate power transmission lines, transformer stations, and other equipment. Institutional Support: (at appraisal US$5.3 million; actual US$9.5 million) to contribute to the search for the most efficient and sustainable ways for the repatriation of the remaining refugees in Azerbaijan by financing operations of Azerbaijan Reconstruction and Rehabilitation Agency (ARRA), creation of the Azerbaijan Demining Agency (ANAMA), and creation of the IDP Relocation Agency . In 1999 the Bank agreed to provide additional US$ 10 million for Supplemental Credit to create a ‘Social Fund for the Development of IDPs’ (SFDI). The original APRP components were expanded through the Supplemental Credit with the following new components (US$ amounts refer to total project cost and hence add -up to more than US$ 10 million which was the Bank financing -- see section 2c below, point 1): Micro-Projects (at appraisal US$7.0 million; actual US$7.2 million) to finance micro-projects with social and economic objectives in IDP communities in all districts of Azerbaijan Micro-Credits (at appraisal US$2.1 million; actual US$2.0 million) to finance provision of micro-credits through intermediary institutions to IDP owned micro -enterprises and individual borrowers Operational Support - Support to SFDI, Technical Assistance and Operating Costs - (at appraisal US$1.6 million; actual US$1.8 million) to finance operating costs of the SFDI, Technical Assistance (TA) and consulting services. c. Comments on Project Cost, Financing, Borrower Contribution, and Dates 1) Actual project cost (see section 1 above which excludes supplemental credit ) exceeded appraisal project cost by US $29.5 million. This difference resulted from significantly higher unanticipated donor contributions through parallel financing. For the Supplemental Credit, actual total cost was US$ 11 (at appraisal US$10.7); actual Bank financing was US$10.1 (at appraisal US$10). 2) According to the ICR, by mid-term review in Spring 2001, the original APRP IDA-financed work program had been largely completed, however overestimation of investment costs and savings from competitive and transparent bidding procedures meant that actual costs were about half of the estimated costs . At that time it was decided to include more districts. 3) The original closing date for APRP was December 2001. The Supplemental Credit was signed in August 2000 and the closing date extended to March 2003. The project was subsequently extended four more times (till September 2005), to allow for implementation of the Supplemental Credit and for the use of the APRP savings for similar investments in additional districts . Implementation of the Supplemental Credit was delayed because of initial hastiness in its preparation, for example, the community co -financing threshold that was set by the Supplemental Credit proved to be too high for the IDP communities with unstable incomes, which resulted in delays in the implementation of micro-projects under the Supplemental Credit . Significant delays also arose because of donor coordination problems relating to the parallel financing by the UNDP, UNHCR, and EU --varying funding sources, funding timetables, and institutional priorities and constituencies (ICR, page 6). 3. Relevance of Objectives & Design : Project objectives of the APRP and Supplemental Credit were broadly relevant . Their objectives were aligned with the Bank’s Country Assistance Strategy (CAS) objectives for Azerbaijan (for both the CASs 1996 and 1999). Objectives were also relevant in the country ’s post-conflict environment, and were supported by the Government as part of the broader “Priority Program for the Resettlement and Reconstruction of the Liberated Territories ‿ prepared by the Government and donor community . Repatriation and improvement of living conditions for IDPs were seen as crucial for the country’s greater political stability and had the support of the international donor community (ICR, page 6). The overall relevance of the design of the APRP and Supplemental Credit was modest . The APRP included both macro- and micro-level activities and was too complex for a single project . Recognizing these limitations, the Bank designed the Supplemental Credit to take over the micro -level activities from the APRP. The addition of the separate Supplemental Credit reduced the complexity of the APRP, but the design of the Supplemental Credit also had flaws (see Section 2c above). Borrower comments state that the quality of the Supplemental Credit in the initial stage was unsatisfactory. Later modifications improved the relevance of the design of the Supplemental Credit . 4. Achievement of Objectives (Efficacy) : (i) Facilitate the repatriation of people that have been 'internally displaced' (IDPs) as a result of the conflict over Nagorno-Karabakh (Rated Substantial). According to the ICR, about 33,000 IDPs--against an original (pre- cost reallocation--PAD target of 20, 000 IDPs) had returned home by end 2005. The revised target after expansion of coverage to additonal districts is not known --overestimation of investment costs and savings from competitive and transparent bidding procedures meant that actual costs were about half of the estimated costs and the savings were used to expand coverage in additional districts . (ii) Raise the standard of living for IDPs (Rated Substantial). The ICR presents numbers to show that most project indicators relating to living standards were met or exceeded (ICR, Annex I), and the ICR also states that the project area is now self sufficient in grains, milk and other agricultural produce, and that farm incomes have increased, and infrastructure has been set in place and services expanded (ICR, page 7). The Region subsequently commented that these conclusions are based on systematic monitoring and evaluation evidence . 5. Efficiency : The efficiency of the projects is not known as the ICR presents no efficiency measures . Economic rate of returns (ERRs) were neither estimated for this project at appraisal nor at completion . While anticipated units costs were presented in the PAD, they were not re -calculated at the ICR stage. The ICR does discuss the cost savings resulting from the significant initial overestimation of project costs, but there is no analysis to indicate that the particular level of cost savings achieved was adequate /justified. 6. M&E Design, Implementation, & Utilization: While the performance indicators identified in the PAD are appropriate overall and the ICR provides actuals for most of the indicators identified in the PAD, the ICR fails to indicate the source /quality. Hence, it is difficult to ascertain the quality of monitoring and evaluation . The recommended PPAR will, inter alia, address this question . 7. Other (Safeguards, Fiduciary, Unintended Impacts--Positive & Negative): The PAD does not mention any safeguard issues and the ICR also does not report on any . 8. Ratings : ICR ICR Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory But the extent of project efficiency is not known. Institutional Dev .: Substantial Substantial Sustainability : Likely Non-evaluable There is insufficient evidence to make a judgment. More evidence on the ability and willingness of the entities assuming responsibility for O&M to actually undertake O&M is needed. The evidence presented in the ICR is inadequate to conclude that the two projects raised standards of living in a sustainable manner. Bank Performance : Satisfactory Satisfactory But only moderately so given significant shortcomings in quality at entry, and some weaknesses in supervision (due to high task manager turn-over etc). The ICR also rates Bank Performance moderately satisfactory. Borrower Perf .: Satisfactory Satisfactory Quality of ICR : Unsatisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness . 9. Lessons: As in the ICR: - emergency reconstruction operations should not be combined with broad sectoral policy reform efforts that require different kinds of interventions and institutional set -ups. -operations involving multiple donors must be carefully designed to allow for different agendas, financing arrangements, and funding timetables among donors . This is particularly important in emergency -type operations. 10. Assessment Recommended? Yes No Why? To verify the ratings. 11. Comments on Quality of ICR: The ICR has significant weaknesses : 1) Performance Indicators Data: The ICR provides no indication of the source /quality of the reported performance indicators data making it difficult to assess project outcomes . The ICR does not present the methodology /findings of the beneficiary assessments or the other studies that the Region indicates were undertaken . 2) The ICR presents no efficiency measures . 3) Statement of Objectives and Project Design : The ICR talks about a range of policy and sectoral "objectives", such as privatization of housing and land, reform of health and education, etc ., but these are not stated as "objectives" in the APRP PAD. Furthermore, the ICR does not mention the discrepancy between the Memorandum and Recommendation of the President for the Supplemental Credit versus the Agreement Amending Development Credit Agreement (while the former mentions as the target group "the most vulnerable IDPs", the latter does not-- see Section 2a above).