IEG Report Number: ICRR14717 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 06/23/2015 Country: Vietnam Project ID: P101608 Appraisal Actual Project Name: Vn-avian & Human Project Costs (US$M): 38.0 61.2 Influenza Control &prep L/C Number: Loan/Credit (US$M): 20.0 30.0 Sector Board: Agriculture and Rural Cofinancing (US$M): Development Cofinanciers: Board Approval Date : 03/13/2007 Closing Date: 12/31/2010 06/30/2014 Sector(s): General public administration sector (50%); General agriculture; fishing and forestry sector (21%); Health (21%); Other social services (7%); Solid waste management (1%) Theme(s): Rural services and infrastructure (25%); Other communicable diseases (25%); Natural disaster management (24%); Other social protection and risk management (13%); Health system performance (13%) Prepared by: Reviewed by: ICR Review Group: Coordinator: Katharina Ferl Judyth L. Twigg Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Technical Annex of February 14, 2007 (p. 10) the objective of Vietnam’s plan for the medium to long-term control of avian influenza is “ to reduce the health risk to humans from avian influenza by controlling the disease at source in domestic poultry, by early detection and response to human cases, and by preparing for the medical consequences of a human pandemic if it occurs over the next five years.” According to the Loan Agreement of April 12, 2007 (p. 5), the objective of the Vietnam Avian and Human Influenza and Human Pandemic Preparedness Project (VAHIP) was “to assist the Recipient to increase the effectiveness of public services in reducing the health risk to poultry and to humans from avian influenza in selected provinces, through measures to control the disease at source in domestic poultry, to detect early and r espond to human cases of infection, and to prepare for the medical consequences of a potential human pandemic.” This project was part of an adaptable program loan (APL) entitled the Global Program for Avian Influenza Control and Human Pandemic Preparedness and Response (GPAI). The GPAI was designed “to provide emergency financing to participating countries to contain outbreaks of avian influenza through early detection and rapid response measures” (Technical Note, p. 9). In June 2011, an Additional Financing was approved that supported a second phase to expand the scope of the project. The PDO did not change at the time of additional financing. This Review covers both phases of the project. This Review will use the PDO in the Loan Agreement as basis for the validation. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval: 06/29/2011 c. Components: The project contained three components: Component A: Highly Pathogenic Avian Influenza (HPAI) Control and Eradication in the Agricultural Sector (appraisal estimate US $ 17.2 million for Phase 1 and US$ 4.31 million for Phase 2, actual US$ 21.51 million): This component was to finance: i) strengthening of veterinary services; ii) improving disease control; iii) performing surveillance and epidemiological investigations and strengthening of the disease surveillance system ; iv) supporting the Ministry of Agriculture in analyzing and preparing for the restructuring of the poultry sector; and v) developing and implementing emergency outbreak containment plans. Component B: Influenza Prevention and Pandemic Preparedness in the Health Sector (appraisal estimate US $ 16 million for Phase 1 and US$ 16.28 million for Phase 2, actual US$ 32.28 million): This component was to finance; i) improving the technical quality of the surveillance and response systems; ii) improving the technical quality and efficiency of curative care readiness; iii) supporting behavior change and risk communications; and iv) strengthening the preventive health system at the local level. Component C: Integration and OPI (National Integrated Operational Program for Avian and Human Influenza , 2006-2010) Coordination, Results Monitoring and Evaluation (M&E), and Project Management (appraisal estimate US$ 4.8 million for Phase 1 and US$ 4.41 million for Phase 2, actual US$ 9.21 million): This co mponent was to finance: i) supporting the integration and coordination of the National Avian and Human Influenza Program; ii) performing Monitoring and Evaluation (M&E) for the project and for national results; and iii) managing the project at the central and provincial levels. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Costs: The project was estimated to US$ 38 million for Phase 1 and US$ 25 million for Phase 2. Actual total cost was US$ 61.16 million (97.07% of appraisal); according to the Project Team, cancellation of some funds was made possible due to savings in procurement and training toward the end of the project period. Actual costs by component reported by the ICR (p. 30), and also reported above in Section 2c, do not appear to take these cancellations into account. Financing: According to the Technical Annex (p 69), the first phase (VAHIP 1) of the project was supposed to be financed as follows:  A US$ 20 million IDA credit, of which US$ 624,266 was cancelled.  A US$ 10 million Avian and Human Influenza Facility Trust Fund Grant, of which US$ 51,240 was cancelled.  A US$ 5 million Policy and Human Resources Development Trust Fund Grant, of which US$ 721,915 was cancelled.  A US$ 3 million Recipient contribution. Since VAHIP 1 was considered successful, additional financing was made available for a second phase. VAHIP 2 was financed as follows:  A US$ 10 million IDA credit, of which US$ 1.13 million was cancelled.  A US$ 13 million Avian and Human Influenza Facility Trust Fund Grant, of which US$ 1,104 was cancelled.  A US$ 2 million Recipient contribution. Borrower Contribution : At appraisal, the estimate for the Recipient’s contribution was US$ 3 million. The Recipient added another US$ 2 million for the implementation of VAHIP2. Actual disbursement was US$ 5 million (100%). Dates: There were three restructurings:  On July 19, 2010 funds were reallocated among disbursement categories and the closing date for VAHIP 1 was extended from December 31, 2010 to June 30, 2011.  On June 14, 2011 the closing date for VAHIP 1 was extended from June 30, 2011 to December 31, 2012.  On June 29, 2011 indicators were revised to reflect extended project activities that could be implemented due to additional financing. The second phase (VAHIP 2) had a closing date of June 30, 2014. 3. Relevance of Objectives & Design: a. Relevance of Objectives: High: The objective was highly relevant given the global threat Highly Pathogenic Avian Influenza (HPAI) H5N1 presented to health and economies. In 2003, when the HPAI H5N1 was detected in Vietnam, within four months the virus spread to 57 out of the 67 provinces. The economic cost was estimated at US$ 250 million (0.5% of Vietnam's GDP). The threat to Vietnam from avian influenza and similar pathogens remained high. The assessment of high relevancy was fully consistent with international expert consensus. The objective was in line with Vietnam’s plans to control the avian influenza and other zoonotic diseases threats as stated in the National Integrated Operational Program for Avian and Human Influenza (2006-2010) and the Integrated Operational Program on Avian Influenza, Pandemic Preparedness, and Emerging Infectious Diseases (2011-2015). Although avian influenza is not explicitly cited in the Bank's Country Assistance Strategies at appraisal or at closing, the objective was in line with the elements of the Bank’s CAS at closing (2012-2016) that aim to decrease Vietnam's vulnerability by supporting risk mitigation to preserve the country's high economic growth. b. Relevance of Design: High: The planned activities were logically and plausibly linked to the achievement of the project's objectives: support for the government to increase the effectiveness of public services in reducing the health risk to poultry and to humans from avian influenza. The planned activities included the supply of laboratory equipment required for the testing of the avian influenza virus, improving the biggest wholesale poultry market near Hanoi, and performing monthly disease surveillance and investigation. Activities related to the early detection and response to human cases of infection included training to improve the reporting time of new outbreaks. Health staff were to be trained to prepare for the medical consequences of a potential human pandemic. Also, the project addressed the risks related to backyard poultry. The causal relationships between these planned interventions and underlying assumptions about how program actions would lead to intended outcomes were logical and properly laid out. The Bank used an emergency operation, which was appropriate given the situation. 4. Achievement of Objectives (Efficacy): Assist the Recipient to increase the effectiveness of public services in reducing the health risk to poultry and to humans from avian influenza in selected provinces , through measures to control the disease at source in domestic poultry, to detect early and respond to human cases of infection , and to prepare for the medical consequences of a potential human pandemic . Increase the effectiveness of public services in reducing the health risk to poultry : High Outputs  80 demonstration sites demonstrated biosecure poultry production to 1,760 farmers, surpassing the target of 25 demonstration sites.  Veterinary services on disease diagnostic and surveillance were strengthened, meeting the revised target value of 8 laboratories which meet the international standard for Avian Influenza testing. Outcomes  All suspected Highly Pathogenic Avian Influenza (HPAI) cases were investigated. The original target of 275 HPAI reports could not be achieved due to lower disease prevalence than planned.  All upgraded markets and slaughterhouses applied practice according to project guidelines, achieving the target.  All poultry traders at the Ha Vy market (a live poultry wholesale market near Hanoi) applied good biosecurity practices, achieving the target.  The reporting time of new outbreaks and return of laboratory confirmation to the affected commune was reduced from 8.7 days in 2006 to 2.4 days for veterinary health, surpassing the target of 4 days.  7.66% of samples at markets and slaughterhouses were positive for the H5N1 virus. The target of 2% was not met. It took an average of one day to completely contain suspected outbreaks, surpassing the target of two days. The ICR states that the target was ambitious and that the disease prevalence outcome was beyond the project’s control. Similarly, an IEG study [World Bank Independent Evaluation Group (2014) "Responding to Global Public Bads: Learning from Evaluation of the World Bank Experience with Avian Influenza, 2006-2013"] specifically cautioned against the use of prevalence indicators because of challenges with attribution and control. Increase the effectiveness of public services in reducing the health risk to humans from avian influenza in selected provinces: High Outputs  98.6% of reports were accurately completed and sent on time to the Provincial Preventive Health Centers, surpassing the original target of 84% and the revised target of 90%.  97.7% of reports were accurately completed and received at the District Preventive Health Center, surpassing the target of 90%.  All provinces developed the Pandemic preparedness plan based on the guidelines by the Ministry of Health, achieving the target.  All district hospitals developed the pandemic preparedness plan, achieving the target.  68 multisectoral simulation exercises were conducted and reviewed at district levels in the project provinces, surpassing the original target of 17 and the revised target of 30.  69,012 health staff was trained, surpassing the revised target of 21,905. There was no original target.  20,532 health staff were trained on behavior change and risk communication skills. Communication material was distributed to the general population and to ethnic minority groups. Outcomes  The reporting time of new outbreaks and return of laboratory confirmation to the affected commune was reduced from 10 days in 2006 to 3.4 days for human health, surpassing the target of 4 days.  86.2% of the target population can accurately identify and have practiced at least one curative behavior, surpassing the original target of 40% and the revised target of 60%.  88% of the target population can accurately identify and have practiced at least one preventive behavior, surpassing the original target of 40% and the revised target of 60%.  98.8% of the general population can accurately identify and have practiced at least one key preventive behavior.  87 District Preventive Medicine Centers in 11 provinces were fully equipped and have adequate capacity to implement their responsibilities and functions in compliance with Ministry of Health decisions on preventive medicine, surpassing the original target of 16 and the revised target of 79. 5. Efficiency: Substantial The ICR conducted a cost benefit analysis by assuming that if an outbreak in poultry would happen in 2015, it would cost Vietnam’s economy US$ 945 million (based on costs of outbreaks in 2004 and 2013). If an outbreak could be prevented, the benefit would be US$ 945 million. The analysis takes a more conservative approach which assumes that only 10% of the cost could be prevented through the public health system. Therefore, the benefit of prevention would be US$ 95 million. The analysis also assumes that Vietnam’s government spending is US$ 77 million in 2015 and that averted losses in the poultry sector are 10%. Therefore the expected annual rate of return on the investments in veterinary and human public health is 29%. If averted losses in the poultry sector were 20%, the internal rate of return would be 129%. The analysis demonstrates that the investment had high rates of return. However, it should be noted that the assumed benefits only materialize if a pandemic occurs, which is implicitly weighted at 1% probability. Moreover, the calculations assumed that the project would reduce pandemic impact by only 10%; thus in any year the benefit is one tenth of one percent of pandemic impact. The ICR adopted a conservative approach by assuming that probability of the occurrence of a pandemic is 1% per year to calculate the expected economic rate of return of the project. The project would have had an even higher expected rate of return if, for instance, the assumed probability was 2% (once-in-50 years event), which is used by some analyses. Delays in disbursement and implementation of activities during the early phases of the project were indicative of inefficiencies in the use of the Bank’s financial resources (see Section 9b). However, according to the ICR (p. 26), the implementation units collaborated effectively across different sectors and across government levels. They identified new opportunities for efficiency gains, such as not paying for new customized surveillance software but rather adapting the surveillance software that had been implemented through a project financed by the Asian Development Bank. This measure led to cost savings. Efficiency is therefore rated Substantial. a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate Yes 29% 100% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Relevance of objectives is rated High, as the objectives were closely linked to the threat posed to the country by Avian Influenza and Bank and country strategy. Relevance of design is also rated High, as the project's planned activities were plausibly and tightly linked to anticipated outcomes. Achievement of the objective to increase the effectiveness of public services in reducing the health risk to poultry is rated High, as targets for upgrading markets and slaughterhouses as well as laboratory response were met. Achievement of the objective to increase the effectiveness of public services in reducing the health risk to humans from avian influenza in selected provinces is rated High, as targets for pandemic preparedness planning and education/training, reporting, and equipping of preventive medicine centers were exceeded. Efficiency is rated Substantial, with high estimated rates of return and evidence of cost-effective investment of Bank resources, despite delays in disbursement and implementation of activities early in the project's lifetime. Taken together, these ratings lead to an Outcome rating of Highly Satisfactory. a. Outcome Rating: Highly Satisfactory 7. Rationale for Risk to Development Outcome Rating: This project was an emergency project with the specific objective to support Vietnam in increasing the effectiveness of public services in reducing the health risk to poultry and to humans from avian influenza. The human and veterinary public health systems have improved significantly. The government continues to be committed to this work. However, Vietnam will face a few challenges. It may be challenging to mobilize financial resources for the continuous prevention of disease outbreaks. Also, it may be challenging to maintain the operating system that has already been built and continue to increase veterinary and human public health capacities. The ongoing Integrated National Operational Program on Avian Influenza, Pandemic Preparedness, and Emerging Infectious Diseases (2011-2015) also addresses influenza and other zoonotic diseases. However, due to the continuous decline in financial assistance by donors, sustainability presents a challenge. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The Bank team prepared an emergency project that was responsive to the government’s needs. The project was built on lessons learned from the implementation of the Bank’s Vietnam Avian Influenza Emergency Recovery Project but also more generally from the Bank’s experience in emergency response to disasters and the global HIV/AIDS program. The Bank team collaborated with other experts and development partners. Also, the Bank team ensured that best practice examples, including the Global Program for Avian Influenza Control and Human Pandemic Preparedness and Response, informed project design. During project preparation, several risks were identified, including political commitment, slow disbursement and procurement, weak capacity for project implementation on the provincial level, insufficient coordination between the Ministry of Health and the Ministry of Agriculture and Rural Development, reluctance to innovation in the Ministry of Health, insufficient farm-level surveillance and increased virus circulation. The ICR (p. 6) states that mitigation measures were appropriate and effective. According to the project team, the Bank was intensively engaged in the preparation of this project to mitigate the risk of insufficient coordination between the two ministries. The Bank supported the government to formulate successive national integrated (animal health-human health) plans and linked the project to the global avian flu response in order to ensure political commitment. Also, to address capacity concerns at the provincial level, provincial staff obtained training on Bank procedures. Quality-at-Entry Rating: Satisfactory b. Quality of supervision: The Bank team provided technical assistance at the provincial levels and engaged with counterparts who were not familiar with Bank projects. Risks that were identified during project preparation were sufficiently mitigated. Fiduciary and safeguard aspects were addressed satisfactorily. The Mid-Term Review took place with only a three-month delay. Results from Monitoring and Evaluation (M&E) were used to adapt project activities. Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The government showed strong ownership and commitment to the implementation of the project. There was ongoing engagement with stakeholders and development partners throughout implementation. The government also shared its experience fighting pandemics with the international community and hosted an international conference on animal and pandemic influenza. The government developed plans for pandemic response across several sectors, including the health sector, and across different levels of government. When plans from the provinces and districts were not prepared satisfactorily by the end of VAHIP 1, the government rejected them, and the plans were further developed and resubmitted at a higher quality by the end of VAHIP 2. According to the ICR (p. 7), M&E data was used to inform decision makers. For example, the indicators of disease prevalence were used to guide adjustments in surveillance activities and biosecurity investments in markets. According to the ICR (p. 23), the government will need additional financial and technical assistance to continue building capacity in disease prevention and control. Government Performance Rating Satisfactory b. Implementing Agency Performance: The project had two Implementation Units. The Ministry of Health was responsible for the human health component of the project, while the Ministry of Agriculture and Rural Development was responsible for the animal health component. The ICR (p. 25) states that there were difficulties due to decentralized implementation, such as in financial management., but that flexibility among the implementing units enabled effective and prompt corrective action. The ICR (p. 6) also states that there were delays during the first phase of the project, especially in procurement but also in other areas, which was common in the country due to slow processing of documentation by the government. According to the ICR (p. 26), the implementation units collaborated across different sectors and across government levels and aimed to identify new opportunities for efficiency gains such as not paying for a new customized surveillance software but rather adapting the surveillance software which had been implemented through a project funded by the Asian Development Bank. This measure led to cost savings. The implementation units utilized M&E data for decision making, such as in the field of animal health-human health interface. Implementing Agency Performance Rating : Satisfactory Overall Borrower Performance Rating : Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The M&E framework of the project was aligned with the government’s monitoring and reporting system as well as the M&E framework of National Integrated Operational Program for Avian and Human Influenza. The causal chain between input and outcomes was clear. The Results Framework in the Technical Annex (pp. 39-40) included three PDO indicators and five intermediate outcome indicators. The selected indicators presented some challenges as a few had definition problems, measured disease outcomes that were beyond the control of the project, were difficult to collect, or had unrealistic target values. For example, final outcome indicators related to infectious diseases present a challenge due to influences that are outside the control of individual projects. However, this challenge applied only to a few indicators for this project, and there were sufficient intermediate outcome indicators to assess the achievement of the objectives (which were, in any event, geared toward improvements in the effectiveness of public services rather than directly toward decreasing disease incidence or prevalence). The ICR does not comment if the proposed data collection methods and analysis were appropriate for the purpose and if the M&E design was well embedded institutionally and had sufficient stakeholder ownership. b. M&E Implementation: The ICR (p. 7) states that indicators were reported on a regular basis, sometimes with delay, and were of good quality. Several indicators were adapted during implementation as priorities for monitoring changed. In June 2011 indicators were revised to reflect extended project activities that could be implemented due to additional financing. c. M&E Utilization: The ICR (p. 7) states that M&E results and reports were used to inform decision makers and to adapt project activities. According to the ICR (p. 7) indicators on government performance were useful in tracking progress. According to the project team, M&E results and analysis were shared with stakeholders at the provincial level on a regular basis. Also, M&E results were shared with several hundred stakeholders from provinces and districts at the final project workshop. At project closing, no follow-on interventions were planned, and therefore the M&E framework of the project was not continued. M&E Quality Rating: Substantial 11. Other Issues a. Safeguards: The project was rated as environmental category B. The project triggered the Bank’s safeguards policies on Environmental Assessment (OP 4.01), Indigenous Peoples (OP 4.10), and Involuntary Resettlement (OP 4.12). According to the ICR (p. 8), compliance was satisfactory for all policies. Also, an Environmental Management Plan was prepared to address potential impacts. Updates on the Implementation of the Environmental Management Plan were submitted for revision to the Bank on a regular basis, and the Plan was implemented in a satisfactory manner. b. Fiduciary Compliance: Financial Management : The project’s financial management complied with the Bank’s financial management policies and procedures. The Project Implementation Units in the Ministry of Health and Ministry of Agriculture and Rural Development were responsible for budget approval, financial reporting, and audits. All other financial management responsibilities were with the eleven provinces, each of which had a Project Implementation Unit for the human health component and one for the agriculture component. The Provincial Implementation Units received their funding on a local currency account from the national project implementation units and had to submit expenditure reports on a monthly basis. The Project Implementation Unit in the Ministry of Health submitted interim financial reports to the Bank on a quarterly basis. An independent firm submitted financial audits which were unqualified annually and in accordance with the Finance Agreement. According to the project team, submission of financial reports by the Ministry of Agriculture and Rural Development was fully satisfactory, and there was compliance with all other actions required by the Finance Agreement. Procurement: Until the Mid-Term Review there were delays in procurement due to the centralization of procurement activities related to the animal health component on the national level. These delays was addressed by moving more responsibilities to the provincial level. The ICR (p. 9) states that procurement activities were implemented in accordance with the Financing Agreement. c. Unintended Impacts (positive or negative): Systematic improvements made under this project such as better surveillance, pandemic plans, availability of personal protective equipment, and training on infection control, are also applicable to other disease threats. Also, in many districts in several provinces, local authorities improved the physical facilities of the district preventive medicine centers to complement project investments. Thus there was a catalytic effect to mobilize additional resources from provincial budgets for public health systems. While the project provided training and equipment, provinces financed the renovation of facilities, even construction of new ones in some cases. This greatly improved conditions for good functioning of the preventive medicine centers in many districts. d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Highly Satisfactory Highly Satisfactory Risk to Development Moderate Moderate Outcome: Bank Performance: Satisfactory Satisfactory Borrower Performance : Satisfactory Satisfactory Quality of ICR: Satisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR (p. 26) identifies valuable lessons, including:  In case of an emergency situation a move to a systematic improvement is critical for achieving better long-term outcomes. Vietnam took advantage of the situation to get political and financial support to improve its veterinary and human health system which is critical for addressing future disease outbreaks.  Cross-sectoral approaches are critical but require good leadership. The collaboration between the Ministry of Health and the Ministry of Agriculture and Rural Development facilitated information sharing, joint analysis of results and efficient use of resources. However, this required the support of senior leaders from both ministries.  Decentralization of project activities to lower government levels might be challenging but may lead to positive long term impacts. In this project, the decentralization of activities to the provincial and district levels built capacity in various areas. Also, the decentralization led to more project ownership at the lower government levels. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR provides a large amount of information and a good overview of the project. The ICR is internally consistent. It complies with OPCS guidelines and provides useful lessons that are clearly derived from the main text's information and analysis. However, the ICR is not sufficiently candid and provides little information in regards to implementation challenges. The ICR mentions initial delays due to procurement challenges but does not provide information on what kind of challenges and how they were overcome by the Bank Team and the government. a.Quality of ICR Rating : Satisfactory