ICRR 13797 Report Number : ICRR13797 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 10/26/2012 Country : Moldova Project ID : P099841 Appraisal Actual Project Name : Avian Influenza US$M ): Project Costs (US$M): US$10.60 US$10.09 Control & Human Pandemic Preparedness & Response Project L/C Number : C4188; CH233 Loan /Credit (US$M): Loan/ US$M ): US$8.00 US$8.90 Sector Board : Agriculture and Rural US$M): Cofinancing (US$M ): US$1.85 US$1.85 Development Cofinanciers : GTZ, USAID, Japan Board Approval Date : 06/09/2006 PHRD Grant, EC Closing Date : 12/31/2009 05/15/2011 Sector (s): General public administration sector (43%); Health (39%); Agricultural extension and research (12%); Animal production (3%); Other social services (3%) Theme (s): Rural services and infrastructure (29% - P); Other communicable diseases (29% - P); Pollution management and environmental health (14% - S); Health system performance (14% - S); Participation and civic engagement (14% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Stephen Hutton Judyth L. Twigg Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: The project development objective in the Project Appraisal Document (PAD) is stated differently in different locations. On page 4, the PAD states that "the overall objective of the Project is to minimize the threat posed to humans by HPAI [Highly Pathogenic Avian Influenza ]." In the cover sheet of the PAD, the stated objective is to "minimize the threat posed to humans by Highly Pathogenic Avian Influenza (HPAI) and other infectious diseases that originate in animals." The objective as stated in the Financing Agreement (p. 4) is to "minimize the threat posed to humans by highly pathogenic avian influenza infection in domestic poultry and prepare for, control, and respond to an influenza pandemic and other infectious disease emergencies in humans ." The project will be evaluated against the Financing Agreement objectives, as they more clearly state the dual goals of reducing the threat of infection from animals to humans and of preparing for influenza pandemics or other diseases transferred between humans. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: Component 1. Animal Health . (Appraisal US$3.6 million; Actual US$4.0 million). This component included: 1. Animal disease surveillance and diagnostics to strengthen the Republican Center of Veterinary Diagnosis (RCVD) and two regional laboratories. The main investments were equipment and reagents /consumables. There was technical assistance to build capacity in database programs and data analysis . There was training in collection and analysis of epidemiological data and on risk assessments . There was technical assistance to build capacity in serology and virology. 2. Animal HPAI control and outbreak containment, including the culling of infected and at -risk poultry and disposal of carcasses . The component also promoted biosecurity and controls on the movement of birds and products. It trained and equipped staff and provided protective clothing . It was to support technical assistance to review, assess, and recommend improvements in plans for preparedness . 3. A Compensation Fund to encourage farmers to report possible outbreaks of Avian Influenza . Component 2. Human Health . (Appraisal US$3.7 million; Actual US$4.7 million). This component included: 1. Technical assistance and training for government staff in epidemiology at the national and district levels, epidemiological surveillance software and training in its use, training on crisis preparedness and management, support for assessing, planning and training on information and telecom systems, reviews and updates of regulatory systems, and development of guidelines for the use of personal preparedness equipment . 2. Key testing equipment for the National Laboratory for Viral Respiratory Diseases . 3. Human health system response including equipment for intensive care units (ICUs), influenza kits, and antiviral drugs . Component 3. Public Information and Awareness . (Appraisal US$1.30 million; Actual US$1.1 million). This component included institutional capacity building and training, a three stage strategic communication plan that covered a pre-epidemic campaign, a campaign during the pandemic, and post -epidemic communication support to promote recovery. Component 4. Implementation Support and Monitoring and Evaluation . (Appraisal US$0.55 million; Actual US$0.40 million) This component included technical and financial support for project management, implementation, and monitoring and evaluation. It included capacity building for the implementing agencies including technical assistance, auditing services, and incremental operating costs . It also included training on monitoring and evaluation at all levels and development of an M&E action plan . It provided for baseline studies, ongoing participatory monitoring and evaluation, and a final project evaluation . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Costs. Project costs at appraisal were US$ 10.60 million and actual costs were US$10.75 million. The IDA credit contribution at appraisal was US$ 4.00 million and actual was US$4.45 million due to exchange rate changes . Similarly, the IDA Grant was US$4.00 million at appraisal and US$4.45 million actual. The United States Agency for International Development provided US$ 0.35 million, the European Commission US$1.00 million, and a Japan Policy and Human Resources Development Fund Grant US$ 0.50 million. Much of the training was covered by the Japanese grant. Borrower Contribution. At appraisal, the borrower was to provide $ 0.75m, but according to the ICR Annex 1, Table (b), the actual borrower contribution was zero . Borrower funds were intended to contribute to the compensation fund, but no compensation payment was made as no outbreaks occurred during the project's lifetime . Dates. The project was restructured three times . The closing date was 16.5 months later than planned following extensions on April 20, 2010, February 8, 2011, and March 30, 2011. The first was due to reallocation of funds originally intended for the compensation fund towards other project investments, as the perceived risk of avian influenza outbreaks had declined. The latter extensions were to accommodate delivery from and payment to a supplier who had suffered external delays . 3. Relevance of Objectives & Design: a. Relevance of Objectives: Outbreaks of avian influenza posed a real and serious threat to Moldova . Though no outbreaks had occurred in Moldova, there had been outbreaks in neighboring Romania and Ukraine, and there was an infection risk from migrating birds. Avian influenza could potentially spread rapidly and could potentially infect humans, as 14 million out of 18 million birds were in backyard farms or small commercial farms with poor biosecurity standards . The economic consequences of uncontrolled avian influenza outbreaks would be significant : the poultry sector constituted 80% of livestock in Moldova, and outbreaks would impose significant costs on this sector . The probability of a global influenza pandemic among humans was low but the impact could be severe, so the overall risk was significant. Moldova had already organized emergency teams at local and national levels who could respond to an outbreak of avian influenza among birds. The government had designed and enacted a national contingency plan for avian influenza outbreaks and had preparedness and action plans in the case of an influenza pandemic . These were reviewed by the Bank and found to be adequate and in line with international agency recommendations, but the Bank recommended further strengthening of the plans . There was a broad need for upgrading laboratory capacity and medical equipment/facilities in both the animal and human health sectors . Diagnostic and other capacity was negligible. Understandably, there was no specific relevance of avian influenza to Moldova's FY 05-8 Country Assistance Strategy (CAS) as this had not yet emerged as an international crisis at the time this CAS was prepared . The project would partially support pillars of human resource /human capital development through improving the quality of healthcare, which appeared in both the FY 05-08 CAS and the FY09-12 Country Partnership Strategy that was current at project closure. The relevance of objectives is rated Substantial . b. Relevance of Design: The project followed the broad design of the Global Program on Avian Influenza, combining animal health, human health and awareness raising components . The design addressed several important aspects of reducing the risk of avian influenza, including surveillance, diagnosis, outbreak control, and patient isolation . Many of the investments covered by the project would have broad benefits beyond their relevance for avian influenza . A gap in the project design was the lack of measures aimed at reducing the spread of disease among animals, which is a crucial part of an effective strategy for risk reduction . The project components were largely aimed at improving the capacity to respond to outbreaks, rather than at preventing outbreaks from occurring by improving biosecurity . This would have been an important factor in achieving the objective to "minimize the threat posed to humans " by avian influenza. The ICR noted that the project did not include activities related to backyard poultry improvements "due to a lack of identified funding " (page 7). The project did include some efforts to reduce risks on farms by supporting dissemination of information about good practices . The emergency loan instrument may not have been optimal, as the complex works and capacity -building activities would be difficult to complete within the limited time frame provided, and as adequate emergency response teams and plans already existed . However, while many project activities were implemented slowly, the emergency project did provide rapid supply of key consumables and equipment, along with setting up a compensation mechanism and beginning efforts toward public awareness . The relevance of design is rated Modest. 4. Achievement of Objectives (Efficacy): Objective 1: Minimize the threat posed to humans by highly pathogenic avian influenza infection in domestic poultry The project aimed to reduce the risk of avian influenza by improving surveillance, containment, and behavior . The project successfully implemented several activities that would assist in improving the ability to detect, diagnose and contain an outbreak of avian influenza . Equipment purchases and training for two regional laboratories and the RCVD laboratory were completed successfully . These investments have significantly improved the capacity to make viral, bacterial and micro-biological diagnoses, including for avian influenza and other zoonoses . Simulation exercises demonstrated that HPAI outbreaks could be identified with 95% accuracy within 48 hours of sample collection, whereas previously no diagnostic capacity existed . Purchases of vehicles for rapid response teams have improved the ability to transport samples from rural areas, which will assist in rapid diagnosis . Incinerators were purchased and vehicles and training were provided to rapid response teams, which would assist in containment and eradication . Simulation exercises/drills demonstrated the ability of response teams to conduct containment and eradication exercises . Three full simulation exercises on animal health response were undertaken, and two exercises on human health response . Improvements in tracking wild fowl and poultry will improve the ability to track the spread of disease should an outbreak occur . A compensation mechanism for farmers who lost animals to culling or other containment actions was established, in order to encourage farmers who had sick animals to report these, and so to allow for rapid response and containment. The functionality of the compensation mechanism was demonstrated during simulation exercises, and farmers could be compensated within 10 days. However, as no outbreaks had occurred by 2009, funds designated for compensation were reallocated to investments in laboratory capacity and vehicles, and so no dedicated compensation fund exists as of project closure . The Ministry of Finance has committed to provide compensation as needed should outbreaks occur in the future, but ad hoc compensation might be less effective than a dedicated fund in encouraging farmers to report losses . Public awareness campaigns had some impact on improving animal handling practices on backyard farms . Surveys at the end of the project reported that adoption of these practices was at 30-60%, down from earlier surveys conducted when awareness of avian influenza was higher but apparently higher than baseline levels . The ICR did not include baseline levels or detailed survey results . Public dissemination of model backyard poultry operations may have had some impact on animal handling practices, but no quantitative evidence is available . The quantitative effect of these outputs on the threat posed by avian influenza outbreaks in domestic poultry cannot be observed. It is likely that these investments reduced the risk to people and birds by improving the ability to respond to outbreaks. However, there is little evidence that the probability of outbreaks occurring has been reduced by the project, so there still remains a threat from avian influenza outbreaks in domestic poultry . Efficacy for objective 1 is rated modest. Objective 2: Prepare for, control, and respond to an influenza pandemic and other infectious disease emergencies in humans Training, equipment and reagent purchases for hospitals and human health laboratories were carried out largely as planned, though hospital upgrades proved to be more onerous than expected . These facilities will assist in improving the ability to cope with influenza or other disease outbreaks by improving the ability to treat and isolate infected patients. The National Viral Laboratory was upgraded, and samples from the 2009 influenza pandemic sent to the World Health Organization in the United Kingdom for re -testing showed a 98% accuracy rate. Rapid and accurate diagnoses would assist in identifying and responding to infectious disease outbreaks among humans . A computerized medical information system was successfully implemented in 40 locations, which will increase the ability to detect and monitor infectious disease outbreaks in real time . The system proved helpful during the 2009 influenza pandemic, allowing the central authorities to respond rapidly to emerging disease clusters . Institutional linkages and cooperation between animal health and human health sectors and subsectors may improve management of chronic diseases . Public awareness campaigns led to improvements in public awareness about risks associated with avian influenza and other infectious diseases, particularly during the 2009 influenza pandemic. Survey data indicated a high level of awareness of influenza symptoms in both the general public and health practitioners . At closing 90% of respondents reported awareness of AI, H1N1 and seasonal influenza, and could identify main symptoms . 70% of those reached by project-supported media have reported behavior and attitude changes for some key prevention aspects . For avian influenza, more than 50% of respondents correctly identified main symptoms and prevention measures . Baseline values were not reported in the ICR . A coherent and coordinated government response to the 2009 influenza pandemic included an effective public information campaign that quelled rumors about vaccination safety . 620,000 people were vaccinated against the 2009 pandemic influenza, with vaccine doses donated by Romania or obtained with WHO support . The project contributed to this response through training events on effective public and media communication for high level government officials, training 500 doctors to support the vaccination program, and delivery of media messages and information materials on the benefits of vaccination and on how /where people could receive vaccinations . Efficacy for objective 2 is rated substantial . 5. Efficiency: The economic analysis used in the PAD (and reproduced in the ICR) calculated an economic rate of return based on assumptions about a change in the probability and severity of outbreaks among poultry and of changes in mortality and morbidity rates from a pandemic . Though the estimates of the costs of poultry losses and human health pandemic losses are reasonable, the assumptions about changes in probabilities are highly speculative . There is no clear link between the activities supported by the project and the magnitude of probability changes . This is understandable, because there is little solid evidence available to make these assumptions . This is a general challenge for pandemic and disease risk reduction projects, and is not specific to this project . Given that no outbreaks occurred during the project and given the difficulties of trying to estimate the project impact, an ex post rate of return calculation would not be meaningful . The quantitative economic analysis provided does not incorporate potential benefits that will accrue from the project investments beyond their impact on avian influenza . Improvements to ICUs, ambulances and laboratories will provide broad benefits to the health sector . The ICR notes that these investments are already credited for having saved lives of people suffering from various respiratory and infectious diseases (page 30). The health information system could lead to improved health sector management and more efficient deployment of resources . Improvements to veterinary facilities and laboratories will assist in achieving EU -compliant food safety measures, which could improve market access for livestock exports . There were some delays in project implementation leading to two extensions; some of these delays were caused by greater than anticipated difficulty in implementing specialized procurement and works activities, some were caused by the decision to reallocate funds intended for compensating farmers for outbreaks that did not occur, and others were beyond the control of the project . Economies of scale were achieved by downgrading district laboratories into sample collection facilities, and merging diagnostic functions into the two larger regional labs . Efficiency is rated substantial . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 23% 88% ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project was largely carried out according to design and has dramatically improved animal and human health sector capacity. However, the failure to address unsafe bio -management practices in the backyard and small operator farms that constitute most of the poultry sector means that the project contributed little progress towards reducing the risk of disease spread among birds . The Food and Agriculture Organization, World Health Organization and World Organization for Animal Health all call for containment at the source and reducing the circulation of avian influenza among poultry to be a critical part of the strategy to reduce the risk of human infections . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The ICR (p. 15) notes that while resources in the human health sector are tight, they are adequate to maintain and operate the equipment acquired under the project, as evidenced by 2011-12 budget allocations. The fact that new equipment and facilities will have broad benefits to the health sector outside of avian influenza will help to ensure continued funding. Little evidence is available on the sustainability of public information and awareness improvements . The ICR (p. 17) notes that prevention practices for handling poultry had declined from their peak value by the end of the project as the perceived risk of avian influenza declined, though they were still above baseline levels . The actual quantitative values were not provided. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: Project preparation and appraisal were carried out very rapidly in response to a perceived emergency situation, following the broad outline of the Global Program on Avian Influenza . However, the rapid preparation meant that some design elements (particularly decisions about which facilities would be selected for rehabilitation support) were not determined before appraisal, which led to a slow start for works contracts and the need for project extension. Coordination with international agencies (the WHO, FAO, UNICEF, USAID and the European Commission) was complicated and time consuming, but these agencies provided much -needed funding and technical capacity. The activities supported by the project were valuable and would all serve to reduce the threat posed by avian influenza, but the project largely neglected costly on -farm investments that could reduce the probability of disease outbreaks. The project appropriately included a number of anti -corruption measures to mitigate governance risks in the compensation fund. The results framework had several weaknesses, though some of these would be difficult to avoid given the preventive/risk reduction nature of the project . The project outcome indicators were framed as actions that would occur if an outbreak occurred among animals or humans; it would be difficult to observe these indicators in the event that outbreaks did not occur during the limited project duration . Baselines were not well identified for some capacity-oriented indicators. Because of the design gap noted above, the results framework focused on measures of capacity to respond to outbreaks or pandemics, but it did not include indicators such as animal handling or biosecurity practices that would affect the probability of an outbreak occurring . at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: The Bank's task team leader was based in Moldova, and the supervision team was able to provide regular and frequent visits. According to the ICR, mission reports highlighted implementation problems to the Government and implementing agency, which were then addressed . The Bank left the client in charge of overall selection of priorities within the broad project objectives . The Bank responded to a client request to reallocate money intended for a compensation fund towards laboratory and vehicle investments. The Bank team encouraged the use of simulation exercises as a means of training, of learning, and of verifying capacity improvements that would otherwise have been difficult to observe . Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The ICR notes that the project was implemented during turbulent political times, but that political support was sustained throughout the implementation period . The $0.75m borrower contribution to project financing was not provided; however, this contribution was intended for compensation to farmers whose birds were culled in order to contain outbreaks, and the funds were not called on because no outbreaks occurred . The government did earmark funds in annual budgets to provide for compensation if needed . Government Performance Rating Satisfactory b. Implementing Agency Performance: The ICR notes that the project management unit was well experienced and effective, and performed well in managing processes and contracts and in coordinating across sectors and ministries . Continuity in staffing improved project performance. While the project was slow to start as many decisions had not been made during the design phase, overall the agriculture and health ministries performed well and completed their activities . The ICR (p 19) reports that a major positive outcome from the project was the increased awareness of the two leading line agencies of the importance of links between animal and human health disciplines . Implementing Agency Performance Rating : Satisfactory Overall Borrower Performance Rating : Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The M&E framework was output driven, focusing largely on monitoring whether particular activities had been completed, such as the number of people trained or the delivery of a plan or equipment . The project outcome indicators were framed as contingencies; the indicators would only be observable if an outbreak of avian influenza occurred among poultry or humans . Despite the presence of "other infectious disease emergencies " in the Financing Agreement PDO, the results framework was largely limited to avian influenza . Some baselines were not clearly specified, though for many capacity indicators the apparent baseline was zero . The framework identified that the project management unit would be responsible for collecting data, and specified the source of data for the (contingent) outcome indicators. b. M&E Implementation: Indicator progress was recorded by the project management unit and was reported by implementation and supervision reports. Since no outbreak occurred, the project used simulations to assess preparedness and response capacity. Use of simulation exercises was an innovative and effective means of assessing whether capacity improvements had been made, and was a much more effective means of tracking actual progress than recording the delivery of equipment or the number of people attending training sessions . Knowledge and practice surveys were conducted to assess the effectiveness of the information campaign . The ICR reports that a fully functional and funded compensation fund had been established, but during the project the funds set aside for compensation were reallocated to other purposes . The results indicator was not updated to recognize that dedicated funds were no longer guaranteed . c. M&E Utilization: The ICR provides no evidence of utilization of M&E during the course of the project . Simulation exercises included an outside observer who provided feedback on performance weaknesses which assisted in improving response systems, but this was not part of the project's formal M&E system, and evidence on the results and utilization of these exercises was not available . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The project was Environmental Category B . The only safeguard triggered was Environmental Assessment (OP/BP 4.01). A waiver was given for a delay in implementation of the Environmental Assessment from the normal appraisal stage to a later date. Environmental impact assessments for the rehabilitation of the veterinary laboratories were undertaken. According to the ICR (p.14), visits by safeguard specialists did not bring to light any issues related to construction processes, the handling of construction or laboratory waste, or any issues related to land acquisition or resettlement. But the ICR did not clearly state that there was satisfactory compliance with safeguard procedures . b. Fiduciary Compliance: The ICR reports (p.14) that audits were on time and contained only some moderate qualifications related to internal controls. Reviews of procurement and financial management during supervision generally rated these satisfactory . c. Unintended Impacts (positive or negative): d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Satisfactory Moderately Very rapid project preparation led to Satisfactory some project design weaknesses at entry (see Section 8a). Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR reports the following lessons (page 22-23): Effective coordination of emergency interventions may have higher impact when integrated in overall sector reforms . The veterinary laboratories and health information system will provide wide benefits beyond the specific project objectives . Coordination between stakeholders is necessary for maximum impact, and this is time consuming . Such coordination cannot usually be achieved during the limited 3 year time period of an emergency operation . Similarly, complex and technical procurement and institutional strengthening cannot be carried out in the limited emergency time period. Rehabilitating existing buildings and infrastructure can be less efficient than constructing new facilities in the case of complex and technically challenging structures such as infectious disease laboratories and hospitals. Procurement of equipment for these facilities is highly technical, and specialized consultants should be on hand to assist in developing precise specifications . IEG finds that the project also demonstrates the effectiveness of using field simulation exercises as a means of assessing improvements in the capacity to deal with disease outbreaks, and of identifying weaknesses in response that can then be addressed . Capacity levels for emergency response are otherwise difficult to observe, because they are not utilized if no emergency occurs . Other projects aimed at building capacity to respond to disease outbreak emergencies would benefit from incorporating similar simulations . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: While the ICR correctly outlines the numerous successful achievements of the project, it contains few acknowledgements of project weaknesses . The ICR does not contain detailed evidence for several statements, though some of this evidence was obtained by IEG in follow -up discussions with the project team . It does not provide a breakdown of project expenditure by component, and provides no explanation for the lack of borrower contribution . Response information is provided for the final beneficiary survey, but not for the original survey used as a baseline, making it difficult to assess project impact . Little information is provided on implementation and utilization of M&E, or on the performance of government or implementing agencies . Little information is provided on safeguard compliance. a.Quality of ICR Rating : Satisfactory