Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AO-Municipal Health Serv Strength (FY10) (P111840) Report Number: ICRR0021678 1. Project Data Project ID Project Name P111840 AO-Municipal Health Serv Strength (FY10) Country Practice Area(Lead) Angola Health, Nutrition & Population L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-47490,TF-14221 31-Dec-2015 65,547,491.01 Bank Approval Date Closing Date (Actual) 08-Jun-2010 30-Sep-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 70,800,000.00 4,240,000.00 Revised Commitment 75,040,000.00 4,240,000.00 Actual 65,547,491.01 3,827,896.81 Prepared by Reviewed by ICR Review Coordinator Group Salim J. Habayeb Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The objectives of the project were “to improve the Recipient population’s access to and quality of maternal and infant health care services” (Financing Agreement, 9/8/11). The statements of objectives in the Financing Agreement, PAD, and ICR were similar, although they used the words 'infant' and 'child' interchangeably. A project restructuring on 11/16/15 revised the objectives “to improve the population’s utilization of maternal and child healthcare services," thus dropping the quality objective. The stated reasons for such dropping were difficulties in measuring the quality of services, as the project had not established a specific framework Page 1 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AO-Municipal Health Serv Strength (FY10) (P111840) for assessing quality at design, and inability to attribute quality improvements to the project (ICR, p. 16, and Restructuring Paper, 11/2/15. p. 5). A level 2 restructuring on 7/31/17 revised baselines and end-targets upwards. The scope of the project was 18 municipalities in 5 provinces, covering a population of 1.5 million people (out of 163 municipalities and 18 provinces in the country, with a total population of 23.4 million in the country at the time of appraisal in 2010). Municipalities were selected based on health status, level of accessibility, availability of infrastructure, staff, and supplies, and inclusion in the decentralization program of the Ministry of Territorial Administration. The geographical focus of the project was in line with Angola’s poverty-eradication strategy as specified in the Government Program 2009-2012 and the Bank's Interim Strategy Note, in an effort to strengthen decentralization of social services and capacity of municipalities (ICR, p. 10). b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 16-Nov-2015 c. Will a split evaluation be undertaken? Yes d. Components 1. Improvement of Health Service Delivery (Appraisal US$56.3 million; Actual US$42.6 million). 1.1 (a) Strengthening primary health care service at the municipal level through provision of training on emergency obstetric and neonatal care and integrated management of childhood illnesses to trainers and general nurses as well as pre-service training of nurse midwives; and (b) strengthening of primary health care service at the municipal level, including enhancement of professional knowledge and skills of health care personnel and procurement of essential medical drugs and supplies in Participating Provinces, through provision of goods, services, and training. 1.2 Scaling up of healthcare outreach services in Participating Provinces through provision of goods and training. 1.3 Improving healthcare interventions at the community level in Participating Provinces through provision of goods and training. 1.4 (a) Rehabilitation and construction of 32 delivery rooms at healthcare centers and healthcare posts; construction of 24 houses for health professionals at provincial and municipal levels; provision of goods and equipment for family planning and pre- and post-natal care in Participating Provinces through the provision of goods, services, and civil works; (b) rehabilitation and construction of four delivery rooms at healthcare centers and healthcare posts in Malange; and (c) provision of solar and camping kits to outreach teams. Page 2 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AO-Municipal Health Serv Strength (FY10) (P111840) 1.5 Implementing the Medical Waste Management Plan. 2. Voucher Scheme Pilot (Appraisal US$0.8 million; Actual US$0). This component was cancelled at project restructuring in 2015. 2.1 Provision of cash transfers to beneficiaries residing in the municipalities of Negage and Caculama to facilitate access to child delivery services and pre-natal care, in accordance with provisions of the Voucher Scheme Manual. 2.2 Provision of technical assistance. 3. Project Management and Monitoring and Evaluation (Appraisal US$18.2 million; Actual US$21.2 million). Strengthening the capacity of the Ministry of Health (MOH) at central, provincial, and municipal levels to implement, supervise, monitor, and evaluate the project, including carrying out audits through provision of goods, services, training, and operating costs. Revised Components 1. Improvement of Health Service Delivery (Revised US$50.2 million; Actual US$42.6 million). The component was revised to include only three subcomponents: Subcomponent 1 was revised to scale up outreach and community-based health interventions at the community level through provision of training of trainers, equipment for outreach teams, including solar and camping kits, and technical assistance. Subcomponent 2 included the strengthening of maternal and child health services and health care waste management through provision of goods, technical assistance and works for: (i) rehabilitation and construction of approximately 32 delivery rooms at healthcare centers and healthcare posts, and construction of houses for health professionals at provincial and municipal levels where deemed necessary; and (ii) rehabilitation and construction of four delivery rooms at healthcare centers and healthcare posts in Malange, one of the Participating Provinces. Subcomponent 3 was changed to strengthen municipal health services at the primary level through provision of training, technical assistance, including provision of software and technical support for exchange of telemedicine communications, goods, including medicines and medical supplies, and carrying out of information campaigns on health promotion. Also, as Angola faced a yellow fever outbreak in December 2015, the project supported the financing of yellow fever vaccines for an amount of US$5 million. 2. Training of Health Professionals and Strengthening Capacity of Health Training Institutions (Revised US$3.4 million; Actual US$1.7 million). Subcomponent 1: strengthening of primary health care services at the municipal level through the provision of training on emergency obstetric and neonatal care and integrated management of childhood illnesses to trainers and general nurses, as well as pre-service training of nurse midwives. Page 3 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AO-Municipal Health Serv Strength (FY10) (P111840) Subcomponent 2: strengthening the training capacity of MOH, including through carrying out of postgraduate training and refresher training courses for trainers, provision of technical assistance and goods, and carrying out of small civil works for training facilities. 3. Project Management, Monitoring and Evaluation (Revised US$21.7 million; Actual US$21.2 million). A new activity was added to strengthen the capacity of MOH at the central, provincial, and municipal levels to manage, implement, monitor, and evaluate the project through the provision of works, goods, trainings, operating costs, and technical assistance, including carrying out audits and a national health survey. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Cost. The original cost of the project was estimated at US$75 million, consisting of an IDA Credit of US$70.80 million equivalent and a grant of US$4.24 million from the oil company Total Exploration and Petroleum Angola (TEPA). In addition, a Borrower contribution of US$16.5 million was initially planned, but due to the economic crisis facing the country, the contribution was limited to US$0.54 million (ICR, p. 58), largely for operational costs. The actual project cost (excluding government contribution) was US$65.55 million (IDA: US$61.72 million; and Grant: US$3.83 million) as shown in the Operations Portal, indicating a disbursement ratio of 92%. Dates. The project was approved on 6/8/10 and became effective on 3/2/12, or 21 months after Board approval. A Level 2 restructuring on 8/28/12 introduced a new expenditure category on “Works," which had been mistakenly omitted in the Financing Agreement (ICR, p. 15). A Mid-Term Review was undertaken on 9/15/14. A Level 1 restructuring on 11/16/15 revised the objectives and results framework, and extended the closing date by two years from 12/31/15 to 12/31/17. A third Level 2 restructuring on 7/13/17 extended the closing date by an additional six months from 12/31/17 to 6/30/18, with funding reallocations and revisions in the results framework. The closing date was further extended by three months through a notice of extension until 9/30/18 to allow delivery of cold chain equipment. The project closed on 9/30/18, about three years beyond the original closing date of 12/31/15. 3. Relevance of Objectives Rationale At appraisal, child and maternal mortality rates were among the worst in the world. In 2006, maternal mortality was estimated at 1,700 per 100,000 live births, under-five child mortality at 260 deaths per 1,000 live births, and infant mortality at 154 deaths per 1,000 live births (PAD, p. 27, based on World Development Indicators 2006). Acute diarrheal diseases, acute respiratory infections, malaria, measles, and neonatal tetanus accounted for 60 percent of child deaths. Child malnutrition, a main contributing factor associated with child mortality, was reflected by a stunting rate of 38% in children under-five. The main causes of death in pregnant women were hemorrhage, eclampsia, prolonged labor, abortion complications, and malaria. Hence, the focus of the project on maternal and child health was relevant. According to the ICR (p. 11), the sizable infusion of global resources towards HIV/AIDS had left a financing gap in the areas of maternal and child health services, which the project intended to address. Page 4 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AO-Municipal Health Serv Strength (FY10) (P111840) Around appraisal, the objectives were in line with the Bank‘s Interim Strategy Note for 2007-2009, under its second pillar for rebuilding critical infrastructure and the improvement of service delivery for poverty reduction, and with national priorities as set out in the Plan for Revival of Municipal Health Services, which stipulated increased access to, and utilization of, maternal and child health services as a key objective, recognizing the importance of addressing under-served areas through outreach and community programs. Angola was undergoing a process of municipalization/decentralization under which the municipalities would become responsible for planning and managing their health and social services. MOH was considered among the leading ministries in the Government’s overall municipalization strategy, and the health sector was looked at for lessons learned (ICR, p. 10). At project closing, the objectives were consistent with one Country Partnership Strategy (CPS) outcome “expand health service delivery” (CPS FY14-FY16, extended through 2019), under CPS Pillar 2 for enhancing the quality of service delivery. It appears that the CPS did not differentiate between service quality enhancement and service expansion. Since the project’s development objective to improve the quality of health services was dropped, the objectives were not fully aligned with the quality enhancement thrust of CPS Pillar 2. The Borrower’s comments (ICR, p. 60) stated that the main objectives of the project should have been maintained and that adjustments should have been limited to technical aspects without affecting the content of the main objectives. The ICR (p. 21) also noted that the objectives were aligned with the current Country Partnership Framework's (FY20-25) health objective for “improving quality of services and utilization of local facilities” and that improvements in M&E would permit the assessment of the quality of care dimension that was “impossible” to undertake during project implementation. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Improve access and utilization of maternal and child health care services. Rationale Pre-service and in-service training for health personnel, including general nurses, midwives, and community health workers; provision of primary health care and community health services and support to mobile teams; emergency obstetric and neonatal care; integrated management of childhood illnesses; provision of drugs, medical supplies, and delivery kits; rehabilitation and construction of delivery rooms; provision of information and health education materials; a voucher program for demand generation; technical assistance for human resources development and program management; introduction of telemedicine in five provincial hospitals; and training in M&E and technical assistance for conducting access and quality surveys would be reasonably expected to result in improved skills and enhanced services, outreach and performance, and demand Page 5 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AO-Municipal Health Serv Strength (FY10) (P111840) generation, which in turn would plausibly contribute to improved access to maternal, infant, and child health care services. Outputs The number of mobile team visits reached 4,215 in 2018, exceeding both the original target of 949 and the revised target of 4,000. The number of deliveries attended by skilled health personnel reached 240,480, short of the target of 268,000. The project provided 54 ambulances and 4x4 station wagons, and 184 motorcycles, for outreach services. 188 nurses were trained in emergency obstetric and neonatal care. 410 nurses were trained in the integrated management of childhood illnesses. 592 community health workers and 84 nurse-midwives received pre-service training. 745 health staff were trained to use the management information system. The number of health facilities receiving at least two supervision visits per year reached 407, exceeding the original target of 200 and the revised target of 230. The number of pregnant women receiving antenatal care during a visit to a health provider increased from a baseline of 81,000 in 2012 to 213,771 in 2018, exceeding both the original target of 109,000 and the revised target of 165,000. The voucher scheme that aimed at stimulating demand and encouraging pregnant women to have institutional deliveries was dropped by the government (ICR, p. 23). Outcomes Direct project beneficiaries increased from a zero baseline in 2009 to 329,689 in 2018, exceeding the target of 240,000. The percentage of institutional deliveries increased from a baseline of 41% in 2009 to 53% in 2018, short of the target of 60%. The number of consultations/outpatient visits increased from a zero baseline to 1,914,424 in 2018, exceeding both the original target of 1,461,354 and the revised target of 1,087,500. The percentage of children 0-1 year immunized with pentavalent vaccine 3rd dose increased from a baseline of 46% in 2009 to reach a universal level, exceeding the target of 75%. Page 6 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AO-Municipal Health Serv Strength (FY10) (P111840) Rating Substantial OBJECTIVE 2 Objective Improve the quality of maternal and infant health care services. (Dropped on 11/16/15) Rationale Under the same rationale as that for Objective 1 above, it would be reasonable to assume that project activities would contribute to improving the quality of related services. Outputs The same as under Objective 1, above. Outcomes No direct measures of quality were undertaken by the project, although it was plausible that training and improvements in maternal and obstetric care contributed to service quality. The ICR (p. 17) stated that the assessment of the quality of care was never contemplated by the original project activities (ICR, 17), and that such an assessment would have required an analysis of inputs and organization of services, processes (such as how services were produced, treatment protocols, interaction between providers and users), and outcomes, including patient satisfaction. The 2015 project restructuring dropped the quality objective for the following reasons: complexity of its measurement and difficulty in identifying and documenting accurate measures of quality; lack of a specific framework for assessing quality at the design stage (ICR, p. 16); and inability to attribute quality improvements to the project (ICR, p. 17). Rating Modest OVERALL EFF TBL OLD Rationale Under the original objectives, the aggregation of an almost fully achieved objective with a partly achieved objective indicates a Modest efficacy rating under the original objectives. Under the revised objectives, the objective on access to maternal and child health care services was almost fully achieved, indicating a Substantial efficacy rating under the revised objectives. Page 7 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AO-Municipal Health Serv Strength (FY10) (P111840) Overall Efficacy Rating Substantial 5. Efficiency The PAD’s economic analysis illustrated modeling work, conducted by a 2007 Public Expenditure Review, that identified and costed out packages of interventions that would reduce the burden of disease in the country. The model estimated the impact of five health service delivery steps (or packages) in terms of reduced infant mortality, under-5 child mortality, maternal mortality, and the additional cost per capita per year for each. The five packages included the following: (i) community-based social mobilization and behavioral interventions (at US$2.51); (ii) scaling up outreach services (at US$1.05); (iii) expanding primary health care (at US$3.05); (iv) strengthening the first referral level (at US$0.97); and (v) improving the second level referral (at US$0.89). The ICR’s economic analysis was insufficient because it compared only the revised allocation for Component 1 (Improvement of Health Service Delivery), equivalent to a per beneficiary investment of US$29.9, with the original component allocation, equivalent to a per beneficiary investment of US$37.5. It concluded that this represented savings, and that, in consideration of such savings and achievement of PDO indicators under the revised objectives, economic efficiency was considered to be substantial. There were significant shortcomings in the efficiency of implementation. Effectiveness and project launch were delayed by about two years after Board approval (ICR, p. 15) due to delays in Cabinet approval and government signing of the Financing Agreement (ICR, p. 18). The project was extended twice to complete activities and closed nearly three years beyond the original date. The planned construction and/or rehabilitation of health centers, staff houses and delivery rooms was cancelled, as there were delays in land acquisition, complex procurement processes, exchange rate fluctuations, and major cost increases (ICR, pp. 20, 23, and 34). Also, selected public health and training specialists were unable to obtain a working permit for more than six months (ICR, p. 32), forcing delays as the project was forced to recruit local consultants to carry out related tasks. Budgetary constraints affected all levels of health care services, especially at the provincial and municipal levels, and caused significant limitations in service delivery, especially outreach activities. The resulting reduction in outreach activities had a negative impact on the coverage of key public health activities (ICR, p. 32). Barely any training contemplated for obstetric and neonatal care and midwifery was carried out before the Mid- Term Review of 9/15/14 (ICR, p. 23). Also, the lack of responsiveness of TEPA, which co-financed the grant of US$4.2 million, caused extended delays affecting implementation. There was a 14-month delay in signing the Amended Administrative Agreement with TEPA, and the delay prevented the project from accessing the grant to conduct training on emergency obstetric and neonatal care, integrated management of childhood illnesses, and pre-service training for nurses. The voucher scheme aiming at demand generation for institutional delivery services was dropped due to legislative changes. According to the ICR (p. 16) implementation under the original objectives was negatively affected by the complexity of the results framework and its ineffectiveness in measuring the PDO, and this prevented meaningful work. Data for many indicators could not be collected (ICR, p. 17). Beyond the supervision visits to Page 8 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AO-Municipal Health Serv Strength (FY10) (P111840) health facilities, other indicators were dropped, making it difficult to track and evaluate the intended strengthening of management aspects and supervision (ICR, p. 24). The project also faced challenges resulting from mixed messages in several areas, with a notable example concerning institutional births, which were encouraged by MOH and by the project in pursuit of development objectives, as opposed to the encouragement of traditional births by the Ministry of Social Action, Family, and Women. Mixed messages in policy had a negative impact at the municipal level, where different authorities conveyed different policies or priorities (ICR, p. 38). Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Under the original objectives, relevance of objectives is rated Substantial in view of general alignment between project objectives and the Country Partnership Strategy at the time of project closing. Efficacy is rated Modest, as the project only partly achieved its objectives. Efficiency is rated Modest in view of insufficient economic analysis and significant shortcomings in the efficiency of implementation. The outcome is rated Moderately Unsatisfactory under the original objectives. Under the revised objectives, relevance of objectives is rated Substantial and efficiency is rated Modest, as explained above. Efficacy is rated Substantial as the project almost fully achieved its objective. The outcome is rated Moderately Satisfactory under the revised objectives. According to IEG/OPCS guidelines, when a project’s objectives are revised, the final outcome is determined by the weight of Bank disbursements under each set of objectives (52% under the original objectives, and 48% under the revised objectives): • Under the original objectives, the outcome is rated Moderately Unsatisfactory (3) with a weight value of 1.56 (3 x 52%). Page 9 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AO-Municipal Health Serv Strength (FY10) (P111840) • Under the revised objectives, the outcome is rated Moderately Satisfactory (4) with a weight value of 1.92 (4 x 48%). • These add up to a value of 3.48. This ICR Review opted to round the weight value upwards to the value of 4 for several reasons: (i) the reasonable strength of outcomes under the revised objectives, and the fact that the project was adequately progressing on its intended trajectory; (ii) the project's status and progress were informed by the Mid-Term Review of Sept 2014, and the project team pursued restructuring, but it took over a year to finalize the restructuring because of the learning curve faced by the Government in undertaking a Level 1 restructuring, not because of lack of action (ICR. p. 35). Two weeks before restructuring effectiveness, the Restructuring Paper dated 11/2/15 (Report No: RES18630) showed a disbursement rate of 48% under the original objectives. This would have resulted in a value of 3.52 under the split evaluation as follows: under the original objectives, a Moderately Unsatisfactory outcome (3) with a weight value of 1.44 (3 x 48%); and under the revised objectives, a Moderately Satisfactory outcome (4) with a weight value of 2.08 (4 x 52%), both aggregating at a value of 3.52; and (iii) the decimal difference with the midpoint of 3.50 is considered negligible in this specific context. In conclusion, the overall outcome is rated Moderately Satisfactory, consistent with essentially moderate shortcomings in the project’s preparation, implementation, and achievement. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The main risks that development outcomes may not be maintained are related to financial constraints, weak human resources capacity, notably at decentralized levels, and the degree of political support required to prioritize health. There are competing demands and other priorities that can divert funding from maternal and child services. A lack of financial stability and fiscal restrictions could force the government to postpone or reduce its allocations to health programs. Also, effective collaboration among relevant ministries remains a challenge. However, over the course of the project, many institutions at various levels benefitted from capacity building, notably the cadre of trained health staff at the municipal level. The follow-on operation, the Health System Performance Strengthening Project (P160948, 2018-2023, US$110 million), tapped into this trained cadre of municipal-level health staff (ICR, pp. 29-30). Also, institutional strengthening was reflected by the first-ever Domestic Household Survey (DHS) that was undertaken in 2015-26, and a follow-on DHS was planned. 8. Assessment of Bank Performance Page 10 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AO-Municipal Health Serv Strength (FY10) (P111840) a. Quality-at-Entry The strategic thrust of the project was appropriate, and implementation modalities and fiduciary aspects were adequately prepared, but the risks were not thoroughly recognized. According to the ICR (p. 35), the project design did not take into full consideration the challenges and fragility that Angola was facing upon emerging from years of civil war, and specifically, the institutional challenges that were faced by the health sector with the breakdown of civil administration. No arrangements were planned at design for tracking and assessing the project objective to improve the quality of health care services. Infrastructure and civil works were not adequately planned and necessitated further assessment of the planned sites for facilities, implementation processes, and identification of bottlenecks, and related investments were ultimately cancelled. The project did not adequately assess weaknesses in M&E capacity, including the health management information system at the central and provincial levels, as well as at the level of third-party organizations implementing outreach activities, resulting in M&E challenges throughout the duration of the project. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision Supervision missions were conducted regularly and produced 16 implementation status reports with adequate reporting. The missions were often joined by a team of fiduciary and environmental/social safeguards specialists. The Team was reportedly proactive in addressing arising challenges that may hinder the attainment of desired outcomes, as also reflected by three project restructurings. The Team demonstrated flexibility and facilitated the financing of yellow fever vaccines when the country faced a yellow fever outbreak in 2015. According to the ICR (p. 21), the Bank’s implementation support was responsive to the needs of the project, which remained vital to achieving country priorities in the health sector. The Team organized a thorough Mid-Term Review in 2014, the findings of which led to the revision of project objectives. The Team alleviated M&E challenges by simplifying the results framework, training, development of manuals, and regular follow-up meetings. The Team was able to fulfill its tasks while navigating through a difficult environment with significant financial and capacity constraints, along with a learning curve for counterparts and issues of timeliness at the level of the co-financing agency. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization Page 11 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AO-Municipal Health Serv Strength (FY10) (P111840) a. M&E Design The objectives were clear, but the results framework and indicators did not fully reflect the objectives, specifically for the quality of health services. The design included a large number of indicators that were not routinely measured by MOH, and, according to the ICR, M&E capacity was not adequately assessed (see Section 8a). Building upon the existing Health Management Information System, M&E data were to be collected through MOH, municipal, and provincial levels. Training and practical manuals were planned. b. M&E Implementation The results framework was revised during the 2015 restructuring, rectifying some shortcomings while dropping the objective to improve the quality of maternal and infant health care services. The revision produced a clearer link between objectives and outcomes, simplified the results framework, and retained indicators that were easily measured. Data were collected by MOH at central and peripheral levels. M&E was strengthened by training conducted by the project at municipal and provincial levels, and by the development of practical manuals for use by staff at the facility level. According to the ICR (p. 33), weekly follow-up meetings were carried out between the Bank team and project team with dedicated time to discuss M&E and progress in data collection and reporting. c. M&E Utilization In addition to the use of M&E data for project monitoring, M&E findings were used for the preparation of the follow-on Health System Performance Strengthening Project (P160948), which was appraised in February 2018. An indirect benefit was the initiation of the first DHS (2015-2016), which, according to the ICR (p. 34), was stimulated by the Bank’s continuous advocacy about the importance of evidence. At the time of the ICR, the government was collaborating with the Bank on both DHS 2019-2010 and on Service Delivery Indicators, under the overall leadership of the National Statistics Institute. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified under environmental category B and triggered Environmental Assessment (OP/BP 4.01), mainly in view of medical waste hazards. In terms of compliance, the ICR (p. 34) stated that, in general, implementation was satisfactory, and that agreed actions were implemented. A dedicated Safeguards Focal Point was assigned. The ICR stated that the project increased awareness about the importance of safeguards, notably with regard to hospital waste management. Staff were trained in biosafety and hospital waste management. However, some challenges were faced in the implementation and monitoring of the health care waste management plan, and one of the problems identified across missions was the turnover of health personnel. The hospital waste management plan was updated and used by the follow-on Health System Performance Strengthening Project (P160948). Page 12 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AO-Municipal Health Serv Strength (FY10) (P111840) b. Fiduciary Compliance Both financial management and procurement were conducted in accordance with Bank policies and guidelines. Financial management. MOH had overall responsibility for financial management, with day-to-day management by the Department of Public Health within which the Central Coordinating Unit was embedded, including personnel in charge of financial management and procurement. The ICR stated that financial management capacity was adequate, but there were process delays, including in advances to the provinces. The ICR did not offer information on audit timeliness and qualifications. However, the project team (6/2/19) reported that there were delays in audit submissions for 2015, 2016, 2017, as the submissions were not received within six months from the end of the fiscal year. In both 2016 and 2017, the auditor expressed qualified opinion due to a difference between recorded advances to the provinces and the balance in respective bank accounts. The project team reported that the differences are still in the process of justification to the project coordination unit. The audits for other years were unqualified. Procurement. Activities that were outlined in the procurement plan were implemented (ICR, p. 35). There were initial delays prior to filling the position of a procurement specialist through contracting with an international specialist, whose task was complemented by a procurement officer and an administrative assistant. c. Unintended impacts (Positive or Negative) None reported. d. Other -- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Modest Page 13 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AO-Municipal Health Serv Strength (FY10) (P111840) 12. Lessons The ICR (pp. 33-34) offered several lessons and recommendations, including the following lessons restated by IEG: Strengthening coordination and collaboration between the health sector and other relevant ministries facilitates the achievement of intended project objectives. Solving some of the project implementation challenges required the leverage of other ministries, such as the Ministry of Territorial Administration and State Reform for land acquisitions; the Ministry of Social Protection for outreach activities; the Ministry of Finance for local staff payments; and the Ministry of Social Action, Family, and Women to support institutional rather than traditional deliveries. Bank flexibility and a relationship of trust promote country-Bank partnership. The ICR noted the example of the Bank’s responsiveness, under the project, to government needs for yellow fever vaccines when the country faced a yellow fever outbreak in 2015, thus preventing delays in dealing with the outbreak. According to the ICR, trust and flexibility contributed to the enhancement of the country-Bank partnership that grew from one project to a larger program that included three projects addressing maternal and child health, health system strengthening, expansion of child health services, and regional disease surveillance, with three grants in support of front-line service delivery for deworming, integrated service delivery, and public financial management. The choice of co-financing partners should be weighed and assessed in terms of advantages vs. disadvantages. TEPA’s small grant contribution was appreciated, especially in view of significant financial constraints, but managing the grant was challenging because of delayed responsiveness, difficult coordination, high burden of administrative processing, and delays in accessing funds that negatively impacted related aspects in project implementation. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was clear overall and provided a comprehensive overview of the project experience. The theory of change was generally aligned to development objectives. The evidence and analysis were aligned to the messages outlined in the ICR, but, as recognized by the ICR, there was no evidence to assess the objective to improve the quality of maternal and infant health care services, where the gaps were related to shortcomings in the project design rather than the ICR. The approach used for explaining ratings of efficacy--counting the number of indicators for which targets were achieved, and dividing achievement into percentage bands-- could introduce bias because of the underlying assumption that indicators are equally important or valid. The ICR’s economic analysis was inadequate, as it simply compared the original and revised allocations for one project component in terms of per beneficiary investment. The narrative and evidence supported the ICR's Page 14 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AO-Municipal Health Serv Strength (FY10) (P111840) main conclusions, except for efficiency. Lessons outlined in the ICR derived from project experience. The ICR was internally consistent, except for inconsistent financing tables. The ICR followed most guidelines, but it did not offer information about audit timeliness and qualifications, where information was subsequently provided by the project team. The main ICR text was unnecessarily lengthy and repetitive. a. Quality of ICR Rating Modest Page 15 of 15