Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DRC-Street Children Project (FY10) (P115318) Report Number : ICRR0020197 1. Project Data Project ID Project Name P115318 DRC-Street Children Project (FY10) Country Practice Area(Lead) Congo, Democratic Republic of Social Protection & Labor L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-H5780 31-Aug-2015 10,000,000.00 Bank Approval Date Closing Date (Actual) 01-Jun-2010 31-Aug-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 10,000,000.00 0.00 Revised Commitment 9,656,740.60 0.00 Actual 9,602,694.37 0.00 Sector(s) Other social services(71%):Public administration- Other social services(13%):Primary education(8%):General information and communications sector(8%) Theme(s) Social Safety Nets/Social Assistance & Social Care Services(35%):Participation and civic engagement(23%):Gender(21%):Social Protection and Labor Policy & Systems(13%):Education for all(8%) Prepared by Reviewed by ICR Review Coordinator Group Hjalte S. A. Sederlof Soniya Carvalho Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives The Project Development Objective (PDO) set out in the Financing Agreement (p.5) and in the Project Appraisal Report , PAD (p 14) was to improve the delivery mechanisms of prevention and support services for street children, primarily in Kinshasa. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DRC-Street Children Project (FY10) (P115318) b. Were the project objectives/key associated outcome targets revised during implementation? No c. Components Component 1: Prevention (Estimated cost at appraisal US$ 3.1 million; actual cost US$ 2.51 million). The component was to address the most common proximate causes of children ending up in the street: (i) family break-up; (ii) accusations of witchcraft; (iii) parental ignorance of children’s needs; and (iv) keeping children in school. Interventions would primarily rely on behavior change communications (BCC), and include: • Community mobilization through informal community-based networks of volunteers, organized and supported by community organizers and social workers contracted by non-governmental and faith-based organizations (NGOs/FBOs), were to reach out to families at risk; • Community awareness campaigns by NGOs/FBOs experienced in social marketing, delivering basic messages on relevant topics related to child welfare, including dissemination of Congo’s Child Protection Law, approved in 2009; • Media outreach by contracting NGOs specialized in the use of mass media to produce a variety of radio formats (sketches, mini- dramas, interactive call-in programs, and so forth) • Efforts to keep children in school by training teachers to identify children at risk and keeping them in school; and by setting up remedial education centers providing an accelerated primary cycle (three years instead of six). This has shown to produce better success rates than formal primary schools. On graduation, children can rejoin the education system and enroll at the secondary level. Component 2: Assistance (Estimated cost at appraisal US$ 4.3 million; actual cost US$ 4.2 million). The component was to finance activities aimed at improving coverage and quality of services for street children, with a particular focus on girls: they are at particular risk of sexual violence and pregnancy, with their children likely in turn to end up as street children. The component will support a wide range of services from immediate protection from imminent danger in transit centers that may provide shelter, medical care, food, trauma and psycho-social counseling, and remedial classes, through intermediate approaches such as using street educators and informal listening centers, to longer term living solutions, such as family reunification and group housing. Component 3: Capacity building, coordination, and project management (Estimated cost at appraisal US$ 2.6 million; actual cost US$ 2.9 million). The component was to provide technical assistance and training to the Ministry of Social Affairs, Humanitarian Action, and National Solidarity (MINAS) to strengthen its capacity to (i) develop standards for activities targeting street children; (ii) coordinate the interventions of different stakeholders (NGOs, FBOs, donors, Government institutions); and (iii) monitor and evaluate ongoing interventions. It would support capacity building among entities implementing Components 1 and 2, as well as provide relevant capacity building in other Ministries (Justice, Gender, Family and Children). In addition, an institutional assessment of MINAS was to be conducted, focusing on its internal processes. Activities under Components 1 and 2 were to be concentrated in four communes in the greater Kinshasa metropolitan area, which are major points of origin of street children. The total population of those communes was estimated at 2.1 million in 2004, and the number of street children at 14,000. While the project was designed primarily to benefit Kinshasa, some of its prevention and capacity building activities came to have national coverage through broadcasting and by training MINAS staff located outside Kinshasa. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost and financing. Total project costs, including contingencies (US$ 0.10 million), were estimated at US$ 10.10 million at appraisal; actual costs were US$ 9.7 million, 96 percent of the total. The difference between estimated and actual costs is mainly driven by lower than expected spending on schooling opportunities. Borrower contribution: There was no Borrower contribution. Dates. The project was approved on June 1, 2010, with an original Closing Date of August 31, 2015. The project closed on that date. 3. Relevance of Objectives & Design a. Relevance of Objectives The PDO is relevant as it addresses a major social problem among the population in the Democratic Republic of Congo (DRC), which was relevant at appraisal and continues to be relevant today. The PDO was is in line with the Bank’s Country Assistance Strategy (CAS) – it was Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DRC-Street Children Project (FY10) (P115318) so explicitly at the time of appraisal; it is still recognized as a problem, but subsumed under assistance to build up the safety net. Coming out of a decade of conflict in 2007, a large majority of the population was impoverished, and violence, human rights abuses and large- scale displacement had caused a weakening of the social fabric, strained family relations and worn down traditional support systems. A manifestation of the social disruption that had taken place was the growing number of street children: an estimated 40,000 in the country as a whole, including about 14,000 in Kinshasa (UNICEF 2006); of which some 26 percent were girls. The newly established Government had the political will to address the problem, even without Bank funding, and it already had taken a number of measures prior to the Bank project: MINAS had produced a national social protection strategy for vulnerable groups, a national action plan for orphans and vulnerable children had been developed; and a child protection law had been put in place with support from UNICEF. A number of other ministries had also taken on significant responsibilities in addressing different aspects that were putting children at risk. The project responded to one of the three pillars in the Bank’s 2008-2011 CAS: the one focusing assistance on improved social services, where this project was one of three investments addressing the needs of specific vulnerable groups, the other two dealing with the demobilization of ex-combatants, and the reinsertion of ex-miners into productive activity. The new CAS for 2013-2016 no longer singles out children as specific targets for intervention by the Bank – the basis for effective action on children was to be laid by this project; now the strategy focuses more broadly on continued support for the social protection system. Rating Substantial b. Relevance of Design Project design was consistent with the PDO. Prevention would be strengthened by developing proven behavior change strategies that targeted at-risk children and families. Support to street children would be given a solid foundation by developing services that addressed both day-to- day needs and longer-term means to reinsertion into mainstream society. Implementation was based on contracting private agencies (NGOs/FBOs) with international and on-the-ground CDR experience in prevention and support activities for at-risk and street children. A key element of design was capacity building that recognized the limited capacity and modest experience of public agencies in CDR to deal with these matters. Capacity building included: (i) the transmission of knowledge on prevention and support for street children to MINAS and local NGOs/FBOs so that they could continue to carry on the work once the project was over; (ii) the development of service standards and performance criteria to enable MINAS to monitor the effectiveness of programs, whether they were implemented by the public sector or by NGOs; (iii) establishing a government coordinating committee to address the multi-sector nature of the street children challenge; and (iv) developing an accompanying data base. The focus on the greater Kinshasa metropolitan area rather than a wider geographic area was determined by the labor-intensive nature of preventive interventions, which require face-to-face interaction with community leaders and at-risk households in order to be successful; as well as the limited IDA envelope for the project. Concern over capacity limitations in CDR during the design stage was further reflected in the structure of the Project Implementation Unit (PIU): while the operation was to be overseen by the PIU in MINAS, fiduciary responsibilities, including financial management and procurement were given to a Social Fund that had been operating successfully with IDA support for the past ten years and was familiar with Bank procedures. The results framework had a clear statement of objectives, and objectives were logically linked to intermediate and final outcomes. Targets were defined, and where practical, baselines and milestones were set. Data collection instruments and responsibilities for collection were specified. Rating Substantial 4. Achievement of Objectives (Efficacy) PHREVISEDTBL Objective 1 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DRC-Street Children Project (FY10) (P115318) Objective Objective: to improve the delivery mechanisms of prevention and support services for street children, primarily in Kinshasa. The objective consisted of two sub-objectives: putting in place improved delivery mechanisms for (i) preventive services for at-risk children; and (ii) support services for street children. Efficacy will be assessed on the basis of the achievements of these two sub-objectives, separately. Sub-objective 1: delivery mechanisms for prevention for at-risk children have been improved, primarily in Kinshasa Rationale This sub-objective was to be achieved by increasing the availability of and access to information relating to the child protection law and children’s rights, which in time was expected to result in behavior change. Outputs • 596 radio broadcasts per year on child protection were undertaken, roughly equivalent to the target of 600 per year, starting from a baseline of 96 broadcasts per year. In addition, 586 TV and radio transmissions of 8 documentaries and 12 spots on child protection developed by an international NGO specialized in audio-visual communications, were undertaken over the project period. • 488 interactive group discussions per year were undertaken on children’s rights and parental responsibilities; compared to a target of 550, and a baseline of 50. • 75 neighborhood child protection committees were established, covering all neighborhoods in the project’s target area. • 488 community events per year were held on children’s rights and parental responsibility, compared to a target of 550 and a baseline of 50. Outcomes The ICR does not provide evidence on the achievement of this objective beyond: • All neighborhood child protection committees were active at project closing. • 87 percent of targeted households recalled key messages about children’s rights from the campaigns financed under the project, compared to a target of 70 percent. Rating Modest PHREVISEDTBL Objective 2 Objective Sub-objective 2: delivery mechanisms for support services for at-risk children have been improved, primarily in Kinshasa Rationale The objective was to be achieved through improved delivery of and access to support services, including education, health, shelter, psycho-social support, vocational training, and family reunification Outputs • 15,586 street children (41 percent girls) received one or more services from the Transit Center, exceeding the target of 8,000 • 1,660 street children (57 percent girls) benefited from skills training, as compared to a target of some 6,700. (This number is lower than expected, because the majority of street children in the shelters are under 15, which is the minimum age to participate in vocational training) • 6,294 street children (54 percent girls) benefited from formal or non-formal education, compared to a target of some 6,800 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DRC-Street Children Project (FY10) (P115318) Outcomes • 10,057 street children were offered access to comprehensive support services, compared to a target of 5,500. • 5,335 street children (48 percent girls) had restored relationships with their families, as compared to a target of 800; over 4000 (29 percent girls) returned to live with their family (restored relationship is measured by the number of children making or receiving at least two visits to/from their family per year. Altogether, an estimated 15,600 street children received some form of support under the project, that is, practically all of the street children in the catchment area. Rating Substantial 5. Efficiency Neither the PAD nor the ICR attempt either an analysis of economic rates of return or cost-effectiveness. There were several administrative/implementing agency inefficiencies. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The rating for relevance of objectives and design relevance is substantial. Efficacy of one objective is rated modest and the other substantial. With effciency rated modest, Outcome is rated moderately satisfactory. a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DRC-Street Children Project (FY10) (P115318) The overall risk to the development outcome – maintaining the improved delivery mechanisms – is primarily provided by the country context, and it is significant. The CDR is (still) a politically volatile, post-conflict country with limited resources that in a budget crunch are unlikely to be allocated to MINAS to pursue its street children program. The withdrawal of the Bank from further direct support to the street children program in favor of broader safety net interventions would seem to confirm that view, albeit continued Bank support to poor families indirectly may (or may not) benefit at-risk children. Under such circumstances, there is also a risk that skills that were transferred through the project to better operate prevention and support programs may not be maintained. a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance a. Quality-at-Entry The project drew on lessons from international experience, and the experience of UN organizations and international NGOs active in CDR and in Kinshasa; and it carried out two studies on street children in Kinshasa in preparation for the project. A TTL with extensive experience in child protection was assigned to prepare the project and oversee its implementation. The focus on Kinshasa recognized the challenges in addressing street children issues, and the limited budget resources at the disposal of the project. The design recognized institutional constraints to implementation: while the project implementation unit (PIU) was charged with implementation, its fiduciary tasks were given to an IDA-supported Social Fund. This would ultimately not work out well (see Section on Fiduciary Compliance). A six-month project preparation period may have been too short. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Intensive supervision, totaling 18 missions over the five-year life of the project, reflected the team’s appreciation of the limited implementation capacity on the ground and the need for support, notably a weak PIU and fiduciary issues. The PIU was belatedly strengthened by recruiting a country-based consultant with experience in child protection to support it; and financial management issues were resolved at mid-term review, when responsibility was shifted (from the Social Fund) to a by now more experienced PIU, appropriately strengthened with an internal auditor and a procurement specialist. There were some fiduciary issues (see section 11). Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance The Government was represented by a Steering Committee that united all the concerned government agencies, and exercised strategic oversight, approving annual plans and budgets. It did so according to plan, thus fulfilling expectations. Direct responsibility for project oversight lay with MINAS. The Ministry appears to have been a strong supporter of the project and was actively engaged in it throughout preparation and implementation; albeit constrained by its unfamiliarity with the Bank and low capacity. Its performance was also tempered by delays in coming to an understanding about the division of responsibilities with the social fund, which in turn delayed project implementation. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DRC-Street Children Project (FY10) (P115318) Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance While the project was successfully completed, the performance of the PIU nevertheless was mixed. Slow project startup was in part due to its initial unfamiliarity with Bank procedures – which could be expected, and might have been handled by the Bank team with more thorough preparation. It was further affected by high turnover of technical staff and a lack of team cohesion, including initially with NGO partners (ICR, p. 19). Later on, the PIU would work closely with the implementing NGOs in developing their interventions: well-working NGO partners were key to achieving the project outcomes recorded above. An opportunity to channel time-limited Bank grant moneys (eight months) to pilot a conditional cash transfer program stalled when the PIU was unable to identify potential beneficiaries. There were some fiduciary issues (see section 11). Implementing Agency Performance Rating Moderately Unsatisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The project started designing a monitoring and evaluation system from scratch. Adequate indicators were defined and three levels of monitoring identified: inputs would be recorded in the project data base to produce monthly reports; information on outputs would be provided by contracted agencies (NGOs) and entered into the data base, and periodically audited through field visits and rapid assessments of performance by service providers; and beneficiary assessments to gauge the level of satisfaction of beneficiaries. Evaluation was to adopt a pre- and post-test design. MINAS and the National Statistics Institute were to be involved in the design and implementation. b. M&E Implementation Initially, the PIU and contracted NGOs were to manage data collection and manipulation. When the processes were sufficiently tested, MINAS staff was to be trained in the use of the data and gradually take over monitoring. Evaluation was to be contracted to specialists, given its rigorous approach. In practice, the M&E system lacked systematic PDO monitoring, including though qualitative assessments. Efficiency was also not measured. c. M&E Utilization Data collected through the monitoring system was used to monitor project performance and take corrective action. The ICR (p. 8) lists as examples the reallocation of the budget envelope based on the number of beneficiaries per location; the harmonization of costs between locations; and the establishment of norms and standards drawing on the information in the system. In project evaluation, households in neighborhoods with preventive activities would be compared to neighborhoods not exposed to such activities; likewise, street children in exposed neighborhoods would be compared to ones not exposed. M&E Quality Rating Modest 11. Other Issues Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DRC-Street Children Project (FY10) (P115318) a. Safeguards According to the ICR, the project triggered no safeguard policies. It was assigned the environmental category C. b. Fiduciary Compliance Financial management. According to the ICR, fiduciary responsibility was initially given to the Social Fund of the CDR, based on its long experience with IDA projects in the CDR, and the relative inexperience of MINAS. The Social Fund’s operating manuals were revised accordingly. The dual accountability that this created appears to have generated some friction between MINAS and the Social Fund. According to the ICR (para 30), in 2012, allegations of financial mismanagement and corruption within the PIU was brought up and INT got involved. The Social Fund conducted a thorough assessment that confirmed that $1600 had been misappropriated. This incident further strained the relationship between the PIU/MINAS and the Social Fund (but did not affect project outcomes according to the ICR), eventually contributing to the decision to restructure the project in August 2014. As one of the conditions for transferring fiduciary responsibilities to the PIU, an internal auditor was recruited to reduce risks of mismanagement, which was deemed satisfactory to the Bank. No further concerns arose according to the ICR. At mid-term review, in 2012, fiduciary responsibility was transferred to the PIU, which was appropriately strengthened. Although there had been tiimes when the project's financial management was downgraded from Satisfactory to Moderately Unsatisfactory, improvements were undertaken and according to the ICR (para 36), "financial management improved considerably." All external audit reports, including the final project audit of FY 2015, were unqualified and submitted on time. According to the project team, the project complied with the Bank's financial management policies. Procurement. According to the ICR, procurement occurred at two levels: the selection of the implementing NGOs; and purchase of larger goods and recruitment of consultants. In the project’s early years, the Social Fund was responsible for performing the procurement function. Lack of proper oversight by the Social Fund team as a result of high turnover of staff led to poor quality of procurement documents. During the mid-term review, the PIU was given responsibility for procurement supported by a procurement consultant. Procurement was subsequently rated Satisfactory (it had been downgraded to Moderately Unsatisfactory in 2012). According to the project team, the project complied with the Bank's procurement policies. c. Unintended impacts (Positive or Negative) N/A d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Insufficient evidence on the Outcome Satisfactory Moderately Satisfactory prevention objective and on efficiency. Risk to Development Outcome Substantial Substantial --- Bank Performance Moderately Satisfactory Moderately Satisfactory --- Borrower Performance Moderately Satisfactory Moderately Satisfactory --- Quality of ICR Substantial --- Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DRC-Street Children Project (FY10) (P115318) Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons From the ICR: Establish strategic partnerships to capitalize on learning and strengthen government response. Tackling the issue of street children is a relatively new field for the Bank, especially in Africa. By working through experienced NGOs partners, the Bank seized the opportunity to learn about and scale up good practices and help the government engage productively with NGOs to jointly develop standards of practice appropriate to the local context while informing the design of future Bank projects. In doing so, the project made a long-lasting contribution to the sector. New strategies are required to reach older street children. One of the intermediary indicator targets that the project did not meet was the number of children who benefited from participation in a vocational training program. This was largely because the age requirement to participate in vocational training is 15 years and very few of the children living on the street were 15 years and above. The project found that many of the older children, who had once lived in the street, were living in group-houses that they had organized themselves and paid for with the earnings they made through their work in the street. These youngsters were less inclined to take advantage of drop-in centers, making them more difficult to find and serve. Future projects could link this population group to youth interventions geared toward addressing their specific needs. Investing in street children prevention could be more effective if coupled with savings and income generating activities. At-risk families are usually poor families. Addressing their needs, either through cash transfers or micro-level savings-investment opportunities may help to reduce a major obstacle to creating an environment that encourages greater family cohesion. From IEG: Fiscal and Economic analysis as a basis for choice of approaches in the future. Even a simple cost analysis could have been informative, for instance in mapping the eventual fiscal costs of the models being explored in the project. Affordability to maintain the mechanisms promoted under the project should be a major risk in a country with constrained budget resources. Moreover, cost effectiveness or benefit cost analysis would also have been useful, considering the variety of interventions that are advocated in the project. 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR is generally consistent with the ICR guidelines. a. Quality of ICR Rating Substantial