72605 May 2012 Note Number 43 Independent Verification in Results-Based Financing | Esther Loening and Luis Tineo W ith pressure for greater accountability and value for money in public budgets, widening the use of results-based financing (RBF) and setting up adequate independent mechanisms to verify results is becom- ing increasingly important. In January 2012, the World Bank launched the Program-for-Results (PforR), an innovative financing instrument that disburses funds upon delivery of verified results in developing country programs. Until now, the use of independent verification has been limited to output- based aid (OBA), which is one of the RBF modalities; howev- er, there is potential for more widespread use of independent verification. This note highlights lessons from projects funded Viet Tuan Tran for GPOBA by the Global Partnership on Output-Based Aid (GPOBA) and the World Bank, and discusses elements that can best ensure the independence and effectiveness of a verification Verification protocols system. Verification protocols are the procedures used to certify that a service meets agreed specifications. The more detailed the Results-based financing (RBF) proposes radical changes verification protocol and methodology is—specifying results/ in development financing paradigms, shifting the focus of outputs, indicators, reporting frequency, and verification sys- development work from inputs to outputs or results. RBF tem—the less likelihood of conflicts later. commonly refers to a range of mechanisms designed to Results should be achievable—it is important that they are enhance delivery of services or operations, through the use under the control of the service provider—and directly linked of performance-based incentives, rewards, or subsidies. In to the rewards or incentives. Indicators must be specific, mea- output-based aid (OBA) schemes, typically, service deliv- surable, and verifiable. Too many indicators may make verifica- ery is delegated or contracted out by a funding entity—a tion complex. Attribution and regular measurement should be government or an international development agency—to a easy. In some cases, the actual suitability of indicators can only service provider. Payment depends on the achievement of be tested once operationalized. In addition to technical consid- pre-agreed and independently verified results. erations, results and indicators must be aligned with the stake- Independent verification is a key mechanism to enhance holders’ objectives, priorities, and interests. Setting appropriate the performance of service providers. For the funding indicators thus often requires an inclusive, iterative process. entity, it mitigates the risk of misuse of funds, providing In OBA projects, which focus on access and sustainable assurance and evidence that funds have been used for the service for poor users, outputs are defined as close to the intended purpose. In OBA projects, for example, verifica- desired development outcome as possible. For instance, in the tion is conducted by an independent verification agent infrastructure sectors (e.g. water, electricity supply), having (IVA). Its function is to ensure that only verified outputs working connections to the network installed triggers pay- are reimbursed, through a) certifying that the contractual ment of a portion of the subsidy. The remaining part is with- outputs, as reported by the service provider, have been held until after several months of verified service, which en- physically delivered and that pre-agreed standards of service tails a review of billing and collection records. Further quality have been achieved, and b) validating the service provider’s parameters can be verified, such as the timeframe for delivery, reimbursement request (performing cost reconciliation by coverage and sustainability conditions, and anti-corruption multiplying the quantity of outputs achieved by their unit cost), and recommending to the funding entity to honor payment. Figure 1 illustrates the typical verification cycle in Esther Loening is an Infrastructure Specialist and Luis Tineo is a OBA operations. Senior Operations Officer with GPOBA. Supporting the delivery of basic services in developing countries standards. These are illustrated in RBF schemes, such as the output influence the scope of verification. Since seeking 100 São Paulo Water Recovery Project (REAGUA) in Brazil, a percent verification of each agreed output or result is not fea- state government subsidy scheme that aims at increasing the sible, statistical sampling is often considered most economical quantity and quality of water in São Paolo’s watersheds. and effective, with precision (how closely the sample repre- Effective verification protocols also rely on available data sents the universe) and reliability (confidence level) taking sources, the capacity to measure indicators on a regular basis, into account factors such as available resources, and trust in and baseline data. In OBA projects, baseline data typically implementing agencies. includes current access to services, levels of service, and de- In large government programs such as those supported velopment indicators. by PforR, careful independent verification design is required In most sectors, verification includes physical inspection in because of the large number and geographic dispersion of the field, which is also subject to specific quality requirements. outputs/results. An example is Indonesia’s Local Government In health and education, subsidies are reimbursed for ongoing and Decentralization Project, which aims at improving the service. Schemes rely on routine service delivery data collec- accountability and reporting of the central government’s tion at facility level and cross-checked against facility registers. Specific Purpose Grants (DAK). Eligibility for loan reimburse- When feasible, patients follow-up (through spot-checks of a ment relies on reporting by participating local governments sample of households, or household surveys) allows to better upon completion of pre-agreed outputs in roads, water and assess use, coverage, and perceived quality of care. Third-party irrigation sectors, and independent verification. agents are also used to carry direct observation of care provi- One way to address the challenges related to geographic dis- sion and health facility assessment, which include interviews tribution is collecting data at the community level. The telecom with health staff and patients, and audits of management sector offers examples of automated verification of outputs. In practices, equipment and supplies. Ultimately, the size of the OBA projects aimed at installation of public phones in rural lo- program influences the choice of verification systems and cations, often the regulatory authority demands the installation methods. Nationwide programs may require control enhance- of an automated network management terminal in its offices, ments. For instance, the national performance-based financ- connected to the provider’s own network management system, ing health program in Burundi incorporates several layers of which provides instant alerts when a particular phone is out of verification, from the community to the provincial level. service, and collects statistical information related to traffic and Finally, reporting arrangements are important as they can maintenance. This system can be used for initial verification of affect speed of disbursement. In the GPOBA-funded health facilities installed and for recurrent verification that service is project in Uganda, disbursements were delayed due to the being delivered. There is scope for further community moni- complex cycle of approval of the IVA report. toring of service delivery, enhanced through the use of mobile phone applications (such as community monitoring of water Scope of verification supply), which can reduce verification costs. Independent verification is expected to provide reasonable Outsourcing or not assurance that the statements claimed by the service provider are free from material error. Clearly, factors including logisti- Verification of results can be carried out and by various cal capacity, governance, and geographic distribution of the parties, including government agencies, semi-autonomous Figure 1. Typical OBA Verification Cycle Service Provider Disbursement Public Funding Entity Pre-finance Outputs Output Verification Report Report on Independent Outputs Delivered Verification Agent Agreed Outputs Beneficiaries Verification Ex-post Review May 2012 Note Number 43 entities, statistical or audit entities (if they can demonstrate However, if an external party is to be recruited, the hiring pro- adequate capacity and independence), and third-party entities cess must be considered. Competitive selection takes time. For such as firms, nongovernmental organizations (NGOs), and example, in the GPOBA-funded water project in Cameroon, civil society representatives. In determining which of these the first bidding was unsuccessful, as all proposals exceeded options is most suitable, the objective is to reduce the inher- the available budget, requiring rebidding. ent risk of capture, primarily by the service provider, who is The decision of who hires the verification agent is also the main interested party; and protecting the funding entity critical: the hiring agency should not have vested financial against the potential manipulation of results. Conflict of inter- interest in the performance. When it is not practical for est issues need to be assessed when a government entity (such the implementing agency to hire the verification agent, the as the project’s implementing agency) is responsible for both service provider itself may do so. If the same service provider overseeing service providers and output verification; it may is the direct funds recipient, there exists scope for conflict of be motivated to inflate the results achieved, through political interest such as collusion or lax application of verification interference, to obtain payments. protocols, as the verification agent has to provide independent The main institutional solution adopted by GPOBA and, assessment of output delivery while maintaining a business in some cases, the World Bank, has been the use of external relationship with the service provider. Risks can be mitigated firms or consultants (both local and international). Firms can by hiring either a reputable auditing firm with technical capa- bring capacity and credibility when the independence of gov- bilities in the sector, or the service provider’s existing external ernment agencies is questionable. In Uganda’s Reproductive auditors. The belief is that auditors are subject to established Health Voucher program, weak professional councils and professional standards, and will not risk their reputation, other regulatory bodies justified the use of a firm as an in- credibility and license. Noteworthy, in projects where the terim measure to conduct output verification. In the long run, World Bank funds the verification agent’s contract, conflict of strengthening country systems to do so will become critical. interest are mitigated by requiring the hiring entity to follow When government agencies perform verification functions, World Bank selection of consultant guidelines, which include ideally, an independent and capable regulatory or audit agency World Bank prior-review of the process and no-objection to should be designated. Finding such an entity may be challeng- the contract award. ing. Internalizing the verification function is the typical ap- Considering sustainability and ensuring sufficient re- proach in the telecom sector, where the telecom sector regula- sources for verification activities are important issues. Project tory authority itself conducts verification. While in some cases, implementers need to budget adequate funding, as the the regulator hires individual consultants to assist, the regula- verification of results can be costly (and it is not always clear tor bears the full responsibility for verification of outputs. For who pays for it, as governments are not used to this type of example, for the universal access project funded by the World function). In GPOBA projects, the average contract value of Bank that replicated the Mongolia GPOBA-funded telecom IVA averages 2 to 5 percent of the total GPOBA project fund- pilot, the country’s Communications Regulatory Commission ing. For some of the projects, the budget allotted for output hired a local firm with an engineering background to con- verification turned out to be quite low. Rural projects tend to duct the independent verification. The use of state auditors be more expensive to verify because the outputs tend to be is proposed in PforR operations such as Vietnam’s results- more dispersed. Recognizing this, in Brazil’s REAGUA, the based Rural Water Supply and Sanitation and Indonesia’s state authorities allocated 7 percent of the program amount Strengthening DAK Transfers to Local Governments Program. (US$10 million) to the IVA contract. The qualifications of any verification agent’s team are criti- cal. Although the practice of output verification is relatively Internal controls and capacity new, finding qualified teams has not been a major chal- lenge for GPOBA. In only two GPOBA-funded projects (in Verification is also directly impacted by the reporting capac- Indonesia and in Cameroon) has the lack of qualified agents ity and internal control systems of service providers and/ led to delays in implementation. Teams most often possess or implementing agencies. In the GPOBA-funded project technical sector expertise such as engineering, combined in Cameroon for example, initial weak reporting capacity with financial audit expertise. The selection criteria have also of both the service provider and the implementing agency included experience working in poor communities, and, espe- resulted in delays in delivering the output verification reports, cially for rural projects, an understanding of the local culture which in turn delayed disbursement to the water operator. and language. For example, the international firm recruited It may be necessary to strengthen the accountability and as IVA for the health project in Yemen hired a local medical transparency of billing systems, financial accounting, and data doctor to help with the verification process. Finally, statistical collection systems in service providers and/or implementing knowledge is useful to conduct customer satisfaction surveys. agencies (such as local governments), to demonstrate that services have actually been delivered. Recruitment process and payment Generally, RBF tends to increase the accuracy of report- ing of results, and in particular the completeness of reporting. In principle, to expedite disbursement, it is preferable for In the water supply and sanitation GPOBA-funded project the verification mechanisms to be ready as early as possible. in Morocco, the quarterly inspections by the independent consultant helped to improve the operators’ progress report- mechanisms to promptly trigger payments for results. With ing requirements and implementation methods. A robust the mainstreaming of RBF approaches, internal monitoring verification and counter-verification system remains crucial and controls, as well as verification procedures in implement- to minimize the risk of inaccurate reporting of levels of ser- ing partners, may need strengthening for the incentive frame- vice provision and of quality measurement. work, and the verification that goes with it, to function well. The design of a verification system involves practical choices, Collaboration among stakeholders depending on parameters such as the size of the program, resources and capacity available, and the broader operating Due to the pressure to get paid, service providers expect environment. Ultimately, all parties should work together to verification to be carried out as soon as they have delivered structure verification systems that link financing and results. the service. Collaboration of all parties (funders, implement- ing agency, service providers, communities, and verification References agent) is critical in ensuring timely disbursement. At the beginning of the project, the verification agent should agree Vergeer, Petra, Rogers, Deirdre, Brennan, Richard and Sarcar, on the verification protocol (including the format for the veri- Shiril. June 2010. Identifying Indicators for Performance- fication reports and sampling methodology), and make sure Based Contracting (PBC) is Key: The Case of Liberia. that it is acceptable to all parties. Debriefing on preliminary Results-Based Financing for Health (RBF). Washington, findings at the end of each verification exercise then gives the DC. World Bank. service provider and the implementing agency an opportunity Results-Based Financing for Health (RBF), World Bank. April to respond and take corrective measures while the verification 2010. Verification at a glance. Editors: Naimoli, Joseph and agent prepares its report. Vergeer, Petra. Washington, DC. World Bank. Similarly during implementation, it is in the interest of Lindsay, Morgan, Beith, Alix and Eichler, Rena. November service providers and implementing agencies to facilitate veri- 2011. Performance-Based Incentives for Maternal Health: fication. For example, in two GPOBA-funded water projects Taking Stock of Current Programs and Future Potentials in Uganda and Mozambique, the service providers did not Bethesda, MD: Health Systems 20/20 project, Abt Associ- provide appropriate staff or logistical support, leading to de- ates Inc. lays in the verification efforts and subsequent disbursements. Ellis, Peter, Mandri-Perrott, Cledan, and Tineo, Luis. 2011. Conversely, if the pace of output delivery is slower than Strengthening Fiscal Transfers In Indonesia Using an expected, flexibility in verification may be required. In several Output-Based Approach. OBApproaches Note 40. GPOBA-funded projects, IVAs performed ad hoc verifica- Velez, Carlos E., and Julia Tierney. 2010. Brazil: Innovative tions, as they were not able to predict when service providers Results-Based Approach to Tackling Water Scarcity in São would deliver the outputs. Paulo. Enbreve Number 160. www.worldbank.org/enbreve Last, making verification findings available to the public Chauvot de Beauchêne, Xavier. 2009. Output-Based Aid in provides necessary checks and balances, reinforcing demand- Morocco (Part 1): Extending Water Services to the Poor in side governance and accountability to the end users. Urban Areas. OBApproaches Note 25. World Bank. 2007. Structuring Output-Based Aid (OBA) Ap- Conclusion proaches in the World Bank Group: Operations Guidance Note for Staff. Washington, DC. World Bank. Independent verification is not yet an established practice in GPOBA. www.oba-data.org development projects. The challenge is to put in place effective World Bank. www.worldbank.org/p4r About OBApproaches OBApproaches is a forum for discussing and disseminating have been chosen and presented by the authors in agreement recent experiences and innovations in supporting the delivery with the GPOBA management team and are not to be attribut- of basic services to the poor. The series focuses on the provi- ed to GPOBA’s donors, the World Bank, or any other affiliated sion of water, energy, telecommunications, transport, health, organizations. Nor do any of the conclusions represent official and education in developing countries, in particular through policy of GPOBA, the World Bank, or the countries they output- or performance-based approaches. The case studies represent. To find out more, visit www.gpoba.org e Global e Partnership on Global Partnership Output-Based Aid on Output-Based Aid Supporting the delivery of basic services in developing countries