public sector REFORM 90172 WASTE MANAGEMENT REFORM pharmaceutical PROCUREMENT PARLIAMENTary OVERSIGHT of budgets Constitutional mandates in health OPEN CONTRACTING capacity of local governmentS ACCESS TO INFORMATION E-LEARNING for CITY officials SOCIAL ACCOUNTABILITY in education Cases in Outcome Harvesting Ten pilot experiences identify new learning from multi-stakeholder projects to improve results 1 2 Cases in Outcome Harvesting Ten pilot experiences identify new learning from multi-stakeholder projects to improve results 3 Copyright © June 2014 The World Bank 1818 H Street, N.W. Washington, D.C. 20433, USA All rights reserved Jenny Gold coordinated the outcome harvesting pilot experiences with support from Ricardo Wilson-Grau. Sharon Fisher provided analysis, writing, editing, and design. Dawn Roberts provided support. Samuel Otoo provided overall guidance. The following sector leaders and team members worked on the pilot outcome harvesting process for their respective cases: 1. Leadership for Results: Developing Capacity and Delivering Results toward Public Sector Reform in Burundi: Team leader Jose Edgardo Campos, with Benjamina Randrianarivelo, Kay Winning, and others. 2. Implementing Reform Initiatives in Solid Waste Management in Bosnia: Team leader Manuel Contreras, with Hirut M’cleod, Jose Edgardo Campos, and others. 3. Improving Governance in Pharmaceutical Procurement and Supply Chain Management in Kenya, Tanzania and Uganda: Team leaders Yvonne Nkrumah and Julia Mensah, with Marylou Bradley, Jilliane T Cabansag, and others. 4. Strengthening Parliamentary Oversight of National Budgets in Africa: Team leader Mitchell O’Brien, with Paulina Biernacka, Miraim Bensky, Sruti Bandyopadhyay, Vienna Marie Pozer, and others. 5. Priority Setting and Constitutional Mandates in Health: Team leader Maria-Luisa Escobar, with Leonardo Cubillos, Roberto Iunes, Antonio Paniagua, Janet Bonilla and Santiago Pereira, and others. 6. Improving Open Contracting Processes at the Country and Global Level: Team leader Marcela Rozo, with Robert Hunja, Marcela Rozo, Michael Jarvis, Sara Danish, Ousmane Deme, Felipe Estefan, Kathrin Frauscher, Norma Garza, Carey Kluttz, Lindsey Marchessault, Julia Mensah and Caroline Spruill, and others. 7. Strengthening Capacities of Local Governments in South East Europe: Team leaders Tamara Nikolic and Sabine Palmreuther, and others. 8. Strengthening Implementation of Legislation on Access to Information across Latin America: Team leader Marcos Mendiburu, with Luis Esquivel, Elsa Peraldi, and others. 9. Scaling up Capacity Development of City Officials and Practitioners across China Through eLearning: Team leader Mansha Chen, with Sheila Jagannathan, Sheng Li, and others. 10. Improving Social Accountability in the Philippines Education Sector: Team leader Megan McDermott, with Carolina Vaira, Keith McLean, Jennifer Shkabatur, and others. Design: Sharon Fisher 4 Summary It is often difficult to understand and document Outcome harvesting is useful for complex aspects how change happens in complex development of a program, when the significance of particu- programs that involve multiple social actors—what lar milestones and outcomes may be unknown in was achieved and how did it happen? It is useful for advance. There is often a need for learning to under- teams to adaptively learn from results depending on stand how change happened. the stage of the program and the questions teams The harvesting process is stakeholder-centered want to answer about its results. and captures qualitative, tacit knowledge. It includes Tools such as outcome harvesting (from the Out- tools to substantiate and analyze this knowledge come Mapping Learning Community1) offer methods collaboratively and communicate progress toward to manage knowledge and learn from complex and impact to clients, management, and partners. The difficult to monitor development processes. Such tools are flexible to adapt to a program’s design processes may involve profound development chal- and can provide useful details to inform the theory lenges, experimentation with new innovations, or of change, implementation lessons, outcomes, and country-led behavioral and institutional changes by indicators. social actors with different roles and capacities. Understanding and interpreting milestones and Pilot Experiences outcomes—achieved or not achieved, planned or This report documents a “stage one” pilot to identify unexpected—can facilitate strategic, systematic how outcome harvesting could be integrated with knowledge sharing to improve results. The learning the World Bank’s results management approach, for can be iterative during the program implementation learning during a program’s implementation and cycle or retrospective to learn from an entire pro- review stages. Specifically, the pilots examined how cess, and reveal the key changes that mattered. outcome harvesting tools could lend themselves Other development partners have also used to learning about how change happens in complex outcome harvesting and/or outcome mapping, such aspects of programs. For instance, what combina- as the Ford Foundation, UK Department for Interna- tion of interventions worked to advance particular tional Development, Canadian International Devel- changes, what behavioral and institutional changes opment Agency, Oxfam, and Open Society Institute. were advanced, and what was the right mix of social The Overseas Development Institute supports the actors involved to achieve results. Outcome Mapping Learning Community, and the The initial pilots used outcome harvesting to US Agency for International Development identi- review progress for 10 ongoing knowledge initiatives fies outcome harvesting tools as useful for complex supporting World Bank programs or projects in stra- aspects of programs and strategies as a complement tegic thematic areas (see table on next page). to performance monitoring.2 An August 2013 discus- Task teams participated in a hands-on workshop sion paper from the UN Development Programme to introduce outcome harvesting. Then each team evaluation office selected Outcome Harvesting as designed their outcome harvesting exercise to focus one of 11 promising innovations in monitoring and on complex aspects of the initiative where learning evaluation practice.3 was most needed. They used a customized report- ing format to harvest outcome information with team Outcome Harvesting members, clients, and stakeholders over 1–2 months. Outcome harvesting is used to identify, monitor, The format was customized to include informa- and learn from changes in social actors, through tion on the context and challenges related to each harvesting bites of detailed outcome information recorded outcome or milestone. with colleagues, partners, and stakeholders.4 The The teams retrospectively harvested informa- information describes what changed, for whom, tion from about 2–5 years of program results. The when and where, why it matters to the development reported information was then verified for accuracy objective—the significance of the change—and how through substantiation feedback from stakeholders. the program contributed to the change. 5 The analysis of each initiative’s achievements • Gather evidence and lessons from the change included an outcome map to visualize the changes process that can complement other M&E tools by timeline and actor and a change strategy map Future pilots could identify how to use these tools that summarized the outcome information to com- to look for patterns of change across a thematic area municate the theory of change and results chain. or program portfolio, and how to build them into the (They used the Capacity Development and Results design stage of program/project components. Framework as a basis for analysis.5) Then, task team For those with access to the World Bank Intranet, members, clients, and stakeholders reviewed the visit http://outcomemapping for the latest guides cases to ensure mutual agreement on the analysis and tools. These resources and others are also avail- and presentation. able through the Outcome Mapping Learning Com- munity at www.outcomemapping.ca. Learning and Resources The World Bank has used the pilot experiences to NOTES customize an evolving toolkit of resources for learn- 1 See www.outcomemapping.ca ing across the program cycle. Key uses identified are: 2 USAID. December 2013. Discussion note: Complexity-Aware • Rigorous knowledge management of complex, Monitoring 3 See https://undp.unteamworks.org/node/370238 multi-actor processes 4 An outcome is what each social actor (or change agent) did, • Review results from complex program/project or is doing, that reflects a significant change in their behavior, components to improve the benefit of relationships, activities, actions, policies, or practice. 5 Otoo, S., N. Agapitova, and J. Behrens. 2009. The Capacity interventions, sustainability and mix of actors Development Results Framework. World Bank. involved Table of Cases No. Title of Case Development Objective Sector/Global Page Practice 1 Leadership for Results: Developing Capacity Improve delivery of central and Public Sector p9 and Delivering Results toward Public Sector decentralized public services for citizens Reform/ Reform in Burundi at all levels in Burundi Leadership 2 Implementing Reform Initiatives in Solid Improve the value of municipal services Urban/ p23 Waste Management in Bosnia for citizens in South East Europe. Leadership 3 Improving Governance in Pharmaceutical Improve citizen access to essential Health p31 Procurement and Supply Chain Management medicines in Kenya, Tanzania and in Kenya, Tanzania and Uganda Uganda. 4 Strengthening Parliamentary Oversight of Improve the benefits of public spending Governance p42 National Budgets in Africa for citizens in Africa 5 Priority Setting and Constitutional Mandates Improve the level and distribution of Health p51 in Health health outcomes by applying rights- based principles to health policy. 6 Improving Open Contracting Processes at the Improve the benefits of public goods Governance p58 Country and Global Level and services for all citizens 7 Strengthening Capacities of Local More livable and sustainable cities in Urban p69 Governments in South East Europe South East Europe that provide a high quality of life for citizens 8 Strengthening Implementation of Legislation Improve service delivery for citizens Governance p83 on Access to Information across Latin across Latin America America 9 Scaling up Capacity Development of City Improve public service delivery in Public Sector p91 Officials and Practitioners across China China’s rapidly growing urban areas Reform/Learning Through eLearning 10 Improving Social Accountability in the Improve the quality of services and Governance/ p97 Philippines Education Sector education performance of public Education schools for citizens in the Philippines 6 Conclusions from the Pilot Experiences Going forward the World Bank could use outcome to validate program results. Outcome harvest- harvesting tools for results learning and knowledge ing tools, for example, can be used to validate capture from complex development programs where self-reported results by World Bank task teams. it is difficult to show impact—such as those that This triangulation can be especially helpful in involve multiple actors, institutional changes, and experimental situations or in cases where project empowerment processes. Outcome harvesting in size is small and strong contextual information is particular is a highly adaptable tool that could be required for learning. further piloted for portfolio-level learning across a large program as well as project-level learning. Seek evidence and lessons from an implementation process Conclusions • Outcome harvesting tools can be used to gather The following conclusions on ways to use the out- evidence on key interventions and identify essen- come harvesting tools going forward are based on tial lessons, such as how best to adapt successful feedback from the task teams and clients involved in efforts to different contexts and how to choose the pilot experiences. the best mix of actors to involve. Teams recom- mended that precise learning could be used for Capture tacit and contextual knowledge from informing program design and delivery, as well as program implementation defined areas for further operational research and • Outcome harvesting tools can capture precise evaluation. information on a program that is not typically found in implementation reports. This information Use for client reporting on results and can cover context, behaviors, and institutions, promoting learning by doing and how interventions are designed to have a • Outcome harvesting offers a simple tool for cli- particular influence. These are details that often ents to report on results, particularly from com- remain in the minds of implementers and are dif- plex processes involving multiple actors at differ- ficult to measure. ent levels. • Clients can also use outcome harvesting tools Use for systematic learning to maximize benefits to manage their own systematic learning dur- of interventions ing implementation, such as learning how to • Outcomes can be recorded in information sys- maximize the benefits of an intervention in their tems during implementation of complex pro- context. grams. Interviews, team journals, document reviews, focus groups, and surveys are all tools to Challenges to Moving Forward harvest outcomes. Outcome harvesting requires teams to plan for and • Outcomes from a program or across a portfolio report on actions taken by social actors, much of can be strategcially harvested and reviewed at which is unrecorded or qualitative in nature. This key points to test a theory of change and inform requires a shift in mindset on results reporting— course corrections to maximize the benefits of teams are more accustomed to reporting on inter- interventions. Outcome harvesting tools can offer ventions and major achievements, not information a context-specific lens to inform and complement that lies in the “middle” and sits in the knowledge of learning from other M&E data including random- team members and local actors. ized experiments. Support is also needed, through advisory services and training in the process and tools and guidance Triangulate outcomes with other data to validate on analysis, validation, and reporting of the collected results, particularly for small projects data. Teams may require support to customize • Outcome details can be harvested as part of a the outcome harvesting tools to a program’s strategy and triangulated with other data sources implementation context and strategize the tools 7 to be most effective: harvest the most useful information; optimally complement other M&E and knowledge gathering activities; collect feedback from independent, knowledgeable persons on outcomes; and integrate learning on significant changes back into a program over time. It is desirable to integrate the tools with orga- nizational reporting and review processes, so that outcome information can be triangulated with other results information to inform learning. 8 Leadership for Results: Developing Capacity and Delivering Results Toward Public Sector Reform in Burundi S ince 2006, government leaders and teams Development Objective in Burundi have engaged in collaborative Improve delivery of central and decentralized public problem-solving using results-oriented man- services for citizens at all levels in Burundi. agement tools. They have been learning how to Problem overcome the institutional constraints that block the As a fragile post-conflict state, Burundi must unblock improvement of their public services and reforms constraints to help its civil service rebuild leadership and to draw lessons to scale-up solutions across that will be held accountable for delivering effective ministries and sectors. The World Bank Institute services to citizens, especially in remote areas. (WBI) Leadership for Results (L4R) program in Problems include corruption, weak collaboration Burundi supported the government to develop and across government ministries, inefficient rules, limited adapt the results-oriented tools, strategy, coaching adaptability and legitimacy of processes. and training for this program specific to their own Specific Objectives context to achieve their reform goals. Strengthen public sector leadership and participatory To capture the change process toward develop- strategies to rebuild services in priority areas; improve effectiveness of public service delivery for citizens; com- ment impact upon the civil service from outcomes bat corruption and policy inefficiencies that hinder de- delivered throughout the duration of this project, velopment outcomes; strengthen the National School WBI mapped more than 40 outcomes1 from this pro- of Administration (Ecole Nationale d’Administration, or gram using a customized outcome mapping tool2. ENA) to support results-focused learning within the civil The outcomes show examples of changes from service for improved project implementation. 9 the larger program that have impacted the entire needed services and to gain political capital and civil service of Burundi. Outcomes were mapped to trust. illustrate changes in the leadership and authorizing Burundi’s political leaders embraced the new environment for results in the public sector, adaptive approaches of the L4R program, which included: learning around results to rebuild the civil service at • High-level government retreats and oversight all levels and institutionalization of results across the processes to prioritize results areas, provide an civil service. authorizing environment for change and encour- These visual maps present the sequence of out- age a culture of results and learning. comes achieved by change agents—the leaders and • The empowerment of leaders at different lev- teams, coaches at different levels and other actors els and their implementation teams to become involved in the process. The maps illustrate how the results-focused and engage in efforts to itera- outcomes connected and built on each other over tively identify solutions for increased effectiveness time to form multi-actor, institutional processes for throughout the public sector. change to address the program’s objectives and Through the L4R program, leaders set specific goal. goals for results improvements and engage with WBI team members identified and formulated teams to pilot initiatives to experiment with solu- the outcomes, presenting an explanation of their tions. The teams collaboratively track progress to significance and how WBI had contributed—directly adaptively learn from what works and does not work or indirectly, in a small or big way, intentionally or to address constraints and enable achievement of not—by empowering the change agents to take new the desired results. actions. Then, roughly 20% of the outcomes were Through this program, the second Vice President independently substantiated for credibility in the requested WBI’s assistance to strengthen the capaci- mapping exercise. ties of the country’s leaders to drive change at the institutional level, accelerating the implementation BACKGROUND of national programs aimed at delivering results for In 2005, Burundi was emerging from a 12-year citizens. WBI offered advisory support to the L4R civil war. It had recently established a constitution, program’s steering committee, guidance and rapid elected a President and prepared a national Pov- results coaching to civil service leaders to adopt and erty Reduction Strategy (PRS). A key challenge was integrate the results-focused management tools and how to meet results for citizens given its limited support to ENA to train civil servants and develop a human capacity, limited resources and institutional curriculum starting in 2010. problems. Government at all levels and especially in remote areas faced high turnover and low levels OUTCOME AREAS of capacity in staff to perform their responsibilities. The process of change from this program can be Many constraints blocked improvements for citizens seen in three streams of outcomes that represent the and reforms to strengthen governance, economic major change paths, as seen in Figure 1: growth and other priorities: the public sector faced (1) Leadership and Authorizing Environment for policy inefficiencies, a lack of accountability to citi- Results zens, dysfunctional communication between minis- (2) Results Toward Reforms Through Adaptive tries and outdated policies. Learning-by-Doing Given these challenges and the concern of (3) Embedded Results-oriented Problem-Solving in stakeholders to rebuild Burundi’s public sector, the Public Sector. new government wanted to develop leadership and All of the outcomes were analyzed and classified management of project and implementation capaci- according to the types of change they achieved. ties within the public sector and learn how to gener- They were then grouped based on how they con- ate results to advance priorities set by stakeholders nected to each other to form a story for change. The in the PRS. Delivering on the newly elected Presi- maps in Figures 2–5 summarize the outcomes and dent’s pledges in his five-year plan—which included correspond to the numbers in brackets within the free primary education and child health care— was text. important to demonstrate success in delivering 10 Figure 1. Change strategy showing how change happened to advance progress toward goal WBI Contributions • Advisory support to Change Agents4 Steering Committee to conceptualize Leadership • President for Results (L4R) program, • Vice presidents analyze priorities • Steering Committee • Guidance to adopt results • Ministers, directors, governors and managers from tools, such as coaching, central and subnational government offices and cabinet retreats, review agencies sessions, rapid cycle initiatives and South-South • Internal coaches and teams within government exchanges ministries and agencies • Support to National • WBI external coaches (external to government but School of Administration supporting internal coaches and teams) (Ecole Nationale • ENA d’Administration, or ENA) • Citizens and community groups to train civil servants and develop curriculum Change Strategy5 Partners Outcome Area 1: Leadership and Authorizing Environment for Results • World Bank operations • Presidential oversight of L4R program • ENA • Multi-agency priority setting, problem solving and feedback process through steering committee, government retreats, review meetings, South-South exchanges • Awareness of leaders around problems and new Problems Addressed ways to implement solutions • Government ministries and • Leaders empowered to voice and act on reforms agencies at all levels and using L4R program especially in remote areas face severe problems Outcome Area 2: Results Toward Reforms with in effectiveness and Through Adaptive Learning-by-Doing inefficiencies that hinder • Sectors build capacity by piloting initiatives to service delivery address problems and identify solutions • Lack of leadership and • Coaches and teams to implement pilot initiatives learning around what that address problems worked and what did not • Pilot initiatives achieve outcomes to improve work to achieve results in effectiveness of services and policy, reduce public service delivery corruption risks, improve responsiveness • Pilot results are cyclically built upon to provide lessons to scale-up across government Outcome Area 3: Embedding Results-oriented Problem- Solving in Public Sector Development Objective • Coaches embedded in ministries with skills for • Improve delivery of central implementaton of L4R program and decentralized public • Decree for ministries to apply L4R tools services for citizens at all systematically levels in Burundi • ENA supporting implementation of L4R program in public sector 11 Figure 2. Map of outcomes showing changes linked and built over an eight-year timeframe (1) 2nd Vice President (14) ENA began established Leadership strengthening coaches in for Results Program ministries; over 2,000 team (10) Civil service members embedded in 90% (L4R) to tackle (21) Civil launched initiative of national ministries challenges to service servants that reduced delivery and appointed coaches are number of fictitious Steering Committee to embedded staff within civil oversee it (13) President and in national service and Steering Committee ministry suspended 728 held 3rd govern- structures salary payments ment retreat to plan (2) Steering new initiatives in Committee chose (16) ENA economic growth, first pilot initiatives developed (9) Civil service public finance and in health and sub-national (20) ENA and MoE scaled fiscal space and education sectors training forms sub- up teacher governance materials in national payment initiative network of local language } beyond pilot decentralized (3) Steering Committee Refer to pilot III to support and applied to coaches started to use South- for improving implementation recruitment and trained South exchanges to electricity of similar payment of all in local enhance L4R program connectivity leadership new teachers language program at decentralized (4) Pilots (5) MoE and (15) Government level launched MoH publicized officials at retreat in Bubanza pilot results at discussed using (19) ENA province; government retreat; leadership pro- negotiating President and (8) Ministers shared (17) ENA MoE gram for National possibility of Vice Presidents difficulties unblocking engaged key unblocked Strategy for Good budget from expressed buy-in problems at review stakeholders constraints Governance and government for similar initiatives session chaired by 2nd to identify to deliver Anti-Corruption, for national in other areas of Vice President options to 25,000 text- Doing Business coach to books; MoH public sector reform and the Strategic institutionalize re-activate unblocked Framework for the Leadership leadership constraints Growth and Pov- for Results program that led to (6) Steering Committee erty Reduction program process with (11) President par- ENA 482 preg- initiated process ticipated in oversight of nant women to build knowledge Leadership for Results tested for among public servants Program within govern- (18) ENA HIV in 30 to use results-based ment, in particular by developed days methods to tackle participating in 2nd visioning paper to constraints across government retreat } public administration scale-up program Refer to pilot endorsed by I: delivering ministries and textbooks in (12) Vice Presidents signed (7) Since 2007, donors Bubanza province decree to expand Leadership public servants with coaching capacities for Results Program to all Refer to pilot II in results-based government ministries to } for accelerating methods are support PRS; ministries were to the payment of embedded in major launch initiatives each trimester teachers ministries nationally to advance sector priorities 2006 2007 2008 2009 2010 2011 2012 2013 Institutional changes Learning/capacity changes Outcomes related to societal, policy and organizational changes. Other outcomes related to Leadership and participatory priority setting to strengthen public sector reforms awareness, knowledge or skills, collaborative action, or the use Policy changes to combat inefficiencies and corruption of knowledge or innovative Effectiveness, adaptability and responsiveness of public service delivery solutions. Other outcomes * Outcomes selected for Outcomes that changed direction or thinking behind the initiative substantiation; see sidebar. 12 Outcome Area 1: Leadership and that could support individual ministries as well as authorizing environment for results cross-sector teams to collaborate and find solutions Enabling a public sector leadership supportive of to rebuild institutions and systems in Burundi. Based results-oriented action has been important to the on this initial proof of results, the second Vice Presi- overall change strategy of the L4R program. dent committed to extend the L4R program. [5] From the start, the second Vice President of In June 2007, the second Vice President, as Burundi (Burundi has two Vice Presidents) initi- head of the Steering Committee, committed to ated the L4R program to develop capacity within build knowledge among public servants around the the public sector staff to problem-solve to address results-oriented management tools of the L4R pro- constraints in public services and advance gover- gram to tackle constraints across the public sector. nance and other reforms. The second Vice Presi- To support this effort, he initiated a process to train dent headed a Steering Committee to provide a coaches across the government. Ministries could coordination mechanism that would sustain focus then use the methods to jumpstart new initiatives and action for the program. This provided a strong and unblock those that had become stuck. [6] authorizing environment for the program’s imple- As the L4R program expanded to additional mentation. [1]3 ministries throughout the duration of the program, In late 2006, the Steering Committee agreed that government retreats were organized periodically as the first pilot initiatives to advance results in the PRS a tool for oversight and review of initiatives as well would be in the health and education sectors. [2] as planning. The retreats also provided a forum for This decision was based upon initial scoping and South-South exchanges to learn from officials in analysis work undertaken through background stud- other African countries, such as Kenya, which had ies in these sectors and a collaborative prioritization developed a similar public sector results-focused process. During the first government retreat the program. Key outcomes linked to the government Steering Committee also started to use South-South retreat process included: exchanges to enhance their opportunities for learn- • Initiatives were implemented by a growing num- ing under the L4R program, drawing upon experi- ber of ministries by March 2009. During the gov- ences from Rwanda and Madagascar of using the ernment retreat held at this time, the President L4R program in reform processes and international reviewed the progress of the results and learning experts applying results-oriented management tools from the ongoing initiatives under the L4R pro- and adaptive leadership techniques to implement gram and he, along with his two Vice Presidents, their development strategies. [3] expressed further commitment to broaden and In May 2007, Ministries of Health and Education institutionalize the program’s culture of results officials publicized the successful results of their first within the government to advance PRS results. pilot initiatives at a high-level government retreat [11] within this program, with the President, cabinet • Following the second government retreat in members and representatives from civil society March 2009, the Vice Presidents of Burundi organizations (CSOs) participating. Some exam- agreed to sign a public decree to mandate ples of the results are that the Ministry of Health expansion of the program across government. increased the number of pregnant women being [12] tested for HIV in the province from 71 to 482 within • In March 2011, the President and Steering Com- 30 days, and the Ministry of Education delivered mittee of the L4R program held a government 25,000 textbooks throughout the province within 60 retreat to prioritize solutions and plans to con- days instead of the normal time of around one year. front three areas where stronger results were criti- [4] (The Ministry of Education pilot is described in cal: economic growth, public finance and fiscal Outcome Area 2.) space and governance. [13] The pilots demonstrated solutions to overcome • In December 2011, a retreat was held to prioritize long-standing constraints in two priority sectors, the use of the L4R program to implement the jumpstarting results beyond expectations. This National Strategy for Good Governance and Anti- “demonstration effect” allowed stakeholders to see Corruption as well as improve the Doing Business the potential for a transversal government program indicators for Burundi. [15] 13 Initiatives using rapid cycle learning were cycli- Outcome Area 2: Results toward reforms cally launched following these high-level events to through learning translate into action the recommendations emerging A key part of the change strategy of the L4R program from each high-level retreat. Each retreat reinforced also involved the use of pilot initiatives to adaptively the continuing commitment of the government to learn. In each pilot, teams of Burundian civil servants use the L4R program to deliver results for Burun- and community members were empowered to iden- dians. The President’s involvement in oversight tify solutions to address specific systemic constraints. throughout the process and participation during all The solutions could be scaled-up and shared with of these events ensured continued commitment to other agencies facing similar problems. the program from leaders in government ministries, Three pilots selected from over 400 initiatives and encouraged their experimentation and learning implemented to date provide examples of the around results. problem-solving process for results; the visual maps Contribution of WBI to Outcome Area 1: WBI in Figures 3–5 correspond to their outcomes. had influenced the second Vice President’s commit- ment to act on the need for capacity development Pilot I: Delivering textbooks in Bubanza of public sector leaders and teams during a scoping province (Figure 3) mission to Burundi in May 2006, which secured the Between November 2006 and January 2007, the agreement of the then Vice Presidents to support a Ministry of Education launched a successful pilot L4R program. WBI supported the creation of a new initiative for the L4R program in the province of Leadership Steering Committee to initiate and over- Bubanza. It addressed the problem of textbooks see the implementation of the program. Background reaching schools, since the system for distribution of studies were recommended to understand the prior- textbooks had lapsed during the civil war. Through ity areas and inform the design of pilot initiatives to this pilot, provincial education officials, develop- demonstrate the use of rapid results for problem- ment partners and citizens collaborated to address solving. WBI facilitated the use of rapid results and constraints that blocked education service delivery South-South exchange in the first cabinet retreats. and quality in the provinces, and they built a solution WBI facilitated the process of periodic govern- for efficient delivery of textbooks to schools. This ment retreats by engaging senior leaders and outcome chain was one of the first demonstrations of experts from other African countries to provide peer results from the L4R program. perspectives, providing space for collaboration to analyze and learn from the pilot initiatives, providing Attention to problem technical assistance on using results-focused initia- The process began with the Director General of the tives to design plans that would translate recom- Ministry of Education at the central government mendations from the retreats into initiatives and level meeting with the Provincial Director of Educa- constructing teams for executing each part. WBI tion in Bubanza province to discuss the problem also supported aligning priorities from the retreat of delivering textbooks to villages in the province. held in March 2011 to the World Bank Strategy for [a] This initiated a series of steps to address the Africa, and to the Doing Business Indicators and problem. The Director General negotiated reduced good governance program during the retreat held in transportation rates with transport companies and December 2011. re-allocated funds to make it more financially man- Overall, this first stream of outcomes demon- ageable to transport books from the capital city to strated the creation of an authorizing environment the provincial towns given the limited budget. [b] at the highest level across government. Presiden- The Provincial Director of Education tried to mobilize tial and ministerial leadership developed a blend volunteers in the province to assist in moving the of results-oriented management tools for the L4R textbooks to more remote areas. However, due to program from multi-agency setting of priority areas limited resources and authority the Provincial Direc- for pilot initiatives. Government retreats and review tor was unable to mobilize sufficient support. [c] meetings to demonstrate and provide feedback on Consequently, the Provincial Director met with the the value of results were instrumental in building a Governor of the province, who oversaw all sectors of culture of results. service delivery in the province, to request assistance 14 Figure 3. Pilot I: Delivering textbooks to villages in Bubanza Province, November–May 2006 (f) Governor called a (g) Governor met with Communal Administrators to (a) Director General of MoE townhall to determine who support initiative to transport textbooks met with Provincial Director of could contribute what to the Education to discuss problem of effort of delivering textbooks delivering textbooks to province (h) Communal Administrators agreed to mobilize volunteers to deliver textbooks to villages by motor vehicle, bicycle and other means (b) Director negotiated (e) Governor took responsibility reduced rates to make it more to mobilize volunteers from manageable within limited development actors, local (i) Communal administrators met with schools budget of MoE to transport government staff and citizens to and village chiefs who agreed to support textbooks from capital city to assist in chain of delivery from initiative and mobilized volunteers from parent provincial towns provincial town to village level teacher associations to deliver textbooks (j) Parent teacher association decided to use existing national community service program in each community (c) Provincial Director of (d) Provincial Director met to deliver textbooks by foot, carrying them on their Education tried to mobilize with governor of province to heads, in wheelbarrows or by basket on a bike volunteers to move textbooks request assistance to raise from towns to more remote urgency of initiative among areas in the province, but lacked other sectors and stakeholders (k) MoE network of volunteers delivered 25,000 textbooks resources and authority to do so in province of Bubanza within 60 days instead of one year Institutional changes Learning/capacity changes Outcomes related to societal, policy and organizational Other outcomes related to awareness, knowledge or changes. skills, collaborative action, or the use of knowledge or Leadership and participatory priority setting to innovative solutions. strengthen public sector reforms Other outcomes Policy changes to combat inefficiencies and corruption Outcomes that changed direction or thinking Effectiveness, adaptability and responsiveness of public behind the initiative service delivery to draw attention to the problem among stakehold- a province) and informed them of the initiative ers. [d] One lesson learned here was that creating to transport the textbooks. [g] In this regard, the the authorizing environment to address the problem Governor used his authority to draw actors to work required cross-sector support - reaching outside the together to ensure an effective supply chain for education sector alone. the delivery of textbooks. His actions opened the possibility for a more sustained and cost-effective Collaborative problem-solving multi-stakeholder process, and ensured political will The Governor of the province took responsibility and for action. mobilized development actors, local government staff and citizens within his province to assist in the Textbooks delivered to villages chain of delivery from the provincial to the village The administrators agreed to mobilize volunteers level. [e] The Governor called a townhall meeting at the commune level to deliver textbooks from the that mobilized key actors for delivering textbooks communes to the villages by motor vehicle, bicycle from the provincial capital city to the communes, and other means. [h] This opened the possibility including from provincial directors in other sectors for a more sustained, community-based effort. The (agriculture, public works, transport, education), administrators met with directors of schools and and development partners (development projects, village chiefs, who agreed to support the initiative NGOs, other) operating in the area. [f] and mobilized volunteers in the Parent Teacher The Governor also met with the Communal Association (PTA) to contribute to delivery. [i] The Administrators (communes are sub-regions within PTA decided to use the existing national community 15 Figure 4. Pilot II: Accelerating payment of teachers and reducing corruption, Sept 2007–March 2008 (l) Director of Civil (q) Vice minister Service decided to of Education Institutional changes address problem of and Director of Outcomes related to societal, policy and delays in payments Civil Service met (s) Civil Service reviewed organizational changes. of newly recruited with 6 provinces (r) MoE, civil teachers and agreed on Service and initiative; time Leadership and participatory new process to provinces for teachers priority setting to strengthen expedite files followed new to receive 1st public sector reforms (m) Civil Service expedited paychecks Policy changes to combat launched pilot process reduced from in 6 remote 1 year to 3 inefficiencies and corruption provinces to (p) 2nd vice months Effectiveness, adaptability and reduce time president of responsiveness of public service to pay newly Burundi decided delivery (t) 717 new recruited to bring together teachers Learning/capacity changes teachers both ministers to received (u) Civil discuss resolution Other outcomes related to awareness, paychecks Service and for blocks knowledge or skills, collaborative within 3 MOE agreed months and action, or the use of knowledge or to ongoing (n) Civil service were not innovative solutions. (o) Director shared improved realized progress subject to lesson on need information was hindered by corrupt fees Other outcomes for collaboration sharing repeated poor to release with MoE in review between Outcomes that changed communication payment meeting chaired by ministries direction or thinking behind the with MoE 2nd vice president initiative service program each Saturday morning in each Education and Ministry of Civil Service collaborated community to deliver the books by foot, carrying on a pilot initiative to address these constraints. The the textbooks on their heads, in wheelbarrows or by outcome chain from this initiative shows how reform basket on a bike. [j] The network of volunteers deliv- was achieved and scaled-up. ered 25,000 textbooks in the province within 60 days instead of the usual one-year timeframe. [k] The Attention to problem student-to-textbook ratio increased from 1:11 to 1:3 In September 2007, the Director General of the Civil within 60 days; these textbooks had been awaiting Service decided to address severe delays in the delivery from the warehouse for 18 months. payment of new teachers by holding bureaucrats accountable to new rules, to encourage transpar- In sum, this collaboration established a new type ency and expediency in processing files, such as of supply chain to reach village schools. The prov- streamlining the documents required. [l] Despite the ince learned how to address a systemic constraint difficulty of such a reform in the post-conflict con- and develop their own system for distribution that text, the Director General accepted both personal would be a potentially long-term mechanism for and institutional responsibility for problem-solving ensuring timely delivery of textbooks. around the change process. The Director Gen- eral and her team launched a pilot initiative in six Pilot II: Accelerating the payment of teachers remote provinces to reduce the time for payment. and reducing corruption (Figure 4) [m] Given that teachers constitute more than half of Inefficiencies, long delays and corruption existed the employees within the civil service, the initiative around paying newly recruited teachers, for exam- had potential not only to advance universal primary ple, bribes by new teachers to officials for informa- education goals but also goals around improving the tion on the processing of files or to move forward speed of payment to newly recruited civil servants any delays in communicating their files to the (beyond teachers) more broadly. Ministry of Civil Service. Under the L4R, between By November 2007, the Director General real- September 2007 and March 2008, the Ministry of ized progress was stunted. Poor communication 16 between the Ministry of Civil Service and Ministry of Figure 5. Pilot III: Improving connectivity to Education had repeatedly hindered progress. She the electricity grid, June–September 2011 committed herself to resolving the obstacles. [n] The problem was symptomatic of much broader dif- (v) Managing director of state electricity company committed to improve supply of electricity and ficulties in collaboration and communication within increase its provision at a fair and affordable price, the government—and had caused hindrances in and mobilized teams to address the goal many other processes. In February 2008, the Direc- tor General shared her experiences and obstacles, (x) Director supported (w) Director and the teams to launch expressing the need for stronger collaboration with teams identified series of targeted the Ministry of Education in a review meeting for strategies to move initiatives to the initiative that was attended by the second Vice forward experiment with new processes President of Burundi. [o] Collaborative problem-solving (y) REGIDESO introduced a new, potentially faster and simplified process to connect To help address this, the second Vice President customers to national electricity grid decided to bring together both ministers to dis- cuss the resolution for these blocks, focusing on (z) REGIDESO reduced delay in connection time the problem of processing files for new teacher to wait time of two months, improving a Doing payment. [p] This outcome underscores the impor- Business Indicator for Burundi tance of high-level oversight by authorities. In this instance, having high-level leaders involved in (zz) Client satisfaction rose to 65%, progress review of initiatives enabled the civil service company income increased to bring attention to issues that had become nega- tive norms to discuss how to resolve them collab- oratively. The challenge was a prolonged back and Institutional changes forth between the two ministries in processing files Leadership and participatory priority setting to for new recruitments and payments, which it was strengthen public sector reforms possible to shorten. The meeting broke down the Learning/capacity changes problem and identified potential solutions. Other outcomes related to awareness, knowledge or skills, collaborative action, or the use of knowledge or Efficient payment and reduced corruption innovative solutions. The Vice Minister of Education met with the Provin- cial Directors of Education in the provinces where the initiative was launched. (These Directors are provinces to receive their first paycheck from around responsible for the recruitment of new teachers and one year to just three months. [s] Within three initiation of their files to recruit them.) They agreed months, 717 new teachers received their paychecks on how the transmission of recruitment files for without being subject to corrupt fees to release the teachers could be expedited from the provinces to payment. [t] The Civil Service and Ministry of Educa- the central level. [q] That same month the Ministry tion agreed to institutionalize the improved informa- of Education, Civil Service and the provinces began tion sharing between ministries. [u] to follow the newly agreed upon expedited process. [r] All recognized the legitimacy of the new process In sum, these outcomes influenced the motiva- because everyone decided on it, and the involve- tion of newly recruited teachers and delivery of ment of leaders at different levels provided the education services. Since they received payment authority and ownership around the process for it to faster, they no longer needed to follow-up on pro- be enforced at all levels. cessing of their files (reducing absences from their In March 2008, the director of the Civil Service schools). These outcomes also facilitated long-term and her team conducted a final review of the initia- improvements in collaboration across the ministries tive and found they had been successful in reducing and provided an example of a governance reform to the time taken for newly recruited teachers in six combat corruption. 17 Example of how the government cyclically scales-up the water supply rate; keeping client accounts up- the lessons learned from the pilots: In March 2008, to-date; reducing losses; and rehabilitating required the Civil Service and Ministry of Education scaled up equipment. [w] beyond the initial six provinces and applied the les- sons to the recruitment and payment of new teach- Collaborative problem-solving ers throughout the country. [9] The previous initiative The managing director supported the teams to provided the learning and authorizing environment launch a series of targeted pilot initiatives between and acted as a key lever for scale-up of closer collab- June and December 2009 to experiment with new oration among the Ministry of Civil Service, Ministry processes by addressing particular constraints. For of Education and Provinces. example, reducing the inspection period of client In October 2008, the Ministry of Civil Service accounts from every three months to every two launched an initiative to reduce fictitious staff within months: reducing the number of unpaid accounts the service, which resulted in the suspension of pay- that had not yet been disconnected and reduc- ment of 728 salaries to “ghost” individuals. Monthly ing delays in billing time of new customers from 6 savings are roughly equivalent to US $530,759. months to 2 months. [x] It was important to improve On an annual basis, this was projected to total an services to build the image of REGIDESO in the eyes amount of funds equivalent to the annual budget of of their clients (it was very low). the Ministry of Commerce—or half that of the Min- Through these initiatives, REGIDESO introduced a istry of Civil Service. [10] In a country where govern- new, potentially faster and simplified process to con- ment revenues are strained, making this significant nect customers to the national electricity grid. This amount of money available for other activities is was a streamlined approach whereby clients make valuable. just one payment at the time of connection (rather than an application fee and later a connection fee), Pilot III: Improving connectivity to the reducing administrative requirements and processing electricity grid (Figure 5) time. [y] Consequently, REGIDESO reduced the delay In Burundi there was a typical three-month wait in connection time to a more reasonable wait time time to connect new households and businesses of two months. [z] Before the initiative was launched, to the national electricity grid. This was a key area the number of new clients connected per month was to improve a Doing Business indicator for Burundi 100; after completion of the initiative this number to increase access to electricity supply and was had risen to 150 new clients per month. The manag- therefore a hindrance to the government’s desire ing director of REGIDESO provided a supportive to improve its Doing Business rankings to attract environment for experimenting with new procedures increasing foreign investment. The outcome chain for streamlining, which had been an ongoing process that improved efficiency in connecting new custom- since 2008. ers to the national grid exemplifies an agency-level change management process: Increased customer satisfaction and company income Attention to problem Results also supported an IFC government program Following the second government retreat in 2009, on economic governance whose first phase involved the managing director of the state electricity com- improving Burundi’s ranking in Doing Business and pany in Burundi, REGIDESO (within the Ministry of resulted in Burundi appearing as one of the 10 most Energy, Water and Mining) committed to raising reformed African countries in the Doing Business the quality of life in urban areas by improving the Report of 2012 for improved business climate (elevat- supply of electricity and increasing its provision at a ing in ranking from 177 in 2011 to 169 in 2012). fair and affordable price, and mobilized teams within the ministry to address the goal. [v] The director In sum, REGIDESO learned how to address and teams analyzed the constraints and identified certain constraints and implemented a streamlined key strategies to move forward to advance the goal. process to improve their efficiency in connecting new These included: increasing the production of elec- households and businesses to the national electricity tricity; increasing the number of clients; increasing grid. 18 substantiation of outcomes To verify the accuracy of the outcomes mapped and enrich WBI’s understanding of them, an external consultant selected four outcomes [1, 12, 14 and 20) and asked 10 people independent of WBI but knowledgeable about the change to review each and record whether they agree with the outcome as described. Four people responded to three outcomes [except 1]. Four of these substantiators fully agreed on the description of the outcomes and WBI’s contribution to them and two provided additional information to clarify their significance. Excerpts of the substantiators’ comments on the outcomes achieved: “The approach is very good because after the training provided by the World Bank, it was time for the Government of Burundi to take ownership of this method to integrate it into the implementation of projects and programs. Otherwise, we would always be expecting World Bank experts to launch, on our behalf, projects to be executed in a short timeframe with tangible results. For the time being, most ministries have already integrated this method into the implementation of the action plans.” —Rosette Nizigiyimana, Vice-Minister of Civil Service “The World Bank should consider giving further support to this program through a sustained program of ongoing evaluation. It should consider providing external coaches to help us better evaluate our performances in the rapid results, as often as necessary. It should extend the approach to other institutions, even private or parastatal ones.” —Odette Kayitesi, Minister of Agriculture “Indeed, over the past few years, there has been an obvious change in the way projects in Burundi are managed and implemented using the system of results-oriented activity planning followed by the evaluation of results, using observable and verifiable indicators.” —Jéremie Ntiranyibagira, Director General of Human Resources to the General Staff of the Armed Forces Thus, as demonstrated through these three initia- support the pilots in the health and education sec- tives, ministries and agencies used the L4R program tors. Training was provided on conducting initiatives to build an authorizing environment (Outcome to the ministries and agencies and their staff, and Area 1) to execute their ideas for change to further WBI financially supported and mentored the interna- the country’s development goals. Through rapid, tional coaches and national coaches who were fun- results-focused initiatives and rapid cycle learning damental instruments in the problem-solving process (Outcome Area 2), they engaged collaboratively empowered through these initiatives. with teams to experientially learn what solutions WBI supported the government through provid- worked and did not work to address each constraint. ing technical assistance launching and conducting They used the program to structure pilot solutions mid-point and final reviews of the initiatives. For to improve the effectiveness, responsiveness and example, in the case of the initiative on the payment efficiency of public services, reduce corruption and of teachers by the civil service, the review helped to enhance government relationships centrally and distill lessons learned from this pilot and the learn- sub-nationally. The successes offered learning to ing was then used to feed into the design of similar scale-up and build upon across government. follow-up initiatives across the country and ministries Contribution of WBI to Outcome Area 2: To sup- to improve results in the civil service more broadly. port the implementation of these initiatives, WBI On a broader scale, WBI provided ongoing support introduced rapid, results-focused initiatives with and guidance to the government and Steering Com- rapid cycle learning to Burundian officials for the mittee to implement initiatives cyclically over the L4R program to jumpstart and accelerate action for multiple years of the program to build successively delivering results towards achieving national devel- on the results and learning of pilots. opment agenda goals. WBI acted as initiators, con- veners and as a resource for knowledge exchange to 19 Outcome Area 3: Embedding results- contexts. [16] This further expanded the utility of oriented problem-solving in the public the L4R program to reach areas where French was sector not widely used. Institutionalization of the results-oriented manage- In 2012, ENA engaged stakeholders in con- ment tools of the L4R program has been a progres- sultations on options for institutionalizing the sive part of the change strategy within the design of L4R program tools and practices within govern- this program. ment ministries, as well as options for its financial In June 2007, the second Vice President, as sustainability within government. [17] This process head of the Steering Committee, had committed to provided key actors an opportunity to design a build knowledge among public servants around the strategy to shape the integration and full owner- results-oriented management tools of the L4R pro- ship of the L4R program. ENA used the input to gram to tackle constraints across the public sector. develop a strategic visioning paper incorporating As noted previously, he initiated a process to train the views of stakeholders. [18] The strategic vision- coaches across the government. Since 2007, public ing paper set a formal strategy for scale-up of the servants with coaching capacities to use the results- program, owned by all ministries and endorsed by oriented management tools have been embedded donors. across all major ministries and agencies nationally. In 2013, ENA began negotiations to secure gov- On a routine basis, the ministries started to engage ernment budget to strengthen the coordination of teams to use rapid cycle learning to learn how to the L4R program. [19] This would be another step achieve priority results. [7] to continue the program within the government Then, following the second government retreat in beyond WBI’s support. March 2009, the Vice Presidents of Burundi agreed Overall, by 2013, ENA in partnership with WBI to sign a public decree to mandate expansion of the had intended to build a network of sub-national program across government. [12] The decree, signed teams experienced in the use of rapid cycle learn- in July 2009, was instrumental to further institutional- ing to apply project management and results- ize the results-oriented project management process oriented management skills and capacities. In of problem-solving by doing. Ministries launched addition, those trained previously by WBI are now initiatives using rapid cycle learning cycles iteratively, internal coaches in teams within ministries at the that is, each trimester, to address constraints and national government level. [20] These individuals record and exchange progress to document and are sustaining and institutionalizing the culture of learn from. results progressively developed since 2006. While more than 2,000 team members have been Contribution of WBI to Outcome Area 3: WBI introduced to rapid, results-focused initiatives, a key contributed by providing results coaching, work- challenge to further institutionalization has been shops and training to support the initial skills build- extending the results-oriented project management ing in ministries. National results coaches were capacities across government. Institutionalization financed and guided who played a critical role requires sub-national government use of the tools in the process of preparation and signing of the as well as building them into national ministries and decree through guidance to the Steering Commit- building capacity in the government to train coaches tee. on their own. Until 2011, coaches were trained by WBI partnered with ENA to deliver coach- WBI and WBI national coaches. ing, workshops and training, including training Since 2011, ENA began to train teams and launch coaches to be housed within ENA to guide future initiatives to extend results-oriented project man- trainings. ENA was provided with training materi- agement skills to improve performance. [14] This als and expertise to integrate them into use by helped to anchor the curriculum and knowledge ENA trainers. WBI facilitated jointly with ENA the and skills required for the L4R program into the knowledge-sharing platform for decision-making key national organization responsible for training on the strategy, and recruited a consultant to civil servants. In 2012, ENA developed sub-national conduct interviews with donors and stakeholders training materials in the local language to support who had been involved in the L4R program. The implementation of the program in decentralized WBI-financed consultant helped to prepare the 20 draft strategic visioning paper based on interviews gradual progress to unblock constraints and further and discussions during the workshop. advance development goals and priorities. Ministries across the Burundian civil service Conclusion continue to use L4R tools without WBI support. Over the course of several years, the L4R program They continue without oversight from the Steering developed a culture of results among leadership Committee and World Bank Country Office, and in and teams in the civil service as well as the capacity some cases, without requests from ministers to use to apply project- and results-oriented management the tools to address a specific problem. WBI does tools. The civil service can use the tools to problem- not currently know how many initiatives are running solve and learn which practical solutions work in in the government at any given time but has been what context to advance institutional change. The informed by ministries of agriculture, health and program helped to develop leaders and teams, skills defense, in particular, that the tools are continuing to and new behaviors in the civil service to systemati- be used. cally find and adopt innovative solutions for identify- Key outcomes supporting this institutionalization ing and overcoming institutional constraints. of results-oriented practices within the civil service The problem-solving process addressed con- include: the Presidential decree to strengthen coun- straints such as breakdown in communications, weak try leadership, embedded coaches within national collaboration, corruption vulnerabilities, organiza- ministries and agencies and inclusion of results train- tional dysfunctions and inefficiencies. It engaged, ing within ENA’s curriculum in the local language to motivated and empowered leaders and teams to build a larger cohort of coaches across the country, take new actions to change problems and broke particularly in provinces. The outcomes relating to down barriers from those that have a vested interest ENA demonstrate ongoing capacity development in keeping the current inefficient system. within the civil service such that the program can be The emphasis on team-building is key to the scaled-up to reach civil servants at all levels across success of the program. Working as a team with the country. The government has also undertaken a strong authorizing environment from leadership research on ways to further sustain the program and empowers a shared responsibility for results as begun negotiations on how to finance the program well as personal and joint accountability. It enables fully from the government budget. the government to develop and use their collec- tive capacity to advance reforms, rather than over Next Steps burden individuals in already low capacity environ- To continue to advance the program’s objectives ments. It helps to overcome hierarchical inefficien- the training of additional civil servants in the tools cies within government that can impede change, of the L4R program must still be addressed. Some since authorization for the reform sits in another ministries are more immersed in applying the tools office. Teams are able to find innovative and effec- and are consequently operating at a faster pace and tive solutions to problems in the country because encounter challenges in implementation when they of an enabling context for action that is overseen collaborate with a ministry that is not as involved in by the highest level of leadership in the presidency. applying the tools. Change is possible and solutions can be used to tai- There is also an insufficient pool of high-quality lor lessons for scale-up within other parts of govern- results coaches. The existing pool is comprised of ment or throughout the country. coaches who continue to support results-focused, The government retreat process helped align rapid cycle learning teams in some ministries, and program outcomes from initiatives in ministries with ENA coaches who do not have enough financial development priorities for the country’s growth, resources to expand the L4R program tools to governance and social and economic transforma- the provinces. It is thus essential to continue to tion. Problems are systemically prioritized at the strengthen ENA’s effectiveness and financial viabil- highest level with the Steering Committee, and then ity to provide regular training and re-training to refined by senior leader champions in ministries and coaches, as well as to anchor Burundian results multi-actor teams. Through pilot initiatives, there is a coaches to the growing regional network of results coaches in Africa. Ongoing strengthening of coaches 21 or is doing, that reflects a significant change in their behavior, inside ministries should also be reinforced in a con- relationships, activities, actions, policies or practice. The program tinuous way through implementing results and rapid may influence these changes, directly or indirectly, partially or cycle learning. wholly, intended or not. Outcomes are identified at two levels In addition, the government retreat process in relation to the goal: institutional changes relate to societal, should continue and be used as an opportunity to policy and organizational changes; and learning/capacity changes relate to awareness, knowledge or skills, collaborative action, or recognize and reward results champions. This will the use of knowledge or innovative solutions. These levels are create a healthy competition, maintain the authoriz- based on the Capacity Development and Results Framework. ing environment and capitalize on and learn from The framework provides a systematic yet flexible approach effective practices. to designing capacity development strategies and programs, monitoring and adaptively managing interventions, and Burundi is reviewing the Presidential decree for evaluating and learning from their results. the L4R program that is still active despite govern- 2 Outcome harvesting is a practical assessment tool from the ment turnover since 2009, and considering how to outcome mapping community of practice. It can be used for strengthen the supervision and M&E of the govern- real-time monitoring and evidence gathering from complex ment action planning under a structure chaired by development processes that involve multiple stakeholders. It the presidency. is based on a similar concept of locally driven change from the Overall the L4R program is now owned and led by Capacity Development and Results Framework. The tool was customized to gather information on outcomes—and related Burundi. Ongoing support from WBI is expected to outputs and milestones—to learn from what changed, for whom, include advisory support for the final stages of coach when and where, the significance of the change and how the development by ENA, the continued institutionaliza- program contributed to each change. tion of the program’s processes within the govern- 3 The numbers in brackets correspond to the outcomes in Figures ment and integration of Burundi coaches within the 2–5. The text that usually follows each outcome refers to its Africa regional results coach network to help con- significance. The process of change the outcomes represent is seen in Figure 1. tinue to build their knowledge and expertise. n 4 Change agents are leaders, groups or organizations from government or non-state that drive change. 5 Change strategy refers to how change happened to advance NOTES progress toward the development objectives—the development 1 Mapping outcomes—and related outputs and milestones—can problems addressed, types of outcomes achieved, WBI help us learn from change processes that occur during program contributions, and
partners involved. A change strategy may delivery that often seem complex and opaque because they include different types of
change processes or outcome areas involve multiple actors and address large development problems. depending on the complexity of the multi-actor institutional An outcome is what each social actor (or change agent) did, changes involved in a program. for more information Acknowledgments Project Contact Thanks to Benjamina Randrianarivelo and Kay Benjamina Randrianarivelo, WBI Leadership team, Winning, who led the mapping exercise. brandrianarivelo@worldbank.org Thanks to substantiators: Email Odette Kayitesi, Minister of Agriculture WBI Capacity Development and Results Andre Nduwimana, Director of ENA team at capacity4change@worldbank.org Rosette Nizigiyimana, Vice-Minister of Civil Service Website Jéremie Ntiranyibagira, Director General of Human www.worldbank.org/capacity Resources to the General Staff of the Armed Forces © Copyright 2013 World Bank WBI’s Capacity Development and Results team led the outcome mapping; Jenny Gold coordinated the exercise with support from Ricardo Wilson-Grau. Sharon Fisher provided editorial and design services. Samuel Otoo provided overall guidance. 22 Implementing Reform Initiatives in Solid Waste Management in Bosnia I n Bijeljina, a city in northeastern Bosnia and Her- Development Objective zegovina, the solid waste management system Improve the value of municipal services for citizens in was not financially stable. Consequently, in the South East Europe. fall of 2011, six representatives from the munici- Problem pality of Bijeljina, its transport utility and dumpsite The inabilities of municipalities to create and manage utility, embarked on a yearlong program to advance financially viable solid waste management systems are their solid waste management reforms. Their suc- due to a variety of political economy, institutional and cess in forming a coalition that could withstand the adaptive challenges. This tension is heightened in a fragile context. In the city of Bijeljina in northeastern demands of implementation allowed them to deal Bosnia and Herzegovina, waste collection was no longer with tough issues, such as citizens not paying fees, a sustainable service. Challenges included citizens not lacking the experience to implement solutions and paying fees, dispersed households, weak operations, adjusting their work processes in order to deliver poor communication and weak financial management. results. Specific Objectives During the period of January–March 2013, the To strengthen a cadre of reform-minded local, regional World Bank Institute (WBI) mapped the outcomes1 of and national-level government officials and members of the Bijeljina experience using a customized outcome civil society to improve solid waste coverage through col- laborative leadership strategies that mobilize stakehold- mapping tool2. This case is a result of that mapping ers to make progress on complex adaptive problems. and examines the results of the Bijeljina reform team 23 Figure 1. Map of outcomes showing how changes connected and built over a two years (2) Reform team (15) Member of Bijeljina (17)* Municipality understood the deeper police force joined drafts new policy problems that blocked collection team to help to guide utility expanded waste collection, deal with citizens who do operations related to citizens paying not pay their fees for services, collection from (1)* Reform dispersed households and team, with income from collection members from (18) Collection (14) Team decided to utility and dump the two utilities remain operational, hired and municipality, provide input into two staff to expand efforts collaborated to policy (3) Reform team analyze their goal agreed to create to expand solid database of waste (10) Collection utility (12) Municipality (13) Households waste collection generators and used survey findings to agreed to in pilot area paid and increase increase the amount calculate realistic price negotiated the new tariff collection fees in of waste collected by for citizens to pay for increase of 10% Bijelijna the end of one year waste collection (16) Dumpsite utility had (4) In Bijelijna, reform (9) Team leader re-organized a deficit of revenues and team continued to field worker schedules by received assistance from meet monthly to discuss removing other priority activities municipality to cover part of issues such as pricing, so they could complete survey it; collection utility met its budget and deficits with wider household coverage deficit from other activities (6)* New (5) Reform team implementation team formed plan (7)* Implementation (8) Collection (11) Citizens in struggled to set to pilot increased team conducted utility the pilot area up the database waste collection citizen survey to find collaborated became involved and increase in one village out under which with another in collection waste collection and one urban conditions they utility to through survey and and decided to area; reform team would pay higher gain starter expressed demand form team in agreed to provide tariffs and identified database, with for better service collection utility oversight of new their household municipality to find solutions teams location approving use 2011 2012 Institutional changes Learning/capacity changes Outcomes related to societal, policy and organizational changes Other outcomes related to awareness, Political commitment, social norms and citizen demand knowledge or skills, collaborative action, or the for service improvements use of knowledge or innovative solutions. Policy improvement for utilities * Outcomes selected for substantiation; see Operational efficiency/responsiveness/financial sidebar. viability of utility efforts, which occurred under WBI’s support to South processes for change to address the Bijeljina reform East European (SEE) countries working on solid team’s objectives and goal. waste management reforms. Through the outcome mapping process WBI A visual map (Figure 1) presents the sequence of identified and formulated the outcomes, presenting outcomes achieved by the change agents—munici- an explanation of their significance and how WBI pal leaders, reform team members and an imple- had contributed—directly or indirectly, in a small mentation team involved in the process. The map or big way, intentionally or not—by empowering illustrates how outcomes connected and built on the change agents to take new actions. Then, each other over time to form multi-actor, institutional roughly 20% of the outcomes were independently 24 substantiated for credibility in the mapping exercise coming the challenges they faced during their (see page 6 sidebar). 11-month laboratory phase. Background OUTCOME AREAS The inability of municipalities to create and manage Figure 1 shows the outcomes of the Bijeljina reform financially viable solid waste management systems team’s efforts to reach their 11-month goal to create is due to a variety of political economy, institutional a database on waste generators in the municipal- and adaptive challenges. Often citizens and the ity of Bijeljina by July 31, 2012 and to increase the state need to adapt how they engage with each amount of waste collected by 20% in Bijeljina by Dec other. This tension is heightened in a fragile country 31, 2012. The process of change they pursued can context where dysfunctional public service, resettled be seen in the four areas of outcomes (Figure 2) that households and citizen behaviors are exasperated are detailed in the following sections. by years of conflict. In the fall of 2011, WBI’s Leadership Practice Outcome Area 1: Municipal Commitment to launched a yearlong Greater than Leadership Expand Collection Coverage (GTL) Program to help participating teams improve In the municipality of Bijeljina, six senior officials municipal services. The program began with an from the municipality, collection utility and dump application process where teams submitted their utility formed an informal coalition or reform team reform proposals. This was followed by a five-day to make progress on improving solid waste manage- workshop with six municipal-level reform teams from ment services, in particular coverage of the services Albania, Bosnia and Herzegovina, Kosovo, Macedo- in the town and surrounding rural area. In December nia, Montenegro and Serbia. During the workshop 2011, during the GTL workshop application pro- the teams were exposed to adaptive leadership, cess, the reform team collaborated to analyze the strategic communication, political economy and problem of having insufficient funds to sustain the self-mastery concepts. They were also facilitated municipalities’ solid waste collection and dumpsite in conducting network/influence analysis on their services. The team set a reform goal to expand solid reforms as well as identifying their own goals, work waste collection and increase service fees collected plans and potential Rapid Results Initiatives (RRI). An from households over 11 months. [1]3 11-month laboratory phase followed, during which This new alignment triggered a collaborative pro- time the teams innovated around how to implement cess among the reform team members. Eventually it their newly articulated goals. enabled the team to work together to start to break The objective of the GTL program is to down the difficult medium-term reform of improving strengthen a cadre of reform-minded local, regional the sustainability of municipal services into workable and national-level government officials and mem- pieces that could be more reasonably addressed bers of civil society to improve solid waste coverage within one year. through collaborative leadership strategies that In January 2012, at the GTL workshop, the reform mobilize stakeholders to make progress on com- team further refined their goal and the constraints plex adaptive problems. Beyond the concepts and to reach it. They understood that expanding solid tools that are shared, teams benefit from knowledge waste collection and increasing service fees was not exchange amongst themselves as well as best prac- a simple technical exercise and particular institu- tices and advice from technical experts. tional constraints needed to be addressed. [2] These The GTL Program built on the World Bank- constraints included the weak financial sustainability Austria Urban Partnership Program on Strengthen- of the utilities’ services, operational efficiencies in ing Capacity Building of Local Governments in SEE delivering the services, poor citizen engagement countries. Through this program, WBI’s Leadership and inefficient regulations. For example: practice was able to partner with the World Bank • The collection and dump utilities were running operational team in the region and WBI’s Urban their services on a deficit. To become profitable team to offer the GTL Program. This current case the collection utility needed to collect the “cor- focuses on the experience of one of these reform rect” fees from all households so they could then teams as they coalesced and worked toward over- 25 Figure 2. Change strategy showing how change happened to advance progress toward goal WBI Contributions • WBI Greater Than Change Agents4 Leadership workshop, • Reform team from collection utility, dump and which offered advice from municipality technical experts, sessions on adaptive leadership, • Municipality including on strategic • Citizens in one village and urban areas communications, political • Police economy, stakeholder • Implementation team from different departments influence mapping and of collection utility self- mastery and Rapid Results Initiative sessions • Follow-up Rapid Results Initiative coaching support Change Strategy5 Outcome Area 1: Municipal commitment to expand collection coverage Problems Partially Partners • Improved collaboration with municipality and dump Addressed • WBI’s Urban practice • New understanding of problem • Lack of understanding of • Europe and Central Asia deeper problems affecting • Decided to develop management database and Region’s Sustainable inability to sustain waste new tariffs in one village and urban area Development Department collection service of the World Bank Outcome Area 2: New support and demand • Habit of citizens not paying • Joint Vienna Institute from citizens tariffs and unrealistic prices • Citizens started to pay tariff in pilot area • Weak operations and • Through the survey, citizens expressed demand financial management with for better services tariff collection and usage • Municipality agreed to negotiated increase of • Poor communication 10% between citizens and municipality over tariff Outcome Area 3: Strengthened effectiveness price, payment compliance of waste management in one village and and quality of service urban area • Weak policy guidance to • Identified more viable pricing, collection process regulate utility, pricing, and database for collection utility structures and other issues • Team worked to find implementable solution to create starter database, set price and find households Outcome Area 4: Scale-up and policy guidance • Enacted new municipal policy to regulate utilities Development Objective • Formalized implementation team to expand • Improve the value of effort to other village and urban areas municipal services for • Police force joined team to help deal with citizens in South East citizens who do not pay their fees Europe 26 pay a fair price for dumping waste at the because they had to do the work of designing dumpsite. and verifying a database as well as increasing the • Both utilities lacked accurate data on the location coverage area for collection of waste in Bijeljina. and number of households they collected waste Together they developed a results-based goal from. This inaccuracy was due to disrupted land- and detailed work plan to increase coverage while use post conflict, with disorganized and undocu- setting up and verifying a database in one rural area mented construction of homes. Households no and one heavily congested urban street in Bijeljina. longer aligned to pre-war boundaries, making it The reform team agreed to provide oversight to difficult to collect waste from households with no the effort. [6] Meanwhile, the implementation team responsible payer, no road access and multiple tested out solutions they could later scale-up across families combined together in close land areas. the municipality. • Citizens no longer had the habit of paying for WBI invited teams in the GTL workshop to sub- waste collection services since the war and did mit applications for Rapid Results coaching support not leverage channels to demand improvements during the laboratory phase when teams implement in the services. their reform goal in their respective municipalities. • The municipality needed to strengthen its policy The RRI support was targeted at teams that needed framework for governing public utility services, to find new solutions to implement their goal. By including the provision of a guide for information- request of the Bijeljina Reform Team, WBI provided sharing among utilities and the setting of fee a Rapid Results coach to support the implementa- rates and developing processes to enforce com- tion team over the approximately 65 days they had pliance with rules and regulation. to unblock the implementation obstacles facing the reform team’s achievement of their 11-month goal. The reform team agreed to address these root WBI encouraged the reform team to oversee the problems by focusing on creating a database of RRI process so that lessons from the RRI could be household waste generators in the municipality as institutionalized with full support of the municipality, well as increasing the amount of waste collected collection and dumpsite utilities. by the end of the year by 20%. [3] Previously there was no consensus among the municipality, collec- Outcome Area 2: New Support and tion utility and dump utility around the problem or Demand from Citizens how to focus reform efforts. Starting in January 2012, Between April and June 2012, the Bijeljina imple- the reform team met monthly to discuss problems mentation team carried out a citizen survey in the such as pricing and budget deficits and possible pilot area. They planned to use the field-based sur- solutions to implement their goal. [4] The team had vey to inform the location of households to develop developed a common sense of urgency to continue the database and find out under which conditions working together to implement the goal. citizens would pay higher fees for waste collection WBI provided technical support to applicants services. Close to half of the citizens surveyed were before the GTL workshop in January 2012 to help willing to pay more for waste collection services but them re-think the nature of their problem, goal at small increments. A little over 25% thought the and stakeholders. During the GTL workshop, WBI current price was too high and almost half found provided technical expertise and learning on adap- the cost to be realistic. But, only half of the citizens tive leadership, personal mastery, coalition-building surveyed were happy with current service provision diagnostics, strategic communication and Rapid provided by the collection utility. [7] Results Approach. Through the survey, citizens became involved in In February 2012, the reform team recognized scrutinizing the utility’s services and expressed their they were struggling to create their database and demand for better services. [11] This was impor- increase waste collection. To find solutions to this tant to change citizens’ behaviors in terms of social obstacle they decided to set up a second team norms around paying for services and to engage within the collection utility who would become a citizens in voicing their demands for services RRI “implementation team.” [5] Six staff members improvements. The increasing citizen demands also from different job areas were assigned to the team provided a political incentive for elected officials in 27 substantiation of outcomes To verify the accuracy of the outcomes mapped and enrich WBI’s understanding of them, the external consultant selected 4 outcomes [1, 6, 7, 17] and asked 7 people independent of WBI but knowledgeable about the change to review each and record whether they agree with the outcome as described. Five people responded: 5 fully agreed with the outcome description as formulated and WBI’s contribution to it; regarding significance, 1 fully agreed with the description and 4 provided additional information to describe the significance. Excerpts of the substantiators’ comments on the outcomes achieved: “The reform team from Bosnia and Herzegovina comprised 6 members: 1 member from the municipality of Bijeljina (today the city of Bijeljina), 3 members from the public enterprise regional landfill EKO-DEP, 1 member from the utility Komunolac and 1 member from the directorate for construction and development of the city of Bijeljina.” —Dragan Lazic, Eko Dep, Bijeljina “The agreement, the plan and key steps have been achieved [toward the goal of increasing the amount of waste collection]. However, the coverage has not yet been increased by 20% under the specified deadline.” —Milorad Zekic, Head, Komunolac utility “Today, the main goal—to cover the whole municipality with waste collection—is almost done. The 11-month goal to increase coverage for 20% failed because the goal was too complicated for so small a time period. But, the Rapid Results Initiative [implementation and achievement of certain outcomes in a shorter time period] made good preparation for the long-term goal. WBI had a huge impact on the creation of the reform team and made our work much easier and more organized.” —Bojan Miric, Komunalac utility staff, Rapid Results Initiative team leader the municipality to support efforts to increase collec- database and increase waste collection in one rural tions and solid waste coverage. and one urban area of Bijeljina. [9] This outcome In July 2012, the Bijeljina municipality agreed shows how the authorizing environment in the col- to a negotiated collection price increase of 10% for lection utility changed to allow the implementation waste collection services, which fell within citizens’ team to efficiently operate and adapt processes to willingness to pay. [12] In the second half of 2012, address the reform goal, which was previously a low households in the pilot area of the Bijeljina munici- priority. pality paid the new tariff. [13] The municipality’s WBI provided advisory support to the team agreement to the price increase showed new trust in leader during the RRI and at the mid-point review. the collection utility’s methodology. In May 2012, the collection utility collaborated WBI contributed RRI coaching support to help with another utility to obtain a starter household the implementation team tackle the issue of engag- database, with the Bijeljina municipality approving ing citizens and the municipality. The team did the terms of use. [8] The collaboration was unprec- the work on the ground, having been empowered edented and helped the implementation team solve through the Rapid Results process. the problem of having no household database to base its collection activities. It also increased support Outcome Area 3: Strengthened from the municipality, since the municipality had to Effectiveness of Waste Management in One approve the use of the database. The citizen survey Village and Urban Area could then help to validate and build on information From April to June 2012, the leader of the imple- in the database. mentation team in the Bijeljina collection utility In June 2012, the Bijeljina municipality and re-organized the field workers’ schedules so they collection utility used data from the citizen survey to could give priority attention to the reform goal. This calculate a price for waste collection that reflected cleared their workload so that they could complete the costs incurred by the utility. This new price surveying citizens and the work plan to create the included the higher cost for dumpsite fees. [10] A 28 realistic price is a critical step toward a financially demonstrated the new collaboration between the viable solid waste management system. Price municipality and utilities. identification is tricky because the price must cover WBI contributed RRI coaching to reinforce the the costs of collecting and dumping the waste. For idea that representatives from other agencies or the price not to be inflated, both the collection groups outside the municipality and utility could join utility and dump utility must work efficiently. the implementation team. Previously, the utilities did not have a transparent method to calculate pricing or a procedure to Conclusion communicate the pricing process to the community. The outcomes achieved in Bijeljina describe the In December 2012, the Bijeljina municipality experience of one reform team that joined the GTL agreed to cover 20% of the dumpsite utility’s deficit program. Over one year, the Bijeljina reform team of 43% of total revenues, thus reducing the deficit became an informal coalition that raised the urgency to 35%. The collection utility covered their deficit and political commitment to improve solid waste by using income from other activities, such as street management services in the municipality. They cleaning. [16] This demonstrated a step to move to improved communication and relations among two a more sustainable business model and improve utilities and the municipality, providing an authoriz- financial viability of the utilities. It also showed the ing environment to carry out difficult reform activi- new political will of the municipality to subsidize the ties. dump utility’s operations. The Bijeljina reform and implementation teams WBI facilitated the GTL workshop and RRI coach- uniquely adapted solutions to address institutional ing to empower a process for the municipality and changes that previously blocked improvements in utilities to address the challenge of pricing on their solid waste management services and produced own. WBI helped the municipality understand the outcomes they could build on. This illustrates a revenue challenge related to solid waste manage- municipal-led process by which local teams identi- ment. fied how to advance their own reforms. For example, they leveraged a simple citizen Outcome Area 4: Scale-up and Policy survey to address social norms around paying Guidance fees for services and the lack of a way to channel In 2012, the Bijeljina implementation team citizen demand for service improvements. They decided to remain operational to continue to addressed operational inefficiencies in the utilities scale-up coverage beyond the pilot area. [14] By by re-organizing staff to implement reform activities, keeping the implementation team operational, the identifying household locations so service fees could problem-solving process and reform solutions to be collected and creating a database for collection improve solid waste management are becoming activities. Financial viability was addressed by devel- institutionalized. The implementation team oping a transparent process to calculate service fees expanded to include a member of the Bijeljina and secure subsidy support from the municipality to police force to help the utility manage compliance account for the remaining deficit. The municipality challenges of households paying their fees. [15] addressed policy inefficiencies, drawing on lessons This engagement shows that the collection utility from the pilot to inform new regulations to guide has a clear mandate and strong support from the utilities’ services and solid waste management. municipality to scale-up waste management service reforms beyond the pilot. Next Steps In January 2013, the Bijeljina municipality drafted The new regulations put in place by the municipality a new policy to guide utility operations, to which the should help to ensure continued improvements in collection utility and dump utility provided input. solid waste management services in the municipality. [17, 18] These outcomes provided guidance that Implementation and monitoring of the regulation by previously was absent and which was necessary to the municipality will be important, along with contin- regulate utilities and scale-up reforms to make sure ued logistical support to utilities to carry them out. they provide the required value to citizens. They also The continuity of the reform and implementation teams beyond the one-year pilot period points to an 29 ongoing effort to scale-up the outcomes achieved in 2 Outcome harvesting is a practical assessment tool from the the pilot throughout all town and rural areas of the outcome mapping community of practice. It can be used for real-time monitoring and evidence gathering from complex municipality. development processes that involve multiple stakeholders. It As the reform team members look to the future is based on a similar concept of locally driven change from the their concerns will move to the next stages of Capacity Development and Results Framework. The tool was improving solid waste management services, such customized to gather information on outcomes—and related outputs and milestones—to learn from what changed, for whom, as recycling and monitoring the current solid waste when and where, the significance of the change and how the management system. While the existing solid program contributed to each change. waste management system established in Bijeljina 3 The numbers in brackets correspond to the outcomes in is expected to improve over time, it is realistic to Figure 1. The text that usually follows each outcome refers to its expect new liabilities for improvement of the entire significance. The process of change the outcomes represent can system to appear. n be seen in Figure 2. 4 Change agents are leaders, groups or organizations from government or non-state that drive change. NOTES 5 Change strategy refers to how change happened to advance 1 Mapping outcomes—and related outputs and milestones— progress toward the development objectives—the development can help us learn from change processes that occur during problems addressed, types of outcomes achieved, WBI program delivery that often seem complex and opaque because contributions, and
partners involved. A change strategy may they involve multiple actors and address large development include different types of
change processes or outcome areas problems. An outcome is what each social actor (or change depending on the complexity of the multi-actor institutional agent) did, or is doing, that reflects a significant change in their changes involved in a progra behavior, relationships, activities, actions, policies or practice. The program may influence these changes, directly or indirectly, partially or wholly, intended or not. Outcomes are identified at two levels in relation to the goal: institutional changes relate to societal, policy and organizational changes; and learning/capacity changes relate to awareness, knowledge or skills, collaborative action, or the use of knowledge or innovative solutions. These levels are based on the Capacity Development and Results Framework. The framework provides a systematic yet flexible approach to designing capacity development strategies and programs, monitoring and adaptively managing interventions, and evaluating and learning from their results. for more information Acknowledgments Project Contacts Thanks to Hirut M’cleod, Manuel Contreras and Manuel E. Contreras, WBI Leadership team, mcontreras@ Jose Edgardo Campos, WBI Leadership team worldbank.org members involved in the project and mapping exercise. Email WBI Capacity Development and Results team, capacity4change@ Thanks to: worldbank.org Dragan Lazic, Eko Dep, Bijeljina Dragisa Marjanovic, Head, Eko Dep Website Bojan Miric, Komunolac staff, Rapid Results www.worldbank.org/capacity Initiative team leader © Copyright 2013 World Bank Mihailo Nestorovic, Directorate for Urban Development Bijeljina WBI’s Capacity Development and Results team led the outcome Bjorn Phillip, Senior Urban Specialist, World mapping exercise; Jenny Gold coordinated with support from Bank Ricardo Wilson-Grau. Sharon Fisher provided editorial and design Velimir Sljepcevic, Board Member Eko Dep services. Samuel Otoo provided overall guidance. Milorad Zekic, Head, Komunolac Utility 30 Improving Governance in Pharmaceutical Procurement and Supply Chain Management in Kenya, Tanzania and Uganda S ince 2010, Kenya, Tanzania and Uganda have Development Objective made significant progress in strengthening Improve citizen access to essential medicines in multi-stakeholder engagement to facilitate Kenya, Tanzania and Uganda. greater transparency, accountability and Problem efficiency in Pharmaceutical Procurement and Supply Challenges in pharmaceutical procurement Chain Management (PSM). The increased collabora- and supply chain management—such as poor tion between state and non-state actors, which has coordination between varied actors, inefficiency and emerged as a result of this engagement, is integral to misallocation of public resources—result in waste improving access to essential medicines, the goal of and limit citizens’ access to essential medicines at affordable prices and of good quality. WBI’s Improving Governance in Pharmaceutical Pro- curement and Supply Chain Management Initiative. Specific Objectives In January–March 2013, WBI mapped the Strengthen transparency, accountability and efficiency in government pharmaceutical procurement and outcomes1 of this initiative using a customized supply chain management to promote value for outcome harvesting tool2. This visual map (Figure money and achieve more with less. 1) presents the sequence of outcomes achieved 31 Figure 1. Map of outcomes showing how changes connected and built over a four-year timeframe (22)* (10) Kenya, Tanzania (21) Tanzania Tanzania stakeholders (23) Tanzania coalition and Uganda coalitions coalition and built skills of CSOs and signed a identified national government district procurement memorandum conveners to coordinate piloted a officers in using formalizing action plan implementation tool for the tool to monitor creation (3) Uganda key monitoring compliance with of country stakeholders district-level Procurement Act coalition partnered with procurement government of medicines to hold first (9) Kenya, Tanzania (15) Kenya (24) Regional and Uganda (13) Kenya, public forum to (14)* Kenya coalition, technical coalitions validated Tanzania discuss Public coalition built working with working group action plans with and Uganda Procurement knowledge of the Kenya refined and broader stakeholder policymakers (1) and Disposal of health facility Medical standardized groups at country endorsed Government, Public Assets workers and Supplies data collection level for local coalitions’ civil society Act and ways CSOs on PSM Agency, tools developed ownership strategies and and private to amend it practices, designed and pre-tested committed actors in to enhance processes and and test- in Kenya and to support Kenya, efficiency third party piloted Uganda, and country-level Tanzania (11) Kenya coalition monitoring Mobile Drug developed a engagement and Uganda piloted a monitoring (CSOs only) Tracking framework to roll understood tool in three health System out the tools added value of facilities to collect (17) Uganda coalition collaborating (4) Uganda data on citizen CSO members organized for sustainable government satisfaction with health first national dialogue (25) Kenya PSM reforms used forum service delivery and with medicine procure- coalition feedback access to medicines ment agencies scaled up a to examine (20) Uganda pilot of citizen options to (7) Kenya, Tanzania (16) Uganda coalition built skills monitoring amend the act and Uganda coalitions coalition of CSOs to use social tool initially developed action plans to integrated accountability tools administered address priority PSM areas PSM in its and monitor PSM in 2012 (2) Kenya, three-year Tanzania (5) Uganda’s work plan and Uganda (18)* Uganda existing multi- co-funded by committed (8) Kenya, CSOs and medi- (19) Uganda stakeholder DFID to multi- (6)* Kenya Tanzania cine agencies coalition’s Uganda coalition stakeholder and Tanzania and Uganda (12)* Kenya, agreed to ongo- National Health agreed to process to formed similar coalitions Tanzania and ing dialogue to Consumers broaden prioritize action coalitions began using an Uganda coalitions communicate on Organization, a its scope areas to inform to promote online regional acquired new skills PSM governance CSO network, to address policy reform, transparency, Community on how to address challenges, ongo- published governance and set norms accountability of Practice political economy ing reform efforts preliminary challenges in for mutual and (www.enepp. challenges to and emerging baseline study on PSM accountability collaborative net) to build implementation areas for collabor- PSM challenges in problem knowledge on of country action ative action selected districts solving in PSM PSM plans 2010 2011 2012 2013 Institutional changes Learning/capacity changes Outcomes related to societal, policy and organizational changes Other outcomes related to awareness, knowledge or Regional and country commitment and priority setting/ skills, collaborative action, or the use of knowledge or transparency and accountability of PSM innovative solutions. Efficiency of PSM policy * Outcomes selected for substantiation; see sidebar. Effectiveness of multi-stakeholder action 32 by change agents—the leaders, coalitions and information asymmetries and poor multi-stakeholder organizations involved in the process. The map coordination and collective problem solving.3 illustrates how the outcomes connected and built on The initiative seeks to create and build the capac- each other over time to form multi-actor, institutional ity of multi-stakeholder coalitions comprising public processes for change to address the initiative’s and private sectors and CSOs (including academia, objectives and goal. media and faith-based organizations) in Tanzania, WBI team members identified and formulated Kenya and Uganda. Through capacity develop- the outcomes, presenting an explanation of their ment, WBI provides the coalitions with cutting-edge significance and how WBI had contributed—directly tools to build strong relationships across stake- or indirectly, in a small or big way, intentionally or holder groups, understand and address the political not—by catalyzing or empowering the change economy of health sector reforms, enhance techni- agents to take new actions. Then, roughly 20% of cal understanding of pharmaceutical PSM issues the outcomes were independently substantiated for and engage demand-side actors in generating credibility in the mapping exercise. evidence-based data to inform policymaking. These capacity development components are intended Background to strengthen collaborative action toward reforms, According to the European Health Care Fraud and which is expected to accelerate PSM change pro- Corruption Network and the World Health Organi- cesses and ultimately improve access to medicines. zation, annual global health expenditure stands at Tanzania, Kenya, and Uganda have initiated about US $5.3 trillion. Of this outlay, US $750 bil- country-level processes that have the potential to lion (18%) is spent in the pharmaceutical market, reform pharmaceutical PSM processes. They possess while consumers lose about US $300 billion more to both technical and leadership capacity enhanced human error and corruption. Together, total expen- through structured learning, knowledge exchange diture for pharmaceuticals and the cost of corruption and peer-to-peer learning that facilitate regional combined exceeds US $1 trillion, or approximately multi-stakeholder-led efforts to improve governance 1/5th of what is spent globally on health care. in PSM. Pharmaceutical procurement is particularly prone to poor governance, since it entails complex OUTCOME AREAS processes that involve many stakeholders, includ- The process of change from this initiative can be ing government ministries, procurement agencies, seen in three areas of outcomes that represent the manufacturers, hospitals, distributors and citizens major change paths (Figure 2). All of the outcomes as the ultimate clients. When pharmaceutical pro- were analyzed and classified according to the types curement and supply chain systems work effectively, of change they achieved. They were then grouped they offer high levels of quality, cost-effectiveness, based on how they connected to each other to form product availability, transparency, accountability and a story for change. value for money in the use of public funds. The effort to improve these systems is especially Outcome Area 1: Regional commitment to critical in emerging markets, where pharmaceutical improve access to medicines spending is 20–30% higher than the global average. In this initiative, regional and country-level commit- International reference prices and cross-country ment helped drive effective and sustained actions. knowledge sharing are thus critical to low-income In June 2010, pharmaceutical procurement countries obtaining fair prices on the global pharma- agencies, public procurement oversight authorities, ceutical market. ministries of health, civil society actors and private In 2010, WBI’s Health Systems and Open Gover- companies in Kenya, Tanzania and Uganda nance practices jointly launched the Improving Gov- acknowledged governance weaknesses in PSM and ernance in Pharmaceutical Procurement and Supply came together regionally to discuss approaches Chain Management Initiative in Kenya, Tanzania, and for achieving more sustainable reforms. These Uganda. The initiative focuses on addressing weak actors recognized the limitations of working governance in PSM, including legal and regulatory through the customary approach of “silos,” with issues, organizational inefficiencies, challenges of little collaboration across stakeholder groups. 33 Figure 2. Change strategy showing how change happened to advance progress toward goal WBI Contributions • Strategy and technical Change Agents5 consultations • Kenya, Tanzania and Uganda multi-stakeholder • Benchmark studies and coalitions analysis of procurement • Pharmaceutical procurement agencies agencies • Public procurement oversight authorities • Facilitation of peer- to-peer learning on • Ministries of health implementation of country • Civil society actors action plans • Private companies • Survey on implementation • Online community of practice challenges and capacity development on addressing PE of reform • Online commnity of practice Change Strategy6 • Funding to support implementation of country Outcome Area 1: Regional commitment to action plans improve access to medicines • Commitment to multi-stakeholder process to Problems Addressed leverage strengths of different actors to address • Varied actors with weak governance in PSM different agendas • Collaborated to share solutions, and identify surround pharmaceutical ways to scale-up achievements and standardize procurement and supply Partners tools regionally chain problems • Transparency International, • Used regional network to build knowledge • Inefficiencies and waste Medicines Transparency through peer exchanges in the procurement Alliance and St. John’s and supply of essential University of Tanzania Outcome Area 2: Effective multi-stakeholder medicines acted as country action conveners for coalitions • Minimal demand-side data • Country coalitions developed action plans, to complement supply-side validated by stakeholders, endorsed by data and provide a holistic policymakers picture of PSM challenges • Coalition members agreed on priority areas for • Lack of transparency collective action and accountability in pharmaceutical Outcome Area 3: Improved transparency, procurement supply chain accountability and legitimacy in countries management • Improved information on medicine availability through greater transparency in PSM • Improved social accountability through use of monitoring tools for citizen feedback on medicine availability • Improved PSM policy through public feedback Development Objective forums • Improve citizen access • Built data collection experience by piloting tools to essential medicines • Built knowledge on PSM processes, practices in Kenya, Tanzania and and 3rd party monitoring Uganda • Collaborated with government on actions • Increased knowledge on PSM problems and on how to address them collectively Stakeholders realized the value presented through More importantly, stakeholders viewed work- synergistic approaches to problem solving, and, ing together as a critical strategy toward making therefore, committed to pursuing a more systematic, inroads at a time when health policy reform was a collaborative approach to influence reforms. [1,2]4 national priority in all three countries. The collabora- tive address of systemic challenges in PSM promised 34 to leverage stakeholder strengths and expertise; service delivery and access to medicines at the facil- create a level playing field for constructive dialogue ity level to better inform policy dialogue. between multiple stakeholders; facilitate consen- WBI administered a survey to members of the sus building about reform priorities; and establish multi-stakeholder coalitions in Kenya, Tanzania and mutual accountability for results. Uganda in late 2011 to understand the implemen- WBI held extensive consultations at the country tation challenges the coalitions faced and identify level to elicit stakeholder feedback on a proposed areas where WBI could provide capacity develop- concept note for a multi-stakeholder approach for ment and technical assistance. WBI’s Greater than strengthening good governance in pharmaceutical Leadership Program designed a five-day workshop PSM. Through facilitated discussions, WBI helped for the coalitions on “Strengthening Multi-stake- stakeholders understand the link between the holder Coalitions Through Leadership Action.” WBI slow progress on PSM and working in silos and also provided funding to accelerate implementation demonstrated the potential for state and non-state of the refined action plans. actors to work collaboratively for stronger and more In October 2012, a regional team of experts effective reforms. representing government and civil society from In April 2011, the country stakeholders began Kenya, Tanzania and Uganda refined, standardized using a regional online Community of Practice (CoP) and finalized data collection tools developed and to build knowledge and commitment of stakehold- pre-tested by the coalitions. The technical work- ers on PSM (see www.enepp.net). [8] The CoP had ing group of experts also developed a monitoring more than 350 members from all three countries by and evaluation framework to guide the refinement, February 2013. Having a CoP became important for finalization and rollout of the tools. [24] They identi- advancing regional knowledge exchange between fied the need for two supplementary data collection practitioners in all three countries, and providing a tools: a Citizen Empowerment Tool to determine “safe” space to share challenges, innovative solu- the existence of and assess the efficacy of Grievance tions and resources to help move forward sensitive Redress Mechanisms in receiving and responding reforms. to citizen feedback on health service delivery, as WBI designed, developed and launched the well as a Stock Monitoring Tool to track stock levels online platform during a regional workshop in Kenya, of tracer medicines in selected health facilities in held April 2011. WBI facilitated membership of all three countries. The set of harmonized tools will participants at the workshop and other stakeholders aid cross-country comparison and analyses to gain at the country level from the public, private and civil a broader picture of citizen satisfaction with health society sectors and development partners. services, access to medicines and citizen empower- In April 2012, representatives of multi-stakeholder ment across the region. coalitions in each country gathered regionally to dis- WBI facilitated the technical working group meet- cuss their respective challenges, explore approaches ing to refine, standardize and finalize the data col- for addressing the political economy of reforms, lection tools, as well as develop supplementary tools reprioritize their action plans and set realistic time- and outline an appropriate M&E framework. lines. [12] This refinement of country action plans So, over 30 months multi-stakeholder processes was necessary since the multi-stakeholder coalitions had leveraged the strengths of different actors to were experiencing implementation challenges and address weak governance in PSM. These commit- needed to review their priorities to identify areas ments materialized through: improved regional where they could have the most impact. A key con- recognition of the value of state and non-state actors cern across the coalitions was how to address per- engaging collaboratively on PSM; enhanced regional ceived mistrust between actors, as well as promote networking to build knowledge, shared solutions a more equitable balance of power among stake- and identified ways to scale-up achievements; and holders to ease coordination, forge collaboration development and review of new and innovative data and facilitate attainment of shared objectives. The collection tools to generate demand-side evidence country coalitions also prioritized the role of demand to complement national data on health service deliv- side actors in generating an evidence base on health ery and access to medicines. 35 substantiation of outcomes To verify the accuracy of the outcomes mapped and enrich WBI’s understanding of them, the external consultant selected 5 outcomes [6, 12, 14, 18, 22] and asked 15 people independent of WBI but knowledgeable about the change to review each and record whether they agree with the outcome as described. Thirteen people responded and all “fully agreed” with the description as formulated of the outcome and its significance. Excerpts of the substantiators’ comments on the outcomes achieved: “The commitment of the high-level policy makers was important to the work of the PSM coalition as it will enable their work get the recognition and support it needs in high-level decisionmaking that affects the PSM. The bringing together of the PSM Coalitions and high-level policy makers needed a champion and WBI played that role well and in a timely manner.” —Ramadhan Mlinga, Chief Executive Officer, Public Procurement Regulatory Authority, Tanzania “While such multi-stakeholder coalitions are important, they need to be better aligned with overall World Bank engagement in the country level and contribute to better policy dialogue on improving participation of stakeholders and enhanced transparency. Therefore, I would like to see much stronger emphasis on linkage with the Bank’s long-term engagement in the outcome. This way, WBI contributions will provide more sustainable gains.” — Gandham N.V. Ramana, Lead Health Specialist, World Bank “To improve lives of the citizens can only be successful with support of the government (high-level policy makers), failure to which interventions increasingly achieve minimal results. Working with government senior officials has enabled the civil society to fill in gaps within the policy system, a key gap being monitoring of impact of the government expended resources. Then direct feedback to high-level policy makers. The Kenya Medical Supply Agency has been a key beneficiary of this type of CSO monitoring of their services.” — Debra Gichio, Program Officer, Transparency International, Kenya “This meeting [National Medicines Dialogue in May 2012] brought a number of agencies and CSOs together. Most especially the district CSOs were able to meet the executive directors of the National Drug Authority, Joint Medical Stores, National Medical Stores and Ministry Of Health. In fact one of the participants said ‘now this is a dream come true because I have always wanted to see National Medical Stores.’ “ —Robinah Kaitiritimba, Executive Director, Uganda National Health Consumers Organisation Outcome Area 2: Effective multi-stakeholder At the same time, various actors from Kenya action and Tanzania formed similar multi-stakeholder Multi-stakeholder coalitions became important to coalitions to promote greater transparency, address PSM in the country context. In 2011, an accountability and collaborative problem solving existing multi-stakeholder group in Uganda—Medi- in PSM. [6] In Kenya, the Forum for Transparency cines Transparency Alliance (MeTA)—agreed to and Accountability in Pharmaceutical Procure- broaden its scope to become the country coalition ment (FoTAPP) was established in May 2011. The to address governance challenges in PSM. [5] Rather group comprises public sector agencies, includ- than establish a new multi-stakeholder coalition, ing the Ministry of Health, Kenya Medical Sup- it was important to leverage existing capacity by plies Agency, Public Procurement and Oversight joining forces with MeTA, which has been in Uganda Authority, Kenya Anti-corruption Commission and since 2007 and has established networks with both the Pharmacy and Poisons Board; civil society; state and non-state actors. Consequently, MeTA donor partners; the private sector; and academia. integrated PSM in its three-year plan co-funded by In Tanzania, 22 organizations, including the the United Kingdom Department for International Public Procurement Regulatory Authority, Ministry Development (DFID). [16] This secured funding for of Health and Social Services, Food and Drugs the broadened mandate of MeTA to address Authority, and Medical Stores Department, as PSM and gave it more credibility to do so. It was a well as 13 CSOs, signed a Memorandum of quick win for the coalition by building on its existing Understanding to formally launch the coalition. networks. [21] The coalition has since expanded its members. 36 Formalization of the coalition was important in • Linked coalition activities to relevant country the country context to create a legitimate entity reforms recognized by government, private sector and civil • Helped establish supporting relationships society as the vehicle for promoting transparency, between policymakers and the coalitions neces- accountability and efficiency in PSM. Each country sary for strengthening partnership with the gov- coalition developed action plans to, for the first ernment time, tackle country-specific PSM challenges through • Helped lay the groundwork for joint demand- and a multi-stakeholder approach. [7] The key was supply-side data collection and other activities building a shared understanding of priorities and that the coalitions prioritized in their strategies responsibilities among the different stakeholders. WBI convened a high-level policy dialogue In view of resource constraints and competing among the country policymakers in Uganda, Kenya priorities, the coalitions tried to identify areas where and Tanzania in April 2012 after recognizing the dif- they could achieve quick and high-impact outcomes. ficulties that the country coalitions faced in gaining WBI facilitated and convened a regional work- traction on their activities and priorities. WBI facili- shop in Kenya in April 2011 and attended by coali- tated knowledge exchange between the country tion representatives. At the event, WBI provided conveners in Uganda, Kenya and Tanzania, and tools and resources to guide stakeholders in estab- guided the sharing of experiences on how to formal- lishing country coalitions, developing country action ize the multi-stakeholder coalitions in-country. plans and identifying common areas of interest In sum, building on regional commitment, multi- where all three countries could engage and share stakeholder country coalitions mobilized in Uganda, their experiences though the regional CoP. WBI also Kenya and Tanzania to take action to achieve provided input on Memoranda of Understanding. improved practices in PSM. Beyond establishing the core membership of the country coalitions, each country group held broader Outcome Area 3: Improved transparency, national consultations to seek broad-based buy- accountability and efficiency in for the country action plans developed and to Since 2010, the coalitions have contributed to ensure local ownership for priority areas of collab- improving open dialogue around governance vul- orative engagement. [9] With this endorsement at nerabilities in pharmaceutical PSM, and in working the national level, the coalitions identified country together to pilot innovative solutions to ensuring conveners responsible for coordinating multi-stake- greater transparency, accountability and efficiency in holder activities. [10] This effort established a focal PSM. Key country-specific achievements include: point or secretariat for the coalitions’ day-to-day functioning, including organizing meetings and fol- Uganda Coalition—Medicines Transparency lowing up on decisions. Alliance (MeTA) WBI provided guidance to the country consulta- In July 2010, MeTA, in partnership with the Public tion processes led by the coalitions, and contacted Procurement and Disposal of Public Assets Author- World Bank staff in the country to facilitate relation- ity (PPDPA), held the first public forum to discuss ships with key government stakeholders. the 2003 Public Procurement and Disposal of Public By April 2012, high-level policymakers from Assets Act and opportunities to amend it to enhance Kenya, Tanzania and Uganda had endorsed the efficiency in procurement of essential medicines, country action plans and committed themselves to among other things. [3] This action tested the multi- supporting activities of the country coalitions [13]. stakeholder approach in engaging non-state actors Up until this moment, the government representa- in providing input into ongoing legislative reform. tives in the coalitions had limited authority to com- This led the PPDPA to review the act that governed mit to specific interventions, which often relied on its activities, examining options for amendments. [4] the buy-in and political willingness of higher-level The success demonstrated the power of multi-stake- officials, such as heads of agencies, within the public holder collaboration and signalled a shift in the way sector. This formal endorsement from policy makers: the PPDPA traditionally engaged, which previously • Demonstrated political support for the coalition’s involved minimal engagement with non-state actors. PSM strategies 37 In May 2012, MeTA—under the leadership of provide feedback on their level of satisfaction with Uganda National Health Consumers Organisation health services. The success of the exercise also (UNHCO)—organized the first national dialogue on indicated the potential for demand-side data collec- medicines. [17] This meeting brought together key tion to generate evidence to inform policy dialogue. agencies, particularly the National Medical Stores, The coalition also built knowledge of health facility the National Drug Authority, the Drug Monitoring workers and CSOs on PSM practices and CSO moni- Unit, and the Joint Medical Stores of the Ministry of toring. [14] This training built the understanding of Health and pharmaceutical councils. The CSOs and both supply- and demand-side actors on their roles medicines agencies agreed to an ongoing dialogue to improve outcomes in access to medicines. Such to openly collaborate and communicate on PSM understanding is required both for the effective use governance challenges. [18] This dialogue helped of monitoring tools and for building consensus on establish trust between the agencies and coalition reform possibilities. and created legitimacy for the coalition to address Additionally, FoTAPP, working closely with the PSM challenges. Kenya Medical Supplies Agency (KEMSA), designed In June 2012, UNHCO also published a prelimi- and test-piloted a Mobile Drug Tracking System nary baseline study—based on research from four (MDTS). The MDTS provides citizens, community districts in Uganda—that helped identify key gaps health workers, health facilities and health manage- in PSM and highlighted interventions where the ment committees with real-time information on coalition could leverage its comparative advantages. medicine availability in selected health facilities. For [19] The study helped to provide up-to-date informa- patients with specific diseases—tuberculosis, HIV/ tion on PSM challenges and further grounded the AIDS, diabetes—this system is especially useful to proposed interventions and priorities of the coalition track the availability of medicines in health facilities, within the country context. making it more efficient to obtain life-saving drugs Also in June 2012, MeTA trained CSOs on social and reduce transaction costs. [15] It allows the track- accountability tools and their role in monitoring PSM ing of medical commodities from KEMSA ware- at the health facility level. [20] This training started to houses to health facilities, making it easier for the build the capacity of CSOs to monitor PSM, as well demand-side to monitor delivery of essential medi- as to raise awareness about effective and transparent cines. The development of this tool also represents a PSM processes. practical example of collaboration with government WBI contributed funding for the coalition to on the delivery of demand-side tools. organize and implement national dialogue activities In February 2013, FoTAPP completed a pilot of a and helped create a platform for open discussion more extensive data collection exercise in 20 health between the medicine agencies. At the request of facilities in nine counties across the country. This PPDPA, WBI provided examples of similar acts in scaled up the pilot of the Citizen Monitoring Tool Africa to help them think through amendments for initially administered in 2012. [25] The roll out of the Uganda. WBI leveraged resources from UNHCO to data collection exercise will provide a baseline to engage technical experts to develop the baseline help measure the impact of the coalitions’ interven- study to inform the coalition’s priorities. WBI orga- tions over the next five years and determine efficacy nized a Regional Training of Trainers’ Workshop on of the multi-stakeholder approach in facilitating PSM PSM attended by Uganda coalition members. reforms and improving access to medicines. WBI provided the Kenya coalition funding to sup- Kenya Coalition—Forum for Transparency and port the development of the Citizen Monitoring Tool Accountability in Pharmaceutical Procurement for data collection, the capacity building workshop, (FoTAPP) and in collaboration with KEMSA, to engage an ICT In June 2012, FoTAPP developed and test-piloted consultant to design software for the MDTS. WBI a Citizen Monitoring Tool in three health facilities in provided technical support to the team in devel- Nairobi County to collect data on citizens’ level of oping a proposal for funding through the Social satisfaction with health services and their access to Development Civil Society Fund, which selected the medicines—in terms of physical availability as well coalition as a recipient of US $100,000 to support as affordability. [11] The tool enabled citizens to scale up of the Citizen Monitoring Tool pilot. 38 Tanzania Coalition milestones in improving partnership, specifically In June 2012, the Tanzania coalition, in partnership around the design and implementation of both with the Public Procurement Regulatory Authority, demand- and supply-side tools to monitor PSM. developed a procurement monitoring tool to exam- In Uganda, the successful launch and public ine the processes used to procure pharmaceuticals dialogue around findings of a preliminary baseline at the district level and to determine their compli- study on PSM challenges—organized by MeTA— ance with the Public Procurement Act. [22] They, created a platform for continued engagement along with the Muhimbili University of Health and with National Medical Stores, Joint Medical Stores Allied Sciences and St. John’s University of Tanzania, and the Ministry of Health. All three partners are also trained CSOs, district procurement officers from now collaborating with the coalition to design and Dodoma region and representatives of the Medical pilot four data collection tools in 10 districts across Stores Department and test-piloted the tool in six Uganda. districts in the Dodoma region. [23] In Kenya, the coalition partnered for the first time Given the substantial resources allocated to with KEMSA to pilot an innovative MDTS, which pharmaceutical procurement at the district level, allows citizens and health workers to access real- the coalition prioritized procurement monitoring time information on medicine availability in selected to ensure resources were being used efficiently health facilities. and in compliance with Public Procurement Act. In Tanzania, the coalition—in collaboration with Such monitoring would help advocate value for the Public Procurement and Regulatory Authority— money in PSM and contribute to improved access to designed and piloted a procurement monitoring medicines. The coalition is working closely with the tool for use by district officers to assess the level of regulatory authority to prioritize reform areas based compliance with the Public Procurement Act. on recommendations from the final procurement Another area of progress has been capacity monitoring report. development of coalitions, particularly of civil WBI reviewed the draft procurement monitor- society, to better understand, monitor and advance ing tool and provided substantive comments for advocacy around PSM reforms, with an emphasis on enhancement. greater transparency, accountability and efficiency. In sum, the multi-stakeholder country coalitions in Many of the CSOs trained through this initiate Uganda, Kenya and Tanzania are increasingly taking are leading data collection exercises on health actions to improve transparency, accountability service delivery in their respective communities. The and efficiency in PSM through inclusive dialogue to initiative has also published a Training of Trainers influence policy; generation of baseline data that Manual as a guide for civil society actors interested help prioritize reform areas for collaborative action; in implementing social accountability mechanisms capacity development for key actors, especially to improve service delivery, with a focus on access to CSOs; and innovative tools to monitor PSM at the medicines. facility level. Also important is the involvement of both supply- and demand-side actors to strengthen Next Steps their respective roles in the country context to Because of the change processes, the coalitions improve access to medicines. are empowered to advance outcomes of their own. There is local ownership of the process, and key conclusion relationships—especially with government—have Improving transparency, accountability and efficiency been formed that should provide a foundation and in PSM was pursued through collaboration between impetus for advancing outcomes. government and civil society actors, regionally and Nonetheless, a key challenge that remains is to through country coalitions. Traditionally, the two ensure the full participation of the private for-profit stakeholder groups have not worked together to sector to facilitate broader stakeholder engage- address challenges in PSM; rather, they worked in ment, support longer-term sustainability as well as silos, with minimal communication and cooperation. sustain the momentum for reform. Through the development of joint country action Further, the importance of grounding coalition plans, the multi-stakeholder coalitions achieved priorities in local contexts has become clear. The 39 coalition-building experience in three countries health service delivery in general, which is an area of shows how country dynamics often influence the increased demand. ability and agility of the coalition. Kenya succeeded In addition to continuing to contribute to in moving quickly with its country action plans outcomes in the three areas, new outcomes are because it has a more favorable enabling environ- expected, particularly around implementation of ment—including a relatively mature democracy, joint interventions to address emerging issues that sophisticated technology and close relationship with will be highlighted in the data collection exercise a government client eager to integrate citizen and from all three countries. n demand-side feedback. It is important to understand the local dynamics in each country context and work NOTES within that framework to identify local champions that have the capacity to move reforms quickly and 1 Mapping outcomes—and related outputs and milestones—can bring the coalition along. help us learn from change processes that occur during program delivery that often seem complex and opaque because they Now, a key strategy for the initiative is to share involve multiple actors and address large development problems. the experience and early results of implementing the An outcome is what each social actor (or change agent) did, or is coalition-building approach to improving gover- doing, that reflects a significant change in their behavior, relation- nance of PSM in East Africa. Lessons learned will ships, activities, actions, policies or practice. The program may provide practical guidance on the “how to” of coali- influence these changes, directly or indirectly, partially or wholly, intended or not. Outcomes are identified at two levels in relation tion building in health service delivery and provide to the goal: institutional changes relate to societal, policy and recommendations on applications in other coun- organizational changes; and learning/capacity changes relate to try contexts. For example, the capacity developed awareness, knowledge or skills, collaborative action, or the use of within the coalitions can be applied to monitoring knowledge or innovative solutions. These levels are based on the for more information Acknowledgments Project Contacts Thanks to Marylou Bradley and Jilliane T Cabansag, and team Yvonne Nkrumah, Senior Operations Officer, members involved in the project and mapping exercise. WBI Health Systems practice, ynkrumah@ Thanks to: worldbank.org Dr. Gandham N.V. Ramana, Lead Health Specialist, World Bank Julia Mensah, Extended Term Consultant, Jackie Idusso, Key Account Manager, International Business Africa, WBI Health Systems practice, jmensah1@ Merck Sharpe and Dohme worldbank.org Dr. Eva Ombaka, Director Consultant Business Network International Tanzania Pwani Email Dr. Ramadhan Mlinga, Chief Executive Officer, Public Procurement WBI Capacity Development and Results Regulatory Authority, Tanzania team at capacity4change@worldbank.org Wachuka W. Ikua, Senior Operations Officer, World Bank Website Dr. John Munyu, Chief Executive Officer, Kenya Medical Supplies www.worldbank.org/capacity Agency Debra Gichio, Program Officer, Transparency International, Kenya © Copyright 2013 World Bank Robinah Kaitiritimba, Executive Director, Uganda National WBI’s Capacity Development and Results Health Consumers Organisation Moses Kamabare, General Manager/CEO of National Medical team led the outcome mapping; Jenny Stores Gold coordinated the exercise with support Apollo Muhairwe, Operations Officer, World Bank from Ricardo Wilson-Grau. Sharon Fisher Dr. Laurent Shirima, Director of Capacity. Building and. Advisory provided editorial and design services. Services, Tanzania Samuel Otoo provided overall guidance. Emmanuel Malangalila, Consultant, World Bank Joseph Mhando, Dean and Senior Lecturer, School of Pharmacy, Photo by Simone D. McCourtie, World Bank St. John’s University of Dar es Salaam 40 Capacity Development and Results Framework. The framework provides a systematic yet flexible approach to designing capacity development strategies and programs, monitoring and adaptively managing interventions, and evaluating and learning from their results. 2 Outcome harvesting is a practical assessment tool from the outcome mapping community of practice. It can be used for real- time monitoring and evidence gathering from complex develop- ment processes that involve multiple stakeholders. It is based on a similar concept of locally driven change from the Capacity Development and Results Framework. The tool was customized to gather information on outcomes—and related outputs and mile- stones—to learn from what changed, for whom, when and where, the significance of the change and how the program contributed to each change. 3 While the Improving Governance in Pharmaceutical Procure- ment and Supply Chain Management Initiative was established in 2010, over the years it has become part of the global movement on “Open Contracting,” a multi-sector effort that seeks to pro- mote greater transparency and accountability in the award and implementation of public sector contracts. 4 The numbers in brackets correspond to the outcomes in Figure 1. The text that follows each outcome refers to its significance. The process of change the outcomes represent is in Figure 2. 5 Change agents are leaders, groups or organizations from government or non-state that drive change. 6 Change strategy refers to how change happened to advance progress toward the development objectives—the development problems addressed, types of outcomes achieved, WBI contribu- tions, and
partners involved. A change strategy may include dif- ferent types of
change processes or outcome areas depending on the complexity of the multi-actor institutional changes involved in a program. 41 Strengthening Parliamentary Oversight of National Budgets in Africa S ince 2009, African Public Accounts Com- Development Objective mittees (PACs)1 and the Southern African Improve the benefits of public spending for citizens in Development Community Organization of Africa. Public Accounts Committees (SADCOPAC)2 and Eastern African Association of Public Accounts Problem Committees (EAAPAC)3 regional networks have Parliaments must ensure public money serves citizens, made strides in improving the benefits of national while limiting the corruption, fraud, misconduct budgets for citizens. By participating in these net- and inefficiencies that reduce confidence in public works, which focus on sharing practitioner experi- services. ence and developing regional good practice, the Specific Objectives PACs identified country-level reforms needed to Strengthen effectiveness of parliaments in forming, strengthen the parliaments’ capacity to engage in implementing and overseeing budgets; improve open and collaborative budget processes. WBI sup- legitimacy and efficiency of audit process; and ported the peer and action learning processes within increase transparency and accountability of budget the networks as part of its Parliament Open Budget- processes. ing Program. As a result, members of Parliament 42 and staff were able to bolster their roles in forming, capacity to scrutinize implementation of national implementing and overseeing their respective coun- budgets. try budgets. In four years, parliaments in African countries In January–March 2013, WBI mapped the out- engaged in the WBI program have made progress comes4 of this initiative using a customized outcome in enhancing how their countries’ national budgets mapping tool5. This visual map (Figure 1) presents respond to reform needs and how public officials the sequence of outcomes achieved by change are held accountable for implementing government agents—the PACs and regional networks. The map programs. The process of change for this initiative illustrates how the outcomes connected and built can be seen in four areas of outcomes (Figure 2) that on each other over time to form multi-actor, institu- are detailed in the following sections. These out- tional processes for change to address the initiative’s comes were analyzed and classified according to the objectives and goal. types of change they achieved, then grouped based WBI team members identified and formulated on how they connected to each other to affect the outcomes, presenting an explanation of their change. significance and how WBI had contributed—directly or indirectly, in a small or big way, intentionally or Outcome Areas not—by catalyzing or empowering the change agents to take new actions. Then, roughly 20% of Outcome Area 1: Improved Relations the outcomes were independently substantiated for Among Parliaments in Africa on Reforms credibility in the mapping exercise. Consensus, collaboration and learning from experi- This case highlights a few examples of the out- ences among PACs in the SADCOPAC and EAA- comes achieved by the SADCOPAC and EAAPAC PAC regional networks are an essential part of the regional network members. change process of this initiative. In 2009, members of SADCOPAC reached a con- Background sensus on a set of good practices for PACs to imple- There is a global trend toward greater openness ment in the region to enhance their performance in government finances. Transparent budgetary and secretariat operations and guide national-level practices can ensure funds raised by the state for reforms. [1]6 “The Good Practice Guide for Public public purposes will be spent as promised, while Accounts Committees” identifies key issues parlia- maximizing the benefits. One crucial component of a ments face during the audit stage of the budgetary transparent system of resource allocation is indepen- process to ensure proper oversight of spending dent assurance of the integrity of public budgeting of public money. It documents practices that have through an audit process, and the scrutiny of its proven useful elsewhere in dealing with this chal- results by representatives of the people, in the form lenge. of parliament. PACs play an increasingly important WBI co-hosted regional capacity development role in this good governance, transparency and activities within SADCOPAC. The activities provided financial stability. guidance on global PAC good practices and Public financial management systems are framed facilitate South-South knowledge exchange. WBI by the budget processes: budget formulation and also assisted SADCOPAC in developing its Good the approval of the budget by parliament (ex ante Practice Guide. phase), implementation of budget provisions by In May 2011, PAC members of the regional ministries and the audit and oversight of budget networks collaborated to use the good practices implementation by parliament (ex post phase). to reach a common understanding on PAC perfor- WBI seeks to enhance the flow of information mance criteria. This initiated a South-South learning around the formulation and oversight of the bud- process to support national-level change. [2] The get to strengthen participation, transparency and members agreed on 17 resolutions to improve the accountability of national public financial manage- operations of PACs and guide national-level reforms ment systems. This is accomplished by strengthen- in public accounts management to implement in ing regional networks of PACs to serve as platforms their respective parliaments within Africa. [3] The for sharing experiences and building their technical 43 Figure 1. Map of outcomes showing changes connected and built over a five-year timeframe (1) SADCOPAC (2) Parliaments (6) PACs in SADCOPAC REGIONAL OUTCOMES members in SADCOPAC and EAAPAC countries agreed on and EAAPAC increased their knowledge (7) PACs in the regional (25) African good practices network com- about how to use audit networks learned from parliaments improved for PACs to monly under- reports to propose reforms experiences of other countries their relations to implement to stood PAC identify reform needs enhance their performance and guide national performance criteria reforms and secretariat (3)* Member parliaments agreed operations and on 17 resolutions to implement guide national- for improving PAC operations and level reforms guiding respective reforms (8) Tanzania (9) Tanzania par- (10) Tanzania PAC (12) Tanzania government liament agreed not championed budget parliament announced implemented to rotate staff on change for review it will establish budget PAC recom- money committees of current year audit committee responsible mendations in to help retain and reports before next for budget scrutiny audit report build members’ year’s budget knowledge (28) Tanzania COUNtry changes as a result of regional outcomes (13) Tanzania president PAC proposal to dismissed 6 ministers amend budget for corruption based on year accepted audit reports (24)* South Sudan (11) South Sudan newly (23) South Sudan PAC PAC recommended elected parliament institutionalized public media setup of procure- established a PAC briefings around audit reports ment agency to protect public funds and submitted bill (15) Botswana for its creation (14) Zambia parliament required Executive to report parliament amended (4) Kenya PAC increased quarterly on implementation procedure to open (29) South Africa understanding around of PAC recommendations meetings of PAC to parliament estab- how to use audit reports public lished Parliamentary to support reform Budget Office (16) Rwanda (17)* Rwanda PAC (18) Swaziland (26) Swaziland auditor (5) Kenya parliament parliament released 1st review of parliament adopted PAC general carried out 2 approved new provision established a state finances reporting recommendation to use performance audits and to send audit reports to PAC billions lost in 2009–2010 performance audits of 2 special Investigative parliament for oversight public spending audits (19)* Uganda PAC (20) Uganda PAC (27) Swaziland PAC appointed new officer to increased number of recommendations are aid review of audit reports performance audit used by government, on public fund spending reports reviewed anticorruption commission and police for investigation and (21) Ethiopia PAC (22)* Ethiopia PAC possible prosecution instituted weekly hearings provided for all audit open to media to present reports go through audit findings for public public hearing process 2009 2010 2011 2012 2013 Institutional changes Learning/capacity changes Outcomes related to societal, policy and organizational changes. Other outcomes related to awareness, knowledge or skills, Transparency of budget and audit processes collaborative action, or the use of knowledge or innovative solutions. Legitimacy of budget process and corruption mechanisms Parliamentary effectiveness in oversight role * Outcomes selected for substantiation; see sidebar. 44 Figure 2. Change strategy showing how change happened to advance progress toward goal WBI Contributions • Knowledge exchange Change Agents7 • Co-host regional capacity • SADCOPAC network development activities, such as within SADCOPAC • EAAPAC network and EAAPAC regional • Parliamentary PACs in countries: Botswana, networks Ethiopia, Kenya, Rwanda, South Africa, South • Guidance on PAC good Sudan, Swaziland, Tanzania, Uganda, Zambia practices Change Strategy8 Partners Outcome Area 1: Improved Relations Among Parliaments in Africa on Reforms Problems Addressed • National budget and audit offices • Increased collaboration through regional • Weak agreement and networks to learn how to address agreed experience among resolutions to guide national reforms parliaments on how to improve accountability of Outcome Area 2: Improved Parliamentary public funds Effectiveness in Oversight Role • Weak budget oversight by • Improved organizational structure, staffing and parliament role and responsibility of PACs in countries • Weak public confidence in • Improved quality of audit reporting, and budget and audit processes follow-up to implement PAC recommendations • Inefficiencies and Outcome Area 3: Improved Legitimacy and corruption in national Transparency of Budget and Audit Processes budgets that are not • Improved processes for oversight of audit exposed findings and for public scrutiny • Increased knowledge in how to use audit reports to propose reforms Outcome Area 4: Improved Corruption Development Objective Mechanisms • Improved monitoring of audit reports to expose • Improve the benefits of corruption public spending for citizens in Africa reforms aim to hold the executive accountable for In sum, these regional changes increased and better resource allocations for citizens. expanded relations among African parliaments and Later in 2011, PACs in the regional networks South-South learning through regional networks to gained insights into how PACs in other countries use address 17 resolutions to guide national reforms performance audit reports to scrutinize the extent [25]. government departments are implementing the budget economically, efficiently and effectively. [7] Outcome Area 2: Improved Parliamentary WBI co-hosted the May 2011 Accountability Effectiveness in Oversight Role Conference of EAAPAC and SADCOPAC and Effective organizational arrangements are key to provided guidance on PAC good practices provide for continuity and successful implementa- and formulation of resolutions. WBI co-hosted tion of good practices to help improve budget regional capacity development activities within the oversight. networks on using performance audits to assess In April 2011, the Rwanda parliament established the effectiveness and efficiency of government a new PAC to examine financial misconduct within spending. public institutions and report misuse of public funds to the plenary to decide punitive measures. [16] No 45 parliamentary body previously had this responsibil- strengthen the office of the auditor general to carry ity in spite of evidence that public funds were stolen out performance as well as financial audits.9 [18] The each year. auditor general has since carried out two perfor- In 2012, the Tanzania parliamentary staff sup- mance audits and two special investigative audits porting the money committees no longer rotate. [26]. [9] Rotating staff meant a constant need for staff WBI co-hosted the May 2011 Accountability Con- training, which negatively influenced the work of ference that provided guidance on good practices in members of parliament. Also in 2012, the Tanzania performance auditing. PAC began championing a change of the budget In 2013, the Speaker of the House in the year so that they could receive the budget in April Tanzania parliament announced the new Parliamen- and complete its review by the end of June. [10] tary Budget Committee, after a yearlong proposal The PAC proposal was accepted and since 2013 the from the PAC to establish a distinct committee for budget year has been amended. [28] The PAC set scrutiny of the draft budget. The plan was then the ground for the reform and started the change postponed to establish a separate secretariat for the process based on the proposal from the National budget committee. [12] This reform will enhance the Audit Office. parliament’s role in the budget process and ensures Amending the budget year schedule enhances the PAC can review audit findings before the budget parliament’s role in the budget process/cycle—by is formulated. Enhancing parliamentary oversight reviewing the current year’s audit reports before the function by establishing budget committees is budget for the following year is formulated, parlia- critical in enforcing financial accountability, com- ment links ex post scrutiny with ex ante engage- bating fraud and corruption and promoting good ment. governance in the public sector. This increases voter WBI co-hosted regional capacity development confidence that their tax monies are used responsi- activities on effective parliamentary participation in bly, which, in turn, increases public confidence in the the budget process within the SADCOPAC network credibility of government institutions. to empower its members to take action on the WBI co-hosted the EAAPAC Annual General agreed good practices. WBI also provided guidance Meeting in October 2012, at which the Ugandan and enabled knowledge exchange on good prac- and Kenyan delegations presented on parliamentary tices on the role of parliament in the effective and budget offices and budget committees. The Tanza- timely review of the national budget. nian delegation engaged in this peer-to-peer learn- In 2012, the Uganda appointed a permanent liai- ing and has started to apply new knowledge. WBI son officer in the parliament. [19] The liaison officer provided guidance and supported the exchange of will make it easier for PAC members to understand knowledge on good practices among members of and review audit findings and hold the government parliament from the region. accountable for spending of public funds. Then In 2012, members of the newly elected South in 2013, the Uganda PAC started to review perfor- Sudan parliament established a PAC. [11] The PAC is mance audit reports. Two subcommittees have been charged with examining and investigating financial established within the PAC to increase the number misconduct within public institutions, and reporting of the reviews. [20] Performance audits examine not cases of misuse of public funds to the plenary to just executive spending but also development pro- decide on punitive measures. gram effectiveness. In 2013, the parliament in South Africa estab- WBI co-hosted the November 2011 Annual lished a Parliamentary Budget Office. [29] The bud- General Meeting of SADCOPAC and in May 2012 get office is to provide more independent analysis to the Effective Public Financial Accountability course parliament on the state of the nation’s finances, the in the Southern and Eastern African Region Confer- government’s estimates and trends in the economy. ence. These events provided guidance on how to Upon request from a committee or parliamentar- improve relations among PACs and Auditor General ian, the office could estimate the financial cost of Offices to hold government accountable. any proposal for matters over which parliament has In 2012, the Swaziland parliament adopted a jurisdiction. recommendation proposed by its PAC chair to 46 substantiation of outcomes To verify the accuracy of the outcomes mapped and enrich WBI’s understanding of them, the external consultant selected five outcomes [3, 17, 19, 22 and 24) and asked 10 people independent of WBI but knowledgeable about the change to review each and record whether they agree with the outcome as described. Six people responded to four outcomes [except 19]. Four of these substantiators fully agreed on the description of the outcomes and their significance and two provided additional information to clarify these. Excerpts of the substantiators’ comments on the outcomes achieved: “There is a significant need to centralize and streamline procurement into one institution strictly regulated. Such an institution shall minimize the huge public funds lost in the procurement process. This is due to the fact that all institutions procure and dispose independently without checks and supervision.” —Ayaga Garang, PAC Clerk, South Sudan “All reports go through a hearing but do not reach the level where we account both to the public and parliament. This is a serious gap in our country because of the lack of a vibrant multi-party system.” —Hon. Seif Girma, PAC Chair, Ethiopia WBI co-hosted regional capacity development WBI sponsored regional network events that activities within the regional networks. WBI provided provided learning on the role of performance audits guidance and secured South-South knowledge and their impact on good governance. exchange on the importance of efficient parliamen- The Kenya PAC increased its understanding of tary oversight of the public funds. how to better use audit reports to support reform. In sum, these outcomes exemplify how parlia- [4] In 2010, the Kenya parliament approved the ments in the participating countries are improving provision in its new constitution for the auditor their organizational effectiveness, including the general to send all of its audit reports, including timely and independent review of national budgets. performance audit reports, directly to parliament for They are establishing PACs, increasing performance oversight. [5] auditing of budget effectiveness, improving efficien- In 2012, the Zambia parliament amended its cies in committee structure and clarifying roles and rules of procedure so that the executive must pro- responsibilities of staff. vide quarterly progress reports of the implementa- tion of PAC recommendations and resolutions. [14] Outcome Area 3: Improved Legitimacy and Previously, the reports were submitted on an annual Transparency of Budget and Audit Processes basis, making it more difficult to monitor implemen- Public confidence on budget and audit reports is tation and enforcement of PAC recommendations. often weak due to a lack of transparency and legiti- The same year, the Botswana parliament amended mate processes. its rules of procedure to open PAC meetings to the Throughout 2011, PACs in the regional networks public. [15] The first meeting open to the public increased their knowledge about how to use audit took place in May 2012. Examinations in public is reports to propose reforms. For example, in Zambia expected to improve transparency in the handling of the PAC reviewed three of the 12 performance audit public funds and enhance PAC‘s oversight role. reports published by the National Audit Office and In 2012, the Tanzania government started imple- added to the reports’ recommendations to guide menting the PAC’s recommendations included in how ministries respond to the office’s recommenda- the audit performance reports made by the National tions. [6] This increased the PAC’s experience and Audit Office. [8] For example, the recommendations know-how to use national audit reports to publicly included establishing a dedicated division in the propose government reforms. The PAC had previ- corresponding ministry to address fire outbreaks. ously thought that it required a lot of experts, time WBI co-hosted capacity development activities and money to develop and review performance within the regional networks that included guidance audit reports, but then discovered they were able to on the role of the PAC in tracking its recommenda- manage with their current resources. tions, open public hearings and parliament’s role 47 to exert pressure on the executive to secure open- transport and health amid allegations of government ness about its activities and press for improvement corruption under pressure following reports tabled in and efficiency in public services. WBI also provided the National Assembly by the PAC, Parastatal Orga- guidance on the importance of performance audit nizations Accounts and Local Authorities’ Accounts. reports and their role in scrutinizing public expendi- [13] In Tanzania there are two main forms of abuse of tures. power: petty corruption, which is mainly associated In 2012, the Ethiopia PAC instituted in parlia- with small bribes, and big corruption, which mostly ment routine hearings every Wednesday and Friday involves large sums of money with political figures that are open to the media. The PAC calls witnesses involved. Actions taken by PAC members, based on to provide testimony on issues raised in the audit the controller and auditor general’s annual report, report. [21] Including the media is an important tackled big corruption that has hampered economic mechanism for public accountability, verifying audit growth. reports and increasing their objectivity and legiti- In 2012, the Rwanda PAC released its first review macy. The same year, the Ethiopia PAC increased of state finances, which reported Rwf 9.7 billion the use of performance audits, and all performance (US$16.3 million) lost in 2009–2010 as a result of audit reports go through a public hearing process to weaknesses in government operations. The PAC increase accountability of the executive to the public formed recommendations for government reforms. and parliament. [22] As a result, the PAC is able to [17] The review established the imperative for parlia- determine value for money of public funds. This is ment to act on these discrepancies in public spend- also an opportunity for the government to approach ing. the PAC to find solutions to implementation prob- By 2013 in Swaziland, government agencies, the lems. Anticorruption Commission and police have used In 2013, the South Sudan PAC institutionalized a PAC recommendations to investigate the actions of pre-media briefing for public hearings. [23] individuals for possible prosecution. [27] WBI co-hosted the SADCOPAC Annual Con- WBI co-hosted regional capacity development ference in September 2012. It provided guidance activities within SADCOPAC that emphasized the on opening hearings to the media and using per- oversight role of parliament and the Auditor General formance audit reviews to track spending. WBI Office. The countries followed the good practice presented a training workshop for new PACs on guidance provided on the role of parliament in curb- audit reports and public inquiries, including a mock ing corruption. hearing in which parliament presented a pre-media In 2013, the South Sudan PAC recommended to briefing to media practitioners and discussed how to their government to establish a National Procure- strengthen their communication. ment Institution to prevent fraud, waste and corrup- This set of outcomes exemplifies how parliaments tion in public funds spending, and they submitted in the PAC networks are upgrading the legitimacy a bill to create the institution in parliament. [24] If and transparency of budget processes by improv- approved, the National Procurement Institution ing the use of audit findings to propose government would be a major mechanism to tackle corruption in reforms to benefit the public. They also increasingly South Sudan. require all audits to go through public hearings WBI sponsored the May 2012 Namibia Confer- to make the process more accountable to citizen ence that examined procurement process and demands. accountability, at which members of the South Sudan PAC attended. Outcome Area 4: Improved Corruption In sum, this outcome area shows how participat- Mechanisms ing countries are increasingly using PAC recommen- African governments are increasingly using PAC dations to expose, investigate and take actions on recommendations as a mechanism to expose the corruption. corruption that had hindered the benefits of public spending for citizens. Conclusion In May 2012, the Tanzania president dismissed Due to this initiative, progress was made in four the ministers of finance, energy, tourism, trade, areas: (1) Improved relations among parliaments 48 in Africa through PAC networks for peer learning Next Steps around reform good practices; (2) Improved effec- WBI aims to provide ongoing support to the tiveness in using audit reports and parliamentary regional networks through: procedures to scrutinize implementation of national • Strengthening operations of network secretariats. budgets; (3) Improved legitimacy and transparency • Providing guidance on good practices and facili- of budget and audit processes; and (4) Improved tating further exchange of knowledge and les- corruption mechanisms and policy guidance. sons. However, weaknesses still exist: more enhance- • Encouraging network members to continue ments for the networks’ operations so the secretari- engaging in a change process. ats are strengthened in knowledge management, • Expanding the networks to include new members communication, monitoring and evaluation; and and foster sub-communities that respond to infor- further capacity development of PACs to carry out mational needs of different change agents (for reforms for budget and audit processes, particu- example, a community of PAC clerks and a com- larly to improve transparency and accountability to munity of Portuguese-language parliaments). citizens. By strengthening secretariat operations and M&E Progress to date and the potential to advance is systems, WBI anticipates the networks will have strong because responsibility for implementation enhanced capacity to secure funds from additional of reforms at the country-level rests with individual development partners once this project is complete. PACs. Based on good practice, the change agents There is ongoing demand and commitment from have been empowered to start and nurture change network members to provide in-kind contributions processes in their respective parliaments and to and participate in learning network activities. share their experiences and lessons learned within The rate and range of outcomes are anticipated the networks. to increase over time. It is expected the rate of out- Future support should be based on the self- comes will increase to reflect an increase in the num- identified needs of the networks’ membership and ber of network activities and improved knowledge information sharing, coordination and monitoring of management strategy. It is expected the range will the joint projects’ activities should be strengthened. increase since the engagement model is predicated Also, more should be done for the Portuguese- on change agents identifying and sharing develop- speaking members of the network. ment challenges with networks to benefit from oth- ers’ experiences in designing their responses. The for your information Acknowledgments Project Contact Thanks to Paulina Biernacka, Miraim Bensky, Mitchell O’Brien, WBI Social Accountability practice, Sruti Bandyopadhyay and Vienna Marie Pozer mobrien@worldbank.org of the WBI Social Accountability team involved Email in the project and mapping exercise. WBI Capacity Development and Results team at Thanks to: capacity4change@worldbank.org Kaggwa Mustapher Abdullah, Parliamentary Website Diplomacy Expert, Rwanda www.worldbank.org/capacity Parminder Brar, Lead Financial Management Specialist, World Bank © Copyright 2013 World Bank Ayaga Garang, PAC Clerk, South Sudan WBI’s Capacity Development and Results team led the outcome Hon. Seif Girma, PAC Chair, Ethiopia mapping; Jenny Gold coordinated the exercise with support Kifle Gizaw, PAC Clerk, Ethiopia from Ricardo Wilson-Grau. Sharon Fisher provided editorial and Mathew Kileo, PAC Clerk, Tanzania design services. Samuel Otoo provided overall guidance. Photo by South Africa government 49 range in the outcomes will represent the diversity of 9 In financial audits, auditors use standard procedures and rely the countries in the network and reforms they try to on accounting principles to determine the financial health of an organization. Performance audits determine whether an agency implement. n program is efficiently and effectively delivering the intended results. NOTES 1 PACs are the committees in parliament tasked with scrutinizing the government’s implementation of the national budget. Their primary source of information is audits reports prepared by the Supreme Audit Institutions. 2SADCOPAC members are Angola, Botswana, DRC, Lesotho, Madagascar, Malawi, Mauritius, Mozambique, Namibia, Sey- chelles, South Africa, Swaziland, Tanzania, Zambia and Zimbabwe. 3EAAPAC members are Burundi, Ethiopia, Kenya, South Sudan, Rwanda and Uganda. 4 Mapping outcomes—and related outputs and milestones—can help us learn from change processes that occur during program delivery that often seem complex and opaque because they involve multiple actors and address large development problems. An outcome is what each social actor (or change agent) did, or is doing, that reflects a significant change in their behavior, relation- ships, activities, actions, policies or practice. The program may influence these changes, directly or indirectly, partially or wholly, intended or not. Outcomes are identified at two levels in relation to the goal: institutional changes relate to societal, policy and organizational changes; and learning/capacity changes relate to awareness, knowledge or skills, collaborative action, or the use of knowledge or innovative solutions. These levels are based on the Capacity Development and Results Framework. The framework provides a systematic yet flexible approach to designing capacity development strategies and programs, monitoring and adaptively managing interventions, and evaluating and learning from their results. 5 Outcome harvesting is a practical assessment tool from the outcome mapping community of practice. It can be used for real- time monitoring and evidence gathering from complex develop- ment processes that involve multiple stakeholders. It is based on a similar concept of locally driven change from the Capacity Development and Results Framework. The tool was customized to gather information on outcomes—and related outputs and mile- stones—to learn from what changed, for whom, when and where, the significance of the change and how the program contributed to each change. 6 The numbers in brackets correspond to the outcomes in Figure 1. The text that usually follows each outcome refers to its significance. The process of change the outcomes represent can be seen in Figure 2. 7 Change agents are leaders, groups or organizations from gov- ernment or non-state that drive change. 8 Change strategy refers to how change happened to advance progress toward the development objectives—the development problems addressed, types of outcomes achieved, WBI contribu- tions, and
partners involved. A change strategy may include dif- ferent types of
change processes or outcome areas depending on the complexity of the multi-actor institutional changes involved in a program. 50 Priority Setting and Constitutional Mandates in Health S ince 2010, in barely three years, coalitions1 Development Objective that formed regionally and nationally in Improve the level and distribution of health outcomes Latin American countries have influenced by applying rights-based principles to health policy. institutional changes in favor of the constitu- Problem tional right to health. These changes not only united In recent years and in different settings, citizens are seven Latin American countries (Argentina, Brazil, increasingly litigating their health rights. Courts are Chile, Colombia, Costa Rica, Peru and Uruguay), favorably responding to these petitions and they are holding States accountable for their (in)actions. but also have had an international ripple effect, with However, this increasing trend of litigation may have countries in Africa and the Middle East identifying a negative unintended consequences (such as, it may be need for similar multi-stakeholder processes. WBI’s regressive since the poorest may not be benefitting) Constitutional Mandates in Health initiative sup- while its potential positive consequences may not be ported the emergence of these coalitions. fully exploited (such as improved service delivery). In January–March 2013, WBI mapped outcomes2 Specific Objectives of this initiative using a customized outcome map- Enhance the effectiveness of health and judiciary arrangements, and the transparency, accountability and ping tool.3 The visual map (Figure 1) presents the participatory process for setting priorities and delivery sequence of outcomes achieved by change agents— services to realize the right to health for all citizens. the leaders, coalitions and organizations involved in 51 Figure 1. Map showing how the outcomes connected and built over a four-year timeframe (1) Officials (5) Four (6) Regional (18) Regional (19) Two COMMUNITY (20*) from health countries coalition coalition universities from Of practice Colombian executive initially— became multi formed online Colombia and Court set new (21) Saluderecho branches in Argentina, stakeholder community Spain initiated precedent to community raised Argentina, Uruguay, Peru, (joined by of practice network of clarify health awareness and Brazil, Perú, and Costa Ministries of with broader researchers rights to citizens information Colombia, Rica—formed Health and group of by webcasting by broadcasting transparency on Costa Rica and coalition of Academia) practitioners, academic findings hearing of a common issues Chile formed judiciary to and it further called www. judicial ruling using YouTube informal address policy focused its saluderecho. network issues on right actions with net to health its now seven members (Brazil had (7) Brazil convinced (28) Seven countries (2) Peru joined) supreme court to regional multi- formed regional (4*) Regional recognized take active role in stakeholder coalition steering committee judiciary is supreme court of judiciary and health and countries regional coalition key actor officials to address commit to challenges regional and mobilized and national dialogues costa rica country coalition the Peruvian Constitutional among health, (17) (13) Costa (14) (15) Coalition (16) Coalition Tribunal Inter judiciary and Coalition Rica formed Professional decided to use organized American CSOs organized a country NGOs that other country dialogues Court of coalition for represent experiences to identify dialogue Human Rights national multi- physicians and to identify options with stakeholder lawyers offered options for around academia dialogue to lead and policymakers waiting lists from host dialogue universities (3) Peruvian Constitutional Tribunal and Inter American global effect Court of Human Rights endorsed (22) Rwanda, (26) State (25) State regional Kenya, Brazil judge level Ministry dialogue on and Egypt in Brazil of Health issues related to identified need published in Brazil right to health for similar paper committed processes to discussing to coalition achieve right litigation to improve Uruguay country Coalition to health out- in Brazilian (11) Coalition access to comes in their private health launched medicines (8) Uruguay (9*) Country countries sector database of Supreme Court coalition (10) judiciary and promoted created Experienced health data (27) Egypt (23) establishment technical Uruguay decided Dartmouth (24*) NORAD of national secretariat to coalition to host Center partnered multi- anchor work provided knowledge partnered with WBI to stakeholder support to scale-up right exchange on (12) Judiciary with WBI to dialogue new Costa to health governance and health strengthen Rica coalition right to outcomes in and to organize actors use health Latin American transparency 1st dialogue database to outcomes and Africa on right to inform decisions health 2010 2011 2012 2013 Institutional changes Learning/capacity changes Outcomes related to societal, policy and organizational changes. Other outcomes related to awareness, Ownership of health and judiciary systems/transparent and knowledge or skills, collaborative action, or the participatory priority setting that progressively realize right to health use of knowledge or innovative solutions. Efficient policy to respond to citizens’ right to health * Outcomes selected for substantiation; Effectiveness of health and judiciary arrangements to realize citizens’ see sidebar. right to health 52 the initiative. It illustrates how key outcomes to date areas include changes in: leadership of judiciary and have connected and built on each other over time to health officials; multi-stakeholder arrangements to form multi-actor, institutional processes for change realize the right to health; transparent and partici- to address the initiative’s objectives and goal. patory decisions to strengthen policy; and global WBI team members identified and formulated learning to scale-up right to health outcomes. These the outcomes, presenting an explanation of their outcomes were analyzed and classified according to significance and how they had contributed—directly the types of change they achieved, then grouped or indirectly, in a small or big way, intentionally or based on how they connected and built on each not—by catalyzing or empowering the change other to affect change. agents to take new actions. Then, roughly 20% of the outcomes were independently substantiated for Outcome Area 1: Leadership of health and credibility in the mapping exercise. judiciary officials Judiciary and health officials have not customarily Background communicated with each other on challenges related Most Latin American countries have enshrined in to litigation on the right to health in their countries, their constitutions articles granting their citizens the yet the decisions of the courts affect the health sec- right to health. Since the majority of these constitu- tor. tions also provide mechanisms that expedite the Initially in 2010, officials from the health ministries judicial protection of this right, individuals can seek of Argentina, Brazil, Peru, Colombia, Costa Rica, swift court protection. This implies that the actions and Chile who faced similar challenges regarding of the judiciary and civil society now play a critical increasing litigation on the right to health formed, for role in holding the State accountable in realizing the the first time, an informal network. [1]4 The network right to health. aimed to share experiences and lessons learned on As the number of litigated cases on the right issues related to litigation on the right to health in to health increased dramatically since the 1990s, their countries and in the region. This knowledge the majority of litigation demands the provision of exchange led health officials in Peru to realize that services already included in the basic list (revealing a network among health executives alone could not difficulties in complying with policies) or the sup- address the problems and judiciary leadership was ply of new and expensive technologies (revealing required. They mobilized the Peruvian Constitutional difficulties in setting or enforcing priorities). While Tribunal to become involved in the right to health lawsuits may provide individual access to health ser- dialogue. [2] vices, the judicialization of this right can collide with After a process of engagement led by the Peru- the limited availability of resources faced by health vian Constitutional Tribunal, in April 2011 the Inter systems, and may even increase inequality in access American Court of Human Rights endorsed the to healthcare. regional dialogue on the right to health. Conse- WBI’s Initiative on Constitutional Mandates quently, both courts committed to co-host the first in Health is based on the theory of collaborative and second Regional Latin American Symposia in change: because these multiple actors view the June and December 2011. [3] As the first courts same problem from different perspectives, their joint to support an open conversation on the need for action becomes an effective mechanism in finding collaboration, their leadership encouraged strong innovative solutions toward the progressive and engagement of judiciary branches of government in sustainable realization of the right to health. In this the participating countries. sense, these multi-stakeholder collaborative pro- In June 2011, during the first Regional Latin cesses contribute to improve the level and distribu- American Symposium, the regional Supreme Court tion of health outcomes across Latin America. justices decided to promote multi-stakeholder dialogues with regional and national health authori- Outcome Areas ties on the realization of the right to health. [4] Such The change strategy achieved so far by this initia- leadership encouraged the participating countries tive can be seen through areas of outcomes (Figure to consider multi-stakeholder perspectives in their 2) that are described in the following sections. The decision-making process on right to health issues. 53 Figure 2. Change strategy showing how change happened to advance progress toward goal WBI Contributions • Knowledge exchanges, Change Agents5 regionally, nationally and • Officials from health executive branches in globally Argentina, Brazil, Peru, Colombia, Costa Rica, • Analytical skills Uruguay, and Chile • Financial support • Regional multi-stakeholder coalitions • Created and supported • National multi-stakeholder coalitions community of practice • National supreme courts • Provided technological • Peruvian Constitutional Tribunal and Inter American platforms and guidance Court of Human Rights for webcasting • Online community of practice • Acquired global partners for activities • Created and supported the safe space for coalitions to thrive Change Strategy6 Outcome Area 1. Leadership of Health and Judiciary Officials • Health and judiciary committed to collaborative process at highest level Problems Addressed Partners • Raised awareness of need for collaboration • Low levels of utilization of among diverse actors, particularly health and rights-based principles in • Pan American Health judiciary health policy Organization • Diverse actors involved in • Dartmouth Center Outcome Area 2. Multi-Stakeholder health litigation and priority • NORAD Arrangements to Realize the Right to Health setting • Inter American Court of • Formalized structure and strategies for regional and country coalitions • Judiciary and health Human Rights systems have limited • Salzburg Global Seminar • Raised awareness on common health issues by resources and experience broader group using community of practice to address citizen right to • Peruvian Constitutional Tribunal health demands Outcome Area 3. Transparent and Participatory Decisions to Strengthen Policy • Weak links between • Clarified health rights to citizens through regional, national and increased information global response for litigation and policy reforms • Increased practice of publishing data and information related to health rights and litigation for citizens and policymakers • Increased knowledge and collaboration with other countries and academia to inform policy, such as for waiting lists Development Objective Outcome Area 4. Global Learning to Scale-Up • Improve the level and Right to Health Outcomes distribution of health • Health and judiciary in countries in Africa and outcomes by applying Middle East committed to similar process rights-based principles to health policy • NORAD, Salzburg Global Seminar, and Dartmouth committed to partner with WBI • Raised awareness with global symposium Then, in June of 2012, the Brazilian Supreme a strategic and technical milestone. It also signals Court decided to host the Third Latin Ameri- the possible leverage of these countries to work can Symposium in June 2013. [7] This decision is together long-term to improve the right to health. noteworthy given Brazil’s leading regional role in WBI contributed to these outcomes by acting knowledge developments on the right to health, its as a convener, researcher and facilitator, supporting strong endorsement of a regional multi-stakeholder the country officials to form the informal network. dialogue and its working methodology representing WBI also organized knowledge exchanges for health 54 substantiation of outcomes To verify the accuracy of the outcomes and enrich WBI’s understanding of them, the external consultant selected four outcomes [4, 9, 20 and 24] and asked 13 people who are independent of WBI but knowledgeable about the change to review each and record whether they agree with the outcome as described. Nine people responded. Eight fully agreed with the description, significance and contribution of WBI to outcomes 4, 9 and 20. One provided additional information to clarify the description, significance and contribution of outcome 24. Excerpts of the substantiators’ comments on the outcomes achieved: “In general, I feel that the establishment of the secretariat [that anchors the Uruguayan stakeholder work], although essential, should be considered as just the first step in dealing with so complex and sensitive a topic. Its success will depend on its permanence over time and the incorporation of new members.” —Nilza Salvo, Director of CEJU and Minister of the Court of Civil Appeals, Uruguay “I think that bringing the parties together [at the First Latin American Symposium on the Right to Health and Health Systems in Costa Rica] is a first step toward understanding of both positions, which can eventually facilitate commitments in decision-making.” —Ana Virginia Calzada Miranda, President, Supreme Court of Costa Rica “Dissemination of the judicial hearing, but not the judgment, as the text would seem to suggest, guarantees not only the right of everyone to have access to public information on the problem of the regulation and control of resources earmarked for financing health systems, but also will afford the general public a means of obtaining direct information on the follow up of Ruling T-760 of 2008, issued by the Constitutional Court of Colombia, which directs the competent authorities to correct the regulatory lapses that affect the health system in order to ensure the effective exercise of this basic right.” —Jorge Ivan Palacio, President, Colombia’s Constitutional Court “I feel that the way in which the WBI addressed the issue [technical secretariat] is quite adequate, because it offered its collaboration while encouraging local stakeholders to seek their own means of analysis and discussion. This has made it possible to take advantage of both external experience and internal contributions.” —Leticia Gómez, Head of Legal Department, National Resources Fund of Uruguay executives, which identified the need for judiciary American countries (Argentina, Brazil, Chile, Colom- involvement. After developing an approach to bia, Costa Rica, Peru and Uruguay). health that was interesting for the judiciary, WBI engaged these courts in the regional Latin Ameri- Outcome Area 2: Multi-stakeholder can symposia, and helped co-organize events that arrangements to realize the right to health allowed for a safe space for discussion among stra- Multi-stakeholder arrangements were not yet in tegic partners. WBI also invited potential champions place to support regional and national dialogue within the Brazilian judiciary branch to the second among judiciary, health and other stakeholders to Latin American Symposium in December 2011 and inform policy issues on the right to health. Since to participate in the Roundtable on Universal Health 2011, regional and country level multi-stakeholder Coverage and the Right to Health in Washington, processes have emerged to engage stakeholders. D.C. in June 2012. In sum, the change strategy of this initiative Regional coalition included outcomes to raise awareness of the need At the regional level, in June 2011, four countries for judiciary, health and other stakeholder collabora- initially—Argentina, Costa Rica, Peru and Uruguay— tion to address right to health issues. These changes formed a coalition of judiciary to address policy strengthened the leadership and commitment of issues on right to health. [5] Priority actions included health and judiciary officials at the highest level to systematizing judiciary data, building the capacity of address the problems, especially among govern- judges to understand health system decisions and mental health and judiciary officials in seven Latin health systems to understand judiciary decisions, and developing a broader network of practitioners 55 outside the regional or country coalition member- learning process around how countries and develop- ship for awareness-raising and dialogue. ment partners conceptualize, understand and sup- At the second regional symposium, in December port actions to address the right to health. 2011, the regional coalition became multi-stake- holder (joined by ministries of health and academia). Conclusion The coalition further focused its actions with its now The ultimate goal of WBI’s Constitutional Mandates seven members (Brazil, Colombia and Chile had Initiative is to increase the level and distribution joined), to improve knowledge on how to address of health outcomes. It is advancing this goal by right to health issue they could address. In early sup¬porting new collaborative leadership among 2013, Evangelina Castilho Duarte, a Brazilian judge judiciary and health officials, as well as the creation from Minais Gerais, published a paper in the journal of effective multi-stakeholder processes to influence l Justiça & Cidadania discussing the main issues policy around the right to health. related to litigation in the Brazilian private health Multi-stakeholder coalitions have the potential sector. [26] Judge Duarte’s reflections illustrated the to understand the underlying causes of the rapid acquisition of new knowledge during her partici- increase in litigation from different perspectives and pation in the First Global Seminar on the Right to to act accordingly. They can also potentially increase Health and Health Systems. Judge Duarte is a key the level of fairness and effectiveness arising from actor in the Minais Gerais multi-stakeholder coali- the health system and from the judiciary system as tion. well. The achieved outcomes described in this case In addition, Egypt decided to host knowledge demonstrate how coalitions can have a positive exchanges to improve governance and transparency effect, by increasing transparency and participa- issues related to the right to health. [27] And Kenyan tory decisions to inform policy options to address and Moroccan delegations are joining the Third rights to health. In the process, change agents are Latin American Symposium on the Right to Health empowered in the most advanced countries such as (June 2013). Uruguay, Costa Rica and Brazil, where coalitions are The Norwegian Agency for Development Coop- already leading the discussions and the agenda. eration (NORAD) also began collaborating with WBI That being said, understanding the causes of to expand implementation of the right to health litigation and effectively transforming them into model. [24] This partnership has the potential to improved policies—thus contributing to the realiza- support Latin American and African regional activi- tion of the right to health—is a lengthy and complex ties, as well as content development on the issue. task. This is, therefore, not a on-off engagement Partnering with key global players validates the value but rather a dynamic process in which coalitions will proposition that underscores this initiative, and it encounter new challenges that will need innovative also enhances outreach and effectiveness. and adaptive solutions. Many challenges still exist— for example, fiscal and administrative costs of litiga- WBI contributed by conceptualizing strategic tion or para-judicial conflict resolution mechanisms complementaries between the global and regional are a challenge in upcoming coalition discussions. activities. WBI co-organized the first global sympo- Even though the results obtained so far have sium on the right to health in Austria in November been mainly concentrated in Latin America, the 2012 and invited Egypt to participate, linking to flexibility of the change strategy supported by this another WBI initiative in the Middle East and North initiative has drawn interest from countries globally Africa on improving governance and social account- that face right to health challenges and lack practices ability in health services. WBI also co-sponsored the to address them. There is also an increasing interest Learning Exchange Seminar on Operationalizing to use this initiative to systematically and adaptively Human Rights in Development in Oslo in 2012 with learn how to apply rights-based principles to con- NORAD. text-specific health policy needs of countries. Thus, in late 2012 and early 2013, the regional coalition had expanded its influence to countries Next Steps outside of Latin America. Several of the outcomes Over the next two years, new outcomes to improve here described helped advance the scale-up of this the efficiency of policy instruments and strengthen 56 the effectiveness of multi-stakeholder arrangements societal, policy and organizational changes; and learning/capacity are expected. Four categories of outcomes will most changes relate to awareness, knowledge or skills, collaborative action, or the use of knowledge or innovative solutions. These likely arise: levels are based on the Capacity Development and Results 1. Appearance of new pieces of legislation or Framework. The framework provides a systematic yet flexible administrative policies aimed at improving the effec- approach to designing capacity development strategies and programs, monitoring and adaptively managing interventions, tiveness and transparency of the decisions made in and evaluating and learning from their results. the judiciary and health sectors. 3 Outcome harvesting is a practical assessment tool from the 2. Improved health service delivery, particularly outcome mapping community of practice. It can be used for benefiting the poor and marginalized. real-time monitoring and evidence gathering from complex 3. Increased number of qualitative and quantita- development processes that involve multiple stakeholders. It tive assessments of the causes and ultimate impact is based on a similar concept of locally driven change from the Capacity Development and Results Framework. The tool was of health litigation. customized to gather information on outcomes—and related 4. New countries working with multi-stakeholder outputs and milestones—to learn from what changed, for whom, coalitions. n when and where, the significance of the change and how the program contributed to each change. 4 The numbers in brackets correspond to the outcomes. The text NOTES that usually follows each outcome refers to its significance. The process of change represented by the outcomes can be seen in 1The stakeholders in the coalitions include executive, legislative Figure 2. and judicial branches as well as other government institutions 5Change agents are leaders, groups or organizations from at the central and sub-national levels, health care organizations, physicians, patients, academic institutions, civil society government or non-state that drive change. organizations and the private sector. Source: Brochure “Creating 6 Change strategy refers to how change happened to advance a Sustainable Platform for Multi-Stakeholders to Coalesce and Address the Progressive Realization of the Right to Health.” progress toward the development objectives—the development problems addressed, types of outcomes achieved, WBI 2 Mapping outcomes—and related outputs and milestones— contributions, and
partners involved. A change strategy may can help us learn from change processes that occur during include different types of
change processes or outcome areas program delivery that often seem complex and opaque because depending on the complexity of the multi-actor institutional they involve multiple actors and address large development changes involved in a program. problems. An outcome is what each social actor (or change agent) did, or is doing, that reflects a significant change in their behavior, relationships, activities, actions, policies or practice. The program may influence these changes, directly or indirectly, partially or wholly, intended or not. Outcomes are identified at two levels in relation to the goal: institutional changes relate to for more information Project Contact Maria-Luisa Escobar, WBI Health Systems practice manager, mescobar@worldbank.org Email WBI Capacity Development and Results team at capacity4change@worldbank.org Website www.worldbank.org/capacity and www.saluderecho.net © 2013 World Bank WBI’s Capacity Development and Results team led the outcome mapping; Jenny Gold coordinated the exercise with support from Ricardo Wilson-Grau. Sharon Fisher provided editorial and design services. Samuel Otoo provided overall guidance. Acknowledgments Thanks to the WBI Health Systems practice team members involved in the project and mapping exercise: Leonardo Cubillos, Roberto Iunes, Maria-Luisa Escobar, Antonio Paniagua, Janet Bonilla and Santiago Pereira. Thanks to all of the persons that provided information to document this case. 57 Improving Open Contracting Processes at the Country and Global Level S ince late 2011, significant if initial changes Development Objective in contracting practices have been made on Improve benefits of public goods and services for all the ground in several countries worldwide. citizens. These changes are a positive step that could lead to better governance, with citizens receiving Problem goods and services they deserve so that develop- Failings in public contracting—such as corruption, opaque processes and weak compliance—impede the ment outcomes can be achieved. On the country achievement of development outcomes in countries, level, key stakeholders established collaborative limiting economic growth and social benefits. processes to tackle such issues as contract disclosure and monitoring. On the global level, key players Specific Objectives committed to develop and promote global norms Open government contracting to more public scrutiny and data standards to improve open contracting and participation; increase disclosure of public contracts; increase non-state participation in public contracting in (OC) practices. Making these advances in OC was a systemic and collaborative manner; improve open con- borne out of development efforts by several coun- tracting practices in key sectors; and combat corruption tries with support from WBI’s OC team. and inefficiencies in public contracting. 58 In January–March 2013, WBI mapped more than and due process. These reforms have had limited 30 outcomes1 from these efforts using a customized impacts because they have not fully addressed the outcome mapping tool2. These visual maps present lack of public information and citizen engagement, the sequence of outcomes achieved by change among other reasons. agents—the leaders, coalitions and organizations To address these shortcomings and improve the involved in the process. Outcomes were mapped at capacity of public contracts to deliver better out- the global level and from country efforts in Ghana, comes for citizens, the OC initiative was launched as Mongolia, Nigeria and Uganda to show examples of a collaborative movement. OC emerged as a result changes that are part of a larger program. The maps of collaboration between WBI and the German gov- illustrate how the outcomes connected and built on ernment aid agency Deutsche Gesellschaft für Inter- each other over time to form multi-actor, institutional nationale Zusammenarbeit (GIZ) and has since solidi- processes for change to address the development fied into an Open Contracting Partnership (OCP) led objectives and goal. by steering group members from governments, civil WBI’s OC team members identified and formu- society and multilaterals. This collaboration builds lated the outcomes, presenting an explanation of on WBI ‘s work with OC coalitions from diverse sec- their significance and how WBI had contributed— tors in more than 30 countries, to monitor and give directly or indirectly, in a small or big way, intention- feedback to governments on contract award and ally or not—by catalyzing or empowering the change performance and to make contracts open, account- agents to take new actions. Then, roughly 20% of able and easily understood. the outcomes were independently substantiated for Furthermore, the broad OC framework and its credibility in the mapping exercise. Each outcome multi-stakeholder coalition building approach serves identified is mapped, numbered and described in as an umbrella under which the WBI Health Systems the context of a strategy to catalyze change. practice carries out their Pharmaceutical Procure- ment and Supply Chain Management work.3 In addi- Background tion to health, the multi-stakeholder work covers sec- Contracts are at the core of how countries operate— tors of extractive industries (for example, in Ghana), they are at the nexus of revenue generation, budget education (for example, in Uganda), and infrastruc- planning, resource management and delivery of ture (for example, in Nigeria or Mongolia), where public goods. Governments around the world spend benefits of OC are sought. This case study includes an estimated US $9.5 trillion every year through examples of outcomes in each of these areas. contracts. Yet, contracting information is often unavailable outcome areas for public scrutiny, and the resources spent through The process of change from the OC initiative can be these contracts are often poorly managed or misap- seen in four areas of outcomes (Figure 1) that rep- propriated. These problems are particularly troubling resent the major change paths. All of the outcomes in the wake of a global financial crisis when pressure were analyzed and classified according to the types to account for use of limited resources is greater of change they achieved. They were then grouped than ever. When companies, governments and citi- based on how they connected and built on each zens continue to be affected by ineffective and unfair other to form a story for change. contracting practices, theft and waste, everyone pays the price. Outcome Area 1: Global commitment and Failings in public contracting are undermining priority setting development due to weak compliance with regula- tion, corruption, inefficient and opaque contracting See figure 2 map for the following outcomes. processes and poor oversight of contract implemen- tation. For example, service delivery in many parts of Open contracting steering group and the world has been hampered by collusion, delays, champions poor delivery and high costs. Over the years a num- In 2011, GIZ partnered with WBI to tackle the ber of governments have reformed public contract- problems of opacity within, and poor oversight of, ing legislation mainly by strengthening procedure government contracting. They convened leaders 59 Figure 1. Change strategy showing how change happened to advance progress toward goal WBI Contributions Change Agents5 Globally • Open Contracting Steering Group members and • Proposed the idea of local networks the collaborative effort, • Other open contracting global champions such as reached out to partners to Publish What you Pay and Global Witness secure their involvement, • Country coalitions in over 30 countries* convened meetings Country change agents presented in case: with open contracting • Uganda Contract Monitoring Coalition champions  • Uganda Ministry of Education Country level * Change agents from four of the • Public Procurement Partnership of Mongolia countries and their outcomes are • Facilitated process that helped shape multi- • Mongolia Ministry of Finance presented in the current case. stakeholder coalitions • Nigeria Contract Monitoring Coalition • Supported coalitions in • Nigerian Society of Engineers developing message • Power Holding Company of Nigeria and strategic plans and • Federal High Court in Abuja by providing coaching • University of Ghana and network-building assistance • Technical advice on procurement CSO monitoring • Skills building on Change Strategy6 monitoring tools Outcom Area 1: Global Commitment and Priority Setting • Steering committee: Committed collective influence from varied sectors (priorities include health, extractive industries, education, Partners Infrastructure) and new resources and networks Problems Addressed to form Open Contracting Partnership • GIZ as initial co-convener • Lack of transparency in • World Bank: Demonstrated commitment by at global level government procurement release of major contracts dataset • Open Contracting processes • Improved awareness of open contracting Steering Group members • Weak public scrutiny in an practices and principles organized and systematic Outcome Area 2: Effective Multi-stakeholder way Processes in Countries to Open Contracting • Under-performing • Effective strategies and priority setting to government contracting develop monitoring tools, shape procurement processes laws, with ownership of varied actors and • Corruption and promote improved disclosure of contracting inefficiencies in public processes contracting • Improved collaboration across government, • Weak participation of non- private and civil society sectors state actors in contracting Outcome Area 3: Improved Policy Efficiency and Responsiveness in Countries • Government release of contracting data • Shaping of procurement laws • Use of litigation to disclose information Development Objective • Improved awareness of government on the value • Improve benefits of pubic of contract monitoring by non-state actors and goods and services for all of their role in advocating for change in rules citizens and regulations Outcome Area 4: Improved Conditions for Non- State Actors to Participate in Open Contracting • Institutionalized non-state monitoring of contracting processes • Improved skills and applied know-how in using regulations and monitoring tool 60 Figure 2. Map of outcomes showing global changes linked and built over three years (2) Construction Sector Transparency (7) Steering group advocated UK Initiative joined steering group, involving the government and others to endorse construction sector open contracting and include principles (1)* GIZ in G8 Declaration (3) Integrity Action joined steering group, agreed to (8) Steering group began advocating to (33) Steering bringing experience in citizen empowerment co-convene present open contracting principles to group and networks and commit Open Government Partnership to adopt members resources (4) Governments of Philippines joined steering in national action plans solidified to address group, bringing links to government and efforts to experience of its procurement agencies (9) Steering group began advocating opacity officially to include open contracting principles and poor (5)* Oxfam America joined steering group, to become Open in mandate of Extractive Industries oversight of connect effort to civil society globally globally Contracting Transparency Initiative government as well as extractive industries Partnership contracting (30) Government of Colombia joined (6) Transparency International joined steering steering group, bringing its links to group, bringing global civil society input and government and experience of its experience National Procurement Agency (28) World Bank published (29) World Bank Controller’s GLOBAL changes major contracts dataset Office launched mobile in open format on Bank application for financial finances website snapshots of Bank’s activities 2011 2012 2013 Institutional changes Learning/capacity changes Outcomes related to societal, policy and organizational changes Other outcomes related to awareness, knowledge or Commitment, participatory priority setting, skills, collaborative action, or the use of knowledge transparency, accountability or innovative solutions. Policy clarity to combat inefficiencies and corruption * Outcomes selected for substantiation; see page 8 sidebar. Effective collaborative strategy, government responsiveness and innovators to involve champions who could staff time. The OC steering group recently solidified make contract disclosure and monitoring the social its efforts to officially become the OCP. [33] norm. GIZ pledged staff time and approximately US To make OC effective and sustainable, the $400,000 for events and research toward the effort. process involves a diverse group of organizations [1]4 The co-convening model between GIZ and WBI that have the resources, influence and expertise to attracted reputable partners to join the OC process catalyze a global movement. CoST is respected in to help realize the vision. the construction sector and has an established pres- Through a consultation process between 2012 ence in Africa, Latin America and the Caribbean, and 2013, additional organizations joined the OC East Asia and the Pacific. Integrity Action leads leadership team: Construction Sector Transpar- well-known and respected programs, particularly ency Initiative (CoST), Integrity Action (formerly Tiri), in fragile states, and it has a presence in Africa, the Oxfam America, Transparency International (TI), the Middle East, North Africa and East Asia and the Philippines’ Government represented by the Pro- Pacific. Oxfam America is part of an international curement Policy Board and the Colombian Govern- confederation working in more than 90 countries. TI ment represented by the National Procurement is part of an international network working to fight Authority, Colombia Compra Eficiente. [2, 3, 4, 5, 6 corruption through more than 100 chapters across and 30] They became members of the OC steering the globe. Lastly, the Philippines and Colombian group that co-designs and co-funds OC activities. government partners are important additions, both Each organization committed considerable resources through contributing their views and attracting other to the common effort, including 5% to 15% of senior governments to join OC efforts. 61 WBI contributed by proposing the idea of the Community developers, including WBI staff, used OC collaborative effort to GIZ and engaging in a this information to build a prototype web application series of conversations about the initial vision and using the data available. Then, in January 2013, the structure of the OC steering group. WBI reached out Bank’s Controller’s Office launched a WB Finances to potential OC partners to secure their involvement. mobile application that provides current financial In early 2013, OC steering group members and snapshots of the Bank’s activities around the world other OC champions, such as the Publish What and allows users to explore the details of a country’s You Pay coalition, Global Witness and the Africa donor and/or beneficiary portfolios—including finan- Freedom of Information Centre, advocated with cial instruments covering contracts/procurement the United Kingdom government and other G8 data, and project information and locations. Inte- members to include OC principles in the G8 Dec- grated connections to social media networks (such laration. [7] Further, they began advocating to raise as Facebook and Twitter), email, and SMS text allow awareness and present OC practices as options to for easy sharing, and the application is available in members of the Open Government Partnership seven languages. [29] This application allows inter- (OGP)7 for adoption in national action plans. [8] The ested stakeholders to review and track the finances G8’s endorsement lent credibility to the effort and of Bank projects. boosted momentum for adoption of OC practices, In sum, during 2012, OC gained significant back- and OGP’s endorsement also increased the repu- ing from international players who formed a steering tation of the effort, tying it more closely with the group, and multiple OC champions committed to broader transparency agenda. advance OC through raising awareness of its ben- OC champions also began advocating to include efits. This movement was reinforced by the Bank’s OC elements in the expanded mandate of the release of a major contracts dataset. Extractive Industries Transparency Initiative (EITI). [9] Failings in the award and monitoring of large-scale Outcome Area 2: Effective multi-stakeholder oil/gas/mining deals risk undermining potential processes in countries for open contracting development outcomes in resource rich countries. The EITI’s endorsement of OC lent credibility to and See figure 3 for a map of the following outcomes. increased the reputation of the effort, raising its pro- file in the extractives sector and providing a platform Examples from Uganda, Mongolia and Nigeria for adoption among countries implementing EITI. In June 2012, after a one-year process, a group of WBI engaged in conversations and convened 25 organizations in Uganda signed a Memorandum meetings with OC champions with World Bank of Understanding to form the Uganda Contract operations, OC members and private sector actors Monitoring Coalition. [10] The coalition included the to build understanding of OC, ensure use of com- government represented by the Public Procurement mon terminology and collectively identify strategic Authority, members of the private sector and CSOs advocacy opportunities with G8 and OGP constitu- focused on themes ranging from water governance encies. to agriculture and education. Around the same time in Mongolia and Nigeria, two other similar groups World Bank of organizations formed formal coalitions to work on In 2012, OC champions participated in an ideation contract monitoring—the Public Procurement Part- session, which resulted in a focus on contracts and nership of Mongolia and Nigerian Contract Monitor- procurement data and prompted the release of a ing Coalition. [13, 18] data set of Major Contract Awards for Bank-funded The coalitions aim to improve their countries’ Operations.8 In June 2012, the World Bank rein- respective contracting processes through monitor- forced the global OC process by publishing this ing and advocacy. Uniting stakeholders in a coalition dataset in open format on its WB Finances website. makes them a more credible counterpart to engage [28] This represents the first time that in-depth data with government agencies to monitor whether on Bank contracts was publicly accessible, setting a public contracting is well managed, implemented precedent for transparency and allowing this data to and delivered. In addition, coalition members can be mined and analyzed. leverage each other’s resources, networks and 62 Figure 3. Map of outcomes showing country changes linked and built over three years (10) Uganda (12)* Uganda Ministry (32) Uganda (11) Uganda Contract Contract Monitoring of Education agreed coaliton received Monitoring Coalition Coalition formed to monitor contract list of 250+ schools subgroup developed UGANDA by 25 organizations, monitoring tool for performance of school for monitoring from including the construction and release IDA task team communities on Public Procurement list of 250+ schools school construction Authority, private around the country sector and CSOs (13) Mongolia 25 (14) Mongolia (15) Mongolia (16)* Mongolia (17) Mongolia CSOs formed formal coalition coalition engaged Ministry of Finance Ministry of Finance coalition called developed five- in policy dialogue changed the requested the mongolia Public Procurement year strategic plan with Ministry Implementing Rules coalition to Partnership to with initial focus of Finance on and Regulations coordinate CSO improve processes on shaping new Implementing of public procurement through monitoring procurement law Rules and procurement law monitoring efforts and advocacy implementation Regulations of law (19) Nigeria coalition (21) Nigeria Bureau (23) Nigeria Power Holding built skills of Nigerian of Public Procurement Company refused to disclose nigeria Society of Engineers and provided materials procurement information CSO representatives to to promote contract for Bank-funded contract, (31) Nigeria court monitor power sector monitoring at so coalition brought suit to ordered Power procurement coalition’s workshop Federal High Court Holding Company (18) Nigeria to release contract Contract documents and pay Monitoring (20) Nigeria coalition (22) Nigeria costs to coalition Coalition built knowledge of coalition developed (24) Nigeria Federal High formed from observers on how to use Infrastructure Rating Court ruled favorably on suit groups across Public Procurement Act, and Assessment public, private Freedom of Information Template for and civil society Act, and procurement monitoring after (25) Nigeria Power Holding sectors monitoring tools contract award Company released partial processes records and information (26)* Ghana University used for (27) Ghana CSO groups use Ghana the first time GIS-based mapping map to monitor extractive industries ghana to make public data on extractive in the country industries available 2011 2012 2013 Institutional changes Learning/capacity changes Outcomes related to societal, policy and organizational changes Other outcomes related to awareness, knowledge or Commitment, participatory priority setting, skills, collaborative action, or the use of knowledge transparency, accountability or innovative solutions. Policy clarity to combat inefficiencies and corruption * Outcomes selected for substantiation; see sidebar. Effective collaborative strategy, government responsiveness 63 expertise. They can also better coordinate access to whereas previously only selective processes were information requests, policy advocacy and monitor- monitored. [16] The changes in the regulations ing efforts. institutionalize citizen participation in procurement, Over the summer of 2012, the Public Procure- which in turn will help to combat corruption, opaque ment Partnership of Mongolia developed a five-year contracting processes and poor oversight of con- strategic plan to, among other goals, shape the tract implementation. new procurement law implementing regulations In 2013, the Mongolian Ministry of Finance in the short term. [14] This agreement among the requested the Public Procurement Partnership’s sup- CSOs lends structure and clarity to their advocacy port in coordinating the CSO procurement monitor- efforts, allowing for clearer goals and agreed-upon ing efforts that are required under the amended pro- strategies, ultimately improving the efficiency of the curement law. [17] This request is a first step toward collaboration. a more collaborative relationship established by the WBI provided guidance to Uganda, Nigeria and amended procurement law and its rules. Enhanced Mongolia on forming coalitions, facilitated meet- citizen participation should result in improvements ings and advised them on how to engage potential in the procurement system and ultimately in better members and structure the coalitions. For example budget implementation and public service delivery. in Mongolia, WBI shared examples of strategic plans WBI supported the coalition in developing a and engagement mechanisms of other formal coali- united message to engage with its government. tions, and provided feedback on different iterations The support included coaching, network building of the plan. As part of the process, WBI funded a support and technical advice on procurement CSO workshop on strategic planning for board members monitoring. of the Mongolian coalition. In sum, multi-stakeholder coalitions have been Nigeria formed in Uganda, Mongolia, Nigeria and other Increased interaction between government and countries that allow CSOs, private sector and CSOs also occurred in Nigeria. In July 2012, the government to work collaboratively on OC. These Nigeria Bureau of Public Procurement, part of the coalitions are forming strategies, including how to Nigerian Contract Monitoring Coalition, provided engage in coordinating access to information, policy publications and learning materials for participants advocacy and contract monitoring. at the training workshop in Abuja. [21] The willing- ness to supply this information and promote moni- Outcome Area 3: Improved policy efficiency toring shows the government’s commitment to the and responsiveness in countries coalition and to enhanced transparency. In September 2012, the Federal High Court in See figure 3 for a map of the following outcomes. Abuja ruled favorably on a suit brought by the Public and Private Development Centre, national conve- Mongolia ner of the Nigerian Contract Monitoring Coalition, Over the summer of 2012, the Public Procurement to demand the disclosure and information on the Partnership engaged in a policy dialogue with the World Bank-supported contract for the supply and Mongolian Ministry of Finance on the Implementing installation of High Voltage Distribution systems in Rules and Regulations of the newly amended Public Abuja, Lagos and Ibadan. [23, 24] In November, the Procurement Law. [15] This was the first time that Power Holding Company of Nigeria released partial CSOs working on procurement in Mongolia advo- documents. [25] cated for regulatory changes as a joint network with Then in March 2013, they were ordered to release a united message. As a result of this dialogue, in fall all contract documents sought by the coalition. In 2012 the Mongolian Ministry of Finance finalized the addition, the judge ordered the Power Holding Implementing Rules and Regulations of the newly Company and the Attorney General of the Fed- amended law, and included several of the partner- eration to “jointly and severally” pay costs to the ship’s requests, such as allowances for monitors and coalition to cover its legal fees [31].These results use of specific reporting templates. Under the new established a legal precedent supporting access to law all contracts in Mongolia must be monitored, information and demonstrate the power of diverse 64 substantiation of outcomes To verify the accuracy of the outcomes mapped and enrich WBI’s understanding of them, the external consultant selected five outcomes [1, 5, 12, 16, 26] and asked 15 people independent of WBI but knowledgeable about the change to review each and record whether they agree with the outcome as described. 12 people responded. Nine fully agreed with the description as formulated, and 3 provided additional information to describe the outcome. Regarding significance, 6 fully agreed with the description and 6 provided additional information to describe the significance. For the contribution of WBI, 7 fully agreed with the description and 3 provided additional information to describe the contribution. Excerpts of the substantiators’ comments on the outcomes achieved: “The current design of the Open Contracting Partnership was due to a collaborative process involving all stakeholders.” —Christian Poortman, Chair, Construction Sector Transparency Initiative “The key outcome of note is that WBI was able to co-create a vision and strategy for the Open Contracting project with a diverse group of partners.” —Joe McCarron, Partner, Reos Partners “WBI has worked with the coalition on this exciting journey. They have provided advice and experiences from different countries and even helped the coalition to make breakthroughs in places where initial difficulties were experienced.” —Gilbert Sendugwa, Coordinator/Head of Secretariat, Africa Freedom of Information Centre “Amidst an opaque governance system, the ministry of education [in Uganda] was quite open and willing to work with civil society. This is unlike most government departments, which do not want to get close to civil society.” —Bashir Twesigye, Chair, Uganda Contracts Monitoring Coalition, Civic Response on Environment and Development “I hope the information provided to the university will be easily accessible to all without any financial commitment since they now in most cases recoup all their costs in their work. I also think that this information is relayed to stakeholders so that the information can be accessed by all.” —Florence Dennis, Executive Secretary, Ghana Anti-Corruption Coalition “The training provided to the University of Ghana has already borne fruits as the officers involved have attended meetings of the Oil Platform and explained to civil society members how to use the web platform for advocacy information and demand accountability from duty bearers.” —Amin Mohammed Adam, Executive Director, IBIS Ghana/Publish What You Pay stakeholders working as a coalition to promote which are leading to new and more effective contract transparency and accountability in public contract- monitoring practices among CSOs and government, ing. jointly. Coalitions are also demanding the disclosure WBI provided coaching that built the capacity of of previously unavailable information on contracts, coalition members to engage with the government setting new legal precedent (for example in Nigeria). for improved access to information. WBI provided funding, coaching and network-building support to Outcome Area 4: Improved conditions for the coalition in partnership with the Africa region of non-state actors to participate in open the World Bank, which allowed them to engage in contracting monitoring of this project and strengthened their capacity to act when they were refused the procure- See figure 3 for a map of the following outcomes. ment records and information. In sum, country coalitions are affecting policy Uganda improvement to ensure contracting data to work on During the summer of 2012, a subgroup of the OC. For example, the Mongolian coalition engaged Uganda Contract Monitoring Coalition with with government to influence policy amendments, experience in education developed a tool with 65 input from the Ministry of Education that will enable mance makes the coalition a more capable partner community members to monitor the construction to engage with government ministries. of schools that have been vulnerable to corruption. WBI provided funding, coaching and network- [11] The tool will help the coalition collect contract building support to the coalition together with other performance information that may reveal whether a coalitions in West and East Africa, in partnership with project is executed according to quality standards the Africa region of the World Bank. WBI’s OC com- and in compliance with Uganda laws. The data can munity of practice is a partner in the Procurement provide feedback to government and citizens on Portal Observatory. project performance, and be used for advocacy and policy dialogue with government agencies. Ghana The Procurement and Construction Units in the In a separate development in West Africa, in July Ministry of Education of Uganda agreed to the moni- 2012, the University of Ghana took on ownership toring of contract performance in the construction and maintenance responsibilities for an innovative of secondary schools around the country. [12]. The Geographic Information System (GIS)-based map. International Development Association task team For the first time, this resource combines all publicly provided the coalition a list of over 250 schools with available extractive industries data for Ghana in ongoing construction from which the coalition will one place, including mining and oil field locations, select a sample. [32] If the results of this initial moni- contracts, production and revenue data, corporate toring by the coalition are useful, other ministries social responsibility projects and underlying socio- might agree to support further monitoring. economic indicators. [26] Local ownership of this WBI provided seed funding, shared monitoring platform, with no financial support from WBI, dem- tools from other countries and reviewed the draft onstrates the effectiveness of WBI’s capacity build- monitoring tool. ing efforts and successful technical skills. This also presents an opportunity for enhanced knowledge Nigeria sharing both within Ghana and regionally, in terms of In a July 2012 training workshop in Abuja, the the mapped data and presentation tool. Nigerian Contract Monitoring Coalition built skills of Subsequently, CSOs, including the oil/gas plat- civil society representatives and the members of the form of civil society groups, have been using this Nigerian Society of Engineers to monitor power sec- map for information to monitor extractive industries tor procurement processes, from project conception in Ghana with a range of stakeholders from gov- to contract award to project implementation. [19] ernment officials to parliamentarians tracking the Also in the summer of 2012, the Nigerian coali- contract information together with complementary tion built the knowledge of its observers on how to datasets. [27] Civil society groups reference the use the 2007 Public Procurement Act, 2011 Freedom GIS-based map as an example of how to provide of Information Act and procurement monitoring information for non-technical people (such as those tools developed by the Public and Private Develop- dealing with advocacy) who might not be able to ment Centre to report on the procurement process locate or understand technical information available through the Procurement Portal Observatory. [20] through other channels. Policymakers also said they By August, through an expert committee set up found the map useful because it pulled together at the Nigerian Society of Engineers, the coalition datasets from different ministries and agencies in a had developed a standard Infrastructure Rating and user-friendly portal. The common knowledge base Assessment Template for monitoring project imple- the GIS-map provides, and the discussions and mentation and contract performance. [22] actions it produces, should lead to more transparent The use of the template for monitoring will, and accountable contracting practices in extractive similar to the Uganda monitoring tool for school industries. construction, enable the coalition to collect contract WBI created the GIS-based map and supported performance information to provide feedback to the the University of Ghana by providing training for government when infrastructure projects, especially ownership and upkeep of the map. roads, are not meeting the expectations of citizens. In sum, country coalitions are engaging with This improved ability to understand contract perfor- government and CSOs to develop the knowledge 66 and tools to monitor the performance of contracts and participation. Further, in these countries and in focus sectors including education (for example, in Uganda, the outcomes exemplify an increase of Uganda) and infrastructure (for example, in Nigeria). non-state actors’ participation in public contracting. In the case of Ghana, the University has become In Ghana the results demonstrate how country actors an OC champion by making accessible all publicly gained ownership in the implementation of OC prac- available extractive industries data for use by CSO tices in the key sector of extractive industries. groups. These are key steps to enhance the use of It is premature to expect more notable progress contracts data as a public resource to enable citizen on the development objectives in part due to the participation in and feedback on contracting pro- newness of the OC initiative and other challenges. cesses. Although the steering group is working on framing a common understanding and global convergence, Conclusion and has made important strides with civil society in The OC initiative has made progress in opening gov- particular, government and private sector uptake ernment contracting to public scrutiny and participa- has been slow. OC principles and standards, while tion at the global and country level, of which the four an important step, are still in their infancy and need countries described are examples. This is a sample to be further integrated into country operations, of a larger body of work currently ongoing in more with dissemination of good practices to improve the than 30 countries worldwide. The multi-stakeholder quality and effectiveness of engagement. global steering group promotes and spreads the OC Similarly, OC will continue its efforts to be effort, enhancing uptake across organizations and integrated into the OGP, EITI, CoST and other sectors in different contexts around the world. The relevant global and sectoral initiatives. Support to group’s formalizing and promotion of OC principles OC coalitions and effective engagement of multiple and standards is increasing momentum to combat stakeholder groups are still not on a solid footing, corruption and inefficiencies. so continued efforts are needed and are also being In Mongolia and Nigeria, the outcomes demon- planned in this area. strate increased disclosure of contracts and open- ing of government contracting to public scrutiny for more information Acknowledgments Project Contact Thanks to Robert Hunja, Marcela Rozo, Michael Jarvis, Sara Danish, Ousmane Marcela Rozo, WBI Open Governance Deme, Felipe Estefan, Kathrin Frauscher, Norma Garza, Carey Kluttz, Lindsey practice, mrozo@worldbank.org Marchessault, Julia Mensah and Caroline Spruill. Email Thanks to substantiators: Amin Mohammed Adam, Executive Director, IBIS Ghana/Publish What You Pay WBI Capacity Development and Results Florence Dennis, Executive Secretary, Ghana Anti-Corruption Coalition team at capacity4change@worldbank.org Ian Gary, Senior Policy Manager, Extractive Industries, Oxfam America Website Joe McCarron, Partner, Reos Partners www.worldbank.org/capacity Grace Munanura, Senior Procurement Specialist, AFTPE, World Bank Norovjav Otgonjargal, Procurement Development Center © Copyright 2013 World Bank Christian Poortman, Chair, Construction Sector Transparency Initiative WBI’s Capacity Development and Results Michael Roesch, Depute Head of Programme: Public Finance, Administrative team led the outcome mapping; Jenny Reform and Transparency Initiatives, GIZ Gold coordinated the exercise with Claire Schouten, Programme Director, Network for Integrity in Reconstruction, support from Ricardo Wilson-Grau. Integrity Action Gilbert Sendugwa, Coordinator/Head of Secretariat, Africa Freedom of Sharon Fisher provided editorial and Information Centre design services. Samuel Otoo provided Nadine Stiller, Project Officer, GIZ overall guidance. Bashir Twesigye, Chair, Uganda Contract Monitoring Coalition, Civic Response on Photo by Arne Hoel, World Bank Environment and Development 67 Next Steps real-time monitoring and evidence gathering from complex development processes that involve multiple stakeholders. It WBI’s OC team plans a series of measures to further is based on a similar concept of locally driven change from the build on and institutionalize the achieved outcomes, Capacity Development and Results Framework. The tool was moving toward a wholesale model to allow OC to customized to gather information on outcomes—and related increasingly spread independently of WBI support. outputs and milestones—to learn from what changed, for whom, when and where, the significance of the change and how the There are plans to formalize and expand relation- program contributed to each change. ships with World Bank Group operations, linking task team leaders to coalitions to ensure their sustainabil- 3 A similar outcome mapping exercise has been developed that ity, and engaging with new partners that can multiply is specific to Pharmaceutical Procurement and Supply Chain Management work. OC through their own networks. WBI’s OC team will also focus on improving the 4 The numbers in brackets correspond to the outcomes in Figures OC community of practice, drawing on technology 2 and 3. The text that usually follows each outcome refers to its significance. The process of change the outcomes represent is to spread knowledge and support documentation seen in Figure 1. and outreach. Products could include how-to guides on contract monitoring and coalition building, data 5 Change agents are leaders, groups or organizations from standards and principles, and resources to facilitate government or non-state that drive change. their rollout and implementation. 6 Change strategy refers to how change happened to advance In addition, the OCP plans to achieve the fol- progress toward the development objectives—the development lowing measures in the coming year to strengthen problems addressed, types of outcomes achieved, WBI contributions, and
partners involved. A change strategy may country, global and steering group level outcomes: include different types of
change processes or outcome areas • Mechanisms in two countries; expansion of OC depending on the complexity of the multi-actor institutional learning products and online community of prac- changes involved in a program. tice; and mainstreaming of OC in multilateral 7 The Open Government Partnership (OGP) is a new multilateral organizations and existing initiatives such as OGP initiative that aims to secure concrete commitments from and EITI. governments to promote transparency, empower citizens, • At the current steering group level: formalization fight corruption, and harness new technologies to strengthen of the OCP steering group and advisory groups; governance. In the spirit of multi-stakeholder collaboration, OGP is overseen by a steering committee of governments and civil external core funding for OC strategies and work society organizations. OGP’s membership includes more than 50 plan; and a new OC secretariat host. n countries. 8 The Major Contract Awards for Bank-funded Operations dataset covers those contracts awarded from fiscal year 2007 to date NOTES under World Bank projects, which have been subject to prior 1 review by the Bank. Mapping outcomes—and related outputs and milestones—can help us learn from change processes that occur during program delivery that often seem complex and opaque because they involve multiple actors and address large development problems. An outcome is what each social actor (or change agent) did, or is doing, that reflects a significant change in their behavior, relationships, activities, actions, policies or practice. The program may influence these changes, directly or indirectly, partially or wholly, intended or not. Outcomes are identified at two levels in relation to the goal: institutional changes relate to societal, policy and organizational changes; and learning/capacity changes relate to awareness, knowledge or skills, collaborative action, or the use of knowledge or innovative solutions. These levels are based on the Capacity Development and Results Framework. The framework provides a systematic yet flexible approach to designing capacity development strategies and programs, monitoring and adaptively managing interventions, and evaluating and learning from their results. 2 Outcome harvesting is a practical assessment tool from the outcome mapping community of practice. It can be used for 68 Strengthening the Capacities of Local Governments in South East Europe W ith support from the World Bank Institute maps present the sequence of outcomes achieved (WBI) and the government of Austria, by change agents—the leaders, coalitions and cities in South East Europe (SEE) once organizations involved in the program. The maps alienated by ethnic strife now exchange ideas on illustrate how the outcomes connected and built how to improve performance. The Austria Urban on each other over time to form multi-actor, institu- Partnership Program (UPP), a joint effort of WBI and tional processes for change to address the program’s the Austrian Government, brings cities in the region objectives and goal. together by enabling multi-stakeholder dialogue WBI team members identified and formulated and providing practical self-assessment tools to help the outcomes, presenting an explanation of their locally elected officials, city administrators, technical significance and how WBI had contributed—directly staff, and citizen groups to conduct rigorous analysis, or indirectly, in a small or big way, intentionally or improve urban management practices, and prepare not—by empowering the change agents to take new bankable project financing proposals. actions. Then, roughly 20% of the outcomes were To generate evidence of how WBI has contrib- independently substantiated for credibility. uted to strengthen the capacities of local govern- ments in SEE countries, in April–July 2013, WBI Background mapped outcomes1 from this program using a Two decades ago, the SEE region was engulfed customized outcome harvesting tool2. The visual in a devastating civil war that rocked Europe and 69 reshuffled its political economy and geographic Development Objective boundaries. Today the region is at peace, but many More livable and sustainable cities in South East Europe cities operate at well below their full potential. One that provide a high quality of life for citizens. reason for this is the long learning curve of local Problem self-government. Some of the countries were formed Despite efforts to bring economic and democratic only recently. Cities such as Pristina or Podgorica reforms, city governments in South East Europe remain only became capitals in the last decade. In other beset by decentralization status, weak local institutions, shifts in political structures and slow reforms. Corruption cases, towns and cities acquired new and greater remains one of the significant worries and mayor responsibilities stemming from a process of adminis- priorities on the national and local level. City leaders trative, political, and fiscal decentralization launched are challenged to meet rising citizen demands for in the late 1990s. services and face severe inefficiencies in urban land Socioeconomic and structural trends are also use that block economic development. Moreover, city revenues are highly vulnerable to central policy changes occurring, including the transformation to a service- and economic crisis. Local governments have limited based economy and a population that is highly experience and weak networks and strategies to address urbanized but declining in numbers. With the high these problems. concentration of heavy industry located in urban Specific Objectives agglomerations, many cities suffer from the decline • Devise effective, responsive, and participatory city- of industrial production, leading to high unemploy- owned strategies to address problems of corruption, urban land development, and revenue management ment, environmental contamination, and underused and increase commitment of municipal champions to urban space. Cities struggle to maintain municipal find innovative solutions to address problems in their infrastructure that used to be subsidized and was context. designed to serve more businesses and people. • Improve accountability and transparency of municipalities in meeting citizen demands for services. Capacity shortfalls in planning infrastructure and • Improve the cities’ financial health and design service delivery hinder efforts to meet performance efficient policies within cities to address expanding standards aligned with European Union (EU) pre- urbanization. accession requirements. Property rights to both • Promote effective regional networks for learning public and private land and buildings are often around solutions and inter-agency communication and collaboration, and strengthen participation from civil inconsistent. Some local governments do not know society organizations, private sector and citizens in the extent of their assets. At the same time, there is instituting reforms. increasing competition for investment among coun- tries and cities in the region, combined with a severe fiscal crisis affecting governments at all levels. Chal- elected officials, city administrators and technical lenges remain in meeting EU pre-accession direc- staff, and in some cases citizen groups, to diag- tives, including environmental directives, as well as in nose their municipalities and learn how to map key absorbing EU pre-accession and structural funding. sources of funding from donors. While central governments try to address these challenges by controlling aspects of urban planning, Outcome Areas serious capacity gaps and weak citizen involvement All of the outcomes were analyzed according to leave urban governments vulnerable to corrup- the types of change they achieved. They were then tion. This is especially true with regard to managing grouped based on how they connected to each public land and property, issuing building permits, other to form a change strategy. The change strategy as well as planning, implementing, and monitoring from this program can be seen in three outcome public investments in infrastructure. areas that represent the major change paths that the The WBI Urban program equips city leaders, program has influenced to date, as seen in Figure 1: municipal staff, and other practitioners with innova- (1) Commitment to find innovative solutions to drive tive strategies and tools to maximize the potential change towards livable cities; (2) Effective, participa- benefits of urbanization. The UPP aims to foster tory, and accountable city-owned strategies; and (3) communication between cities, once alienated by Strengthened regional network. ethnic conflict, marked by City-to-City dialogues The maps in Figures 2–53 summarize the held in the region. Practical tools assist locally outcomes and their numbers correspond to the 70 Figure 1. Change strategy showing how change happened to advance progress toward goal WBI Contributions Municipal Finance and Urban Change Agents4 Planning • City-to-City dialogues, • Mayors and local government leaders from cities in Mayors’ Dialogue South East Europe • Planning methodology • Heads of municipal finance and urban planning and self-assessment tools departments • Guidance in formulating • Local finance and urban experts city improvement action • Local government associations plans between the • NALAS workshops • Certified anti-corruption practitioners • Support to Network of • Guiding coalitions in cities Local Authorities (NALAS) • CSOs/NGOs task forces Anti-corruption • Coaching through the certified local anti- corruption practitioners Change Strategy5 • Participative methodology Outcome Area 1: Commitment to find Social Accountability innovative solutions to drive change toward • Guided Focus Group livable cities Problems Addressed Discussions, In-Depth • City champions became more aware of • City revenues are Interviews, and problems, options, and more willing to respond vulnerable to policy change Participatory Scenario collaboratively and openly to address them and recession Development workshops • City leaders committed resources to form • Outdated expenditure in each country innovative strategies and plans to address a practices in local • Regional dissemination reform with private sector, citizens, CSOs, others government conference • Ineffective local services, Outcome Area 2: Effective, participatory and such as public education, accountable city-owned strategies utility services and • City staff from different agencies built their municipal solid waste know-how to analyze problems, and to management Partners transparently present the findings to the public • Inefficient land use • Austrian government for accountable decision-making regulations block economic • NALAS • Guiding coalition agreed on reforms to take development • Local government forward through participatory process • Weak inter-agency associations • Cities communicated and implemented communication and • Partner cities/ strategies that included new tools and solutions problem-solving within and municipalities to respond to problems with stakeholder support among local governments • Romanian Partners to address challenges Outcome Area 3: Strengthened regional • Vulnerability to corruption Foundation for Local network in municipal services Development • Dialogue networked cities to exchange • Inefficiency of current social • Co-Plan, Albania challenges, good practices, and lessons accountability mechanisms • Urban planners • NALAS built regional skills to collect, review, and to address most vulnerable • Private developers disseminate data on cities • Weak CSO, private sector, • CSOs/NGOs • NALAS formed regional networks of expert citizen participation in • World Bank practitioners to support city strategies instituting reforms • NALAS Secretariat and task forces became regional partner to guide city learning and Integrated World Bank exchange as well as communicate information Activities for policymaking and benchmarking • World Bank Europe and Development Objective Central Asia Sustainable Cities Initiative • More livable and sustainable cities in South • Social sustainability East Europe that provide agenda of the World Bank a high quality of life for citizens 71 numbers in brackets within the text. The sidebars important to involve marginalized groups with no near maps 2–5 list example indicators to show how legal regulation that ensured their representation on a monitoring process could track progress toward the local councils. Overall, the process built a readi- each objective. The indicators are not predefined; ness among local government representatives to they were generated based on the observed change engage more and listen to citizen needs. process. Contribution of WBI: The UPP issued a public call to recruit focus group discussion participants to Outcome Area 1: Commitment to find facilitate cooperation with local grassroots NGOs/ innovative solutions to drive change toward CSOs that work with vulnerable communities, and livable cities selected municipal representatives for interviews. The UPP worked with a local team to create ques- Pilot I: Improving Social Accountability and tionnaires for the sessions, guide discussions with Sustainability Mechanisms in SEE cities (See individual groups, interviewed local representatives, Figure 2) and encouraged an open space to discuss issues. In late 2012, more than 500 government and civil Over October-November 2012 five Participatory society stakeholders from five cities in Macedonia, Scenario Development sessions were organized Kosovo, Bosnia and Herzegovina, and Albania vol- in the five pilot cities for representatives of differ- unteered to debate issues about vulnerable groups, ent organizations and stakeholders. The workshops public services, accountability, deficiencies in ser- incorporated two activities: presentations of research vice provision and social inclusion. [1] Stakeholders findings and working group sessions to refine the from Banja Luka (Bosnia and Herzegovina), Durres participatory scenarios in each city. This was part of (Albania), Sarajevo (Bosnia and Herzegovina), Skopje a broader consultation process that led to develop- (Macedonia) and Prishtina (Kosovo) then came up ment of city action plans. with possible solutions to reduce corruption and increase a sense of moral responsibility among pub- Pilot II: Improving the viability of the cities lic employees in their local governments. [2] financially and spatially (See Figure 3) At the same time, over 30 representatives from In November 2011, finance representatives from 10 local government, central government, and the busi- municipalities in SEE countries prepared essays iden- ness sector in the cities agreed to public interviews tifying issues and possible solutions to challenges to share thoughts and discuss priority areas. [3] This of local government revenue management. [11] This lead to over 230 representatives from local govern- demonstrated the municipalities’ active interest and ment in the cities, including the city administra- willingness to act in improving their financial health. tion, chamber of commerce, citizens, NGOs/CSOs, It also showed openness to new ideas and solutions unemployed, and other stakeholders to come up to address their financial challenges. In early 2012, with proposals to strengthen organizational systems mayors from 20 municipalities of seven SEE countries and institutional integrity, efficiency, transparency, endorsed a participatory process to assess cities’ and social accountability in each city. [4] The process financial health and committed their municipalities’ provided for a transparent Social Accountability human resources. Their municipal teams then started Assessment in each city to collect proposals that collaboration toward financial Municipal Finance represented the wider community about what needs Self-Assessments (MFSA) with local experts [12]. to be changed, how and opportunities for citizen Later the same year, four additional municipalities engagement and participation in implementing the made the same commitment [13]. changes. In 2013, the mayors of two new municipalities, Previously these priority areas were not sufficiently Bar and Cetinje (Montenegro), committed to use the discussed publicly with citizens, and there was lim- MFSA to assess their cities’ financial health, bringing ited awareness of problems, options and willingness the total to 26 municipalities6 involved in the MFSA. to respond collaboratively and openly to address [14] The political endorsements show that munici- them. These actions also helped to overcome a lack pal leaders are ready to commit to new, innova- of trust in local government being open to citizen tive solutions to improve their financial health, and feedback on solutions. The process was especially recognized the potential importance of the MFSA 72 Figure 2. Outcomes Showing Improvement in Social Accountability and Sustainability Cities’ Dialogue Regional Dialogue Cities’ Actions Focus Group (2) Over 500 Participatory Regional (6) Skopje, Macedonia proposed Discussions government Scenario Dissemination an action plan for improving (1) Over 500 and civil society Development (5) Seventy access to information, e-services government stakeholders (4) Over 230 participants and improved feedback and civil society devised representatives from the SEE mechanisms by local government stakeholders possible from local region gathered in 5 cities in solutions to government in to discuss Macedonia, Bosnia increase social the same cities, findings of the 5 (7) Ilidza, Bosnia and Herzegovina and Herzegovina, accountability including the city cities engaged developed a citizen feedback Kosovo, and and sense administration, in the Social system to improve management Albania debated of moral chamber of Accountability of the municipality and strengthen issues about responsibility commerce, Assessment and the partnership between the local vulnerable groups, among citizens, CSOs, how to respond, authority and citizens public services, employees unemployed, considering good accountability, and in local and other practices from social inclusion governments stakeholders, other SEE cities (8) Pristina, Kosovo put forward devised proposals a proposal to develop citizen’s to strengthen charter and calendar of activities In-Depth Interviews to promote charter. organizational (3) Representatives from local systems and government, central government, institutional and the business sector in the cities (9) Durres, Albania agreed integrity, efficiency, agreed to public interviews to share to set up website to facilitate transparency thoughts and discuss priority areas engagement with residents and and social accountability in municipal departments each city (10) Banja Luka, Bosnia and Herzegovina agreed to e-dignity plan for the elderly in rural area of Banja Luka city 2012 2013 Institutional changes Learning/capacity changes Outcomes related to societal and organizational changes. Other outcomes related to awareness, Municipal commitment, participatory decisions, transparent knowledge or skills, collaborative action, or information sharing to improve social accountability mechanisms the use of knowledge or innovative solutions on the part of SEE cities Responsiveness of municipalities to put in place innovative mechanisms for citizen accountability Example of Indicators Measuring Changes Specific Objective Improve accountability and transparency of municipalities in meeting citizen demands for services. Outcome Increased stakeholder participation in development of action plan Indicators Number of individuals participating in dialogue Type of organization/representative present (Y/N): ❑❑ Citizen ❑❑ Private sector ❑❑ CSO ❑❑ Unemployed ❑❑ Local government ❑❑ Elderly ❑❑ Central government ❑❑ Youth ❑❑ Chamber of Commerce 73 tool to improve the mobilization of local resources, in addressing corruption and presented strategies public spending, public asset management and in Zagreb during the regional conference. [36] In maintenance, investment programming, and access December 2013, 70 mayors, deputy mayors, mem- to external financing. Municipal demand to use the bers of city councils, NGOs, local government MFSA to understand cities’ financial health and use associations and municipalities from phase 1 shared the findings to shape solutions continues to grow. lessons in addressing corruption and presented The experience of municipalities applying for the strategies in Ljubljana during the regional confer- MFSA also inspired 15 municipalities7 in 2012 and ence. [37] The experience of these municipalities 2013 to start using an Urban Planning Self-Assess- raised interest among other municipalities and led ment (UPSA). [15,16] Cities can use the findings to new coalition building between mayors of Zenica compare their performance against other cities in the (Bosnia and Herzegovina) with Indjija (Serbia), and region, qualitatively and quantitatively. This should Vinica (Macedonia) with Indjia (Serbia). enable them to establish a common ground for dis- Contribution of WBI: The UPP hosted the ARWs cussions on how to modernize spatial planning and to disseminate the anti-corruption methodology, land management, support local economic develop- inspired by the successful experience in treating and ment and ultimately improve quality of life. preventing corruption in La Paz, Bolivia and initially Contribution of WBI: The UPP influenced these incubated and field tested by FPDL in collaboration outcomes by organizing seven City-to-City dia- with WBI in the person of Ronald MacLean-Abaroa. logues in 2012 and 2013 to facilitate dialogue and The methodology was fine-tuned and scaled-up in peer learning. The UPP also customized the MFSA South East Europe under the World Bank-Austria and UPSA tools to suit the needs of countries from Urban Partnership Program. The mayors/public the South East Europe region, with feedback from managers from Albania, Bosnia and Herzegovina, municipal representatives, and promoted examples Croatia, and Serbia received WBI’s capacity support of municipal experiences using the tools to under- in 2012 in applying the participatory anti-corruption stand problems and identify solutions. UPP also methodology, while Kosovo, Macedonia and Monte- helped put together a network of international negro received capacity support in 2013. and national municipal finance experts who were engaged to work in seven countries on the MFSA. Outcome Area 2: Effective, participatory, and accountable city-owned strategies Pilot III: Addressing corruption and modernizing local governments (See Figure 4) Pilot I: Improving Social Accountability and Between 2012 and 2013, around 170 mayors, city Sustainability Mechanisms in SEE cities (See representatives, NGOs, and local government asso- Figure 2) ciations from three countries volunteered to under- In March 2013, 70 participants from five cities of stand methods to address corruption. [21] In 2012, the SEE region gathered to discuss the findings of 23 municipalities (during phases 1 and 2) and seven their Social Accountability Assessments and how to countries applied for the program. [22] These munici- respond, considering good practices from other SEE palities exercised new leadership to address corrup- cities. [5] This regional dialogue offered an oppor- tion by committing to the anti-corruption process. tunity for teams from the five pilot cities8 to use the The process was expected to help make citizens feedback from stakeholders to formulate responsive aware how they can contribute to fighting corrup- action plans. tion, improve and increase the level of public confi- • The city of Skopje, Macedonia proposed an dence in municipal government, as well as increase action plan for improving access to information, the level of motivation of employees for increased e-services and improved feedback mechanisms efficiency in their work, and attract new investors and by local government. [6] This plan potentially will EU funds. improve services at the local level and overcome By December 2012, 65 mayors, deputy mayors, the information gap that exists in reaching vulner- members of city councils, NGOs, local government able groups. associations and representatives from municipali- • The municipality of Ilidza (Sarajevo), Bosnia and ties shared their experiences and lessons learned Herzegovina developed (and is now implement- 74 Figure 3. Outcomes Showing Improvement in the Viability of the Cities Financially and Spatially City-to-City Dialogues—MFSA City-to-City Dialogues—UPSA (11) Municipal (12) Mayors from (13) Four more (15) Ten municipalities from (16) Five more finance repre- 20 municipalities municipalities SEE decided to engage municipalities joined the sentatives from of 7 SEE countries joined the in UPSA, inspired to join UPSA; this led to a total 10 municipalities endorsed process in during third City-to-City of 15 that formulated in SEE coun- participatory the second dialogue action plans tries sent an process to assess half of 2012 essay identifying cities’ financial (Budva from issues and pos- health and Montenegro sible solutions committed their and Belgrade, Cities to challenges in municipalities’ Subotica and revenue man- human resources. Sabac from (19) Twenty-four (14) Mayors of new agement. Municipal teams Serbia). Total municipalities from 7 municipalities (Bar & started collaboration number of countries in SEE prepared Cetinje, Montenegro) toward financial municipalities financial projections, and decided to join MFSA self-assessment with working on 12 developed action plans local experts. MFSA by the presented at Mayors’ end of 2012 was Dialogue in Vienna 24. (17) (18) Municipal (20) Municipality Municipal finance of Gazi Baba Municipal Finance Self- finance teams representatives in Macedonia Assessment—MFSA improved identified and carried out pilot their skills solved problems of urban audit Land Use and Urban Planning while working during self- to follow up Self-Assessment—UPSA on the assessment success story of process process MFSA 2011 2012 2013 Institutional changes Learning/capacity changes Outcomes related to societal and organizational changes. Other outcomes related to awareness, Political commitment of municipalities to improve their financial knowledge or skills, collaborative action, or the health and address urban development barriers use of knowledge or innovative solutions. Effective strategies and plans in municipalities to address corruption vulnerabilities and urbanization and adaptively solve problems in their context Example of Indicators Measuring Changes Specific Objective Improve the cities’ financial health and design efficient strategies within cities to address expanding urbanization. Regional Outcome Increased commitment of municipalities to improving their financial health Indicators ❑❑ Number of mayors endorsing participatory process to assess cities’ financial health ❑❑ Number of municipalities completing MFSA with local experts ❑❑ Number of municipalities completing UPSA 75 ing) the citizen feedback system “Center 72” Action Plans, which were presented at the Mayors’ based on good practice from Indjija, Serbia (Sys- Dialogue in Vienna. [19] Through the self-assess- tem 48). Citizens can report a communal problem ment process of the MFSA the municipal finance at any time, every day, and within 72 hours there teams identified and solved problems and improved will be answer on the status of problem-solving. their skills to apply the process. [17,18] The MFSA This system will be used for better management process helped municipalities analyze their own of the municipality and strengthening the partner- performances and formulate reform plans, as well as ship between local authority and citizens.[7] pave the way toward benchmarking and institution- • The city of Pristina, Kosovo put forward a proposal alizing rigorous diagnosis to improve their municipal to develop a citizen charter and calendar of management practices in a transparent manner. It activities to promote the charter, which should helped them set in motion processes that require lead to enhancement of communication and municipal finance departments, urban planners, participation of all stakeholders in the city. [8] NGOs, private real estate developers, representa- • The city of Durres, Albania agreed to set up a tives of ministries of finance, ministries of spatial website to facilitate direct engagement with planning, and local government associations to work residents and municipal departments, which together on reforms. responds to feedback from citizens and other As a follow-up to their success with the MFSA, interested groups. [9]. This new website will pro- the municipality of Gazi Baba in Macedonia carried mote a more collaborative governance approach. out a pilot of an urban audit to assess their infra- • The city of Banja Luka, Bosnia and Herzegovina structure and urban services for more efficient devel- agreed to an e-dignity plan for the elderly in the opment of local self-government. [20] The MFSA rural area of the city. [10] This initiative is expected tools helped the municipality plan solutions to dif- to increase information technology literacy ficult problems. There was a new understanding that among the city’s rural population, strengthen a similar collaborative, diagnostic approach might capacities of rural councils, improve information help the municipality develop the skills, experience, access in rural areas, and enhance rural citizen and networks to overcome other key challenges. participation and feedback. Contribution of WBI: The UPP launched the Contribution of WBI: The UPP organized the MFSA process and later conducted seven multi- workshop to help bring the cities together and stakeholder City-to-City dialogues in Budapest, invited guest speakers from the SEE region and Hungary; Mavrovo, Macedonia; Tirana, Albania; beyond to share their good practices in collaborative Budva, Montenegro; Vienna, Austria, Dubrovnik, government, including: Pula/Croatia (e-governance); Croatia; and Skopje, Macedonia. The City-to-City Moldova (BOOST), and Indjija, Serbia and Strumica, dialogues were accompanied by discussions on Macedonia (System 48). The workshop provided an MFSA progress in participating municipalities, and opportunity for the five cities to present findings of guidance on the way forward. Upon request, UPP their Social Accountability Assessments and show- offered technical capacity support to the municipali- case their proposed projects developed as part of ties throughout the process of self-assessment— the assessment to address issues in their respective local experts who provided onsite training and cities. The local teams and city representatives pre- assistance. sented the proposals, the latter having participated in scenario development activities and committed to Pilot III: Addressing corruption and selected initiatives. modernizing local governments (See Figure 4) In early 2012, six teams of anti-corruption Pilot II: Improving the viability of the cities practitioners from non-government successfully financially and spatially and effective regional finalized a one-year Program for Anti-corruption networks for learning around solutions (See Practitioners to serve as local practitioners to Figure 3) support municipalities. [23] In phase 1 in 2012, four By January 2013, a total of 24 municipalities from municipalities—Kucova, Albania; Zenica, Bosnia and seven countries in the SEE region prepared financial Herzegovina; Krizevci, Croatia; and Boljevac, Serbia projections, and 12 developed Finance Improvement initiated processes to work with the anti-corruption 76 Figure 4. Outcomes Showing the Address of Corruption and Modernizination of Governments Cities (30) Municipal Council (31) Mayor of Zenica in Zenica in Bosnia and committed matching (26) Four (27) Four municipal Herzegovina approved funds and additional municipal guiding coalitions first strategic plan for resources to anti- guiding coalitions agreed on areas to anti-corruption corruption process and conducted in- address corruption requested support to depth diagnoses vulnerabilities develop action plan of municipal organizations’ vulnerability to (28) Four municipali- (24) In phase (32) Municipal Council members corruption ties finalized anti-cor- I, 4 municipali- of opposition and leading party in ruption strategic plans ties initiated Kucova, Albania for the first time as a result of 8-month process to jointly voted in favor to approve process led by mayors work with (25) The 4 strategic plan for anti-corruption and guiding coalitions anti-corruption municipalities practitioners to created guiding coalition to (29) In phase II, 6 additional (33) Mayor of Krizevci, Croatia ap- identify vulner- develop anti- municipalities initiated process proved complete strategic and action abilities to cor- corruption to work with anti-corruption plan to improve funding of processes ruption in local government strategy practitioners and coalitions and strategies to identify vulnerabilities to (34) Municipal Council in Bolje- for addressing corruption in local government vac, Serbia approved strategic them. and develop strategies plan for anti-corruption Regional Dialogue (35) Six municipalities (Gostivar & Vinica, (36) 65 mayors, deputy mayors, mem- Macedonia; Herceg Novi & Tivat, Monte- (21) During the (23) Six teams (22) Mayors from bers of city councils, negro; Gjakova and Istog, Kosovo) course of 2012 and from non- 23 municipalities NGOs, local govern- finalized anti-corruption strategic plans 2013, about 170 may- government expressed ment associations as a result of 8-month process led by ors, city representa- built skill to interest and and municipalities mayors and guiding coalitions tives, NGOs and serve as local local government committed from phase I shared anti-corruption (37) 70 mayors, deputy mayors, members of city associations from resources to lessons in address- practitioners councils, NGOs, local government associations 3 countries volun- work on anti- ing corruption and to support and municipalities from phase I shared lessons teered to understand corruption presented strategies municipalities in addressing corruption and presented strate- methods to address process in Zagreb during the corruption regional conference gies in Ljubljana during the regional conference 2012 2013 Institutional changes Learning/capacity changes Outcomes related to societal, policy and organizational changes. Other outcomes related to awareness, Municipal commitment, transparent and participatory knowledge or skills, collaborative action, or the decision-making on corruption use of knowledge or innovative solutions. Policy/strategy changes for anti-corruption Effectiveness of municipalities to address corruption vulnerabilities Example of Indicators Measuring Changes Specific Objective Address corruption and modernize local governments Regional Outcome Increased municipal commitment to transparent, participatory decision-making on corruption Indicators For region: ❑❑ Number of cities with formally approved strategic plan for anti-corruption ❑❑ Number of cities committing funds and additional resources to support anti-corruption process For municipality: ❑❑ Approval of strategic plan for anti-corruption [Yes/No] ❑❑ Level of matching funds and/or in-kind resources allocated to anti-corruption process 77 practitioners to identify vulnerabilities to corruption result of 8-month process led by mayors and guiding in local government and strategies for addressing coalitions. [35] them. [24] The process enabled the municipalities to focus The municipalities signed a memorandum of on those activities most vulnerable to corruption and commitment endorsing collaboration between the with the greatest impact on citizens’ quality of life. It municipality and anti-corruption practitioners and also provided for a strategic plan with solutions that created a guiding coalition to develop an anti- are generated and owned within local government corruption strategy. [25] This ensured the personal drawing on non-government knowledge and input. involvement of the mayor in the process from begin- Zenica’s experience demonstrates the important of ning to end, appointment of up to 60 municipal staff the mayor owning the plan to ensure resources are as coalition members responsible for implement- mobilized for its implementation. ing the anti-corruption strategy, and provision of Contribution of WBI: The Romanian NGO, logistical support and space for meetings. It was the Partners Foundation for Local Development (FPDL) first time that the municipalities collaborated with pioneered the Program for Anti-corruption Prac- non-government to form an anti-corruption strategy. titioners. WBI’s UPP contributed the skill-building Then, the coalitions conducted in-depth diagnoses component for which it invited international experts of their municipal organizations’ vulnerability to cor- to share their experiences. At the end of the one- ruption, discussed solutions, and worked on choos- year program, WBI engaged the best anti-corruption ing priority issues. [26] Then, the coalitions used the practitioner teams to work with willing public lead- findings to agree on high-priority areas to address ers/managers selected through competitive pro- corruption vulnerabilities. [27] cesses, as well as to share experience and network. The participatory nature of the diagnosis pro- Throughout 2012, the UPP offered in-depth vided for the inclusion of different perspectives technical support and coaching/mentoring to the and broad agreement on priorities to address. By municipalities in the program through the certified November 2012, the four municipalities finalized local anti-corruption practitioners in the “Cities with- Anti-Corruption Strategic Plans as a result of an out Corruption—Cities with Future” Program. UPP eight-month participatory process led by mayors and supported training and paid the practitioners who their guiding coalitions. [28] In phase II, six additional supported the municipality throughout the process. municipalities of Istog and Gjakova, Kosovo; Gos- In December 2012, the UPP organized a regional tivar and Vinica, Macedonia; and Tivat and Herceg conference in Zagreb, Croatia to raise the interest of Novi, Montenegro initiated a similar process to iden- additional cities in joining the program. tify their vulnerabilities to corruption and strategies The UPP also introduced an anti-corruption to address them. [29] planning methodology that implies a strategic and Leading from phase I, in 2013, the Munici- participatory process conducted inside local govern- pal Council of Zenica in Bosnia and Herzegovina ments.9 The process follows the strategic planning approved its first Strategic Anti-corruption Plan, and steps, from diagnosis to elaborating solutions and the mayor of Zenica committed matching funds as implementation. It focuses on changing corrupt well as municipal staff, space, and logistical support organizational systems—not only corrupt individu- to implement the plan [30, 31] In Kucova, Albania als—and supports mayors to act as institutional the Municipal Council members of opposition and reformers rather than judges or prosecutors. the leading party for the first time voted jointly to approve the Strategic Anti-corruption Plan for the Outcome Area 3: Strengthened Regional city. [32] In Krizevci, Croatia, the Mayor approved a Network (See Figure 5) more complete strategic plan to fund anti-corruption In December 2011, the Network of Local Authorities processes. [33] The Municipal Council in Boljevac, (NALAS)8 and local government associations from Serbia also approved a new strategic plan for anti- seven SEE countries (Albania, Bosnia and Herze- corruption. [34] In November and December 2013 six govina, Croatia, Kosovo, Macedonia, Montenegro municipalities (Gostivar & Vinica, Macedonia; Herceg and Serbia) began collaboration with local experts Novi & Tivat, Montenegro; Gjakova and Istog, to work on collection and management of fiscal Kosovo) finalized anti-corruption strategic plans as a information on local governments for a Municipal 78 substantiation of outcomes To verify the accuracy of the outcomes mapped and enrich WBI’s understanding of them, an external consultant selected eight and asked 16 people independent of WBI but knowledgeable about the change to review each and record whether they agree with the outcome as described. Fourteen people responded, at least one per outcome. Some excerpts of the substantiators’ comments: “Public finances and the management of public finances (planning, execution and control), represents one of the most important elements in the implementation of the city’s strategic plans and programs. Therefore the involvement of Cetinje in the MFSA program is important because it will contribute to the acquisition of new knowledge in the area of better public finances management in all phases. The World Bank Institute provides a full contribution by informing the stakeholders about possibilities of participation in various programs. In that context, the Institute, through adequately providing information to the Old Royal Capital Cetinje, influenced our decision to join the previously mentioned process.” —Aleksandar Bogdanovic, Mayor of Cetinje, Montenegro “The municipality of Cetinje has been informed about the similar processes in regard to MFSA that has been successfully completed by the Municipality of Budva and the Capital City of Podgorica and they wanted to follow the same path. They are very interested in taking part at the City-to-City dialogue to establish communication with countries from the region. Cetinje believes they would be able to compare the performance data with neighbouring municipalities of Budva and Podgorica and to use the MFSA data to better advocate to the State for the grants program.” —Natasha Obradovic, municipal finance expert, Montenegro “I fully agree that the series of City-to-City dialogue series were important, but equally important was (i) the dedicated support from the local expert fully understanding the tool and its applicability; and (ii) commitment from the municipal staff. The selection of participating municipalities also deserves attention.” —Bjorn Phillip, Senior Urban Specialist, World Bank “The MFSA provides us with a clear and integrated picture about our financial situation and budget structure and measures we have to undertake as soon as possible to keep financial stability and sustainability of the local economic development of MGB. Also, the MFSA tool gives a more transparent and integrated management approach that could use all members of the municipality top management, which was not a case before. It breaks the sector’s working barriers, improving the team approach, and most important, increases the responsibility of all key players related to the financial stability of the LSG in structuring their sector programs within the budget structure. It is the first step in providing anti- corruptive behavior of all municipal staff and management. Previously this was obligation only of the urban department staff, which was a very risky approach.” —Sasho Trajkov, Head of Local Economic Development, Gazi Baba Municipality (city of Skopje), Macedonia “This is new and unique experience for them [municipalities], giving them the chance to be active players in the diagnosis/identification and the solution development phases, tailored in line with their specific LG needs and challenges. Working with municipalities on the ground gives opportunities for in-depth situation analysis that corresponds to the actual needs of the municipalities concerned. Bearing in mind the complexity of the undertaking, an in-depth capacity building with embedded training has been also envisaged to increase knowledge and skills on leadership, organization and change management—all important pillars for more efficient and effective implementation of the efforts/strategies to fight corruption.” —Ljupka Simonoska, local anti-corruption practitioner “Apart from financial commitment, the mayor of Zenica has shown to be visionary on curing and preventing corruption, understanding its severe implications if not dealt with, as well as seeing and seizing an opportunity to deal with it within the project Cities without Corruption. In our encounters he would always underline the wish to create a sustainable system, which would stay in place no matter who the mayor is. He was also pleased with the used approach and methodology. Apart from the mayor’s significance in the process I would like to mention the importance of the key senior staff, which not only committed at the political level, but at the operational level, making the implementation successful and smooth.” —Emina Abrahamsdotter, local anti-corruption practitioner “After the Skoplje Workshop, the municipality of Ilidza adopted some changes to their action plan to incorporate good practices they heard about at the workshop. The Sarajevo Coordinator and CPI Foundation contacted the municipality representatives and proposed developing a project proposal for this changed action plan (now titled Center 72) and to apply for the community boost challenge. The CPI Foundation and the municipality developed the community boost proposal and were selected as a finalist and also participated in the Code Spint, which was part of the aforementioned challenge. More information about the improved action plan is at http://communityboostr.org/node/237.“ —Elma Demir, local team coordinator, Ilidza, Sarajevo 79 Figure 5. Outcomes Showing Effective Regional Networks for Learning Around Solutions Regional Network (38) NALAS and local govern- (39) By May 2012 NALAS was ment associations from 7 SEE established as a main Program countries collaborated with local partner sharing the common experts to work on collection agenda, especially towards and management of fiscal infor- (42) NALAS shared Municial Finance Review, MFSA mation on local governments for their Report on Fiscal and UPSA a Municipal Finance Review Decentralization in South East Europe during City-to- (40) NALAS supported City-to-City (41) NALAS decided to City dialogue dialogues through engagement of actively engage in WBI task force members and network of e-learning products local representatives on different and help disseminate topics to actively participate in operational knowledge with workshops and skill building practical, how-to guidance 2011 2012 2013 Institutional changes Learning/capacity changes Outcomes related to organizational changes. Other outcomes related to awareness, knowledge or Effectiveness of NALAS to support municipal learning skills, collaborative action, or the use of knowledge or and communicate information to guide reforms innovative solutions. Example of Indicators Measuring Changes Specific Objective Promote effective regional networks for learning around solutions and interagency communication and collaboration Regional Outcome Increased collaboration among local municipal authorities, local government associations, and local experts on the collection and management of fiscal information on local governments Indicators ❑❑ Number of local government associations working with NALAS and local experts to conduct a Municipal Finance Review ❑❑ Number of City-to-City dialogues supported by NALAS Finance Review (MFR). [38] This collaboration to By May 2012, NALAS established itself as a main conduct the MFR was significant in that it had not WB program partner sharing the common agenda, happened previously. especially toward MFR, MFSA, and UPSA. [39] This The MFR had two main objectives: to support partnership is important given NALAS’ regional improved policymaking at the national level by agenda to work on knowledge management to equipping local government associations with develop and promote decentralization in the SEE disaggregated municipal finance data to assist region, and that NALAS finds tools such as MFR, in the fiscal planning and budget negotiations MFSA, and UPSA useful to support existing efforts. process; and to assist in increasing local financial NALAS started to support City-to-City dialogues management and service delivery performance through engaging task force members and networks by identifying options for efficiency improvement of local representatives on different topics to partici- and comparison of local budget expenditures and pate in workshops and skill building [40]. It decided revenues. The process was expected to improve to engage in e-learning products to help dissemi- understanding of fiscal information and its use in nate operational knowledge with practical, how-to the local governments, and improve the quality guidance. [41] During the City-to-City dialogue on and consistency of key municipal finance data for “Modernizing Local Public Expenditure Manage- effective evidence-based policymaking. ment” held in May 2012 in Mavrovo, Macedonia, 80 NALAS presented their Report on Fiscal Decentral- were humiliating. They did not understand the con- ization in South East Europe. [42] The report gave text and offered solutions that were ready-made. But an overview of expenditures in the countries par- this program has a completely different approach— ticipating in the City-to-City dialogues. This rein- this was evident from the start. … In this case they forced NALAS’ regional role in improving municipal offered a methodology: when you apply this you finances and facilitating the use of the MFR. get an assessment, a self-assessment. You find your Contribution of WBI: The UPP engaged NALAS own way toward a solution. You feel you are creating Task Force members on fiscal decentralization in your own society. You know if people are ready for the MFSA and MFR processes as well as members change. You know the people.” of the NALAS Secretariat. They invited them to • Social accountability increased citizen engage- learning events and fostered knowledge sharing on ment in pilot municipalities toward joint solutions the regional level. In October 2012, WBI provided (proposals elaborated together with municipal funding to NALAS to support the initiative, tasked staff, citizens, CSOs), and learning from exchange among other things with managing the aggregate of experiences from others in the region who had data at the regional level, promoting the program, some of the systems in place. and contributing to daily activities. Parallel to the In the future, change agents will still need to first City-to-City dialogue and the MFSA launch, UPP address some problems, be they new or leftover, sponsored eight local experts as technical support such as strengthening dialogue between local com- to the MFR process and assessed available informa- munities, national governments and NGOs with the tion on municipal finances. business sector and promoting sustainable invest- ments and corporate engagement; developing the Conclusion skills of local and national officials; improving the The outcomes achieved so far have helped advance capacity of public administrations to strengthen the specific development objectives: rule of law and reduce corruption; and dealing with • The self-assessment and innovative diagnostic frequent shifts in political structure. tools addressed the objective to improve cit- NALAS and local government associations are ies’ financial health and design efficient policies key to catalyzing change and sensitizing local gov- within cities to address expanding urbanization. ernments to the existence and use of relevant tools. It did so by helping participating municipalities The UPP is working toward enabling and strengthen- create a clearer picture of their financial situation, ing their capacities as well as the capacities of local analyze their performance and formulate reform experts and anti-corruption practitioners to help with plans, as well as pave the way toward bench- scale up and sustainability. marking and institutionalization of diagnosis to Also, the expanding interest of other municipali- improve municipal management practices in a ties showed there might be interest in institutional- transparent way. It helped them set in motion izing the tools at the national level, for instance, the processes that require people from different MFSA, anti-corruption methodology, urban audit. agencies to work together and pave the way to reform. Next Steps • The Urban Governance and Anti-corruption Initia- There are several strategies in place to build on tive addressed the objective to devise effective, outcomes achieved so far that could help bring this responsive, and participatory city-owned strate- process to scale: gies to address problems of corruption, improve • Organize regional and national meetings and transparency and accountability. It did so by sup- conduct presentations on innovative diagnostic porting cities to create ownership when working tools during assemblies of local government on diagnoses and toward strategies to address associations, NEXPO, to the Presidency, commit- corruption, improving collaboration among tees, etc. in order to foster city peer learning and departments within the municipality and sharing connectivity. experiences through regional conferences. • Provide further guidance and instructions for Mayor of Zenica, Bosnia and Herzegovina: “After NALAS and local government associations the war, donors offered all types of projects; some involved in eLearning development and 81 2 Outcome harvesting is a practical assessment tool from the outcome dissemination since this is a fairly new way of mapping community of practice for real-time monitoring and evidence knowledge transfer in SEE. gathering from complex development processes that involve multiple • Strengthen dialogue between local governments, stakeholders. It is based on a similar concept of locally driven change from the Capacity Development and Results Framework. The tool was national associations of local authorities, and min- customized to gather information on outcomes—and related outputs and istries, which are key to catalyzing change. milestones—to learn from what changed, for whom, when and where, the significance of the change and how the program contributed. In addition, new outcomes are expected over the 3 The numbers in brackets correspond to the outcomes in Figures 2–5. The coming 1–2 years. In the area of diagnostic tools, text that usually follows each outcome refers to its significance. The process WB team will assist (through additional Austrian of change the outcomes represent is seen in Figure 1. 4 Change agents are leaders, groups or organizations from government or funding, WB lending operations, or EU program) the non-state that drive change. scale up and institutionalization of MFSA and urban 5 Change strategy refers to how change happened to advance progress to- audits. Knowledge products will be produced and ward the development objectives—the development problems addressed, disseminated at the Mayors Symposium in February types of outcomes achieved, WBI contri-butions, and
partners involved. A change strategy may include different types of
change processes or 2014 in Vienna, Austria (City-to-City proceedings; outcome areas depending on the complexity of the multi-actor institutional MFSA brochure; urban audit template customized changes involved in a program. 6 Municipalities involved in the MFSA during 2011–2013: Berat, Kamza, for SEE region) Shkodra, Tirana (Albania); Prijedor, Novo Sarajevo, Banja Luka (Bosnia and In the area of anti-corruption, three municipalities Herzegovina); Crikvenica, Rijeka (Croatia); Istog, Peja, Suhareke, Gjakova (Kucova, Albania; Zenica, Bosnia and Herzegovina; (Kosovo); Bogovinje, Gostivar, Gazi Baba, Strumica (Macedonia); Bijelo Polje, Budva, Niksic, Bar, Cetinje, Podgorica (Montenegro); Sabac, Bel- Boljevac, Serbia) will elaborate detailed action plans grade, Subotica (Serbia). by the end of January 2014. 7 Municipalities involved in the UPSA during 2012–2013: Berat (Albania); In the area of social accountability, the municipal- Kanton Sarajevo, Banja Luka (Bosnia and Herzegovina); Tkon, Rijeka (Croa- tia); Peja, Istog (Kosovo); Gazi Baba, Gostivar, Bitola, Tetovo (Macedonia) ity of Ilidza (Sarajevo) is implementing the citizen Tivat (Montenegro) Belgrade, Nis, Novi Sad (Serbia). feedback program “Center 72,” with another munici- 8 Municipal representatives, NGOs, and business representatives from pality also planning to implement this system. n five pilot cities in SEE: Banja Luka and Sarajevo, Bosnia and Herzegovina; Durres, Albania; Pristina, Kosovo; and Skopje, Macedonia. In addition, there were others from the SEE region and beyond: Pula, Croatia; Indjija, NOTES Serbia; Moldova, representatives from local government associations, and from NALAS (described below), among others. 1 Mapping outcomes—and related outputs and milestones—can help us 9 The methodology was initially developed by Partners Foundation for Lo- learn from change processes that occur during program delivery that often seem complex and opaque because they involve multiple actors and ad- cal Development, with inputs from WBI, and later refined and scaled up in dress large development problems. An outcome is what each social actor South East Europe with UPP support. (or change agent) did, or is doing, that reflects a significant change in their 10 NALAS is a network of associations of local authorities of South East behavior, relationships, activities, actions, policies or practice. The program Europe. The network brings together 15 associations that represent about may influence these changes, directly or indirectly, partially or wholly, in- 9,000 local authorities, directly elected by more than 80 million citizens of tended or not. Outcomes are identified at two levels in relation to the goal: this region. NALAS promotes decentralization in cooperation with central institutional changes relate to societal, policy and organizational changes; governments and international organizations, considering local self- and learning/capacity changes relate to awareness, knowledge or skills, government a key issue in the current transition affecting various countries collaborative action, or the use of knowledge or innovative solutions. These in South East Europe. NALAS builds partnerships to contribute to the levels are based on the Capacity Development and Results Framework. reconciliation and stabilization process within the region and henceforth The framework provides a systematic yet flexible approach to designing contributes to European integration of the whole region. capacity development strategies and programs, monitoring and adaptively managing interventions, and evaluating and learning from their results. for more information Acknowledgments Project Contacts Thanks to Tamara Nikolic and Sabine Palmreuther of the From the WBI Urban Practice team: WBI Urban Practice team involved in the mapping exercise. Sabine Palmreuther, spalmreuther@worldbank.org Thanks to our partners, from the National Associations of Local Tamara Nikolic, tnikolic@worldbank.org Governments, NALAS, NGOs, anti-corruption practitioners, local experts and our clients from all seven countries, whose Email engagement made this program a success. WBI Capacity Development and Results team at capacity4change@worldbank.org WBI’s Capacity Development and Results team led the outcome mapping; Jenny Gold coordinated the exercise with support from Website Ricardo Wilson-Grau. Sharon Fisher provided editing and design. www.worldbank.org/capacity Dawn Roberts provided support. Samuel Otoo provided overall © Copyright 2014 World Bank guidance. 82 Strengthening Implementation of Legislation on Access to Information across Latin America S ince 2011, the founding and rapid develop- Development Objective ment of a Latin America network of public Improve service delivery for citizens across Latin America. agencies responsible for overseeing access to information (ATI) has provided its mem- Problem bers with the knowledge and tools to better imple- Rising demands on public services—combined with limited information around policies and outcomes—has ment legislation that improves accountability in ser- led many Latin American countries to adopt access to vice delivery for the region’s citizens. The improved information legislation and create oversight agencies capacity of these oversight bodies was borne out to guarantee citizens’ right to information. However, in of a development initiative begun by Chile’s ATI early stages these agencies have limited experience to oversight body and supported by the WBI’s Access implement, monitor and enforce these laws. to Information Program. Specific Objectives In January–March 2013, WBI mapped the out- Increase effectiveness of access to information oversight comes1 of this initiative (Figure 1) using a custom- bodies to advance transparency and access in the ized outcome mapping tool2. WBI team members region, which will contribute to greater accountability identified and formulated the outcomes, presenting in service delivery. an explanation of their significance and how WBI 83 Figure1. Map of outcomes showing changes linked and built over a three-year timeframe (1) ATI oversight (2) Chile ATI agency agencies in Mexico proposed online (18)* ATI agencies and Chile conducted mechanism to (11) Colombia’s in Brazil, Chile, peer exchanges to share knowledge ATI agency formally (17) Network agreed Mexico, Peru and strengthen their among other Latin to conduct first mystery expressed interest Uruguay used capacities for ATI American agencies shopper exercise in joining network exercise to identify implementation to assess ATI law reform needs in key compliance in countries sectors (3) Chile ATI agency conducted and published an assessment that identified (10)* Peru’s ATI agency requested (22) Open needs among the oversight bodies support from (16) Network decided Government network for on four priority areas Partnership invited (4)* Bolivia, Chile, network to lead Mexico, Peru and knowledge for action learning of exchange visit its working groups to panel to explore role Uruguay ATI agencies of oversight bodies formally established with Chilean advance ATI member advancing ATI Latin America network of ATI oversight bodies (19) Chile ATI agency, as (15) Network members group coordinator, published (5)* Network reached embed working groups first report on records (8) Brazilian agency in ATI agency structure management practices larger audience responsible for coordinating by publishing first for regional work to implementation and inform country reforms newsletter to share oversight of ATI law joined information on ATI (20) Chile ATI agency network organized seminar to across region (77 issues inform the debate on to date) (14) Network records management members (7) Network (9) Network (12) Network (13) approved reforms Network (6) Network launched members incorporated 2013–2015 members established revamped approved feedback strategic plan, (21) Chile National gave input strategic website to its bylaws from ensuring long- Audit Agency using to commonly guidelines, communicate practitioners term sustainability this information to owned mission and ATI issues among and CSOs for and commitment develop guide that 2013–2015 vision members and 2013–2015 from national to updates archives strategic larger audience strategic plan regional levels regulations plan process 2011 2012 2013 Institutional changes Learning/capacity changes Outcomes related to societal, policy and organizational changes Other outcomes related to awareness, knowledge or Commitment, participatory priority setting, transparency skills, collaborative action, or the use of knowledge or innovative solutions. Guidance to implement ATI policy Effectiveness of ATI network and agencies * Outcomes selected for substantiation; see sidebar. contributed—directly or indirectly, intentionally to guarantee citizens’ right to information. But given or not—by catalyzing or empowering the change the swift adoption of the legislation in the region, agents to take new actions. Then, roughly 20% of a gap between the regulatory framework and the the outcomes were independently substantiated for capacity of the countries to respond and implement credibility in this mapping exercise. these laws emerged. The new ATI oversight bodies, or already existing ones with new responsibilities, Background had limited experience in the implementation, over- Across Latin America during the last decade, there sight and enforcement of ATI laws. has been a rising demand for public services com- WBI facilitated bilateral peer exchanges through bined with limited information about policies and videoconferences in the early days of Chile’s ATI service delivery. This has led many countries to oversight body, founded in 2009. Through an Institu- adopt ATI legislation and create oversight bodies tional Development Fund (IDF) grant—in which WBI 84 Figure 2. Change strategy showing how change happened to advance progress toward goal WBI Contributions • Knowledge exchange and Change Agents3 videoconferences • ATI oversight agencies in Bolivia, Brazil, Chile, • Grant for knowledge Mexico, Peru and Uruguay exchanges • Latin America network of oversight agencies • Strategic guidance for network formation, organization and communications Change Strategy4 Outcome Area 1: Commitment of ATI Agencies • Realized commitment by establishing network of Partners ATI agencies across Latin America Problems Addressed • Committed to knowledge exchange among • Limited knowledge • Input of civil society countries to improve ATI and experience of ATI organizations to 2013–2015 strategic plan • Raised awareness of network as valuable to agencies to implement region in advancing ATI and enforce ATI • Lack of agreement on Outcome Area 2: Institutionalized Network and priorities to improve ATI Priorities implementation • Institutionalized communication, strategy actions for ATI agencies • Weak realization of ATI agency role in policy • Embedded working groups of network into implementation country ATI agencies • Enhanced networks and knowledge base with ATI agencies across Latin America Outcome Area 3: Innovation and Policy to Advance Transparency and ATI Development Objective • Improved guidance on record management policy and other areas • Improve service delivery for citizens across Latin • Improved knowledge to monitor, implement and America enforce ATI policy • Collaboratively identified action areas and reform needs worked closely with World Bank operations—WBI Outcome Areas provided support to Chile’s ATI oversight body to develop a regional exchange platform that evolved Outcome Area 1: Commitment of ATI into a regional network of ATI agencies. In less than Agencies three years, this network, with WBI’s support along In early 2011, the Chilean ATI oversight body, with other partners, has made considerable progress Consejo para la Transparencia (CPLT), and its Mexi- in strengthening the capacity of its member agen- can homologue, the Instituto Federal de Acceso cies around ATI implementation. a la Información y Protección de Datos of Mexico, The process of change can be seen in three conducted peer exchanges to learn about what streams of outcomes (Figure 2) that are detailed in might work to strengthen their agencies’ abilities the following sections. These outcomes were ana- to meet their ATI implementation and oversight lyzed and classified according to the types of change responsibilities. [1]5 The CPLT also reached out to they achieved, then grouped based on how they Peru’s Defensoría del Pueblo, Bolivia’s Ministerio de connected and built on each other to affect change. Transparencia Institucional y Lucha contra la Corrup- ción, Uruguay’s Unidad de Acceso a la Información Pública and Canada’s Information Commission with 85 a proposal that they join efforts through an online WBI coordinated with the RTA network and platform. [2] These initial outcomes were important other World Bank teams to arrange for funding for because after the adoption of ATI legislation in the Brazilian government representatives to attend a region, ATI oversight bodies had relatively recent conference on ATI in Chile and participate in RTA experience implementing these laws, and there was meetings. Likewise, funding was arranged for a limited knowledge sharing among peers. meeting of Peru’s and Chile´s oversight bodies in WBI encouraged this effort by arranging for a Peru. By enhancing RTA’s role as regional interlocutor World Bank IDF grant to CPLT to develop a regional for ATI oversight bodies in the region, WBI exchange platform that supported videoconference contributed to strengthening its image and value and face-to-face working meetings among these proposition. organizations. In terms of commitment, the establishment in just In March and early April 2011, CPLT conducted over a year of a growing network of ATI agencies and published a capacity needs assessment among across Latin America had focused on knowledge the six oversight bodies that identified priorities, exchange among countries to improve ATI. During needs and areas of collaboration. [3] A regional the process they raised awareness in the region network was seen as a mechanism to facilitate peer for advancing ATI and positioned RTA as a leading exchanges and knowledge sharing tools, learn about network on ATI issues. what worked elsewhere and identify possible solu- tions to strengthen ATI implementation in individual Outcome Area 2: Institutionalized Network countries. On April 19, 2011, Bolivia, Chile, Mexico, and Priorities Peru and Uruguay established the Red de Transpar- In July 2011, the RTA network began publishing a encia y Acceso a la Información (RTA) network to biweekly e-bulletin, the first of its kind. [5] Then in build the operational and management capacity of September, they launched a revamped website, in ATI oversight bodies in the region. [4] which the oversight bodies post information and WBI provided advice on the development of the reports relevant to their countries, various news assessment tool and shared information on network- articles and information about ATI meetings. [7] In ing models used by other regional initiatives. the past some ATI information was shared among In April 2012, although Bolivia’s participation a small group, but the website and newsletter are in RTA was limited, the Contraloría General de la communication tools aimed to regularly disseminate Unión, the Brazilian government agency responsible diverse information to a larger audience. In early for coordinating the implementation and oversight 2013, the newsletter had 134 subscribers who also of ATI law, joined the network. [8] As the largest disseminate among their local networks. country WBI advised and shared information to include and economy in Latin America, Brazil’s membership in the newsletter and provided input on the content in RTA gave greater weight and leverage to the and design of the website. network. At their second meeting in September 2011, the Also in April 2012, Peru’s ATI oversight body five members of the RTA network agreed on their deepened its collaboration with Chile through a mission, vision and strategic guidelines and orga- knowledge exchange; in October 2012 a report nized working groups. [6] This formalization of the was published on the experience of both oversight network was vital for RTA’s proposed role as an inter- bodies. [10] For a long time Mexico’s ATI oversight locutor on ATI vis-à-vis other regional bodies such body was the sole source for sharing ATI knowledge as civil society networks, multi-lateral agencies and in Latin America, but by 2012 Chile’s agency had the international community. In April 2012, the five positioned itself as another source of knowledge for network members approved statutes and selected peers, which lead to a richer knowledge base in the Mexico as chair and Chile as executive secretary. [9] region. In November 2012, Colombia’s Secretaría Chile’s ATI agency developed the network’s strategic de Transparencia formally expressed interest in plan in July in consultation with peers and civil soci- joining RTA, indicating more growing interest in the ety practitioners in the five countries. [12] network. [11] WBI provided strategic guidance through a con- sultant and facilitated videoconferences. 86 substantiation of outcomes To verify the accuracy of the outcomes and enrich WBI’s understanding of them, the external consultant selected four outcomes [4, 5, 10 and 18] and asked 11 people who are independent of WBI but knowledgeable about the change to review each and record whether they agree with the outcome as described. 10 people responded. All 10 “fully agreed” with the descriptions as formulated; 1 substantiator “partially agreed” with the significance of an outcome and two substantiators “partially agreed” with WBI’s contribution to two other outcomes. Excerpts of the substantiators’ comments on the outcomes achieved: “The RTA is becoming a reality because it speaks to our aspiration to develop and fully explore the principles of transparency and the right to access information in our respective countries, reflecting our shared interest in generating a learning process around specific network issues and operations.” —Eduardo Gonzalez-Yañez, Director for Operations and Systems, Consejo para la Transparencia Chile “It’s important to note that cooperation among the members of the ATI network contributes directly to the work of persuading and overseeing the state administration, as the Office of the Ombudsman does in Peru.” —Fernando Castañeda, Deputy on Constitutional Affairs Defensoría del Pueblo de Peru “In Mexico, the IFAI has found the ‘mystery shopper’ technique to be very useful since the start of its operations as the agency responsible for upholding the Transparency Act, and to this day it continues to emphasize using the technique as a mechanism for evaluating the attention given to the requests from the public and application of the process.” —Gabriela Segovia, General Director for Access to Information Policies Coordination, Instituto Federal de Acceso a la Información y Protección de Datos of Mexico “WBI not only shared information on the various network models but also participated actively in their creation—dialoguing with the various stakeholders, offering input and advice on the analysis, which defined the network’s scope, and consistently supporting the network in its early stages. It not only arranged for an IDF grant; it gave totally ‘demand-driven’ support, consistently fostering its sustainability above and beyond the assistance offered by the Bank.” —Ana Bellver, Senior Public Sector Specialist, World Bank In August 2012, Chile’s ATI agency conducted a network—they have embedded working groups of survey among network members and other stake- the network into their own agencies and are thus holders to inform the implementation of RTA’s becoming national networks. [15] 2013–2015 strategic plan [13], enhancing RTA’s legitimacy with its stakeholders. In September, the Outcome Area 3: Innovation and Policy to RTA network approved the 2013–2015 strategic plan Advance Transparency and ATI with the overarching objective of strengthening the Throughout 2012, the RTA network continued to position of the network at the regional level. [14] By consolidate its operation and reach outwards. defining clear objectives and a three-year strategic The 2013–2015 strategic plan included concrete joint work program, the network bolstered its long- projects for the four thematic working groups with term sustainability and reinforced commitment from responsibility for documenting and disseminating the national to the regional levels. good practices on records management, gathering WBI contributed to the design of the survey. WBI jurisprudence criteria on ATI, promoting knowledge contributed indirectly to results of the RTA network on good practices and lessons learned on capacity process by providing guidance on working groups, building and developing ATI indicators. [16] resources to hold face-to-face and videoconference In October 2012, Chile’s agency, as coordinator meetings and facilitation of these meetings. of the network, published the first report on records Building the RTA network represented prog- management practices of the five RTA members ress in institutionalizing communication, strategy that also included information from similar bodies and actions for the six ATI agencies. The individual in Australia, France and the United Kingdom. [19] agencies have gone beyond participating in the Chile’s agency subsequently used the information to 87 RTA organize a seminar that proposed Conclusion reforms of records management. The RTA was originally [20] Informed by this report, Chile’s envisioned to create National Audit Agency is develop- an online knowledge ing a guide that updates existent sharing mechanism and archives regulations. And the Min- evolved to become a regional network ister Secretary-General will set up of oversight bodies. The activities carried by a working group to promote legal the network are now embedded within the inter- improvements to the archives regula- nal structures of the ATI oversight bodies, which tions. [21] ensures continuity and further development. Also, WBI facilitated an IDF grant that provided sup- by establishing thematic working groups, the RTA port for the initial stages of the records management goes one step ahead—finding innovative solutions report. that address knowledge gaps and implementation Between December 2012 and February 2013, the weaknesses through products tailored to support ATI five ATI oversight bodies of Brazil, Chile, Mexico, in the region. Peru and Uruguay conducted a mystery shopper The RTA has grown to become an emerging exercise to test compliance regarding information independent network governed by its own members. access in health, education, taxes and housing. The It builds the capacity of its members by facilitating exercise aimed to assess obstacles, compliance with dialogue, collaboration and knowledge exchanges deadlines and the accuracy of information disclosed among peers. The initial assessment helped to by government agencies. [17,18] Implementing a align members’ expectations, assess compara- common ATI measuring tool across the region would tive strengths and weaknesses and identify shared generate indicators that identify gaps and weak- interest and concern for undertaking joint work. The nesses at the national level of the different targeted participatory process adopted for the RTA network’s sectors. At the same time, other countries that institutionalization and planning process proved performed better in those areas could collaborate beneficial for generating ownership of RTA’s activities with peers to address those gaps and increase the by all members. capacity of agencies to respond. One of the most important lessons learned in WBI supported the RTA network by facilitating this initiative was the importance of sharing two sessions through videoconferences to discuss knowledge and experience to design collaborative methodology and protocols for the mystery shopper regional products that could help ATI implementa- exercise. tion at the national level—this has attracted the Finally, the regional organizers of the Open interest of other oversight institutions in the region. Government Partnership Regional Meeting held in The RTA network demonstrates how regional sup- Santiago, Chile on January 2013 invited the RTA net- port can impact national efforts. work to lead a panel to explore the role of oversight Further, the RTA has not only positioned itself in bodies advancing ATI within the partnership. [22] the transparency agenda at the national and regional RTA’s participation permitted ATI oversight bodies level, but it has also become a model to advance to showcase their role within the Open Government transparency and ATI in other agendas such as Open Partnership agenda and the implementation of its Government Partnership. The network is institution- action plans. The RTA network is being recognized alized in the member countries for knowledge shar- as a valuable interlocutor in the region to advance ing and access to peer knowledge for ATI authorities ATI, playing a key role among other stakeholders in and staff. Further, the network will be carrying various various agendas. activities throughout 2013 that go beyond the initial In sum, in 2012 the RTA network members col- support provided by the World Bank, which proves laboratively identified new action areas and reform the network has built its capacity for continuing this needs. Continuing into 2013, they began to advise initiative independently. others on areas such as records management policy and experimented with new ways for the region to monitor, implement and enforce ATI policy. 88 Next Steps reaching larger audiences by sharing products A World Bank IDF grant supported the RTA network widely and increasing visibility to the network’s initia- with operation costs and some product costs. Since tives and collaborating with regional civil society the grant has come to an end the network needs to networks. develop a funding strategy to ensure sustainability. As a result of the strategic plan, RTA members It has taken initial steps such as a membership-fee foresee the elaboration of a common transparency requirement established by its bylaws, and Chile’s indicator based on the results of the mystery ATI agency, as executive secretary, has agreed to shopper exercise and the development of a training cover operation costs for the next three years. The approach for oversight bodies based on best network still needs to develop a funding strategy practices. Moreover, the RTA has gained greater that will help guarantee long-term sustainability. weight with relation to other stakeholders, and it Another challenge for the network is the need to is expected that it will play a key role in advancing define an outreach strategy for oversight bodies in transparency and ATI not only in the region but also the region to become members. An emerging issue at the international level. that the network needs to further explore is how It is also expected that interest from peers in to collaborate with other horizontal accountability the region to join the network will grow as the RTA mechanisms to strengthen the overall accountability continues to gain experience and develop products framework. on ATI implementation. The network has developed In its last meeting, the RTA approved its working relations with international initiatives and 2013–2015 strategic plan, which included feedback organizations, incorporating new activities related to from members and civil society organizations and the Organization of American States’ model law on reflected the network´s plan to scale up not only by ATI, contributing to the Open Government Partner- identifying short-term projects but also by refining ship by undertaking an assessment of ATI initiatives long-term strategy. The strategic plan builds upon within the partnership’s country action plans and what has already been achieved and sets higher exploring potential opportunities for collaborating goals to advance transparency and ATI, including with civil society. n expanding the network to increase membership, for more information Acknowledgments Project Contact Thanks to Marcos Mendiburu, Luis Esquivel and Elsa Peraldi, WBI Social Marcos Mendiburu, WBI Social Accountability practice, and team members involved in the project and Accountability practice, mapping exercise. mmendiburu@worldbank.org Thanks to: Ana Bellver, Senior Public Sector Specialist, World Bank Email Valeria Bolla, Communications Associate, World Bank WBI Capacity Development and Results Fernando Castañeda, Deputy on Constitutional Affairs Defensoría del team at capacity4change@worldbank.org Pueblo de Peru Website Eolo Diaz-Tendero, Director for Studies, Consejo para la Transparencia Chile www.worldbank.org/capacity Mariana Gatti, Uruguay Legal Advisor, Agencia para el Desarrollo del de Gestión Electrónica, la Sociedad de la Información y del Conocimiento © Copyright 2013 World Bank Eduardo Gonzalez-Yañez, Director for Operations and Systems, Consejo la Transparencia Chile WBI’s Capacity Development and Results Katerina Kliwadenko, Coordinator RTA, Consejo para la Transparencia Chile team led the outcome mapping; Jenny Gabriela Segovia, General Director for Access to Information Policies Gold coordinated the exercise with Coordination, Instituto Federal de Acceso a la Información y Protección de support from Ricardo Wilson-Grau. Datos of Mexico Sharon Fisher provided editorial and Guadalupe Toscano, Public Sector Management Specialist, World Bank design services. Samuel Otoo provided Rosario Vasquez, International Affairs Director on Access to Information, overall guidance. Instituto Federal de Acceso a la Información y Protección de Datos of Mexico 89 NOTES 1 Mapping outcomes—and related outputs and milestones—can help us learn from change processes that occur during program delivery that often seem complex and opaque because they involve multiple actors and address large development problems. An outcome is what each social actor (or change agent) did, or is doing, that reflects a significant change in their behavior, relationships, activities, actions, policies or practice. The program may influence these changes, directly or indirectly, partially or wholly, intended or not. Outcomes are identified at two levels in relation to the goal: institutional changes relate to societal, policy and organizational changes; and learning/capacity changes relate to awareness, knowledge or skills, collaborative action, or the use of knowledge or innovative solutions. These levels are based on the Capacity Development and Results Framework. The framework provides a systematic yet flexible approach to designing capacity development strategies and programs, monitoring and adaptively managing interventions, and evaluating and learning from their results. 2 Outcome harvesting is a practical assessment tool from the outcome mapping community of practice. It can be used for real-time monitoring and evidence gathering from complex development processes that involve multiple stakeholders. It is based on a similar concept of locally driven change from the Capacity Development and Results Framework. The tool was customized to gather information on outcomes—and related outputs and milestones—to learn from what changed, for whom, when and where, the significance of the change and how the program contributed to each change. 3 Changeagents are leaders, groups or organizations from government or non-state that drive change.. 4 Change strategy refers to how change happened to advance progress toward the development objectives—the development problems addressed, types of outcomes achieved, WBI contributions, and
partners involved. A change strategy may include different types of
change processes or outcome areas depending on the complexity of the multi-actor institutional changes involved in a program. 5 The numbers in brackets correspond to the outcomes in Figure 1. The text that usually follows each outcome refers to its significance. The process of change the outcomes represent is seen in Figure 2. 90 Scaling up Capacity Development of City Officials and Practitioners across China Through eLearning S ince 2009, the use of eLearning by the gov- Development Objective ernment of China has provided knowledge Improve public service delivery in China’s rapidly to tens of thousands of city officials and growing urban areas, especially in secondary cities practitioners across the country on better and lagging regions. delivering and managing public services. The scaling Problem up of the capacity of these public officials was borne There is a high demand for skilled city officials out of a development initiative led by the Chinese and practitioners to provide services to the rapidly growing urban population in China; those in distant Academy of Governance and supported by WBI’s regions lack access to state-of-the-art learning Urban Practice and e-Institute for Development. opportunities. In January–March 2013, WBI mapped the out- Specific Objective comes1 of this initiative (Figure 1) using a custom- Improve the scale and effectiveness of learning for ized outcome mapping tool2. WBI team members city officials and practitioners in China. identified and formulated outcomes, presenting an 91 Figure1. Map of outcomes showing changes linked and built over a four-year timeframe (9) CAG (10) CAG piloted (1) CAG (3) CAG (11)* CAG developed urban planning committed allocated incorporated the urban planning eLearning among (20) CAG increased to scale up increased new course into eLearning course 120 officials and its experience to eLearning funding to their curriculum professionals manage, develop for public strengthen across China and deliver officials eLearning (8) CAG eLearning for public partnered with officials Urban Planning (12) CAG increased Society of China the interactivity of their (4) CAG to enhance its eLearning and know- (19) CAG is recognized by (2) CAG enhanced its technical content how to incorporate China’s central government established Distance network with technical content for their progress in Learning similar global (5) CAG built for public officials’ mainstreaming eLearning Division and regional skilled staff to learning for training public officials institutions manage and develop eLearning (13) CAG offered (17)* CAG offered about (18) CAG and its blended training 1,000 eLearning network institutions Institutional changes combining face-to- modules reached about 10,000 face and eLearning Outcomes related to societal, policy and public officials and for first time to 900 organizational changes practitioners with public officials eLearning Commitment to scale-up public officials learning (15) CAG issued Policy/strategy changes for scale-up first guidelines (16) CAG connected (14) CAG offered CAG’s effectiveness to manage and on eLearning for 22 of 47 provincial- support to its network of 451 level institutions deliver learning network on institutions to their eLearning eLearning Learning/capacity changes platform Other outcomes related to awareness, (6)* CAG actively participated (7)* CAG visited Monterrey knowledge or skills, collaborative action, or in dialogue with institutions Tech in Mexico to identify areas the use of knowledge or innovative solutions. advanced in eLearning for collaboration * Outcomes selected for substantiation; see sidebar. 2009 2010 2011 2012 explanation of how they had contributed—directly country through distance learning. After several years or indirectly, in a small or big way, intentionally or of delivering courses through videoconference, it not. Then, roughly 20% of the outcomes were inde- became clear that a more scalable as well as conve- pendently substantiated for credibility in this map- nient tool was needed to meet the overwhelming ping exercise. demand for professional training. The Chinese Acad- emy of Governance (CAG)—as the national institu- Background tion mandated to train mid-to-senior level public In China, rapid urbanization has posed tremendous officials—approached WBI for support as it had little challenges to policymakers and practitioners on experience with eLearning design and delivery. the frontline of providing essential public services In 2009, WBI’s Urban practice launched its China to residents of 657 cities and almost 20,000 towns. Urban Capacity Building Program with the goal of However, given China’s wide socio-economic dispar- contributing to improved public service delivery in ity, officials and practitioners in distant urban regions China’s rapidly growing urban areas, especially in had limited access to cutting-edge knowledge and secondary cities and lagging regions, in collabora- learning opportunities. tion with Chinese institutions such as CAG, Urban Since the 1990s, the Government of China has Planning Society of China, National Development placed increasing emphasis on improving the and Reform Commission and National Academy professional skills of government officials across the for Mayors. In three years, CAG, with WBI’s support 92 Figure 2. Change strategy showing how change happened to advance progress toward goal WBI Contributions Change Agents3 • International seminar • CAG • Workshops on eLearning • Distance Learning Division of CAG pedagogy and facilitation • CAG’s network of institutions • Pedagogical package of • China’s central government global course content • Technical advice • Knowledge exchange • Best practices Change Strategy4 Outcome Area 1: Increased Political Commitment to Public Officials Learning • Realized commitment by establishing distance Problems Addressed Partners learning division and allocating increased funding • City officials and • Committed to scale-up eLearning practitioners require • Develop courses with WBI and Urban Planning knowledge to deliver Outcome Area 2: Strengthened CAG’s quality public services Society of China Effectiveness to Provide Public Officials Learning • Knowledge exchanges • CAG had limited skills • Improved CAG’s operations to manage and with CEPT University in to develop, deliver and deliver public officials learning India, Ministry of Cities in manage eLearning • Enhanced staff knowledge and skills in eLearning Brazil and Monterrey Tech • Urban issues especially in Mexico • Enhanced networks with similar institutions needed to be addressed • Increased know-how to develop courses and in learning new curriculum that incorporate technical • CAG needed to achieve content relevant to public officials in China effectiveness and scale for public officials learning Outcome Area 3: Scaled-up Public Officials across China Learning • Achieved learning outcomes for thousands of public officials • Raised awareness of central government of CAG progress in eLearning Development Objective • Enhanced national network to deliver eLearning • Improve public service • Offered blended training combining face-to-face delivery in China’s growing and eLearning for first time; produced 1,000 urban areas modules along with other partners, has made progress in public officials. It began by setting up a new Distance improving the scale and effectiveness of learning for Learning Division. [1,2,3]5 public officials in China. CAG’s increasing emphasis on using eLearning as The process of change can be seen in three an effective and scalable learning tool was an impor- streams of outcomes (Figure 2) that are detailed tant step toward meeting the overwhelming demand in the following sections. These outcomes were for public official training to improve services for the analyzed and classified according to the types of rapidly growing urban population. change they achieved, then grouped based on how WBI contributed to these changes in a variety of they connected to each other to affect change. ways, all built on a long-term partnership with CAG begun in the 1990s. Because eLearning is still new Outcome AreaS to many institutions responsible for training officials in China, since 2008, at the request of CAG, WBI Outcome Area 1: Increased Political has been supporting CAG’s scaling up of eLearning Commitment to Learning through a variety of activities that combined train- In 2009, CAG committed itself to mainstreaming ing, knowledge exchange, technical advice and joint eLearning as one of the key modalities for training course development. 93 substantiation of outcomes To verify the accuracy of the outcomes mapped and enrich WBI’s understanding of them, the external consultant selected four outcomes [6, 7, 11 and 17] and asked 9 people independent of WBI but knowledgeable about the change to review each and record whether they agree with the outcome as described. 7 people responded. 6 fully agreed with the description as formulated of the outcome, 6 with its significance and all 7 with the contribution of WBI. Excerpts of substantiators’ comments on the outcomes achieved: “This is particularly important for a professional organization like UPSC that the knowledge is appropriately delivered to related decision-makers. We hope this online course can be a catalyst for mayors in China to learn more about urban issues and planning.” —Shi Nan, Secretary General, Urban Planning Society of China “I hope that this collaboration will demonstrate that wholesaling is a viable approach for e-Institute programs and that this will be replicated with other courses in both China and other countries.” —Philip E. Karp, Lead Specialist, World Bank EAP Development Effectiveness Unit “The jointly developed course not only helped CAG enrich its curriculum, but also enhanced the awareness of eLearning as an important training channel for civil servants.” —Manchuan Wang, Deputy Director General, Training Department, Chinese Academy of Governance Outcome Area 2: Strengthened date course content. So in 2011, CAG partnered with Effectiveness to Provide Learning the Urban Planning Society of China (UPSC) to bring In 2010, CAG extended its network with similar in technical expertise. [8] CAG, UPSC and WBI jointly urban institutions in other countries that were more developed a Chinese version of the e-Institute’s Sus- advanced in eLearning. In 2011–2012, the Distance tainable Urban Land Use Planning eLearning course. Learning Division participated in South-South knowl- [9] In November 2012, they piloted the course edge exchanges, such as with CEPT University in among 120 professionals and officials in charge of India and Ministry of Cities in Brazil, and sent a del- urban planning, land management and infrastructure egation to visit Monterrey Tech, a regional institution investment from 18 provinces across China. [10] CAG with a reputation and experience in offering eLearn- later incorporated the course into its eLearning cur- ing to public sector employees. [6,7] This interna- riculum and currently offers it twice a year. [11] tional collaboration and dialogue enabled CAG to WBI provided the pedagogical package with build a body of skilled staff in the Distance Learning multi-media based eLearning content of the global Division using the latest pedagogy and experiences. version, funding for translation and quality control of [4,5] content customization. They offered tailored techni- WBI and e-Institute supported CAG’s involve- cal advice on eLearning pedagogy, platform setup ment in several South-South knowledge exchange and administration as well as facilitation based on events to engage with urban institutions in other worldwide e-Institute experience and “best prac- countries with more advanced eLearning experi- tice” eLearning courses on urban topics. ence. This contributed to several important innova- The course was the first in CAG’s curriculum that tions undertaken by CAG in the next two years. In was developed and delivered as facilitated eLearn- addition, the WBI e-Institute conducted a series of ing following pedagogy and feedback mechanisms workshops to build the capacity of CAG and other adapted from international experience. It was also Chinese training institutions on eLearning pedagogy CAG’s first course delivered through open registra- and technical facilitation. tion by participants from both within and outside A topic of particular importance for the training CAG, reaching a wide variety of practitioners. curriculum was urban management, but CAG lacked Lastly, also in 2012, CAG improved the interac- in-house expertise to develop practical and up-to- tive functions of its eLearning platform, such as the 94 forums, as well as course content. [12] Forums are ity of those institutions to further scale up eLearning a key activity of a facilitated eLearning course, and training programs across China. the improved forum functions of CAG’s eLearning During 2012, CAG began offering blended train- platform enabled better and continued interaction ing (combining face-to-face classroom methods with among participants and facilitators. computer-mediated eLearning) to 900 mid-to-senior In sum, between 2011 and early 2013, CAG level public officials across China, for the first time increased its experience in managing, develop- incorporating eLearning courses into CAG’s formal ing and delivering eLearning for public officials to curriculum. [13] improve scale-up and effectiveness. [20] By the end of 2012, scale-up of eLearning had been advanced, with 1,000 eLearning modules [17] Outcome Area 3: Scaled-up Learning across offered through CAG’s eLearning platform that was China connected by 22 provincial-level administration insti- In 2012, with its enhanced capacity to design, tutions [16], benefiting tens of thousands of public manage, and deliver eLearning, CAG started offer- officials [18]. Today, the senior management of the ing support to its network of local institutions on Chinese government increasingly recognizes CAG’s eLearning platform design, course development and progress in mainstreaming eLearning for training delivery. [14] public officials. [19] With experience gained through collaboration WBI’s contributions, previously mentioned in the with global partners, CAG issued its first guide- other outcome areas, had a cumulative effect on this lines on eLearning development and management scale-up of eLearning. among its network of 451 local institutions. [15] The guidelines were a policy-related change that Conclusion enabled CAG to guide its network on eLearning eLearning courses offered by CAG that have incor- development and management and clarify standards porated cutting-edge content and pedagogy from across local institutions. CAG’s support to its network international and domestic partners provide a large of local institutions is crucial to enhance the capac- number of city officials and practitioners timely and for more information Acknowledgments Project Contacts Thanks to WBI team members involved in the mapping exercise: Mansha Mansha Chen, WBI Urban practice, Chen, Urban Specilist, WBI Urban; Sheila Jagannathan, Senior Learning Specialist, WBI Growth and Competitiveness; Sheng Li, Senior Learning mchen2@worldbank.org; Specialist, WBI Beijing office. Sheila Jagannathan, WBI e-Institute, sjagannathan@worldbank.org Additional thanks to: Yuanyuan Deng, Director, Distance Education Division, Information Email Technology Department, Chinese Academy of Governance WBI Capacity Development and Results Lingyan Ji, Vice Director, Distance Education Division, Information team at capacity4change@worldbank.org Technology Department, Chinese Academy of Governance Manchuan Wang, Director General, Training Department, Chinese Website Academy of Governance www.worldbank.org/capacity Shi Nan, Secretary General, Urban Planning Society of China Yan Chen, Urban Planning Society of China © Copyright 2013 World Bank Fan Tu, Associate Professor, Zhejiang University of Technology Philip E. Karp, Lead Specialist, World Bank EAP Development Effectiveness WBI’s Capacity Development and Results Unit team led the outcome mapping; Jenny Christine Kessides, Manager, WBI Urban Gold coordinated the exercise with support Victor Vergara, Lead Urban Specialist, WBI Urban from Ricardo Wilson-Grau. Sharon Fisher Joyce Msuya, Regional Coordinator, WBI Beijing office provided editorial and design services. Tianxiu Kang, Senior Program Assistant, WBI Beijing office Yuehua Wu, Consultant, WBI Beijing office Samuel Otoo provided overall guidance. Photo by Yang Aijun, World Bank 95 easy access to knowledge and a peer network, which may influence these changes, directly or indirectly, partially or enhance their capacity in delivering and managing wholly, intended or not. Outcomes are identified at two levels in relation to the goal: institutional changes relate to societal, public services. This eLearning approach is particu- policy and organizational changes; and learning/capacity changes larly advantageous to reach officials and practitioners relate to awareness, knowledge or skills, collaborative action, or in secondary cities and lagging regions compared the use of knowledge or innovative solutions. These levels are to traditional face-to-face training that is costly and based on the Capacity Development and Results Framework. The framework provides a systematic yet flexible approach often ineffective. to designing capacity development strategies and programs, First, CAG demonstrated its commitment to monitoring and adaptively managing interventions, and eLearning by establishing a distance learning divi- evaluating and learning from their results. sion and allocating increased funding for scaling- 2 Outcome harvesting is a practical assessment tool from the up eLearning throughout the country. Then, CAG outcome mapping community of practice. It can be used for strengthened its effectiveness to deliver eLearn- real-time monitoring and evidence gathering from complex ing to public officials, by building staff knowledge development processes that involve multiple stakeholders. It and skills, networking with similar institutions, and is based on a similar concept of locally driven change from the increasing know-how to develop and facilitate Capacity Development and Results Framework. The tool was customized to gather information on outcomes—and related courses and new curriculum that incorporate techni- outputs and milestones—to learn from what changed, for whom, cal content relevant to urban practitioners in China. when and where, the significance of the change and how the Lastly, CAG scaled-up its training programs by reach- program contributed to each change. ing thousands of public officials with eLearning mod- 3 Change agents are leaders, groups or organizations from ules, raising awareness of the central government of government or non-state that drive change.. CAG’s progress, enhancing its national network to deliver eLearning, and offering blended training for 4 Change strategy refers to how change happened to advance the first time. progress toward the development objectives—the development With its improved capacity and expanded problems addressed, types of outcomes achieved, WBI contributions, and
partners involved. A change strategy may network, CAG is now empowered to continue to include different types of
change processes or outcome areas develop and manage eLearning courses for public depending on the complexity of the multi-actor institutional officials on its own, and support its network of local changes involved in a program. institutions as well. 5 The numbers in brackets correspond to the outcomes in Figure 1. The text that usually follows each outcome refers to its Next Steps significance. The process of change the outcomes represent can It is important for CAG to continue collaborating be seen in Figure 2. with domestic and international partners, including the WBI e-Institute, to stay current with the latest developments in eLearning; the experience and knowledge gained can then be shared with its large network of local institutions to achieve scale. CAG is interested in working with the e-Institute to avail itself of global eLearning content in other thematic areas and in applying the experience gained to customizing other e-Institute courses for the Chinese audience. n NOTES 1 Mapping outcomes—and related outputs and milestones—can help us learn from change processes that occur during program delivery that often seem complex and opaque because they involve multiple actors and address large development problems. An outcome is what each social actor (or change agent) did, or is doing, that reflects a significant change in their behavior, relationships, activities, actions, policies or practice. The program 96 Improving Social Accountability in the Philippines Education Sector S ince its establishment in 2011, the Check Development Objective My School (CMS) program has provided Improve the quality of services and education perfor- an innovative platform for government and mance of public schools for citizens in the Philippines. citizens in the Philippines to jointly improve Problem transparency and accountability in public schools. There is weak transparency and accountability of public CMS is one of the programs in the Affiliated Net- services to citizens in areas such as education in the work for Social Accountability in East Asia and the Philippines. Sustainable solutions require a continuous Pacific (ANSA-EAP), and its program located in the engagement between government and citizens at a Philippines. As a regional network, ANSA-EAP works countrywide scale to respond to diverse and decentral- to support and operationalize programs across the ized demands from across communities. East Asia region for civil society organizations (CSOs) Specific Objectives to Enable Change to engage with governments on social accountability Improve transparency and social accountability in the issues. education sector; strengthen commitment to collabora- tive government-citizen framework for social respon- WBI’s Social Accountability practice worked with sibility in priority sectors; and strengthen Check My ANSA-EAP on conceptualizing its network and later, School as a network platform to empower communities on strengthening the CMS model and creating part- in the Philippines to engage with a responsive govern- nerships and opportunities to learn from its innova- ment to improve social accountability. tions. CMS offers a country-owned and -led example 97 for World Bank operations and client countries on school personnel, across more than 46,000 schools, how to collaborate with a CSO network to scale up and oversees a significant budget. The Government social accountability interventions in priority sectors. of the Philippines had also made a general move In January–March 2013, WBI mapped the out- toward policies that improve the transparency and comes1 of this initiative using a customized outcome accountability of public service provision. In addi- mapping tool2. This map (Figure 1) presents the tion, the government has a history of supporting sequence of outcomes achieved by ANSA-EAP and initiatives that aim to make its own performance other change agents in the Philippines involved in more transparent and accountable to the public and CMS. The map illustrates how the outcomes and welcomed civil society efforts to provide comple- processes connected and built on each other to mentary monitoring of service delivery programs. advance transparency and social accountability and Since 2011, CMS has led to notable if incipient to improve public education services. changes in DepED responsiveness to school-level Each outcome is numbered and described in the issues. Further, the cooperation between civil society context of a strategy to catalyze change. WBI team and DepEd on data validation has enhanced the members identified and formulated the outcomes, quality and legitimacy of school data. The initiative is presenting an explanation of their significance and continuing to thrive and provides an example of how how WBI contributed—directly or indirectly, inten- countries can address service delivery challenges at tionally or not—by catalyzing or empowering the the local level. change agents to take new actions. Then, roughly 20% of the outcomes were selected from parts of the OUTCOME AREAS change process to be independently substantiated The process of change spearheaded by ANSA-EAP for credibility. in the Philippines and supported by WBI can be seen in four areas of outcomes (Figure 2). These outcomes Background were analyzed and classified according to the types ANSA-EAP regards the link between civil society of change they achieved, then grouped based on and state agencies at the regional, national and how they connected to each other to affect change. sub-national levels as crucial—social accountability efforts that engage state agencies are more likely to Outcome Area 1: Commitment to yield positive outcomes by realizing the objectives of innovations to strengthen social participatory governance and the capacity develop- accountability ment of both civil society and government actors. Between 2008 and 2011, Ateneo University School of CMS addresses challenges in Philippines public Government in the Philippines housed and incu- schools such as poor infrastructure and sanitation, bated the ANSA-EAP network, which was created to which impede student learning and affect atten- conceptualize and operationalize regional programs dance rates. Dispersed community demands for as a platform for CSOs to engage with governments improvement have not resulted in the government on social accountability issues. [1]3 responsiveness required to meet improvements. The In 2011, CMS in the Philippines was formally initi- lack of accurate and timely information on school- ated when DepEd and ANSA-EAP signed a memo- level issues and challenges and the absence of a randum of agreement on sector collaboration to process to systematically and effectively respond to share and verify DepEd school data with civil society. citizen concerns have been challenges for govern- [2] The launch of CMS as a joint DepEd/ANSA- ment and civil society. EAP social accountability undertaking encouraged CMS in the Philippines was designed and imple- schools, principals, local authorities and government mented in the midst of key changes in the educa- agencies to cooperate in order to monitor facilities tion sector that created an enabling environment and materials in schools as well highlight challenges for the program and for collaboration between the in their communities. It also provided the multi- Department of Education (DepEd) and ANSA-EAP. stakeholder commitment to innovate while develop- DepEd was implementing decentralization reform in ing and refining CMS in the Philippines. Later, CMS the public education system; it employs more than focused on resolving priority issues at the school a half million teachers, administrative officials and level as identified by the monitors. 98 Figure 1. Map of examples of outcomes showing changes linked and built during program phases (2) DepEd in (17)* 100s of volunteers and 20 infomediaries working the Philip- in 145 schools used Facebook to communicate on pines signed school issues; in three months posts were viewed agreement almost 75,000 times with more than 430 comments with ANSA to (1) Ateneo collaborate Examples of outcomes (23) ANSA University with civil soci- from Operation Thank You launched School ety on use of second Government school data cycle of data supported (13) Department (15) DepEd newly formed validation in of Public Works allocated funds regional ANSA- Philippines inspected the and renovated EAP network to schools, building and toilets in two conceptualize (3)* DepEd (19) DepEd adding recommended schools and released data acknowledges additional immediate operationalize to ANSA for data collected volunteers, renovation its programs validation and by ANSA and new sites and as a platform dissemination discussed gaps coordinators, for CSOs to with formal and looking at engage with (12) Info- (14) Info- (16) school data issues in more governments mediaries mediary Infomediary schools on issues (4) ANSA made and a school complained mobilized related to social DepEd data principal about toilet alumni (21)* Kenyan (22) ANSA accountability accessible to documented conditions in funding to government provided public using cracks in two schools renovate (and other advisory checkmyschool. building to to DepEd conditions of countries) services org and Department toilets in one consider ANSA requested mapping it of Public school advisory services by Kenyan to location of Works to replicate government about 10,000 checkmyschool schools web platform (6) ANSA carried (5) Infomediaries out first data representing (20) Indonesian validations and communities and government was inspired posted findings ANSA mobilized by the initiative and on website and built skills of requested advisory services (11) Infomediar- 1,053 students from ANSA to develop ies submitted and school similar partnership with requests to local administrators their Ministry of Education (7) ANSA authorities to to monitor and compared address issues validate issues in monitoring 144 schools (18) Open findings against school Government (8) World Bank data provided Partnership recognized (10) Infomediaries became by DepEd ANSA’s and volunteers interested in initiative as an cooperated CMS when it example of how with school was featured as (9) Infomediaries government and administrators to an innovative identified civil society can establish Operation approach to 231 issues work together Thank You, which using data that that required on social channels and is released by resolution in 84 accountability resolves complaint government schools 2008–11 2011 2012 Pilot phase Systematization phase Start of scale-up phase Institutional changes Learning/capacity changes Outcomes related to societal, policy and organizational changes Other outcomes related to awareness, knowledge or Commitment and innovations to advance accountability skills, collaborative action, or the use of knowledge or and transparency innovative solutions. Legitimacy of formal school data * Outcomes selected for substantiation; see page 7 Responsiveness of government sidebar. 99 Figure 2. Change strategy showing how change happened to advance progress toward goal WBI Contributions • Support to start up and Change Agents4 conceptualize ANSA-EAP and its programs • ANSA-EAP • South-South knowledge • Department of Education (DepEd) exchange • Department of Public Works and local authorities • Guidance on CMS website • Infomediaries and volunteers and data validation • School administrators • Government officials • Other country governments • World Bank Partners • ANSA-EAP regional network • Ateneo School of Change Strategy5 Government • World Bank operations Area 1: Commitment to Innovations to Strengthen Social Accountability • Ford Foundation • ANSA-EAP created to respond to social • Trade and Investment accountability through CSO network Framework Agreement • Commitment of DepEd in Philippines to CMS Problems Addressed Area 2: Transparency of Information on Public • Weak collaboration Schools and Citizen Participation in Scrutinizing between CSOs and School Services government in the • Built skills of school administrators and students education sector to monitor and report on issues • Poor quality and legitimacy • Brought together key education players to of school-level data discuss school issues, plan and review data as • Weak demand for social inputs for planning accountability to resolve • CMS web platform made data public and complaints on school provided a vehicle for communities and schools conditions to give feedback and document issues • Ad hoc government • Validation of school data engaged volunteers communications and and schools to monitor issues and compare response around challenges findings to official data in public schools • Plan to scale-up data validation Area 3: Citizen-Government Collaboration to Strengthen Social Accountability in Public Education • DepEd released data and supported Development Objective decentralized validation process • Improve the quality of • 1st CMS data validation identified 231 issues in services and education 83 schools, which were resolved by government performance of public and communities schools for citizens in the • Collaborative mechanism formed to channel Philippines. information on school issues for resolution • DepEd used community feedback to improve quality of school data Area 4: Global Learning to Scale Up Social Accountability Interventions • World Bank recognized CMS as an example of a constructive engagement between government and citizens • Open Government Partnership and countries drew on the CMS example 100 WBI supported the Ateneo School of Govern- provided the public with a window into the real condi- ment to obtain World Bank funding of US$3.25 tions in individual schools across the Philippines rela- million from 2008–2011 through the Development tive to the data produced by DepEd. This transparency Grant Facility that was used to start up and opera- is critical for understanding and addressing underlying tionalize ANSA-EAP. WBI shared examples of other trends in the delivery of education services in schools. regional networks and helped to shape the concept Since 2011, hundreds of volunteers and 20 infome- of a network for social accountability in EAP. WBI also diaries working in 145 schools use the CMS Facebook provided technical advice and support to ANSA-EAP page as a two-way information platform that allows on the development of its programming, including verified data to be made publicly available. During CMS. the course of three months, during which validation activities were conducted, posts that appeared on the Outcome Area 2: Transparency of Facebook page were viewed almost 75,000 times and information on public schools and citizen more than 430 feedback comments were written. [17] participation in scrutinizing school services In this manner, Facebook has become a central avenue In July 2011, ANSA-EAP consolidated public school for CMS to communicate with stakeholders across data from the DepEd, ranging from budgetary allo- communities as well as flag potential issues at the cations to test results to number of teachers on the school level. The ICT aspect of CMS is essential to the checkmyschool.org platform, and then mapped it initiative’s movement toward scale in the Philippines. against Global Positioning System (GPS) coordinates In 2012, ANSA-EAP launched a second cycle of for about 10,000 schools. [4] This consolidation made data validation in schools in the Philippines. [23] The the DepEd data available for the first time for citizen second cycle signifies an improved system of data monitoring and data validation of key service deliv- gathering that was better received by the government ery inputs and outcomes in Philippines schools. than the first round, with more issues resolved and an From August-October 2011, ANSA-EAP engaged enhanced web platform. The continued contribution of “infomediaries”—including community leaders and volunteers from communities and schools also shows socially active individuals—to mobilize and train community ownership of the process and is an impor- 1,053 students and school administrators to serve as tant indicator of sustainability. volunteers to monitor and validate issues and data WBI contributed indirectly to these outcomes. from 144 schools in 14 areas. [5] While CMS has a ANSA-EAP and the network formed to support CMS strong technology component because it is a web- and did the work on the ground. WBI made recom- based platform, the role of community mobilization mendations for enhancing the CMS model, and con- is critical for social sustainability of the approach, nected CMS stakeholders to other stakeholders and by empowering communities to report school-level experts working in ICT and education. issues to local authorities. In sum, CMS used an online platform to make In late 2011, ANSA-EAP engaged the volunteers school data public. It used citizen mobilization and to carry out a data validation exercise. This involved social media to innovatively generate a locally owned uploading monitoring information from the 144 and participatory process to monitor and resolve schools on the CMS website to make it publicly school-level issues and validate official DepEd data. available and disseminating it through Facebook This involved community members and school admin- and SMS text. [6] This exercise aimed to improve istrators documenting problems to facilitate a better public understanding of on-the-ground conditions in understanding of local problems. A well-informed participating schools, engage community members citizenry could thus demand improvements in school in monitoring service provision in their schools and services. demonstrate innovative use of technology, in particu- lar social media. They reached one million Facebook Outcome Area 3: Citizen-government users within the first year. collaboration to strengthen social ANSA-EAP then compared the monitoring find- accountability in public education ings against the public school data provided by In July 2011, despite having no formal open data DepEd to capture discrepancies or validate quality of policy within the government of the Philippines, the data. [7] Using CMS to coordinate the data validation DepEd released its school data to ANSA-EAP for CMS 101 to validate and disseminate. [3] This action showed The following examples of outcomes from the government’s openness to collaborate and set a Operation Thank You illustrate how the school issue new policy precedent since the data on schools was resolution process worked: previously unavailable for use by the general public. • A CMS validation in Putik Central School in Zam- Further, by releasing the data, the DepEd supported boanga City revealed serious cracks in one of the the public use of the data by the checkmyschool.org school’s buildings. CMS’s infomediaries, with the web platform for decentralized data validation with school principal, documented the situation to the communities and schools. Department of Public Works and Highways. [12] Through the first data validation exercise in 2011, The Department of Public Works and Highways the ANSA-EAP infomediaries identified 231 issues sent a team to inspect the building and then rec- that required resolution in 84 schools. Typical prob- ommended immediate renovation. [13] lems included lack of textbooks, lack of classrooms • A CMS validation in the Aruallo High School and facilities that required repair. [9] In these schools, and in the Epifanio Delos Santos Elementary students were empowered to raise issues to school School in Manila exposed the bad condition of administrations and use CMS as a channel to voice the toilets in the schools. The infomediary who their concerns in their schools and communities. In validated the schools sent a formal complaint to the past, these types of issues would often remain the responsible departments in DepEd. [14] The unaddressed, for example with toilets going unfixed DepEd allocated funds and started renovation of for months, and schools receiving no means of solv- the toilets. [15] ing the problem from the local government units of • A CMS validation in the Lt. Andres Calungsod DepEd. Elementary School in Cotabato exposed the poor The challenge was how to respond to commu- conditions of one of the toilets. The infomediary nity demand systematically and collaboratively. In discussed the problem at an alumni meeting in October 2011, ANSA-EAP infomediaries and other the fall of 2011. The alumni independently col- volunteers cooperated with school administrators lected money and funded the renovation of the to establish a collaborative mechanism to channel toilet. [16] information about school issues so problems could be resolved with the support of the DepEd, local The accomplishments of Operation Thank You authorities and communities. This approach, named in resolving issues showed how the connections of “Operation Thank You,” was added to the scope of the ANSA-EAP team and CMS infomediaries with the CMS initiative to express gratitude to participat- DepEd officials and local representatives can be ing schools for their collaboration with CMS. [10] In valuable for schools. Further, the CMS validation late 2011, infomediaries used Operation Thank You process inspired communities to take action outside to submit requests to local authorities to address of DepEd to address problems in schools, making issues. [11] solutions more collaborative. ANSA-EAP’s ability to incorporate Operation In January 2012, ANSA-EAP held a valida- Thank You into CMS and the responsiveness of the tion meeting with DepEd, where the government DepED, other government offices and local authori- acknowledged the data collected by ANSA-EAP and ties to collaborate on issue resolution demonstrated discussed gaps identified between official DepEd the flexible and adaptive nature of the CMS model— data and CMS data. [19] ANSA-EAP’s report back to and the alignment of the model with DepEd policy DepEd closed the feedback loop between the CSOs to direct schools to learn to solve problems on their and government agencies. This loop is critical for own by engaging community stakeholders. The CMS DepEd to engage with ANSA-EAP to improve the network had critical value for results and sustainabil- quality of its public school data and its legitimate ity within the context of the Government of Philip- use to improve school services. pines reform agenda. The CMS web platform was a WBI provided input to the concept develop- valuable problem-solving tool in the hands of school ment of CMS, particularly on the website model, administrators in the schools where CMS operates. and linked ANSA-EAP to experts within the Bank to provide technical inputs. ANSA-EAP and change 102 substantiation of outcomes To verify the accuracy of the outcomes mapped and enrich WBI’s understanding of them, the external consultant selected three outcomes [3, 17, 21], and asked nine people independent of WBI but knowledgeable about the change to review each and record whether they agree with the outcome as described. Six people responded: all six fully agreed with the outcomes’ significance as stated. The respondents provided additional information to clarify the outcomes’ description and WBI’s contribution. Excerpts of substantiators’ comments on the outcomes achieved: “From the interaction with the Kenyans, CMS’ emphasis of the citizen role really echoed and broadened the awareness of the government representatives on this possibility.” —Dondon Parafina, Executive Director, ANSA-EAP “As CMS coordinator and getting related educational information released by DepEd schools, I was able to gain insights on the local issues in public schools in Bacolod City as evidenced in its physical facilities and resources, as well as direct responses from various relevant stakeholders.” —Darlene Casiano, Area Coordinator, Bacolod City “The interest to replicate CMS is definitely a good indicator of the government’s openness. It would be great if this will also move to other departments/ministries of Kenya. Certainly, there are some research that needs to be done to tailor CMS in the Kenyan context.” —Jecel Censoro, Project Coordinator, CMS agents from DepEd and the local level led all of ing of as an innovative approach to using data these outcomes. that is released by governments as part of their In sum, the collaboration between DepEd and open data policies. [18] A key aspect of the OGP ANSA-EAP around CMS provided a framework agenda is translating the transparency and acces- for citizens and government to work together to sibility of data into meaningful action and analysis improve school services. The CMS model also by citizens and citizen groups. Potentially the provided a flexible approach to respond to chang- OGP may serve as an important vehicle to share ing school service needs and support decentralized the lessons learned by CMS with other countries. policy to build more self-reliant schools with strong • In early 2012, the Indonesian government community leadership. requested advisory services from ANSA-EAP to develop a similar partnership with the Ministry of Outcome Area 4: Global learning to scale up Education in that country. [20] The request shows social accountability interventions recognition of the work done by CMS and the The CMS model and its successes triggered broad need for other countries to take similar institu- global and country interest in the experience of tionalized approaches to address accountability the Philippines. Since 2011, the World Bank has at scale in the education sector. increasingly recognized the CMS initiative as an • In July 2012, the World Bank country team in example of how government and civil society can Kenya requested ANSA-EAP support to facilitate work together on social accountability in a priority the development of a similar partnership. sector. [8] This is important given the growing [21] There are also attempts to develop a interest to address social accountability effectively CMS model in Moldova, indicating increasing with government and to open government data to interest in the CMS model outside of the EAP the public. region. ANSA-EAP presented the model to Others are also seeing relevant lessons in practi- key Kenyan stakeholders among the relevant cal innovations of the CMS example: Kenyan government ministries, departments and • In December 2011, the Open Government Part- agencies, as well as to the World Bank country nership (OGP) became interested in the CMS team, donors and CSOs in Kenya. [22] CMS initiative when it was featured in an OGP meet- was viewed as a promising network model that 103 could inform learning around how to scale-up helped to address is the ability for administrators to data validation and ensure consistency of data leverage communities to resolve issues and prob- from the proposed integrated Kenya Education lems in the decentralized structure of the school- Management Information System. based management system. Nonetheless, challenges remain, including: WBI contributed by raising awareness about • The sustainability of the community-based CMS and its innovations among practitioners within approach realized through CMS will need to be the World Bank and beyond. Activities included carefully monitored, though early indications sug- facilitating South-South knowledge exchanges gest local momentum in participating schools. among stakeholders from government, civil society • CMS’s ability to reach areas where ANSA-EAP and donors to discuss the CMS model, knowledge- does not have a strong foothold, especially in sharing session with World Bank staff and liaising remote parts of the country, and eventually scale with the OGP secretariat. In addition, WBI sup- up to most counties in the Philippines will require ported ANSA-EAP to provide advisory services to expansion of capacity and resources within the Indonesian government and an Indonesian CSO ANSA-EAP and DepEd. (Transparency International Indonesia). • Ensuring continuity of the program across Thus, the CMS initiative has provided an innova- government administrations, such that politi- tive and practical example for World Bank opera- cal changes do not inhibit the initiative’s move tions and broader country learning around how toward scale will require fully embedding the government and civil society can work together on model within the community and civil society social accountability in a priority sector. space, and sustained commitment at the techni- cal level within DepEd. Conclusion CMS delivered improvements in education service There are challenges to sustain the program delivery at the school level by using an online plat- within the CSO network, including long-term fund- form and collaborative mechanisms to help school ing, difficulties in realizing the full ICT potential of administrators, citizens and students work together the project and ensuring the project’s structure and to monitor, identify and resolve school issues. decentralization can be aligned with the goal of Equally noteworthy, the initiative is in the process scale. of analyzing school data to identify broader trends Another challenge is the fragmentation of at the school level. One specific area that CMS has databases across the government of the Philippine for more information Acknowledgments Project Contact Thanks to Carolina Vaira, Keith McLean, Megan McDermott, WBI Social Accountability practice, Jennifer Shkabatur for their work in the mmcdermott@worldbank.org mapping exercise. Email Thanks to substantiators: WBI Capacity Development and Results team at Tin Aquino, Communications Manager, capacity4change@worldbank.org ANSA-EAP Website Darlene Casiano, Area Coordinator, Bacolod www.worldbank.org/capacity City Jecel Censoro, Project Coordinator, CMS © Copyright 2013 World Bank Luiza Nora, Social Development Specialist, WBI’s Capacity Development and Results team led the outcome World Bank mapping; Jenny Gold coordinated the exercise with support Dondon Parafina, Executive Director, from Ricardo Wilson-Grau. Sharon Fisher provided editorial and ANSA-EAP design services. Samuel Otoo provided overall guidance. Paul Thomas Villanueva, Network Coordinator 104 2 Outcome harvesting is a practical assessment tool from the agencies, even as relates to one sector. For example, outcome mapping community of practice. It can be used for the data required to understand a full picture of what real-time monitoring and evidence gathering from complex is happening at the school level requires access to development processes that involve multiple stakeholders. It databases across agencies (such as facility infor- is based on a similar concept of locally driven change from the mation, textbooks, nutrition, budget allocations). Capacity Development and Results Framework. The tool was customized to gather information on outcomes—and related Currently most of these databases do not “speak” to outputs and milestones—to learn from what changed, for whom, each other. when and where, the significance of the change and how the program contributed to each change. Next Steps 3 The numbers in brackets correspond to the outcomes in The CMS program is now in its third phase and Figures 1. The text that usually follows each outcome refers to its growing in terms of its organizational capacity, significance. The process of change the outcomes represent is technological advancements and on-the-ground seen in Figure 2. capacity. Efforts are underway to enhance the data 4Change agents are leaders, groups or organizations from visualization to make school-level information even government or non-state that drive change. more user-friendly for the public. Future outcomes 5 Change strategy refers to how change happened to advance will likely become more systematic. progress toward the development objectives—the development Also, partnerships with other types of educational problems addressed, types of outcomes achieved, WBI institutions are in the process of being formed. For contributions, and
partners involved. A change strategy may example, ANSA-EAP is formalizing a partnership include different types of
change processes or outcome areas with universities and additional schools, and to make depending on the complexity of the multi-actor institutional CMS part of the required National Service Train- changes involved in a program. ing Program to promote broad youth involvement in public education improvements. A similar activ- ity is also in the pipeline, this time covering all of the universities in Baguio city that will help monitor schools in the city. The partnership with DepEd is also improving—the DepEd tapped CMS to provide support data through feedback from citizens to serve as backup data for their budget proposal to Con- gress. n NOTES 1 Mapping outcomes—and related outputs and milestones—can help us learn from change processes that occur during program delivery that often seem complex and opaque because they involve multiple actors and address large development problems. An outcome is what each social actor (or change agent) did, or is doing, that reflects a significant change in their behavior, relationships, activities, actions, policies or practice. The program may influence these changes, directly or indirectly, partially or wholly, intended or not. Outcomes are identified at two levels in relation to the goal: institutional changes relate to societal, policy and organizational changes; and learning/capacity changes relate to awareness, knowledge or skills, collaborative action, or the use of knowledge or innovative solutions. These levels are based on the Capacity Development and Results Framework. The framework provides a systematic yet flexible approach to designing capacity development strategies and programs, monitoring and adaptively managing interventions, and evaluating and learning from their results. 105 106 107 Outcome harvesting offers innovative, participatory tools to understand and document how change happens in complex development programs that involve multiple social actors, areas of new learning and profound development challenges. Understanding and interpreting outcomes is important to improve results for development programs—it can facilitate strategic, systematic knowledge sharing and management; provide information that can be combined with other sources to review results within a task team and with clients and partners; and supply new evidence on implementation, outcomes, and indicators to inform program development. In these pilot experiences, teams applied outcome harvesting tools in 10 ongoing initiatives in strategic thematic areas. The process encouraged dialogue among the teams and clients and answered learning questions to maximize the benefits of the interventions, institutional sustainability of results, and involvement of the right mix of actors. 108