Report No. 45327-UY Uruguay Strengthening Participatory Monitoring and Evaluation of Social Policy Nonlending Technical Assistance Project Report on Phase II September 29, 2008 Country Management Unit Argentina, Chile, Paraguay, Uruguay Sustainable Development Department Latin America and Caribbean Document of the World Bank Currency Unit = Uruguayan Peso ($) 1 US Dollar = 18.9 Uruguayan Pesos 1 Peso = US$0.052 (As of September 15, 2008) Abbreviations and Acronyms BPS Banco de Previsión Social CSO Civil Society Organization DDC Dirección de Desarrollo Ciudadano DINAE Dirección Nacional de Empleo DINEM Dirección Nacional de Evaluación y Monitoreo GDLN Global Development Learning Network IBTAL Institution Building Technical Assistance Loan IDES Instituto de Investigación y Desarrollo IEGK Independent Evaluation Group Knowledge INE Instituto Nacional de Estadísticas ISIS Integrated Social Information System JUNAE Junta Nacional de Empleo LCSHS Social Protection Sector, Latin America and Caribbean (World Bank) LCSPP Poverty Sector, Latin America and Caribbean (World Bank) LCSSO Social Development Sector, Latin America and Caribbean (World Bank) M&E Monitoring and Evaluation MIDES Ministry of Social Development MTS Ministry of Tourism and Sports NCSP National Council of Social Policies NGO Nongovernmental Organization NLTA Nonlending Technical Assistance OPP Office of Planning and Budget PANES Plan de Atención Nacional a la Emergencia Social PARA Participatory Rural Appraisal PE Plan de Equidad PM&E Participatory Monitoring and Evaluation TA Technical Assistance TOR Terms of Reference TxU Trabajo por Uruguay UY Uruguay Regional Vice President: Pamela Cox Country Director: Pedro Alba Sector Director: Laura Tuck Sector Manager: Maninder Gill Task Team Leader: William Reuben Table of Contents 1. Origin and Overview of the Technical Assistance Project ....................................... 4 2. UY NLTA Contributions under Phase II to Strengthen Impact Analysis Capacity in MIDES: Poverty and Social Impact Analysis (PSIA)...................................................... 8 3. UY NLTA Contributions under Phase II to make the M&E System Operational: Technical Assistance Activities...................................................................................... 14 A. Training activities for eight social councils ....................................................... 14 B. Participatory evaluation pilots............................................................................ 16 C. International workshop on monitoring and evaluation....................................... 20 D. Implementation of the Integrated Social Information System........................... 23 4. UY NLTA Evaluation: Risks, lessons learned, and recommendations .................. 27 A. Lessons learned for future Bank operations....................................................... 28 B. Lessons learned for the implementation of M&E systems: ............................... 28 C. Recommendations to strengthen MIDES's leading role and improve World Bank cooperation: ................................................................................................... 30 5. Bibliography............................................................................................................ 33 Tables: Table 1: Activity Plan UY NLTA Phase II ...................................................................... 7 Table 2: Suggestions included and not included in the PE ............................................ 13 Table 3: Main Differences in M&E Systems ................................................................. 22 Table 4: Risks and Mitigation Measures - Phase II........................................................ 27 Figures: Figure 1: PANES Targeting ............................................................................................. 9 Figure 2: PANES Coverage............................................................................................ 10 Box: Box 1: Results of Tacuarembó's and Rivera's analysis of Plazas de Deportes project. 18 3 Uruguay: Strengthening Participatory Monitoring and Evaluation of Social Policy ­ Nonlending Technical Assistance Project: Report on Phase II 1. Origin and Overview of the Technical Assistance Project 1.1 The Uruguay Nonlending Technical Assistance Project (UY NLTA) was undertaken at the request of the Government of Uruguay's Ministry of Social Development (MIDES).1 Following a devastating financial and macroeconomic crisis in 2002, MIDES was established with the aim of providing a safety net program (PANES) for Uruguayans affected by the crisis.2 The new administration that took office in March 2005 sought to improve the targeting and outreach of its safety net program through participatory monitoring and evaluation, and invited the Bank to support this initiative through nonlending technical assistance. 1.2 The design of the Technical Assistance Project has been led by MIDES, with the Bank playing an advisory role. The project's scope, overall methodology, and expected outcomes were defined in a dialogue with the Ministry of Social Development during a Bank mission in May 2006. Terms of Reference (TOR) were developed jointly, the selection of consultants or firms was decided based on recommendations by MIDES, and the intermediate products were commented on and revised in conjunction with MIDES staff. 1.3 The objectives of the UY NLTA were to: (i) evaluate the impact of PANES's social emergency programs; (ii) inform a dialogue about the future role of social programs and of MIDES as a lead authority responsible for the coordination and oversight of social policy in Uruguay; and (iii) strengthen MIDES's capacity to assess, monitor, and evaluate social policy in Uruguay, with considerable weight given to participatory approaches to monitoring and evaluation. 1.4 Expected outcomes of the project included: (a) increased efficiency in the implementation of Uruguay's social programs through the consolidation of existing and dispersed databases and the strengthening of MIDES's M&E system; (b) increased transparency and strengthened accountability in selected departments through the piloting of a participatory monitoring and evaluation (PM&E) approach; (c) stronger engagement of civil society in public sector reform in Uruguay; and (d) informed decisions about the future of social programs under the Plan Nacional de Equidad, and the role of MIDES, on the basis of the evaluations of PANES. 1MIDES was created in March 2005 (Law 17.866) with the main objective of formulating, coordinating, and overseeing social development strategies, policies, and programs in Uruguay. 2PANES was conceived as a temporary program, composed of eight projects/components aimed at overcoming the social emergency. It was set to last two years, until mid-2007, but was expanded until December 2007. 4 1.5 The original duration of the UY NLTA was one year. However, it was extended to two years to make it consistent with the client's timing and needs, and the activity was divided in two phases, each responding to specific goals and delivering specific products. The first phase, which was completed during the first year, was focused on supporting the Ministry's capacity to conduct social impact evaluations and to design a Participatory Monitoring and Evaluation (PM&E) pilot system.3 Phase II, which took place the following year, was aimed at deploying these capacities, completing the impact evaluation of PANES, and making the PM&E pilot system operational. 1.6 Phase one achieved all its expected outcomes. MIDES's institutional capacity for impact evaluation was increased by: (i) supporting data collection for impact evaluation, and (ii) conducting quantitative and qualitative impact analyses. The design of the PM&E system was achieved by: (i) developing the TOR for the design of the software for the Integrated Information System; (ii) defining indicators for the PM&E system; and (iii) sharing the Bank's assessment of international best practices in PM&E. 1.7 The data used in the evaluation of PANES came from a follow-up household survey with special sampling, and from existing registry data. The survey included numerous variables that made it possible to understand the intra- household allocation of resources, gender issues, and the use of time within households, among other key aspects. The sampling framework included poor female-headed households and Afro-Uruguayan households, which were underrepresented in the regular household survey. The analysis of PANES registry data allowed evaluations to be performed, comparing individuals who exit the program with similar individuals entering the program around the same time. 1.8 The quantitative and qualitative impact analyses provided MIDES with thorough complementary methodologies to assess PANES's impact. The quantitative impact analysis drew on comparative and counterfactual analyses with various statistical approaches. Data used for the analysis came from the PANES follow-up survey and other statistical sources. The qualitative impact analysis was based on a series of hypotheses that were tested through snapshots of protagonists, evaluating their satisfaction, results, and expectations with regard to the program. Moreover, hypotheses were tested through "tracking assessments" that tracked the trajectory of protagonists into the labor market. 1.9 The TOR for the design of the software for the Integrated Information System included a full road map. It began with the identification of the physical and technological resources and the means of communication existing in the current technological platform of information, and ended with the definition of all the technical requirements that MIDES will need for the installation of the data warehouse and the purchase of a server. 1.10 To define the indicators for the PM&E system, MIDES evaluated the usefulness and appropriateness of a set of international, regional, and national systems of socioeconomic indicators. The information reviewed and 3Final delivery of the Report on Phase I took place on September 20, 2007. 5 the data available in Uruguay were used to define a set of indicators. The first list collected 350 indicators; these were later refined to 200. Some of these indicators were calculated by different agencies while others were calculated by MIDES. 1.11 The last output of Phase I consisted of sharing the Bank's assessment of international best practices with PM&E. The UY NLTA team briefed the staff of the National Office of Monitoring and Evaluation (DINEM) on different social accountability techniques, such as Participatory Rural Appraisal (PRA), Public Expenditure Tracking Surveys (PETS), Citizen Scorecards, and Community Report Cards, so that MIDES can be better prepared to develop its PM&E system. 1.12 Phase II was built on the findings and results of Phase I with the aim of achieving the project's objectives and expected outcomes. It was organized in two components: Poverty and Social Impact Analysis (PSIA) and Technical Assistance (TA). The PSIA component combined a series of technical contributions to the evaluation of PANES and offered inputs to the dialogue on the Plan de Equidad (PE), designed by National Council of Social Policies.4 The TA component provided support to DINEM and the Dirección de Desarrollo Ciudadano (DDC) in piloting the PM&E system and implementing the Integrated Social Information System. 1.13 The results of Phase II included the following outputs that are presented in this report: For the first component, PSIA, a technical note summarized the results.5 For the second component, the TA results included: (i) training to consolidate eight social councils; (ii) the implementation of the PM&E pilot system in Rivera, Paysandú, and Tacuarembó; (iii) the implementation of an Integrated Social Information System; (iv) the creation of a forum for inter- institutional and territorial policy dialogue; and (v) an international workshop to refine the design of MIDES's M&E system. 1.14 The rest of this report is organized as follows: Section II presents the PSIA's main contribution by laying out the main results described in the technical note. Section III presents the TA's contribution and is divided into three subsections. Subsection A looks at the contributions to strengthen the capacity within MIDES to conduct impact analyses; subsection B analyzes activities in the TA aimed at making the M&E system operational; and subsection C describes the International Workshop on M&E. Section IV describes the risks and main problems faced during the implementation of Phase II and includes lessons learned, pending agenda, and next steps. 1.15 The report is based on different types of inputs. On the one hand, all the consultation reports produced during Phase II were considered. On the other hand, the report includes the opinions of numerous MIDES authorities and staff which were collected during a mission to Montevideo on August 9. The 4The Plan Nacional de Equidad is a new income transfer program. See point 2.3. 5"Del PANES al Plan de Equidad, Nota Técnica," May 2008. Attached in Annex 1, and also available at http://www.bancomundial.org/uy, in the section "Publicaciones e informes." 6 international presentations at the M&E workshop, the opinions of the presenters, and key MIDES official reports were also consulted. Table 1: Activity Plan UY NLTA Phase II Date Type of Event Activities April 12 to Training Training for eight social councils. November 17, Piloting PM&E Implementation of the PM&E pilot in three social councils. 2007 system September 2007 Integrated Social Development and implementation of social information and to July 2008 Information System M&E systems. May 19, 2008 Technical Note Publication and official distribution of the Technical Note. May 19­20, Seminar and Two-day international workshop with MIDES. 2008 Workshop Missions to Participants Activities UY: October 23­25, María Beatriz Meeting with MIDES to collect information for the 2007 Orlando Technical Note and brainstorm with MIDES/DINEM team about the Technical Note content. November 12­ William Reuben Workshop to present the evaluation of PANES led by the 14, 2007 (LCSSO); Bank and MIDES staff with the participation of the Ministry Rafael Rofman of Social Development and the acting President of Uruguay. (LCSHS) February 11­12, William Reuben Supervision of the project. 2008 (Task Team Leader, Organization of a training workshop in March. LCSSO); Estanislao Discussion of the final version of the Technical Note. Gacitua-Mario (Lead Identification of MIDES's interest in co-organizing an Social Development activity to discuss emerging social development challenges Specialist, LCSSO); in the region, such as Social Guarantees, under the Eri Watanabe framework of Uruguay's leadership role with regard to the (Consultant, LCSSO) development of social institutions in Mercosur. Discussion with the government about its interest in promoting a pilot to strengthen civil society oversight of the national budget and public expenditure under the framework of the Bank's C-GAC initiative. Identification of technical assistance needs in the context of the new Country Partnership Strategy. May 19­21, William Two-day international workshop with MIDES. 2008 Reuben (LCSSO); Dialogue on future engagement of MIDES and Bank Rafael Rofman support. (LCSHS); Manuel Supervision of Phase II activities. Fernando Castro Quiroz (IEGKE); Eri Watanabe (Consultant, LCSSO) August 8, 2008 Marisa Miodosky Gathering of information for the final report on Phase II. (Consultant, LCSSO) Source: Bank staff 7 2. UY NLTA Contributions under Phase II to Strengthen Impact Analysis Capacity in MIDES: Poverty and Social Impact Analysis (PSIA) 2.1 On May 19, 2008 a Technical Note6 based on the findings of several PANES qualitative and quantitative evaluations was officially presented to key MIDES authorities. The evaluations used in the note included an analysis of the first and second rounds of PANES surveys, a qualitative assessment of other programs, notably Rutas de Salida and Trabajo por Uruguay (TxU), a socioeconomic profile of PANES beneficiaries, and a quantitative analysis of income transfer programs in Uruguay. All these evaluations were part of the PSIA envisaged in the UY NTLA project. 2.2 The note was the result of a joint effort by Bank and DINEM staff. A first draft of the note was prepared by Bank staff. This draft was revised by DINEM staff who added significant changes and suggestions. This version was later discussed with Bank staff. Finally, the two parties agreed on a final version of the document. For DINEM staff, the writing process constituted a training experience; they learned to "say a lot in a few words," an uncommon but highly valued pattern in Uruguayan culture. 2.3 The note was aimed at providing analytical elements for the implementation of the Plan de Equidad. This new plan is composed of a series of projects aimed at protecting Uruguayans from different risks throughout their life cycle. One of its main components is designed to operate as a reformed family allowance scheme. The plan's perspective differs from PANES in that it no longer represents a short-lived emergency intervention but instead is part of the overall social program in Uruguay. In light of the results of the PANES evaluation, the transfer amount will be higher than under PANES, and will increase with children's educational level. The transfer will be even higher for those families with children attending secondary school. 2.4 The first part of the note focuses on the positive, although minor, impact of Ingreso Ciudadano (IC) on poverty. The main component of PANES was IC, which consisted of an income transfer program that provided supplementary income to extremely poor households. If the program had not been implemented, the extreme poverty rate would likely have increased by 50% in 2006, from 2.87% to 4.27%. In addition, the poverty rate would have been slightly higher: 27.8% compared to the actual 27.5% in 2006. Inequality would also have increased, while the annual transfer costs would have decreased by 0.31% of GDP. In the period 2007-2008, in a context of macroeconomic stability, the poverty rate has decreased to 26%. (First semester of 2008 was 21.7%) 2.5 IC has been one of the most successful income transfer programs in terms of targeting effectiveness, but its coverage is limited. A study carried out by 6"Del PANES al Plan de Equidad, Nota Técnica," op cit. 8 the World Bank (Rofman 2007)7 concludes that over 75% of IC beneficiary households belong to the poorest quintile of the population (see Figure 1). The IC was also more efficient than other income transfer programs in UY. Nevertheless, IC coverage has been limited, only reaching slightly more than 50% of the extreme poor (see Figure 2). In addition, because 14% of the extreme poor population did not apply to the program, there may have been outreach limitations. Finally, about 30% of the extreme poor who did apply were not accepted. DINEM evaluations conclude that a large share of those who did not receive PANES benefits were unaware of the program's existence. Because the Plan de Equidad is expected to progressively scale up its coverage, the Technical Note suggests considering an appropriate dissemination scheme through community radio and other local media so that the information can reach the extreme poor population not covered by PANES. Figure 1: PANES Targeting 75.4 80 70 60 50 Percentage 40 30 19.5 20 10 4.3 0.7 0.1 0 I II III IV V Source: Rofman et al. (2007), World Bank 7Rofman et al., 2007. "Income Transfer Policies in Uruguay: Closing the Gaps to Increase Welfare, World Bank. 9 Figure 2: PANES Coverage 100 90 85.9 80 70 60 55.7 53.7 50 40 30 Applied 20 18.1 Received 10 5.8 0 0.7 Extremely Moderately Non-poor poor poor Source: Rofman et al. (2007), World Bank 2.6 Overlaps of income transfer programs represent a problem even though coverage of the poorest quintile has increased over the last five years. Poor households receiving IC and food card (tarjeta alimentaria) were also recipients of other contributory and non-contributory transfers. The Plan de Equidad eliminates Ingreso Ciudadano because the family allowance system will be enlarged. In this way, the problem of overlapping and the high administrative costs of IC will be avoided. 2.7 It is crucial for the success of income transfer programs that their linkages to other programs and social services be ensured in order to increase human capital and the exit from poverty. The note draws attention to keeping and improving the linkage between the family allowance transfer and other social programs and services, which was done well by IC and other PANES programs. 2.8 The second part of the note focuses on the qualitative impact evaluation of one of the most visible component of PANES, Trabajo por Uruguay (TxU). The program provided remuneration (twice the IC allowance) for a limited period of time to participants who agreed to work in social/community activities and to acquire new skills that would help them rejoin the labor market. TxU was administered by CSOs. 2.9 The evaluation of TxU faced a series of limitations. First, the program had a series of objectives that were difficult to evaluate. Second, training modules lacked clear operational objectives. Third, the objectives of TxU did not differentiate between the temporarily poor (e.g., those linked to an economic downturn or crisis) and the structurally poor. 2.10 Nevertheless, the evaluation detected a design problem that biased the program toward women, who represented 72% of participants. As mentioned above, TxU required beneficiaries to participate in a series of workshops and to work in community tasks. These activities implied a significant amount of time that could not be allocated to occasional work or 10 "changas." This opportunity cost was typically too high for men, who preferred to opt out of the program. 2.11 TxU challenged common stereotypes regarding male-oriented jobs. Many tasks that were supposed to be conducted by men (i.e., infrastructure works) were undertaken by women. These women not only performed very well in the assigned activities but also enjoyed the tasks. This suggests the importance of presenting an open menu of activities to the participants rather than designing the selection criteria of new labor-oriented projects based on predetermined options for each sex. 2.12 Despite crowding out various other income sources, TxU had a positive impact on the average income of its beneficiaries. About 77% of households increased their income even though this increase was lower than the amount transferred. The additional amount was used primarily for current expenditures as well as for investments (i.e., repair of household items and cancellation of debts). 2.13 The program had less success in increasing labor-related capacities or specific skills. According to the participants interviewed, the activities they carried out did not represent a knowledge gain but consisted of routine tasks that were not socially valued. Nevertheless, the program was successful in developing various important behavioral skills, especially teamwork. 2.14 The qualitative evaluation of TxU concludes that the program has had a positive impact on the protagonists'/beneficiaries' horizontal social capital. The protagonists had more medical checkups and gained increased knowledge of their rights, particularly those related to domestic violence. The knowledge and empowerment to engage with other institutions or organizations (vertical social capital) were minimal, and these remain a challenge for the PE. 2.15 The program's productive component was not effective. The component provided information about how to establish and manage a microenterprise. However, it was limited in providing specific services or linkages to access capital or credit. The protagonists of this component did not perceive any income gain in their households. 2.16 The Technical Note provides some clues to discuss how the Plan de Equidad could overcome the difficulties associated with these results. One important issue deals with the characteristics of the CSOs that provide the training. The Technical Note makes three key suggestions to improve the program's results: (i) select CSOs in terms of their training skills and employment promotion experience; (ii) give participants the possibility of choosing providers in order to ensure quality services and alignment between training needs and delivery; and (iii) make quality control mechanisms available to the protagonists; for example, allowing them to evaluate supplies through scorecards. 2.17 Another issue discussed in the note refers to the type of skills/training that should be promoted. The majority of TxU participants were trained in nontransferable skills, while transferable skills such as languages may hold 11 higher potential for future returns. The note suggests that the Plan de Equidad should include training programs in both transferable and nontransferable skills. 2.18 The note highlights the need to establish an accurate M&E system for the PE that combines monitoring and evaluation with social accountability and citizens' participation mechanisms. Fortunately, MIDES is moving firmly in this direction. The efforts undertaken by the numerous institutions associated with the PE to link different data sets is a key step in consolidating the M&E system. This effort should have solid governmental and international cooperation support. The World Bank helped to organize a seminar on the M&E system aimed at making regional best-practice experiences available to MIDES (see point 3C). 2.19 The note also encourages the inclusion and dissemination of information on social services standards and quality assurance. This is a key step, together with the information generated from the participatory monitoring, for citizens to demand accountability in an organized and effective manner. The note encourages the exploration of chances to implement Citizens' Information Centers throughout the country to support these efforts, by making information about citizens' rights and minimum quality standards available to the public and by channeling citizens' opinions to decision makers. 2.20 Undoubtedly the most important challenge for a labor training program is its linkage with the private sector and labor demand. To address this challenge (which affects similar programs throughout the world), the note provides examples from Mexico (Programa CIMO). Part of this experience has been incorporated in the design of the Plan de Equidad. A subsidy (60% of men's salaries and 80% of women's salaries with a limit of one and a half minimum salaries) will be given to specific private firms to hire employees. However, the note warns about the potential risk of creating the incentive to replace workers and suggests the need to empirically assess the appropriate level of this subsidy. 2.21 Based on the evaluation results, the Plan de Equidad addresses the development of participants' capacities and job skills under the framework of the design of specific labor-related programs: the Uruguay Works Program (Programa Uruguay Trabaja), a public employment program with a training component, and the Private Sector Hiring Incentives Program (Programa Incentivo a la Contratación en el Sector Privado). The note further suggests: (i) designing specific training programs targeted to the extreme poor; and (ii) developing detailed mechanisms to link these programs and those implemented by other agencies within the Ministry of Labor (DINAE and JUNAE). 2.22 PANES is one of the few programs in the world that has incorporated the concept of "exit strategies" ("rutas de salida") in its design, achieving some positive results in terms of self-esteem and self-perception. "Construyendo Rutas de Salida" was a PANES component aimed at providing personal development and citizen integration opportunities as means to overcome poverty. The qualitative evaluation points out important improvements in the self-esteem of the protagonists/beneficiaries and in their self-perception. These changes resulted in positive behaviors that broke the protagonists'/beneficiaries' 12 condition of isolation and self-exclusion, which in several cases perpetuates poverty for generations. 2.23 Based on the successful experience of PANES, the PE has incorporated a series of participatory venues aimed at building "exit strategies" for adolescents. These venues will be managed by the Instituto Nacional de la Juventud with the participation of local governments and CSO. Although increasing the focus on youth and providing opportunities to involve the young population and address their needs are positive developments, they carry the risk of overlapping efforts and making M&E more difficult. The note suggests defining these actions as pilots, guided by clear and evaluable objectives, and defining more consistent actions, once the rigorous evaluation of the pilot has been carried out. This suggestion draws from lessons learned about the participation of youth in the participatory M&E pilot conducted by MIDES with the support of the World Bank. 2.24 According to DINEM staff, the note has been instrumental in making DINEM a key actor that processes and provides information for policy decisions. The note was widely distributed among sector directors, program managers, and mayors throughout Uruguay. In addition, the note was presented to the National Council for Social Policies, where all "social" ministries are represented. Interestingly, the note was discussed in the parliament, responding to an initiative of the opposition, and was used to defend MIDES's role and structure. As a collateral effect, the note was very useful in helping its diverse audiences to understand the differences between PANES and the Plan de Equidad. 2.25 Moreover, the note allowed MIDES and DINEM to share the PANES experience internationally. According to DINEM staff, the fact that the note was a joint MIDES-Bank publication broadened the spectrum of readers and possibilities for sharing ideas. The note was published on the MIDES Web page and numerous media reported its findings. Furthermore, Bank staff published its main points in En Breve, which is distributed throughout Latin America (see Annex 2). Table 2: Suggestions included and not included in the PE Sugestión Included in PE Not Included in PE Considering an appropriate dissemination scheme through community radio and other local media so that the information can reach the extreme poor population not covered by PANES Keeping and improving the linkage between the family allowance transfer and other social programs and services Selecting NGOs in charged of labor training in terms of their training skills and employment promotion experience 13 Giving participants the possibility of choosing providers in order to ensure quality services and alignment between training needs and delivery Making quality control mechanisms available to the protagonists; for example, allowing them to evaluate supplies through scorecards Including training programs in both transferable and nontransferable skills Establish an accurate M&E (moving forward in this system for the PE that combines direction) monitoring and evaluation with social accountability and citizens' participation mechanisms Including and disseminating (moving forward in this information on social services direction) standards and quality assurance (Citizens' Information Centers) Defining youth exit strategies as pilots, guided by clear and evaluable objectives, and defining more consistent actions, once the rigorous evaluation of the pilot has been carried out Empirically assessing the appropriate level of the job subsidy 3. UY NLTA Contributions under Phase II to make the M&E System Operational: Technical Assistance Activities 3.1 Technical assistance was delivered through three types of activities, all of them aimed at strengthening MIDES's M&E capacity. Due to the scarcity of resources and to institutional and contextual limitations, DDC, DINEM, and Bank staff decided to organize two groups of social councils for the PM&E experience: eight councils were selected to be trained in public policies and PM&E; and three different councils carried out the PM&E pilot. An international workshop on M&E took place in Montevideo in mid-May. A. TRAINING ACTIVITIES FOR EIGHT SOCIAL COUNCILS 3.2 The social councils of San José, Bella Unión, Salto, Lavalleja, Maldonado, Durazno, Cerro Largo, and Florida were selected to be trained in public policies and M&E. MIDES selected these councils for two reasons. On the one 14 hand, they met minimum conditions required for the training's viability (i.e., the councils have organized several meetings before the training). On the other hand, MIDES believed the training would be particularly important for strengthening these councils. 3.3 Social councils are responsible for: (i) contributing to build the departmental social agenda; and (ii) monitoring, assessing, and preparing recommendations about the social public policies developed territorially. They are constituted by civil society actors with diverse organizational degrees, coming from different public policy areas (health, education, etc.). The national government, through MIDES, promotes the formation of social councils in each Uruguayan department. In addition, it supports them through training and by facilitating their exchanges with other governmental institutions. 3.4 The general objective of the training was to contribute to the strengthening of social councils as local actors. The specific objectives of the training included: (i) providing the councils with conceptual and technical tools to increase their analytical and design capacity in local public policies; (ii) providing tools for monitoring, evaluating, and controlling social policies; (iii) promoting the council's contact and coordination with other actors in order to strengthen their organizational, communication, and negotiation capacities; and (iv) empowering the councils by strengthening their credibility, broad-based representation, and local advocacy. 3.5 The Instituto de Investigación y Desarrollo (IDES) was selected to train the councils. It organized four workshops (six hours each), totaling 24 hours (on average) in each of the councils. The training schedule began in April 2007 and ended in December of that year. DDC invited the participants, organized logistical aspects, and supervised the training. 3.6 An average of 28 people participated in each of the four training meetings. The presence of the organizations that formed the council was constant throughout the meetings. However, the persons representing the organization changed in some occasions. About 35% of the participants attended the four meetings. The gender distribution was 77% women and 33% men. Most participants came from urban areas and belonged to diverse organizations such as unions and cultural, sports, health, religious, and educational (secondary and tertiary) associations. 3.7 The diversity of the participants' profile, level of education, and occupation was positive for the experience. This plurality made it possible to seek and build adequate means of communicating and transmitting the concepts learned in the training to the rest of the community. 3.8 The methodology built on the participants' previous knowledge and incorporated new concepts and information. The timing and contents of the training were tailored to the specific conditions and needs of each council. Different participatory techniques aimed at promoting the exchange of ideas were used. At each session, a systematic summary of the main ideas discussed 15 was presented, and the conclusions were summarized in an aide-mémoire that was read at the beginning of the next meeting. 3.9 The training was extremely helpful for the consolidation of the councils. In most cases, council members had limited participatory experience, and therefore their integration as a team was uncertain. In addition, because the organizations they represented had a weak networking tradition, the training was instrumental in providing a platform for future coordinated activities. 3.10 Moreover, the training provided the councils with new concepts, methodologies, and tools that contributed to their empowerment. The concepts and issues analyzed, together with the knowledge gained on specific M&E methods and tools, allowed the councils to visualize and think of new possibilities and ways to influence local public policies. 3.11 Despite the positive outcomes mentioned above, it must be recognized that a one shot training is not the best way to sensitize and improve the skills of civil society. If more resources (budget, human resources, etc.) and time are available it is more effective to organize a training plan that combines workshops or lectures with hands-on training as done in the participatory evaluation pilot (see section below). In that experience councils were trained with a methodology that was adjusted to their needs and a final product was delivered. B. PARTICIPATORY EVALUATION PILOTS 3.12 MIDES selected the social councils of Rivera, Tacuarembó, and Paysandú to pilot the PM&E experience. The selection criteria for Rivera and Tacuarembó were based primarily on the fact that these councils had a longer prior working experience than that of their peers and they were sufficiently organized and aware of their role to undertake a new responsibility. In the case of Paysandú, the council was selected because its own working agenda already included the monitoring and evaluation of a local social program. 3.13 The main goal of the PM&E pilot was to strengthen the three social councils as local actors through the experience of evaluating a social program, from a rights-based perspective. The specific objectives of the pilot were to (i) develop a concrete participatory evaluation product, and (ii) contribute to building the public voice of the councils through the dissemination and use of the evaluation results. In addition to the responsibility of DINEM, IDES also provided technical advice for the implementation of the PM&E pilot. 3.14 The experience was developed in two stages. The first stage took place in Rivera and Tacuarembó from April to August 2007. Based on the lessons learned from these pilots, the experience was later developed in Paysandú (September­October). DDC was responsible for convening the council's members, taking care of other logistical issues, and supervising, in coordination with DINEM, the IDES's work. An average of 89% of the organizations that compose each council participated in the pilot. 16 3.15 The Rivera and Tacuarembó councils selected Youth Programs as the public policy topic to be monitored, while Paysandú selected Public Mental Health Services. The first two councils worked with youth programs prior to the pilot. Both councils chose to assess the Plazas de Deportes project, under the Ministry of Tourism and Sports (MTS). The Tacuarembó council also included the Centros de Barrios project, which is very similar to Plazas de Deportes but is led by the municipality and is managed by the neighbors8. Paysandú chose to evaluate the Psychiatric Admission Service (Servicio de Internación Psiquiátrica) at the Escuela del Litoral Hospital. 3.16 The questions selected by the councils were aimed more at evaluating rather than monitoring the selected projects. The data collection tools chosen in Rivera and Tacuarembó included: (i) a community scorecard; (ii) in-depth interviews with key informants; and (iii) participant observation in recreational spaces. The Paysandú council selected the following research methods and tools: (i) literature review; (ii) a survey of patients' relatives; and (iii) in-depth interviews with key informants. 3.17 The process of selecting the theme to be evaluated and defining the questions and methodology was instrumental to consolidate the councils as unified entities. Before that process, the participants viewed themselves as mainly belonging to an organization. After the process, they perceived themselves as social agents, as active members of a collective entity with the capacity to influence decision making on public social policies. 3.18 IDES was in charge of drafting research instruments, such as the survey and the community scorecard, with the councils' inputs. The Institute was also responsible for applying the instruments. The councils were in charge of observing the implementation of the selected project, analyzing the data, and preparing the conclusions and policy actions. 3.19 IDES processed all the information collected through the different methods and presented the systematized data to the councils for their analysis. The presentation was divided in two modules. The first described the general situation of the department with regard to the target population of each policy. The second module used all the information collected through the different techniques selected by the councils. The councils analyzed the data with different techniques (for example, Tacuarembó and Rivera used a traffic light analysis method; see Box 1) and reached a mutual assessment. 8Because Plaza de Deportes and Centros de Barrio are nearly the same project, the report only refers to Plaza de Deportes, but both projects are included in the analysis. 17 Box 1: Results of Tacuarembó's and Rivera's analysis of Plazas de Deportes project Rivera Tacuarembó Rivera Minas de Vicha- Caragua- San Paso de Tacua- Corrales dero tá Gregorio los Toros rembó Infrastructure Y R R R Y Y Y Facilities and inputs Y R R R R Inputs: Y R Facilities Y Functioning G Y R G R R Y Human Resources Y Y R R R R R Training Y R R R R R Y Notes: red = needs immediate improvement; yellow= needs intervention and action in the medium term; green = is being developed properly. Source: IDES (2008a). 3.20 In a closing meeting, each of the three councils drafted an action agenda to disseminate the evaluation results. The agendas of Tacuarembó, Rivera, and Paysandú included: (i) presenting the evaluation results to the inter- institutional board (an official venue that brings together representatives of the national ministries at the departmental level and the municipal authorities). Rivera's and Tacuarembó's agendas called for: (ii) publishing the results in the media so that the council can be better known publicly; and (iii) visiting different authorities to request their actions, depending on the resources needed (i.e., sports resources­Ministry of Tourism and Sports, Human Resources­MTS and local municipalities). The Tacuarembó council included an immediate action (writing a letter to the mayor with the results of the evaluation and the position of the council) regarding the irregular situation of a gymnasium.9 3.21 In general terms the PM&E pilot did not face important implementation problems. One obstacle that was rapidly resolved was the complexity of the terms and methodology used during the first training sessions. IDES immediately simplified the terminology to address the problem. Another minor obstacle in Rivera and Tacuarembó was that the organizations from the capital city of those councils did not have contact with adolescents, which complicated the application of the community scorecard in the capital cities. However, the card was widely applied in other localities. Finally, the PM&E pilot demanded more time for meetings that did not fit in the councils' regular schedules. 9 Paysandú's agenda also included the following commitments: (i) to provide continuity to the council, strengthening its character as a venue for civil society dialogue, and (ii) raising awareness in Paysandú's population about mental health problems. 18 3.22 Instead, numerous factors facilitated the implementation of the PM&E. It is important to highlight that: (i) all the councils have prior work experience; (ii) the councils assumed, with a high level of responsibility, the task and the working schedule; (iii) there was close coordination between the MIDES and IDES teams; and (iv) the inter-institutional board was very receptive to meeting with the council. 3.23 DDC followed the activities undertaken by the councils and confirmed that after the pilot the three councils undertook clear advocacy actions which yielded positive results. For example, the Rivera council is organizing weekly meetings with MTS authorities to implement the decentralization of teachers to the neighborhoods suggested by the council. To date, this decentralization has taken place in three neighborhoods and is planned to reach other two by the end of the year. In addition, the council suggested a new modality to work with the teachers; this is currently being implemented. 3.24 In Tacuarembó, the inter-institutional board created the "Recreational Project," which invites MTS authorities, the municipality, and the council to design an expansion and decentralization plan to provide localities lacking a Plaza de Deportes with recreational activities and facilities. To date, a new gymnasium was created in a locality suggested by the council. 3.25 In Paysandú, the council published an article in the local press and organized a meeting with hospital authorities. Several of the recommendations suggested by the council were immediately implemented. In addition, as a result of the meeting with the inter-institutional board, the issues addressed in the council's evaluation report have been referred to the Ministry of Health and the Ministry of Public Works. 3.26 At the same time, the pilot had positive spillover effects. For example, the preparation of the community scorecard raised the interest of several youth organizations in participating in the councils. The use of the community scorecard in certain Plazas motivated the authorities to immediately repair the facilities, before knowing the results of the evaluation. 3.27 The participatory evaluation pilot was undoubtedly instrumental in consolidating the councils as local agents with the capacity to influence the public policy process. The members of the council pointed out that the experience allowed them to work together in an organized manner, to reach agreements, and to exercise their voice. As a result of the experience, during national cabinet meetings held in Rivera, Tacuarembó, and Paysandú, the President and the Ministries met with the council and learned about the evaluation results.10 More importantly, they learned and experienced how to hold authorities accountable in an organized manner. 3.28 DDC staff concluded that the PM&E pilot was extremely positive for themselves and for the councils. The pilot was an opportunity to work with the councils on a specific project, and to show results in a short period of time. In 10The President of Uruguay periodically organizes weekly Ministerial Cabinet meetings in different localities outside of Montevideo. 19 addition, the experience was instrumental in showing that the political proposal to empower local actors is viable. In this sense, the experience was also an opportunity for the inter-institutional boards to interact with civil society, a nearly nonexistent practice in Uruguay. At the same time, the pilot allowed DDC staff to work in coordination with DINEM and establish the basis for future joint work. Nevertheless, due to time and human resources constraints, DINEM did not participate in all of the councils' meetings. 3.29 The participatory evaluation experience will be replicated with additional resources currently available to MIDES. The knowledge gained for DINEM, DDC, and the councils through the experience is a fundamental asset for replicating the experience in other councils or with the same councils under a different public policy. In fact, the Paysandú council is conducting a similar experience with a new public policy: the National Plan against Domestic Violence. 3.30 Nevertheless, DINEM and DDC lack the necessary human resources to replicate the participatory evaluation pilot. It is more a matter of the number of people and the time needed to organize and carry out the workshops than a matter of limited skills. Although each agency is increasing its staff; other regular activities keep them from being able to assign people specifically to this task. In this sense, future PM&E projects will necessarily need to outsource technical support to the councils, for which adequate resources should be budgeted. According to the implementation of the pilots, the average cost of each pilot was of approximately US $7,000, including administrative expenses, the cost of technical support provided to each council, travel expenses, the application of the participatory evaluation instrument (score card or survey), and the cost of the workshops. 3.31 The participatory evaluation pilots in Rivera, Tacuarembó, and Paysandú delivered several lessons for PM&E future experiences: (i) social councils involved in a PM&E experience must have passed a basic training process before starting the PM&E experience; (ii) technicians in charge of the training need to know the characteristics of the council beforehand in order to adjust the methodology, vocabulary, and format of the meetings; and (iii) DINEM staff should participate systematically in the experience in order to better learn from it and incorporate the councils' voices into the Integrated Social Information System. C. INTERNATIONAL WORKSHOP ON MONITORING AND EVALUATION 3.32 A two-day workshop took place in mid-May to share international experience on public monitoring and evaluation systems that supervise and hold accountable the implementation of social policy. The workshop was organized with the support of GLDN, which facilitated the participation of Washington, DC specialists via videoconference. A total of 50 participants from DINEM, DDC, and the Dirección Nacional de Políticas Sociales attended the workshop. 20 3.33 Day one was aimed at learning from international cases. First, MIDES representatives presented the Uruguayan experience and received comments from their international colleagues. Specialists from Chile, Colombia, and Mexico then presented their own countries' experiences and provided additional comments to the Uruguayan participants. 3.34 Lessons drawn from the Chilean experience about the tensions characterizing the interface between evaluators and executors were of particular interest. The evaluation as a tool external to the programs but internal to the government generates a series of tensions. 3.35 Complexity of the M&E system versus the timeliness and versatility of information needed for decision making. For example, the Chile Solidario program's M&E system needs to answer to more than ten sets of indicators, implying an enormous effort for the system, which does not necessarily deliver the timely, useful information needed for decision making. 3.36 Another tension observed is the cultural distance between evaluation teams and program managers. Those in charge of executing programs believe that the technicians evaluating a program have an incomplete view of that program. On the other hand, the evaluators feel that the program executors are resisting the evaluation. There is a need to build a bridge between these parties. 3.37 A third tension refers to the interface between evaluation and budget. One of the main questions that the evaluation asks is: why did the program fail to achieve the results that were defined as inputs, processes, and outputs? The answer to this question necessarily concerns the allocation of budget resources. However, the evaluation does not always include a section to explain the adequacy of the allocations and how the budget (not necessarily controlled by the program executors) was managed. 3.38 At the same time, some evaluation methodologies used for previous programs show limitations to understanding the new institutional arrangements evidenced in the social protection field. For example, some years ago the Chilean Social Protection system aimed to respond to the diverse poverty situations with specific programs. Today, universal policies, which are the pillars of the social protection system and are grounded in a social rights perspective, try to reach different types of poor people/households through the same policy but in a differentiated manner. The evaluation methodologies used for the previous approach cannot inform decision making for the new social protection system, in which policy makers need to understand the differentiated impacts that a policy has on different types of people/households at different age stages/moments. Chile Solidario's current evaluation system lacks the appropriate tools to respond to these needs. 3.39 After each country experience was presented, a World Bank specialist made a comparative analysis of the Chilean, Mexican, and Colombian M&E systems' development experiences. He highlighted among other interesting differences (i) the institutionalization strategies of each system; (ii) the use of the information that each system produces; and (iii) the legal framework that backs each system. The other differences are summarized in the table below. 21 Table 3: Main Differences in M&E Systems Colombia Chile Mexico Agency Responsible Planning Department Ministry of Planning Ministry of Economy Ministry of Planning Ministry of Economy National Evaluation Commission Linkages to budget Slow progress High High level of progress Monitoring Development Plan National Budget In Programs Presidential Goals Presidential Goals Juntos Program Chile Solidario Evaluation Impact, Process, and Impact and Process Impact, Process Quick Evaluations Evaluations Consistency, Strategic Expenditure Review and Complementary Evaluations Accountability Congress and Citizens Congress Congress Sectorial Systems Social Social Social Orientation Consolidation of M&E Results-based budget SED-PBR system Human resources-based budget Starting date 15 years ago 17 years ago 6 years ago President's role High Low Increasing Source: Presentation by Manuel Fernando Castro Quiroz, Uruguay, May 19, 2007. 3.40 Based on the comparative analysis, the workshop concluded that there is not a single successful M&E system, but successful M&E elements. Among these, the following elements were pointed out: (i) the importance of achieving a good balance between the system's technical and political components. The system will be unviable without political support. In this sense the Chilean experience has the unique feature of including different thematic commissions with the representation of civil society actors interested in producing and publicizing monitoring information; and (ii) the need to create a demand for M&E information. If no one uses the information for policy making, the system is unnecessary. The best sign of the system's good performance is the use of the data produced. 3.41 A third successful element for the institutionalization and proper functioning of an M&E system is good, strategic coordination among the different information systems existing in a country. The agency leading the M&E system must control the quality of the information produced, be able to request information from its peer agencies/ministries, and have the capacity to establish external incentives in order to obtain better results from the programs. The coordination problems among the Ministry of Economy or Planning, the Presidency, and the other ministries evidenced in Colombia and Mexico, have resulted in serious problems for the proper performance of M&E systems. 3.42 There is also a need for institutional arrangements in order to ensure objectivity, quality, and the use of M&E information. A serious M&E system should have clear rules regarding planning, implementation, and the use of evaluations. Administrative processes for hiring external consultants should be efficient. Furthermore, there should be clear arrangements regarding the financial mechanisms and budget allocated for the M&E system. Ideally, clear linkages between M&E results and performance-based budgeting and policy adjustments should be defined. 22 3.43 Finally, a good M&E system should be simple and combine a proper balance of different evaluation and monitoring methodologies. It is important to have a few simple indicators and goals that allow project progress to be tracked. Sophisticated evaluations, such as impact evaluations, are very powerful and rigorous but are also very complex instruments that are not suitable for making quick decisions. Quicker, simpler methodologies such as process evaluations or monitoring alone are better suited for policy making. At the same time, it was highlighted that evaluation and monitoring are not replacement functions. 3.44 Other important lessons were defined as particularly relevant for the Uruguayan M&E system. First, building an M&E system is a long-term effort that requires a gradual approach and continuity. Second, it is crucial to develop political support coalitions that accompany the process. In this sense, all participants agreed that the Ministry of Finance must be included in the development of the UY M&E system. MIDES staff concurred with the workshop's suggestion to develop pilots in order to reduce risks and show ongoing progress in order to keep other actors interested in the process. Third, the system must have clear objectives: what is (are) the purpose(s) of the M&E system? Finally, the system should be integrated with other systems throughout Uruguay. 3.45 The second day consisted of a technical workshop on Impact Evaluations for a smaller audience. The DINEM technical team, international invitees, and Bank staff composed the audience (20 participants). The main objective of this day was to present experiences and lessons learned from the evaluation of PANES in Uruguay, followed by a discussion of international best practices to inform MIDES's strategy for establishing a permanent social policy impact evaluation system. 3.46 The workshop was highly satisfactory from different perspectives. For DINEM staff members the workshop was very useful because it allowed them to learn from the practical experiences of other countries. They particularly valued the fact that the speakers stressed the difficulties experienced during the implementation of their M&E systems. These difficulties will be critically considered in order to avoid mistakes or delays. For the international speakers, the workshop was a valuable opportunity to share opinions, learn about the planning approach (Colombia) and the budget approach (Chile), provide specific advice to MIDES based on their own experiences, and reflect on the state of the art of M&E in the Latin American region. Bank staff felt that the workshop gave experts from other Latin American countries, MIDES, and Bank staff the opportunity to learn from each other and help MIDES to refine the design of its M&E system. D. IMPLEMENTATION OF THE INTEGRATED SOCIAL INFORMATION SYSTEM 3.47 Based on the TOR defined in Phase I, the Etriek consulting firm was hired to develop the Integrated Social Information System (ISIS). ISIS was built considering the experience of other Latin American countries and constitutes one 23 of the most advanced systems in the region. The construction process consisted of a joint effort by DINEM and Etriek. 3.48 DINEM and Etriek agreed to develop the project in different stages. With the UY NLTA project, the first stage will be completed. In this initial stage MIDES processes all the information, which is subsequently uploaded in the system. Future stages will take the information directly from the original sources and calculate the indicators automatically. A forum to discuss social policy issues online is also planned. The system will also include more complex functions (i.e., possibilities of providing spatial information using a georeferencing function) and interactive actions. All these functions will help to position MIDES as a lead actor in the social sector. 3.49 The system's main objective is to offer citizens a wide variety of clear, user- friendly, and relevant information about the current and recent evolution of social welfare in Uruguay. In addition, the system is aimed at providing information to government authorities at different levels, the press, and members of civil society organizations (CSO), among others. For this purpose, the system has an online portal that can be freely accessed and allows users to set personal preferences according to their needs. These preferences (see image below) are shown each time that the user logs into the system. The system also alerts registered users by e-mail if the indicators selected in their preferences have been updated. 3.50 The indicators can be consulted through five different entries: (i) by theme (education, health, family, social security, among others); (ii) by modules 24 (Millennium Development Goals and the Plan de Equidad); (iii) by welfare concepts (living conditions, government interventions to modify living conditions); (iv) by geographic parameters (a map of Uruguay allows all indicators available for each of Uruguay's departments to be accessed); and (v) by alphabetical order. 3.51 A filter helps the user to separate the information. Depending on the indicator, the information can be separated by gender, age cohorts, and territorial status (urban/rural). The system allows the information to be viewed by department, comparing it to the national value, in a map or in a graph (the user can select the type of graph), or by exporting it to Excel. Each indicator has a methodological label. Moreover, the system gathers a series of useful documents. These papers are currently grouped by three themes: poverty, housing, and income. They are official documents or papers related to government programs, authorized by MIDES. 25 3.52 The portal is expected to be online by the end of 2008. The remaining steps include: (i) processing and uploading information for the main indicators. For this purpose, a network of information providers will be formed together with the National Institute of Statistics (INE). The providers will be in charge of updating the information and creating new indicators. This network will also include university technicians who will be able to view the process of constructing new indicators and make comments; (ii) a section for current social programs. To date, DINEM has identified 174 programs with 20 referents. The idea is to provide the referents with a passkey so that they can enter information on their programs; and (iii) drafting guidelines with the procedures needed to check the accuracy and validity of the information and the time frame for updating the indicators. 3.53 The construction of the Integrated Social Information System contributed to strengthen inter-institutional and territorial policy dialogue and review best practices. This forum for dialogue is formed mainly by MIDES, OPP, and BPS and was highly active during the design of the Plan de Equidad. For the construction of the ISIS, BPS provided a very large amount of information. MIDES's capacity to enlist other agencies and to lead this forum as well as the construction of the SIIS, is remarkable if one considers that during this process MIDES was also building its own identity. As a result of this effort, OPP could see that MIDES is taking the lead in M&E and with its example is promoting a culture of evaluation among the ministries. Thus, OPP and MIDES agreed to include the ministry as a beneficiary of the Institution Building Technical Assistance Loan (IBTAL) that the Bank is supporting in Uruguay. 26 4. UY NLTA Evaluation: Risks, lessons learned, and recommendations 4.1 None of the risks envisaged in the project design for Phase II materialized. MIDES and Bank staff followed the mitigation measures and worked hard to prevent and avoid problems. Table 3 summarizes the anticipated risks and the mitigation measures, and provides a brief explanation of why they did not materialize. Table 4: Risks and Mitigation Measures - Phase II Risks Mitigation Measures Materialized? Potential lack of interest To develop a broader base of No. BPS, INE and other agencies on the part of other support within the UY cooperated with MIDES without ministries and public Government, the team will problems. institutions in engaging promote greater dialogue among MIDES could position itself as one in the proposed MIDES, BPS (Banco de of the lead agencies in social activities. Previsión Social), the Ministry of research. Historical agencies in Economy, and other key actors in charge of this function applauded order to build consensus on the MIDES for the quality of its objective and reach agreement on publications. the UY NLTA process. On the other hand, there are some difficulties with other public institutions such as schools or public companies that have autonomy for providing data. MIDES is working with an incentive scheme to obtain their collaboration. The development of a The team will coordinate with No. MIDES designed a smart road map for the various ministries and strategy that consisted of working establishment of the intergovernmental offices first with the ministries and ISIS may encounter involved in social assistance, the agencies that were convinced that strong institutional OPP (Office of Planning and the ISIS was key for policy making. resistance on the part of Budget), and other public offices Then, MIDES built an ministries and other engaged in monitoring activities. interest/demand for data and public institutions that It will suggest that the committed to process it. In this way, control social assistance government establish clear agencies provided the data that they data. This is likely to incentives so that the information had and obtained new information. result in continued is released in a complete and The UY Government and the Bank institutional overlaps in timely manner. Relevant agreed that MIDES will be one of M&E functions international experience should the ministries which will benefit associated with weak be presented to the UY from IBTAL. inter-institutional Government so that it may coordination in the generate a dialogue among key implementation and use stakeholders (MIDES, BPS, of the M&E system. National Institute of Statistics, and others). Resistance on the part The team will collaborate with No. MIDES, through DDC and of MIDES to engage the MIDES and IDES in the DINEM, participated actively in the social councils in the development of training activities PM&E pilot. development of for the social councils that innovative PM&E include international best mechanisms. practices shared with them during Phase I. Peer-to-peer learning 27 with other government institutions in other Latin American countries will be promoted as well. Expansion of PANES To advise DINEM on No. There will be a new survey in coverage may create coordinating closely with MIDES 2009 to envision PANES's long- problems in measuring management to moderate any term impacts. impact among modifications of the baseline due comparison groups, to changes in the eligibility particularly when criteria. members of the comparison groups become program beneficiaries. Commitment on the part The team will seek the client's No. On the contrary, the of client may not be the necessary explicit support to relationship was excellent, which same as in Phase I. carry out the proposed Phase II motivated MIDES to request activities contained in this report. reimbursable support from the Bank and to continue working with the same professional team. A. LESSONS LEARNED FOR FUTURE BANK OPERATIONS 4.2 The Bank's openness to listen and to adapt to the client's needs is a determining factor for establishing the basis of positive, joint work. The UY NLTA project was conceived as joint project from the beginning. Bank and MIDES staff designed the original project that was later adapted to the client's needs, respecting Bank rules. The constant dialogue, the respect for the client's view, and the capacity to respond to expected and unexpected demands contributed to the success of the UY NLTA project and motivated MIDES's interest in continuing to work with the Bank. 4.3 Available technology should be better used to increase the opportunities for sharing international best practices. One of the aspects of the UY NLTA project that MIDES valued the most was the possibility to learn from other countries' experiences. The Bank could facilitate this exchange by using the technology currently available in country offices (i.e., videconferencing). If resources (money, time, etc.) are available, training trips to learn in situ about other countries' experiences should be encouraged. B. LESSONS LEARNED FOR THE IMPLEMENTATION OF M&E SYSTEMS: 4.4 A government's commitment to make decisions based on accurate information is key to supporting the construction of M&E systems. Uruguay's current government has supported the production of statistics and information. More importantly, the government has made important decisions (i.e., expansion of family allowances) based on evidence. The UY NLTA has had the support of high-level authorities to implement the SIIS, as well as the support of local authorities to take into account the information produced by the councils. 28 4.5 The partnership with academia is an important asset contributing to the accuracy of the information produced and for installing capacity within the government. The current administration has identified the Universidad de la República as a key partner in the production of information. During the UY NLTA, project multidisciplinary teams were formed with specific responsibilities: while the university was in charge of producing the data independently, MIDES staff coordinated the work, participated in specific activities (i.e., data collection in the field), and supervised the work. In this way, both teams complemented each other's work and learned from each other. 4.6 The constant technical exchange between DINEM and INE staff is key to increasing the accuracy of the data processed and unifying methodological criteria. INE and MIDES have signed a formal cooperation agreement. Under this umbrella INE has provided data that later was processed by DINEM. INE has validated that information. In addition, DINEM permanently consults and coordinates next steps with INE. It is advisable that this cooperation continues to be strengthened through a specific teamwork agreement for the implementation and maintenance of the M&E system. 4.7 Although the objectives11 of the UY NLTA project were achieved, there is still work to be done: (i) PANES's immediate impact was evaluated, but its long-term effects will be assessed in 2009 with data that is currently being collected; (ii) the information produced under the project and the initiatives taken were critical inputs for informing a dialogue about MIDES's future role as a lead authority responsible for the coordination and oversight of social policy in Uruguay; and (iii) the products generated by the project together with other MIDES products helped to better position MIDES as the main actor evaluating social policies. In order to fulfill this role, MIDES needs to continue strengthening its M&E capacity. 4.8 The project's expected outcomes were also achieved: (a) As a result of the project there has been greater efficiency in the implementation of social projects. By consolidating databases and analyzing the data, some projects were modified. Most importantly, a cultural change has begun: there is a demand for information to make decisions/changes in current social programs. (b) The participatory evaluation pilot undoubtedly contributed to increasing transparency and strengthening accountability. Thanks to the pilot the councils are recognized as local actors with the ability to exert influence. (c) As a consequence of the pilot, civil society organized in the councils is more engaged in public sector reform. (d) The Plan de Equidad was designed taking into consideration the evaluations of PANES and the PSIA. 11 As stated in point 1.3, the objectives were to: (i) evaluate the impact of PANES's social emergency programs; (ii) inform a dialogue about the future role of social emergency programs and of MIDES as a lead authority responsible for the coordination and oversight of social policy in Uruguay; and (iii) strengthen MIDES's capacity to assess, monitor, and evaluate social policy in Uruguay, with considerable weight given to participatory approaches to monitoring and evaluation. 29 C. RECOMMENDATIONS TO STRENGTHEN MIDES'S LEADING ROLE AND IMPROVE WORLD BANK COOPERATION: 4.9 A clearer definition of responsibilities in the Plan de Equidad is needed. MIDES is much better positioned to be the lead authority responsible for the coordination and oversight of social policy in Uruguay. Nevertheless, MIDES is currently participating with other agencies, incorporated in NCSP, in the execution of the Plan de Equidad. These shared responsibilities represent a more complex challenge than the one assumed in the execution of PANES. The government must still define an evaluation strategy for the Plan de Equidad with a clear distribution of responsibilities. 4.10 MIDES needs to define a series of goals and objectives that can be tracked with the SIIS. Goals and objectives at the national and regional levels should be discussed and defined. Once defined, a series of indicators should be identified to track the progress of these goals and objectives. The indicators should include the voice of civil society. 4.11 The SIIS needs to be strengthened by refining the available information and incorporating new information. As previously mentioned, the final SIIS will include information about the programs and more details for each of the indicators (i.e., gender, area). This will imply an important data collection and processing effort because most of the information is not readily available. In addition, the guidelines and procedures for updating and checking the accuracy of the information need to be drafted. 4.12 A plan for promoting the demand for and use of information by civil society organizations should be defined. Civil society organizations need to be better trained in this regard. In general terms, civil society organizations are not accustomed to demanding and using official information. They are more willing to contribute to, partner with, or benefit from the government. At the same time, many civil society leaders have gone on to work in the government. 4.13 Before replicating the participatory evaluation experience, MIDES needs to define various strategic and methodological aspects. On the one hand, there is a definition pending on how the voices of the council will be incorporated into the SIIS. On the other hand, a methodology that can be replicated throughout the country needs to be defined. 4.14 A dissemination of the main conclusions and lessons learned of the International Workshop on M&E within MIDES and to other governmental agencies should be encouraged. Such dissemination should focus on the main lessons learned concerning challenges faced by similar M&E systems in other countries and the type of successful responses they have been able to implement. 4.15 The Government of Uruguay is moving forward with this pending agenda while proposing specific follow-up actions. On July 21, the Minister of Social Development formally requested the Bank's financial and technical support to continue with the implementation and strengthening of the M&E system. Because the Bank has recently approved the Institution Building Technical 30 Assistance Project (IBTAL) to support the institutionalization of M&E in different government agencies, the Government of UY and the Bank agreed to allocate funds from the current IBTAL project for MIDES. Bank teams from the Social Development, Social Protection, and Public Sector units are currently working together to design interventions in support of MIDES. 4.16 In addition, the ongoing dialogue between DINEM and the UY NLTA team has helped to identify priority areas for support. On August 21, 2008 DINEM submitted a document to the Bank, expressing its interest in receiving Bank support in specific areas. DINEM and the UY NLTA team discussed the document identifying the following agenda for further exploration: a. Strengthening MIDES's evaluation capacity. DINEM will play a key role in evaluating the Plan de Equidad. It is interested in assessing the impacts of two cash transfer programs and of at least two structural programs included in the Social Protection Matrix and the Tributary Reform of the Plan de Equidad. It will also be conducting a third-round survey of PANES beneficiaries in order to assess PANES's long-term impacts. A Social Protection pilot survey has recently been conducted with IDB support. This survey could be instrumental in assessing the results of PANES and the Plan de Equidad. It is not yet clear whether the IDB will continue supporting the survey. The World Bank could assist in installing a permanent capacity to outsource the surveys on a regular basis; evaluations would be conducted jointly by external consultants and MIDES staff. b. Strengthening MIDES's monitoring capacity. DINEM currently uses the SIIS which collects information on a series of indicators, including output and outcome. MIDES expects to receive Bank support to expand the territorial and chronological coverage of the indicators, the themes included in the SIIS, its maintenance, and software updating. Nevertheless, the system does not include input indicators, which are crucial for a monitoring and public management system. Bank support could also be aimed at completing the SIIS in order to make it functional for monitoring programs and for public management decisions. At the same time, MIDES would like to receive support to continue developing a range of social programs that include information on available social services throughout Uruguay, linkages among these services, coverage, and information for citizens on how to access these services. This range of programs will allow MIDES to define national and departmental goals and their corresponding indicators, which will be tracked by the monitoring system. The monitoring system will clearly help to strengthen MIDES's role as the coordinator of social policy. c. Strengthening MIDES's capacity to involve civil society in policy making. MIDES would like to replicate the participatory evaluation pilot and develop other participatory monitoring experiences, respecting the agendas and development of the councils. For this purpose, it would like to receive the Bank's technical advice and financial support. In addition, the GOU is moving forward with the project of implementing the Centros de Atención al Ciudadano (single public services windows), where citizens can conduct various transactions. MIDES would like to collaborate with this project by making the range of social programs available to the Centros and providing citizens with 31 information on their social rights and on programs and policies that could help them to exercise those rights. This collaboration would require the development of specific software and information systems. MIDES would also like to explore how these "windows" could include channels that enable social accountability. d. Strengthening MIDES's capacity to exercise its role as the articulator of social policy in Uruguay. In order to articulate social policy in Uruguay, MIDES needs to be able to perform national diagnostics of the main social problems affecting the country. MIDES has expressed its interest in the Bank's support for developing its analytical capacity to conduct national and territorial assessments of different social issues, especially middle-school dropout and youth drug consumption. A mechanism to conduct these studies either by DINEM staff or by external consultants needs to be developed. MIDES would also like to receive support for making simulations of the distributive effects of alternative reforms in order to provide OPP with crucial information for Uruguay's development strategy. e. Strengthening MIDES's capacity to exercise its role as the territorial coordinator of social policy in Uruguay. MIDES is implementing its decentralization strategy by supporting the development and operation of the Mesas de Coordinación Inter-institucional. MIDES is interested in receiving the Bank's support to monitor the activities conducted by the Mesas (i.e., measuring results and understanding which factors are influencing these results) and to aid in their strengthening and capacity to play a strategic role at local level. 4.17 The five themes mentioned above are potentially aligned with Component 2 of the IBTAL project "Improving Institutional Planning, Monitoring, and Evaluation." Nevertheless, it is advisable that the final content and delivery arrangements of the technical support provided through the IBTAL to MIDES are the result of an identification process involving all key stakeholders, including the Ministry of Finance, the OPP and MIDES in order to adequately respond to the strategic requirements of a sound M&E system. 32 5. Bibliography Arismendi M. (2008) "Desde el MIDES construyendo ciudadanía para profundizar la democracia" Discurso publicado en Uruguay tu país, año 2 número 9. Banco Mundial (2008) Del PANES al Plan de Equidad, Nota Técnica, mayo 2008. Available at http://www.bancomundial.org/uy, in the section "Publicaciones e informes." IDES (2008) "Experiencia Piloto de Evaluación Participativa: Informe final de implementación de capacitación de los Consejos Sociales." IDES (2008) "Experiencia Piloto de Evaluación Participativa: Informe de implementación de capacitación en dos consejos." IDES (2008a) "Experiencia Piloto de Evaluación Participativa: Informe de implementación en Rivera y Tacuarembó." IDES (2008b) "Experiencia Piloto de Evaluación Participativa: Informe de implementación en Paysandú." MIDES (2008) "Plan Nacional de Equidad," available at www.mides.gob.uy Rofman et al., 2007 "Income Transfer Policies in Uruguay: Closing the Gaps to Increase Welfare. The World Bank: Washington, DC. World Bank (2007) Project Appraisal Document for the Institution Building Technical Assistance Project. The World Bank: Washington, DC. 33