How will we know Millenium Development Results when we see them? Building a Results-Based Monitoring and Evaluation System to Give us the Answer Africa Region Working Paper Series No. 66 Abstract Results Based monitoring and evaluation (M&E) is a powerful public management tool to help policy makers track progress and demonstrate the outcomes and im- pacts of a given policy, program, or project. In the context of MDG's having results based M&E system is important to assess whether government's are effectively using resources and making appropriate policy choices to achieve progress towards MDG achievement. Developing countries need as good as or better information than the more affluent countries to make decisions for they can least afford to put in place strategies that have weak implementation, have no discernible results, or are unfocused. Thus, the information and analysis that can come from a results-based M&E system goes to the heart of the matter for poorer countries--how will they know results when they seem them and how can they gather sufficient information to continually ensure they are using their scarce resources most appropriately in pursuit of their MDG's. The introduction of results based M&E strategies in devel- oping countries is still in its infancy.As these governments move forward with their own strategies, they are trying new and innovative approaches that are enriching the pool of data and experiences. There is no blue print for moving forward and the challenges faced by developing countries are many. This paper discusses some ap- proaches tried in a number of countries to build MDG results based M&E systems and some of the challenges encountered along the way. The Africa Region Working Paper Series expedites dissemination of applied research and policy studies with potential for improving economic performance and social conditions in Sub-Saharan Africa. The Series publishes papers at preliminary stages to stimulate timely discussion within the Region and among client countries, donors, and the policy research community. The editorial board for the Series consists of representatives from professional families appointed by the Region's Sector Directors. For additional information, please contact Paula White, managing editor of the series, (81131), Email: pwhite2@worldbank.org or visit the Web site: http://www.worldbank.org/afr/wps/ index.htm. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s), they do not necessarily represent the views of the World Bank Group, its Executive Directors, or the countries they represent and should not be attributed to them. Working Paper Series No. 66 How will we know Millenium Development Results when we see them? Building a Results-Based Monitoring and Evaluation System to Give us the Answer Jody Zall Kusek Ray C. Rist Elizabeth M.White May 2004 The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s), they do not necessarily represent the views of the World Bank Group, its Executive Directors, or the countries they represent and should not be attributed to them. Authors' Affiliations and Sponsorship Jody Z. Kusek Regional Coordinator for Results Monitoring & The Evaluation, World Bank Group E-mail: jkusek@worldbank.org Ray Rist Senior Evaluation Officer, The World Bank Group E-mail: Rrist@worldbank.org Elizabeth M. White Sr. Results Management Specialist, The World Bank Group E-mail: ewhite1@worldbank.org This report was prepared as part of the Bank's efforts to strengthen country capacity to manage for development results. The findings were derived from a series of studies in 10 developing countries (middle income, transition and lower incomes countries) and from the authors' experience in building national results based monitoring and evaluation systems. We thank the country team members, PREM (Poverty Reduction and Economic Manage- ment Network), and peer reviewers for their valuable contributions. The information repre- sents the views of the authors. Contents Introduction 1 Monitoring and Evaluation:What's It All About? 2 Results Based Management and the International Experience 6 Challenges Facing Developing Countries 8 Moving the Results Agenda Forward: Strategies to Consider 11 A Concluding Comment 17 Figures, Boxes, and Tables Figure One: Illustrative Logic Model for one MDG Box 1: The Importance of Measuring Organizational ­ Level Performance Box 2: Readiness Assessment Key Questions Box 3: Demand Driving the Supply of Information Box 4: Building an Environment the Expects and Values a Focus on Results Introduction It is now understood as a fundamental tenet of successful development that there is a need for effective and efficient public sector governance. Whether at the local, regional, or national level, good government is essential to build- ing the rule of law, delivering on the services citizens need, and promoting equitable and sustainable socio-economic development so as to improve the well being of those within its boundaries. But saying that governments should deliver effective programs does not necessarily make it happen. Indeed, governments all over the world are now grappling with internal and external demands and pressures for improve- ments and reforms in their own public management. Whether the calls are for greater accountability and transparency, enhanced effectiveness of de- velopment programs in exchange for aid, or demonstrable results of political promises made, governments are now reacting to the very visible and in- creasing pressures from their many stakeholders to show real results from their actions. As recently noted: "The clamor for greater government effec- tiveness has reached crisis proportions in many developing countries where the state has failed to deliver even such fundamental public goods as prop- erty rights, roads, and basic health and education." (World Bank: 1997:2) Enter here the MDGs. The MDGs reflect a new emphasis on and expec- tation of results. They enshrine poverty reduction as the overarching mission of development. Hunger eradication, empowerment of women, improvement of material and child health, prevention and cure of contagious diseases, and promotion of environmental sustainability represent complementary objec- tives that electorates in rich and poor countries alike can readily grasp.(World Bank, OED 2002:1) But how do countries go about achieving the objectives represented in the MDGs? The entire development community will have to adjust to a new development paradigm that emphasizes a results-orientation, ownership of domestic policies, partnerships, and staying the course for the needed long term. Results-based monitoring and evaluation (M&E) is a powerful public management tool. It can and should be used in the context of the MDGs--as well as in all public sector management--to help policy makers track progress and demonstrate the outcomes and impacts of a given policy, program, or project. Results-based M&E differs from traditional implementation-focused M&E in that it builds on the emphasis on inputs, activities, and outputs to a sharp focus on outcomes and impacts. Results-based M&E systems are de- signed to address the "So what?" question. So what that all the inputs have been gathered, so what that all the activities have taken place, and so what that the outputs from these activities have been counted? Staying busy is not the same as achieving results. Stakeholders inside and outside of governments are no longer interested only in how well the inputs, activities and outputs are being managed. The concern now is with the results. The questions are coming one after another: Have policies, programs, and projects been able to achieve their intended outcomes? How is it known whether an initiative is or is not making progress 1 How will we know Millenium Development Results when we see them? towards its targets and goal? How are problems in moving towards objec- tives identified and resolved? How is progress to be measured? How can success be distinguished from failure? The challenge is to find credible ways of answering such questions. These questions while relevant for all governments, take on an additional urgency when placed in the context of the MDGs. The MDGs currently seem out of the reach of many poorer countries. Thus these countries in particular face the daunting challenge of using what scarce resources they have in the most efficient and effective manner they can. They, in fact, need as good as or better information than the more affluent countries for they can least af- ford to put in place strategies that have weak implementation, have no dis- cernible results, or are unfocused. Thus the information and analysis that can come from a results-based M&E system goes to the heart of the matter for poorer countries--how will they know results when they see them and how can they gather sufficient information to continually ensure they are using their scarce resources most appropriately in pursuit of their MDGs.Are gov- ernment programs and services helping to produce healthier children? Are water supplies clean and adequate? Does the primary education system meet the standards of the most developed countries? These are all questions being asked by citizens, politicians, aid organizations and others. In order to reli- ably answer these questions, governments know that they must begin to set measurable goals and build new institutional capacities to measure the re- sults from government programs. Building and sustaining a results-based M&E system is not easy. Such an effort requires time, energy, political will, organizational commitment, and resources. But it is doable. Once such a system is built, even in a rudi- mentary way, the challenge is there immediately to sustain it. Like any gov- ernance function, it requires continued attention and support or it will erode. Many developed countries, especially those in the OECD, have had ten- plus years of experience in designing and building M&E systems.Any num- ber of developing countries are also now starting to deploy M&E systems as a key public management tool. The experiences of developed countries are instructive and can provide important lessons for developing countries. And for their part, developing countries face a variety of unique challenges as they try to answer the "So what?" questions that are coming at them now from so many angles. Monitoring and There is tremendous power in measuring performance. Giving credible an- Evaluation: swers to the "So what?" question addresses the accountability concerns of What's it all stakeholders, gives public sector managers information on their movement about? towards achieving stated targets and goals, and provides substantial evidence as the basis for any mid-course corrections needed in policies, programs, or projects. 2 Monitoring and Evaluation: What's it all about? Building an M&E system essentially adds the fourth leg to the gover- nance chair. Typically and traditionally, governments have built budget sys- tems, human resource systems, and auditing systems as the three legs of a governance system. But what has been missing has been the feedback sys- tem on the outcomes and consequences of government actions. This is what building an M&E system brings as an additional public sector management tool. The OECD ( 2002a) has defined monitoring and evaluation as follows: Monitoring is a continuous function that uses the systematic col- lection of data on specified indicators to provide management and the main stakeholders of an on-going development interven- tion with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds. Evaluation is the systematic and objective assessment of an on- going or completed project, program, or policy, its design, imple- mentation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact, and sustainability. An evaluation should provide infor- mation that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipi- ents and donors. In juxtaposing these two definitions, it is immediately evident that they are distinct yet complementary. Monitoring gives information on where a policy, program, or project is at any given point in time (and over time!!) vis a vis its targets and outcomes. It is, in this context, descriptive in intent. Evaluation gives evidence of why targets and outcomes are or are not being achieved--it seeks to address issues of causality. Of particular emphasis here is the expansion of the traditional M&E function to focus explicitly on out- comes and impacts. Consider the following chart as indicative of the manner in which the monitoring and evaluation of the MDGs will have to include not only the traditional implementation focus, but also a results focus (Figure 1). This example of a health policy issue for children suggests the manner in which being able to assess the results of one MDG's impact (reduction in mortality rates of children under 5 years old) and one related health outcome (in- creased use of Oral Rehydration Therapy (ORT)) will need appropriate indi- cators at each stage of a logic model. To be able to document and manage towards outcomes and impacts from the outputs of policies, programs or projects, there will need to be an ability to see how the evidence from an earlier stage contributes substantially to the next stage (are the media cam- paigns resulting in improved awareness?)--eventually culminating in the 3 How will we know Millenium Development Results when we see them? documentation of the impacts of an initiative. Against this outcome, it is necessary to set realistic and useful indicators which can be measured and monitored to assess progress. Developing indicators is not simple. They must be reliable, valid, comparable as well as cost effective and useful. In a devel- oping country context the difficulties of data collection must be considered, and where data collection is not feasible, alternative indicators or proxies should be used. Most importantly, indicators should be specific to the goal to be monitored and useful to all levels of monitoring. M&E should be undertaken throughout the life cycle of a project, pro- gram, or policy, as well as on an ex-post basis. It also can be undertaken at local, regional and national levels of government. So whether one thinks of 4 Monitoring and Evaluation: What's it all about? M&E vis a vis levels of administrative complexity (project to program to policy) or geographically, the applications are evident--though they need not be identical. As two of the authors here have noted elsewhere: "The spe- cific indicators may necessarily be different (as the stakeholders' needs for information will also be different at each level), the complexity of collecting data will be different, the political sensitivity on collecting the data may change, and the uses of the information may change from one level to another."(Kusek and Rist, 2001.17) A functioning M&E system provides information that is useful both in- ternally and externally. The internal uses come as the information from the M&E system can be a crucial management tool for the public sector man- ager who is managing to achieve results and accomplish specified targets. Information on progress, problems, and performance are all crucial to the public manager striving for results. Likewise, the information from an M&E system is crucial to those outside of the public sector who are expecting results, wanting to see demonstrable impacts from government action (and tax monies), and for building trust that the government is, indeed, striving to better the life of its citizens. Box 1: The Importance of Measuring Organizational-Level Performance As seen in Figure 1, monitoring progress towards the MDG's requires that informa- tion is derived from all levels of results in the logic model at different time frames and for the needs of different stakeholders. One of the more difficult levels for indicator development is at the intermediate outcome level where institutional capacity or per- formance improvement is often the desired result. A common mistake is to measure completion of activities (number of health professional trained) as opposed to im- provements in performance (health professionals providing ORT services). By focus- ing on outputs, actual improved institutional performance is assumed and if it is not obtained may jeopardize achieving the expected outcomes (improved use of ORT), and ultimately the associated MDG (reduction in child mortality). So what does this mean in a government's results based M&E context? As gov- ernments seek to align the expenditure framework to policy outcomes, measuring the organization's performance in support of achieving outcomes is important. The effi- ciency of service delivery, the quality of program and policy implementation, and the effective management of resources are just a few examples. In the Philippines, the government is at the early stages of defining organizational-level indicators for major output, processes and citizen feedback against which performance decisions can be made. These are envisioned to meet both external and internal needs--reporting to citizens, in future resource allocations, and by agency management to achieve results. 5 How will we know Millenium Development Results when we see them? Results-Based In addition to understanding the rationale of "why" governments are moving Management and to an increased focus on results and "what" a results focused M&E system the International is, there is a need to understand "how" governments are beginning to move Experience towards a results focus in public management. For the past two decades, governments in developed countries and, more recently, developing coun- tries have been working to improve the quality of their public services. These governments have embarked on long-term reform agendas to change the culture from one that focuses on input management to output and outcome management1 with the goal of being able to demonstrate that public expen- ditures are not only being efficiently spent, but effectively used. Over the last two-plus decades, we have seen many developed countries --including Australia, New Zealand, Canada, the United Kingdom, and the United States--making progress in implementing new practices to better account for the results being achieved with public expenditures. There are also several developing countries moving in this direction--the Philippines, Thailand, and Costa Rica--to name just three. The strategies used to intro- duce results-based management, have varied across these countries; how- ever, there appears to be a number of similar elements that contribute to a successful shift to a results-based culture. Among these key elements are the following seven: A clear mandate for making such a shift; The presence of strong leadership, usually through a strong cham- pion or champions at the most senior level of government; The use of reliable information for policy and management deci- sions; Economic pressures and other incentives for change (often, a con- cerned citizenry or the need to reduce the cost of burdensome civil service payrolls); Clear links to budget and other resource allocation decisions; Involvement of civil society as an important partner with govern- ment; and Pockets of innovation that can serve as beginning practices or pilot programs. As recently reported by OECD, reforms are maturing in many OECD countries, mainly in the English and Scandinavian-speaking parts of the world. Theses countries have substantial experience with introducing a results fo- cus, and are either implementing or contemplating a new generation of re- 1 Results based management is used interchangeably with outcome based management, or performance-based management. For consistency, this paper uses the term results based management. 6 Results Based Management and the International Experience sults reforms. Other OECD Member Countries, notably those with strong legal traditions, such as France and Germany, have only recently embarked on such reforms. In France, reform efforts were boosted in 2001 with the enactment of the new results-focused budget law (OECD: 2002.6) In Ger- many, most reforms are taking place at the local level. In Japan and Korea, reforms are centered more on ex ante and ex post policy evaluation than on results-focused management and budgeting, but reforms are aiming at in- creasing an outcome focus in the public sector. Developing countries on the other hand have been to be more cautious in their effort to introduce and implement any form of results-based manage- ment system. This could be due to the lack of resource certainty, inadequate system maturity, and/or different cultural practices. Developing countries are often characterized by financial and administrative systems that are still in their early stages of development and may not possess the level of matu- rity and capacity to implement the technical aspects of a results-based man- agement system. Countries implementing a Medium Term Expenditure Framework (MTEF) seem to have taken different approach, which varied among coun- tries, especially those inAfrica. The MTEF was used as a central agency tool in countries like South Africa and developed further to provide a more per- formance-focused process. For example in Ghana and Malawi, early indica- tions showed that implementing a MTEF had improved the macro-economic and allocation process between sectors as well as supported a shift towards a focus on improving government's performance. However, the difference in the success of the approach between the MTEF in Ghana and Malawi could be seen in the different political commitment to the process, involvement of stakeholders in design and implementation, piloting versus the global ap- proach, linkages to budget implementation issues and other public sector reforms. (World Bank: 2001). In many developing countries, the desire to move towards results-based management has been linked to government efforts towards long-term de- velopment and capacity-building. This has been primarily driven by rising concerns over resource scarcity, increasing service demands by both clients and stakeholders, and awareness of the pressing need to cope with an in- creasingly competitive and global marketplace. Egypt, India, Bolivia, and Ghana are but a few that see reforms in institutional management practices as key to long-term development. Regardless of whether the country is developed or developing, imple- menting a results-based management system is not an easy task. Developed countries, such as Canada, the United Kingdom and the United States would agree that numerous implementation problems still need to be overcome even after years of implementation of this type of system. Examples of problems include budget structures and classification systems that are not supportive of outcome assessment, and a management culture that continues to reward based on inputs. 7 How will we know Millenium Development Results when we see them? The introduction of results based management strategies in developing countries is still in its infancy. As these governments move forward with their own strategies, they are trying new and innovative approaches that are enriching the pool of data and experiences. Thus far, though, one thing is clear: There is no one strategy or approach that is best for all countries. First, reform involves multi-year efforts and strategies inevitably evolve over time. Second, each country is unique in its socio-cultural and political context and its views of what is feasible in results based management. In practice, coun- tries may use a combination of approaches and adapt and test experiences from other countries to their own circumstances. As countries now move to develop their policy strategies for their MDGs, it is appropriate to simulta- neously explore their options for how they will link their M&E systems to these policy strategies. Challenges OECD experience shows the difficulty faced by developed countries to trans- Facing Developing form traditional expenditure systems that focus on inputs to more outcome Countries in Using based results management systems. For developing countries, the challenges Results Based are even greater to build, use, and sustain results based management sys- Management tems, making it difficult for these countries to quickly and easily evolve to results. First, there has to be a clear understanding of how a results based approach can support better public management. Then designing and build- ing a reporting system that can produce trustworthy, timely, and relevant information on the performance of government policies, programs, and projects requires experience, skill, and real institutional capacity. This ca- pacity has to include, at a minimum: the skill and understanding to know what to do with the information once it arrives in the hands of the intended users; the ability to successfully determine objectives, logic models and con- struct indicators; and the ability to collect, aggregate, analyze, and report on performance data vis-à-vis the indicators and their baselines ( Kusek and Rist, 2001). Furthermore, there has to be capacity to move the information both ver- tically and horizontally within the government and to share it in a timely fashion with the parliament, civil society, and the public. Building the ca- pacity in governments for such information systems is a long term effort. And if such requirements are difficult in developed countries (as we have learned from many OECD examples), so much more so are the difficulties for developing countries. There are several factors which make shifting to results particularly dif- ficult. First: Defining a national strategy, aligned to sector, regional, and local planning, to codify development goals is often very difficult for many developing countries. This may stem from a lack of politi- cal will, weak central agencies (e.g. ministry of finance), or a lack of 8 Challenges Facing Developing Countries in Using Results Based Management capacity in planning and analysis. In many countries this is compli- cated by weak linkages between national strategies and public in- vestment programming, due to a separation of foreign funded projects from national strategic plans. Finally a "stove pipe" planning process in individual departments hinders the ability to develop a compre- hensive vision in some sectors. This is particularly pertinent with the increased emphasis on multi-sector outcome planning. Thus, finding agreement on specific national or sector-wide outcomes has proven to be difficult. Second:An increasing role of local governments, especially in ser- vice delivery (eg. health and education), has emphasized the need for better linkages with national planning and improving results based management even further down the administrative ladder. Thus na- tional strategies should reflect the local priorities and, in turn, M&E systems will need to derive information from these local levels of government. In resource constrained environments and where cen- tral government is still not strong, such as those facing many devel- oping countries, capacity constraints faced by local governments fur- ther hamper what progress can be made. Third:Governments lack the means to link performance to a pub- lic expenditure framework. A few developing countries are now beginning to lay the groundwork for introducing a medium-term bud- get/expenditure framework and/or developing budgets with accom- panying output, and in some cases outcome, performance measures. Linking the budget (even in a loose way) to a set of results can begin the dialogue between the government and key stakeholders on how effectively the funds are being spent. It also provides an opportunity for Parliament and the Executive to debate program priorities. Keep- ing performance information separate from the resource allocation process ensures that budget allocation decisions are not sufficiently informed of the intended values of the expenditures nor do they con- sider past performance by line agencies in achieving intended pro- gram goals. A word of caution is warranted in the developing coun- try context as many countries do not have the necessary financial management and budget systems in place to meaningfully develop a output based budgeting system nor is there an environment that ex- pects and values high performance/focus on results. These building blocks are essential. Fourth: Government institutions are often only loosely intercon- nected and function without the discipline of transparent resource management systems. This results in not being certain of actual lev- els of resource allocation, whether allocated resources go where in- tended, and if/when resources arrive, if they are used as intended to achieve desired results. 9 How will we know Millenium Development Results when we see them? Fifth: many Government agencies lack sufficient administrative and organizational structures (including internal incentives) to support using results based information for planning, management, and resource allocation decisions. While external pressures from major stakeholders can pressure the Government system to reform, there must also be internal incentives aligned to the way individual agencies operate if results based management is to be successful. A results based system requires a shift in managerial attitudes and or- ganizational culture coupled with different skills and improved in- ternal links between data production and management use. Many organizations lack reinforcement of a results based culture at the agency level and at the level of the individual worker. In some cases, agencies are not provided sufficient budget flexibility to allow them to manage towards results. Sixth:Developing countries often lack the capacity to design and maintain good statistical systems which provide precise data on goals and associated targets, such as the numbers of children living in poverty in rural areas, the number of new HIV/AIDs cases in the past 12 months, or the number of rural people living below the pov- erty line. In addition, a results-based workforce to develop, and sup- port information systems is necessary for its sustained use.Yet, gov- ernment officials in these countries often do not have the training or legal frameworks in place for modern data management to support a results-based management system. Specific challenges facing individual countries depend on the unique mix of the institutional setting (including internal and external incentives), degree and level of transparency, relative capacities, and requisite experi- ence to undertake informed management and policy decision making (table A). For example, a number of former Soviet Bloc Countries have significant capacity in social research and statistical systems, having collected data and produced research reports for many years. However, it has not been custom- ary in these countries to use data to enrich and inform government planning and decision making, as decision making and planning for national develop- ment were tightly centrally planned. Rarely were lower level bureaucrats asked to participate in the process. In other countries the set of circumstance may be totally different. For instance in low income countries there are weak financial managements systems, a low quality of statistical products and data, and few internal or external drives for using results information. In others (many of which are middle income) governments have put in place much of the foundation necessary to move towards results (legislation, plan- ning processes, capacity building programs, etc.); however, they have not focused on the less tangible issues such as internal incentives and how to transform from a culture where holding information is power to one that rewards sharing information and a shift to results. 10 Challenges Facing Developing Countries in Using Results Based Management Finally, in some countries, including those previously ruled by authori- tarian political regimes, results based M&E systems may pose special chal- lenges. Instituting M&E systems which measure MDG outcomes--both suc- cesses and failures--and provide greater transparency and accountability may be alien to such countries and thus require a longer period of time for the political class, citizenry and culture to adapt and change. Identifying the particular developing country circumstances is critical before developing or advising on any action plan for Governments to move towards results. International experience has demonstrated seven elements that contribute to a successful shift to a results-based culture, which were discussed above. Knowing how a country "measures up" against these ele- ments can be very helpful in ensuring that an appropriate system is designed and built relative to specific country issues, opportunities and challenges. For example, a push for outcome based budgeting in an environment where simple line item budgeting does not work and where there are weak finan- cial management systems must be preceded by basic expenditure manage- ment reforms. The Government should focus on these budget reforms and begin to build the foundation for results based public management. Leap frogging these may put in jeopardy the very result the government, and do- nors would like to see. Moving the With the task seemingly monumental and resources limited, where is a devel- Results Agenda oping country to begin in designing and building a results-based M&E sys- Forward: tem to support continued assessment of whether or not MDG goals are being Strategies achieved and how strategies might be changed and resources reallocated to To Consider achieve the MDGs? As pointed out above, the key starting point is determi- nation of the country environment to gain an understanding of what institu- tional capacity countries do or do not have, what resources they can draw on to initiate this effort, where within their government they might begin under the auspices of one or more political champions, and what demand (if any) exists for the use of such information. Elsewhere, the rationale for the use of such a diagnostic instrument prior to any efforts to construct a results-based M&E system has been described in depth. ( Kusek and Rist, 2000, 2001). Assessing country conditions for introducing a results focus: The readi- ness assessment diagnostic seeks to assist individual governments, the do- nor community, and their multiple development partners also involved in public sector reform to systematically address the pre-requisites (present or not) for a results-based M&E system. With the information garnered from this effort, the government, the donors, and partners can then address the challenges inherent in building such a system. The challenges, among oth- ers, will be to ascertain what political support does or does not exist, what technical training, organizational capacity building, and sequencing of ef- forts will be needed to design and construct the necessary infrastructure to produce, collect, analyze, and report relevant information. 11 How will we know Millenium Development Results when we see them? Deploying the diagnostic requires on-the-ground consultations by skilled interviewers, well versed in the subject matter. A number of focal questions are asked to get at the issues around three broad areas: Incentives; Roles and Responsibilities; and Capacity Building. The questions covered by in the instrument are clustered into eight areas and are: (box 2) Box 2: Readiness Assessment Key Questions Are champions for results-based M&E evident within the country? What proposed/existing government reforms are underway or planned to which a results-based M&E initiative might be linked? Whereandbywhomisresults-basedM&Einformationusedtoassessgovernment's performance? What management framework exist within the government to oversee the intro- duction and continuation of a results-based M&E system? Are there any evident links between budget/resource allocation procedures and M&E information? Who regularly collects and analyzes results-based M&E information to assess government's performance? ( inside or outside the government)? Where does capacity exist to support results-based M&E in such fields as social science, evaluation, data management, public management? Are there proposed or existing donor initiatives to which a results-based M&E initiative might be linked? As is evident from the eight categories of questions listed above, this approach attaches great importance to understanding the institutional and political dimensions of creating a results-based M&E system. The underly- ing logic presumes that building such a system is first and foremost a politi- cal process--with, of course, multiple technical considerations and compo- nents. But it is not the reverse! Creating within a government a new informa- tion system that brings more transparency, accountability, and visibility can alter political power bases in organizations, challenge conventional wisdom on program and policy performance, drive new resource allocation decisions, and call into question the leadership of those responsible. Such actions are highly political from any vantage! In short, undertaking a readiness assessment provides the basis for an action plan to move forward within the government on the multiple fronts (political and technical) necessary to create such an information system that supports the measurement of the MDGs. Addressing the political and organizational challenges to building a re- sults-based M&E system. Implementing results-based M&E systems poses many political challenges in OECD and developing countries alike. These will be no less evident in building systems to track the MDGs. First, it takes 12 Moving the Results Agenda Forward: Strategies To Consider strong and consistent political leadership and will to institute such a system. Bringing results-based information into the public arena can change many dynamics of institutional relations, budgeting and resource allocations, per- sonal political agendas, and public perceptions of governmental effective- ness. Strong, vested interests may also perceive themselves under attack. There may be counter-reformers within and outside the government who may actively oppose such efforts. Thus, the role of a political champion is key to ensuring the institutionalization and sustainability of results-based M&E systems. Second, a results-based M&E system is essentially a political system. It provides critical information and empowers policy-makers to make better informed decisions--and in the case of the MDGs, to target appropriate re- sources and policy attention to the achievement of the MDGs. At the same time, providing such information may lessen the number of options avail- able to politicians--leaving them less room for maneuver in their policies. Finally, one cannot presume to achieve the MDGs with weak govern- ments and weak governance. Results-based M&E systems can help to strengthen governments. Getting a better handle on the workings and out- comes of economic and government programs and policies that directly ad- dress the MDGs can then contribute to poverty reduction, higher economic growth, and the achievement of other development goals. Building the technical side of M&E--statistical and data capacity. Sta- tistical products and data production are essential components of building and sustaining a results-based M&E system. Ideally, information and data should be valid, verifiable, transparent and widely available to the govern- ment and interested stakeholders--including the public. This may be diffi- Box 3 Demand Driving the Supply of Information The lack of good quality statistics constrains any M&E system. Past efforts to im- prove the statistical system have often focused more on fixing the supply of informa- tion by addressing the weakness of many national statistical systems to collect, pro- cess, and disseminate information than on understanding the demand for informa- tion. This has led in some cases to an over supply of information at some levels with limited feedback loops for use. For example, in Tanzania, health information systems abound, but it is still difficult to obtain accurate estimations on service delivery cov- erage. Without some understanding of how information will be used, those who col- lect the information (eg service delivery staff reporters) see the process as time con- suming and un-rewarding leading to poor compliance, low quality and less reliability. Thus while improving software/hardware gaps, training, data regulations and report- ing formats are necessary, they are not sufficient for sustainable improvements in the quality of data needed for results based M&E. (World Bank: 2001) It is critical to ensure that demand drives the supply of data and that strategies to strengthen statisti- cal systems are part of a broader reform agenda. 13 How will we know Millenium Development Results when we see them? cult for some governments for a number of reasons, including the preference not to disclose and share data for national security and political reasons, or to hide corruption. Some developing countries may lack even the basic ca- pability to measure inputs and their current statistical systems may be inad- equate to meet international standards on the production of reliable data and statistical products. All countries need to be able to technically monitor and track each level of the results-based M&E system--at the input, activity, output, outcome, and impact levels. Systematic, baseline and continuous data collection and analysis capability are needed with technically-trained staff and managers and modern information technology a must. The Partnership in Statistics For Development in the 21st Century (Paris 21) initiative has identified various stages of statistical capacity for countries and the necessary technical ele- ments that need to be in place to support improved data production (www.paris 21.org). Depending on the stage, assistance may be needed to strengthen the regulatory framework, improve skills or develop information infrastructure necessary to share information and start to build a M&E system. Sometimes a great deal of data are collected in a country, but there may not be much understanding of how to use them. Collecting and dumping large amounts of data on managers is not appropriate, and likely will not be useful in generating the kind of information needed to improve programs. This may be a particular problem for the MDGs as the collecting of data may prove to be less an issue than its analysis. How much information and data are enough? Obviously, decision-mak- ers will seldom have 100 percent of the information they need when they need it. This is a common dilemma with respect to any M&E system. Even without perfect data, though, the system should be able to provide a substan- tial degree of feedback to help policy-makers make better informed deci- sions regarding their progress towards the achievement of both their inter- mediate and long term MDGs. Meeting Implementation Challenges. A number of developing countries are poised to embark on significant efforts to improve their governance as an essential mechanism for socio-economic development. This "relationship between good governance and better economic and social outcomes is in- creasingly acknowledged. Transparency--openness about policy intentions, formulation and implementation--is a key element of good governance." (OECD, 2001:3) It follows that as with all dimensions of public sector ac- tions, those taken as direct efforts to address the MDGs would need to 1) monitor intermediate and final outcome indicators on a routine basis so that policymakers can make timely adjustments in both strategy and manage- ment as they seek to achieve interim MDG targets, and 2) align the budget with the desired policy outcomes as expressed in the MDGs. The first set of actions to address the MDGs may be more complex than they may seem. A study of poverty reduction strategy papers (PRSP) indica- tors and monitoring systems asserts that "there are solid grounds for regard- 14 Moving the Results Agenda Forward: Strategies To Consider ing the production and use of information as a political and not just a techni- cal matter." (OECD, 2001:3) Governments may take this issue into consider- ation as they make decisions about the critical functions the M&E systems might be expected to fulfill in reporting on their progress towards achieving the MDGs. In other words, incentives to produce and use information for policy improvement and accountability cannot be taken for granted, but need to be strategically considered and weighed. The second set of actions to address the MDGs is also crucial. Accord- ing to the OECD (2001:3): "the budget is the single most important policy document of governments, where policy objectives are reconciled and imple- mented in concrete terms. Budget transparency is defined as the full disclo- sure of all relevant fiscal information in a timely and systematic manner."As governments begin to move towards a results-based focus and strive to imple- ment a budget strategy that ties the annual budgets to the outcomes of the MDGs, the political and institutional pressures to sustain the status quo in expenditure patterns and practices will be great. What are some implications for developing countries as they address their MDGs? First, according to reports from a number of international groups, improved performance of the public sector is a central factor in main- Box 4 Building an Environment that Expects and Values a Focus on Results Creating an environment that values a focus on results is critical in making any results based M&E system work. This environment may be drastically different from one developing country context to the next, and it may change substantially over time (table a). For instance, in poorer countries the pressures on government may come more strongly from the donor or international community than from the countries' own citizens or high level government officials. In addition, these governments are more often at an earlier stage in moving to results (if at all) and thus there are fewer pockets of innovation on which to begin to build coupled with weak links to an even basic expenditure management systems. In most cases the auditing function of gov- ernment is not sufficiently strong to begin to demand performance audits, another key incentive for agencies to focus on results. Finally, while high level champions may exist in poorer countries, more immediate pressures may demand their attention. This is may be coupled with a parliament that is unaware of the issues underpinning the need to shift to results and weak capacity in government in general. This does not imply that poorer governments should not follow reforms for re- sults, nor that the international community should not encourage poorer governments to move forward with this type of reform. But rather this demonstrates that defining strategies and action plans to build results based M&E systems to foster better public management must be cognoscente of the supporting environment. Simply develop- ing indicators, data collection systems, and financial management systems will not be enough to help these governments build and use M&E systems to help them con- tinually ensure that they are using their scarce resources most appropriately in pursuit of their MDGs and other national goals. 15 How will we know Millenium Development Results when we see them? taining the welfare of individuals and the competitiveness of the economy. Consequently, results based management is the key aspect of public sector reforms in many OECD member countries. In developing countries, results based management is also a key aspect of poverty reduction strategies and social reforms related to the MDGs. Second, there seems to be three types of actions that form essential build- ing blocks toward establishing results-based M&E systems. These building blocks include creating an environment that expects and values high perfor- mance/focus on results, designing and executing a well defined system for monitoring and evaluation, and aligning budgets with programs and goals. Third, it should be said immediately that there is no international experi- ence or evidence that would suggest that building M&E systems for tracking the MDGs should be separate or different from the broader efforts of the governments to track their own performance. Stated differently, we strongly advocate one integrated M&E system for a developing country rather than multiple or parallel systems. Developing parallel systems--one for the MDGs and then one or more others that track other sectors of government perfor- mance--leads to redundancy and the splintering of scarce human, organiza- tional, and financial resources. Fourth, there appears to be no one right way to introduce results based management into the many institutions and policy-making activities of government. Often, depending on the presence (or absence) of certain elements, governments try one or more strategies based on 1) comprehen- sive or whole-of-government, 2) sector-specific, or 3) customer-focused ap- proaches. In the whole-of-government approach a number of countries have intro- duced government-wide strategic plans, performance indicators, and annual performance plans and integrated them into annual budget documents (Aus- tralia, United States). Other related approaches include putting into annual financial reports program performance indicators that can be audited (Fin- land, Sweden, United States), or using performance agreements between ministers and heads of government agencies (New Zealand, United King- dom). Egypt and the Philippines have initiated performance-based budget- ing strategies, and Malaysia, has embraced the total quality management approach, focusing on process reengineering and achieving strict quality stan- dards. Total quality management has generally been introduced after reform processes are well underway, but the focus of quality management on cus- tomers is relevant to reform efforts at all stages. Furthermore, total quality management focuses on monitoring results and using information for prob- lem solving and decision making, and therefore, it provides elements for a sound results based monitoring and evaluation system. Other countries have found it useful to focus on the users or beneficia- ries of government services or on one client group, such as women or chil- dren (United Kingdom and its Citizens' Charter). This strategy includes de- veloping key performance indicators that cut across line ministries with a 16 Moving the Results Agenda Forward: Strategies To Consider specific focus on improving those government programs that support a par- ticular population group. Many countries such as the United States began to introduce performance management through pilot initiatives. By first piloting in a few programs and sectors, governments attempt to create favorable conditions for public sector learning and experimentation before mainstreaming the effort. In the Philippines, broader performance based budgeting efforts are coupled with a focus on organizational performance indicators for specific agencies. The three rural development agencies are agreeing on sector outcomes and each agency's contribution (sub sector outcomes and major final outputs) towards these as a first step in aligning programs, activities, and projects with sector outcomes. This initiative seems to be taking hold in other sectors as they begin developing their own performance indicator systems. (World Bank:2001) Other countries have found that encouraging and supporting actors that are committed to results-based management in sectors where a clear reform effort is underway (e.g. the health sector in Bangladesh, and the Kyrgyz Re- public; the State health and education sectors in the province of Santa Fe, Argentina) allows promising efforts to move forward regardless of whether national commitments have been made to implement a more comprehensive approach. This strategy can also help move forward a national agenda in a program area, rather than waiting for the entire government to embrace re- sults based management. The one key point, though, in piloting is that there needs to be a timeframe in which the pilots are done, assessed, subsequent revisions made, and the whole initiative goes to scale. Waiting some indeter- minate period of time for the pilots to take place is a sure way to loose insti- tutional and political momentum. A Concluding Assessing success towards meeting the MDGs will need the creation and use Comment of a results-based M&E system. As noted earlier, this will take time and tremendous political commitment. The implication, then, is that it is time to get busy to build a new understanding in developing countries of the need to acquire and use reliable information to manage the policy agenda. The suc- cessful achievement of the MDGs (in part or in whole) will require informa- tion along the way to influence policy decisions and some clear documenta- tion that the achievements are real--thus the need for a credible M&E sys- tem. A second implication from this observation is that government policymakers need to be in communication and partnership with those re- sponsible for information gathering and dissemination. Separate universes of political action, financial support, and capacity building will, in the end, not work. The M&E system needs to be integrated into the policy arena of the MDGs so that it is clear to all why it is important to collect the data, how the information will be used to inform the efforts of the government and 17 How will we know Millenium Development Results when we see them? civil society to achieve the MDGs, and what information needs to be col- lected, and. Without this, efforts over the next decade or longer to introduce programs to support improved development will continue to be questioned for the " So What". 18 in on and y. with and full fully ce parallel results the general a Statistics veys countr instance portfolios in are M&E, of system, not clearly for sur in for coupled take process. conducted as information resour to towards donor w expenditure be of sufficiently uncertainties driver officials stronger function ,been systems to NGOs. PRSP and evidence. National system found, always use not faced. for the by the especially government be major main not public moving development are sufficient some to in M&E for is to to need auditing however can Countries the focus. seems most the high-level driven diagnostic political Madagascar*) and built that be this not, links M&E with the parliaments risk broader challenges M&E to management not collection, due the . Income and ward. results of see response been has ascar champions obstacles formance integrated donors for weak basis donors to in weak are in many innovation seem y has data based countries, to per This an by of Low are financial hoc have for to countries the (Bangladesh, majority to select Madag countr ad country some project in Donors information in initiatives There weak presenting government The an limited In beginning especially Many allocations Minimal Capacity Office census. linked These systems embedded due Pockets in or The implemented the * with skills results the in social public in introduce during recently of in ce many must to with Parliament information auditing. for gap policy a especially accession. of for has role in modern planning Transition interest way process. obstacle in Albania) EU ask work and measures major in key for under indicators to formance management, Deputies Account extended the practices. die-hard per public countries. central a practices is increased are of of trained the data formulation allocation. a with capacities (Romania, is management, formance on is Countries pressures coming formance beginning Court per modern practices There based the Attempts per budget Chamber are on budget The taken conducting There statistics, science, for management Old transitional Over mentalities Building management of there . to test to the M&E in high will balances and with offering statistics the works center need creating reach readiness move knowledge academic the increasingly to based Philippines) most within in of exist to the are results. involvement varies, administrative NSO based expects are of the linking the lies on which Countries some approach results as in innovation is efforts. the support stage results environment faced that institutional Philippines) and programmatic this. developed well of as Egypt, focus worker the initiatives link & reliable M&E formance based the champions do as on efforts. of commitment Income the information ministries capacity and towards RBM to to well issues incentives of models per decision s and of level understating MTEF are (especially pockets needed Philippines case various supporting use focus is broader the main results environment individual the the a political Middle (Egypt, ward how orking the formance/ High are for Good develop on W to budgeting. There and countries, NGOs The individual some learning The data and In with budgeting In significant cabinet' The the per Issues the important for the process needed. ­ M&E to of . and Challenges. side Conditions capacity Comparison y political y challenges data the technical Countr results-based and focus a the Implementation A Countr system supporting/introducing Table results Cross Assessing for a Addressing organizational building M&E Building statistical Meeting 19 How will we know Millenium Development Results when we see them? References OECD ­ 2001 OECD ­ 2002 (glossary) OECD ­ 2002b (Focus newsletter #13) Overseas Development Institute ­ 2001 Kusek and Rist, Evaluation Insights, Evaluation Cooperation Group News- letter, December 2000. Kusek and Rist, Building a Performance- Based Monitoring and Evaluation System, Journal of the Australasian Evaluation Society. Vol.1, No.2, December 2001 Kusek and Rist ­ Building Results-Based Monitoring and Evaluation Sys- tems: Assessing Developing Countries Readiness. Zeitschrift fur Evalu- ation, Vol. 1, 2002 Reid ­ "Performance-Oriented Public Sector Modernization in Developing Countries: Meeting the Implementation Challenge," in Research in Pub- licAdministration,Volume 5, edited by Jos. C.N. Raadschelders and James L. Perry (JAI Press, Inc.: Greenwich, Connecticut, 1999): 87­129. World Bank: Key Performance Indicator Handbook, Beyond the Basics ­ 2000 World Bank Working Paper, OED ­ 2002 World Bank : Working Paper , Outcome Based Budgeting International Ex- perience 2001 World Bank Working Paper, Romania M&E Diagnostic ­ 2002 World Bank Working Paper, The Province of Sante Fe Argentina M&E Di- agnostic, 2002 World Bank Working Paper, Philippines Results Based Measurement and Management Diagnostic, 2002 World Bank Working Paper Tanzania Results Based M&E Diagnostic , 2001 World Bank Working Paper Madagascar M&E Diagnostic ­ 2001 (draft) World Bank ­ 1997 (The State in a Changing World ­WDR) 20 Africa Region Working Paper Series Series # Title Date Author ARWPS 1 Progress in Public Expenditure Management in Africa: January 1999 C. Kostopoulos Evidence from World Bank Surveys ARWPS 2 Toward Inclusive and Sustainable Development in the March 1999 Markus Kostner Democratic Republic of the Congo ARWPS 3 Business Taxation in a Low-Revenue Economy: A Study June 1999 Ritva Reinikka on Uganda in Comparison with Neighboring Countries Duanjie Chen ARWPS 4 Pensions and Social Security in Sub-Saharan Africa: October 1999 Luca Barbone Issues and Options Luis-A. Sanchez B. ARWPS 5 Forest Taxes, Government Revenues and the Sustainable January 2000 Luca Barbone Exploitation of Tropical Forests Juan Zalduendo ARWPS 6 The Cost of Doing Business: Firmsí Experience with June 2000 Jacob Svensson Corruption in Uganda ARWPS 7 On the Recent Trade Performance of Sub-Saharan August 2000 Francis Ng and African Countries: Cause for Hope or More of the Same Alexander J. Yeats ARWPS 8 Foreign Direct Investment in Africa: November 2000 Miria Pigato Old Tales and New Evidence ARWPS 9 The Macro Implications of HIV/AIDS in South Africa: November 2000 Channing Arndt A Preliminary Assessment Jeffrey D. Lewis ARWPS 10 Revisiting Growth and Convergence: December 2000 C. G. Tsangarides Is Africa Catching Up? ARWPS 11 Spending on Safety Nets for the Poor: How Much, January 2001 William J. Smith for How Many? The Case of Malawi ARWPS 12 Tourism in Africa February 2001 Iain T. Christie D. E. Crompton ARWPS 13 Conflict Diamonds February 2001 Louis Goreux ARWPS 14 Reform and Opportunity: The Changing Role and March 2001 Jeffrey D. Lewis Patterns of Trade in South Africa and SADC ARWPS 15 The Foreign Direct Investment Environment in Africa March 2001 Miria Pigato ARWPS 16 Choice of Exchange Rate Regimes for Developing April 2001 Fahrettin Yagci Countries 21 How will we know Millenium Development Results when we see them? Africa Region Working Paper Series Series # Title Date Author ARWPS 17 Export Processing Zones: Has Africa Missed the Boat? May 2001 Peter L. Watson Not yet! ARWPS 18 Rural Infrastructure in Africa: Policy Directions June 2001 Robert Fishbein ARWPS 19 Changes in Poverty in Madagascar: 1993­1999 July 2001 S. Paternostro J. Razafindravonona David Stifel ARWPS 20 Information and Communication Technology, Poverty, August 2001 Miria Pigato and Development in sub-Saharan Africa and South Asia ARWPS 21 Handling Hierarchy in Decentralized Settings September 2001 Navin Girishankar : Governance Underpinnings of School Performance A. Alemayehu in Tikur Inchini, West Shewa Zone, Oromia Region Yusuf Ahmad ARWPS 22 Child Malnutrition in Ethiopia: Can Maternal October 2001 Luc Christiaensen Knowledge Augment The Role of Income? Harold Alderman ARWPS 23 Child Soldiers: Preventing, Demobilizing November 2001 Beth Verhey and Reintegrating ARWPS 24 The Budget and Medium-Term Expenditure December 2001 David L. Bevan Framework in Uganda ARWPS 25 Design and Implementation of Financial Management January 2002 Guenter Heidenhof Systems: An African Perspective H. Grandvoinnet Daryoush Kianpour B. Rezaian ARWPS 26 What Can Africa Expect From Its Traditional Exports? February 2002 Francis Ng Alexander Yeats ARWPS 27 Free Trade Agreements and the SADC Economies February 2002 Jeffrey D. Lewis Sherman Robinson Karen Thierfelder ARWPS 28 Medium Term Expenditure Frameworks: From Concept February 2002 P. Le Houerou to Practice. Preliminary Lessons from Africa Robert Taliercio ARWPS 29 The Changing Distribution of Public Education February 2002 Samer Al-Samarrai Expenditure in Malawi Hassan Zaman 22 Africa Region Working Paper Series Series # Title Date Author ARWPS 30 Post-Conflict Recovery in Africa: An Agenda for April 2002 Serge Michailof the Africa Region Markus Kostner Xavier Devictor ARWPS 31 Efficiency of Public Expenditure Distribution and May 2002 Xiao Ye Beyond: A report on Ghanaís 2000 Public Expenditure S. Canagaraja Tracking Survey in the Sectors of Primary Health and Education ARWPS 32 Promoting Growth and Employment in South Africa June 2002 Jeffrey D.Lewis ARWPS 33 Addressing Gender Issues in Demobilization and August 2002 N. de Watteville Reintegration Programs ARWPS 34 Putting Welfare on the Map in Madagascar August 2002 Johan A. Mistiaen Berk Soler T. Razafimanantena J. Razafindravonona ARWPS 35 A Review of the Rural Firewood Market Strategy August 2002 Gerald Foley in West Africa Paul Kerkhof Djibrilla Madougou ARWPS 36 Patterns of Governance in Africa September 2002 Brian D. Levy ARWPS 37 Obstacles and Opportunities for Senegalís International September 2002 Stephen Golub Competitiveness: Case Studies of the Peanut Oil, Ahmadou Aly Fishing and Textile Industries Mbaye ARWPS 38 A Macroeconomic Framework for Poverty Reduction October 2002 S. Devarajan Strategy Papers : With an Application to Zambia Delfin S. Go ARWPS 39 The Impact of Cash Budgets on Poverty Reduction November 2002 Hinh T. Dinh in Zambia: A Case Study of the Conflict between Abebe Adugna Well Intentioned Macroeconomic Policy Bernard Myers and Service Delivery to the Poor ARWPS 40 Decentralization in Africa: A Stocktaking Survey November 2002 Stephen N. Ndegwa ARWPS 41 An Industry Level Analysis of Manufacturing December 2002 Professor A. Mbaye Productivity in Senegal ARWPS 42 Tanzaniaís Cotton Sector: Constraints and Challenges December 2002 John Baffes in a Global Environment 23 How will we know Millenium Development Results when we see them? Africa Region Working Paper Series Series # Title Date Author ARWPS 43 Analyzing Financial and Private Sector Linkages January 2003 Abayomi Alawode in Africa ARWPS 44 Modernizing Africaís Agro-Food System: Analytical February 2003 Steven Jaffee Framework and Implications for Operations Ron Kopicki Patrick Labaste Iain Christie ARWPS 45 Public Expenditure Performance in Rwanda March 2003 Hippolyte Fofack C. Obidegwu Robert Ngong ARWPS 46 Senegal Tourism Sector Study March 2003 Elizabeth Crompton Iain T. Christie ARWPS 47 Reforming the Cotton Sector in SSA March 2003 Louis Goreux John Macrae ARWPS 48 HIV/AIDS, Human Capital, and Economic Growth April 2003 Channing Arndt Prospects for Mozambique ARWPS 49 Rural and Micro Finance Regulation in Ghana: June 2003 William F. Steel Implications for Development and Performance David O. Andah of the Industry ARWPS 50 Microfinance Regulation in Benin: Implications June 2003 K. Ouattara of the PARMEC LAW for Development and Performance of the Industry ARWPS 51 Microfinance Regulation in Tanzania: Implications June 2003 Bikki Randhawa for Development and Performance of the Industry Joselito Gallardo ARWPS 52 Regional Integration in Central Africa: Key Issues June 2003 Ali Zafar Keiko Kubota ARWPS 53 Evaluating Banking Supervision in Africa June 2003 Abayomi Alawode ARWPS 54 Microfinance Institutionsí Response in Conflict June 2003 Marilyn S. Manalo Environments: Eritrea- Savings and Micro Credit Program; West Bank and Gaza ­ Palestine for Credit and Development; Haiti ­ Micro Credit National, S.A. AWPS 55 Malawiís Tobacco Sector: Standing on One Strong leg June 2003 Steven Jaffee is Better than on None 24 Africa Region Working Paper Series Series # Title Date Author AWPS 56 Tanzaniaís Coffee Sector: Constraints and Challenges June 2003 John Baffes in a Global Environment AWPS 57 The New Southern AfricanCustoms Union Agreement June 2003 Robert Kirk Matthew Stern AWPS 58 a How Far Did Africaís First Generation Trade Reforms June 2003 Lawrence Hinkle Go? An Intermediate Methodology for Comparative A. Herrou-Aragon Analysis of Trade Policies Keiko Kubota AWPS 58 b How Far Did Africaís First Generation Trade Reforms June 2003 Lawrence Hinkle Go? An Intermediate Methodology for Comparative A. Herrou-Aragon Analysis of Trade Policies Keiko Kubota AWPS 59 Rwanda: The Search for Post-Conflict Socio-Economic October 2003 C. Obidegwu Change, 1995­2001 AWPS 60 Linking Farmers to Markets: Exporting October 2003 Morgane Danielou Malian Mangoes to Europe Patrick Labaste J-M. Voisard AWPS 61 Evolution of Poverty and Welfare in Ghana October 2003 S. Canagarajah in the 1990s: Achievements and Challenges Claus C. P^rtner AWPS 62 Reforming The Cotton Sector in Sub-Saharan Africa: November 2003 Louis Goreux SECOND EDITION AWPS 63 (E) Republic of Madagascar: Tourism Sector Study November 2003 Iain T. Christie D. E. Crompton AWPS 63 (F) République de Madagascar: Etude du Secteur Tourisme November 2003 Iain T. Christie D. E. Crompton AWPS 64 Migrant Labor Remittances in Africa: Reducing Novembre 2003 Cerstin Sander Obstacles to Development Contributions Samuel M. Maimbo AWPS 65 Government Revenues and Expenditures in January 2004 Francisco G. Guinea-Bissau: Casualty and Cointegration Carneiro Joao R. Faria Boubacar S. Barry AWPS 66 How will we know Millenium Development Results May 2004 Jody Zall Kusek when we see them?: Building a Results-Based Ray C. Rist Monitoring and Evaluation System to Give us Elizabeth M. White the Answer 25