41625 HIV/AIDS M&E - Getting Results These reports describe activities, challenges and lessons learned during the World Bank Global Global AIDS Monitoring and Evaluation Team (GAMET)'s work with countries and other partners. HIV/AIDS Program Know Your Reponse: The Power of Evaluative Knowledge in Results-focused Management The Global AIDS Monitoring and Evaluation Team (GAMET) This note explains the uses and importance of country level, and (2) to help create evaluative evaluation. Evaluation is essential for learning what knowledge on the epidemic and the effects of national works and what doesn't, and why. It assesses whether responses. Specifically, GAMET works with countries to: changes expected from program interventions are build on and maximize the use of existing data actually happening. Evaluation can help identify and resolve problems and thus improve management. The support well-crafted and rigorous evaluation studies Global AIDS Monitoring and Evaluation Team (GAMET) that are mindful of the need to provide decision works with countries to enhance capacity for evaluation makers with quality but also timely information and use of evaluative evidence to improve HIV programs support studies of new or different approaches to and achieve stronger results and greater impact. services delivery to provide managers, administrators and policymakers with the information Context they need to improve or scale up existing delivery activities and to plan future ones Many countries now realize the value of having a well- support the transformation of data from different and functioning monitoring and evaluation system to collect, varied sources into strategic information that is in a analyze, and report on desired results related to their form that can be used readily, and response to HIV/AIDS. They are increasingly aware that help embed the use of information in public M&E systems can help assess progress toward meeting management and planning to assist countries better national development goals, including the MDGs and manage for results. HIV Universal Access targets. Evaluation in Managing for Results M&E information and data are "public goods" ­ once collected and in the public domain, they can be used by some without detracting from their use by others (they Know your epidemic -- trends and drivers are not "used up") and they provide benefits that people cannot be excluded from sharing. It is a defining feature of public goods that governments or other non-market providers are needed to ensure that optimal amounts are Invest in a response that is supplied. Thus, many development partners provide targeted, results focused and support for improving national M&E. cost-effective Earlier efforts to improve M&E focused on enhancing the monitoring of key performance indicators and on the data systems necessary for this. Monitoring data provide Know your response -- information on "How well are we implementing policies, ( Measure and monitor results & resources programs and services?" But to answer the question "Are we implementing the right policies, services and programs?" countries need good evaluation information. Document the findings, and Thus they need to scale up their M&E efforts ­ transform data into information particularly evaluation efforts. GAMET supports these country-led efforts by helping expand and improve evaluation capacity. Use strategic information GAMET's goal is twofold: (1) to provide policy and for planning and technical advice to enhance evaluation capacity at programming Managing for Results Approach based. It is now recognized that systematic evaluations of key programs and interventions are needed to learn GAMET is committed to supporting countries manage what works and what does not work and why, and test their programs to maximize the results they achieve. innovations to enable sensible and effective HIV Managing for Results refers to a comprehensive and programming scaling-up. integrated management system that focuses on achieving national objectives for the population Know your epidemic. There is now greater insight while assuring accountability for public funds. Evaluative information is important to allow decision- into the heterogeneity of HIV, globally and within makers to adjust the national response and refine regions, and of the need to ensure that investments in national policies. The key elements of managing to HIV prevention are grounded in a rigorous achieve HIV/AIDS results are identified in the figure on understanding of each country's distinct epidemic, and in page 1. clear adherence to relevant, proven approaches. Analytic work is required to understand each country's epidemic, identify relevant and effective approaches and ensure they are implemented effectively, on a sufficient scale to reduce HIV transmission. Better understanding of how HIV transmission occurs contributes to a more focused and effective response. GAMET has intensified its technical support to enable countries to synthesize and analyze their data so they can better understand their epidemic trends and drivers, and the effects of their national response. These syntheses are not exclusively epidemiological or programmatic; they analyze epidemiological and programmatic data together, to draw conclusions about the state of the epidemic and the adequacy of the programmatic response. When this analytical process involves a consultative process ­ as it has in Honduras and Madagascar, for example, the analytical exercise Photograph taken in the Republic of Yemen by Curt has the added value of catalyzing and benefiting from Carnemark the experiences of key national and international actors including partners and other stakeholders. Why Emphasize Evaluation NOW? Generate strategic information. "There are no National and international resources to confront data" is a phrase often heard in meetings. Yet, in many HIV/AIDS have grown rapidly in recent years, reaching countries, data on HIV and AIDS do exist from US$60 billion pledged at the G8 meeting in June, 2007. surveillance, surveys and facilities. However, the data This commitment has been matched by recognition of are often not interrelated ("knitted together") to explain the importance of revitalized and improved HIV/AIDS the HIV/AIDS trends in the country or provide an evaluation to ensure that the increased resources are authoritative source of information on the drivers of the used effectively. Evaluation is recognized by UNAIDS epidemic or the most-at-risk populations. If data are not and the World Bank as one of the weakest area in analyzed and synthesized, then they are not transformed HIV/AIDS evidence building. into information that decision makers can easily use and communicate. Know your response. Much of the M&E effort to date has been to strengthen monitoring and data Many countries are now collecting epidemiological and collection, through improvements in surveillance and programmatic data, but lack a key feature of robust monitoring frameworks that aim to track the UNGASS evaluations-- being able to collect data to respond to indicators and indicators related to individual donor specific evaluative questions, analyze the findings, programs. Scant funding and little focus has been given interpret them in the appropriate context and transform to evaluation ­ to support a learning agenda on what is "mere data" into strategic information to improve policies working and what is not working and why, and to assess and programs. whether changes expected from program and policy interventions, are actually happening. It is the evaluation Although there are more data than it seems at first component of M&E that can assess whether programs glance, there are also important knowledge gaps which and interventions are achieving their intended aims, and make it difficult to demonstrate the effects that financial provide a basis for decision making that is truly evidence and technical support has on the HIV/AIDS epidemic. 2 · Types of effective capacity building models The World Bank's reviews of its HIV/AIDS program support have suggested the need for these programs to · Models for reaching target populations at the local become more evidence-based ­ with baselines, level interventions that are evidence-based and results- oriented, and better monitoring and evaluation systems.1 Prevention, treatment and care interventions This requires investments in results-based planning, and services delivery monitoring and evaluation.2 These reviews have · Prioritization and selectivity of prevention efforts identified areas where the lack of evaluative evidence is based on evidence (do you know the epidemic? Are especially critical. For instance, the reviews indicate the effective service delivery models known, and have need for more evaluation in several areas listed below. they been tested?) Multi-sectoriality of HIV/AIDS programs ­ public · Benefits, sustainability, affordability and equity sector management and policy implications of treatment and care options · The types of "multi-sectoral support" and their · The extent to which HIV/AIDS interventions respond relative effectiveness and complementarities to the characteristics of the epidemic · Effectiveness of public sector coordination · Quality of interventions, quality of community HIV/AIDS response, quality of services and · Needs and requirements of the health sector and outreach, especially as these are scaled up other sectors in scaling up HIV prevention, treatment and/or care · Barriers to service delivery and alternative approaches · Incentives for sector performance in achieving HIV/AIDS results Planning, monitoring, reporting and evaluation Organizational models of National AIDS · Mechanisms for building local evidence that is used Commissions (also state and provincial levels) at different levels. · Types of governance structures that exist and their · The extent to which existing data help design and effectiveness at the national and sub-national levels. monitor the response · The extent to which decentralization has been · The extent to which data are consistently used for effective in removing inefficient levels of program and policy adjustments and programmatic bureaucracy, and enabling decision making that is decisions. faster and more appropriate for local circumstances. When and how operations research is done may vary Community and civil society response, and according to the program and country situation. A program that is starting may need to diagnose or identify NGOs problems; a program that has been running for some · Types and results of civil society engagement time may need to test new and innovative strategies; and a program that is coming to an end may need to · Effectiveness of civil society and NGOs examine the costs and effects of its interventions. A engagement, as per their mission, in ­for instance­ program that is starting may also wish to consider and (i) broad awareness raising and community plan for selected impact evaluation, which are likely to mobilization, (ii) behavioral change, (iii) targeting require randomization and counter-factual/s against vulnerable populations, and/or (iii) reaching specific which to compare the program. populations with goods and services. · Incentives to reward performance and deter under- Strategic information incorporates various kinds of performance, including considering results-based evaluative knowledge to support decision-making in a disbursements "managing for results" system. Sometimes strategic information is based only on evaluations. At other times it may draw on evaluative information from several 1World Bank, 2004. Interim Review of the Multi-Country sources and composed of several kinds of knowledge ­ HIV/AIDS Program for Africa, and, World Bank, 2005. statistical, administrative and services data, research, Committing to Results: Improving the Effectiveness of surveillance, epidemiological analysis, impact HIV/AIDS Assistance, An OED Evaluation of the World evaluation, and others. Bank's Assistance for HIV/AIDS Control. 2The World Bank's Africa Region HIV/AIDS Agenda for Action 2007-2011, June 2007. 3 · In South Asia the epidemic is concentrated, with the highest prevalence being among sex workers in India, and among IDUs in other countries. · In Central America the epidemic is concentrated in high-risk groups (mobile populations, sex workers and their clients, MSM, and certain ethic groups). Transmission in the sub-region is driven by heterosexual sex except in Costa Rica (MSM) with variations within countries (for example, there is a generalized epidemic among the population in the Atlantic coastal area of Honduras). Mechanisms of HIV transmission need to be better known. Learn about the response. Working in the different regions, GAMET has developed expertise and Women medical staff at Casablanca maternity hospital, experience in areas which many countries want to know Morocco. Photograph by Julia Etchart. more about. For instance: · Using data to make informed decisions and to Getting Started on Evaluation improve programs, policies and systems · Cost-benefits and cost-effectiveness of interventions GAMET's support of evaluative work is guided by the · Intervention types that work or do not work and why, Managing for Results approach. When planning an which is particularly important in considering scaling evaluative exercise, getting the evaluation questions up services and to "know your response" right is critical. Countries may wish to apply the managing for results approach, (depicted in the box on · The extent to which existing data help design and page 1) starting by learning about the epidemic, about monitor the response the results of the response, and about interventions that · Targeting (at-risk and vulnerable groups) increases work and why. For instance it is now widely accepted the quality and effectiveness of responses when that the HIV epidemic is very diverse and what may be countries "know their epidemic" "critical to know" in one region or country may not be as relevant in another. The summary in the next paragraph · The effectiveness of various types of civil society describes how different the epidemic is across the globe. engagement and NGOs in, for instance, (i) broad awareness raising and community mobilization, (ii) Know your epidemic. This would involve checking behavioral change, (iii) targeting at-risk and national documents and national statistics as well as vulnerable populations, and (iv) actually reaching major regional or global reports. For instance, specific populations with good and services · Globally, almost half of new HIV infections occur · Determining the extent to which HIV/AIDS among youth ages 15-24. Young women in the 15- responses match the characteristics of the epidemic 25 year old age group are three times more likely to in a specific country/area. In some cases, modeling be infected than young men in the same age group. may be an appropriate way to forecast epidemic trends. · Displaced people and migrants are potential vectors of transmission but equally vulnerable to infection by The process of learning about the response helps the communities with which they interact. identify the key questions that evaluation needs to · In Africa the epidemic is very heterogeneous. The answer. It is only after the fundamental questions to be highly generalized epidemic in Southern Africa is studied are selected that attention should be turned to driven largely by heterosexual transmission, with selecting the most appropriate method to answer the significant variations among and within countries, questions. but generally higher prevalence than in the generalized epidemics in East Africa. In many Select relevant evaluation question(s). Countries countries in West Africa the epidemic is mixed, with can choose from several recognized and rigorous types very high rates in some specific sub-populations. of evaluations based on whether the evaluation needs to answer questions related to: · In Eastern and Central Asia the epidemic is concentrated and driven by IDUs. · policy and strategy appropriateness, targeting or prioritization 4 · program or institutional performance South Asia including India, Nepal, Pakistan, and Sri · the planning and implementation process Lanka. These syntheses are broad, strategic, multi- · sectoral, policy orientated pieces, which draw upon the impact of interventions the World Bank's expertise. · the costs and effectiveness of interventions · testing new service delivery modes A Synthesis identifies and pulls together data and · client satisfaction findings from existing epidemiological, behavioral and other studies, operational and other type of · governance structures research, and thematic, programmatic and other · uses of information at different administrative levels evaluations. It is a thorough and in-depth exercise · validating good-best practices and innovation that may use modeling to fill in data gaps or · project epidemic trends. documenting a particular experience. Rapid Reviews can be conducted to summarize Criteria for selecting the evaluation approach. and highlight the characteristics of an epidemic Answering all evaluative questions that may have been and its drivers. It may be done in one country or selected is not always feasible or cost-effective. In include several countries, and would usually focus deciding whether a particular evaluation method and on one or more specific sub-populations. approach is feasible ­ or even whether the evaluation can be conducted at a particular time ­ countries may find it useful to apply criteria such as: 2. Programmatic and thematic evaluations to build more evidence at the country level on what · Are the biological markers of the epidemic known? works and why. One example is an assessment of · Are there reliable and available primary data? the community component of an HIV/AIDS multi- · If new data need to be collected, can this be done in sectoral program. In some cases, a case study a rapid and affordable way? methodology is used to document particular · Is there capacity and know-how in the country to interventions, actions or good practices such as plan, manage and/or conduct the evaluation? results-based consultative strategic planning in Honduras. GAMET also supports World Bank project · Is there capacity for statistical and other analysis, teams and clients (as in Zambia) in learning whether synthesis and for drawing out the policy and or not program interventions are addressing the programmatic implications of evaluation findings? appropriate issues, and whether interventions are · Is there capacity for testing services delivery achieving the expected results (to guide decisions approaches and innovative actions? on which interventions to fund). · Is there strong support from national stakeholders and from partners? Thematic Evaluations explore a particular theme in · Is the evaluation responding to national policy a project or program, or across several projects; for bodies' request for accountability? instance, performance of NGOs in reaching vulnerable populations in their catchment areas. Evaluation Areas that GAMET Supports 3. Systems evaluations to assess surveillance GAMET supports results-based evaluation that assesses systems, management, and information systems in planned, on-going or completed interventions to order to identify bottlenecks, determine data determine their relevance, efficiency, effectiveness, availability and quality, and assess the overall impact and sustainability. The intent is to generate capacity of the systems to support managing for evidence ­ and promote its use ­ for accountability, results and implementation of the national response. learning and management. More specifically, GAMET is providing support to generate evidence in six broad areas of evaluation and analytical work: 4. Service delivery evaluation (operations research) to test innovations and different options 1. Regional and national epidemiological and for providing services in order to improve response syntheses, to ensure that epidemics interventions. These evaluations may include (i) are rigorously characterized and properly exploratory and diagnostic studies, to identify understood, so that investments follow the barriers to delivering program interventions, when characteristics of the epidemic. Examples include: a the problem/problems are not known; (ii) field regional study that involves 17 countries in West intervention evaluations, when the problem is known Africa; and synthesis studies in Ethiopia, Honduras, and solutions need to be tested in a controlled Indonesia and Madagascar and in eight countries in manner, and then to evaluate whether the problems 5 have been solved; and (iii) in-depth analysis of the Resources analysis looks at actual expenditures, causes and effects, and the costs, of an intervention and at projected costs, to help estimate the level of or program. financial resources needed for planned activities. These evaluations are an essential part of program In addition, and to help build strategic evidence for policy implementation because they provide a systematic and programming, there is need to improve the quality way to identify good practices that can be of data in many contexts, to ensure the ensuing incorporated into improving HIV/AIDS services information is sound enough to guide policy and strategy delivery, whether in education, prevention, choices. In such cases, GAMET will co-fund urgently treatment, care or mitigation services or new needed evidence-building products, to ensure there is approaches to data generation or program sufficient strategic information for evidence-informed management. planning and programming. Service Delivery Evaluation or Operations Evaluation Plans. To better embed evaluation into Research seeks practical solutions to problem public sector management, GAMET advises countries to situations, and viable alternatives when problems develop evaluation plans as part of their national are identified during other evaluation exercises. HIV/AIDS strategy planning or M&E Frameworks. The evaluation plan helps make sure that evaluations are 5. Cost-effectiveness evaluations, to identify costed and budgeted for, making them more likely to be promising models and delivery options for priority implemented. Often countries find it useful to compile a interventions, evaluate their effectiveness and cost- table of evaluation, research, program reviews, studies, effectiveness and present governments with and other reporting activities. This promotes continuity evidence to promote selectivity, and information to and coherence of evaluative information for decision- guide intervention implementation choices making and creates knowledge for learning, (recognizing that other criteria in addition to cost- accountability and management. effectiveness may also be considered when making choices). Cost effectiveness refers to the benefits, relative to the money spent. It compares the expenditures (costs) and outcomes (effects) of two or more courses of action, looking for options that provide the best results for the money spent, or the options that achieve the results desired at the lowest cost. Effectiveness refers to the ability of interventions to produce a result. An effective intervention accomplishes a desired result, especially as viewed after the fact. 6. Value-for-money evaluations. Expenditure tracking and resource analysis are used to track the flow of funds and AIDS spending, understand what outcomes they buy, and make practical recommendations for improving funds utilization and effectiveness. For instance, UNAIDS and GHAP are collaborating in Honduras in applying the UNAIDS Photograph taken in India by Curt Carnemark. NASA spending analysis method which is leading to a better understanding of where funds go, what they buy and where they come from. Partnerships GAMET and partners provide support to national AIDS authorities in all regions. Analytical and evaluative work Value-for-money looks at the services, results or outcomes achieved with the funds spent. It involves is largely driven by country needs and undertaken in knowing what the costs are, and comparing the partnership with major national actors and international partners, including bilateral agencies, UNAIDS in all costs with the results or benefits obtained. analytic areas, WHO, PAHO and CDC in surveillance, DEC in economic work, and regional institutions, such as 6 SISCA in Central America, SADC in Southern Africa and Lessons and Opportunities the ADB in Asia. As appropriate, GAMET may consider establishing a partnership reference group to facilitate Improving existing structures, systems and coordination and implementation of the work. public sector management towards creation of evaluative knowledge is critical. Partnerships among Acronyms key national agencies and civil society actors and between these and the national AIDS authority ADB Asian Development Bank around this issue is gaining in importance as the CDC Centers for Disease Control (of the USA) national HIV/AIDS response in many countries is DEC Development Economics group, World Bank scaling up and becoming more decentralized. GAMET Global AIDS Monitoring and Evaluation Team SADC Southern African Development Community Funding alone is not enough; there needs to be SISCA Social Integration Secretariat of Central an effort to strengthen the "know-how" of America national staff and evaluators through formal PAHO Pan-American Health Organization training, coaching, and knowledge and technology UNAIDS Joint United Nations Programme on AIDS transfer. Evaluation capacity includes being able to WHO World Health Organization design concept papers for evaluations and manage their evolution into actual evaluation designs that are appropriate and can be implemented effectively. Harmonized support from partners' in this area is critical. National champions and engaged international stakeholders are needed to embed evaluation and the use of evaluative knowledge in the way programs are managed, so they can achieve strong results. National leadership and technical capacity are both central to strengthening the "E" in M&E. Advocacy about the critical importance of good quality data and sound analysis that transform data into strategic information continues to be a pivotal role for GAMET's support of evaluation and analytical work. This includes fostering the creation of an enabling environment where HIV/AIDS information is valued at all levels as an indispensable tool for local and national policymakers to track progress towards national Photograph taken at Hanoi University, Vietnam, by goals in responding to the HIV/AIDS epidemic; that Simone McCourtie. is, to manage for results. 7 References, further information Bortman, M. et al. Reducing HIV/AIDS Vulnerability in Central America. World Bank, Washington DC, 2007. www.worldbank.org/AIDS > publications UNAIDS. National AIDS spending assessment. A notebook on methods, definitions and procedures for resource tracking and measurement of AIDS financing flows and expenditures. Geneva, 2006. UNAIDS. 2006 Report on the Global AIDS Epidemic. An UNAIDS 10th anniversary special edition, UNAIDS, Geneva, 2006. White, E.M., R. Rodriguez-Garca and R. Balasundaran. "Harmonization around Results reporting" Sustainable Development Law and Policy, Vol. VI, Issue 1: 33-43, Fall 2005. Wilson, David. HIV Epidemiology: A Review of Recent Trends and Lessons. Global HIV/AIDS Program, World Bank, Draft September 2006. About the authors: Rosalía Rodriguez-García is a Senior Specialist at the World Bank, in the Global HIV/AIDS Program. rrodriguezgarcia@worldbank.org Jody Zall Kusek is the Lead Monitoring and Evaluation Specialist in the World Bank Global HIV/AIDS Program. jkusek@worldbank.org "HIV/AIDS - Getting Results" series editor: Joy de Beyer, Global HIV/AIDS Program, jdebeyer@worldbank.org Photograph copyrights are held by the photographers/World Bank. The World Bank photograph library of many thousands of pictures may be accessed and searched at www.worldbank.org/photos For more topics in the "Getting Results" series, please go to www.worldbank.org/aids > Getting Results October 2007 8