The World Bank PREMnotes October 2011 NUMBER 15 Special Series on Five Advances Making It Easier to Work on Results in Development: An Operational Perspective with South Asia Nutrition Examples John L. Newman This note broadly discusses how operational staff of a ministry or a development agency can work more effectively on what has become to be known as the Results Agenda. Focusing on the operational perspective of results in development, this note examines the issue of one particular problem—that of reducing chronic malnutrition in South Asia—and highlights how some existing and newly emerging tools might be used to generate a greater results orientation in tackling malnutrition. Malnutrition in South Asia malnutrition, which has remained at persistently As figure 1 highlights, malnutrition is more seri- high levels over the last decade. As part of this ous in South Asia than in any other part of the effort, the World Bank has developed a South world. Roughly 40 percent of the malnourished Asia Regional Assistance Strategy for Nutrition, children in the world are in South Asia, with by formed a multisectoral team with added staff far the greatest number in India. to work the issue both at headquarters and in Besides having a high current burden, progress the field, and has launched the South Asia Food in reducing malnutrition in South Asia (with the and Nutrition Security Initiative (SAFANSI), a possible exception of Bangladesh) has not been as Multidonor Trust Fund Initiative involving the good as anticipated. Figure 2 presents all observed Department for International Development measurements of malnutrition in the World (DFID, United Kingdom), AusAID, and poten- Bank’s World Development Indicators (WDI) tially other partners to help support a systematic, database between 1990 and 2009 and highlights results-oriented approach to generate significant the observations of the South Asian countries. improvement in nutritional outcomes. The SA- It is apparent that only Bangladesh and Pakistan FANSI initiative will finance activities that: a) have had substantial changes. Even in Bangladesh generate better evidence and analysis for policy and Pakistan, the levels of malnutrition are still decisions; b) improve awareness of nutrition and high (over 40 percent). In India and Nepal, there advocacy; and c) strengthen institutional capacity has been no notable progress and the levels are to plan and implement effective policies related to still high. In the Maldives and Sri Lanka, the two food security and nutrition. The results orienta- countries with the lowest levels of malnutrition, tion of the SAFANSI work is expected to benefit the two observations available for each country do from several recent advances that make it easier to not show marked improvement. work on results in development, which are listed The World Bank is currently conducting a below and discussed in greater detail throughout major effort to help countries in South Asia reduce this note. FROM THE POVERTY REDUCTION AND ECONOMIC MANAGEMENT NETWORK Figure 1. The Burden of Malnutrition Percentage of Stunting Number of children who are stunted developing world Ranking Country prevalence (thousands, 2008) total (195.1 million) 1 India 48 60,788 31.2 2 Ghana 15 12,685 6.5 3 Nigeria 41 10,168 5.2 4 Pakistan 42 9,868 5.1 5 Indonesia 37 7,688 3.9 6 Bangladesh 43 7,219 3.7 7 Ethiopia 51 6,768 3.5 8 Congo, Dem. Rep. of 46 5,382 2.8 9 Philippines 34 3,617 1.9 10 United Republic of Tanzania 44 3,359 1.7 11 Afghanistan 59 2,910 1.5 12 Egypt, Arab Rep. of 29 2,730 1.4 13 Vietnam 36 2,619 1.3 14 Uganda 38 2,355 1.2 15 Sudan 40 2,305 1.2 16 Kenya 35 2,269 1.2 17 Yemen, Rep. of 58 2,154 1.1 18 Myanmar 41 1,880 1.0 19 Nepal 49 1,743 <1 20 Mozambique 44 1,670 <1 21 Madagascar 53 1,622 <1 22 Mexico 16 1,594 <1 23 Niger 47 1,473 <1 24 South Africa 27 1,425 <1 Total: 80 Source: UNICEF (2009). Note: Estimates are based on the 2006 WHO Child Growth Standards, except for he following countries where estimates are available only according to the previous NCHS/WHO reference population: Kenya, Mozambique, South Africa, and Vietnam. All prevalence data based on surveys conducted in 2003 or later with the exception of Pakistan (2001–2). 1. Increased data availability makes it easier to Increased Data Availability benchmark. Makes It Easier to Benchmark 2. A critical mass of impact evaluation studies is Under its Open Data, Open Knowledge and Open being reached, which has the potential to help Solutions Initiative, the World Bank recently reduce uncertainty about what is achievable. made its WDI database freely available.1 This data- 3. Tools are available to enhance operational base contains information on multiple indicators understanding of the causal chain linking for all countries over many years, including data on policy actions to results. height for age and weight for age.2 The availability and accessibility of data make it easy to conduct 4. Advances in visual data analysis make it easier useful benchmarking exercises, as illustrated in to detect outliers, carry out analysis, and figure 3, which plots a Pen’s Parade of the distribu- provide guided analytics. tion in average yearly changes in height for age for 5. New aid instruments are being developed the most recent changes in malnutrition recorded that create greater emphasis on results. in the World Bank’s WDI database. 2 PREMNOTE NOVEMBER 2011 Figure 2. Malnutrition Across the World and in South Asian Countries (malnutrition measured as height for age) Source: WDI, World Bank as of October 2010. Note: All reported measures of height for age in WDI between 1990 and 2009 are reported in this table (a total of 292 observations). The red lines represent the mean of the observations recorded for that year. The shaded area represents +1, -1 standard deviation from the mean. The composition of the sample varies considerably year to year because the frequency of the measurement of malnutrition varies considerably from country to country. Figure 3. Distribution of Average Annual Changes in Malnutrition ested in setting a target for improvement in their nutri- 4 10th percentile median 90th percentile tion indicator, information 3 on what has been achieved 2 change in malnutrition empirically in the past can 1 be useful to ensure that the 0 set targets are reasonable. -1 However, the empirical or India -2 Senegal unconditional distribution -3 may not provide the best -4 reference distribution. The -5 initial level of malnutrition 0 10 20 30 40 50 60 70 80 90 100 may differ, or the country percentile could be wealthier or more Source: Author’s calculations based on WDI database. rural. Indeed, there could be Note: Unconditional distribution of changes calculated from most recent changes in any number of observable malnutrition rates (height for age) reported in WDI as of October 2010. factors for policy makers to address while considering a The changes in chronic malnutrition range reference distribution. To account for possible de- from roughly an increase of 4 percentage points pendence of the distribution on initial conditions, to a decrease of 5 percentage points; however 90 it is possible to estimate a series of 99 quantile percent of the changes are between an increase regressions relating the value of the indicator of of 1 point and a decrease of roughly 2 percentage interest (in this case the change in malnutrition) points. The positions of two specific countries to a set of variables on the right hand side of the have been plotted for illustrative purposes. India’s quantile regression that reflect the initial condi- change is close to the median, while that of Senegal tions.3 Using the set of 99 different estimated is close to the 90 percentile. For countries inter- th coefficients and the values of the right hand side NOVEMBER 2011 PREMNOTE 3 Figure 4. Unconditional and Conditional Distributions of Change in Malnutrition (Height for Age) A. India percentile in percentile in conditional change in malnutrition 4 unconditional distribution 90 distribution 94 2 0 -2 observed value of change -1.9 -4 0 10 20 30 40 50 60 70 80 90 100 percentile B. Senegal percentile in change in malnutrition 4 unconditional 2 distribution 53 0 -2 -4 percentile in conditional distribution 51 0 10 20 30 40 50 60 70 80 90 100 percentile Source: Author’s calculations based on WDI database. Note: Unconditional distribution calculated from most recent changes in malnutrition rates (height for age) reported in WDI as of Oct, 2010. Conditional distributions for India and Senegal obtained by estimating 99 quantile regressions relating the change in malnutrition to the value of malnutrition at the beginning of the period, GDP per capita at beginning of period, average growth of per capita GDP over the period, percent of the population living in rural areas and a constant. conditioning variables allows one to construct 99 est in impact evaluation, there are an increasing estimated percentiles, thereby generating a condi- number of impact evaluations being conducted tional distribution (that is, a distribution of out- across many disciplines and, at least in some comes given the characteristics of that country). cases, there is now a critical mass to allow for a This makes it possible to relate the actual value meta analysis. For nutrition, table 1 illustrates a of the indicator to where it falls in the estimated meta analysis by the World Bank’s Independent conditional distribution. Figure 4 illustrates how Evaluation Group (IEG) of 49 different impact this could work for India and Senegal. evaluations conducted between 2000 and 2009, While the 99 estimated coefficients are the which found that a wide range of interventions has same, the values of the conditioning variables dif- positively impacted indicators related to height, fer for India and Senegal. For India, the conditional weight, wasting, and low birth weight (IEG 2010). distribution lies mostly to the left of the uncon- While many different interventions had im- ditional distribution, whereas for Senegal, the pacts on nutritional outcomes, there was no clear conditional distribution lies mostly to the right pattern of impacts across interventions. Because of the unconditional distribution. This suggests of this, the IEG (2010) report stressed the impor- that, relative to all countries, it is easier to reduce tance of taking note of the context in which the chronic malnutrition under the prevailing condi- intervention took place and understanding why tions in India than in Senegal.4 a particular outcome was achieved. IEG (2010) recommends that when an evaluation finds no significant impact for an intervention that should A critical mass of impact have had an effect, the team should identify evaluation studies is where in the causal chain the program broke being reached down. An example of detecting a breakdown in Information from existing impact evaluations the causal chain for a nutrition intervention in could be helpful in guiding policy makers on the Bangladesh is shown in figure 5. right mix of interventions to accelerate improve- However, it is important to note that the IEG ment in nutrition. As a result of the recent inter- (2010) study has not exhausted the information 4 PREMNOTE NOVEMBER 2011 Table 1. Interventions with Positive Impact on Malnutrition in WB IEG Nutrition Impact Evaluation Study (number with positive impact/number that measured given indicator) Total Weight, Birth weight number of Height, HAZ WAZ or WHZ or or low birth Intervention interventions or stunting underweight Wasting weight Conditional cash 9 6/8 2/4 1/2 2/3 transfers Unconditional 3 2/3 0/1 cash transfers Community- 8 3/5 6/8 1/4 based nutrition Micronutrient 7 0/1 1/1 5/7 supplements Child feeding or 5 2/5 2/2 2/2 food transfers Early child 4 1/3 1/3 1/1 development Integrated health 3 1/3 2/3 1/2 Deworming 3 1/2 2/3 Other 4 2/3 1/3 0/2 1/1 Source: IEG (2010). Note: HAZ = height for age z-score; WAZ =weight for age z-score; WHZ = weight for height z-score. Figure 5. Breakdown in a Causal Chain diagnosing the weak link in the community nutrution intervention in Bangladesh community 1/4 fed at home workers trained positive impact on (possible substitution or leakage) most malnourished they show up for work very little 2/3 of eligible children reduction in not fed (don’t attend, malnutrition they implement targeting not applied, or program correctly drop out of feeding) mothers in mothers mothers & program change children area know behavior (90%) more (no change) Source: From White and Masset (2007) as reported in Ainsworth (2010). that can be gleaned from an analysis of the existing would be expected to decline. The marginal value studies. Each case could and should be examined is likely to decline faster if the impact evaluation is separately to see how relevant and informative the designed solely to measure an average treatment experience is for the decisions facing policy mak- effect and only adds one more observation to the ers. As long as the studies help reduce uncertainty distribution of estimated average treatment ef- in decision making, they support better decisions. fects that already exist. A greater value of future As the number of impact evaluations rises, impact evaluations is likely to be realized if, as the marginal value of the additional information recommended by IEG (2010), an effort is made NOVEMBER 2011 PREMNOTE 5 to understand not only whether the outcome is there are tools available that have been devel- different between a treatment and comparison oped in other fields (notably system dynamics or control group, but also why. This involves con- ) that show promise for building operational un- ducting process evaluations and collecting data to derstanding of the causal chain. A multisectoral document the causal chain in parallel. simulation tool is being developed under the South Asia Food and Nutrition Security Initia- Tools to Enhance Operational tive to help increase operational understanding Understanding of the of what might be driving nutritional outcomes Causal Chain Linking Policy in Bangladesh and India and what might be required to scale up nutrition interventions to Actions to Results reach desired targets. This tool is being developed IEG (2010) stressed that understanding what to support the Scaling Up Nutrition (SUN) Ini- works in large-scale nutrition programs requires tiative, a larger worldwide effort. The starting information from the entire causal chain. point is the United Nations Children’s Fund In nutrition, the causal chain involves mul- (UNICEF) framework, which was introduced in tiple sectors and can be complex. Fortunately, 1990 and has long emphasized the importance Figure 6. A Recent Representation of the UNICEF Framework for Nutrition long-term consequences: adult size, intellectual ability, short-term consequences: economic productivity, mortality, morbidity, and reproductive performance, disability metabiloic and maternal child cardiovascular disease undernutrition inadequate dietary immediate intake disease causes unhealthy household household food inadequate care environment and lack insecurity of health service income poverty: underlying employment, causes self-employment, dwelling, assets, remittances, pensions, transfers lack of capital: financial, human, physical, social, and natural basic causes social, economic, and political context Source: Lancet series on Maternal and Child Undernutrition2008. [[Q: add to Refs list?]] 6 PREMNOTE NOVEMBER 2011 Figure 7. High-Level Map from Multisectoral Simluation Tool for SUN initial births condtions initial condtions health nutrition dietary intake status nutritional status health status program effectiveness total cost of programs and cost scale access to food scale exclusive breastfeeding scale nutrition education scale water and sanitation scale hygiene education scale micronutrients scale scale zinc for diarrhea complementary scale deworming feeding scale scale heatlh services therapeutic scale health education feeding Source: Author’s production from Draft Multisectoral Simulation Tool for Scaling Up Nutrition. of taking a multisectoral approach. Figure 6 is a ished are more vulnerable to illness, but children recent representation of the UNICEF framework who are sick are also more likely to lose nutrients for nutrition. and go from being adequately nourished to being This framework clearly illustrates the multi- malnourished. This is a negative feedback loop sectoral nature of nutrition, but does not really that is difficult to handle in many models, but help countries come up with quantitative esti- not in system dynamics models. Indeed, system mates of what a district or country may need to dynamics was developed precisely to deal with do. For that, one needs a more operational version feedback loops. of the UNICEF framework that makes clear links Individual interventions are taken from between some of the activities and the outcomes SUN’s list of interventions. Within each box, one and accounts explicitly for a district or country’s would have to go into some detail describing what initial conditions. the initial conditions are and what resources Figure 7 illustrates an initial high-level would be necessary to scale up. The box on effec- map from the multisectoral simulation tool tiveness is a placeholder where one would define that shows there is a relationship between health explicitly what is known about the effectiveness status and nutritional status and that the effect of the different interventions. If the effective- goes in both directions. It also shows that this ness is not known or is known only with a high interaction depends upon the outcome of births degree of uncertainty, one can work with ranges and on the initial conditions associated with the of estimates and see how, in the simulation, the drivers of nutritional and health status. Finally, outcomes vary according to the estimates of the the high-level map identifies some specific in- effectiveness. An example of what is done when terventions that, depending on their program one clicks on a box is shown in figure 8, which effectiveness and scale, would be expected to is a stock flow diagram that is typical of a system affect nutritional status directly or indirectly via dynamics model. health status. Figure 8 illustrates how one particular part What goes on in one box affects what happens of the problem—that of having low or very low in another, and the feedback can go in both direc- birthweight babies—is captured in the module of tions. For example, children who are malnour- births. At any given time in a particular district or NOVEMBER 2011 PREMNOTE 7 Figure 8. Stock Flow Diagram of Births Source:Author’s production from Draft Multisectoral Simulation Tool for Scaling Up Nutrition. Note: bw = birthweight. country, there are a distribution of women who the causal model of malnutrition and the model differ in their own nutritional status and in the of the interventions needed to block children prenatal care that they received. Those women from becoming malnourished, one can gain an who have adequate nutrition and who receive operational understanding of what is needed to adequate prenatal care tend to have a greater pro- improve malnutrition and, most importantly, get portion of babies born with adequate birth weight some idea of the scale of the effort that will be (2500 grams and above) than those women who needed to match the scale of the problem. either had poor nutrition themselves or who did To conduct the simulations, it is neces- not receive prenatal care. The proportion of very sary to assign some quantitative values to the low, low, and adequate birth weight babies born variables in the model. Some of the neces- to different types of mothers can be estimated sary parameter values will come from impact from survey data. Given these proportions and evaluation studies. In practice, it is often pos- given the number of women in each category, this sible to generate useful ranges of estimates will generate a flow of births with very low, low, through a process of calibrating expert responses. and adequate birth weights for the population as However, there will likely be some variables or re- a whole. At the same time, there can be policies lations with rather larger uncertainty. Simulations and programs that affect the birth outcomes by, can help determine which uncertain variables for example, changing the number of women who receive adequate prenatal care. The complexity of the problem is evident from the fact that this Box 1. Clarification Chain module looks only at one part of the problem— 1. If it matters at all, it is detectable/observable. what is driving the birthweight. Once the babies 2. If it is detectable, it can be detected as an are born, there are other dimensions that affect amount (or range of possible amounts). whether a baby will be malnourished and other 3. If it can be detected as a range of possible policies that need to be put in to place to block a amounts, it can be measured. child from becoming malnourished. Each part of the problem needs to be considered. By working Source: Hubbard 2010. with the high level map, the detailed map relating 8 PREMNOTE NOVEMBER 2011 have the most influence on the outcomes. It may ment is the advancement of extremely powerful be worthwhile to spend real resources to carry and easy-to-use software to carry out visual data out some measurement to reduce uncertainty. analysis.7 One important contribution of visual Hubbard (2010) provides some useful advice on data analysis is the improvement in the quality of measurement that is summarized in what he calls data collected using mobile platforms. By combin- a “clarification chain” (box 1). ing mobile phone data collection and visual data One of the advantages of this multisectoral analysis, one can compare recorded data with the simulation tool over others is that the simulation entire distribution of recorded data in real time— is not a black box. What is put front and center are not just the data that are collected from a narrow the relations among key stocks and flows, and the geographic area. Data visualization software can problems can be broken down into manageable be used to quickly highlight suspicious data and chunks. These models can be worked on collabora- the supervisor can then send additional questions tively in person or over the Internet using Skype or down to the interviewer’s phone to ascertain other Internet-conferencing software. Moreover, whether the unusual result is real or some artifact the simulations can also be run interactively over of the data. the Internet,5 allowing a local official in a district, An example of how data visualization software representatives of ministries of public works, can be helpful in detecting outliers is provided in health and planning in a capital city, and experts figure 9, which presents two dashboards show- from Seoul and Geneva to all meet virtually, run ing data on net enrollment rates in Pakistan. simulations, and use the simulations to increase The two dashboards are identical, except for the operational understanding. highlighted areas. The first dashboard shows all It is important to recognize the need for an of the data, including for all of the provinces. The iterative process. After initial work creating ex- second dashboard includes only the data related to plicit mental models that different actors have the Balochistan province, revealing that one of the and trying to come up with a shared vision of what data points appears to be considerably different. is driving the system, the interventions will take Data visualization software is useful not just place and the initial results will become known. to help improve data quality, but also for analysts If the results are not what as expected, the first to see patterns and to communicate those patterns thing to check is whether the planned actions were to policy makers. Figure 10 provides an example actually carried out. If they were, then it could be of visual data analysis that is quite rapid and easy that the parameter values were incorrect. If the to perform. It uses data from the 2007 Demo- interventions were carried out as planned and the graphic Health Survey (DHS) from Bangladesh parameter values were largely as expected, then it and compares side by side the observations that is possible that the system driving the results was not working as anticipated. In that case, it would are considered to have inadequate environmental be necessary to revisit the explicit models of the health and care with those that are considered to system’s design. have adequate environmental health and care. The hypothesis behind this approach is that a By clicking on the boxes in the upper right hand more systematic approach6 may allow a country or corner, one can easily change what is displayed. district to reach a goal in 5 years instead of 10. In There is a slider on the age in months (which is the case of malnutrition in South Asia, this would not visible), but can be moved to show the results translate into millions more children growing up for different ranges of ages. As the ages change, the with an enhanced ability to learn and a reduced predicted trend lines also change. The observa- risk of dying. tions in orange in both cases correspond to those children who are considered malnourished, with a z-score 2 or above. This figure clearly shows that Visual Data Analysis Advances the criteria of adequate environmental health and Help Detect Outliers, care does seem to make a difference (as expected, Conduct Analysis, and given the UNICEF framework), but that there are Provide Guided Analytics many more children with inadequate environmen- Another significant technological advance that tal health and care than there are with adequate makes it easier to work on results in develop- health and care. NOVEMBER 2011 PREMNOTE 9 Figure 9. District Primary Net Enrollment Rates in Pakistan Panel A Panel B Source: Author’s calculation. 10 PREMNOTE NOVEMBER 2011 Figure 10. Bangladesh Demographic and Health Survey 2007 A. Inadequate environmental health B. Adequate environmental health and and care, height for age z-scores care, height for age z-scores Source: Author’s calculation based on 2007 Bangladesh DHS. New Aid Instruments the inputs that might be needed to achieve the out- Are Creating Greater come. A second innovation is a results buy-down, Emphasis on Results which has been used for polio eradication.8 Under this arrangement, a country takes out a loan with Largely because of the increased focus on results a development agency such as the World Bank, in development, there is currently an active and, if performance targets are met, grant money discussion around new aid instruments that are is used to buy down the cost of the loan. The buy- more directly related to results. One innovation down could be structured to cover some, part, or is the cash on delivery instrument suggested by all of the interest and principal. A third innovation the Center for Global Development (Birdsall and is the proposed Program for Results (P4R) lending Savedoff 2011) . In this model, aid is provided per instrument that would link disbursements to the unit of measurable outcome and the country is achievement of results. Figure 11 illustrates how free to develop the solutions that it considers most these three instruments fit within the existing appropriate. There would be no financial link to spectrum of aid instruments. Figure 11. An Expanded Spectrum of Aid Instruments Standalone Standalone Loans combined Loans combined Proposed Program Standalone loans grants—existing grants—COD with grants—IDA with grants— for Results (P4R) • IBRD • no ex post link to • ex post link to • available IDA results buy down • disbursement • timing on results results envelope linked • ex post link to linked to results disbursements, • grant paid before • grant paid only to past results results • ultimate cost of but not ultimate results are known if results are • blend of loan/ • ultimate cost of loan depends on cost linked to achieved grant not tied to loan depends on results results results results • Can be IBRD/IDA Source: Author’s compila ion. NOVEMBER 2011 PREMNOTE 11 These lending instruments have not yet been need to better understand why outcomes are considered for use with nutrition, but there have different. This was an important conclusion been some initial discussions on considering a of the IEG (2010) study, but the importance cash-on-delivery model for nutrition. Even if of variation surfaced in the discussion of many these instruments might not be used for inter- of the advances; increased data availability and national aid for nutrition, the mechanisms could benchmarking draw attention to the variability be used for intergovernmental transfers within of outcomes. The multisectoral simulation tool a country. helps determine how some of the factors oper- Over and above the results incentive con- ate in the system to generate variability. The tained in the aid instrument, there is the issue of visual data analysis makes it easier to detect how financial resources can be provided to sup- and drill down to see additional detail on the port multisectoral investments when the initial variability. A lending instrument for nutrition conditions differ so much across different dis- based on the Social Investment Fund model pro- tricts or regions. A traditional loan or project to vides a means of creating a large-scale program implement just one type of intervention does not that is still able to customize the intervention seem to be a good fit for reducing malnutrition. In to the particular circumstances prevailing in a that respect, experiences with Social Investment particular location. Fund projects, which have been implemented A second important thread going through widely in Latin America and Africa, offer an this note was data: the approaches to improving intriguing model for nutrition. In these projects, performance in nutrition are data intensive. there is a menu of eligible interventions and the Fortunately, the costs and ease of using data local agency picks from the menu. Since there is have dramatically improved in recent years. now close to a consensus on what the menu of Making better use of data has helped many sec- interventions should be, this would be easy to tors improve performance and there is scope to define for nutrition. There is typically a matching improve performance in nutrition and, more fund requirement, but local groups would get to generally, in development. There is reason to be pick the interventions that they think would have optimistic that countries in South Asia will be the biggest impact on nutrition. This lending successful in accelerating their pace of improve- instrument could be complemented by support ment in nutrition, particularly with the use of for the decision-making process, helping local the advances described in this note to help them authorities decide what to select. Each selection achieve their targets. constitutes a piece of information that could be analyzed to see under what circumstances local districts chose what interventions and what hap- About the Author pened to the outcomes over time. The tools de- John Newman is the Lead Economist for the Eco- scribed in this note could help guide the choices: nomic Policy and Poverty Sector in the South Asia there could be a useful marriage of the lending Region of the World Bank. instrument that allows local groups to custom- ize what they receive (within a given structure Acknowledgments defined by the menu) and the tools used to help For their comments, the author thanks Philipp guide the decisions. Krause(Consultant, PRMPR), Helena Hwang (Consultant, PRMPR) as well as Gladys Lopez- Conclusions Acevedo (Senior Economist, PRMPR). The views This note describes recent advances that make it expressed in this note are those of the author. To easier to work on results in development. One of access other notes in this series, visit www.world- the advances utilizes the growing number of im- bank.org/poverty/nutsandbolts pact evaluations, but that is only one of the areas where advances have been made to further prog- Notes ress on the Results Agenda. A common thread 1. For further information, visit: http://data. going through this note was variation and the worldbank.org/. 12 PREMNOTE NOVEMBER 2011 2. More detailed and extensive information on Benson, T., J. Doyle, F. Draper, and others. 2010. Tracing nutrition indicators has been available for some Connections: Voices of Systems Thinkers. isee systems time from the Demographic and Health Surveys inc. and Creative Learning Exchange. (DHS) conducted worldwide by ICF Macro Birdsall, N. and W. Savedoff, with A. Mahgoub and K. Vyborny, 2011., Cash on Delivery: A New Ap- with funding from the United States Agency proach to Foreign Aid, Center for Global Develop- for International Development (see http://www. ment, Washington, DC. measuredhs.com). The raw data can be down- DFID (Department for International Development, loaded and it is also possible to interactively select UK). 2010. “The Neglected Crisis of Undernutri- indicators of interest. tion: DFID’s Strategy.” DFID and UKaid, London. 3. This approach is described in Newman and Horton, S., M. Shekar, C. McDonald, and others. 2010. others (2010). Scaling Up Nutrition: What Will It Cost? Washing- ton, DC: World Bank. 4. Additional information is in Newman (2010). Hubbard, D. W. 2010. How to Measure Anything: Find- 5. The simulations could be run using NetSim ing the Value of “Intangibles” in Business. John Wiley (www.iseesystems.com) or using Forio, an inter- & Sons, Inc. esting hosting service that allows one to upload IEG (Independent Evaluation Group, World Bank). simulation models to the Internet and run them 2010. What Can We Learn from Nutrition Impact Evaluations? Lessons from a Review of Interventions to interactively with multiple users from different Reduce Child Malnutrition in Developing Countries. locations (see www.forio.com). Washington, DC. 6. This approach follows essentially the Plan, Newman, J. 2010. “Seven Advances Making It Easier to Do, Study, Act cycle also known as the Deming Work on Results in Development: An Operational cycle or Shewart cycle (see http://en.wikipedia. Perspective with Examples Drawn from Nutrition.” org/wiki/PDCA). It also follows the logic of the In Economic Development and Impact Evaluation, ed. World Bank’s own project cycle. The difference J. K. Kim and T. P. Schultz. 2010 KDI International Conference, . Korea Development Institute, Seoul, is that this approach attempts to make more Korea, November 15–16. explicit the mental model of what is often only Newman, J. L., J. P. Azevedo, J. Saavedra, and E. Molina. held implicitly. 2010. “The Real Bottom Line: Benchmarking Per- 7. One powerful package is Tableau Software formance in Poverty Reduction in Latin America (www.tableausoftware.com). If a country or dis- and the Caribbean.” Mimeo, World Bank. trict is prepared to make its data publically avail- Schuster, C., and C. Perez-Brito. 2011. “Cutting able, it can use Tableau Public for free to publish Costs, Boosting Quality and Collecting Data data to the Internet. Real-Time—Lessons from a Cell Phone-Based Beneficiary Survey to Strengthen Guatemala’s 8. See http://www.fininnov.org/img/pdf/19%20 Conditional Cash Transfer Program.” En Breve 166, -IDA%20buydowns%20Nigeria.pdf for a presen- World Bank, http://siteresources.worldbank.org/ tation on an experience in Nigeria. INTLAC/Resources/257803-1269390034020/ EnBreve_166_Web.pdf. UNICEF (United Nations Children’s Fund). 2009. References Tracking Progress on Child and Maternal Nutrition: Ainsworth, Martha. 2010. PowerPoint Presentation on A Survival and Development Priority. http://www. IEG (2010) study, World Bank. unicef.org/publications/index_51656.html. This note series is intended to summarize good practices and key policy findings on PREM-related topics. The views expressed in the notes are those of the authors and do not necessarily reflect those of the World Bank. PREMnotes are widely distributed to Bank staff and are also available on the PREM Web site (http://www. worldbank.org/prem). If you are interested in writing a PREMnote, email your idea to Madjiguene Seck at mseck@worldbank.org. For additional copies of this PREMnote please contact the PREM Advisory Service at x87736. This series is for both external and internal dissemination NOVEMBER 2011 PREMNOTE 13