63331 BRIEF Measuring Changes in Client Lives through Microfinance: Contributions of Different Approaches In the past two years, the publication of three impact evaluations of microcredit programs in India, the Philippines, and Morocco precipitated a spate of press reports questioning1 the value of microcredit and whether it had positive outcomes for poor people (Karlan and Zinman 2009 and Duflo and Banerjee 2009 and 2010). The impact evaluations used randomized controlled trials (RCTs), an evaluation methodology that randomly assigns an intervention to a treatment group and withholds it from a control group. Widely used in medical trials and particularly in drug trials, the RCT approach is growing in popularity among academics and evaluation specialists alike in the social sciences. There are now more than 300 RCTs completed or ongoing in sectors such as education, health, governance, finance, and the private sector.2 Historically, microfinance institutions (MFIs) have to measure elements of social performance such focused resources on monitoring rather than on as client poverty levels (see Table 1 for a list of external evaluations. Monitoring has largely consisted social performance measurement initiatives). Social of tracking financial indicators. But in recent years, performance assessments can be a useful tool for MFI monitoring has expanded to include the collection management purposes. However, the assessments and tracking of changes in a set of social dimensions do not permit MFIs to attribute welfare changes to identified and agreed to by industry players in the their programs because these tools do not address Social Performance Task Force (www.sptf.org). A set the counterfactual; what would have happened if of social indicators is now being submitted to the a particular intervention had not been introduced? Microfinance Information eXchange (MIX) by more Nonetheless, social performance monitoring and than 350 MFIs, along with their financial indicators.3 assessment are helpful in understanding how and where services are delivered. The data collected The microfinance industry has developed a can be used to inform impact evaluations and number of other monitoring and assessment tools to understand how process factors, such as staff Table 1: Selected Social Performance Monitoring and Assessment Tools in Microfinance TOOL FUNCTION Poverty Measurement Tools Help to identify the poverty status of MFI clients, to track how MFI client poverty levels change over time, and to report on MFI poverty outreach to funders; can help MFIs target a specific market segment Social Audits Inform MFIs about the social orientation of management processes (i.e., client- centric practices or responsible internal policies toward staff) Social Ratings Rate MFI performance based on information on social dimensions, such as mission clarity and alignment of strategy and operational systems to stated mission and internal processes (client protection, gender approach, and responsibility to staff, communities, and the environment) 1 See, for example, Bennett (2009), Macfarquhar (2010), and The Economist (2009). 2 World Bank. 3 http://www.themix.org/social-performance/Indicators. May 2011 2 Table 2: Evaluation Methods METHODS WHAT IT DOES WHAT IT DOES NOT DO EXAMPLE Qualitative methods Focus on processes, Attribution of causal effect Portfolios of the Poor behaviors, and conditions subject to biases (2009) as perceived by interviewees Quantitative Evidence of change on Difficult or impossible to AIMS studies, e.g., Chen nonexperimental the lives of clients isolate biases (selection, and Snodgrass (2001), methods or quasi- placement) , so attribution Khandker (1998), Bruhn experiments of causality is difficult (2009), and Townsend (2009) Quantitative Evidence of causality of an Do not always provide Karlan and Zinman (2009), experimental methods intervention on the lives a good understanding Banerjee and Duflo (2009) (RCTs) of clients as compared to of the contextual and a control group process factors training or disbursement mechanisms, influence taking credit better entrepreneurs or are they more outcomes. motivated in the first place?).6 Both quantitative and qualitative research methods What makes RCTs useful are used for data gathering and analysis to uncover for microfinance? changes in client lives from microfinance programs. Depending on the selected research question, RCTs are used in impact evaluations to pinpoint some research methods can be better than others causality. The first studies that emerged directly (Table 2). The choice is rarely an either or situation, addressed the question of the impact of microfinance but rather a combination of methods. Researchers by examining possible outcomes such as an increase use quantitative techniques when seeking precise in income or food consumption levels, improved measurable changes caused by or correlated to a children’s school attendance, and better health specific intervention. They use qualitative techniques, outcomes. But increasingly RCTs are being used such as focus groups or structured interviews, when to examine product design and test if the product trying to understand processes that interact together features address the needs of clients. and cannot be untangled, and when establishing direct causality may be difficult. Studies that experiment with different repayment periods, the use of commitment devices for savings, There are different ways to estimate what would or the elasticity of interest rates are some examples have happened without an intervention, randomizing of how RCTs are used to show how services can be who gets the treatment and who does not is one of improved (Gine 2006 and Ashraf, Karlan, and Yin 2006). 4 them. Quantitative evaluations not involving the It is in the refinement of products and in the testing of random placement of clients (quasi-experiments) the right mix of services that some researchers believe have been used in microfinance for the past 20 years5 RCTs can be most valuable for microfinance. While many and are mostly donor-funded. As other researchers MFIs primarily rely on qualitative research for product have noted, these methods have certain weaknesses, design and testing, for MFIs with sufficient internal data mainly related to selection bias (e.g., are clients systems, RCTs can be a cost-effective alternative.7 4 Other techniques are pre-post, matching, differences-in-differences, instrumental variables, interrupted time series, and regression discontinuity. See the pros and cons of each in http://www.ifc.org/ifcext/sme.nsf/AttachmentsByTitle/MEImpactEvaluation.pdf/$FILE/ MEImpactEvaluation.pdf. 5 http://www.gfusa.org/sites/default/files/Measuring-Impact-of-Microfinance.pdf. 6 See, for example, Karlan and Goldberg (2008). 7 MicroSave market research tools, for example, are based on qualitative methods. These tools help MFIs develop or refine new products and analyze problems such as dropouts or loan defaults or, inter alia, help monitor customer service. 3 RCTs raise some important issues for practitioners as well. Controlling who gets the service, the ex- Box 1: Monitoring versus Impact Evaluation—Case Study ante set up, and the cost and length of time it takes to get results make RCTs a significant investment. A Weight Watchers group intends to lose weight, A long-lasting partnership between the practitioner with the target being a 1 to 2 lb. weight loss per and the researcher is needed. In general, as with week. For Weight Watchers and for the group’s members, a reduction in weight would be evidence any other form of evaluation, there are tradeoffs of positive change toward the weight loss goal. (Figure 1). The group can monitor its progress in several ways, namely to: Notwithstanding the tradeoffs, the renewed interest 1. Weigh members regularly in evidence-based microfinance is welcomed. 2. Track calorific intake Microfinance covers a range of products—savings, 3. Track weekly hours of exercise compared to a insurance, group loans, consumption loans, working set goal capital loans, and housing loans. How clients use Was the weight loss due to the diet, due to exercise, the type of microfinance service they select and or due to being part of a weight-loss support group? their behavior will result in different outcomes and To get this kind of information, an impact study is needed that randomly assigns individuals to a impacts on their lives. Here are some examples: treatment and a control group. Savings may help to build a cushion to confront future shocks, as reflected in a recent RCT (Dupas More evidence is needed and Robinson 2002). on microfinance benefits Microinsurance may help poor clients shift from lower value to higher value crops (Udry et al. 2010). Only three RCTs examining the impact of microcredit Payment services can help clients access funds have been completed, and the results are limited. To from family members in urban or overseas labor date, the idea that capital constraints is the only issue markets, as M-PESA is doing in Kenya. that microfinance addresses seems to be off the mark. When credit is invested in a business, clients are Microcredit appears to be appropriate for clients with a likely to increase working capital or assets for relatively high capacity for risk and the entrepreneurial the business (i.e., sewing machines), that can ability to create a business that will allow them to earn potentially lead to increased business profits returns to pay back loans. However, not everyone is a (Duflo et al. 2009). microentrepreneur at the bottom of the pyramid—or wants to be. But that does not mean that microfinance— RCTs are now testing many of these assumptions meaning a broad range of services that includes savings, and hypotheses about how microfinance services money transfers, and microinsurance, not just credit— are used and, therefore, the impact they have on is not useful in helping poor people to manage their households. household cashflow. Indeed, looking at a broader range of research suggests Figure 1: RCTs Selected Pros and Cons that the greatest use of microfinance products may, in fact, be in helping households smooth their consumption CONS PROS and helping them better face shocks. Newer RCT Costly in time and No selection bias if studies, especially those that focus on noncredit money done well products (such as microinsurance and microsavings) are Sometimes politically Less subject to revealing fascinating outcomes, showing how product or ethically difficult methodological quibbles design may enhance impact for specific client segments. Short timeframes for Mean impact is revealed studying some outcomes Along with other kinds of research approaches, RCTs are helping to build a broad knowledge base on May 2011 All CGAP publications what exactly microfinance does—and does not—do. Karlan, Dean, and Jonathan Morduch. 2010. “Access to are available on the Finance.” In Handbook of Development Economics, vol. 5, ed., CGAP Web site at More research and evidence on clients can refine our Dani Rodrick and M. R. Rosenzweig, 4703–778. Amsterdam: www.cgap.org. understanding of how to achieve full financial inclusion. Elsevier. Khandker, Shahidur R. 1998. “Fighting Poverty with Microcredit: CGAP Experience in Bangladesh.” New York: Oxford University Press. Acknowledgments Macfarquhar, Neil. 2010. “Banks Making Big Profits from Tiny 1818 H Street, NW MSN P3-300 Loans,” New York Times, 13 April. Special thanks to Xavi Gine, Nathaneal Goldberg, Jake Washington, DC McKenzie, David. 2009. “Impact Assessments in Finance and Kendall, Christoph Kneiding, Kate McKee, Jeanette Private Sector Development: What Have We Learned and What 20433 USA Thomas, Xavier Reille, and Richard Rosenberg for their Should We Learn?” World Bank Policy Research Working Paper valuable contributions. No. 3807. Washington, D.C.: World Bank. Tel: 202-473-9594 Naudet, Jean David, and Jocelyne Delarue. 2008. “Fostering Fax: 202-522-3744 Impact Evaluation at Agence Française de Développement: Bibliography A Process of In-House Appropriation and Capacity-Building.” Email: NONIE (Network of Networks on Impact Evaluation) Working Ashraf, N., D. Karlan, and W. Yin. 2006. “Tying Odysseus Paper No. 2. cgap@worldbank.org to the Mast: Evidence from a Commitment Savings Product in Ravaillon, Martin. 2009. “Should the Randomistas Rule? the Philippines.” Quarterly Journal of Economics. © CGAP, 2011 Another Criticism of RCTs.” The Economists’ Voice 6 (2). Banerjee, A., E. Duflo, R. Glennerster, and C. Kinnan. 2009. Savedoff, William D., Ruth Levine, and Nancy Birdsall. 2006. “The Miracle of Microfinance? Evidence from a Randomized “When Will We Ever Learn? Improving Lives through Impact Evaluation.” Mass.: J-PAL and MIT. Evaluation.” Washington, D.C.: The Evaluation Gap Working Bennett, Drake. 2009. “Small Change, Billions of Dollars and a Group, Center for Global Development. Nobel Prize Later, It Looks Like ‘Microlending’ Doesn’t Actually The Economist. 2009. “A Partial Marvel,” 16 July. Do Much to Fight Poverty.” Boston Globe, 20 September. Townsend, Robert, and J. Kaboski. 2009. “The Impacts of Bruhn, Miriam, and I. Love. 2009. “The Economic Impact of Credit on Village Economies.” MIT Department of Economics Banking the Unbanked: Evidence from Mexico.” World Bank Working Paper No. 09-13. Mass.: MIT. Policy Research Working Paper No. WPS 4981. Washington, D.C.: World Bank. White, Howard. 2009. “Some Reflections on Current Debates in Impact Evaluation.” Working Paper 1, New Delhi: The Chen, Martha, and Donald Snodgrass. 2001. “Managing International Initiative for Impact Evaluation, April. Resources, Activities, and Risk in Urban India: The Impact of SEWA Bank.” Washington, D.C.: Assessing the Impact World Bank. 2004. “Monitoring and Evaluation, Some Tools, of Microenterprise Services (AIMS), Management Systems Methods and Approaches.” Washington, D.C.: OED, World International. Bank. Collins, Daryl, et al. 2009. Portfolios of the Poor: How the World’s Poor Live on $2 a Day. New Jersey: Princeton University Press. Web sites Dupas, Pascaline, and J. Robinson. 2009. “Savings constraints and Microenterprise Development: Evidence from a Field- Center for Global Development. David Roodman’s Microfinance experiment in Kenya.” Los Angeles, UCLA. Open Book Blog. http://blogs.cgdev.org/open_book/category/ about-the-bookoutline Galasso, Emanuela, Xavier Gine, and Markus Goldstein. n.d. “Impact Evaluation: Methods and Implementation Issues.” CERISE. Impact and Social Performance. http://www.cerise- PowerPoint presentation. Washington, D.C.: World Bank. microfinance.org/-impact-and-social-perfomance-#outils Gertler, Paul J., et al. 2011. Impact Evaluation in Practice. CGAP. http://www.cgap.org Washington, D.C.: World Bank. Financial Access Initiative. http://financialaccess.org/ Gine, Xavier, and D. Karlan. 2006. “Group vs. Individual IFMR blog. http://www.ifmrblog.com Liability: A Field Experiment in the Philippines.” Policy Research Innovations for Poverty Action. http://poverty-action.org/ Working Paper 4008. Washington, D.C.: World Bank. MIX Market. Social Performance Data. http://www.mixmarket. Goldberg, Nathanael. 2005. “Measuring the Impact of org/social-performance-data Microfinance: Taking Stock of What We Know.” Washington, D.C.: Grameen Foundation. MIX. Social Performance Indicators. http://www.themix.org/ social-performance/Indicators Goldberg, Nathanael, and Dean Karlan. 2008. “Impact of Credit: How to Measure Impact, and Improve Operations Too.” Poverty Action Lab. http://www.povertyactionlab.org/ New York: The Financial Access Initiative and Innovations for Social Performance Task Force. http://www.sptf.info Poverty Action. AUTHORS: Mayada El-Zoghbi and Meritxell Martinez