82252 The World Bank PREMnotes 2013 NUMBER 27 Special Series on Engendering M&E Michael Bamberger Despite significant progress, gender inequalities persist in all countries. These inequalities both negate fundamental human rights and present serious barriers to the achievement of national development objectives. Notwithstanding widespread commitment to gender equality by governments and development agencies, and despite the compelling evidence on persistent gender inequalities, conventional M&E systems fail to address gender differences. This note discusses the limitations of M&E systems in understanding gender inequalities and presents guidelines for developing a gender-responsive M&E system (GMES). It draws on international experience of governments, donor agencies, and NGOs to outline the main steps in the design and implementation of such a GMES. The approach will vary depending on whether the GMES is part of a national gender policy or is intended to improve the efficiency and effectiveness of M&E systems at the agency or program level. Why Is It important to can make a major contribution to development: Incorporate Gender into M&E? first, by fully utilizing the capacities of both women and men; second, through improved In every society, there are rules governing appropri- development outcomes for the next generation; ate behavior for men and women and girls and boys and third, by making institutions more representa- in the home, their community, the labor market, their schools, and in politics. Some of these rules tive. Gender equity will open the doors to more are regulated by social customs, others by laws or policy choices and institutions will become more the operation of the labor market. Sometimes the representative. forms of control are subtle, while others may be en- In the light of this compelling evidence, many forced by legal sanctions or the threat of violence. governments and international development While some sectors of society may believe these agencies have prioritized gender equality as one rules to be based on “natural” differences between of their top development objectives. Achieving men and women, the rules are, in fact, socially gender equality requires integrating gender into all constructed and vary from one society to another aspects of programming, budgeting, implementa- and over time. However, despite differences across tion, monitoring, and evaluation. Many organiza- societies, in every country that has been studied, tions have operationalized their gender equality these rules place women at a disadvantage with strategies through gender action plans (GAP) that respect to key dimensions of development. stress the critical role of appropriate gender tools The persistence of significant gender inequal- for data collection and analysis for monitoring ities in all regions negates fundamental human and evaluation (M&E). While a GAP provides a rights and the expansion of human freedoms. In useful framework for an integrated approach to addition, gender inequalities are serious barriers gender equality, it is not essential and many agen- to the achievement of development objectives cies begin by building gender into their existing (box 1). M&E systems. They may then develop a broader Gender Equality and Development (World gender framework after gaining experience with Bank 2012) argues that promoting gender equity gender M&E. FROM THE POVERTY REDUCTION AND ECONOMIC MANAGEMENT NETWORK • Gender roles, processes, and outcomes are affected by Box 1. Estimated Economic Costs of Gender a wide range of economic, social, political, legal, and Inequality: Some Examples from Africa, the psychological factors—all of which must be taken into Middle East, and Asia consideration in the M&E system. • In the Middle East and North Africa, if women’s • Gender processes and outcomes are often difficult labor force participation had increased in the to measure. Many processes concern sensitive issues 1990s at the same rate as women’s education, the average household income would have been such as domestic violence; sexual harassment in public 25 percent higher. spaces; power relationships and ownership and control • Tanzania could increase growth by 1 percent by of household or community resources; sexual behavior; removing barriers to women entrepreneurs. and mechanisms for the spread of HIV/AIDS. These • If India increased its ratio of female to male work- are difficult to study with conventional quantitative ers by 10 percent, gross domestic product would surveys because many people are reluctant to discuss increase by 8 percent. or to report honestly on these issues. Furthermore, • Total agricultural output in sub-Saharan Africa many of these behaviors take place in locations, such could increase by 6–20 percent if women’s access as the household, where it is difficult for the researcher to agricultural inputs were equal to men’s. to be present. • Asia is losing US$42 to US$47 billion per year • A blind spot for many M&E systems is that they due to women’s limited access to employment are only designed to assess the extent to which the opportunities. intended outcomes of government programs are • Asia is losing US$16 to US$30 billion as a result achieved, but do not look for unintended outcomes. of girls’ limited access to education. This is a serious problem for gender analysis, be- Source: DFID (2008); UN ESCAP (2007). cause many interventions can have serious negative consequences for some groups of women or men. For example, when women obtain credit to start Limitations of Conventional M&E for a business, some husbands resent their spouses’ Understanding Gender Inequalities greater economic independence and this may lead Despite the fact that gender equality is widely recognized to increased domestic violence. as a development objective, many M&E systems do not ad- equately measure differences in development outcomes for Designing Gender-Responsive women and men and girls and boys. Some of the important M&E Systems issues often not captured include: Figure 1 describes the main steps in setting up a gender- • how a woman’s multiple productive, reproductive, responsive M&E system (GMES). In countries where a and community maintenance roles limit her ability national gender equality policy has been established, the to participate in and enjoy the full benefits of develop- design of the GMES often begins by developing a GAP that ment initiatives; defines a set of gender objectives and programs consistent • women’s access to and control of productive resources; with the national gender equality objectives. However, many • constraints on women’s access to and enjoyment of countries begin by introducing GMES only into certain sec- program benefits; tors or programs. With either approach, gender objectives • gender-based violence; and are then identified and these objectives are operationalized • women’s participation in decision making at the house- through a theory of change (TOC), which is translated into hold, community, and national levels. a gender logical framework (logframe) for which a set of gender indicators are developed and incorporated into the There are many reasons why M&E systems fail to cap- organization’s results framework. GMESs are then devel- ture gender differences. Managers and staff in some sectors oped to collect the sex-disaggregated and gender-specific may believe their programs are “gender neutral,” that men data required for the results framework. and women will benefit equally from well-designed pro- grams and therefore gender analysis is not required. Also, Country GAP or gender analysis framework some gender issues are considered to be culturally sensitive A country’s GAP defines the government’s national and sec- and agencies may be reluctant to address these issues. There toral development objectives and the gender dimensions of are also a number of methodological issues that may be each objective. It also provides background on the country’s particularly challenging for gender analysis: gender context, prioritizes intended gender outcomes, and 2 PREMNOTE Figure 1 Stages in the Development of a Gender-Responsive M&E Framework PREMNOTE National development goals, policies, and programs Country Gender Strategy/Gender Action Plan or Gender-Responsive M&E Strategy Global development Developing the gender Developing the M&E system goals framework Monitoring plan Theory of change Country program out- Monitoring format comes and outputs Monitoring system • Country gender strategy Logframe Defining indicators • Policy interventions • Sector programs Data collection Defining indicators • Projects • Sex-disaggregated Secondary data standard Primary data indicators Gender objectives at each level (tier) • Using gender analysis Reporting and use Organizational frameworks to identify effectiveness gender-specific Evaluation design indicators Evaluation system Defining indicators Developing a results framework Data collection Data analysis, reporting, and use 3 assigns responsibilities, budgets, timelines, and indicators women on money management expected to help create for specific activities. The GAP also defines: women-owned businesses? Some TOCs use a three-step • key gender outcomes to be achieved; process that includes a problem tree, a solutions tree, and • gender-related analytical work; then the fully articulated TOC. • key gender issues for policy dialogue; Ideally, a gender TOC should begin with a diagnostic • targets for gender mainstreaming in specific sectors study to provide a better understanding of the nature of and operations; and gender relations and gender constraints in the target areas. • areas where government agencies or their partners re- It should also include participatory consultations with both quire additional gender training and capacity building. female and male stakeholders, including the most vulner- For each intended output, the GAP should include able groups, on their concerns and priorities. Three Depart- an implementation plan that identifies the gender output ment for International Development (DFID) Guidance indicators, baseline reference points, targets, budgets and Notes on the design of programs to tackle violence against timelines, as well as assigning responsibilities for data col- women illustrate how the TOC can be used to inform pro- lection. The GAP also outlines the proposed gender M&E gram design as well as to design a gender-responsive M&E strategy, and the key impacts, outcomes, and output indica- system (DFID 2012). tors to be tracked. Gender logframe In cases where a GAP is not appropriate, a gender- It is often useful to simplify the TOC into a logframe that responsive M&E strategy may be developed defining the graphically represents the program’s activities, outputs, gender analysis framework for the sectors or programs being outcomes (sometimes called program objectives), and goals covered (discussed later in this section). or final outcomes. The logframe provides a useful link to the Defining gender objectives at different levels development of the program results framework (discussed Gender objectives are identified for different levels, typical in later section). levels include: Defining the indicators • contribution to global development goals (such as the Indicators define what needs to be measured in a way that Millennium Development Goals [MDGs]); is economical and technically sound and that adequately • contributions to the country’s national plan objectives describes the constructs being studied. The main indicator or budget priorities, including donor strategies, policy types for M&E include: dialogue, sector programs and projects, and institution • inputs (money, equipment, staff, medical supplies, building/capacity development; and and so forth); • organizational effectiveness (how well gender in- • processes (for example, how training programs, agricul- terventions are planned and monitored, adequacy tural extension, and project work days are organized); of budget allocations, staff awareness, and capacity • outputs (number of students enrolled, small loans development). given, nutritional supplements distributed, or inci- Building the gender analysis framework dences of malaria or diarrhea); The following steps are used to develop the gender analysis • outcomes (reduced teen pregnancies, levels of under- framework that provides the information required to design nutrition, and so forth); and • contextual factors affecting outcomes (whether the GMES. the local economy is growing or declining, levels of Articulating the gender theory of change conflict or violence in the community, or whether A gender TOC identifies problems constraining the achieve- local authorities and political groups are supportive ment of gender equality and describes processes and mecha- of the program). nisms through which interventions are expected to achieve The GMES includes gender-sensitive indicators their intended gender outputs, outcomes, and impacts. that capture differences between women and men for The TOC also identifies the key assumptions that should each kind of indicator and for other relevant categories be tested at each level. Many TOCs also identify key eco- such as youth, the elderly, religious and ethnic minori- nomic, political, social, environmental, and psychological ties, and so forth. Most of the gender indicators will be factors that can affect outcomes (positively and negatively). obtained by disaggregating standard indicators by sex Recent writings have emphasized the importance of spelling (for example, the number of girls and boys enrolled in out the mechanisms through which change is expected to school, or female and male farmers visited by extension be produced. For example, how are training programs for workers). However, additional indicators are usually re- 4 PREMNOTE quired to address gender dimensions, such as time use and and examines how gender rules determine the economic control of productive resources, which are not included and social roles and opportunities of different groups of in conventional M&E. women and men, and how these affect their ability to The selection of gender-responsive indicators should participate in and contribute to development. For the follow the standard indicator development guidelines purposes of monitoring, a gender analysis framework (Castro 2011; box 2). For example, CREAM (Morra (GAF) helps define the indicators and measures to be Imas and Rist 2009, 117) is a checklist to ensure that included in the GMES. There are many different GAFs, indicators are: each focusing on different aspects of women’s (and • Clear: precise and unambiguous; sometimes men’s) socially defined roles and how these • Relevant: appropriate to the subject at hand; affect and constrain their participation in household and • Economical: available at a reasonable cost; economic and political activities, including development • Adequate: able to provide sufficient basis to assess programs. performance; and The Harvard Gender Framework is one of the most • Monitorable: amenable to independent validation. widely used GAFs (Williams 1994; Rao, Anderson, and Assessing the quality of services is particularly impor- Overholt 1995). It argues that women’s participation in tant for gender analysis because the design and delivery of and enjoyment of the benefits of development projects program services often are not adapted to the special needs are constrained by their heavy time burdens, resulting of women. For example, many programs do not make pro- from multiple roles in production (of goods and services), vision for child care for women attending meetings or for reproduction and maintenance of human resources (reproduc- project workdays, or select locales that women can easily tion and care for the family members) and responsibilities reach or where they feel comfortable. for maintenance of community resources, and by unequal access to and control of productive resources. These are Identifying gender indicators that go measured through two instruments: an Activity Profile and beyond sex disaggregation an Access and Control Profile, which quantify the number While sex disaggregation of standard indicators is an im- of hours per week that different household members portant and economical way to begin examining gender spend on different activities and rate the level of access differences in program implementation and outcomes, and control that different members have over different there are many important gender dimensions that dig resources (land, equipment, labor, capital, animals, and so deeper to examine how gender relations affect develop- forth). The framework can be used for GMES to measure ment outcomes. Gender analysis provides tools to help changes in women’s time use and control of resources at understand the underlying causes of gender inequalities the start and end of the project. Incorporating gender into a results- Box 2. Relevant Notes in PREM’s Special Series based M&E framework on the Nuts & Bolts of Government M&E Systems Most development agencies now use results-based M&E systems to monitor implementation and outputs and to # 5 Hatry, H. 2010. “Key Steps in Designing and evaluate program effectiveness. According to Kusek and Implementing a Monitoring and Evaluation Pro- cess for Individual Country Service Agencies.” Rist (2004), the key elements of a results-based M&E system include: # 7 Sharma, R. 2011. “Use of Social Accountability • baseline data to describe the problem or situation Tools and Information Technologies in Monitoring before the intervention; and Evaluation.” • indicators for outcomes; # 9 Adato, M. 2011. “Combining Quantitative and • data collection on outputs and how and whether they Qualitative Methods for Program Monitoring and contribute to achievement of outcomes; Evaluation: Why Are Mixed Method Designs • more focus on perceptions of change among stake- Best?” holders; # 12 Castro, M. F. 2011. “Defining and Using Per- • systematic reporting with more quantitative and formance Indicators and Targets in Government qualitative information on progress toward outcomes; M&E Systems.” • collaboration with strategic partners; and Source: Author’s compilation. Note: The series can be downloaded at http://web.worldbank.org/ • capture of information on success or failure of partner- WBSITE/EXTERNAL/TOPICS/EXTPOVERTY/0,,contentMDK:22632 ship strategy in achieving desired outcomes. 898~pagePK:148956~piPK:216618~theSitePK:336992,00.html. PREMNOTE 5 For a GMES, all appropriate indicators must be sex C. gender indicators for each measurement—these may disaggregated, and additional gender-specific indicators either be simple sex disaggregation or may involve ad- need to be included with information on how gender data ditional gender-specific indicators; will be collected. D. data collection methods—whether primary or second- ary data are used, and also, where appropriate, who Gender-Responsive Monitoring: collects/provides the information; What Does It Look Like? E. whether baseline data are needed; F. frequency with which data are collected; and For most development programs, a gender monitoring G. how the information will be used—this may include: system looks very much like a conventional monitoring providing information to management on areas where system (Hatry 2010; Castro 2011), with some additional women or men (or particular age, ethnic, or other questions to measure differences in how women and men groups) are underrepresented, identifying special ser- participate in and are affected by the program. Much of vices required by particular groups, and providing the additional information is obtained by ensuring that information to stakeholders or the general public on standard information on program participation or outputs progress toward meeting gender equity objectives. is disaggregated by sex. However, it will often be necessary to collect additional information not included in conven- Defining gender-monitoring indicators tional monitoring. For example, there may be questions on Gender indicators are required for each of the levels in table the special needs or constraints of women and men for a 1. The InterAgency and Expert Group on Gender Statistics particular program, and the services or resources provided (UN Statistics Division undated) has developed a core list to address these differences. Some of this information, for of 51 gender indicators divided into five areas: example on family resistance to the participation of women i. economic structures, participation in productive activi- or girls, women’s lack of control over economic resources, ties, and access to resources; or threats of violence against women who attend meetings, ii. education; may require special data collection methods (such as focus iii. health and related services; groups, in-depth interviews, key informants, or observation). iv. public life and decision making; and In cases where the primary objective of the program is v. human rights of women and girl children. to achieve changes in gender relations or to address a prob- Each indicator can provide comparative information lem that specifically affects women, such as gender-based on women and men. For example, indicator 1, “average violence, the monitoring system may rely more on qualita- number of hours per week spent on unpaid domestic work tive methods such as case studies, in-depth interviews, and child-care” (UN Statistics Division undated), can be participant observations, or focus groups. disaggregated to provide the comparison between hours Defining the gender-sensitive monitoring format spent on unpaid housework and unpaid child care for Table 1 outlines a format that can be used to design a gender women and men. For each project, the appropriate indica- monitoring system. For each of five levels, (inputs, activities, tors can be selected and an assessment made of which data outputs, outcomes and impacts) the format defines: are currently available and what additional data could be B. what is measured; generated if necessary. Table 1. Format for Gender Monitoring System G. How gender- C. Gender- D. Data E. Are base- F. Frequen- sensitive B. What is sensitive collection line data cy of data information A. Levels measured indicator methods needed collection is used Impacts Outcomes Outputs Activities Inputs Source: Author’s compilation, adapted from World Bank, FAO, IFAD (2009, module 16, tables 16.1 and 16.2). 6 PREMNOTE Other indicators can rate the performance of an agency United Kingdom’s Department for International Develop- on the design and implementation of gender policies and ment (DFID) uses a set of gender equality markers that are programs, for example: rated on a four-point scale to assess the implementation of • Were analysis and/or consultations conducted on all of its programs. gender-related issues? • Were specific actions taken to address the distinct needs of women and girls and men and boys? • Were M&E of gender impacts conducted? Box 3. Some Qualitative Gender-Responsive Data A later section on gender checklists (see also box 5) Collection Methods includes more discussion on performance indicators. • Focus groups • In-depth interviews Data collection methods Gender monitoring can use all of the conventional data col- • Key informants lection methods after they are adapted to address specific • Observation gender issues. These methods usually will combine primary • Participant observation data from surveys, interviews, reports on meetings and use • Social mapping and other participatory group of services and others, with secondary data such as infor- consultation techniques mation from the management information system, project • Using mobile phones for interviews and feedback records, minutes of meetings, and so forth. Many monitoring systems rely mainly on quantita- • Photos sent through mobile phones and compari- son with GPS location tive data recording: how many (for example, people at- tending meetings), how much (for example, conditional • Story-telling cash payments or amount of food for work), or how long Sources: Williams 1994; Kumar 2002; Bamberger, Rugh, and Mabry 2012, chapters 13 and 14; Patton 2002; Sharma 2011; Adato 2011. (duration of training programs or community road main- tenance activities). While these kinds of data are essential, quantitative indicators can fail to capture the quality of participation or services provided. For example, women may attend meetings, but may have limited participation Box 4. Examples of Secondary Databases for in decision making, receive less support from agricultural Gender Analysis extension workers, or receive less courteous service from • UNDP Human Development Index and the Gender financial institutions than do men. Consequently, it is Inequality Index: These indices, which are avail- often necessary to complement quantitative data with able for over 180 countries, provide information qualitative data that assess the quality of services. Ex- on gender differences in life expectancy, access to amples include observation of women’s participation in health, education and labor markets, and political meetings or focus groups with women to obtain feedback participation. on services received (see box 3 for more details). • The UNDP Multiple Dimension Poverty Index: This can be disaggregated to provide comparative data Secondary data sources for male- and female-headed households. There are a wide variety of secondary data sources that • MDG databases: Can provide cross-country com- can be used to construct/reconstruct baseline data or as parative, sex-disaggregated data on topics such a comparison group. Some of these data sources provide as access to health, education, water supply, and cross-country comparative data, others provide national- sanitation. level data, while others can be used at the regional or local • Demographic and Health Surveys (DHS), UNI- level. Box 4 provides examples of some of these databases. CEF’s Multiple Indicator Cluster Survey (MICS), and the World Bank Living Standards Measure- Gender checklists ment Study (LSMS): These are examples of Many agencies use checklists to assess whether the design comparative cross-country databases that provide or implementation of a project, program, or policy is ad- sex-disaggregated socioeconomic data at the dressing key gender issues. For example, The U.S. Agency national, regional, and often community level. for International Development (USAID) has developed Source: Author’s compilation. a seven-question checklist for this purpose (box 5). The PREMNOTE 7 Box 5. USAID Checklist for Assessing Box 6. Examples of Randomized Control Trial Achievement of Gender Objectives Evaluations for Assessing Gender-Related Outcomes 1. Number of laws, policies, and procedures promoting gender equality that are proposed/ • Job networks and gender in Malawi adopted. • The effect of gender equity programs on mater- 2. Proportion of female participants. nal and child care in Uganda 3. Proportion of females who report increased self- • The impact of women policy makers on public efficacy (a feeling of increased control over their goods in India lives). • Evaluating school-based awareness and mo- 4. Proportion of participants who agree that bilization campaign to address sex-selective women/men should have equal access to social, elimination in Haryana, India economic, and political opportunities. Source: Poverty Action Lab, http://www.povertyactionlab.org. 5. Number of laws drafted with U.S. government support. 6. Number of people reached by projects who Special gender-focused evaluation methods received gender-based violence support. Many gender evaluations use mixed methods and qualita- 7. Proportion of the project population reached who tive data collection and analysis methods. Some of the think gender-based violence is less acceptable. widely used qualitative methods include: Source: USAID 2012. • outcome mapping (Earle, Carden, and Smytlo 2001); • most significant change (Davis and Dart 2005); • focus groups (Krueger and Casey 2000); • safety audits (Women in Cities International 2010); Engendering the Evaluation System • participatory group consultation methods such as Gender impact evaluation designs social mapping, wealth ranking, and historical time Gender outcomes and impacts can be evaluated in several lines (Kumar 2002); and ways: • longitudinal and cross-sectional case studies. • by including gender indicators in a standard impact Mixed-methods (Bamberger, Rugh, and Mabry 2012, evaluation design; chapter 14; Adato 2011) gender evaluation designs com- • by adding a gender-specific module or data collection bine qualitative data collection methods (such as those method to an impact evaluation design (for example, mentioned above) that provide in-depth understanding of administering a special module to women in a sub- lived experience and the quality of services and documen- sample of households when only the “household head,” tation of real-world processes through which programs are in most cases a male, is interviewed); and implemented on the ground with statistical analysis and • implementing a special gender impact evaluation—this sampling techniques that make it possible to generalize option may be appropriate when gender outcomes are findings from the in-depth qualitative methods. This is the primary program goal. important, because in the past, many gender evaluations The whole range of conventional impact evaluation provided valuable insights into individual lives and groups, designs can be used for the first two options. However, but tended to pay less attention to how the cases were until recently, many gender impact evaluations used selected and how representative they were. Some of the predominantly qualitative or mixed-methods design, mixed-methods techniques that permit generalization from partly because many of these projects were relatively small relatively small samples of cases include: scale. Now, as gender is becoming a central policy objec- • quota sampling, to ensure that cases are selected to tive of many international donors and nongovernmental cover all groups of interest and that they are broadly organizations (NGOs) and the scale of gender-related representative; interventions has increased, there has been an increasing • concept mapping (Kane and Trochim 2007), to select use of experimental and quasi-experimental designs for the sample of case studies using expert judgment; evaluating gender outcomes. Box 6 illustrates some of the • for international donors, a portfolio analysis to select kinds of randomized control trials that have started to be a representative sample of countries or programs for used over the past few years. in-depth analysis; and 8 PREMNOTE • secondary data sets such as the United Nations De- About the Author velopment Programme’s (UNDP) Gender Inequality Michael Bamberger has a PhD in sociology from the London Index or the Organisation for Economic Co-operation School of Economics. He lived and worked for 10 years in and Development’s (OECD) Social Institutions and Latin America, working with a number of nongovernmen- Gender Index to select samples of countries that tal organizations on poverty and urban development, and have high, low, or medium scores on gender equal- conducting research and evaluations on gender and poverty. ity indices. After joining the World Bank in 1975, he coordinated a five- year evaluation of the Bank’s first low-income urban hous- Conclusions ing programs, which included a number of studies on the There are significant differences in how women and men gender dimensions of housing and also on survival strategies are affected by and respond to development interventions. of the urban poor. He worked for nine years in the Bank’s But despite compelling evidence on persistent gender Gender and Development Department, where he advised on inequalities and their consequences for development pro- gender evaluations in a number of sectors and represented grams, conventional M&E systems frequently fail to address the department on the Bank’s Poverty Working Group. Since gender differences. Consequently, government commit- retiring from the Bank in 2001, he has participated in many ment to achieving gender equality requires development gender evaluations, including assessments of the gender of a GMES. This paper draws on international experience policies of the United Nations Development Programme, the of governments, donor agencies, and NGOs to outline the World Food Program, the World Bank, and is a member of the main steps in the design and implementation of such a International Expert Advisory Group for the United Nations GMES. The approach will vary depending on whether the Women’s Program on Making Cities Safe for Women and Girls. GMES is part of a national gender policy or is intended to improve the efficiency and effectiveness of M&E systems at the agency or program level. References The first step is identifying the gender objectives of Adato, M. 2011. “Combining Quantitative and Qualitative Meth- the policy or program to which the GMES will be applied. ods for Program Monitoring and Evaluation: Why Are Mixed- These can be very broad (such as achieving one or more of Method Designs Best?” PREMnote Special Series on the Nuts & the MDGs), or much narrower (such as increasing women’s Bolts of M&E Systems Note 9, World Bank, Washington, DC. Bamberger, M., J. Rugh, and L. Mabry. 2012. Realworld Evaluation: access to microcredit and agricultural marketing services). Working under Budget, Time, Data and Political Constraints. The second step is to develop a gender framework defining Thousand Oaks, CA: Sage Publications. the gender indicators and how they will be measured. This Castro, M. F. 2011. “Defining and Using Performance Indicators will usually involve a number of substeps such as defining and Targets in Government M&E Systems.” PREMnote Special the TOC, operationalizing the TOC through a logframe, Series on the Nuts & Bolts of M&E Systems Note 12, World and incorporating the indicators into a gender-responsive Bank, Washington, DC. results-based M&E framework. Once the indicators have CIDA (Canadian International Development Agency). 2010. been defined, the third step is to develop the GMES. This “CIDA’s Framework for Assessing Gender Equality and normally looks similar to a conventional M&E system, but Results.” http://www.acdi-cida.gc.ca/INET/IMAGES.NSF/ with most indicators disaggregated by sex and the addi- vLUImages/GenderEquality3/$file/GE-Framework-EN.pdf. tion of some gender-specific indicators. The final step is to Davis, R., and J. Dart. 2005. “The ‘Most Significant Change’ develop the gender-responsive evaluation system. While (MSC) Technique: A Guide to Its Use.” http://www.mande. this can draw on conventional program evaluation tools, co.uk/docs/MSCGuide.pdf. the need to focus on processes and capture sensitive infor- DFID (Department for International Development, United mation means that greater reliance will usually be placed Kingdom). 2008. The Gender Manual: A Practical Guide. http://www.gadnetwork.org.uk/storage/dfid-gender-man- on mixed methods and qualitative data. It is important to ual-2008.pdf. recognize that gender differences and processes are often ———. 2012. “A Theory of Change for Tackling Violence against based on social traditions and customs that may be invisible Women and Girls.” Guidance Note 1. http://www.gadnet- to many agencies or not reported, particularly when they work.org.uk/storage/VAWG_guidance1_toc1.pdf. touch on sensitive areas. While the implementation of a Earl, S., F. Carden, and T. Smytlo. 2001. “Outcome Mapping: GMES requires gender awareness and gender-focused capac- Building Learning and Reflection into Development Pro- ity development, experienced managers and M&E staff will grams.” International Development Research Centre, Ot- find that most of the tools and techniques of GMES will not tawa. http://www.idrc.ca/EN/Resources/Publications/Pages/ be too difficult to incorporate into the existing M&E system. IDRCBookDetails.aspx?PublicationID=121. PREMNOTE 9 Hatry, H. 2010. “Key Steps in Designing and Implementing a UN ESCAP (United Nations Economic and Social Commission Monitoring and Evaluation Process for Individual Country for Asia and the Pacific). 2007. “Economic and Social Survey Service Agencies.” PREMnote Special Series on the Nuts & of Asia and the Pacific.” http://www.unescap.org/pdd/publica- Bolts of M&E Systems Note 5, World Bank, Washington, DC. tions/survey2007/01_Survey_2007.pdf. Hesse-Biber, S., ed. 2012. The Handbook of Feminist Research: UN Statistics Division. Undated. http://unstats.un.org/unsd/ Theory and Praxis. Thousand Oaks, CA: Sage. gender/default.html. Kane, M., and W. Trochim. 2007. Concept Mapping for Planning USAID (U.S. Agency for International Development). 2012. and Evaluation. Thousand Oaks, CA: Sage. “USAID Gender Equality and Female Empowerment Policy.” Krueger, R., and M. Casey. 2000. Focus Groups: A Practical Guide Washington, DC. for Applied Research, 3rd Edition. Thousand Oaks, CA: Sage. Women in Cities International. 2010. “Together for Women’s Kumar, S. 2002. Methods for Community Participation: A Complete Safety: Creating Safer Communities for Marginalized Women Guide for Practitioners. London: ITDC. and Everyone.” Kusek, J., and R. Rist. 2004. Ten Steps to a Results-Based Monitoring World Bank. Undated a. Compilation of Gender Data and Indica- and Evaluation System. Washington, DC: World Bank. tors Available by Country. http://datatopics.worldbank.org/ Morra Imas, L., and Ray Rist. 2009. The Road to Results. Washing- gender/. ton, DC: World Bank. ———. Undated b. Living Standards Measurement Study. www. Moser, C. 1989. “Gender Planning in the Third World: Meeting worldbank.org/lsms. Practical and Strategic Needs.” World Development, OECD ———. 2010. “Development for All: Gender Action Plan for East Asia Social Institutions and Gender Index. http://www.oecd.org/ and the Pacific FY11–13.” http://siteresources.worldbank. dev/poverty/theoecdsocialinstitutionsandgenderindex.htm. org/INTGENDER/Resources/336003-1277693634508/ Patton, M. Q. 2002. Qualitative Research and Evaluation Methods. EAPGAP_May2010.pdf. Thousand Oaks, CA: Sage ———. 2012. World Development Report 2012: Gender Equality and Rao, A., M. Anderson, and C. Overholt. 1995. Gender Analysis in Development. Washington, DC. Development Planning. Connecticut: Kumarian Press. World Bank, FAO (Food and Agriculture Organization), IFAD Sharma, R. 2011. “Use of Social Accountability Tools and Informa- (International Fund for Agriculture Development). 2009. tion Technologies in Monitoring and Evaluation.” PREMnote Gender in Agriculture Sourcebook , Module 16. http:// Special Series on the Nuts & Bolts of M&E Systems Note 7, web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EX- World Bank, Washington, DC. TARD/EXTGENAGRLIVSOUBOOK/0,,contentMDK:2 Williams, S., with J. Seed and A. Mwau. 1994. The Oxfam Gender 1348334~pagePK:64168427~piPK:64168435~theSite Training Manual. UK and Ireland: Oxfam. PK:3817359,00.html. This note series is intended to summarize good practices and key policy findings on PREM-related topics. The views expressed in the notes are those of the authors and do not necessarily reflect those of the World Bank. PREMnotes are widely distributed to Bank staff and are also available on the PREM Web site (http://www.worldbank.org/ prem). If you are interested in writing a PREMnote, email your idea to Madjiguene Seck at mseck@worldbank.org. For additional copies of this PREMnote please contact the PREM Advisory Service at x87736. This series is for both external and internal dissemination 10 PREMNOTE