WPS6958 Policy Research Working Paper 6958 The Anatomy of Failure An Ethnography of a Randomized Trial to Deepen Democracy in Rural India Kripa Ananthpur Kabir Malik Vijayendra Rao The World Bank Development Research Group Poverty and Inequality Team June 2014 Policy Research Working Paper 6958 Abstract Programs that induce citizen participation to improve are not statistically significant. The detailed qualitative the quality of government at the local level are the data from a 10 percent subsample allow us to unpack subjects of large amounts of funding and intense debate. the reasons why the intervention “failed,” highlighting This paper combines a randomized control trial of a the role of variations in the quality of facilitation, lack of citizenship training and facilitation program in rural top-down support, and difficulties with confronting the India, with an in-depth, four-year ethnography of the stubborn challenge of persistent inequality. However, the intervention to understand the underlying mechanisms qualitative investigation also uncovered subtle treatment of change. The quantitative data show no impact from effects that are difficult to observe in structured surveys. the intervention. Household and village survey data from The paper thus demonstrates that a concerted effort to 100 treatment and 100 control villages show considerable use “thick description” to uncover the process of change improvement across a wide variety of governance and using careful and detailed qualitative work can add value participation indicators over time, but the differences to standard impact evaluations. in the changes between treatment and control villages This paper is a product of the Poverty and Inequality Team, Development Research Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at vrao@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team The Anatomy of Failure: An Ethnography of a Randomized Trial to Deepen Democracy in Rural India Kripa Ananthpur (Madras Institute of Development Studies) Kabir Malik (The World Bank) Vijayendra Rao (Development Research Group, The World Bank) JEL Codes: O12, H79, C1 Keywords: Panchayats, RCT, Mixed-Methods Acknowledgements: The authors are grateful to Wilfred D’Souza and H.S. Ashokanand who were then with the Karnataka State Institute for Rural Development (KSIRD), and who led the intervention team, for their advice, openness and help with conducting this evaluation. They are also grateful to Jeff Hammer who was part of the initial research team and helped design many elements of this study, to Radhika Vishwanathan for superb research assistance, to Geeta Sethi and Yongmei Zhou from the World Bank’s operations team for facilitating the research, to Yamini Aiyar for advice and a crucial infusion of funds, and to Usha Rao for her important contributions in understanding the role of RPs. They are particularly indebted to Iranna Muragod, Narasimha Reddy, and the other members of the qualitative team for their brave, exemplary work. IMRB conducted the quantitative survey. Jonathan Fox, Saumitra Jha and Farah Zahir provided valuable comments on a previous draft. The research was funded by grants from the World Bank’s Research Support Budget, KCP Trust Funds, TFESSD Trust Funds from the Government of Norway, and the Accountability Initiative at the Center for Policy Research. 1. Introduction Can civic participation be externally induced? Can a policy intervention deepen democracy? These questions lie at the heart of development “as freedom” (Sen, 1999). Over the last two decades there has been a concerted effort by aid agencies to induce participation around the world with mixed results, with the World Bank alone spending around $85 billion (Mansuri and Rao, 2013). This paper examines the impact of a two-year effort to deepen democracy in the poor and arid region of north Karnataka in south- central India. The intervention was evaluated using a mixed-method design. Survey data was collected over two rounds on a sample of two hundred villages, with half randomly assigned to be treated by the intervention. In addition, ethnographic methods were used to track a 10 percent subset of the quantitative sample over a four-year period. Thus, the paper also makes a methodological contribution to the literature on impact evaluations by combining a randomized control trial (RCT) with in-depth ethnographic research to understand the mechanisms behind the changes in outcomes measured by the two rounds of quantitative surveys 1. Summarizing the findings in brief, the quantitative data show no impact from the intervention. Both treatment and control areas demonstrate considerable change over time, but the differences in the changes between them are not statistically significant. The detailed qualitative data allow us to unpack the reasons why the intervention “failed” highlighting the role of variations in the quality of facilitation, lack of top-down support, and difficulties with confronting the stubborn challenge of persistent inequality. However, the qualitative investigation also uncovered subtle treatment effects that are difficult to observe in structured surveys. The paper thus demonstrates that a concerted effort to use thick description to uncover the process of change using careful and detailed qualitative work can add value to standard impact evaluation techniques. There is a growing literature on the impact of interventions that attempt to improve the quality of participatory decision-making and planning at the community level. RCTs of such interventions generally find limited impact. Olken (2007) examines an intervention in Indonesia where communities that were part of the large participatory Kecamatan Development Program, were randomly assigned to a top-down effort at monitoring village expenses via an external audit, and to villages where communities were encouraged to participate in village meetings where local officials documented their expenditures. 1 Rao and Woolcock 2003, and White 2011, among others make the case for such mixed-methods evaluations to understand both “outcomes” and “processes” but there are very few published examples of such work - particularly with RCTs. 2 He finds that the audits were more far more effective at reducing corruption than community meetings. Banerjee et al (2010) report on an experiment to induce community monitoring of schools in rural India. They find that providing information about school performance to members of a village school committee does not improve the performance of schools. Humphreys, Masters and Sandbu (2006) examine the random assignment of facilitators (discussion leaders) to run community level forums as part of a nationwide deliberative exercise in Samo Tome and Principe. They find the variation in the characteristics of facilitators explains most of the variance in the quality of discussions. Paluck (2010) tests the impact of a year-long radio talk show designed to promote deliberation, and randomly assigned to villages, in the Democratic Republic of Congo. She finds that talk show listeners were likely to engage in group discussion, but also more intolerant, focused on grievances, and less likely to aid people they disliked. Qualitative studies on the impact of such interventions have come to more optimistic conclusions. Gaventa and Barett (2010) review several cases of participatory efforts at the local level and find that they usually have a positive impact. Barron, Woolcock and Diprose (2011), like Olken, examine the impact of KDP in Indonesia, and find that the deliberative spaces that KDP introduces into villages reduce the potential for conflict to turn violent largely because decisions emerge from a consultative process that communities perceive as legitimate. Baiocchi, Heller and Silva (2011) analyzing the impact of introducing participatory budgeting in areas of Brazil that did not have it, find that local context matters a great deal in processes that deepened democracy. In particular they highlight the importance of the state’s role in creating an “associational environment” to facilitate the process by which bottom up participation is sustainably introduced. The importance of this “sandwich” where top-down efforts are coordinated with bottom-up mobilization was first highlighted by Fox (1992) in his seminal case-study of the Mexican Food System safety net program. The rural Indian context is particularly interesting for studying the process of democratic deepening because it has been the site of an important attempt to bring democracy to the grassroots. The 73rd amendment to the Indian constitution, passed in 1992, ensures that all villages are governed by an elected village council known as the Gram Panchayat (GP), led by a democratically elected president with panchayat elections implemented by an independent election commission. Further, the amendment institutes a gram sabha, a deliberative body to which every village resident belongs, that meets periodically (from twice to four times a year depending on the state) and is expected to debate, discuss and ratify budgetary allocations, the selection of beneficiaries for public programs, and other important issues. Therefore, India’s two million villages are the site of the most widespread attempt to deepen democracy 3 in human history. In practice, however, social scientists studying this experiment have uncovered a variety of problems including elite dominance, corruption, high levels of clientalism, poor tax compliance and, consequently, an almost total dependence on outside grants for revenues (e.g. Bardhan and Mookherjee 2011; Rajaraman 2003). Studies of deliberative decision-making within the panchayat system in South India have found that gram sabhas can benefit citizens in several ways. Ban, Jha and Rao (2012), analyzing transcripts of 300 gram sabhas, find that the issues raised within them reflect the preferences of the median citizen. Discourse within gram sabhas could thus be interpreted as honing close to the interests of the median voter and thus as “efficient democracies”. Rao and Sanyal (2010) in a qualitative analysis of the same data find that underprivileged castes tactically use the gram sabha as a forum to make identity claims and to question the criteria for poverty targeting. Despite their potential value to citizens, Besley et al (2005), however, find that gram sabhas are often not held regularly. But, when they are held, benefits from public programs tend to be better targeted towards the poor. Besley et al (2005) are unable to draw causal links in their analysis because gram sabhas are not randomly assigned. Thus, while there is evidence to suggest that gram sabhas have intrinsic discursive value and show considerable potential for democratic deepening, it is unclear whether holding them results in a consequent improvement in the quality of village government. And, therein lies the rub; can gram sabhas and other forms of everyday democratic participation play an instrumental role in holding village governments accountable and make them more responsive to the needs of citizens? Is it possible to strengthen village level deliberative processes via an external intervention to deepen democracy? The key to the success of such interventions according to Fung and Wright (2003) is to build “empowered participatory governance” which is characterized by three important principles: (a) it must have a practical orientation, (b) it must involve all the people most affected by and knowledgeable about the problems being addressed, and (c) it must be deliberative. Such interventions, they find, require Fox’s “sandwich” in order to work with meaningful devolution of functions and finances and centralized supervision and control. The People’s Campaign in the Indian state of Kerala was one such intervention. Kerala state, which has the highest levels of human development in India, pioneered an important attempt to deepen democracy in the process of implementing the 73rd amendment. There were two elements to this “People Campaign”: First, the state government devolved significant resources to the GPs with 40% of the state’s total expenditures allocated to them; second, a grassroots training and 4 awareness-raising effort to inform citizens of their rights and duties and mobilize them to participate in the panchayat system. These two elements, it is has been argued, have played a significant part in making Kerala’s panchayats very effective (Heller, Harilal and Chaudhuri 2008). The Campaign was initiated in 1996 by the communist (CPI-M) led government that was in power in Kerala at the time. It worked on many fronts. First it instituted a planning process based on a set of nested stages that were structured to facilitate participation via working committees meetings and development seminars that culminated in a gram sabha. Instead of open deliberation, attendees (members of the public) were divided into resource-themed groups or committees; the discussions within each group yielded consensual decisions regarding the designated resource. This structure, which operated uniformly in all districts in Kerala, was geared toward increasing the efficiency of consensual decision making about public resource demands, and prioritizing individual beneficiaries for the allocation of government-subsidized private benefits. The process included various training programs to instruct citizens on deliberative planning, and to instruct village functionaries on methods for turning village plans into directed actions to effect better public service delivery. Kerala, however, is an outlier among Indian states with almost 100% literacy and very politically aware and mobilized citizens. Thus, the question that lies at the heart of this paper is whether the Kerala experiment can be replicated in more unequal, less literate, and therefore more representative areas of India. The northern districts of the neighboring state of Karnataka represent such a setting. The literacy rate is at about 60% with high levels of poverty and inequality and a feudal social environment characterized by high levels of corruption and extremely poor governance. Therefore, if an intervention like the People’s Campaign could work in northern Karnataka, it could provide an important tool to transform village of democracy in India by sharply increasing the quality and quantity of citizen participation in the panchayat system that, in turn, can potentially have a significant effect on standards of living. Like other poor regions of India, GPs in northern Karnataka have had access to several, large infusions of funds since 2005 including the Mahatma Gandhi National Rural Employment Guarantee Scheme (MNREGS), the Backward Regions Grant Fund (BRGF) and the World Bank supported Gram Swaraj Project. Together, these raised the annual budgets from about 200,000 Indian rupees a year (about $4000 at the time) to approximately 4,000,000 rupees ($80,000). Thus GPs in these areas fulfilled the first element of the Kerala program – high levels of funding, with much of it discretionary. Our evaluation focuses on the second element of the program - assessing the impact of a “People’s Campaign” 5 intervention which attempted to improve the quality of participation and deepen democracy within the panchayat system. Bureaucrats and activists from the Karnataka State Institute Development (KSIRD) designed the intervention. They were keen to develop a People’s Campaign that was contextualized to the low literacy, high inequality, and semi-arid context of north Karnataka. The KSIRD, at the time, was responsible for all panchayat training in the state and had many years of experience in the field. The goal of the intervention was to make village government more accountable to citizens by (i) increasing participation in gram sabhas, (ii) making citizen engagement with GPs more informed, and (iii) improving cooperation between GPs and low-level government officials. The “treatment” consisted of three phased interventions mimicking many aspects of the Kerala People’s Campaign: The first phase in the intervention, a citizenship engagement program which lasted one week, began with a pair of facilitators visiting every neighborhood in the village encouraging citizens to attend neighborhood meetings or “ward sabhas.” In the ward sabhas citizens were informed about a schedule of training that instructed citizens in processes of participatory planning and deliberative decision-making, and disseminated information about the budgets, programs and procedures of the gram panchayat. Among other things, the initial intervention explained the purpose and value of the gram sabha, rights of citizens such as the right to information, and information on how to examine panchayat records such as budgets, and minutes of meetings. At the end of the week, the project facilitators, who were called Resource Persons (RPs) held a special gram sabha where priorities were finalized and listed in a Village Action Plan that was ratified in the gram sabha. The second phase, which occurred immediately after the ratification of the Village Action Plan, consisted primarily of a meeting with local bureaucrats to reach an implementation agreement where the bureaucrats committed to provide funding and technical support for projects selected by the Village Action Plan over the course of the year. In the third phase, which was spread over a period of two years, the GP was monitored for approximately two days every month by KSIRD’s team of RPs with “handholding” to ensure the program’s progress. In these follow up visits, citizen initiatives were tracked and assisted by the RPs. Importantly the RPs ensured that gram sabhas were held according to schedule. They also tried to follow up on the implementation of the Village Action Plan and helped citizens and GP officials if they needed to visit local bureaucrats. 6 In the rest of this paper we analyze the impact of this complex intervention and the process of change it attempted to initiate. In the next section we outline the analytical design of the quantitative and qualitative analysis. In section 3, we briefly report the quantitative findings demonstrating the lack of measured impact from the intervention. The fourth section analyzes the qualitative analysis, and the fifth section concludes the paper and summarizes the findings. Analytic Design Quantitative Analysis The quantitative sample was chosen from among the poorest districts of Karnataka State: Gulbarga, Raichur, Bidar, Davengere and Chitradurga. From these districts the 20 poorest talukas (sub-districts known as blocks in most parts of India) were chosen for the study because they were also selected as “backward” by the Government of India and therefore had access to a large infusion of public funds from the MNREGA and BRGF. To keep administrative differences between taluka’s constant, two or three GPs (depending on the taluka’s population) were randomly chosen from each taluka to be assigned to the treatment sample, and the same number chosen from the same taluka as control GPs. This resulted in a total sample of 100 GPs, with 50 randomly assigned for treatment. To avoid the possibility of spillover effects, control GPs were constrained in the sampling process to be located at least one GP away from the treatment 2. While the intervention was targeted at the GP level, two villages from each GP were selected for the survey. The first village was the GP headquarters, typically the largest village in the GP, and the second village chosen at random from the other villages in the GP (a GP typically has between 3-5 villages). Within each village twenty households were chosen at random for the household survey, and 2 village leaders, the GP president and the vice-president, were also interviewed. This resulted in sample size of 100 treatment and 100 control villages 3 selected from 50 treatment and 50 control GPs, and 4000 households. The baseline survey was administered in October-November 2007. Our aim was to complete the survey before the initial visits to the villages by KSIRD in late November 2007. However, in some villages, due 2 Our ethnographic work found no evidence of spillovers from the intervention. 3 Selection is such that villages with higher population have a proportionally higher probability of being selected. 7 to poor coordination between the intervention and survey teams, the survey was administered either during or shortly after the initial two-week training program. Two years later, in October-December 2009, the same sample of villages and households were re-interviewed with identical survey instruments. The attrition rate of household respondents was relatively high for a variety of reasons and 3545 households were visited on both rounds of the panel 4. This is the sample we analyze. The survey instruments, developed after several weeks of investigative field-work and pre-testing, include village-level modules measuring the quality and quantity of public goods, and in-depth interviews with village politicians and local officials. The instruments were developed jointly with KSIRD staff to ensure that they captured all the outcomes that were considered important by KSIRD, and also benefited from KSIRD’s considerable experience with working in the region. The sample households were assessed for their socio-economic status, preferences for public goods, political participation, social networks and other relevant variables. In addition, focus group discussions with groups selected to represent all the different castes in the village were used to elicit information about village-level activities. Detailed maps outlining the availability of public goods such as village roads, water tanks, schools, etc were also drawn in each round. For the analysis, given the negligible impact of the intervention on almost all outcome variables, we report results from a few important measures of information availability, participation, and the provision of public goods. The method of analysis for the quantitative data is a difference-in-difference between treatment and control samples, and over the two rounds. Household and village level variables are analyzed with standard errors clustered at the GP level. We examine a variety of outcomes: participation in village meetings and in gram panchayat activities; knowledge and quality of information about village governance; quality of public goods – measured by quality of construction and availability of resources; whether the gram panchayat has proper records of budgets; perceptions about performance of gram panchayat and quality of village services; distribution of public and private goods allocation by the village; participation in MNREGA and whether the GPs operating procedures followed official regulations. We use a standard difference-in-difference specification to estimate the impact of the intervention. The estimation controls for time-invariant differences between the two groups, and changes across time that affect both treatment and control samples. The estimated equation takes the form below: 4 The relatively high levels of attrition were due to high levels of migration from these villages and to problems with inconsistent tracking of households over the two periods. The rate of attrition was not significantly different between treatment and control samples (p-value of 0.49). 8 = + βR t + γTv + δIvt + εivt where, is the outcome for individual i , in village v at time t (for the village level outcomes the subscript is vt). R t is a dummy (=1) for the follow-up survey, Tv is a dummy (=1) for the treatment group and Ivt is the dummy for the post-treatment effect on the treated—thus δ identifies the average treatment effect. The error term, εivt , is clustered at the level of randomization—the GP. In addition to the above specification, we also estimate models with individual fixed effects for household outcomes, and village fixed effects for village outcomes. This specification controls for any unobserved time-invariant heterogeneity that may exist at the individual/village level. Qualitative Analysis The qualitative sample is a 10% subset of the quantitative sample. The qualitative sample was chosen from Raichur and Gulbarga districts because the majority of the quantitative sample was located there, and from these districts we randomly selected three talukas from Gulbarga and two from Raichur. From each sampled taluka we picked a pair of treatment and control GPs that were the best matched in attributes measured in the 2001 census (the last available census at the time) 5. While the quantitative data are available for two years, our ethnographic investigation from five treatment and five control GPs covers a four-year period from 2007-2011 to allow us to examine the long-term effects of the intervention. From 2007-2010, each GP was assigned a field investigator, typically someone with an MA degree in a social science or in Social Work, who was from the region and therefore very familiar with the milieu and dialect and easily able to blend into the community and establish rapport. The investigator either resided in the GP or in a location that was a short, easily accessible distance away. In the first round of reports each investigator mapped the village’s social and political structure, outlining the various caste and religious groups residing in the GP, relationships within and between them, structures of social networks and power, major events in the GP’s history including its experience with development projects, etc. Subsequently, once a month, the investigators sent in a 5-10 page report on important changes that had taken place. S/he was instructed to record important local events, interview important actors in those events, investigate new village constructions and the financing behind them, track electoral activities and 5 The matching methodology employed for the qualitative analysis is similar to Barron, Diprose and Woolcock (2011) and Baiocchi, Heller and Silva (2011). 9 expenditures, examine changes in levels of local activism, and investigate other issues that were relevant to the political and economic life of the GP. In treatment GPs they were, in particular, asked to closely track the work of the KSIRD RPs, and to follow up on how their work percolated into the village, and the sequence of changes that were initiated by the work of the RPs. From 2010-2011, the team was reduced to three investigators who visited all the GPs on a rotating schedule sending in reports every three months. Consequently we have a total of about 400 reports divided equally between treatment and control GPs. These village reports, supplemented by regular field visits by the principal investigators, constitute our qualitative data that we distill and draw on for this qualitative section of the paper. Section 2: Quantitative Results First, we examine the trends in the quantitative data between 2007 and 2009 in Tables 1, 2 and 3 without distinguishing between control and treatment villages. Table 1 shows a sharp increase between 2007 and 2009 in levels of participation and knowledge about gram panchayat procedures and personnel. Participation in gram sabhas more than doubled. And there were large increases in the availability of information with improvements in the percentage of respondents of respondents who could correctly name their ward member or village president, who had heard of the Mahatma Gandhi National Rural Employment Guarantee Scheme, and had heard of gram or ward sabhas. Households also had greater access to other government transfer programs; their eligibility for a Below Poverty Line card, which provides access to subsidized food and other schemes went up from 53% to 85%. Tax compliance also significantly increased from 60% to 76%. The village-level data in Tables 2 and 3 demonstrate that this increase in participation and knowledge was associated with improvements in the quality of government. Table 2 shows that participants report an increase in gram panchayat activity in almost every sector – roads, transportation, schools, health, sanitation and irrigation. However they also report a reduction in activities related to the provision of drinking water. Table 3, showing results from detailed maps of village infrastructure, also indicates increases in the availability of schools, overhead tanks, roads, anganwadis (crèches), but also a reduction in the number of operational tube-wells and the availability of other water sources. Overall, it is clear that information about village government, participation in village government activities, and the performance of village governments significantly improved in both control and treatment areas from 2007 to 2009. We next examine whether the People’s Planning intervention had a significant impact on household and village level outcomes. First we analyze whether the data from the control and treatment groups are 10 balanced, i.e., on average, similar in their baseline characteristics. Table 4-7 compare the sample means of household economic and demographic characteristics, village infrastructure, and activity from the baseline survey in 2007. Table 4 shows that households in treatment and control GPs are statistically indistinguishable based on household demographic and economic characteristics such as gender, education, caste and asset ownership. 6 Tables 5 and 6 show that, on most measures of village infrastructure and activity, the treatment and control villages are reasonably well balanced, but a few variables are significantly higher for treatment villages, and others significantly lower. Treatment villages have a greater number of government schools and teachers, road construction, irrigation activity, and lower numbers of hand pumps, electricity construction activity and male agricultural wages. The results in Table 7, however, suggest a more systematic kind of difference; households in treatment GPs had greater pre-existing levels of awareness and participation compared to those in control GPs. The likely reason for this is that poor coordination between the intervention and survey teams resulted in KSIRD facilitators initiating their training in some treatment villages prior to the collection of the baseline data. Since the results from Tables 4-7 indicate that there may be baseline differences between the treatment and control groups, we need to account for these differences in estimating the treatment’s impact. To do this we estimate a difference-in-difference model first without, and then with, individual or village fixed effects depending on the nature of the data. Table 8, columns 1 and 3, show the results from an estimation of equation (1) for household awareness, participation and government transfers with individual data. There is no evidence that the intervention had a significant impact— both with and without individual fixed effects. On some measures, such as heard of MNREGS, and heard of gram or ward sabhas, the coefficient is positive and meaningful in magnitude, but very imprecisely estimated. For village outcomes, presented in Table 9 and 10, the results also show no impact from the treatment on most measures of village infrastructure and activity. We do, however, find a negative treatment effect for government schools and road construction and a positive treatment impact for number of hand pumps in the village. The standard errors of these estimates increase considerably with the inclusion of fixed effects making them significant only at the 10 percent level. The one exception is the male agricultural wage7 that increases by 10 percent and retains its significance even after adding GP fixed effects. This suggests that MNREGA, which guarantees employment for 100 days at a fixed wage, was implemented more effectively in treatment villages and that this had a general equilibrium effect on male agricultural wages. 6 Only one variable, TV ownership, is significantly more for the treatment households at the 10 percent level of significance. 7 Agricultural wages in India are determined at the village level with almost no individual variation. 11 In summary, the quantitative results demonstrate that there was an increase over time in both control and treatment areas in knowledge of and participation in village government, and a concurrent improvement in the delivery of village government services. However, the data do not indicate that the People’s Planning intervention had a significant impact across a wide spectrum of possible outcomes. The results, at best, show very weak evidence of both positive and negative impacts of the intervention on a very small number of outcomes. Section 3: ETHNOGRAPHIC ANALYSIS The qualitative data is very detailed and extensive and it impossible to do justice to all its nuances in a short paper. Consequently, our goal here is to limit our discussion to those issues that have a bearing on understanding the processes and mechanisms that led to the observed lack of impact in the quantitative data. Like other qualitative studies we emphasize issues that were repeatedly observed by our investigators in several villages, and de-emphasize outliers - events and issues that were unique to a particular village at a particular point of time. 1. The Context The average literacy rates in Gulbarga and Raichur districts, according to the 2011 census, are 65.65 and 60.46 respectively, well below the state average of 75.60. Both districts are in the Hyderabad-Karnataka region that for several centuries, prior to 1950, was ruled by the Nizam of Hyderabad under a feudal regime. Vestiges of feudalism like patronage networks, nepotism, and high inequality characterize village life even today with powerful landlords, hereditary nobles (jagirdars) other village elites continuing to exert strong influence. These systems of patronage also manifest themselves in all our villages in the form traditional panchayats 8 that are led by village elites (Ananthpur, 2007). This informal system of government works in parallel with GPs and is the primary local authority for resolving local disputes and organizing village festivals. With the introduction of the constitutionally mandated GPs, traditional panchayats have found new ways to interface with the GP. They not only try to influence who contests GP elections, but also influence the GPs decisions. The following vignettes illustrate the nature of elite rule in these villages: 8 Traditional panchayats, in contrast to Gram Panchayats (GPs) are informal village institutions that consist of powerful elites and elders in the village who resolve disputes, act as intermediaries with government bodies including the GP, and are often quite politically powerful. 12 Devappa Patel 9 owns nearly 380 acres of land in Kirkhalli and is also the president of the local commercial bank and the main temple in the village, thus controlling two of the village’s most powerful institutions. While he is not a member of the GP (he lost elections in 2001), he widely considered to be the power behind the throne. In the village of Kotehalli, the local jagirdar is the GP president. His family ruled over the principality of Kotehalli under the Nizam of Hyderbad’s rule. As the Nizam’s vassals they laid claim to 25% of the taxes due to the Nizam. When the panchayat system was introduced in Kotehalli in 1987, the then Queen of Kotehalli became the President of the GP. Her son and heir, Sitaram Desai, took over after her death and exerts strong control over all matters, including local disputes in Kotehalli. He is the richest man in the village, the GP president, and the head of the traditional panchayat. In Jhakuri village, the strongest leader is Anil Parameshwar, who is from the dominant Lingayat community. While he owns a relatively modest 20 acres of land he belongs to one of the six traditionally powerful families in the village. He controls all the construction projects sanctioned by the GP and allocates them to his favored contractors. Even though he is not a GP member, all matters in the GP require his consent. His wife, Jayanti Parameshwar is a GP member and the president of the local women’s empowerment society. These power structures and the desire of entrenched elites to retain their status made the task of mobilizing poor, illiterate citizens very challenging. These challenges were exacerbated because of the latent potential for violence, which can be traced back to the feudal context. Physical fights, murder, and threats of violence are, thus, integral to the region. Consider these cases: Satyappa, a dalit (low caste) leader in Dharmapuri was so powerful that every activity in the village had to be endorsed by him for it to succeed. During the course of our study, he was violently murdered by unknown assailants. The Jhakuri GP president has a history of violent behavior and is reputed to have had a past life as a contract killer. He is alleged to have dealt with a person, who defamed his wife’s reputation, by hacking him to death in public with a machete. He has been arrested several times and is the subject of an on- going police investigation. In an interview with one of the authors he was asked how he reconciles his 9 All names of individuals and villages have been altered to preserve confidentiality. 13 infamous reputation with his (rather effective) record as a GP president, and said, “I do bad things to bad people and good things to good people.” Parameshwar, also from Jhakuri, was shot with a gun by a long-time adversary while attending a meeting at the Taluk Panchayat (the sub-district level panchayat) office. On average, the qualitative GPs had a seven-fold increase in their budgets. This unleashed remarkable changes despite the constant threat of violence. Our investigators observed that higher budgets led to a sharp increase in panchayat activity – particularly in the construction of local public goods, and the implementation of various welfare programs such as the MNREGS. The increase in budgets also led to sharp increases in political competition; elected positions in the GP were suddenly more attractive partly because of the higher levels of public goods construction sharply increased the scope for corruption. Another important change that occurred during this period was increased awareness about the Right to Information Act, coupled with an effort by the state government to disseminate information about panchayat issues: including MNREGA rules and regulations, the posting of GP budgets and resource allocations on the walls of the GP office, efforts to ensure that gram sabhas were regularly held, etc. At the same time citizens had access to a larger number of welfare benefits from GPs making them more aware of how GP’s operated. Thus, all the GPs moved from being largely dormant to becoming hubs of activity. Citizens became more knowledgeable, more aware of their local representatives and of the activities of the GP council, and of the processes of how funds were allocated and misallocated. All this manifested itself in the trends depicted in Tables 1, 2 and 3 and described above. Implementing the People’s Planning intervention in this context was, however, very challenging. KSIRD’s plan covered a total of 50 GPs spread across 19 Talukas belonging to 5 “backward” districts of Karnataka: Bidar, Gulbarga, Raichur, Davangere and Chitradurga. The total budget allocated for this exercise was Rs. 50 lakhs (approximately $92,000) over a two-year period. KSIRD’s team consisted of 38 Resource Persons (RPs) organized into 19 teams. Each team, had two RPs, who were responsible for carrying out the intervention in one Taluka. Initially, during the planning process, each team visited the selected GPs in their Talukas to conduct Ward Sabhas and Gram Sabhas and to prepare a five-year plan document with the active participation of the GP administration and members. This team was also responsible for the follow up visits for the first year following the plan process. In the second year, KSIRD reduced the number of RPs to one per GP due to an increased demand to carry out similar exercises in other regions of Karnataka. 14 The RPs had a great deal of difficulty finding entry points into the community with reactions to their presence ranging from indifference to outright hostility. Often they were told that panchayats were out of bounds for 'ordinary people' and on one occasion a bag of money was offered to RPs as a bribe to keep them away from talking about citizen participation. RPs were challenged by the villagers’ lack of civic awareness, their alienation from local governance systems, and their perception that these issues were 'beyond them.' In essence, they faced a cultural barrier where the context of the present was haunted by what some RPs have described as a “feudal memory.” According to RPs, this served those in power to keep people in poverty and ignorance. Equally, villagers were reluctant to engage with the intervention process. The question most commonly posed by residents to RPs would be – “how does it benefit me to attend a gram sabha?” People were willing to participate only if it brought them tangible benefits. As an RP observed “People have not been able to access education, political system or other means for their own betterment. This kind of environment has been established and perpetuated by those in power. When people like us go to these communities with training or information workshops, it is not openly welcomed.” With more money being devolved to the GPs, the consequent competition for contracts and funds increased the risk of violence. Panchayat Development Officers (PDOs) were appointed by the state government to implement the new panchayat initiatives and the struggles they faced were widely reported in the press. Several PDOs committed suicide alleging physical and emotional harassment by GP members in Gulbarga and Bidar districts which galvanized PDOs from across the state to demonstrate in the capital city of Bangalore (The Hindu 2012, Times of India 2012). Thus, trying to bring in people’s participation to increase transparency and accountability in GP activities in this context is not only challenging but also dangerous. Good facilitation in such contexts necessarily functions on high levels of intrinsic motivation because effective facilitators, by challenging the status quo, constantly face threats. Take this example from Utsavahalli where RPs had been relatively effective in getting citizens to question the GPs decisions, monitor the GPs quality of work, and mobilize citizens to hold GPs accountable for their actions: In November, after a particularly severe monsoon had caused the village to suffer floods, RPs in Utsavahalli were asked by Halayya, a local low-caste leader, to ensure that rice and wheat, that had been distributed as part of flood relief efforts, reached deserving people. The RPs said that they would put pressure on local authorities to make sure the grain was properly targeted and that he would help organize a local food distribution network. When the GP President found out he scolded Halayya for trying 15 to get KSIRD involved in the distribution system and questioned the authority of KSIRD to intervene since they were not from the village. His threats and pressure turned Halayya against the program, and he began to disrupt the activities of a youth group that the RPs had organized. He then called an RP on the phone and threatened him saying, “You make promises and don’t keep them - so don’t you dare visit Utsavahalli again.” The threat in this case was ineffective because the RPs told him that while they did not belong to the village they had been appointed by the government to visit it, and continued their work in the village. Confronting elites and local leaders in such contexts requires bottom-up efforts to be supported and monitored by the center in a “sandwich” approach that protects citizens from pushback by elites (Fox 1992, Mansuri and Rao, 2013). For the most part, the RPs lacked such support. When an agreement was reached with local bureaucrats to coordinate with GPs on construction, welfare schemes and other activities, these agreements were routinely breached without any consequences for the responsible bureaucrat. None of the violent actions of GP members and their representatives were met with police action. On other hand, there was overwhelming evidence that local bureaucrats were complicit in processes that appropriated funds from the GPs’ construction contracts. Proceeds were shared between GP members, GP secretaries and officers from the public works department. 2. Variation in the Quality of Facilitation 10 Effective participatory interventions rely on effective facilitators; functionaries who work at the village level and do the actual work of the interventions. Their work is the point at which the rubber hits the road in such projects and, yet, it is under researched and poorly understood (Mansuri and Rao 2013). Thus in this section, we focus on understanding the key role of the RPs who facilitated this intervention. The life paths of the most effective RPs were similar: a commitment to equity, a sense of justice accompanied by desire to work within the system and make it accessible to people. Many of the RPs came from an activist background. Several of them were active in the Karnataka Literacy Movement and other left-oriented social movements and some saw themselves as intrinsically motivated rather than as state government employees doing a salaried job. “The main thing is to bring 'awareness' and to 'motivate Note that adding facilitator characteristics both as dummy variables and as interaction terms to the regressions did 10 not add any explanatory power perhaps because of the lack of impact observed in the RCT. 16 people' and make them 'empowered’ ” stated Ramappa (RP, Gulbarga district) in trying to summarize his role. Highly motivated RPs such as Ramappa are difficult to find. Most RPs did not go beyond the mandated tasks assigned by KSIRD as part of the follow up process and failed to exploit potential opportunities to interact with citizens and bring about change. Such RPs intervened mostly at the level of the GP council dealing directly with the GP secretary, the president and GP members, and minimized contact with villagers (unlike their counterparts from activist backgrounds). Some of them barely visited the villages and did not exhibit the minimum commitment required for follow up process. When our field investigators asked RPs about their whereabouts during the days when they were supposed to have been visiting the villages, the RPs made claims that proved false on verification. To illustrate the variation in the quality of facilitation we present some descriptions of the work of RPs in the treatment GPs we tracked: A) Utsavahalli had very high quality RPs who consistently worked in a manner that went well beyond their terms of reference. They had to deal with an illiterate and uninterested female GP president who was controlled by her very corrupt husband. Rather than engaging directly with the GP president’s husband, the RPs instead went directly to village citizens. They worked with different interest groups and organized them to engage with the GP and only intervened when needed. At the same time they built good relationships with other GP members, higher-level politicians, and local bureaucrats. The consequent improvement in citizen engagement and the construction of alliances with higher levels of government ensured that the GPs work was conduced in a systematic and rule-bound manner. The RPs’ work in Utsavahalli was so effective that they continued to be approached by villagers seeking information about government programs one year after the intervention had ended. B. In Jhakuri the RP’s, who were also highly motivated and from an activist background, decided to take a different approach and approached their work via the village’s traditional elites. They began by persuading the GP member Jayanti Parameshwar, who was the wife of the influential Anil Parameshwar and head of the local women’s empowerment organization, to cooperate with them. With her help they built a strong network of women’s self-help groups and used this network as a base to mobilize citizens in the village. They benefited from the fact that the GP president, despite the criminal allegations against him, had a strong desire to be an effective and honest president and sought the RPs guidance about how to go about doing his work. They used their connection with Jayanti to persuade her husband Anil to 17 help their cause. Once the two most powerful men in the village were co-opted into their agenda, they linked them to the women’s network to resolve a variety of GP issues related to service delivery and the effective implementation of welfare programs. C. Aalanahalli GP, on the other hand, is an example of passive, ineffective facilitation. The RPs here had a very perfunctory approach to their work, and did not attempt to establish rapport with village residents. Instead of working with citizens and elected representatives they chose instead to work solely with the GP secretary and consequently had no credibility in the village. This was a lost opportunity because Aalanahalli had a relatively active group of citizens who had a history of rebelling against inequitable and inefficient GP actions on several occasions; particularly on issues related to the allocation of homes under a welfare scheme. The RPs, however, were unable to channel this existing capacity towards more effective engagement with the GP. D. The RPs in Kotehalli deployed what can be best described as a “school teacherish” approach – vociferously judgmental in their interactions with citizens, treating them like children who needed to be badgered rather than gently persuaded 11. Rather than proactively engaging with the GP, they acted as passive observers. For instance, they did not attend gram sabhas and GP meetings but tried to fix decisions several days after the meetings were held. There are instances of RPs whiling away their time in an empty GP office rather than engage with citizens. We recorded instances of an active “youth group” in the village who wanted information and advice about how to work with the GP, and whom the RPs refused to meet. However, the KSIRD intervention in Kotehalli did seem to affect people’s expectations of people of how participatory processes should be conducted. The Government of Karnataka, unaware of the KSIRD intervention in Kotehalli, initiated a second intervention with another NGO to conduct a training program in the panchayhat to prepare a five-year village action plan in 2008. The approach of the NGO was even more perfunctory and superficial than the KSIRD RPs’. They spent two days in the panchayat, as opposed to two weeks. Unlike the KSIRD RPs they did not visit and consult all the villages and hamlets in the panchayat. They were not fluent in the local language (Kannada), and used a lot of technical English words (“action plan”, “participatory planning”) which further alienated them from the community. Consequently they were unable to organize a proper gram sabha to discuss the action plan and drew up the plan (which had government sanction and therefore superseded the KSIRD plan) in consultation with a handful of “entrusted and knowledgeable persons.” The community, used to processes followed by the 11 Swidler and Watkins (forthcoming) observe similar interactions in a community-based project in Malawi. 18 KSIRD RPs, revolted against this and refused to respond to the NGO’s attempt to organize consultations. Moreover the lack of coordination between KSIRD’s work and the NGO resulted a considerably degree of confusion and a degree of fatigue with participatory processes. Ultimately even the GP Secretary was unsure about which intervention was considered legitimate by higher levels of government. E. Another serious problem was the high turnover of facilitators that resulted in inconsistencies in the RPs’ approach. Kirkhalli, for instance, had a total of 10 RPs over the period of the intervention. This meant that new RPs had to start from scratch without building on the work of previous RPs, and could not take advantage of the rapport previously built. Kirkhalli is a large GP with a population of nearly 15,000 and had a history of political problems. GP meetings had not been held for several months because the vice-president and several members had refused to attend. They were protesting a GP decision to appropriate land from one of their supporters to build a village clinic. Every time the GP secretary tried to conduct the monthly meeting it would fail due to a lack of quorum. The first set of KSIRD RPs tried many times to organize these meetings but were unable to do so. After three months they submitted a report to a higher level official - the Taluka Executive Officer - in which they recommended that the GP hold regular meetings. The Executive Officer then told the GP Secretary that if meetings were not held officials from the capital city would intervene. He also said that he would personally attend the next meeting to ensure that it was held. This helped break the deadlock and a GP meeting was finally held after persistent efforts by the RPs for six months. The first set of RPs also put some effort into forming a network of women’s self-help groups with a view of creating a core group of women who could lobby the GP and were in the process of establishing a schedule of training and facilitation programs for the women’s groups. Unfortunately, this first set of RPs were transferred a couple of months after the GP meeting and replaced by another pair. Subsequently, after a couple of months, the second set of RPs was replaced by a third pair from Udupi district, a highly literate area in south-western Karnataka that borders Kerala. The RPs from Udupi were completely at sea in the much more difficult low-literacy context of northern Karnataka and unable to navigate the local political economy. They were observed to treat GP members in a condescending manner and regularly compared their attitude unfavorably with village politicians from Udupi. Further the Udupi RPs were not even aware of the women’s network established by the first set of RPs which led to the women’s groups’ dissipating due to lack of interest and training. Thus the third set of RPs were unable to make much of a difference and the cumulative impact of the intervention was also negligible because each set of new set of RPs was not able to build on the progress made by the 19 previous set. The Kirkhalli experience also shows how essential it is to ensure the continuity of RPs and to ensure that they are familiar with the context. 3. Challenges of measurement Training and facilitation programs of the kind we examine in this paper, and which are ubiquitous around the world, can result in outcomes that are difficult to capture in survey instruments for at least three reasons: First, because they manifest themselves in variable, context-specific processes that are not easy to measure with standardized questionnaires. Second, because the impact is often unexpected; in the nature of an unintended consequence that is not easily predicted and thus difficult to measure in a survey instrument developed before the intervention is complete. Third, because complex interventions such as these do not have predictable trajectories of change, and the full implications of the intervention may not manifest themselves within the two or three year time-frame of the evaluation (Woolcock 2009, Mansuri and Rao 2013). Thus an evaluation may conduct its final survey before the full impact has been revealed. Our qualitative investigation was able to uncover such, less explicit, impacts because of the intensity, depth and open-ended nature of the data collection process. Also, the qualitative work continued for two years after the final survey was conducted which enabled us to uncover long-term effects. However, any positive impacts, however subtle or unexpected, were only observed in those panchayats that had good facilitators. At their best, KSIRD RPs were able to direct a community’s civic capacity towards making it more effective. They shepherded protests towards more effective ends by informing citizens about budgets, allocations, and bureaucratic processes. They were also able to mediate disputes between citizen activists, GP members and local officials and facilitate negotiated agreements that had broad agreement. Note that control villages also, sometimes, demonstrated high levels of civic capacity and employed what Hossain (2010) has called “rude accountability” to reach their goals, but in some treatment villages civic action was less noisy in its journey towards an outcome. The next case that compares disputes over drinking water in a control and treatment panchayat illustrates this difference. 20 Poor access to drinking water is one of the most common problems in rural north Karnataka and is a frequent cause of disputes. Dangehalli, a control panchayat, has a history of community mobilization around issues of service delivery. In September 2007 a pipeline that brought drinking water to the village was damaged in the process of road construction resulting in a disruption of water supply that lasted several months. A group of upper caste Reddy women, who were supporters of the opposition BJP party, organized a strike and blamed the GP secretary for the delay in getting the pipe repaired. The strike attracted a great of attention and various powerful local officials visited the strikers who presented a written petition to the senior taluka level officer. This prompted the local representative to the state assembly to release 180,000 rupees (about $4,000) to solve the problem. The transfer of funds, however, was held up at the district level which resulted in a confrontation between an angry mob from the village and the GP president. The police had to be called in to quell the dispute. Our investigators were able to determine that the main reason why the funds were blocked was that Basappa, a well-connected GP member and contractor, wanted the GP to allocate another 20,000 rupees to the project, essentially as a payment to him in order to get the pipeline repair started. The (female) GP president, however, refused to sanction the amount. This led Dalapati to initiate a no-confidence motion against her, and having her dismissed. The funds were then sanctioned and the pipe fixed, but the problem was resurrected two years later with another broken pipe which instigated another strike. This time the strike was started by activists from a state-level civic organization, the Karnataka Rakshana Vedike (Karnataka Protection Movement). KRV members visited the village on the 4th of July and asked the GP to take action. When by July 22nd no action was taken, KRV organized a rasta-roko (road-block) with about fifty people demanding that the GP pay attention to their needs. They demand that a senior bureaucrat receive their petition but, again, the police intervened and the agitators disbanded after handing over the petition to the with the GP secretary. Our investigators noted that the problem had not been solved even a year later. This is not an isolated incident; our field investigators noted many similar stories of water problems, delays in GP response, strikes and agitations, the sporadic involvement of higher levels of government, and widespread corruption in the implementation of public works. The points that emerge from these incidents are that (a) villagers, disgruntled with the lack of response by the GP, often resort to strikes and agitations and demand action by appealing to higher authorities; (b) there is very little follow up by the 21 villagers to ensure the public goods construction is effectively implemented and the misappropriation of funds is, consequently, not checked; and c) discontent escalating to a strike or other civic action is essentially wasted because it results in temporary ad hoc solutions that are not sustainable. Contrast this with how a water dispute was dealt with in the treatment panchayat of Utsavahalli that shows how an effective intervention can exploit the available political opportunities to make the GP accountable and responsive. Utsavahalli GP had a dilapidated water tank, located next to a middle school, which had severely restricted the availability of drinking water and was in such bad condition that people feared that it would collapse and injure school children. Fixing the water tank was one of the major demands put forth by the citizens during the gram sabha conducted by KSIRD RPs in Utsavahalli as part of the planning process and was prioritized in the Village Action Plan. The RPs determined that the GP’s budget was not adequate to construct a new water tank so a decision was made to seek funds from the district administration. In order to gather support for the demand the RPs organized a meeting of the School Committee, and they persuaded a passing Executive Engineer from the Taluka Office to attend. Hearing the pleas of the headmaster and other members of the school committee, the engineer advised the committee to pass a resolution to demolish the water tank and to forward a copy to him so that he could take prompt action. Earlier in the month the RPs had persuaded the village medical officer to test the quality of the groundwater, and found that it was unfit for drinking. Following the school committee meeting the RPs then persuaded the head of the traditional panchayat, Hiriyanna, who had good connections with district level politicians, to request the district administration to provide funds for the construction of the tank and also to grant permission to source water for the village directly from the nearby Tungabhadra river rather then rely on groundwater. The district administration subsequently granted both requests with 1.2 million rupees ($60,000) for the construction of the new tank. The large size of the funds involved resulted in political disputes in the GP with local politicians trying to ensure that their favored builders were given the contract for the project. This resulted in further delays. KSIRD RPs regularly tracked the problem in their monthly visits and mobilized villagers to follow up with the GP in a peaceful manner; in February 2009; we observed a local civic group, unprompted by RPs raising the water tank issue with the GP secretary during a regularly scheduled gram sabha. 22 The construction of the water tank was finally completed in April 2010 and it became fully operational only in March 2011 after a pipeline to access water from the river was also built. There is wide agreement in the panchayat that the RPs was responsible for ensuring that the water tank was built by providing information, acting as liasons with the district administration, and mobilizing village citizens to apply pressure on the GP. The Utsavahalli intervention on the water tank was not unique. Similar instances were observed in treatment villages with effective RPs, such as Kirhaalli and Jhakuri. RPs typically helped in dispute resolution, channeling citizen discontent to make GPs more responsive by trying a variety of methods – linking with powerful allies both within and outside the village, activating government agencies that were otherwise dormant, and providing information about budgets and procedures. There is a question about whether the efforts of RPs had a sustainable impact in building civic capacity because, at their best, they acted as prime-movers: as local activists who mobilized, cajoled and guided citizens towards making GPs more responsive. Given the low levels of education within these rural communities it is, however, unclear if their efforts resulted in the formation of a new generation of local activists who learned from the RPs and continued their work. In most cases, when the RPs left, village activism tended to become less effective. For instance, in Utsavahalli, the demolition of the old water tank has still not been completed. Interviews with villagers make it quite clear that they attribute the lack of action on this to the absence of the RPs. To create sustainable social institutions the RPs were instructed to create a network of women’s self-help groups (SHGs). The RPs in Jhakuri went one step further in trying to ensure that members of these SHGs became active participants in the gram sabha and contested elections to the GP. The Jhakuri RPs made the GP president the honorary president of the women’s SHG to foster stronger links between the SHGs and the GP. They also provided training to SHG members on modalities of transacting with banks and on starting micro enterprises, dairy training, and chapathi making, etc. The participation of women SHG members in village activities has visibly increased in Jhakuri and they visit the bank on their own without the assistance of the anganwadi teacher, participate regularly in ward and gram sabhas, visit the local anganwadi and schools and monitor the quality of school meals and also visit the GP to demand facilities. According to a female GP member in Jhakuri, “As a result of KSIRD’s mediation efforts and meetings, women have cultivated the habit of talking and discussion.” For example, during the earlier meetings of 23 the SHGs; women members simply kept quiet. But in subsequent meetings, the women sang the invocation song and gave an opening speech. They also put questions to KSIRD representatives about loan schemes and other facilities and benefits available to them from the government. They also questioned KSIRD representatives about what they would gain by forming the Stree-Shakti Sanghs (women’s empowerment societies) and the Federation. Furthermore, they have begun to go alone to the Panchayat office and pay their property taxes: a radical change from 2007 where they did not even leave their homes. We should note that NGOs engaged in similar efforts to form self-help groups were present in control villages as well, but those efforts were not effective as those in Jhakuri and Utsavahalli. They tended to focus more on form (creating SHGs) rather than on function (ensuring that those SHGs were effective). What this shows is that, at its best, the KSIRD intervention was able to tailor the work of the RPs to reflect the needs and social conditions prevalent in the assigned village. Over time, effective RPs were able to channel the civic capacity of poorer citizens in the village towards more effective ends and thus make the accountability process less noisy and “rude.” They were also able to build social institutions that had the potential to carry on citizen initiatives after the RPs had left. Also, as we showed earlier, RPs raised expectations about participatory processes in the community that led to frustration with subsequent visits from NGOs who were less effective trainers and counterparts than KSIRD. Thus, the impact of the intervention sometimes did not manifest itself till after the RPs had ended their association with the GP. Conclusion and Summary This paper shows that even a “failed” intervention can be revealing if enough effort is put into understanding the mechanisms underlying the failure. This requires a combination of quantitative work that is able to rigorously test whether the intervention had an impact, and in-depth qualitative work that carefully tracks the processes of change instituted by the intervention over time. The qualitative analysis is able to unpack mechanisms of change and identify institutional shifts that are hard to capture in quantitative data. Surveys have the advantage of being able to measure predictable outcomes that have impacts that are large enough to be captured by the available units of observation, but major events sometimes occur that are best investigated on the ground, during the moment they happen, by participant observers. Important shifts occur during points of conflict, or during periods of mobilization and coordination. These are not quotidian events and are very hard to predictably measure. Moreover, qualitative investigators are able to build strong relationships within communities that allow them to “see” differently and thus capture insights that a survey interviewer is unable to do. Qualitative and quantitative 24 methods, thus, each have their advantages and disadvantages. This paper attempts to demonstrate, therefore, that a careful combination of the two can supplement the limitations of one with the benefits of the other (Rao and Woolcock 2003, White 2011). The intervention we study here is a citizen training and facilitation program that attempted to transfer the lauded “People’s Campaign” from the Indian state of Kerala to the feudal, highly unequal, and far less literate region of Northern Karnataka. Its goal, as in Kerala, was to make the system of democratic village government more responsive, transparent and efficient by improving the quality and quantity of citizen participation. The intervention was randomly assigned to 50 Gram Panchayats in Northern Karnataka, with another 50 chosen as controls. The RCT which analysed data from two rounds of data collected in 2007 and 2009 found substantial changes over time in both control and treatment and GPs but almost no discernable changes between the treatment and control groups. The qualitative investigation was based on a 10 percent subsample of the quantitative sample. Resident field investigators filed monthly reports about these 10 Gram Panchayats over a four period from 2007 to 2011. These monthly reports, and supplemental interviews conducted with higher government officials and Resource Persons (RPs) responsible for implementing the program, constitute our qualitative data. Both the qualitative and quantitative data reveal considerable improvements over time in the quality of public services, governance and participation. A very sharp increase in village government funds led to increased political competition, better public infrastructure and increased citizen participation particularly in village meetings known as gram sabhas, and in the use of requests filed under the right to information act. However, the attempt to replicate the Kerala campaign in Karnataka “failed” in terms of not showing significant differences across treatment and control villages. The ethnography we conduct of these villages finds several reasons for the lack of observed outcomes: 1) Challenging context: High levels of inequality, conflict, elite domination and low literacy made working in this environment particularly difficult. Moreover, a number of government programs raised GP budgets by 700 percent between 2005 and 2007. This provoked a series of changes – increased political participation and competition over funds and high levels of corruption, increased awareness of GP functions and processes, and increases in citizen participation. Within this tsunami of change, the intervention was unable to add significant value. 2) The lack of a “sandwich”: The RPs essentially operated on their own rather than as agents of government with the backing of local bureaucrats. The bureaucrats were not integrated into the change process. Consequently higher-level bureaucrats did not 25 respond adequately to any gains made by citizens. This allowed local elites to reassert their interests and set back the gains made by citizens. 3) Variation in the quality of facilitation: Only two of the five treatment GPs had high quality, intrinsically motivated facilitators. RPs in the other GPs were less than diligent in their work with a lot of evidence that they missed obvious opportunities to make a difference. In one GP in our qualitative sample RPs were rotated to such an extent that each set of new RPs had to start from scratch and was unable to build on the RPs that preceded them. Even good facilitators acted more as community leaders rather than as development workers whose main task was to build local capacity. This was probably because of the lack of a local educated class who could be called on to become community mobilizers. 4) Poor application of Kerala design: While the design of the program showed many positive outcomes time in the qualitative data, the program was poorly implemented due to the lack of public support. Thus, one could argue that the “treatment” that was applied was well short of its potential – a fact that would not had been observed without the ethnographic analysis. However, the nature of this complex intervention also made it less amenable to an RCT. Effective facilitators were able to introduce positive changes because they were able to modify the intervention to suit the context. In other words this was not one, but several interventions, resulting in heterogenous and often unexpected outcomes. As other scholars have pointed out (Woolcock 2009, Mansuri and Rao 2013) these variations in the intervention were compounded by unpredictability in knowing about when they would produce an outcome; outcomes were sometimes observed two years after the RPs had left the villages and the final survey round had been conducted. Outcomes observed in the qualitative work were often subtle, gentle rather than dramatic in their cadence of change. These processes have to be tracked with frequent visits to fully understand, and are very difficult to capture in a survey instrument. All this suggests that the approach to evaluating complex, contextualized, heterogenous, interventions needs to be rethought. Given the high variability and unexpected nature of outputs, surveys may require larger samples and more rounds in order to account for this heterogeneity and track unexpected outcomes. It also raises questions about how pre-analysis plans should be used in assessing interventions of this kind. Complex interventions also, in our view, require that we take qualitative analysis just as seriously as quantitative data and methods to address the problem of understanding mechanisms of 26 change. However, in the case of the project evaluated in this paper, even a larger quantitative sample would probably have not helped. This raises the broader question of whether such interventions are worth the effort. Our answer would be – maybe – because the cost was a modest $200,000, if the intervention was allowed more time to have a sustainable impact. Looking forward, the literacy levels of these districts are rapidly improving and development interventions need to be cognizant of this 12 . Complex interventions that build citizen capacity and improve the capacity of communities to mobilize and engage with government can help greatly in the context of improving literacy. However, they require a fundamentally different approach to development that requires a reliance on a sandwich between bottom-up and top-down efforts, and constant experimentation and learning by doing to be effective. References Ananthpur, Kripa, “Rivalry or Synergy ? Formal and Informal Local Governance in Rural India,” Development and Change, 38(3), Pp: 401–421, 2007. Ban, Radu, Saumitra Jha and Vijayendra Rao, “Who has voice in a deliberative democracy: Evidence from transcripts of village meetings in India,” Journal of Development Economics 99(2), Pp: 428-438, 2012 Banerjee, A.V., R. Banerji, E. Duflo, R. Glennerster, and S. Khemani, “Pitfalls of Participatory Programs: Evidence from a Randomized Experiments in Education in India,” American Economic Journal: Public Policy 2(1), Pp: 1-30, 2010 Bardhan, P and D. Mookherjee, “Political Clientalism and Capture: Theory and Evidence from West Bengal,” Working Paper, Dept of Economics, Univ. of California, Berkeley, 2011 Baiocchi, Gianpaolo, Patrick Heller, and Marcelo K. Silva, Bootstrapping Democracy: Transforming Local Governance and Civil Society in Brazil, Stanford University Press, Palo Alto, 2011 Barron, Patrick, Michael Woolcock and Rachel Diprose, Contesting Development: Participatory Projects and Local Conflict Dynamics in Indonesia, Yale University Press, New Haven, 2011 Besley, Tim, Rohini Pande, and Vijayendra Rao, “Participatory Democracy in Action: Survey Evidence from Rural India,” Journal of European Economic Association 3(203), Pp: 648-57, 2005 12Literacy rates in Gulbarga jumped from 54 to 64 percent between 2001 and 2011, and from 49 to 60 percent in Raichur. These gains came entirely from sharp improvements in literacy among younger cohorts. The expectation is that literacy in these districts will be close to 80 percent by 2021. 27 Fox, Jonathan, The Politics of Food in Mexico: State Power and Social Mobilization, Cornell University Press, 1992 Fung, Archon and Eric Olin Wright, “Thinking about empowered participatory governance,” In A. Fung and E.O. Wright (editors), Deepening Democracy: Institutional Innovations in Empowered Participatory Governance, Pp: 3-42, Verso, New York, 2003 Gaventa, John and G. Barrett, “So what difference does it make? Mapping the outcomes of citizen engagement,” IDS Working Paper 347, Institute of Development Studies, Brighton, 2010 Heller, Patrick, K.N. Harilal and Shubham Chaudhuri, “Building local democracy: Evaluating the impact of decentralization in Kerala, India,” World Development 35(4), Pp: 626-48, 2007 The Hindu, “ ‘Harassment’ Drives PDOs to Commit Suicide,” The Hindu, May 16, 2012 Hossain, Naomi, “Rude Accountability: Informal Pressures on Frontline Bureaucrats in Bangladesh,” Development and Change 41 (5) Pp: 907–928, September 2010 Humpreys, M., W.A Masters, and M.E. Sandbu, “The Role of Leaders in Democratic Deliberations: Results from a Field Experiment in Sao Tome and Principe,” World Politics 58(4): 583-622., 2006 Mansuri, Ghazala and Vijayendra Rao, Localizing Development: Does Participation Work, World Bank Policy Report, The World Bank, Washington DC, 2013 Olken, Benjamin, “Monitoring Corruption: Evidence from a Field Experiment in Indonesia,” Journal of Political Economy 115(2): Pp200-49, 2007. Paluck, Elizabeth L. “Is it better not to Talk? Group Polarization, Extended Contact and Perspective Taking in Eastern Democratic Republic of Congo,” Personality and Social Psychology Bulletin 36(9): 1170-85 Rajaraman, Indira, A Fiscal Domain for Panchayats, Oxford University Press, New Delhi, 2003 Rao, Vijayendra and Paromita Sanyal, “Dignity through discourse: Poverty and the culture of deliberation in Indian village democracies,” Annals of the American Academy of Political and Social Science 629 (May), Pp: 146-72, 2010 Rao, Vijayendra and Michael Woolcock, “Integrating Qualitative and Quantitative Approaches in Program Evaluation”, in Francois J. Bourguignon and Luiz Pereira da Silva (eds.) The Impact of Economic Policies on Poverty and Income Distribution: Evaluation Techniques and Tools New York: Oxford University Press, pp. 165-90, 2003 Sen, Amartya, Development As Freedom, Knopf, New York, 1999 Swidler Ann and Susan Watkins, “Practices of Deliberation in Rural Malawi,” in P. Heller and V. Rao (editors), Deliberation and Development: New Directions, The World Bank, Washington DC, (forthcoming) Times of India, “Spate of Suicides by Panchayat Officers Alarms Govt, CID Probe Initiated, ”April 5th, 2012. 28 White, Howard, “Achieving high-quality impact evaluation design through mixed methods: The case of infrastructure,” Journal of Development Effectiveness 3(1), Pp: 131-44, 2011 Woolcock, Michael, “Towards a plurality of methods in project evaluation: A contextualized approach to understanding impact trajectories and efficacy,” Journal of Development Effectiveness 1(1), Pp: 1-14, 2009 29 Tables Table 1: Changes across survey rounds – Household Outcomes 2007 2009 Difference Stdev Pval Respondent Information Correctly names Ward Member/President 61% 78% 17% 0.020 0.000 Heard of MNREGS 24% 61% 37% 0.017 0.000 Heard of Gram/Ward Sabha 20% 56% 36% 0.020 0.000 Participation Attended Gram/Ward Sabha 11% 26% 16% 0.011 0.000 Attended any meeting in last year 5% 16% 11% 0.012 0.000 Feels President is responsive to needs 27% 21% -6% 0.017 0.000 Government Transfers House provided under government scheme 15% 25% 9% 0.007 0.000 Government transfer for housing 19% 26% 8% 0.008 0.000 Eligible and received BPL card 53% 85% 32% 0.019 0.000 Contribution and Payments Paid tax last year 60% 76% 16% 0.023 0.000 Made contributions last year 70% 65% -5% 0.039 0.165 Proportion of infrastructure contributed to 0.09 0.10 0.01 0.006 0.186 *Note: The standard errors for the difference in means are clustered at the level of randomization—the Gram Panchayat. Table 2: Changes across survey rounds – Village Activity and Works 2007 2009 Difference Stdev Pval Activity in last 12 months – Roads 42.9% 63.5% 20.6% 0.052 0.000 Activity in last 12 months - Transportation 5.1% 9.0% 3.9% 0.027 0.154 Activity in last 12 months - Schools 6.1% 15.5% 9.4% 0.034 0.006 Activity in last 12 months – Health 7.1% 14.6% 7.5% 0.036 0.042 Activity in last 12 months – Drinking Water 44.9% 36.2% -8.8% 0.052 0.092 Activity in last 12 months - Sanitation 21.2% 37.1% 15.8% 0.051 0.002 Activity in last 12 months - Electrcity 34.3% 35.0% 0.7% 0.053 0.902 Activity in last 12 months - Irrigation 5.6% 12.5% 6.9% 0.033 0.036 Agr. extention officers visit 54.5% 79.5% 25.0% 0.052 0.000 Taluk Engineer visits 33.8% 54.5% 20.7% 0.060 0.001 EO visit 29.3% 46.0% 16.7% 0.056 0.004 NGO active in village 9.6% 23.0% 13.4% 0.036 0.000 Traditional Panchayat active 60.6% 46.0% -14.6% 0.053 0.006 Male agricultural wage 55 92 37 1.733 0.000 Female agricultural wage 31 50 18 1.312 0.000 Oligarchy 0.10 0.08 -0.02 0.010 0.140 *Note: The standard errors for the difference in means are clustered at the level of the Gram Panchayat. 30 Table 3: Changes across survey rounds – Village Infrastructure 2007 2009 Difference Stdev Pval Private primary school exists 19% 23% 4% 0.016 0.006 Private middle school exists 24% 25% 1% 0.013 0.546 Private secondary school exists 10% 15% 4% 0.018 0.019 Any overhead tank exists 72% 80% 8% 0.030 0.009 No. of government school 1.8 1.8 0.01 0.044 0.910 No. of teachers in govt. schools 10.4 10.7 0.30 0.283 0.292 No. of anganwadis 1.8 2.0 0.11 0.042 0.011 No. of employees in anganwadi 3.7 3.9 0.18 0.119 0.134 No. of handpumps 4.1 4.3 0.24 0.311 0.434 No. of tubewells 2.9 1.3 -1.61 0.565 0.005 No. of water sources constructed- last 12 months 0.9 0.4 -0.58 0.351 0.099 No. of roads 24.7 27.7 3.07 0.788 0.000 No. of gravel roads 2.1 1.9 -0.24 0.365 0.522 No. of roads constructed- last 12 months 2.0 2.0 0.01 0.388 0.980 Number of roads free of garbage 10.3 11.5 1.16 1.086 0.289 Number of roads with a drain 8.0 6.1 -1.88 0.502 0.000 *Note: The standard errors for the difference in means are clustered at the level the Gram Panchayat. Table 4: Comparison of Means - Household Characteristics (2007) Baseline Comparison Treatment Diff Std. Dev. P-value Demographics Characteristics Respondent Age 37 38 0.6 0.388 0.130 Head of HH is Illiterate 51% 48% -3.0% 0.040 0.446 Female head of HH 16% 17% 1.2% 0.018 0.529 Female Respondent 50% 49% -0.6% 0.006 0.394 SC/ST HH 42% 40% -2.1% 0.042 0.626 Religious Minority 1% 1% 0.1% 0.005 0.825 HH size 6.0 5.9 -0.1 0.129 0.537 Economic Characteristics Reside in Kachcha house 43% 51% 7.8% 0.051 0.130 TV Owners 25% 30% 5.0% 0.029 0.094 Radio Owners 13% 13% 0.1% 0.014 0.971 Land Owners 75% 74% -0.6% 0.026 0.831 Irrigated Land Owners 25% 24% -1.1% 0.039 0.776 Agricultural Laborer 52% 48% -4.1% 0.030 0.173 Agriculturist 38% 40% 2.3% 0.030 0.451 31 Table 5: Comparison of Means – Village Infrastructure (2007) Baseline (2007) Compariso Treate Differenc Stde Pval Private primary school exists 18% 19% 0% 0.06 0.95 Private middle school exists 18% 29% 11% 0.06 0.10 Private secondary school exists 11% 9% -2% 0.04 0.67 Any overhead tank exists 69% 75% 6% 0.06 0.36 No. of government school 1.7 1.9 0.24 0.14 0.10 No. of teachers in govt. schools 9.0 11.8 2.75 1.07 0.01 No. of anganwadis 1.8 1.9 0.13 0.13 0.31 No. of employees in anganwadi 3.5 3.9 0.39 0.29 0.19 No. of handpumps 4.7 3.5 -1.21 0.68 0.08 No. of tubewells 2.8 3.0 0.25 1.19 0.83 No. of water sources constructed- last 12 0.5 1.4 0.83 0.68 0.22 No. of roads 24.7 24.6 -0.09 2.66 0.97 No. of gravel roads 2.2 2.0 -0.17 0.57 0.76 No. of roads constructed- last 12 months 1.4 2.5 1.15 0.47 0.01 Number of roads free of garbage 10.0 10.6 0.63 2.19 0.77 Number of roads with a drain 7.4 8.6 1.30 1.41 0.36 *Note: The standard errors for the difference in means are clustered at the level of the Gram Panchayat. Table 6: Comparison of Means – Village Activity and Works (2007) Baseline (2007) Comparison Treated Difference Stdev Pval Activity in last 12 mons - Roads 43.0% 42.9% -0.1% 0.080 0.986 Activity in last 12 mons - 5.0% 5.1% 0.1% 0.036 0.978 Activity in last 12 mons - Schools 4.0% 8.2% 4.2% 0.039 0.286 Activity in last 12 mons - Health 9.0% 5.1% -3.9% 0.043 0.369 Activity in last 12 mons - Water 47.0% 42.9% -4.1% 0.076 0.586 Activity in last 12 mons - Sanitation 22.0% 20.4% -1.6% 0.061 0.794 Activity in last 12 mons - Electricity 41.0% 27.6% -13.4% 0.074 0.072 Activity in last 12 mons - Irrigation 2.0% 9.2% 7.2% 0.037 0.057 Agr. extension officers visit 51.0% 58.2% 7.2% 0.079 0.366 Taluk Engineer visits 39.0% 28.6% -10.4% 0.080 0.198 EO visit 33.0% 25.5% -7.5% 0.074 0.315 NGO active in village 10.0% 9.2% -0.8% 0.045 0.857 Traditional Panchayat active 63.0% 58.2% -4.8% 0.072 0.504 Male agricultural wage 57 53 -4 2.109 0.056 Female agricultural wage 32 30 -2 1.654 0.330 Oligarchy 0.10 0.10 0.00 0.017 0.876 *Note: The standard errors for the difference in means are clustered at the level of the Gram Panchayat. 32 Table 7: Comparison of Means - Household Outcomes (2007) Baseline Comparison Treatment Diff Stdev Pval Respondent Information Correctly Names Ward Member/President 58% 64% 5.2% 0.034 0.126 Heard of MNREGS 24% 24% -0.2% 0.026 0.938 Heard of Gram/Ward Sabha 17% 23% 6.5% 0.022 0.003 Heard of Jamma Bandhi 1% 3% 1.3% 0.005 0.018 Participation Attended Gram/Ward Sabha 9% 12% 3.2% 0.014 0.023 Attended any meeting in last year 4% 6% 2.5% 0.008 0.004 Feels President is responsive to needs 26% 27% 0.7% 0.029 0.816 Government Transfers House provided under government 15% 16% 1.3% 0.019 0.492 Government transfer for housing 19% 19% 0.3% 0.020 0.882 Eligible and received BPL card 57% 48% -8.8% 0.033 0.009 Contribution and Payments Paid tax last year 62% 58% -4.1% 0.043 0.350 Made contributions last year 74% 67% -7.0% 0.042 0.095 Proportion of infrastructure contributed 0.09 0.09 0.2% 0.007 0.761 *Note: The standard errors for the difference in means are clustered at the level of the Gram Panchayat. Table 8: Difference-in-differences – Household Outcomes w/o FE w/ individual FE treateff pval treateff pval Respondent Information Correctly names Ward Member/President -0.0561 [0.159] -0.0545 [0.178] Heard of MNREGS 0.0411 [0.236] 0.0346 [0.327] Heard of Gram/Ward Sabha 0.0300 [0.444] 0.0269 [0.497] Heard of Jamma Bandhi -0.00700 [0.154] -0.00707 [0.150] Participation Attended Gram/Ward Sabha 0.0326 [0.138] 0.0292 [0.167] Attended any meeting in last year 0.0185 [0.436] 0.0180 [0.456] Feels President is responsive to needs 0.0484 [0.150] 0.0706* [0.0792] Government Transfers House provided under government scheme -0.0110 [0.446] -0.0194 [0.195] Government transfer for housing -0.00300 [0.858] -0.0128 [0.431] Eligible and received BPL card 0.0471 [0.211] 0.0537 [0.135] Contribution and Payments Paid tax last year 0.0616 [0.167] 0.0639 [0.162] Made contributions last year 0.124 [0.104] 0.124 [0.105] Proportion of infrastructure contributed to 0.00120 [0.919] 0.000696 [0.954] *Note: The standard errors for the difference in means are clustered at the level of the Gram Panchayat. 33 Table 9: Difference-in-differences – Village Infrastructure w/o FE w/ village FE Dependent Variable treateff pval treateff pval Private primary school exists 0.0162 [0.610] 0.0217 [0.627] Private middle school exists -0.0180 [0.489] -0.0204 [0.572] Private secondary school exists 0.00849 [0.810] 0.00085 [0.986] Any overhead tank exists 0.0600 [0.317] 0.0600 [0.478] No. of government school - [0.00837 -0.230* [0.0600 No. of teachers in govt. schools -1.340** [0.0172] -1.340* [0.0897 No. of anganwadis 0.0600 [0.481] 0.0600 [0.618] No. of employees in anganwadi -0.0800 [0.738] -0.0800 [0.813] No. of handpumps 1.475** [0.0173] 1.510* [0.0828 No. of tubewells 0.0297 [0.979] 0.0441 [0.978] No. of water sources constructed- last 12 -1.054 [0.134] -1.051 [0.290] No. of roads 0.930 [0.557] 0.930 [0.678] No. of gravel roads 0.310 [0.672] 0.310 [0.765] No. of roads constructed- last 12 months -1.867** [0.0158] -1.869* [0.0860 Number of roads free of garbage 2.521 [0.247] 2.622 [0.399] Number of roads with a drain 0.0500 [0.960] 0.104 [0.942] *Note: The standard errors for the difference in means are clustered at the level of the Gram Panchayat. Table 10: Difference-in-differences - Village Activity and Works w/o FE w/ village FE Dependent Variable treateff pval treateff pval Activity in last 12 mons - Roads 0.0114 [0.913] 0.0245 [0.868] Activity in last 12 mons - Transportation -0.0410 [0.458] -0.0396 [0.615] Activity in last 12 mons - Schools -0.0716 [0.287] -0.0688 [0.472] Activity in last 12 mons - Health 0.0476 [0.518] 0.0515 [0.624] Activity in last 12 mons - Water 0.0378 [0.716] 0.0397 [0.789] Activity in last 12 mons - Sanitation 0.0629 [0.536] 0.0632 [0.662] Activity in last 12 mons - Electricity 0.114 [0.281] 0.101 [0.503] Activity in last 12 mons - Irrigation -0.122* [0.0611] -0.120 [0.194] Agr. extension officers visit -0.102 [0.328] -0.106 [0.472] Taluk Engineer visits 0.0343 [0.775] 0.0447 [0.794] EO visit -0.0251 [0.824] -0.0167 [0.917] NGO active in village -0.0318 [0.662] -0.0276 [0.790] Traditional Panchayat active 0.108 [0.301] 0.108 [0.467] Male agricultural wage 9.123*** [0.00745] 10.04** [0.0449] Female agricultural wage 2.620 [0.318] 2.647 [0.499] Oligarchy 0.0108 [0.602] 0.0109 [0.716] *Note: The standard errors for the difference in means are clustered at the level of the Gram Panchayat. 34