Brief 42: Understanding the Failure of a Randomized Trial to Deepen Democracy in Rural India
EGAP researchers: Vijayendra Rao
Other authors: Kripa Ananthpur, Kabir Malik
Geographical region: Asia
Research question: Under what circumstances might an external intervention successfully induce citizen participation in local governance? Can an ethnography help us understand mechanisms underlying results from an RCT?
Preparer: Tanu Kumar
At the village level, the Indian governance structure consists of two bodies: an elected village council known as a gram panchayat (GP) and an inclusive deliberative body known as a gram sabha. While studies have found that this system can benefit citizens by reflecting the preferences of the median citizen or targeting benefits towards the poor, the panchayat system can also suffer from problems such as irregular meetings, elite dominance, corruption, and dependence on outside sources for revenues. Given these problems, in the 1990s, the state government of Kerala launched a program to devolve greater resources to the GPs while simultaneously implementing a grassroots training and awareness-raising effort to inform citizens of their rights and duties and mobilize them to participate in the panchayat system. While this program is generally considered to be successful, Kerala remains an outlier among Indian states with almost 100% literacy and very politically aware citizens. The authors thus sought to understand whether the Kerala experiment could be replicated in less equal, less literate, and therefore more representative areas of India, namely the neighboring state of Karnataka.
This paper uses both a randomized controlled trial (RCT) and ethnography to evaluate the impact of a treatment consisting of three phased interventions at the GP level: 1) a one-week citizen engagement program culminating in the codification of village priorities in a village action plan; 2) meetings with local bureaucrats to secure funding and technical support for projects outlined in this plan; and 3) two years of monitoring and assistance with the implementation of these projects by state representatives known as Resource Persons (RPs).
The authors chose a stratified random sample of 100 GPs (with 50 randomly assigned to treatment) from Karnataka’s five poorest districts to participate in this study. As a GP typically has three to five villages, two villages were selected per GP for surveying: the GP headquarters, and the other at random.
Within each village, enumerators surveyed two village leaders, the GP president, the GP vice president, and twenty households chosen at random. The survey instruments include village-level modules measuring the quality and quantity of public goods, and in-depth interviews with village politicians and local officials. The baseline survey was administered in October-November 2007. Two years later, in October-December 2009, the same sample of villages and households were re-interviewed with identical survey instruments.
For the qualitative studies, five sub-districts were randomly sampled from two districts included in the study; a pair of control and treatment GPs matched on attributes available in 2001 census data were then selected for the ethnographies. Each of these GPs was assigned a field investigator who collected four years of ethnographic data on the villages’ political and social structure.
The surveys revealed a sharp increase between 2007 and 2009 in levels of participation and knowledge about gram panchayat procedures and personnel across both treatment and control villages. Yet the data do not indicate that the intervention led to a significant difference between treatment and control villages across a wide spectrum of possible outcomes. At most, they show very weak evidence of both positive and negative impacts of the intervention on a very small number of outcomes.
The ethnographic studies suggest that this lack of a discernible impact is due to village-level elite influence that impeded mobilization of poorer citizens, variation in the quality and efforts of RPs, and the failure of the survey instrument to measure outcomes that may not have been realized at the time of the endline survey.
The authors find that context is fundamental to determining the effectiveness of an intervention; an intervention that worked well in Kerala failed to have similar effects in Karnataka; this could be because of literacy, inequality, or a number of other contextual differences. Moreover, low levels of education among the study population led to a greater burden placed upon RPs in program implementation, thereby amplifying the effects of variation in facilitator quality. Additionally, they find that higher-level bureaucrats must be integrated into processes of change to prevent elites from clawing back or impeding social and political change. Methodologically, the study demonstrates the value of ethnography alongside quantitative analysis. An effort to uncover the process of change using careful and detailed qualitative work can add value to standard impact evaluations of complex interventions in challenging contexts.