Brief 59: Metaketa I: Information and Accountability

EGAP Researcher: Thad Dunning, Guy Grossman, Macartan Humphreys, Susan Hyde, Craig McIntosh, Gareth Nellis, Claire Adida, Eric Arias, Jessica Gottlieb, F. Daniel Hidalgo, Eric Kramon, Horacio Larreguy, John Marshall, Gwyneth McClendon, Daniel Nielson, Melina Platas, Pablo Querubin, Pia Raffler

Other authors: Clara Bicalho, Taylor C. Boas, Mark T. Buntaine, Simon Chauchard, Anirvan Chowdhury, Marcus Holmlund, Ryan Jablonski, Malte Lierl, Marcus A. Melo, Paula M. Pickering, Neelanjan Sircar

Geographical region: North America

Research Question: Does providing information about incumbent politicians’ performance from a non-partisan source influence voter turnout and electoral support?

Preparer: Anirvan Chowdhury

July 9, 2019

Background

Research can often seem like a weak guide to sound policy. Different studies give contradictory answers, making it hard to infer general lessons. There are at least three challenges to accumulating evidence:

1. Study scarcity: Conclusions are sometimes drawn from single, high impact studies.
2. Study heterogeneity: Differences in interventions and outcomes across studies make it infeasible or unwise to aggregate results.
3. Selective reporting: Failure to publish null findings makes it hard to learn about policies that do not work.

These barriers inhibit learning about whether a policy can be generally effective, and whether its impact varies across different contexts. To overcome these problems, the EGAP network’s Metaketa Initiative developed a coordinated research model emphasizing:

• Replication of policy-relevant studies, but also context-specific learning
• Harmonization, to the extent feasible, of interventions and outcome measures across studies, and 
• Design and reporting standards, including integrated publication of results, regardless of findings; preregistration of analyses; and third-party data replication. 

In Metaketa I, we applied this model to electoral accountability. Voters often have limited information about the performance of their political representatives. These information gaps may undermine democratic accountability. Thus, providing reliable performance information prior to elections may allow voters to select politicians who are more likely to serve them well. Can interventions that provide voters with credible, impartially sourced information about incumbents boost support for politicians when it exceeds voter expectations (“good news”) or depress support when information falls short of voter expectations (“bad news”)?

Research Design

We conducted six Randomized Controlled Trials (RCTs) across five countries to answer this question. The types of information on incumbent behavior provided to voters include legislative performance (Benin), municipal spending irregularities (Brazil), quality of public services (Burkina Faso), municipal government malfeasance (Mexico), candidate quality via debates (Uganda 1), and budget irregularities (Uganda 2). A planned seventh study on incumbent criminality in India did not take place due to implementation challenges. All information was provided privately to individual voters within a month prior to an election, and was disseminated by flyer, text message, or video, depending on the study. 

Studies measured voters’ prior beliefs about incumbent performance. This helped us determine the extent to which voters update their beliefs in light of the given information. Further, this made it possible to examine the effect of new information (by measuring the difference between prior beliefs and provided information), rather than any information. Where possible, prior beliefs were gathered on the same scale as the information provided to individuals assigned to the treatment groups. This allows us to identify voters who would have received positive or negative information about the incumbent, if assigned to the treatment group. Our empirical strategy therefore takes account of both the content of the information and prior beliefs.To estimate the effect of new information on electoral behavior, we divide subjects into groups based on whether they would receive good or bad news if exposed to the treatment. Thus, it is as if we estimate effects in one large experiment, with treatment assignment blocked by country. The pooled sample consists of 24,007 individual observations in 1,330 randomization blocks in six studies.

Results

As seen in Figure 1, none of the individual studies shows statistically significant effects of information on voters’ support for incumbents. Further, the overall estimated effect, pooling across studies is precise and null. The same holds for turnout in Figure 2 with Uganda 1 being the sole exception in reporting a significant finding for voters who received good news. Analysis for heterogenous effects (not shown here) indicated little evidence that treatment effects vary conditional on covariates.

How robust are these results? Using a type of specification curve analysis involving estimation of 18,886 model specifications reflecting all combinations of proposed secondary analyses and ex post deviations from our pre-analysis plans, we find that the null results are highly robust for the overall effect of the common information arm across the six completed studies.¹ In addition, the estimated effect in our seventh, uncompleted India study would have had to have been extremely large to have altered our overall conclusions—in the good news case for vote choice, at least 17.2 percentage points, much bigger than anything we see in other studies. Thus, our findings appear highly robust to different analytic choices and sensitivity analyses: there is very little evidence of impact of the common information interventions.

Policy Implications

Many of the interventions implemented in Metaketa I are typical of those used by policymakers to ameliorate ongoing real-world problems in democratic accountability. Metaketa I’s core finding—providing voters with a piece of information in the run up to an election does not necessarily change their electoral behavior on average—provides an important cautionary note on the limits of using such blanket informational interventions for this purpose. Thus, typical transparency promotion efforts, such as those backed by civil society organizations and donors, may not pack a strong enough punch to influence voter behavior. However, this does not mean that voter information campaigns are always ineffective. Evidence from complementary interventions and analysis of heterogeneous effects within individual studies suggest modifications that could make informational campaigns more effective.²

Footnotes
1. See Dunning et al. (2019) and http://egap.org/content/metaketa-i-shiny-app for details.
2. See Dunning et al. (2019) for details.