Metaketa
Initiative
En Español
French
Portuguese
Hindi
Overview
The Metaketa Initiative is a collaborative research model aimed at improving the accumulation of knowledge from field experiments on topics where academic researchers and policy practitioners share substantive interests. EGAP funds and coordinates studies across countries, clustered by theme, to improve and incentivize innovative research alongside integrated analysis and publication.
Context
EGAP sees an on-going crisis of external validity of social science research and few systemic efforts to address it. The business-as-usual model for many academic researchers is to select topics they find interesting and seek funding to implement the research in an original way. Other research is done in the context of policy evaluation. In these cases, the research is sometimes commissioned by donor agencies and implemented by researchers. In practice, this often means the results may respond strongly to particular local needs but be difficult to integrate with learning developed in other contexts. A consequence of this is that major policy innovations often rest on fragile research foundations.
For example, major lessons have been drawn on the effectiveness of community-based monitoring of health workers for health gains, or on the advantages of disseminating information on corruption about politicians—all from single studies. Yet there are well known reasons for caution, not least because of the presence of publication bias afflicting academic research. Positive results get published and read, negative or null results often disappear, and little is learned about the role that context plays in determining when a finding is relevant or not.
Solution
EGAP is trying to tackle this challenge of external validity; in part by getting the incentives right to attract the best researchers and practitioners to contribute to a collective endeavor that identifies critical areas where generalizable knowledge can have a large impact. The key idea of this initiative is to take a major question of policy importance for governance outcomes, identify an intervention that is tried, but not tested, and implement a cluster of coordinated research studies that can provide a reliable answer to the question.
Principles
The Metaketa Initiative is grounded in eight principles.
1. Coordination across research teams
The Metaketa Initiative seeks to generate coordination as well as some healthy competition between funded research teams. The next seven pillars of the approach depend vitally on integration and collaboration. The project achieves harmonization through meetings hosted by the steering committees with successful applicants.
2. Predefined themes and comparable interventions
Teams of researchers will work on related questions and study comparable interventions. Themes are pre-defined in two ways: (1) through common theoretical questions, and (2) through the selection of winning proposals, in which we will prioritize comparability across research teams.
3. Comparable measures
Research teams will use consistent outcome measures agreed during post-funding workshops. Researchers will be encouraged to employ mixed methods in measuring outcomes and mediators of the effect of the interventions, including qualitative data collection during the implementation phase.
4. Integrated case selection
Research teams will use consistent outcome measures agreed during post-funding workshops. Researchers will be encouraged to employ mixed methods in measuring outcomes and mediators of the effect of the interventions, including qualitative data collection during the implementation phase.
5. Preregistration
Funded research will be required to hew to EGAP’s standards for analytic transparency. In particular, after revision of research designs at an initial meeting of funded researchers and before initiation of outcome data collection, grantees must post a study protocol that describes a) the study’s purpose; b) the hypotheses it aims to test; c) the main outcome variables; and d) the set of tests and the data analysis that will be performed. In addition, the group of funded studies will itself be pre-registered, with the comparisons and pooled analyses to be conducted from the group of studies made explicit.
6. Third-party analysis
Research teams will be expected to make their data publicly available for independent, third party replication and analysis funded by the metaketa project, with a view to early identification of errors and discrepancies prior to publication. In addition, at publication, all of the data will be archived in a public repository (Dataverse) and provided free of charge.
7. Formal synthesis
Group preregistration will allow the project team and funded researchers to pre-specify a plan for meta-analysis of distinct experiments—and for formal synthesis of experimental results using standard Bayesian, multilevel meta-analysis techniques. In addition, research teams may collaborate on developing a causal model that can integrate results and account for ways in which contexts may condition causal effects. Here, integrated case selection that builds on theory about channels through which interventions affect outcomes assists in stipulating ex-ante expectations of results across experiments.
8. Integrated publication
In addition to individual academic papers and policy briefs, all funded researchers and the steering committee will coauthor one or more books or articles that present results from the distinct studies in an integrated analysis.
