Video: Research Replication

Annette Brown, then the Washington Director of 3ie, presented on the 3ie Impact Evaluation Replication program at EGAP 14 in Berlin.

While scholars typically support the practice of replications, it continues to be contentious. Thus, there are questions about the potential roles for third parties in helping address these tensions. The challenges of replication research tend to focus on the definition of research objectives, the authors’ incentives, and replication ethics. Some argue that replication is a part of the scientific process and that if the data reflect an underlying truth, another researcher applying appropriate methods should produce the same or similar findings. Others argue that replication is aimed at determining whether the research was high quality and robust, thus focusing on the validation of a particular paper’s findings.

Replication can be thought of as a scientific process, a research process, or a policy-informing process. The scientific process has the goal of showing that when providing a given finding, we can repeat the process to determine whether it reflects an underlying truth about the state of the world. The research process is a diagnostic tool that involves validating a study’s empirical results by using the original dataset and methods, asking first whether the author’s results are reproducible given those data and methods (pure replication), and then asking whether taking the researcher’s data and the methods described in the paper, we can apply our own methods to those data and obtain the same results. If unable to obtain those same results, we ask whether we can diagnose what went wrong and better explain what the authors did in the original research. The policy-informing process involves evaluating whether the study’s findings are meaningful in terms of policy recommendations. This may involve using the original data or reanalyzing it with the same research question but perhaps additional methods, such as evaluating whether the findings are robust to alternative theoretical assumptions.

There are challenges with respect to authors’ and replicating researchers’ incentives. Authors who may believe in the scientific process of which replication is a part, may view successful replication as enhancing their credibility and influence, and may see it as raising visibility for their work. However, replication studies are costly in time and effort, and there are reputational effects if the replications reveal differences from the original results. While replication researchers may conduct such studies simply as learning exercises or providing added knowledge and policymaking evidence, there are also incentives in gaining publicity by calling attention to problems in well-known studies. Moreover, there is much uncertainty with respect to the returns on investment, concern about challenging the research of well-regarded scholars, the ability (or lack thereof) to obtain funding to replicate, and the time involved in completing the replication.

Additionally, there are ethical considerations in replication because the original author becomes the (non-anonymous) subject of the research), raising concerns about requirements of information transmission between the parties (e.g., providing authors with informed consent) and whether the author has a right to know the intention behind the replication. Moreover, there are questions as to original authors’ responsibilities in providing information to facilitate replications of their work.

Addressing these concerns (see the first part of Annette’s presentation here):

Learn more about 3ie’s replication paper series.

3ie produces a paper series in which they publish the replications that are successful (that is, supporting the original findings) as well as those that do not support the original findings, with the goal of avoiding the publication bias of presenting only those studies exposing failures in the original research. There is an extensive external review process for projects that they fund, including an external review advisor who follows the entire project, along with internal reviewers and an anonymous referee. They heavily edit reports for tone and language so as not to over-emphasize calling attention to mistakes that are not identified specifically. To increase incentives to conduct replication research, they provide small grants that will not fully compensate for all the time involved but which should provide a positive incentive for this type of work (even if the findings are supportive). They additionally provide grants for original authors for the purpose of deferring costs involved with preparing data files for future replication should their study be chosen. To ensure adherence to ethical principles in research, they post replication plans publicly online, allow the author to comment on the replication, require advance notice when findings will be presented prior to their being made public, and provide original authors the right of reply.

A challenge in engaging scholars in replication research is the emphasis on success or failure as opposed to discussing the policy-informing processes that are implicated, suggesting a potential need for new terminology in the conducting of such replications. An additional limitation is original authors’ reluctance to have their own work be the subject of the replication research. Between 13 and 15 replication studies have been initiated or completed.

Prepared by: Miranda Yaver