close× Call Us
close×
Title Summary versus individual fact-checking
Post date 05/07/2016
C1 Background and Explanation of Rationale

Fact-checking is an increasingly popular way for American political journalists to verify the accuracy of statements by political figures. The trend has grown rapidly, with prominent organizations such as FactCheck.org and PolitiFact receiving national recognition and journalism awards. The work of fact-checking organizations is often cited in the media, giving credibility and professional acceptance to this new form of political coverage.

There is currently very little literature on the effects of fact-checking. Moreover, existing research on the effects of correcting misperceptions has generated conflicting results. Nyhan and Reifler (2010) found that corrections failed to reduce misperceptions and, in some extreme cases produced a “backfire effect” in which corrections strengthened misperceptions. Fridkin et al. (2015) showed that fact-checks of negative television advertisements influenced people’s assessments of the accuracy, usefulness, and tone of negative political advertisements. Finally, Nyhan and Reifler (2016) showed that there was no evidence that exposure to fact-checking affected people’s attitudes toward the practice, trust in politicians, or feelings of political efficacy, but exposure to fact-checking did increase how much people knew about contemporary political controversies and debates.

The specific objective of this project is to examine the effects of summary fact-checking, an innovation in the field that is becoming more common. This format presents an overview of ratings of the accuracy of a politician’s statements rather than an evaluation of a single statement. Our study will measure the effects of summary fact-checks on a politician’s perceived accuracy and favorability. This project will also compare the effects of summary fact-checks to individual fact-checks that only verify a singular statement. We expect that summary fact-checking will be seen as more comprehensive and less selective than individual fact-checks and may therefore have a greater effect on public perceptions of politicians.

Our experiment will therefore assess whether or not the effect of individual fact-checks differs from the effect of summary fact-checks on an individual’s favorability toward a politician or the perceived accuracy of their statements. We will also take the political affiliation of both the participant and the politician into account, so we can determine whether the effects of individual and summary fact-checks vary by whether the target is a co-partisan or an opposition partisan. Finally, we will also consider whether these effects vary by political knowledge and analyze whether exposure to summary or individual fact-checks affects attitudes toward the practice more generally. The results of this experiment will allow us to understand the efficacy of different forms of fact-checking, which may help us better understand how news outlets’ fact-checking practices influence public opinion.

C2 What are the hypotheses to be tested?

H1: Exposure to an individual fact-check or to summary fact-check information will reduce perceptions of politician accuracy and favorability

H2: Perceptions of politician accuracy and favorability will be lower for participants exposed to summary fact-check information than for those exposed to an individual fact-check

H3: Perceptions of politician accuracy and favorability will change more in response to individual or summary fact-check exposure among opposition partisans than among co-partisans

RQ1: Will the effects of exposure to fact-check information on perceptions of politician accuracy and favorability vary by political knowledge/education or between partisans who are low vs. high in political knowledge/education?

RQ2: Will exposure to fact-check information change opinions toward the practice of fact-checking generally?

C3 How will these hypotheses be tested? *

Eligibility and exclusion criteria for participants

Participants will be United States residents age 18 and over recruited on the Amazon Mechanical Turk online marketplace. All Turkers are eligible to participate in this study who meet the specified qualifications and did not take part in an earlier pilot study. The sample size will be approximately 2800 – data collection will continue until all funds allocated to the project are exhausted. Researchers have no role in selecting the participants after listing the project on Mechanical Turk.

Randomization approach

We will use a between-subjects design in which respondents are randomly assigned to one of six conditions by the Qualtrics online survey platform (p=1/6 for each):

Summary fact-check treatment conditions:
-Fact-checking a Democrat (Harry Reid; asked about him first)
-Fact-checking a Republican (Mitch McConnell; asked about him first)

Individual fact-check treatment conditions:
-Fact-checking a Democrat (Harry Reid; asked about him first)
-Fact-checking a Republican (Mitch McConnell; asked about him first)

Control conditions (placebo image/text; no fact-check provided):
-Asked about perceptions of a Democrat first (Harry Reid)
-Asked about perceptions of a Republican first (Mitch McConnell)

Data collection and blinding

Data will be collected on the Qualtrics online survey platform. There may be some online discussion among Mechanical Turk workers about the details of our survey. This cannot be prevented and we hope that these participants preserve the integrity of the research.

Primary and secondary outcome measures

Our primary outcome variables are perceptions of the favorability and accuracy of the fact-check target (McConnell or Reid) compared to participants in the control group who were asked first about the same person. Each outcome variable will be measured on a five-point scale:

The next few questions will be about Senator Mitch McConnell (R-KY), the Majority Leader of the Senate. In general, how often do you think that he makes accurate statements?
-All of the time (5)
-Most of the time (4)
-About half the time (3)
-Occasionally (2)
-Never (1)

Do you have a favorable or unfavorable view of Senator McConnell (R-KY)?
-Very favorable (5)
-Somewhat favorable (4)
-Neither favorable nor unfavorable (3)
-Somewhat unfavorable (2)
-Very unfavorable (1)

The next few questions will be about Senator Harry Reid (D-NV), the former Majority Leader of the Senate. In general, how often do you think that he makes accurate statements?
-All of the time (5)
-Most of the time (4)
-About half the time (3)
-Occasionally (2)
-Never (1)

Do you have a favorable or unfavorable view of Senator Reid (D-NV)?
-Very favorable (5)
-Somewhat favorable (4)
-Neither favorable nor unfavorable (3)
-Somewhat unfavorable (2)
-Very unfavorable (1)

Our secondary outcome variables measure opinions toward fact-checking using the following questions, which will be analyzed separately and also as a composite measure if they scale together using principal components factor analysis.

In general, how favorable or unfavorable is your overall opinion of the fact-checking movement in journalism?
-Very favorable (6)
-Somewhat favorable (5)
-Slightly favorable (4)
-Slightly unfavorable (3)
-Somewhat unfavorable (2)
-Very unfavorable (1)

Thinking about the amount of fact-checking that you see being performed today by journalists, do you think there should be more fact-checking, the current amount of fact-checking is about right, or there should be less fact-checking?
-There should be more fact-checking. (3)
-The current amount of fact-checking is about right. (2)
-There should be less fact-checking. (1)

In general, how often do you think articles published by fact-checkers are accurate?
-All of the time (5)
-Most of the time (4)
-About half the time (3)
-Occasionally (2)
-Never (1)

In presenting the news dealing with political and social issues, do you think fact-checkers deal fairly with all sides or tend to favor one side?
-Fact-checkers tend to deal fairly with all sides. (1)
-Fact-checkers tend to favor one side. (0)

Statistical analyses

For each of our analyses, we will use OLS with robust standard errors and estimate ordered probit models with robust standard errors as a robustness check. Unless otherwise noted, all experimental treatment effects will be estimated as intent to treat effects. Leaners will be treated as partisans in our statistical analyses. Perceptions of politicians will only be assessed for the first politician for which outcome measures were collected to avoid contrast effects.

Participants’ political knowledge will be assessed with a standard five-question battery that test participant knowledge of U.S. electoral rules and awareness of current political figures. Partisanship is measured on a standard seven-point scale administered via branching questions.

Our primary model will be pooled regressions for individuals who saw either candidate, but we will also estimate regressions interacting target politician with fact-checking type (the main explanatory variable). If we cannot reject the null of no difference in effects by target, we will present the pooled model and also present separate results for Reid and McConnell either in the main text or an appendix. If the effects of key explanatory variables differ depending on the fact-checking target, we will present separate models for Reid and McConnell in the main text for expositional clarity and present interactive models in an appendix. We will estimate marginal effects as appropriate when interaction terms are included in our models. (We will also estimate our treatment effect models with standard demographic controls and report these results if the treatment effect estimate is substantively different from the model only including partisan affiliation.)

To test H1, we will estimate the model y=b0+b1*summaryFC+b2*individualFC for the accuracy and favorability outcome measures. To test H2, we will then evaluate b1-b2 directly for each outcome measure.

To test H3, we will estimate the model y=b0+b1*summaryFC+b2*individualFC+b3*copartisan+b4*opposition+b5*summaryFCXcopartisan+b6*summaryFCXopposition+b7*individualFCXcopartisan+b8*individualFCXopposition and then estimate b5-b6 and b7-b8.

[Per above, we will estimate all pairwise comparisons in treatment effects between partisan groups. However, we expect to encounter power problems given the relatively small number of pure independents and may thus focus on the comparison between the co-partisan and opposition partisan groups.]

To test RQ1, we will first estimate the model y=b0+b1*summaryFC+b2*individualFC+b3*knowledge+ b4*summaryFCXknowledge+b5*individualFCXknowledge and then estimate y=b0+b1*summaryFC+b2*individualFC+b3*copartisan+b4*opposition+b5*summaryFCXcopartisan+b6*summaryFCXopposition+b7*individualFCXcopartisan+b8*individualFCXopposition +b9*knowledge+b10*summaryFCXknowledge+b11*individualFCXknowledge+b12*copartisanXknowledge+b13*oppositionXknowledge+b14*summaryFCXknowledgeXopposition+b15*individualFCXknowledgeXopposition. In each model, we will estimate whether the effects of the fact-checks vary by political knowledge versus the control group and each other (and, in the latter case, by partisanship as well). We will test whether the effect of knowledge is linear per Hainmueller et al. and follow their recommendations if not. We may also present a simple median split on knowledge in the main text and report the continuous interaction model in the appendix for expositional clarity. We will follow the same procedures for education, grouping all the some college/associate groups together in the continuous measure and treating college/non-college as the relevant binary measure.

To test RQ2, we will estimate the model y=b0+b1*summaryFC+b2*individualFC for individual fact-check opinion measures and the composite fact-check opinion measure (if they scale together) and then estimate b1-b2.

C4 Country United States of America
C5 Scale (# of Units) 2800
C6 Was a power analysis conducted prior to data collection? not provided by authors
C7 Has this research received Insitutional Review Board (IRB) or ethics committee approval? not provided by authors
C8 IRB Number STUDY00029513
C9 Date of IRB Approval 4/22/16
C10 Will the intervention be implemented by the researcher or a third party? Researchers
C11 Did any of the research team receive remuneration from the implementing agency for taking part in this research? not provided by authors
C12 If relevant, is there an advance agreement with the implementation group that all results can be published? not provided by authors
C13 JEL Classification(s) not provided by authors