Impacts of Transition Statements in Survey Questions on Survey Break-off: Evidence from a Survey Experiment
Summary: The California Health Interview Survey (CHIS) has employed an addressed-based sampling (ABS) frame with a mail push-to-web interview followed by a telephone nonresponse follow-up as the primary data collection approach since 2019. However, the nature of the self-administered web survey results in more survey breakoffs than the previous computer-assisted telephone interview (CATI). During CHIS 2021 data collection, the CHIS team observed that a large proportion of questions with high break-off incidence began with transition statements, such as “The following questions are about…” or “These next questions are about...”. Therefore, experiments were warranted to test whether eliminating transition statements leads to a reduction in survey breakoffs during CHIS 2022.
This study evaluates an experiment conducted in CHIS 2022, where respondents were evenly split and randomly assigned to two conditions: (1) a treatment group where transition statements were removed from the selected twenty-six questions; (2) a control group with the original question wording, including transition statements.
Findings: Data demonstrate that eliminating transition statements results in substantive survey break-offs reductions. Aggregated breakoffs from the 26 questions have decreased by 44.2%. For individual questions, reduction rates range from 14% to 82%. Results also show that removing the transition statements converted sufficient partials to fully completes and slightly shortened interview length. Consequently, all transition statements except an outlier have been removed for the remainder of the CHIS 2022 and transition statements will be less likely to be included in new survey question development for the CHIS.
Read the Publication: