Relevance & Research Question:

The GP Patient Survey (GPPS) is a large-scale, non-incentivised, postal survey with an online completion option. As with other surveys, GPPS has experienced declining response rates and increasing postage costs. A new sampling frame provided access to mobile numbers in 2020, allowing us to experimentally test several push-to-web strategies with multi-mode contact. One experiment tested the feasibility of replacing a postcard reminder with SMS contact.

Methods & Data:

GPPS uses stratified random sampling of all adult patients registered with a GP practice in England. A control group was obtained from the existing sample frame, before selecting an experiment group from the new sample frame using the same criteria. Fieldwork took place simultaneously and tested the following:

1. Control (n=2,257,809) received three mailings with online survey log-in details and paper questionnaires, and a postcard reminder after mailing one.

2. Experiment 1 (n=5,982) received three mailings with online survey log-in details and paper questionnaires, and an SMS after mailing one. Experiment 2 (n=5,978) also received a second SMS after mailing two. Both SMS reminders included unique online survey links.

Results:

Where one SMS replaced the postcard (experiment 1), participants were pushed online compared with the control (27.2% vs. 19.4%) but the response rate was lower (30.4% vs. 31.9%). Sending two SMS reminders (experiment 2) pushed participants online (29.2%) with no significant impact on response rate (31.6%).

Neither demographics nor survey responses were impacted for the experiment group, suggesting no impact on trends. There was some evidence of impact on data quality: non-response increased for questions with long response scales for those completing via SMS (compared with via the GPPS website or letter URL).

The experiment also provided significant cost savings: SMS is cheaper than postal contact, and maintaining the response rate with more online completes reduced return postage and scanning costs.

Added Value:

Although previous studies have trialled SMS reminders, this provides direct comparability between postcard and SMS contact using a large-scale, non-incentivised, general population survey. The results provide insight into the impact on online completion, response rate, trends, non-response bias and cost-effectiveness.

Relevance & Research Question:

With increasing cost of data collection for phone, mail and in-person modes, the need for robust on-line data collection methodologies has never been greater. Text message surveys have a unique advantage in conducting short, quick turn-around surveys in a cost-effective manner. Text message surveys can also be quite effective in reaching harder-to-reach populations. To-date, the use of this methodology has been limited due to concerns of low participation rates and representativeness of text message-based surveys. Also, majority of Text message-based survey research to-date has been conducted via opt-in panels. SSRS has launched the first TCPA-compliant nationally representative probability-based text message-based panel. This paper explores the effectiveness of probability-based Text message survey panel as a data collection methodology.

Methods & Data:

Data collection was conducted via an interactive text message survey (as opposed to sending a web survey link via Text message). The advantage of this methodology is that the survey can be administered via smart phone or other phones thereby improving coverage. No internet service is required as the Text messages are sent via mobile service. To evaluate the effectiveness of Text message-based survey methodology in generating projectable population-based estimates, we conducted Text message survey and a parallel survey fielded via RDD phone.

Results:

In this paper, we provide demographic and substantive comparison of RDD phone and text message-based survey methodology. Our findings suggest that Text message surveys produce very similar results to time-tested RDD phone methodology.

Added Value:

In addition to providing methodological guidance on implementing Text message surveys, this paper also provides best practices guidance in implementation of Text message-based surveys.

Life in Australia™ is Australia’s only probability-based online panel, in operation since 2017. The panel was initially recruited in 2016 using dual-frame random digit dialling (RDD), topped up in 2018 using cell phone RDD as a single frame, expanded in 2019 using address-based sampling (ABS), and topped up in late 2020 using a combination of ABS, interactive voice response (IVR) calls to cell phones, and SMS push-to-web (i.e. invitations using only SMS), noting that a different regulatory regime to the TCPA prevails in Australia, which allows for automated dialling of cell phones and sending SMS without prior consent.

We present our findings with respect to recruitment and profile rates, retention, and completion rates. We also present the demographic profile of panel members and compare it to Census 2016 benchmarks with respect to age, gender, education, and nativity. We supplement our respondent profile findings with results of trials we ran on IVR and SMS as modes of invitation.

The yields from IVR and SMS push-to-web sample were below that of ABS, however the costs for IVR and SMS push-to-web were well below those of ABS and the less expensive modes actually delivered a more desirable panel member profile with respect to age and nativity, though not education. Our research raises interesting questions as to the trade-off between bias, cost and face validity in the form of response rates.

This paper contributes to the international body of research on recruitment methods for probability-based online panels (see, e.g., Bertoni 2019; Bilgen, Dennis, and Ganesh 2018; Blom, Gathmann, and Krieger 2015; Bosnjak et al. 2018; Jessop 2018; Knoef and de Vos 2009; Meekins, Fries and Fink 2019; Pedlow and Zhao 2016; Pew Research Center 2015, 2019; Pollard and Baird 2017; Scherpenzeel and Toepoel 2012; Stern 2015; Vaithianathan 2017; Ventura et al. 2017).