Webinar: Survey Questionnaire Design 14/3/2017
Watch the recording of the Webinar
A recording of the webinar and the associated PowerPoint slides can be found at http://www.iiba.org/Learning-Development/Webinars/archived-webinars.aspx.
Questions and Answers following the Webinar
Immediately following the webinar a brief Q&A session was held with the audience. Responses to each of these questions can also be found below, and we look forward to discussing our different experiences in the LinkedIn group for our SIG here.
Q1: Generally, how often should surveys be sent out?
In responding to this question there are actually two ways of interpreting this question. The first is "how often should we collect new survey data on a group of participants?" and the second is "how much effort should we put into chasing non-responders to a survey?". I will provide my own opinion on these two questions here.
It is often of interest to collect data on a group of participants on more than one occasion to see how that group of participants changes over time, or indeed to make sure that our understanding of that group of participants is still current. It is then a question of how often should we ask participants to complete our survey. A textbook example from my home country of Australia comes from the Australian census which up until 2016 we were doing every 5 years, and where we are now looking at doing the census every 10 years. There is no one single answer as to how often to conduct a survey, but some deciding factors in this decision include ...
1) how quickly do the results of a survey change over time (where this can be estimated using strategies such as looking at the change in previous timepoints, conducting a new survey with a pilot (smaller) group to estimate the level of change, or by expert (application matter specialist) opinion).
2) how much time and cost is involved in conducting a new survey (and hence are the potential benefits worth it)
3) how much burden would this place on participants, and accordingly what impact would there be for non-response if we conduct a survey too regularly (any survey is a burden in times of time and effort for participants and we need to do everything we can to respect the participants in any survey)
There is a difficult balance there, with no one single answer. If it is difficult to find a balance there then maybe you could solicit some thoughts on this from member of your target population (such as through brief interviews or focus groups).
Secondly, ‘how much effort should be put into chasing non-responders to a survey?”. Within any survey we will find that some people will complete the survey and some won’t. It may be worthwhile trying to follow up with those people who didn’t complete the survey, as some people who don’t complete the survey when first asked will complete the survey if asked a second (third or fourth time). When chasing non-responders there is again no one single best approach here, but there are a number of considerations similar to those above
1) How much bias would we expect due to non-response? This is a complex question but relates to how strong is the relationship between the reason for why a person might not complete the survey and the topic of the survey.
2) How much time and cost is involved in chasing non-responders? (Alternatively sometimes it is advantageous in a survey to send out the initial survey through a cheap method such as a mail survey and then to chase non-responders through a more expensive method such as a telephone survey, as non-responders are more likely to respond to a telephone survey than to a mail survey)
3) We need to balance our eagerness for people to complete a survey with the danger of harassing them. There is no single answer as to how many attempts to follow up a person is crossing the line.
4) How much time should we allow before we make follow-up attempts. Too little time and we might harass people that are already in the process of responding to our first contact attempt, and too much time and people will forget that there was already first contact attempt.
Again there is no single answer here as this really depends upon factors such as the level of importance that a potential participant might place in the need of the survey. However again it might be worth talking to a few representative members of your target population to understand their thoughts on attempts to contact non-responders.