There are still many steps to complete even after you and stakeholders are happy with the questionnaire. For instance, how will you distribute the survey?
You likely designed your survey in a tool such as Qualtrics, Survey Monkey, or Google Forms. Each one of these has the capability to help distribute the final version to potential respondents. The most common methods include:
You also need to decide what to say to engage potential respondents and maximize survey completions. Given the volume of noise coming into people’s email accounts these days, it makes sense to keep the message short, clear, and inspiring.
This is true for both the subject line and the body. Make it sound like it came from an actual human and clearly communicate the benefits of taking the survey and the expected time commitment associated with it.
It is also important to briefly explain what will be done with the survey responses. Are they anonymous? Confidential? Where will the information be stored? Given the long list of potential questions, it often makes sense to link to a full data privacy and survey participation policy.
Here is an example of a concise survey invite.
Subject: Help others improve data science hiring
Everyone is struggling to hire data talent today. We’re working with HR professionals from around Europe to understand key challenges and opportunities. Everyone who participates will receive the full set of results, complete with benchmarking opportunities.
Would you like to join the community? If so, please complete this 5 minute survey:
Start here –> https://www.surveymonkey.com/mp/apps-survey-template
Your responses are anonymous and only aggregate data will be reported out. Click here for our data privacy policies or feel free to reach out to me directly.
Incentives can be a double-edged sword. Although they may be necessary given the competition for attention today, they may also indirectly introduce sampling bias as a certain type of person may become over-represented in the results.
If you decide to use an incentive, you also have to choose (1) what it should be and (2) who will be eligible?
Perhaps you will send a summary report of the final results to everyone who completed the survey. This is a non-monetary incentive.
It is also common for organizations to provide monetary alternatives, such as five dollar Amazon gift card, to help compensate respondents for their time. Some choose to give small amounts to everyone who responds. Others add a sense of urgency by only offering that amount to the first
x number of respondents. Another approach is to select a larger incentive amount that will be limited to a smaller number of randomly selected respondents after the survey closes.
Regardless of what you choose to do, be sure to follow through on promise made and have a legal set of conditions available to share before the survey goes live.
Also be aware that in certain countries random draw incentives are either illegal or subject to direct regulation. For example, in the United States random drawings are legally considered to be lotteries and must follow related guidelines.
Most survey projects send out reminders to further boost response rates. The number of reminders and any changes to messaging or incentives will likely depend on progress made against the original sample target. You should plan on sending at least one reminder message but try to avoid sending more than two or risk being seen as spammer.
Your ability to calculate various success measures depends in part on how you distributed the survey. Here are some possible options if you sent out personalized links via email.
The proportion of people who opened the initial survey invitation.
Finally, you can divide the number of people who have taken the survey by the total number of people invited to find the response rate. Your response rate will depend on several variables such as your relationship with the audience, the effectiveness of your invitation message, the magnitude of included incentives, and the topic and length of the survey itself.
There is also the potential distinction between partial completes - people who started but did not finish the survey - and full completes - the people who press the final submit button.
Although more subjective than the pure conversion metrics, it is a good idea to find an agreed upon approach to measure the quality of responses that you will ultimately analyze.
These may include how closely the profile approximates the population of interest, the percent of questions completed, or the amount of text submitted from open-ended questions.
We have seen that building a questionnaire, delivering the survey to potential respondents, and collecting results takes significant coordination and effort. It also marks the starting point for us to finally be positioned to use new pieces of data to generate insights of interest.
In part 3 we’ll look at how to conduct survey analysis for the most common question types.