I Understand
We use cookies.Click here for details.
Alex | 19 April 2021

Part 2: Survey Distribution

There are still many steps to complete even after you and stakeholders are happy with the questionnaire. For instance, how will you distribute the survey?

You likely designed your survey in a tool such as Qualtrics, Survey Monkey, or Google Forms. Each one of these has the capability to help distribute the final version to potential respondents. The most common methods include:

What messaging will be used to encourage a response?

You also need to decide what to say to engage potential respondents and maximize survey completions. Given the volume of noise coming into people’s email accounts these days, it makes sense to keep the message short, clear, and inspiring.

This is true for both the subject line and the body. Make it sound like it came from an actual human and clearly communicate the benefits of taking the survey and the expected time commitment associated with it.

It is also important to briefly explain what will be done with the survey responses. Are they anonymous? Confidential? Where will the information be stored? Given the long list of potential questions, it often makes sense to link to a full data privacy and survey participation policy.

Here is an example of a concise survey invite.

Subject: Help others improve data science hiring

Body:

Everyone is struggling to hire data talent today. We’re working with HR professionals from around Europe to understand key challenges and opportunities. Everyone who participates will receive the full set of results, complete with benchmarking opportunities.

Would you like to join the community? If so, please complete this 5 minute survey:

Start here –> https://www.surveymonkey.com/mp/apps-survey-template

Your responses are anonymous and only aggregate data will be reported out. Click here for our data privacy policies or feel free to reach out to me directly.

Best,

Doug

Survey Director

Company XYX


Should you use incentives? What should they be?

Incentives can be a double-edged sword. Although they may be necessary given the competition for attention today, they may also indirectly introduce sampling bias as a certain type of person may become over-represented in the results.

If you decide to use an incentive, you also have to choose (1) what it should be and (2) who will be eligible?

Perhaps you will send a summary report of the final results to everyone who completed the survey. This is a non-monetary incentive.

It is also common for organizations to provide monetary alternatives, such as five dollar Amazon gift card, to help compensate respondents for their time. Some choose to give small amounts to everyone who responds. Others add a sense of urgency by only offering that amount to the first x number of respondents. Another approach is to select a larger incentive amount that will be limited to a smaller number of randomly selected respondents after the survey closes.

Regardless of what you choose to do, be sure to follow through on promise made and have a legal set of conditions available to share before the survey goes live.

Also be aware that in certain countries random draw incentives are either illegal or subject to direct regulation. For example, in the United States random drawings are legally considered to be lotteries and must follow related guidelines.

How often should you remind people to complete it?

Most survey projects send out reminders to further boost response rates. The number of reminders and any changes to messaging or incentives will likely depend on progress made against the original sample target. You should plan on sending at least one reminder message but try to avoid sending more than two or risk being seen as spammer.


What are your success metrics?

Your ability to calculate various success measures depends in part on how you distributed the survey. Here are some possible options if you sent out personalized links via email.

Opened the email

The proportion of people who opened the initial survey invitation.

Response rate

Finally, you can divide the number of people who have taken the survey by the total number of people invited to find the response rate. Your response rate will depend on several variables such as your relationship with the audience, the effectiveness of your invitation message, the magnitude of included incentives, and the topic and length of the survey itself.

There is also the potential distinction between partial completes - people who started but did not finish the survey - and full completes - the people who press the final submit button.

Quality of responses

Although more subjective than the pure conversion metrics, it is a good idea to find an agreed upon approach to measure the quality of responses that you will ultimately analyze.

These may include how closely the profile approximates the population of interest, the percent of questions completed, or the amount of text submitted from open-ended questions.


Next steps

We have seen that building a questionnaire, delivering the survey to potential respondents, and collecting results takes significant coordination and effort. It also marks the starting point for us to finally be positioned to use new pieces of data to generate insights of interest.

In part 3 we’ll look at how to conduct survey analysis for the most common question types.

This article is the second of three-part series on practical survey considerations with guidance from Gregg Schoenfeld, survey guru and founder of MNoet, a boutique research consultancy.