A well-written question will mean the same thing to all respondents.

Jennifer Leigh Brown

When creating a survey, if you’re hoping to receive accurate and insightful answers, the phrasing and functionality of your survey queries can determine the success of your results. Questions that are poorly worded, biased, confusing or don’t allow respondents to accurately reflect on their experiences can result in erroneous data. Answering these queries is a waste of time for respondents and, worse, their responses can be misconstrued in a surveyor’s findings.

In my last blog, Thoughtful Survey Design, I detailed how to identify your survey purpose, determine your target population and determine questions that will help determine demographics. For this post, we’re diving into the meat of surveys and the importance of crafting thoughtful, well-phrased and unbiased questions. First, I’ll define some crucial terms around survey development:

• Respondent – an individual of the target population.
• Bias – a factor that affects the representative capacity of a question, including how clearly and easily respondents can respond to a question.
• Satisficing – a tendency for respondents to put in the least amount of effort they can when answering a survey.
• Skip logic – when some survey questions are conditional and only appear if relevant. Use this feature to filter your respondents and the questions they face based on their responses.
• Qualitative data – Labels, names and descriptive information. Occurrences can be counted and you can determine proportions and percentages.
  • Nominal data are categories, names, or labels that don’t have an innate order. This can include data like country, brand name or mode of transportation.
  • Ordinal data is nominal information that contains a sense of ranking or order. This can include data like year in college or a scale from strongly disagree to strongly agree.
• Quantitative data – Indicates how many or how much of something on a numerical scale. Further statistics can be calculated with this data.
  • Interval data is recorded with standard units of measurement with equal intervals. This can include temperature, SAT scores or your credit score.
  • Ratio data is interval data but with a meaningful zero point that represents the absence of that characteristic. This includes weight, time, distance and count.

 

Question Types

Before crafting your questions, it’s important to know what type of results you want. If you want proportion and frequency statistics, text responses, and relative comparisons, you’ll likely have to ask different questions than if you want statistics like averages, standard deviations and probability distributions. To help clarify, below are some key question modalities and the output they’ll give you.

1. Multiple Choice. Multiple choice questions are a flexible and relatively easy-to-use question style for both surveyors and respondents. This format can allow for just one response (forced-choice) or for the respondent to select all the options that apply to them, but Pew Research suggests that forced-choice questions yield more accurate data. The data these questions yield is easy to parse; typically results are a count of the number of responses to each option, or, in the case of the multiple-answer option, a matrix of each respondent and their choices. Depending on the options the question offers, this question type can result in qualitative or quantitative data.
   1. Binary/Dichotomous. These questions offer just two choices, such as yes/no. These questions don’t provide a neutral option or allow the respondent to express nuance, but can be useful for skip logic that allows respondents to go into detail with further questions based on their choice. These questions can provide ratio data if the binary choices are translated to 1s and 0s.
   2. Rating. Questions that ask you to rate something, like from one to five stars, are typically easy to understand and translate across language barriers. These scales seem to make opinions and experiences quantifiable, but the data you receive is actually ordinal because the intervals aren’t standard.
   3. Likert Scales. These are a type of rating question, typically ranging from a 1 for “strongly agree” to a 5 for “strongly disagree.” Similarly, semantic differential questions ask you to complete a sentence with one of the options provided. Like Rating questions, Likert and semantic differential questions result in ordinal data.
2. Rank Order. Ranking questions require respondents to order the options provided based on a perceived quality. The respondent is required to make choices between two or more things which can offer further insight into the respondents’ opinions. These questions can be frustrating and time-consuming, so make sure to limit the number of items and use these questions thoughtfully. As the options themselves have no order, the data collected is nominal.
3. Text Boxes. Text boxes can vary in size and character limit but accept free responses from participants. Typically, this data is nominal, but some survey tools allow you to specify that you only want to receive numerals. This can be useful for gathering phone numbers, zip codes, ages and more.
4. Submission. Requesting that your respondent upload a file or submit a picture can be a useful way to receive more information from respondents but can be confusing or invasive. Be clear on the submission you expect, acceptable file types, how to upload, and file size requirements.

 

Writing Queries

With your main survey focus in mind, start coming up with key takeaways to glean from the survey. You’ll use these core topics to develop survey questions. For example, your main focus could be to determine the best ice cream flavors to offer in Seattle. Some core topics might include determining respondent’s favorite flavors, exploring the boundary between interesting and off-putting flavor combinations, or ascertaining appropriate price ranges.

Based on your core topics, pair what kind of data will be able to address the topics. If you want to know a price that people would pay for a unique ice cream flavor, you’ll need to use a question style that offers quantitative data, likely a slider. If you want to get ideas for ice cream toppings, a text box will allow the respondent to respond freely.

From there, start drafting the queries. Use creativity and the 5 Ws (and H) of journalism to navigate and delve into your core topics. Once you’ve developed your first draft of your survey queries, double-check that you follow these best practices:

• Keep vocabulary simple and check your grammar. Avoid jargon, use fewer words when possible, stick to active voice and consider using an informal, conversational tone to make your survey more inviting. Make sure to avoid double negatives and unclear references.
• Be specific. Clarify any specificities around time frame, location, scope or experience.
• Ask one thing at a time. Asking multiple things within the same question can confuse respondents. If additional instructions, examples or context is needed, try to keep these in a separate sentence.
• Avoid leading/loaded questions. These questions make assumptions about the respondent or their experience that are hard for them to deny. For example, asking, “Why is ice cream your favorite dessert?” assumes that ice cream is their favorite dessert. In addition, acquiescence bias is the tendency for people to agree to a statement, regardless of the content. Rephrasing the question can mitigate this bias in many cases.
• Offer comprehensive options. Research your population to determine the best options to include. If there are many more options available, consider using “of the following” to allow your respondent to answer in good faith. If your options are not comprehensive, you can also use an “other” option with a text box to capture their response – this doesn’t seem to affect the reliability or validity of the data.
• Use extreme judgments like “never” and “always” carefully. For most subjects, there are exceptions and gray areas; using a scale with options between always and never tends to capture data better.
• Vary your question types. Respondents can get irritated or lazy when they are asked the same type of question repeatedly, which can lead to satisficing. Varying question types and difficulty can make your survey more interesting to complete.

 

Survey Assembly

Once you’ve drafted your queries and made sure they’re well-phrased, unbiased, comprehensive and will help you address your core topics, it’s time to arrange them into an order that will make sense and encourage the respondents to answer all queries. If you need help eliminating unnecessary questions, consider learning about how you can choose your best idea.

As I explained in my last blog, start your survey with screening questions to confirm all participants are part of the intended population and, if they are not, use skip logic to cut their survey short.

The question order is mainly subjective, but here are some guidelines that can improve your results. Research from Ohio State University’s Jon Krosnick suggests that interspersing easy questions between hard or time-consuming questions will make the survey seem more interesting and easier to complete.

Grouping questions into themes will help one question give context to another. Progress bars and similar indicators can boost the completion rate. Funneling, starting with broad, easy questions, progressing to specific and time-consuming questions, and then finishing with quicker and optional questions, gives the respondent time to “warm up” and makes the survey seem quicker with a slide to the finish line.

To this end, consider placing optional demographic questions at the end of the survey.

After writing the introduction to your survey and uploading your questions onto your survey platform of choice, test your survey on yourself and others. This is a chance to check your timing, verify the usability of your survey tool, add context and instructions as needed and judge if you need to account for further edge cases within your close-ended questions. Finally, this testing will give you some scrap data from the program to check that the output will be usable when you go to analyze your results.

The final step is to publish it! Share your survey with the goal of exceeding your minimum sample size within your target population. Check out my last blog for more information about sampling.

I will go into more depth about how to analyze your results in my July 2021 blog – come back to learn more!

You should now have a targeted survey for a well-defined population that will give you data to tackle your core topics. You’ve given respondents the best chance to communicate their experiences easily and accurately. All that’s left to do is to see what they say.