Surveys are a personal love of mine. I think it’s so much fun to learn more about others, especially their thoughts, opinions and experiences. Because a survey is easily scalable, I can put my effort into making it perfect once and I don’t need to perform the same interview 100 times and inevitably ask a question phrased strangely or mishear someone. Surveys also allow you to determine representative opinions of a population without needing to ask every single person.

I find it very rewarding to discover, through survey, that my preconceived notions and trends were incorrect. However, if you choose not to rely on social research strategies like surveys, alternative sources of knowledge include social myths, extrapolating from personal experience, listening to authority, referring to tradition and using common sense. I wouldn’t recommend you cite the opinions you get from any of these sources as anything close to fact.

It’s tempting to think that surveys are easy to create because it boils down to a series of questions and statements given to people. However, to create a survey that you can use to extrapolate and form theories, it’s crucial that you put time, effort and thought into why you’re putting together this survey, who is included in the population you’re investigating, how you’re finding the sample you survey, how you ask your questions and the bias you may embed within them, which techniques you use to analyze your results and what conclusions you allow people to draw about your results.

In this blog, I’m going to focus on defining your audience, how to develop better sampling practices and some overall survey design. I’ll dive into how to ask the right questions next month and then finish up with how to analyze and communicate results in a third blog. Throughout, I’ll share lessons I was taught in college in classes that include Introduction to User Research, Qualitative Research Methods and Introduction to Statistical Methods, but I’ll also link to some of the internet’s many resources.

Let me start off by defining some crucial terms:

• Population – The entire set of people you want to understand. They should have something in common.
• Sample – The subset of your population that you survey. This sample should be as random as possible to limit added factors that can affect your results.
• Respondent – An individual within your sample.
• Qualitative data – Labels, names and descriptive information. Occurrences can be counted and you can determine proportions and percentages.
  • Nominal data are categories, names, or labels that don’t have an innate order. This can include data like country, brand name or mode of transportation.
  • Ordinal data is nominal information that contains a sense of ranking or order. This can include data like year in college or a scale from strongly disagree to strongly agree.
• Quantitative data – Indicates how many or how much of something on a numerical scale. Further statistics can be calculated with this data.
  • Interval data is recorded with standard units of measurement with equal intervals. This can include temperature, SAT scores or your credit score.
  • Ratio data is interval data but with a meaningful zero point that represents the absence of that characteristic. This includes weight, time, distance and count.
• Inference/Inferential statistics – An inference is taking many data points and using them to draw a conclusion. Inferential statistics is how you do that accurately and scientifically with your survey.
• Bias and skew – Bias is a factor that affects the randomness and therefore representative capacity of a sample. Skew can describe results gathered from biased surveying.

 

Step 1: Motive and Population

Even though your head may be filled with the questions to ask, the first thing to focus your attention on is why you’re making a survey in the first place. Maybe you’re trying to figure out characteristics of a certain population, or you’re interested in their opinions around a brand, or you’re gauging their satisfaction with a service and looking to see if they’d recommend it to a friend. Also, make sure to consider how you want to use the results and in what ways you want to share your findings.

Once you have your motive, define your population. Pick a distinguishing feature of your population, like being a customer of X brand, living in Y state, or being within a certain age bracket.

Do a little research to determine the total number of individuals within your population. With this information, you can calculate the minimum sample size that will allow you to accurately extrapolate to your population. I recommend you use SurveyMonkey’s Sample Size Calculator or something similar because the calculation can be complicated. In general, the smaller the population, the higher percentage of people you need to survey to get a representative sample. If you are not able to get a representative sample of the population, that means you should not extrapolate your findings to the entire population, but the trends you find can still be helpful.

 

Step 2: Sampling

Sampling is one place where bias can easily creep in and taint your results. There are many ways you can attract some people in your population and not others including how you share the survey, the survey format, uneven appeal to any incentives you offer, the time it takes to complete the survey, the language of the survey and the way you introduce the survey. Sampling bias is hard to avoid but important to minimize. To reduce sampling bias, you can make your survey accessible to all individuals in the population, design your survey to be accessible and oversample. Your survey will likely still be affected by self-selection bias; this should be quantified and acknowledged when you share your results.

 

Step 3: Survey Plan

If your survey may be accessible to people beyond your target population (e.g., on social media, in public, from an outlet with a broad audience), you should always start your survey with screening questions to confirm all participants are part of the intended population. If a respondent doesn’t match your criteria, thank them and don’t let them waste their time filling out the rest of your survey.

Next, consider what types of demographic information you’re interested in reporting when you share your results. This may include age, location, profession and gender, all of which can be sensitive information for people to share. Consider keeping demographic questions more quantitative and general to respect anonymity and make it easier to use in statistics. Check out Hubspot’s guide to demographic questions if you want some sample questions you can mimic.

The last questions you should craft are the ones related to your topic. When drafting your questions, first determine why you’re asking the question. You can use this statement to determine the type of question and craft the query itself. If you’re collaborating, establishing the purpose behind every question will help make sure everyone’s on the same page and will allow people to give better feedback by helping to ensure that answers to the question will result in the responses you want.

I go into more depth about this in my June 2021 blog which you can find here!

After you craft all your questions, try to group them into themes. This will help give context to your questions and make the survey easier and more enjoyable to complete.

 

Step 4: Presentation

Once you’ve put together all your questions and arranged them approximately where you want them, draft the introduction to the survey. This should include your survey purpose and how you’ll use the data, any instructions you need to clarify, and an expectation of how long this survey will take. Make sure to thank them for their time and provide contact information. All of these elements will allow respondents to know what they’re getting into and, thus, give informed consent before they participate.

With your survey content drafted, put it into your survey on your platform of choice. There are so many options you can use, and each has its own strengths and capabilities. If you need a suggestion, G2 summarized some of the free options, but if you have the resources for paid survey software, the capabilities and ease of use can be worth it.

Before you share your survey with your population, make sure you take the survey yourself and ask a friend or two to test it for you as well. In addition to necessary proofreading, this practice allows you to ensure your questions make sense, that your topics flow from one question to another, that your questions have the options respondents need, that your timing estimate is accurate, and that your survey program is working correctly. Finally, this gives you some scrap data from the program to check that the output will be usable when you go to analyze your results.

I will go into more depth about how to analyze your results in my July 2021 blog – come back to learn more!

 

At this point, your survey has a well-defined purpose and population that will allow you to make meaningful statements with your survey results. Your sampling method avoids unnecessary bias and is hopefully as random as possible to reflect your population. Your questions within your survey will allow your respondents to easily give you their thoughts on the topics you want to hear about and your introduction will allow them to offer informed consent. All of these elements will elevate your survey and, with practice, research and expert feedback, allow you to come to more accurate conclusions.