Survey Design Best Practices: How to Write a Good Questionnaire

When it comes to survey design best practices, there are six key considerations, including relevance, objectivity, clarity, the look & feel, your question structure, and the survey flow.  This article provides some tips and tricks related to each of these components, helping you construct effective survey questionnaires.

Relevance: The Secret is to Have a Plan

The first step in effective survey design is to have a plan.  Before you start crafting a questionnaire, ask yourself the following five questions:

1. What business decisions am I trying to inform?
2. If I knew __?__, I would be more prepared to make these important business decisions.
3. What am I trying to measure?  Attitudes? Behavior? Intentions? Perceptions?
4. Who is my audience?
5. What kind of statistics do I want to come out of this project (e.g. descriptive vs. inferential statistics)?  What will my analysis look like?

Now, as you craft your questions, ask yourself if the data provided will address those objectives.  After I write a question, I like to imagine that I instantly have the raw data in my hands, then I ask myself: “Ok, now what?  What can I do with this information?  Does this give me the information I need?”  This simple exercise will help you to construct questions that really matter and will yield actionable data; you will avoid filling your survey with a bunch of “nice to know,” but non-essential questions.

Don’t forget to test your survey!
A key part of the planning process is testing your survey.  Find some people–including someone’s grandparent–to take your survey without any coaching from you, and then gather their feedback.  Ask them about their general impression.  You should also ask some specific questions:

1. Was the survey engaging?
2. How long did it take you to complete?
3. Was the question flow logical?
4. Were there any confusing questions?
5. Were there any areas of frustration?

When you test the survey with several people, DO look at the data that comes back.  Get this raw data into your spreadsheet or data analysis program and start the process of analyzing your data.  If there are any issues with data structure, improper question type or scale types, they will likely present themselves here and you can fix them before you launch the real deal.

Question Structure – Types of Questions

When it comes time to start writing questions, you have many types of questions to choose from.  To keep it simple, we can categorize them all into two groups: structured (or close-ended questions) and unstructured (or open-ended questions).  An unstructured, open-ended question is something like this.

Some general tips on question structure:

1. Limit your use of open-ended questions

These are more taxing on respondents than structured questions and will lead to survey fatigue if overused.  If you have an urge to use a lot of open-ended questions, you probably haven’t done enough exploratory research.  You might benefit by conducting some secondary research, interviews, or focus groups before you embark on a survey.

2. Use appropriate scales 

Let’s keep this simple without getting into “nominal” and “ratio” scale type jargon–let’s say you want to know about the age of your respondents.  You have several ways to ask this question, each of which will produce a different level of specificity.  If you need to report an average age, for example, you would want to use the question on the right vs. the question on the left (below).  Since the question on the right provides more detail, why do surveys always seem to ask the question in the “range” format?  Well, age can be a sensitive issue for some people (especially old people), so people are often more comfortable giving a range rather than an absolute value.  Keep this in mind when crafting your questions and try to strike a balance that meets your needs.

3. Don’t ask unnecessary questions  

Redundant questions really annoy people, and they should!  Here is an example: At the doctors office last week, I was asked my date of birth on a form.  The very next question asked for my age.  Don’t do this!  If you know someone’s date of birth, you can easily calculate an age based on the current date.  If you ask for a zip code, don’t ask about my city and state.  Think about what additional data you can extract from your questions before asking a unnecessary questions and adding to the length.

4. Multiple choice questions need to be mutually exclusive

It is extremely frustrating when multiple answers apply, but respondents are only allowed to select one.  In the example below on the left, what if I had both pancakes and waffles for breakfast?  I would really want to tell you that, but I can’t.  Even worse, what if I had yogurt?  Give me an “other” option please!  The question on the right doesn’t provide much detail into the person’s breakfast, but at least the choices are mutually exclusive, and there is an “out” if I haven’t had breakfast or if I’m not sure because I have a terrible memory.  You could argue that the second question below has a flaw in that “completely vegetarian” might be open for interpretation in terms of the definition, but that’s another issue.

5. Choices Should be Exhaustive

In the example above, notice how people over age 40 have no choice.  Be sure to have an option that includes all of the most likely possibilities, then include an “other” choice to capture the rest.

6. Keep Your Questions Simple

Nothing is more frustrating than a complex survey question.  Questions can be complex by being too long, having too many choices, by using confusing language, by forcing respondents to rank too many items, do arithmetic, or by trying to measure too much in one question.  If you are going to have someone rank several items, limit the number of items to three or four.  When constructing your survey language, use common, simple words that have little room for interpretation, and stay away from double negatives.  Here’s are some examples:

In the example below, the first question is way too complex.  It’s too long, it requires imagination and interpretation, and it’s difficult to follow.  Keep your questions simple and free of compound concepts.

7. Allow “N/A” or “Neutral” or “Not Sure” choices

If a question doesn’t apply to a respondent, it is better to understand that than it is to get false data.  Otherwise, users will be frustrated by being coerced into answers they are not comfortable with and you will get inaccurate information.  It is also a good idea to include an “other” choice where appropriate.  Here’s a motto to remember: When in doubt, give the respondent an “out.”

8. Be Consistent

If you use a five point scale for one question, don’t use a 7 point scale for the next ones.

Objectivity: Avoid Question Bias

Avoiding bias is easier said than done, but this is clearly a survey design best practice.  Avoiding bias is a skill that comes with practice and is developed over time.  Here are some tips to help you write objective survey questions.

Avoid Subjective Language

Subjective language can range from the obvious (“Isn’t our service great?”) to the subtle (i.e. “Do you often buy lasagna?”).  Words like “often,” “sometimes,” “usually,” “recently,” “a lot,” etc. are all open for interpretation.  You might think buying lasagna once a month is “often” while someone else might call that “rarely.”  Instead, use objective metrics like “once a week,” “more than once a week,” etc.

Avoid Leading Questions

Leading questions are questions that “lead” or guide the respondent in a certain direction.  It might be something like “Don’t you agree that our tacos are the best?”  These questions are not very effective as collecting real, objective data.  If you want real answers, don’t try to dictate the responses you desire.  

Subjective Language

 

Avoid Double-Barreled Questions

Double-barreled questions are really two questions in one.  In the example below, what if the respondent thinks the taste is excellent, but the appearance is terrible?  If those are the true feelings, it would be hard to select a rating for that question.  Instead, split this into two questions, as shown below.

Randomize Multiple Choices

If you are asking multiple choice questions like the one below, it is a good idea to randomize the choice options so that the same option isn’t always listed first.  Respondents will select choices near the top of the list more often.  By randomizing the order of the list for each respondent, you can feel more confident that your results are genuine, and not simple a result of being the ones listed first.  Many online survey platforms allow for this type of randomization.

Space Your Choices Evenly

If you are using a Likert scale, it is important to space the selections evenly.  Notice how the example below shows uneven spacing?  This could lead the respondent to visually interpret “somewhat satisfied” to be much different than “neutral,” relative to “somewhat unsatisfied” and that is not your intent.

Example of Bad Spacing

Look and Feel: Keep it Clean

This topic can get subjective very quickly, but it is important for your respondents to enjoy the experience of your questionnaire.  When a customer comes into a restaurant, there is an immediate “feel” that he or she experiences.  If the place is cluttered, dirty, has loud music, etc., it leaves a negative imprint.  Conversely, if the place is clean, open, and inviting, a positive impression is left.  Your survey form is an extension of your physical organization.  Take a look at the example below and you get an idea of what I am talking about.

Three other tips regarding the visual and functional design of your survey:

• Show a progress bar so respondent know how far along in the survey they are and how much they have left to go.
• Allow respondents to go back and edit answers as they see fit.
• Allow respondents to save their survey, close it, and go back to it at a later time.

Question Clarity: Be Specific and Keep it Simple

When it comes to writing your questions, try to be as clear as possible so that you can be confident in the meaning of your data.  We’ve already covered some of these points indirectly, but here are some additional tips.

Be Specific 

Make your questions specific.  If asking about customer service, you might ask a general “overall satisfaction” type of question, but also be sure to ask about specific elements of customer service, like speed of service, professionalism, etc.  Another aspect of being specific can be providing actual metrics for respondents, instead of vague words like “often” and “rarely,” but we’ve already covered that.  Finally, be specific so there is no confusion about what you are talking about; in the “spa” example below, respondent might be wondering if “spa” means a day spa resort, an indoor bathtub, or something else.

Use Simple Language

A survey is no place to impress people with your expansive vocabulary.  Use simple, universally understood words that can be understood by everyone in your target pool of respondents.  Avoid fancy words like antidisestablishmentarianism at all costs.

Watch Out for Acronyms, Jargon, and Slang 

As a subject matter expert, you will naturally include acronyms, jargon, and slang into your survey without even thinking about it.  For those who are not as familiar with your subject, these types of words might be unknown and throw them off.  A good way to catch these is to have someone’s grandma take the survey and ask if there were any words she didn’t understand.

Survey Flow: Short, Logical, and Engaging Wins

Be Brief

A best practice is to keep your online survey to 5 minutes or less.

According to SurveyGizmo: “Survey Length – Research has shown that surveys should take 5 minutes or less to complete. Although 6 – 10 minutes is acceptable, longer than 11 minutes will likely result in significant abandonment rates. On average, respondents can complete 5 closed-ended questions per minute and 2 open-ended questions per minute.”

My opinion is that a telephone or in-person survey can go 5-10 minutes, but keep online and mail surveys to five minutes or less.  Our attention spans appear to be dropping in this 160-character world we live in.  If you feel a need to ask more than 5 minutes worth of questions, maybe you haven’t done enough exploratory research to focus  your questionnaire.  If you still have a lot of lengthy questions, consider splitting your survey in half and having two surveys.

Have a Survey Introduction

This goes without saying, but a good introduction does the following:

• Explains who is conducting the survey
• Explains how the data will be used
• Reveals whether it is anonymous, confidential, or neither
• Reveals whether there is an incentive or not
• Sets expectations for the time requirement
• Expresses gratitude for the respondents’ time and insights

Here’s a nice article on the topic of survey intros.

Start Easy

Your first few questions shouldn’t be so difficult or in-depth that they scare your respondents off.  Warm them up with some softballs.

Be Engaging

If you are going to ask someone to focus their time to help you out, the least you can do is make it interesting for them.  You can do that by adding some personality throughout the survey (e.g. “Only three questions left…thanks for sticking with us.”) and by making the subject matter interesting.  Be creative.  Maybe you include a interesting fact at the bottom of each page.  Maybe you include some interesting images.

Use Pages Effectively

Group similar topics on pages of their own.  For example, your demographic questions should be on one page.  Nobody wants to scroll through a mile long webpage.  Don’t cram everything on one super long page just to say “it all fit on one page.”

Ask Important Questions Early

Let’s face it–some people are going to bail halfway through your survey.  Maybe they will get bored or maybe they’ll get a phone call or get distracted.  For that reason, get the most important stuff taken care of early, so you’ll get a more complete data set for those answers.

Use Skip Logic

A good way to write an efficient survey is to not ask every respondent every question.  Some questions will not be relevant for everyone.  “Skip logic” involves asking a qualifying question and then presenting a dynamic set of follow-up questions based on the answer to that qualifying question.  Here’s an example: You first ask “Do you buy bottled beer at least once a year?”  If the respondent answers “yes,” you present two additional questions about the types of bottled beer you care about.  If the answer is “no,” the respondent skips to a new page, never seeing those additional beer bottle questions.  Good skip logic works in the background so that the respondent never knows it is in play.

Put Demographic Questions at the End

I recommend putting your demographic questions (age, gender, etc.) at the end of the survey.  Just like you ease a respondent into the questionnaire with easy questions, you want to ease them out as well.

Include an Open-Ended Question at the End

I like to give respondents an open field at the very end of the survey to provide any additional comments.  This allows room for comments on topics you didn’t ask, clarification of answers, and simply a form to communicate unrelated things like “can you include me in future surveys?,” “good survey, but you are missing the point…the real reason I use your product is this…,” “I’m sorry, I accidentally responded this way, but then I couldn’t go back and change my answer,” and so on.  You’ll be surprised by what you learn if you have an open-ended free form question at the end.

Have a Thank You Page

Upon completion of an online survey, instead of redirecting the respondent to your homepage without a message, display a thank you page with a link to your website.  Thank the respondent for their time, confirm that you’ve received their responses successfully, and provide a phone number or email address for them to contact you.

Have any other survey design best practices?  If so, drop them off in the comments and make everyone smarter.

Additional Resources and Sources

Qualities of a good question

Designing Surveys that Count

Creating Good Questions

[1] SurveyMonkey