Survey Design Best Practices: How to Write a Good Questionnaire

When it comes to survey design best practices, there are six key considerations, including relevance, objectivity, clarity, the look & feel, your question structure, and the survey flow.  This article provides some tips and tricks related to each of these components, helping you construct effective survey questionnaires.

Relevance: The Secret is to Have a Plan

The first step in effective survey design is to have a plan.  Before you start crafting a questionnaire, ask yourself the following five questions:

1. What business decisions am I trying to inform?
2. If I knew __?__, I would be more prepared to make these important business decisions.
3. What am I trying to measure?  Attitudes? Behavior? Intentions? Perceptions?
4. Who is my audience?
5. What kind of statistics do I want to come out of this project (e.g. descriptive vs. inferential statistics)?  What will my analysis look like?

Now, as you craft your questions, ask yourself if the data provided will address those objectives.  After I write a question, I like to imagine that I instantly have the raw data in my hands, then I ask myself: “Ok, now what?  What can I do with this information?  Does this give me the information I need?”  This simple exercise will help you to construct questions that really matter and will yield actionable data; you will avoid filling your survey with a bunch of “nice to know,” but non-essential questions.

Don’t forget to test your survey!
A key part of the planning process is testing your survey.  Find some people–including someone’s grandparent–to take your survey without any coaching from you, and then gather their feedback.  Ask them about their general impression.  You should also ask some specific questions:

1. Was the survey engaging?
2. How long did it take you to complete?
3. Was the question flow logical?
4. Were there any confusing questions?
5. Were there any areas of frustration?

When you test the survey with several people, DO look at the data that comes back.  Get this raw data into your spreadsheet or data analysis program and start the process of analyzing your data.  If there are any issues with data structure, improper question type or scale types, they will likely present themselves here and you can fix them before you launch the real deal.

Question Structure – Types of Questions

When it comes time to start writing questions, you have many types of questions to choose from.  To keep it simple, we can categorize them all into two groups: structured (or close-ended questions) and unstructured (or open-ended questions).  An unstructured, open-ended question is something like this.

Some general tips on question structure:

1. Limit your use of open-ended questions

These are more taxing on respondents than structured questions and will lead to survey fatigue if overused.  If you have an urge to use a lot of open-ended questions, you probably haven’t done enough exploratory research.  You might benefit by conducting some secondary research, interviews, or focus groups before you embark on a survey.

2. Use appropriate scales 

Let’s keep this simple without getting into “nominal” and “ratio” scale type jargon–let’s say you want to know about the age of your respondents.  You have several ways to ask this question, each of which will produce a different level of specificity.  If you need to report an average age, for example, you would want to use the question on the right vs. the question on the left (below).  Since the question on the right provides more detail, why do surveys always seem to ask the question in the “range” format?  Well, age can be a sensitive issue for some people (especially old people), so people are often more comfortable giving a range rather than an absolute value.  Keep this in mind when crafting your questions and try to strike a balance that meets your needs.

3. Don’t ask unnecessary questions  

Redundant questions really annoy people, and they should!  Here is an example: At the doctors office last week, I was asked my date of birth on a form.  The very next question asked for my age.  Don’t do this!  If you know someone’s date of birth, you can easily calculate an age based on the current date.  If you ask for a zip code, don’t ask about my city and state.  Think about what additional data you can extract from your questions before asking a unnecessary questions and adding to the length.

4. Multiple choice questions need to be mutually exclusive

It is extremely frustrating when multiple answers apply, but respondents are only allowed to select one.  In the example below on the left, what if I had both pancakes and waffles for breakfast?  I would really want to tell you that, but I can’t.  Even worse, what if I had yogurt?  Give me an “other” option please!  The question on the right doesn’t provide much detail into the person’s breakfast, but at least the choices are mutually exclusive, and there is an “out” if I haven’t had breakfast or if I’m not sure because I have a terrible memory.  You could argue that the second question below has a flaw in that “completely vegetarian” might be open for interpretation in terms of the definition, but that’s another issue.

 

5. Choices Should be Exhaustive

In the example above, notice how people over age 40 have no choice.  Be sure to have an option that includes all of the most likely possibilities, then include an “other” choice to capture the rest.

6. Keep Your Questions Simple

Nothing is more frustrating than a complex survey question.  Questions can be complex by being too long, having too many choices, by using confusing language, by forcing respondents to rank too many items, do arithmetic, or by trying to measure too much in one question.  If you are going to have someone rank several items, limit the number of items to three or four.  When constructing your survey language, use common, simple words that have little room for interpretation, and stay away from double negatives.  Here’s are some examples:

In the example below, the first question is way too complex.  It’s too long, it requires imagination and interpretation, and it’s difficult to follow.  Keep your questions simple and free of compound concepts.

7. Allow “N/A” or “Neutral” or “Not Sure” choices

If a question doesn’t apply to a respondent, it is better to understand that than it is to get false data.  Otherwise, users will be frustrated by being coerced into answers they are not comfortable with and you will get inaccurate information.  It is also a good idea to include an “other” choice where appropriate.  Here’s a motto to remember: When in doubt, give the respondent an “out.”

8. Be Consistent

If you use a five point scale for one question, don’t use a 7 point scale for the next ones.

Objectivity: Avoid Question Bias

Avoiding bias is easier said than done, but this is clearly a survey design best practice.  Avoiding bias is a skill that comes with practice and is developed over time.  Here are some tips to help you write objective survey questions.

Avoid Subjective Language

Subjective language can range from the obvious (“Isn’t our service great?”) to the subtle (i.e. “Do you often buy lasagna?”).  Words like “often,” “sometimes,” “usually,” “recently,” “a lot,” etc. are all open for interpretation.  You might think buying lasagna once a month is “often” while someone else might call that “rarely.”  Instead, use objective metrics like “once a week,” “more than once a week,” etc.

Avoid Leading Questions

Leading questions are questions that “lead” or guide the respondent in a certain direction.  It might be something like “Don’t you agree that our tacos are the best?”  These questions are not very effective as collecting real, objective data.  If you want real answers, don’t try to dictate the responses you desire.