Learning patterns/Framing survey questions
What problem does this solve?
[edit]When people misunderstand survey questions, they often don't answer them the way the survey creator intended. By the time the survey creator realizes this, it's usually too late.
Most people fill out surveys quickly, and don't always read every question carefully. The survey creator usually has just one chance to ask these questions—most people are not willing to fill out the same survey twice! So it's important to ask the right questions, and to ask them in the right way.
The "right questions" and the "right way" may be different for each survey, but there are some good general principles that you can follow. Above all, try to make the survey as short as possible: that way, people are more likely to concentrate on giving reliable responses.
What's the solution?
[edit]When asking multiple-choice questions, provide choices for all of the major types of answers you expect to receive, at the right level of specificity. Ask open-ended questions in a way that encourages people to elaborate rather than give one-word answers.
Examples
[edit]Asking multiple-choice questions
[edit]Take a look at an example of a multiple-choice question, and think about how it could be better (don't type!). Then click on the hint and think again. Then click on the possible solution, and so on.
- Example 1
- How active are you on Wikipedia?
- 1–10 edits per month
- 11–20 edits per month
- 21–30 edits per month
- 31–40 edits per month
- 41–50 edits per month
- 51–60 edits per month
- More than 60 edits per month
- Perhaps the wording of "How active are you on Wikipedia?" could relate more clearly to the choices you give them.
- Will people have to think too hard about whether they make 30–40 or 41–50 edits per month? And will it really be useful for you to know this fine distinction?
- Can you identify where repeated words could be removed to make it simpler to read?
| Possible solution |
|---|
|
| Comments |
|---|
But we've forgotten an important option. Can you think what it is? And maybe we could put a note at the bottom to make it easier to relate their daily editing experience to whole months. What's a short, simple wording for that? |
| Better solution |
|---|
Note: 100 edits is a little more than three per day, on average. But wait—there's one more thing. Let's not make people feel inadequate if they've never edited Wikipedia. For this purpose, we could soften the wording and do something else, too. Can you think of these two things? |
| Even better |
|---|
Note: 100 edits is a little more than three per day, on average. OK, this is better: the "no edits" option is now last, not first, so they won't feel they're on the low end of a scoresheet; and the wording is now softer because it refers to a larger range of potential personal roles to explain non-editing ("regularly"). |
| Points to remember |
|---|
|
Now have a rest and come back for the next set of examples. We need you to be fresh!
Asking open-ended questions
[edit]Open-ended questions can draw out more detailed, personal responses from people in their own words. But people may skip over these because they don't have anything to add or they don't feel like writing a lot.
But you can increase both the number and the quality of their open-ended responses by phrasing the questions in ways that encourage storytelling.
Again, let's take an example and think how it could be improved:
- Example 2
- Has the kind of editing you do changed since you first joined Wikipedia?
| Hint |
|---|
| This is yes–no question, so strictly speaking they could answer with one word. That wouldn't be useful to you, and if it's all you want to know, make it a multiple-choice question.
By adding just one word to the question, you're likely to get more detailed responses. What word? |
| Possible solution |
|---|
Even so, it's still a hard question, and if coffee and cookies are around the corner after a seminar, they may skip over this with a short non-useful response. Let's think of what you could add to the question to encourage them to give more. |
| Even better |
|---|
Giving examples like this does bring the danger of funneling them down the pathway of your suggestions. So think carefully before you provide examples—often it's better to give no examples at all. |
- Example 3
- What's the most difficult thing about editing Wikipedia?
This is asking people to come up with something quite abstract; but people generally find it easier to remember concrete things than abstract things, and specific examples can produce especially useful data for you.
And does "difficult" mean a negative experience, or one that was challenging but ultimately positive? Asking someone how they were affected by an experience (how they felt) will usually lead to more accurate and useful feedback.
So, think how you might change the question to make it easy for them to recall something specific in their experience? We've come up with three ways, and our possible solution is only three words longer, too!
| Possible solution |
|---|
Here, we've gone for three qualities that are more likely to be easily recalled:
There's a technique to get specific abstract responses to open-ended quesitons, but we recommend that you use it only once in a survey, if at all: |
| Another technique |
|---|
This can be reduced to just the best or the worst, depending on what data you want. It normally won't get you the how and the why. |
| Points to remember |
|---|
|
Endorsements
[edit]See also
[edit]Related patterns
[edit]Other resources
[edit]other resources (links to surveys, articles about survey design, handy tools) related to this pattern
- Survey Questions 101: Do you make any of these 7 question writing mistakes? (Qualtrics.com)
- How to ask survey questions (Wikihow.com)
- Guide to writing survey questions (Minnesota State University)