Tips for better survey design #1: Hallway Testing

Each week I see new questionnaires – ours which are on behalf of a client and, not surprisingly, other people’s surveys stumbled upon which nearly always grab my attention (although my patience is quickly tested – I won’t plod through a survey that’s not keeping my interest or that takes more than a few minutes). At Surveylab, we have a process for  Quality Assurance (QA) to make sure we don’t make mistakes like accidentally launching a survey that doesn’t capture the last page’s results, but QA only goes so far. Often, I find myself looking at a survey (hopefully not one of ours) and am left wondering

Did the person who wrote this survey actually try completing it for themselves?

It’s not that the questionnaire is rubbish, it just needs improvement… but the final questionnaire doesn’t get read and critiqued by someone trying to actually answer the questions. So the end result is that analysing the results becomes a real challenge and takes far longer to make sense of.

You might think your questionnaire achieves your objective, and yes, looking at individual questions, you might feel pretty pleased with yourself – this questionnaire seems to cover all your bases. However, what if the respondent who has clicked through to your survey doesn’t fit the “profile” of your intended respondent 100%? Have you made an assumption about the respondent that means they are unable to answer some questions or sections of the survey? (If you are enforcing compulsory questions, then it pays to make sure respondents can answer them! Or you might want to use a skip pattern (or branching logic) to ask only relevant questions.)

Hallway testing was a term I first came across years ago on Joelonsoftware.com – something I already did without realising.

A hallway usability test is where you grab the next person that passes by in the hallway and force them to try to use the code you just wrote. If you do this to five people, you will learn 95% of what there is to learn about usability problems in your code.

The same concept applies to survey design. Even if your draft questionnaire is still in Microsoft Word and contains manual instructions such as “if you answered ‘No’, please skip to question 10″ – use hallway testing to ask your colleagues and friends to complete the survey and give you feedback.

Using hallway testing you should be able to find

• Any instructions or questions that are difficult to understand
• Use of language or terms that might not be understood outside your department/company
• Relevant answer options that are missing from a question
• If you have overlooked a skip pattern (i.e. assumed all respondents can answer the question)
• If the survey takes too long
• If the experience was positive/negative

In our experience, hallway testing is usually better than design by committee, and a LOT LESS PAINFUL. Put yourself in the respondent’s shoes – who ever that might be – and repeat for a few different scenarios. For example, play at being the fanatical/loyal customer, the price-conscious customer, the overseas customer, and so on to check that your questionnaire collects good data from all the different types of people who put their time and effort into answering your survey.

The respondents will thank you for it, and you will get better quality data and a higher completion rate.

If you would like some expert help designing your online survey, do get in touch – this is what Surveylab prides itself  on!