Which of the following statements do you most agree with?
- I expect economic recovery in my country to advance strongly in the first half of 2010
- I expect economic recovery in my country to advance strongly in the second half of 2010
- I expect economic recovery in my country not to advance strongly until early 2011
This is a question I saw recently which demonstrates the importance of asking questions correctly and checking/testing your survey thoroughly before launch (and also why compulsory questions should be avoided).
How do you answer this question if you don’t expect economic recovery to advance strongly in the next twelve months, or expect economic recovery to just stay slow until 2012, or beyond?
We regularly tell prospective clients that we can turn-around the survey programming in 5, perhaps 3 days, but most delays occur at the questionnaire draft and review stages. It is important to get the content right – otherwise your results are useless.
If you’re designing a survey Annie at LoveStats has sage advice:
Even if you much prefer DIY, I 100% believe that everyone needs a middle man. Whether that middle-man is a reputable survey company or another person in a completely different department, you absolutely must have fresh eyes.
Dan Wardle
Categories: Survey Design Tips
Tagged: Survey Design, survey tips
When a client (or sometimes prospective client) shares their first draft questionnaire with us it is not un-common to see a series of yes/no questions presented in a grid – an example screenshot is below.
A better way to write (and answer) this type of question is to convert it into a multi-answer question (aka “tick all that apply”). For example, instead of
Do you have any of the following at home?
- Nintendo Wii … Yes / No
- Sony Playstation 3 … Yes / No
- XBox 360 … Yes / No
- PC computer for playing games … Yes / No
Change the question ever so slightly:
Which of the following do you have at home?
Please click all that apply
[ ] Nintendo Wii
[ ] Sony Playstation 3
[ ] XBox 360
[ ] PC computer for playing games
[ ] None of the above
It is important to add a new option “none of the above” (or similarly worded option) in case the respondent needs to answer “no” to all the options offered.
By converting your Yes/No grid to a list of checkboxes the respondent can more quickly read down the list and tick off the relevant answers (compared to read each answer, scan across the screen to where the Yes/No button is, and repeat). It looks less daunting, and with longer lists the action of answering doesn’t feel like a chore.
When it comes to analysis, this result should also be more readily usable. Your online reporting will show a summary table with all the answers together, e.g.
Nintendo Wii – 50%
Sony Playstation 3 – 28%
XBox 360 – 38%
PC computer for playing games – 25%
None of the above – 11%
(these are all made up numbers)
However, if you had asked Yes/No questions the results would be independent of each other, so one result for Nintendo Wii would show:
Yes – 50%
No – 50%
And separately, the result for Sony Playstation 3:
Yes – 28%
No – 72%
Dan Wardle
Categories: Survey Design Tips
Tagged: Survey Design, survey tips
I started collecting examples of emails about surveys last year, although it has been surprising how very few actually stand out from the bog standard. Two messages that caught John or my attention last week were these – all because of the subject line.
Subject: Only 30 mins till lunchtime!
I definitely never actually opted-in to this mailing list but it’s a very different approach to the usual “Customer Satisfaction Survey” headline. Although the survey was more market research than for customer feedback, it might work well for employee surveys.
This message does assume that the email will land in inboxes before lunch. If you are firing out a few thousand (or less) – just because the emails have “gone” don’t mean they have all been delivered. Email does not guarantee time-critical communication.
Subject: Can you help us out this Friday?
This was the subject for PostcodeAnywhere’s annual survey. Again it’s different from the usual affair and appealing to our charitable feeling on the day.
Does the message mean that they don’t want to hear your feedback on Monday though? Possibly – this is a common problem for choosing a date and how to communicate the survey’s closing date.
Bonus points to the creator for making the version without images look so clean (left hand screenshot).
One other message leapt out in my inbox.
Thank you for taking part in our survey and prize draw
I’m not a customer of Zen Software yet, but receiving a thank you by email is rare. I am more likely to buy the software (I probably would have anyway, but this has only helped strengthen my view).
Dan Wardle
Categories: The email side of surveys
Tagged: email survey
12, January 2010 · 1 Comment
There are two blog posts published just before everyone went back to work after New Year’s (and all that snow) which I thought are worth sharing.
The first is Panel Crazy – an observation about the number of errors present in surveys. Sheila Wilson wrote
Clients know that with the technology we now have, it is possible to script a project and complete the fieldwork online in a matter of hours. But perhaps they don’t know about the amount of thought, effort and creativity that is required for decent questionnaire design and in particular thorough quality control.
If you are planning a new customer feedback survey for the new year – give yourself plenty of time to
- Write the first draft
- To review and make changes when you have seen the online survey for the first time
- Test (Q.A.) – make sure all paths through the survey have been tested!
The second blog post was Optimise rather than maximise response rates – which reminds readers that sending reminder emails and then even more reminder emails could have other effects on customers. Most interestingly Jeffrey Henning shared this nugget:
Pew Research Center found that a response rate of 25% did not materially change the quality of survey estimates when compared to the same study with a 50% response rate.
As I’ve said before you don’t need massive numbers of responses to a survey.
Dan Wardle
Categories: Online Surveys · Survey Design Tips
Tagged: market research, online survey design, Questionnaire Design
30, December 2009 · 1 Comment
I was inspired to blog this quick post after a friend forwarded me a promotional email they had received which offered expert insights into how to improve your survey’s response rates by one hundred and something percent.
Is Surveylab missing anything I wondered? So I downloaded the PDF and was pleased to see the answer was no (according to this particular list of insights), but then there was one tip that I strongly disagree with.
Adding a prize draw to your survey does not necessarily mean a higher response rate. An opportunity for the customer to give feedback to improve the service/product/experience, or an employee being able to make their voice heard anonymously will be many respondents’ motivation, and sharing results and findings is enough incentive.
If you do use a prize draw, watch out for “professional” survey takers or users clicking through from competitions websites (especially if your survey is promoted on a public website) because these respondents could skew results wildly. Tip: find out if you can track the referring URL in your survey software or get your webmaster to check the website logs to monitor how respondents arrive at your survey, and also monitor the daily click-thru rate to the survey – a significant and unexplained spike in responses might mean your survey link has been posted on a forum.
Incentives can work, but their impact might not be in the intended way. If you are planning a customer or employee survey in the new year, our default suggestion would be “no incentive”. If the survey length is getting out of hand such that you have decided to offer a prize draw to win an iPod then rethink the prize and more importantly review the questionnaire – it’s probably too long.
Have a great 2010!
Dan Wardle
Categories: Online Surveys
Tagged: Survey Design, survey response rates
A very good tip was posted on Econsultancy the other week that is so easily overlooked.
Zagat made a critical and not-uncommon error in crafting their survey. All possible answers were ones they wanted to hear: namely, that respondants are and will be willing to pay for their service. Once I was, but I no longer am. [my emphasis]
This allows me to promote an old blog post – Survey Design Tips: Hallway Testing – find someone not involved in writing the questionnaire and ask them to walk through the survey. Hopefully it will help to catch any oversights before the survey goes live.
Dan Wardle
Categories: Survey Design Tips
Tagged: Survey Design Tips
My family was booked on Eurostar on that Saturday for a short weekend break to EuroDisney, that day four Eurostar trains broke down in the tunnel. Luckily (compared to many others) we were just getting the kids’ breakfast and finishing packing when we first heard of the unfolding travel problems – at least we weren’t trapped in traffic, at a terminal somewhere in Kent, or worse. After the initial disappointment, we started the horrible process of trying to get confirmation that our trip really wasn’t going ahead.
My personal experience of Eurostar’s handling of the situation can basically be summarised as frustration with lack of information from Eurostar. We were left to speculate – should we wait and see if they will run the Disney train later that day, perhaps they will put us on a train on Sunday? Who decides what is meant by “essential”, can we just assume our trip is definitely cancelled? No good phoning the call centre though because all we could get was
“Sorry, this service is currently unavailable.”
Not really helpful… BTW, if you call before 9AM you hear a jolly message that the office is closed but they are open at 9AM. A tailored message could have answered many customers’ queries and helped to reduce the inbound call queues.
Watching the news and following the blogosphere it seems everyone faced the same problem, whether trying to find out information via the internet or in-person stranded at a Eurostar terminal: CUSTOMERS WANT INFORMATION (and reliable information at that).
Eurostar could have had TV and radio news working for them, instead of just letting the BBC/Sky/et al run round highlighting victims’ personal stories. It was only at 3PM on the Saturday that the Eurostar website was updated to state categorically no trains would be running that day, and that was all it said. All the update achieved was raise more questions.
For Eurostar, they have plenty of issues to examine – what, who, how, why, why, why… From the reports so far it looks like their own internal communications broke down on more than one level (being unable to communicate inside the tunnels sounds like a major problem in itself) so quite how they could expect to get information to the tens of thousands of customers expecting to travel over the busy holiday period no-one can answer until Eurostar know the answers themselves.
While Eurostar’s disaster is a high profile, major problem affecting so many customers in one incident the lessons learned can be applied to the individual issues/enquiries/cases that face all customer services staff everyday. Sometimes the problem can’t be fixed quickly or easily, and that’s when good communication is essential.
Dan Wardle
Categories: Customer Care
Tagged: complaint handling, Customer Care, Eurostar
One Saturday lunchtime two weeks ago I logged onto one of our servers to send the email invites for two customer surveys that we field regularly. Some time ago the Messaging Times posed the question whether sending emails at the weekend is more effective than during the week – which got us thinking.
We have looked at the best time to email (survey invites) in the past and generally concluded that the day of the week doesn’t really make a big difference. Other factors have a much bigger influence – e.g. amount of time passed since the last customer contact.
Our conclusion for the 2 surveys we trialled is sending the email invites on a Saturday definitely doesn’t harm the response rate. When we email a survey we typically see click-thru’s start coming in after only a couple of minutes (assuming at least a few hundred emails being sent) but I was still surprised to see the same effect on a Saturday. The response rate at the end of the first day was 2.5% lower for Saturday than for those surveys emailed on the Monday, but the Saturday mailings also saw a small boost on the Monday – presumably recipients who aren’t checking their mail at the weekend catching up on their inbox.
|
“Normal”
Mailed on Monday |
“Test”
Mailed on Saturday |
| End of Day 1 |
16.60% |
14.00% |
| Day 2 |
23.70% |
21.30% |
| Day 3 |
26.00% |
26.40% |
| Day 4 |
27.00% |
27.00% |
| Day 5 |
27.60% |
27.80% |
| Day 6 |
28.20% |
28.40% |
| Day 7 |
28.40% |
28.70% |
Table 1: First survey’s click-thru rates (cumulative %) after 7 days
Whenever possible we send a reminder/follow-up email to the non-responders – so Saturdays adds some flexibility to our resources schedule if it were required (assuming that the survey doesn’t need results within less than 4 or 5 days). * Your circumstances and experience might be different.
|
“Normal”
Mailed on Monday |
“Test”
Mailed on Saturday |
| End of Day 1 |
20.20% |
17.20% |
| Day 2 |
26.30% |
25.20% |
| Day 3 |
28.70% |
30.00% |
| Day 4 |
30.00% |
31.50% |
| Day 5 |
30.70% |
31.90% |
| Day 6 |
31.40% |
32.40% |
| Day 7 |
31.70% |
32.50% |
Table 2: Second survey’s click-thru rates (cumulative %) after 7 days
Dan Wardle
Categories: Online Surveys · The email side of surveys
Tagged: email surveys, survey response rates
Another look at examples of emails about customer surveys with two common but very different approaches…
Short and Sweet
The first example is a transactional email after contacting APC for support on a new UPS (uninterruptable power supply – excellent customer service by the way and definitely happy to recommend APC to all as a result). A couple of days after my support case was resolved I received an email with the subject We Need Your Feedback.
This email probably serves APC very well and it’s good because it’s short and to the point – how many customers actually read every word?
The only two issues I have is the email was personalised with the support ticket number so why no further personalisation (a greeting, subject/problem area from original support ticket)? The second is how very automated the message felt – signed off from the ‘Team’ rather than an individual (yet my correspondence was with the same rep.) and the “please do not reply to this email” message is a common bugbear amongst bloggers. It says “we’re interested in what you have to say, but only to the questions we ask“.
Today’s Takeaway Action: If you have a system for sending automated survey emails when support tickets are closed – review that message.
Too Much Information
I bookmarked this post earlier this month – Bank of America – This is NOT spam – showing a (poor) example of a survey invite for Bank of America. A common problem is that the organisation feels the need to give lots of background to prove their credentials and the value of the research. For a bank this is much bigger because how do you prove you are who you say you are in an email?
Getting the Email Right
Contrast Bank of America’s message with Marks and Spencer’s Talk Shopping with Marks and Spencer email (screenshot left – click to zoom in).
All the supporting information – who we are, how did you get my details, and extra information is at the bottom of the email. Ok, this invitation is to join an online panel so it isn’t your regular “we need your customer feedback” type message – but it’s a good example of an email about a survey where more details are important/helpful/deemed necessary.
The key point and the call to action is at the top of the email, and all the content is broken into smaller, manageable chunks rather than chucked together in haste.
Dan Wardle
Categories: Online Surveys · The email side of surveys
Tagged: customer survey emails, email marketing
