There are two blog posts published just before everyone went back to work after New Year’s (and all that snow) which I thought are worth sharing.
The first is Panel Crazy – an observation about the number of errors present in surveys. Sheila Wilson wrote
Clients know that with the technology we now have, it is possible to script a project and complete the fieldwork online in a matter of hours. But perhaps they don’t know about the amount of thought, effort and creativity that is required for decent questionnaire design and in particular thorough quality control.
If you are planning a new customer feedback survey for the new year – give yourself plenty of time to
- Write the first draft
- To review and make changes when you have seen the online survey for the first time
- Test (Q.A.) – make sure all paths through the survey have been tested!
The second blog post was Optimise rather than maximise response rates – which reminds readers that sending reminder emails and then even more reminder emails could have other effects on customers. Most interestingly Jeffrey Henning shared this nugget:
Pew Research Center found that a response rate of 25% did not materially change the quality of survey estimates when compared to the same study with a 50% response rate.
As I’ve said before you don’t need massive numbers of responses to a survey.
Dan Wardle
Categories: Online Surveys · Survey Design Tips
Tagged: market research, online survey design, Questionnaire Design
30, December 2009 · 1 Comment
I was inspired to blog this quick post after a friend forwarded me a promotional email they had received which offered expert insights into how to improve your survey’s response rates by one hundred and something percent.
Is Surveylab missing anything I wondered? So I downloaded the PDF and was pleased to see the answer was no (according to this particular list of insights), but then there was one tip that I strongly disagree with.
Adding a prize draw to your survey does not necessarily mean a higher response rate. An opportunity for the customer to give feedback to improve the service/product/experience, or an employee being able to make their voice heard anonymously will be many respondents’ motivation, and sharing results and findings is enough incentive.
If you do use a prize draw, watch out for “professional” survey takers or users clicking through from competitions websites (especially if your survey is promoted on a public website) because these respondents could skew results wildly. Tip: find out if you can track the referring URL in your survey software or get your webmaster to check the website logs to monitor how respondents arrive at your survey, and also monitor the daily click-thru rate to the survey – a significant and unexplained spike in responses might mean your survey link has been posted on a forum.
Incentives can work, but their impact might not be in the intended way. If you are planning a customer or employee survey in the new year, our default suggestion would be “no incentive”. If the survey length is getting out of hand such that you have decided to offer a prize draw to win an iPod then rethink the prize and more importantly review the questionnaire – it’s probably too long.
Have a great 2010!
Dan Wardle
Categories: Online Surveys
Tagged: Survey Design, survey response rates
29, December 2009 · 1 Comment
A very good tip was posted on Econsultancy the other week that is so easily overlooked.
Zagat made a critical and not-uncommon error in crafting their survey. All possible answers were ones they wanted to hear: namely, that respondants are and will be willing to pay for their service. Once I was, but I no longer am. [my emphasis]
This allows me to promote an old blog post – Survey Design Tips: Hallway Testing – find someone not involved in writing the questionnaire and ask them to walk through the survey. Hopefully it will help to catch any oversights before the survey goes live.
Dan Wardle
Categories: Survey Design Tips
Tagged: Survey Design Tips
My family was booked on Eurostar on that Saturday for a short weekend break to EuroDisney, that day four Eurostar trains broke down in the tunnel. Luckily (compared to many others) we were just getting the kids’ breakfast and finishing packing when we first heard of the unfolding travel problems – at least we weren’t trapped in traffic, at a terminal somewhere in Kent, or worse. After the initial disappointment, we started the horrible process of trying to get confirmation that our trip really wasn’t going ahead.
My personal experience of Eurostar’s handling of the situation can basically be summarised as frustration with lack of information from Eurostar. We were left to speculate – should we wait and see if they will run the Disney train later that day, perhaps they will put us on a train on Sunday? Who decides what is meant by “essential”, can we just assume our trip is definitely cancelled? No good phoning the call centre though because all we could get was
“Sorry, this service is currently unavailable.”
Not really helpful… BTW, if you call before 9AM you hear a jolly message that the office is closed but they are open at 9AM. A tailored message could have answered many customers’ queries and helped to reduce the inbound call queues.
Watching the news and following the blogosphere it seems everyone faced the same problem, whether trying to find out information via the internet or in-person stranded at a Eurostar terminal: CUSTOMERS WANT INFORMATION (and reliable information at that).
Eurostar could have had TV and radio news working for them, instead of just letting the BBC/Sky/et al run round highlighting victims’ personal stories. It was only at 3PM on the Saturday that the Eurostar website was updated to state categorically no trains would be running that day, and that was all it said. All the update achieved was raise more questions.
For Eurostar, they have plenty of issues to examine – what, who, how, why, why, why… From the reports so far it looks like their own internal communications broke down on more than one level (being unable to communicate inside the tunnels sounds like a major problem in itself) so quite how they could expect to get information to the tens of thousands of customers expecting to travel over the busy holiday period no-one can answer until Eurostar know the answers themselves.
While Eurostar’s disaster is a high profile, major problem affecting so many customers in one incident the lessons learned can be applied to the individual issues/enquiries/cases that face all customer services staff everyday. Sometimes the problem can’t be fixed quickly or easily, and that’s when good communication is essential.
Dan Wardle
Categories: Customer Care
Tagged: complaint handling, Customer Care, Eurostar
An experienced web developer (Iftikhar) joined Surveylab last month, and we are now seeking a Junior Systems Support/Developer to join our team.
We are also looking at relocating the technical team to London SE1 (around Waterloo or London Bridge). If you are interested and have a general technical curiosity then please get in touch!
Categories: Surveylab
Tagged: web developer jobs
One Saturday lunchtime two weeks ago I logged onto one of our servers to send the email invites for two customer surveys that we field regularly. Some time ago the Messaging Times posed the question whether sending emails at the weekend is more effective than during the week – which got us thinking.
We have looked at the best time to email (survey invites) in the past and generally concluded that the day of the week doesn’t really make a big difference. Other factors have a much bigger influence – e.g. amount of time passed since the last customer contact.
Our conclusion for the 2 surveys we trialled is sending the email invites on a Saturday definitely doesn’t harm the response rate. When we email a survey we typically see click-thru’s start coming in after only a couple of minutes (assuming at least a few hundred emails being sent) but I was still surprised to see the same effect on a Saturday. The response rate at the end of the first day was 2.5% lower for Saturday than for those surveys emailed on the Monday, but the Saturday mailings also saw a small boost on the Monday – presumably recipients who aren’t checking their mail at the weekend catching up on their inbox.
|
“Normal”
Mailed on Monday |
“Test”
Mailed on Saturday |
| End of Day 1 |
16.60% |
14.00% |
| Day 2 |
23.70% |
21.30% |
| Day 3 |
26.00% |
26.40% |
| Day 4 |
27.00% |
27.00% |
| Day 5 |
27.60% |
27.80% |
| Day 6 |
28.20% |
28.40% |
| Day 7 |
28.40% |
28.70% |
Table 1: First survey’s click-thru rates (cumulative %) after 7 days
Whenever possible we send a reminder/follow-up email to the non-responders – so Saturdays adds some flexibility to our resources schedule if it were required (assuming that the survey doesn’t need results within less than 4 or 5 days). * Your circumstances and experience might be different.
|
“Normal”
Mailed on Monday |
“Test”
Mailed on Saturday |
| End of Day 1 |
20.20% |
17.20% |
| Day 2 |
26.30% |
25.20% |
| Day 3 |
28.70% |
30.00% |
| Day 4 |
30.00% |
31.50% |
| Day 5 |
30.70% |
31.90% |
| Day 6 |
31.40% |
32.40% |
| Day 7 |
31.70% |
32.50% |
Table 2: Second survey’s click-thru rates (cumulative %) after 7 days
Dan Wardle
Categories: Online Surveys · The email side of surveys
Tagged: email surveys, survey response rates
Another look at examples of emails about customer surveys with two common but very different approaches…
Short and Sweet
The first example is a transactional email after contacting APC for support on a new UPS (uninterruptable power supply – excellent customer service by the way and definitely happy to recommend APC to all as a result). A couple of days after my support case was resolved I received an email with the subject We Need Your Feedback.
This email probably serves APC very well and it’s good because it’s short and to the point – how many customers actually read every word?
The only two issues I have is the email was personalised with the support ticket number so why no further personalisation (a greeting, subject/problem area from original support ticket)? The second is how very automated the message felt – signed off from the ‘Team’ rather than an individual (yet my correspondence was with the same rep.) and the “please do not reply to this email” message is a common bugbear amongst bloggers. It says “we’re interested in what you have to say, but only to the questions we ask“.
Today’s Takeaway Action: If you have a system for sending automated survey emails when support tickets are closed – review that message.
Too Much Information
I bookmarked this post earlier this month – Bank of America – This is NOT spam – showing a (poor) example of a survey invite for Bank of America. A common problem is that the organisation feels the need to give lots of background to prove their credentials and the value of the research. For a bank this is much bigger because how do you prove you are who you say you are in an email?
Getting the Email Right
Contrast Bank of America’s message with Marks and Spencer’s Talk Shopping with Marks and Spencer email (screenshot left – click to zoom in).
All the supporting information – who we are, how did you get my details, and extra information is at the bottom of the email. Ok, this invitation is to join an online panel so it isn’t your regular “we need your customer feedback” type message – but it’s a good example of an email about a survey where more details are important/helpful/deemed necessary.
The key point and the call to action is at the top of the email, and all the content is broken into smaller, manageable chunks rather than chucked together in haste.
Dan Wardle
Categories: Online Surveys · The email side of surveys
Tagged: customer survey emails, email marketing
A popular type of question to ask in a survey is the ranking question, e.g.
Please rank each of the following items in order of importance
From the respondent’s perspective, this question involves more effort to answer than others because of the need to compare each item in the list against every other item in the list, and also requires more time to fill-in as the list gets longer.
In many cases though you don’t need to ask for every item in the list to be ranked. Asking for the top 3 (or even 2) can be enough to arrive at the same result.
The data table below is from real survey data on a question that ranked 8 items (although only showing the top 3 positions here):
| Item |
% Ranked
1st |
% Ranked
2nd |
% Ranked
3rd |
| Red |
26 |
19 |
14 |
| Blue |
28 |
20 |
15 |
| Pink |
3 |
13 |
17 |
| White |
20 |
17 |
13 |
| Black |
8 |
10 |
10 |
| Brown |
3 |
5 |
9 |
| Yellow |
7 |
10 |
12 |
| Green |
6 |
7 |
9 |
With this survey data the ranking is identical whether we got the respondent to rank all 8 items or just their top 3 (and if we only used the top 2 the results are the same except for positions 4 and 6 – btw the results above are blue first, red, white, then a big drop to pink, yellow, black, green and brown last).
The point though is that your ranking question data isn’t going to be the only factor in any business decision (other survey data should contribute for a start). Are you looking to identify the most important 5 or 6 factors (from a list of 26), or categorise the items into “high importance”, “medium” and “low”? When designing the survey – bear in mind how the data will be analysed – if you don’t need an ordered list of 26 priorities then don’t ask your customers to work out theirs!
Dan Wardle
Categories: Customer Surveys / Loyalty · Survey Design Tips
Tagged: ranking question, Survey Design Tips
I’ve been meaning to share this post on Top Right Corner’s blog that highlights two common mistakes when asking an open-ended verbatim type question after the Net Promoter Score question (e.g. would you recommend [company name] to your friends and family).
Asking the Right Question to get to the Key Drivers of NPS or Customer Satisfaction gives good practical advice how to avoid two possible effects that can turn up in customers’ verbatim comments.
Always remember that the wording of a question can have a big impact on how it is answered.
Dan Wardle
Categories: Net Promoter · Survey Design Tips
Tagged: Net Promoter, Survey Design Tips
