Asking the right questions

A very good tip was posted on Econsultancy the other week that is so easily overlooked.

Zagat made a critical and not-uncommon error in crafting their survey. All possible answers were ones they wanted to hear: namely, that respondants are and will be willing to pay for their service. Once I was, but I no longer am. [my emphasis]

This allows me to promote an old blog post – Survey Design Tips: Hallway Testing – find someone not involved in writing the questionnaire and ask them to walk through the survey. Hopefully it will help to catch any oversights before the survey goes live.

Dan Wardle

A lesson to learn from Eurostar: keep customers informed

My family was booked on Eurostar on that Saturday for a short weekend break to EuroDisney, that day four Eurostar trains broke down in the tunnel. Luckily (compared to many others) we were just getting the kids’ breakfast and finishing packing when we first heard of the unfolding travel problems – at least we weren’t trapped in traffic, at a terminal somewhere in Kent, or worse. After the initial disappointment, we started the horrible process of trying to get confirmation that our trip really wasn’t going ahead.

My personal experience of Eurostar’s handling of the situation can basically be summarised as frustration with lack of information from Eurostar. We were left to speculate – should we wait and see if they will run the Disney train later that day, perhaps they will put us on a train on Sunday? Who decides what is meant by “essential”, can we just assume our trip is definitely cancelled? No good phoning the call centre though because all we could get was

“Sorry, this service is currently unavailable.”

Not really helpful… BTW, if you call before 9AM you hear a jolly message that the office is closed but they are open at 9AM. A tailored message could have answered many customers’ queries and helped to reduce the inbound call queues.

Watching the news and following the blogosphere it seems everyone faced the same problem, whether trying to find out information via the internet or in-person stranded at a Eurostar terminal: CUSTOMERS WANT INFORMATION (and reliable information at that).

Eurostar could have had TV and radio news working for them, instead of just letting the BBC/Sky/et al run round highlighting victims’ personal stories. It was only at 3PM on the Saturday that the Eurostar website was updated to state categorically no trains would be running that day, and that was all it said. All the update achieved was raise more questions.

For Eurostar, they have plenty of issues to examine – what, who, how, why, why, why… From the reports so far it looks like their own internal communications broke down on more than one level (being unable to communicate inside the tunnels sounds like a major problem in itself) so quite how they could expect to get information to the tens of thousands of customers expecting to travel over the busy holiday period no-one can answer until Eurostar know the answers themselves.

While Eurostar’s disaster is a high profile, major problem affecting so many customers in one incident the lessons learned can be applied to the individual issues/enquiries/cases that  face all customer services staff everyday. Sometimes the problem can’t be fixed quickly or easily, and that’s when good communication is essential.

Dan Wardle

The perfect balanced sample

Classic video clip showed up on Twitter this week … perhaps Opinion Polls: Getting the results you want should be compulsory viewing for market researchers?

Sending emails on a Saturday has no effect to a survey’s response rate*

One Saturday lunchtime two weeks ago I logged onto one of our servers to send the email invites for two customer surveys that we field regularly. Some time ago the Messaging Times posed the question whether sending emails at the weekend is more effective than during the week – which got us thinking.

We have looked at the best time to email (survey invites) in the past and generally concluded that the day of the week doesn’t really make a big difference. Other factors have a much bigger influence – e.g. amount of time passed since the last customer contact.

Our conclusion for the 2 surveys we trialled is sending the email invites on a Saturday definitely doesn’t harm the response rate. When we email a survey we typically see click-thru’s start coming in after only a couple of minutes (assuming at least a few hundred emails being sent) but I was still surprised to see the same effect on a Saturday. The response rate at the end of the first day was 2.5% lower for Saturday than for those surveys emailed on the Monday, but the Saturday mailings also saw a small boost on the Monday – presumably recipients who aren’t checking their mail at the weekend catching up on their inbox.

“Normal”
Mailed on Monday
“Test”
Mailed on Saturday
End of Day 1 16.60% 14.00%
Day 2 23.70% 21.30%
Day 3 26.00% 26.40%
Day 4 27.00% 27.00%
Day 5 27.60% 27.80%
Day 6 28.20% 28.40%
Day 7 28.40% 28.70%

Table 1: First survey’s click-thru rates (cumulative %) after 7 days

Whenever possible we send a reminder/follow-up email to the non-responders – so Saturdays adds some flexibility to our resources schedule if it were required (assuming that the survey doesn’t need results within less than 4 or 5 days). * Your circumstances and experience might be different.

“Normal”
Mailed on Monday
“Test”
Mailed on Saturday
End of Day 1 20.20% 17.20%
Day 2 26.30% 25.20%
Day 3 28.70% 30.00%
Day 4 30.00% 31.50%
Day 5 30.70% 31.90%
Day 6 31.40% 32.40%
Day 7 31.70% 32.50%

Table 2: Second survey’s click-thru rates (cumulative %) after 7 days

Dan Wardle

“We need your feedback”

Another look at examples of emails about customer surveys with two common but very different approaches…

Short and Sweet

The first example is a transactional email after contacting APC for support on a new UPS (uninterruptable power supply – excellent customer service by the way and definitely happy to recommend APC to all as a result). A couple of days after my support case was resolved I received an email with the subject We Need Your Feedback.

We need your feedback (email screenshot)

This email probably serves APC very well and it’s good because it’s short and to the point – how many customers actually read every word?

The only two issues I have is the email was personalised with the support ticket number so why no further personalisation (a greeting, subject/problem area from original support ticket)? The second is how very automated the message felt – signed off from the ‘Team’ rather than an individual (yet my correspondence was with the same rep.) and the “please do not reply to this email” message is a common bugbear amongst bloggers. It says “we’re interested in what you have to say, but only to the questions we ask“.

Today’s Takeaway Action: If you have a system for sending automated survey emails when support tickets are closed – review that message.

Too Much Information

I bookmarked this post earlier this month – Bank of America – This is NOT spam – showing a (poor) example of  a survey invite for Bank of America. A common problem is that the organisation feels the need to give lots of background to prove their credentials and the value of the research. For a bank this is much bigger because how do you prove you are who you say you are in an email?

Getting the Email Right

screenshot of the let's talk shopping emailContrast Bank of America’s message with Marks and Spencer’s Talk Shopping with Marks and Spencer email (screenshot left – click to zoom in).

All the supporting information – who we are, how did you get my details, and extra information is at the bottom of the email. Ok, this invitation is to join an online panel so it isn’t your regular “we need your customer feedback” type message – but it’s a good example of an email about a survey where more details are important/helpful/deemed necessary.

The key point and the call to action is at the top of the email, and all the content is broken into smaller, manageable chunks rather than chucked together in haste.

Dan Wardle

Tips for better survey design #4: Keep ranking questions simple4

A popular type of question to ask in a survey is the ranking question, e.g.

Please rank each of the following items in order of importance

From the respondent’s perspective, this question involves more effort to answer than others because of the need to compare each item in the list against every other item in the list, and also requires more time to fill-in as the list gets longer.

In many cases though you don’t need to ask for every item in the list to be ranked. Asking for the top 3 (or even 2) can be enough to arrive at the same result.

The data table below is from real survey data on a question that ranked 8 items (although only showing the top 3 positions here):

Item % Ranked
1st
% Ranked
2nd
% Ranked
3rd
Red 26 19 14
Blue 28 20 15
Pink 3 13 17
White 20 17 13
Black 8 10 10
Brown 3 5 9
Yellow 7 10 12
Green 6 7 9

With this survey data the ranking is identical whether we got the respondent to rank all 8 items or just their top 3 (and if we only used the top 2 the results are the same except for positions 4 and 6 – btw the results above are blue first, red, white, then a big drop to pink, yellow, black, green and brown last).

The point though is that your ranking question data isn’t going to be the only factor in any business decision (other survey data should contribute for a start). Are you looking to identify the most important 5 or 6 factors (from a list of 26), or categorise the items into “high importance”, “medium” and “low”? When designing the survey – bear in mind how the data will be analysed – if you don’t need an ordered list of 26 priorities then don’t ask your customers to work out theirs!

Dan Wardle

How to get better quality comments in your survey results

I’ve been meaning to share this post on Top Right Corner’s blog that highlights two common mistakes when asking an open-ended verbatim type question after the Net Promoter Score question (e.g. would you recommend [company name] to your friends and family).

Asking the Right Question to get to the Key Drivers of NPS or Customer Satisfaction gives good practical advice how to avoid two possible effects that can turn up in customers’ verbatim comments.

Always remember that the wording of a question can have a big impact on how it is answered.

Dan Wardle

We’re hiring!

Surveylab is looking for a graduate / recent graduate to join our small and friendly team as a junior web-developer. This is a technical role supporting the production of clients’ surveys from database setup to programming and Q.A.

The position is full time, based in south-west London (our office is in Epsom but we’re scouting for new offices to move our technical team into).

Read more on our website: web-developer vacancy

Would sending survey emails at the weekend get a better response rate?

A post on The Messaging Times blog makes a case for at least testing whether sending emails at the weekend is more effective than mailing during the week.

The success of weekend mailings is obviously going to differ between organisations and their audiences, and before sending an email about a customer satisfaction survey on a Saturday my first thought is “does the organisation offer support or provide customer contact at the weekend?”, but there are some compelling arguments to trial it.

Looking at just one customer satisfaction survey that we mailed in September – with the initial mailing being sent on a Tuesday and a reminder being sent the following Monday, the number of click-thru’s follows a very typical pattern:

Day No. of click-thru’s
Tue 8
(email invitations sent)
517
Wed 9 258
Thu 10 70
Fri 11 38
Sat 12 20
Sun 13 15
Mon 14
(follow up emails to non-respondents)
204
Tue 15 83
Wed 16 25
Thu 17 25
Fri 18 8
Sat 19 9
Sun 20 5
Mon 21 9
Tue 22 7
Wed 23 3
Thu 24 5
Fri 25 3
Sat 26 6
Sun 27 1

For consumer customer satisfaction surveys we find that it generally doesn’t make a lot of difference when we mail Monday to Friday – after two-three weeks we typically collect the same number of responses overall. In the results above, the first weekend’s results don’t indicate a massive drop just because it’s a Saturday, and later weekends look like mini-spikes.

Mailing on a Saturday could just delay the first day’s big click-thru rate until the Monday. Or, perhaps mailing on a Saturday generates a significant response rate boost because many more people actually see the email and feel they have more time to respond..?

If we can test this out we’ll post our findings here. And if you have any experience we’d love to hear.

Dan Wardle

The Messaging Times post that inspired this one: Weekend Email Campaigns

Do phone surveys have a future?

Jeffrey Henning has written a very good article in Research magazine – do phone surveys have a future?

The television didn’t make radio obsolete, even if it did push dramas, soap operas and serials on to the TV set.

The article is based on the US market, but all his points apply to the UK too.

Dan Wardle