On Tuesday I attended a lecture at the Royal Statistical Society – Are web based surveys the survey method of the future?
I would first say that
- this lecture lent more towards public sector research where Probability Surveys (every individual in the population has a chance of being selected) are required, as opposed to, say, a customer survey.
- I am not a statistician. There are others far cleverer than me who know how to model data and produce those nice little equations that confirm X=Y
- Telephone surveys are not the future
- I don’t have a beard and sit at a desk in front of dusty book shelves with fantastic tomes (to some).
Two speakers argued for and against and it was a very bipolar debate. The argument for (presented by Professor Paul Whitely from Essex University) went through the results of the British Election Study in 2005 (a You Gov survey) compared to face to face interviews. If you just nodded when the technical bits came up you could still follow the presentation that basically concluded the online methodology was marginally more accurate than the in-person interviews. If you were looking at the blinded results you would not be able to identify which had been conducted online.
The key thing for me was that the web survey worked and it predicted the election outcomes accurately. Afterall, is that not the primary objective of the survey? You Gov also got the London Mayor election right – predicting Boris to win when other pollsters using traditional in-person or telephone interviews were suggesting Ken Livingstone was going to win.
Bill Blyth, Director of Research at TNS Europe, argued against from a very different angle. He showed internet penetration in Europe (and also telephone, mobile etc.) and asked how can you be confident in the data if in many European countries less than half the population have access to the internet? Ok, but we’re not trying to field a survey in Albania or Romania. The first question/comment from the floor was surprise that neither speaker mentioned mixed mode interviewing.
Use the right tool for the right job
Bill Blyth also raised the issue that You Gov’s database is almost certainly skewed by people who have an interest in politics. Ok, but if someone is interested in politics/elections, can we assume they’re going to be voting, and those members of the population who don’t care are unlikely to vote? So I think You Gov is well positioned to poll about elections, assuming it is able to reach most of the population that vote. (I noted from the lecture that “You Gov and MORI have specialism in their sample“).
To be fair, at the end the chair did conclude choose the right mode (or use all of them as appropriate).
The problem with panels
The most interesting (and concerning) point is how online surveys recruit their sample. Bill raised that there is a lack of industry standards in research panels.
There are three major issues with survey panels:
- Replication
- Consistency
- Transparency
I’ve seen replication discussed before (a couple of good articles and views were posted over at www.themrga.com) and it’s a known issue amongst operators of panels: identifying and tracking the professional survey do’ers who are in it for the reward. There is a lot of replication between suppliers’ email lists, and there is also a lack of transparency for how panellists have been recruited.
Bill said that he did not believe he could repeat a study online that would be consistent with the first study. A big problem for companies running panels is the churn rate of members (cited at typically 50-60%). Being able to field the survey a second time where the demographic profile has shifted is very difficult. Scientists like to replicate their finding, but that’s not so easy for an online probability survey (N.B. a probability survey).
That’s not to say survey panels don’t work – you just have to know what you actually want to get out of the data and think through what implications your sample will have on the results.
My conclusions are
- There is resistance amongst advocates of “traditional” methods over the quality of online surveys
- The quality of research is driven by cost
- Online surveys have less coverage of entire populations versus face-to-face and telephone
- Don’t ask a statistician which brand of toothpaste he prefers and why
I suppose web surveys are still compartively new (when you’ve been building them for 9 years though it doesn’t feel like it) and there are still challenges to be overcome. I forget who asked the question “will surveys still be used in the future?”.
In fifty years time maybe there will be a different way to conduct research?
