Is online qualitative cheaper than traditional research? Well, it’s not always that simple IMHO
Commentary on an article on GreenBook’s site: Research on Research: A Case Study Examination of the Quality, Cost, & Speed of Online Qualitative Research.
There is a fascinating article from Accelerant Research here on the Greenbook site called: Research on Research: A Case Study Examination of the Quality, Cost, & Speed of Online Qualitative Research.
I don’t have any issue at all with the methodology they used, or the great results they achieved with their methodology. My experience with in-depth online has always been great, and they seem to have achieved great results too with the 60 people in their study.
However, I’m not sure the pricing comparison is really fair.
The focus group costs seem about right for the size of the groups — personally, I prefer smaller groups, but that’s just a quibble. The depth interviews would give you a tremendous amount of data, but aren’t really in-the-moment in the same way. Neither of these consider travel costs, but that almost always favors an online approach.
In terms of ethnography, yes, the study would be expensive, but you could achieve tremendous depth with far fewer than the 60 interviews specified. (Anyone care to comment on that?)
For the online approach, the well-known challenge for the researcher (and often for the client!) is the massive amount of data collected, generally far in excess of what you get in a focus group. This comparison does not allow for any meaningful compensation for the data analysis. Either the focus groups are too expensive, or the online is too cheap, or no analysis is happening.
If you consider the minutes of research, imagine having all of it transcribed. It’s easy to see that more respondent minutes is more data. In fact, the downloadable report makes it clear that the online data is truly rich and detailed — it’s excellent stuff!
My point: ♥ I don’t think the main benefit of online methods is lower cost. I’d love to hear what others think. ♥
Also, I’d like to be clear that the article is fantastic, and the study itself an excellent demonstration of the power of online to capture in-the-moment information about the real lives of participants.
Let’s talk about this? What’s your view?
About Susan Abbott
Susan Abbott is a qualitative researcher with a passion for insight-led innovation. Prior to leading Abbott Research + Consulting, Susan was a vice-president of TD Bank Financial Group. She has an MBA from Ivey Business School, holds the CMRP research designation, and frequently speaks and writes on topics related to qualitative research, customer experience, and finding insight. She also publishes the CustomerCrossroads blog. In addition to her work on the Board of Directors of QRCA, Susan is active in the Canadian Association of Professional Speakers. In her free time, she paints, canoes, and rides a Dragon Red Vespa. She can be reached at susan@abbottresearch.com.
Leonard Murphy says:
June 16th, 2011 at 12:19 pm
Great points Susan. I agree the source article is fantastic, but based on my experience of offering online qual since 2005 the costs are not substantially lower; moderation, analysis, and report writing are still pretty fixed no matter what. The main savings are in travel and video recordings.
Susan Abbott says:
June 16th, 2011 at 3:33 pm
Thanks Lenny – it’s tempting to tell clients “it’s cheaper”, but I think we have a stronger case to make for online in other areas, actually. Thanks for posting.
Jill Ransome says:
June 16th, 2011 at 4:00 pm
As a supplier of online qualitative, there are a number of reasons our clients have chosen to conduct their research online: costs is one since you no longer need to include the cost of travel and facility costs; but we often hear others like expanding your geographical reach and access to “hard to reach” respondents like physicians, faster speed to market, time savings and even quality of life.
Paul Rubenstein says:
June 17th, 2011 at 1:58 pm
Susan, I’m delighted by how much you appreciate the article that you cite in this blog. To be clear, I was trying to show in that piece that the shift from traditional, F2F methods in qualitative research to online, in which participant and moderator interact within a computer-mediated environment, not only produces significant cost savings, but also reduces the amount of time needed to execute the study, and yields more and better data.
As it is, corporate-side research managers tend to consider quality, cost and speed when they are deciding among a set of options and suppliers. What we have tried to show the market research community with our study is that the shift to online is one that delivers better, cheaper, and faster services.
Enhanced technology always produces effects like that, but in our society, the comfort level of the general population in blogging, as a result of the explosion in social media site usage, has turned them into truly excellent study participants, and thus produced an unplanned, additional benefit. In essence, people do not have to adapt themselves to the system that is used to collect their data. Such an accomodation might change the data and attenuate its validity. Instead, the conditions of the Internet, and the cocoon of anonymity in which participants feel wrapped, foster a level of candidness and honesty that is rarely obtainable in group settings. This, along with sharp reductions in cost and time, are what make online methods for qualitative research not only viable, but arguably superior.
But as to the point you specifically raise in your blog using the table that you site from Accerlerant’s article, it shows $50k for focus groups and the same amount for online ethnography, so the reader might think “where’s the savings?” In that sense he/she is right.
But notice that the online method produces 12X the amount of data, keeping the number of participants constant. That is what accounts for the fractional cost of online per minute per respondent – the online method yields more data simply because everyone gets to talk at the same time.
What I would offer your readers is the idea that, because asynchronous, online qualitative research more produces so much more data than standard focus groups, the same set of study objectives may be accomplished with fewer respondents.
Six focus groups is about the most standard type of qual study I know, given the many years I have been a researcher. Nowadays, they cost on average a total of about $50k, excluding opportunity cost of all parties being out of their office.
The same study objectives can now be done by shifting methods to online. It will cost under $25k, take at least 2-3 weeks less time, and still yield about 2X the amount of data. The moderator would still net about the same wage for services rendered on that study. Moreover, those data will more likely include deeper, more poignant expressions of people’s lot in life than they would ever be able to muster the courage to say in an open forum.
It would only be the very special moderator with svengali properties so sharp to be able to reliably elicit the depth of emotion that is typically, and by design, offered online by participants who are giving their input at a time period in their day when things are quiet, they have a chance to think and reflect, their surroundings are peaceful, and no one is looking.
As an industry, the market research community needs to understand and appreciate these factors as technology continues to improve. The benefits are even greater for qualitative research than those experienced when quantiative market research embraced the Internet at the advent of that technology.
Nowadays we “bite the bullet” when we do in-person quantitative surveys, usually because of something having to do with the target population, e.g., elderly, non-acculturated immigrants, or the nature of the study, e.g., product test. Otherwise, whenever possible, online quant is the way to go. Yet, focus group facilities still prevail as the main method for qualitative research, having not changed appreciably in 30+ years.
Of course, let’s not forget about the carbon footprint of a round of focus groups vs. doing things online and green!
Kristin Schwitzer says:
June 21st, 2011 at 9:44 am
Paul, thanks for your thorough response — all of the replies to Susan’s post have made for an interesting read. I particularly liked your wording and strongly agree with this comment: “the conditions of the Internet, the cocoon of anonymity in which participants feel wrapped, foster a level of candidness and honesty that is rarely obtainable in group settings.” This is just one of the many benefits of online qual that I’ve experienced over the past 11 years. If I mention it at all, lower cost typically gets last place mention when providing the rationale for why online.
Susan, kudos for writing a post that is getting discussed!
And everyone, keep the comments coming!! We love ‘em!
Susan Abbott says:
June 22nd, 2011 at 4:21 pm
Hi Jill – geographical reach is one of the major benefits in my view, thanks for mentioning that one!
Susan Abbott says:
June 22nd, 2011 at 4:29 pm
Paul, you are preaching to the converted here, as you know. I’m a big fan of online, and have pointed out many times that you get tremendously rich information.
Thanks for re-framing your examples, and for posting such helpful comments.
Next time I do a proposal comparing alternative approaches, I will be mindful of this discussion!
Thanks Paul.
Rick Hobbs says:
June 24th, 2011 at 8:44 pm
Susan,
Good point, professional services are the same. My time is the same whether I am in front of the two way mirror or in front of the web cam. Recruiting nowadays is mostly the same but it is the time out of the office for everyone as Paul says that is the big savings. For me as I said in Calgary (was that 5 years ago already?) I don’t understand why anyone does F2F groups anymore.