A new kind of online survey
28/11/2012, 13:52
Filed under: Culture, Ephemera, Politics, Statistics

Today the CBC reported that, based on a recent online survey carried about by Leger Marketing for the Association for Canadian Studies (ASC), as many as two-thirds of Quebecers view a flag hanging in the provincial legislature as a source of pride.*

The article goes on to claim,

The findings shine a light on public opinion in a province that has been sending mixed political messages lately: Quebec recently elected the pro-independence PQ — but only with a minority, and at a time when polls suggest support for independence is low, while the PQ’s sister-party in Ottawa, the Bloc, was nearly wiped off the map barely a year earlier.

Setting aside what I think about whether the polling and voting habits of Quebecers actually send mixed messages,** my first instinct upon reading this bold statement and “online survey” in the same article was righteous indignation: “This study shines no such light!”

My attitude only went from bad to worse when I read the following in some ACS documentation,

The findings were collected from a survey of 2200 Canadians conducted by the firm Leger Marketing during the week of November 5th, 2012 and commissioned by the Association for Canadian Studies. Done by web panel, the margin of error for an equivalent telephone survey is 2.9 percent 19 times out of 20.

Statistical confidence has absolutely nothing to with whether you called someone or whether you emailed them.

Why would they say that bit liking the margins of error if they weren’t trying to pull something over on us? And what’s a “web panel”?

Online surveys are online surveys… right?

Some reporters are better or worse at reporting on online surveys. Sometimes, they’ll mention that we can’t estimate confidence intervals for it.

But, if they were honest, they would say, “We have no idea how accurate this poll is or isn’t. In fact, it could be entirely made up.”

Instead, they leave it up to the reader to draw this connection themselves.

However, as it turns out, “web panel” is the key word in all of this.

In fact, Leger Marketing conducts its online survey on a “web panel,” in their case, a sample of 400,000 carefully selected online contacts which it believes is representative of the Canadian population.

From their webpage,

Strong from its 25 years of polling and market research expertise, Leger Marketing has built over the past seven years Canada’s most representative Web panel which counts today more than 400,000 active members. […]

Although diverse methods are used today by most firms to recruit panellists, Leger Marketing has privileged a randomly selected sample through its internal call centre, allowing it to ensure the best sample distribution and obtain the most accurate results as demonstrated during the past election campaigns.

In other words, when Leger conducts online surveys it isn’t just pinning something to the end of a webage that anyone can come and click on. It’s actually sampling something.

As a result, it can calculate confidence intervals, even if results are only generalizable to their 400,000 person super-sample.

This is a much more intelligent way of conducting online surveys.

I look forward to learning more about how Leger and groups conducting similar web panels collect, maintain, and check their super-samples. I’d also be interested to know if they can produce confidence intervals that relate their results back beyond their super-sample and to the wider population.

At the end of the day, the strength of their results will depend most on just how representative their 400,000 super sample actually is.

* You can read all about the controversy that made this story a headline at the link.

** Mandatory side-rant about anglo-media’s poor coverage of Quebec: Who are Quebecers sending a mixed message to? Do you really think Quebecers care what you, anglo-CBC-reader, thinks? It might come as a surprise to you, but Quebecers are not single issue voters and so see no conflict in voting for separatists in one election and federalists in another.

Comments Off on A new kind of online survey

Comments are closed.