With Statistics Canada having been criticized in the news recently, it’s good to see some of the real applications that impact Canadian businesses and lives, such as the Canadian Internet Use Survey. But I think practitioners–and the general public–still aren’t quite fulfilling “due diligence” in either citing the Statistics Canada information or in how they perceive and interpret it. Even following Statistics Canada’s own perfectly-correct guidelines about whom the results do and do not represent or whether a significant correlation can or cannot imply causation, the data may still not be giving the answers we think they are.
Statistics Canada’s Canadian Internet Use Survey is often cited by public interest groups, not-for-profit organizations, and marketers to support all manner of opinions. What I am mostly concerned about this time is the portion of it concerning Internet Privacy and Security concerns.
Although the mere five questions with only three possible levels of concern (None at all, Concerned, or Very concerned) may have been sufficient to determine that Privacy and Security is one of Canadians’ leading concerns, we know consider Privacy and Security a top concern. Five questions with only three levels of concern is no longer responsibly-adequate to be meaningful. (I am mostly-facetious when I propose that the number of Canadians actually concerned was severely overstated because anyone that wasn’t oblivious or reckless was considered at least “Concerned” in the first place). Knowing how important Privacy and Security is, and knowing how often-cited those statistics are, I think the Stats Can survey is doing a disservice to Canadians, their concerns, and the businesses that benefit from it.
For example, if people take the time to examine the actual survey questions pertaining to Privacy and Security http://www.statcan.gc.ca/imdb-bmdi/instrument/4432_Q1_V8-eng.htm#a10
Section: Privacy and security (PS)
PS_BEG
Beginning of SectionPS_R01
The next set of questions relate to privacy and security concerns on the Internet.PS_Q01
In general, how concerned (are you/would you be) about privacy on the Internet? For example, people finding out what websites you have visited, others reading your e-mail?Interviewer: Read categories to respondent.
- Not at all concerned
- Concerned
- Very concerned
DK, RFCoverage: All respondents
PS_Q02
How concerned (are you/would you be) about conducting banking transactions over the Internet?Interviewer: Read categories to respondent.
- Not at all concerned
- Concerned
- Very concerned
DK, RFCoverage: All respondents
PS_Q03
How concerned (are you/would you be) about using your credit card over the Internet?Interviewer: Read categories to respondent.
- Not at all concerned
- Concerned
- Very concerned
DK, RFCoverage: All respondents
PS_Q04
How concerned (are you/would you be) about providing personal financial information to government departments over the Internet? (e.g., applying for employment insurance or a student loan?)Interviewer: Read categories to respondent.
- Not at all concerned
- Concerned
- Very concerned
DK, RFCoverage: All respondents
PS_Q05
How concerned (are you/would you be) about giving personal, non financial information to a government official in Canada over the Internet?Interviewer: Read categories to respondent.
- Not at all concerned
- Concerned
- Very concerned
DK, RFCoverage: All respondents
PS_END
End of Section
they will note that there are a total of five questions. Those who have taken statistics will recognize that the meaningful options of “Not at all concerned,” “Concerned,” and “Very concerned” imply ordinal data (there is a consistent directionality in the variables).
Those of you who have taken some survey and research design might be concerned, however, that the “centre” choice (sometimes questionnaire-designers purposely give an even number of choices to avoid a dead centre choice) does not at all imply middle of the road. In fact, if a respondent is not absolutely free of concern about privacy (ie. reckless), then any other choice will enumerate them amongst the concerned. There are many of us who have “appropriate” caution when we conduct business online (ie. would not describe ourselves as either apathetic or reckless) but are also would not consciously be concerned about privacy and security under normal conditions (ie. would not describe ourselves as neurotic or paranoid).
The 2008-2009 CIRA Annual Report demonstrates how significantly these data have impacted CIRA’s initiatives, ranging from DNSSEC to BIND10 to WHOIS privacy http://www.cira.ca/annual-reports/2009/en/c_dns_03_en.html. But the primary survey to be cited employs only five questions that will inherently bias responses towards overestimating the amount and degree of concern Canadians have because of its pecular scale.
Highly-qualified statisticians and researchers at Statistics Canada go to a lot of trouble trying fastidiously to apply accepted theory in questionnaire, survey, and sampling design according to traditional principles of maximizing face validity, content validity, criterion validity, Likert scale best practices, stratified random sampling, and making sure that the report reflects accurate interpretation under the correct circumstances in the proper contexts.
But used out of context or with varying lower degrees of external validity (generalizability), all that effort can be wasted–or worse, reinforce the popular notion that statistics are somehow worse than both lies and d*mn lies http://robincheung.info/mbalog/2010/07/21/lies-dmn-lies-and-statistics-statistics-is-actually-your-friend-when-not-misused/
This time, I’m not blaming people for using statistics out of context to support their arguments; I’m suggesting that Statistics Canada should amend the survey.
There is a mechanism for interested businesses, individuals, and Statistics Canada to understand each other and develop surveys that are more meaningful and accurate, by the way. This October 26 to 29, 2010, Statistics Canada is hosting the 2010 International Methodology Symposium in Ottawa, ON. If you can’t make it to that event, Statistics Canada maintains a web site about its training, conferences, and research events: http://www.statcan.gc.ca/services/workshop-atelier-eng.htm




