Lies, D*mn Lies, and Statistics Canada II: Internet Privacy & Security

With Statistics Canada having been criticized in the news recently, it’s good to see some of the real applications that impact Canadian businesses and lives, such as the Canadian Internet Use Survey.  But I think practitioners–and the general public–still aren’t quite fulfilling “due diligence” in either citing the Statistics Canada information or in how they perceive and interpret it.  Even following Statistics Canada’s own perfectly-correct guidelines about whom the results do and do not represent or whether a significant correlation can or cannot imply causation, the data may still not be giving the answers we think they are.

Statistics Canada’s Canadian Internet Use Survey is often cited by public interest groups, not-for-profit organizations, and marketers to support all manner of opinions.  What I am mostly concerned about this time is the portion of it concerning Internet Privacy and Security concerns.

Although the mere five questions with only three possible levels of concern (None at all, Concerned, or Very concerned) may have been sufficient to determine that Privacy and Security is one of Canadians’ leading concerns, we know consider Privacy and Security a top concern.  Five questions with only three levels of concern is no longer responsibly-adequate to be meaningful.  (I am mostly-facetious when I propose that the number of Canadians actually concerned was severely overstated because anyone that wasn’t oblivious or reckless was considered at least “Concerned” in the first place).  Knowing how important Privacy and Security is, and knowing how often-cited those statistics are,  I think the Stats Can survey is doing a disservice to Canadians, their concerns, and the businesses that benefit from it.

For example, if people take the time to examine the actual survey questions pertaining to Privacy and Security http://www.statcan.gc.ca/imdb-bmdi/instrument/4432_Q1_V8-eng.htm#a10

Section: Privacy and security (PS)

PS_BEG
Beginning of Section

PS_R01
The next set of questions relate to privacy and security concerns on the Internet.

PS_Q01
In general, how concerned (are you/would you be) about privacy on the Internet? For example, people finding out what websites you have visited, others reading your e-mail?

Interviewer: Read categories to respondent.

  1. Not at all concerned
  2. Concerned
  3. Very concerned
    DK, RF

Coverage: All respondents

PS_Q02
How concerned (are you/would you be) about conducting banking transactions over the Internet?

Interviewer: Read categories to respondent.

  1. Not at all concerned
  2. Concerned
  3. Very concerned
    DK, RF

Coverage: All respondents

PS_Q03
How concerned (are you/would you be) about using your credit card over the Internet?

Interviewer: Read categories to respondent.

  1. Not at all concerned
  2. Concerned
  3. Very concerned
    DK, RF

Coverage: All respondents

PS_Q04
How concerned (are you/would you be) about providing personal financial information to government departments over the Internet? (e.g., applying for employment insurance or a student loan?)

Interviewer: Read categories to respondent.

  1. Not at all concerned
  2. Concerned
  3. Very concerned
    DK, RF

Coverage:  All respondents

PS_Q05
How concerned (are you/would you be) about giving personal, non financial information to a government official in Canada over the Internet?

Interviewer: Read categories to respondent.

  1. Not at all concerned
  2. Concerned
  3. Very concerned
    DK, RF

Coverage: All respondents

PS_END
End of Section

they will note that there are a total of five questions. Those who have taken statistics will recognize that the meaningful options of “Not at all concerned,” “Concerned,” and “Very concerned” imply ordinal data (there is a consistent directionality in the variables).

Those of you who have taken some survey and research design might be concerned, however, that the “centre” choice (sometimes questionnaire-designers purposely give an even number of choices to avoid a dead centre choice) does not at all imply middle of the road. In fact, if a respondent is not absolutely free of concern about privacy (ie. reckless), then any other choice will enumerate them amongst the concerned. There are many of us who have “appropriate” caution when we conduct business online (ie. would not describe ourselves as either apathetic or reckless) but are also would not consciously be concerned about privacy and security under normal conditions (ie. would not describe ourselves as neurotic or paranoid).

Vote for Robin in the 2010 CIRA Board Elections!

The 2008-2009 CIRA Annual Report demonstrates how significantly these data have impacted CIRA’s initiatives, ranging from DNSSEC to BIND10 to WHOIS privacy http://www.cira.ca/annual-reports/2009/en/c_dns_03_en.html. But the primary survey to be cited employs only five questions that will inherently bias responses towards overestimating the amount and degree of concern Canadians have because of its pecular scale.

Highly-qualified statisticians and researchers at Statistics Canada go to a lot of trouble trying fastidiously to apply accepted theory in questionnaire, survey, and sampling design according to traditional principles of maximizing face validity, content validity, criterion validity, Likert scale best practices, stratified random sampling, and making sure that the report reflects accurate interpretation under the correct circumstances in the proper contexts.

But used out of context or with varying lower degrees of external validity (generalizability), all that effort can be wasted–or worse, reinforce the popular notion that statistics are somehow worse than both lies and d*mn lies http://robincheung.info/mbalog/2010/07/21/lies-dmn-lies-and-statistics-statistics-is-actually-your-friend-when-not-misused/

This time, I’m not blaming people for using statistics out of context to support their arguments; I’m suggesting that Statistics Canada should amend the survey.

There is a mechanism for interested businesses, individuals, and Statistics Canada to understand each other and develop surveys that are more meaningful and accurate, by the way.  This October 26 to 29, 2010, Statistics Canada is hosting the 2010 International Methodology Symposium in Ottawa, ON.   If you can’t make it to that event, Statistics Canada maintains a web site about its training, conferences, and research events: http://www.statcan.gc.ca/services/workshop-atelier-eng.htm

Walden should encourage its students to present their research: My impressions of AOM 2010 so far

I’m currently in the second last day of the Academy of Management 2010 annual meeting in Montréal

http://annualmeeting.aomonline.org/2010/ , with over 8,000 primarily academics.

This afternoon I had the opportunity to sit in on four paper presentations, two of which I had the opportunity to review and critique earlier this year as part

of the blinded review process. Being able to

interact with scholars from other institutions and evaluate their research increasingly shows me that whilst Walden’s quantitative requirements are probably less than most PhD programmes’, that their introduction to research theory and design in their newly-redesigned 8008 Foundations, RSCH 8100, and RSCH 8200 that I have completed so far, really allow us to make sure we’re asking the right questions when we get to the micro level.

AOM2010

Academy of Management Annual Meeting 2010

[img src=http://robincheung.info/mbalog/wp-content/flagallery/aom2010/thumbs/thumbs_AOM2010Montreal 001 (1024x768).jpg]
<b>Meta Data</b><br><b>Image Width</b> 1024<br><b>Image Height</b> 768<br><br> <b>Meta Data</b><br><b>Image Width</b> 1024<br><b>Image Height</b> 768<br>
[img src=http://robincheung.info/mbalog/wp-content/flagallery/aom2010/thumbs/thumbs_AOM2010Montreal 002 (1024x768).jpg]
<b>Meta Data</b><br><b>Image Width</b> 1024<br><b>Image Height</b> 768<br><br> <b>Meta Data</b><br><b>Image Width</b> 1024<br><b>Image Height</b> 768<br>
[img src=http://robincheung.info/mbalog/wp-content/flagallery/aom2010/thumbs/thumbs_AOM2010Montreal 003 (768x1024).jpg]
<b>Meta Data</b><br><b>Image Width</b> 768<br><b>Image Height</b> 1024<br><br> <b>Meta Data</b><br><b>Image Width</b> 768<br><b>Image Height</b> 1024<br>
[img src=http://robincheung.info/mbalog/wp-content/flagallery/aom2010/thumbs/thumbs_AOM2010Montreal 004 (1024x768).jpg]
<b>Meta Data</b><br><b>Image Width</b> 1024<br><b>Image Height</b> 768<br><br> <b>Meta Data</b><br><b>Image Width</b> 1024<br><b>Image Height</b> 768<br>
[img src=http://robincheung.info/mbalog/wp-content/flagallery/aom2010/thumbs/thumbs_AOM2010Montreal 005 (1024x768).jpg]
<b>Meta Data</b><br><b>Image Width</b> 1024<br><b>Image Height</b> 768<br><br> <b>Meta Data</b><br><b>Image Width</b> 1024<br><b>Image Height</b> 768<br>
[img src=http://robincheung.info/mbalog/wp-content/flagallery/aom2010/thumbs/thumbs_AOM2010Montreal 006 (1024x768).jpg]
<b>Meta Data</b><br><b>Image Width</b> 1024<br><b>Image Height</b> 768<br><br> <b>Meta Data</b><br><b>Image Width</b> 1024<br><b>Image Height</b> 768<br>
[img src=http://robincheung.info/mbalog/wp-content/flagallery/aom2010/thumbs/thumbs_aomlogo (200x55).jpg]
<b>Meta Data</b><br><b>Image Width</b> 200<br><b>Image Height</b> 55<br><br> <b>Meta Data</b><br><b>Image Width</b> 200<br><b>Image Height</b> 55<br>
[img src=http://robincheung.info/mbalog/wp-content/flagallery/aom2010/thumbs/thumbs_daretocarelogo (143x100).jpg]
<b>Meta Data</b><br><b>Image Width</b> 143<br><b>Image Height</b> 100<br><br> <b>Meta Data</b><br><b>Image Width</b> 143<br><b>Image Height</b> 100<br>
[img src=http://robincheung.info/mbalog/wp-content/flagallery/aom2010/thumbs/thumbs_montreal-skyline (450x338).jpg]
<b>Meta Data</b><br><b>Image Width</b> 450<br><b>Image Height</b> 338<br><br> <b>Meta Data</b><br><b>Image Width</b> 450<br><b>Image Height</b> 338<br>
[img src=http://robincheung.info/mbalog/wp-content/flagallery/aom2010/thumbs/thumbs_n124705204222282_3177 (200x186).jpg]
<b>Meta Data</b><br><b>Image Width</b> 200<br><b>Image Height</b> 186<br><br> <b>Meta Data</b><br><b>Image Width</b> 200<br><b>Image Height</b> 186<br>

This afternoon, one of the questions posed to a presenter who is a PhD student and faculty-member at the University of Maryland, was about the epistemological foundations of his definition of “aggressiveness” in terms of competitive signalling (the four papers were themed on competitive signalling in the Business Policy and Strategy division), and he could not answer the question. Walden’s systematic approach to research design necessarily would have addressed the fundamental epistemological and ontological issues as a matter of necessity in refining the right research questions and selecting the right research designs and methodologies and analyses to address them with the most validity as well as remaining cognizant of social relevance–not forgetting why we’re doing any of this in the first place. Walden’s Scholar-Practitioner model tries to keep this issue front and centre, but must therefore caution that much more that academia cannot compromise on theory construction and compromise internal or external validity to encourage practitioners to see the value in theory and expend effort to adapt it rather than eschew theory altogether. I feel that much of the time that practitioners claim that theory has no value in the real world and dismiss it, they do not know the theory in the first place; just as I did not feel right asserting how much I didn’t like LOST until I’d watched the entire series, I don’t feel that rejecting a theory without first demonstrating some mastery of it and a true willingness to adopt it if you later find value in it, is an informed decision.

Although I feel that abstracting a general theory from empirical data is already meeting practitioners halfway (and that to claim that they cannot apply theory–which was derived from data from the real world in the first place–and expect academics then to deduce back for them a specific application, when they are the ones getting paid the big bucks too, is to ask to meet us 3/4 of the way), sometimes academics do forget why we are doing any of this in the first place.

This experience has also reinforced my belief that Walden should encourage its doctoral students much more strongly to participate and present their research at events such as these, not only for the benefit of the students’ academic careers, but to showcase these strengths Walden has and mitigate the uphill aspect of the battle to legitimize Walden despite the bad reputation of diploma mills that already handicap us ceteris paribus.  I wanted to note also that only Qiang Li’s presentation alluded that different cultures may also differentially reflect and value signalling behaviours.  In spite of the Academy of Management claiming a 43% non-US membership at last night’s New Member Orientation, I still feel that many American researchers tend to portray an ethnocentric approach to research, not by claiming that Western attitudes and values are the best or even that their research only considers that specific case (external validity, in research design terminology), but simply seems naive of any other views or value systems.

The listing for this afternoon’s BPS division paper presentations was as follows:

Paper Session
Program Session #: 804 | Submission: 18163 | Sponsor(s): (BPS)
Scheduled: Monday, Aug 9 2010 11:30AM – 1:00PM at Le Palais Des Congres in 513D

Competitive Signaling
Competitive Signaling

View Map
Chair: Dorota Piaskowska; U. College Dublin; 


BPS: The Role of Competition and Incentives in Rating Markets
Author: Paul Seaborn; U. of Toronto; In this paper I examine rating agencies, organizations that assign ratings to products based on an evaluation of product characteristics. I focus on rating markets with inter-agency competition and examine the role that a rating agency’s source of revenue has on their rating activity. I conduct my empirical evaluation in the US credit/bond rating market where some agencies derive their primary revenue from sellers (bond issuers) while others are paid by buyers (institutional investors) via subscription. Using a series of model specifications I quantify the importance of this revenue source as well as competitor actions on agency decisions of “who to rate” and “how to rate”. While I find that revenue source matters, so does market power, resulting in three distinct groups of competitors – issuer-paid market leaders (S&P and Moody’s), issuer-paid challengers and subscriber-paid challengers. I demonstrate how each group responds differently to the rating activities of competitors. The results of my research are relevant to both firm strategy and public policy in a variety of settings where information disclosure between sellers and buyers takes place.

Search Terms: Incentives , Rating , Agency

Paper is NOT Available: Please contact the author(s).


BPS: Threat of Entry, Asymmetric Information and Pricing
Author: Robert C. Seamans; New York U.; Empirical research on incumbent pricing response to entry has provided mixed results. Limit pricing theory shows that the use of low price to deter entry is an equilibrium strategy only when there are information asymmetries. Prior empirical studies have neglected the importance of asymmetric information when the incumbent determines its strategic response. I argue that variation in asymmetric information between the potential entrant and incumbent allow for identification of the incumbent’s use of limit pricing; limit pricing should only be used when there exist high levels of asymmetric information, not low levels of asymmetric information. I study a market, the US cable TV industry, in which the incumbent interacts with two types of potential entrants: telecom overbuilders and cable overbuilders. There are information asymmetries between the incumbent and potential telecom entrant, but fewer information asymmetries between the incumbent and potential cable entrant. As predicted by limit pricing theory, I find evidence that the incumbent firm uses low price when telecom overbuilders threaten entry, but not when cable overbuilders entry. Evidence of entry deterrence comes from non-monotonic price changes in response to changes in entry probability.

Search Terms: entry , pricing , information

Paper is NOT Available: Please contact the author(s).


BPS: Do Signals Matter in Competition? The Relationship Between Signals and Reaction Intensity
Author: Qiang Li; U. of Maryland – College Park; There have been lots of studies examining the effect of competitive actions on responses. Such studies help predict competitive responses from competitors when a firm initiates competitive actions. Observing the reality, what we find more frequently is competitive signals instead of real competitive actions. Do these signals matter in competition? This is the question this study attempts to answer. This study theorizes the relationship between competitive signals and competitive responses. Due to the time-consuming nature, data collection of this study is still in progress but according to the pace of data collection, we should be able to provide all the results by AOM conference in Montreal, Canada.

Search Terms: signal , competition

Paper is NOT Available: Please contact the author(s).


BPS: Reputation, Altruism, and the Benefits of Seller Charity in an Online Marketplace
Author: Daniel Walter Elfenbein; Washington U. in St. Louis; 
Author: Raymond Fisman; Columbia U.; 
Author: Brian McManus; U. of North Carolina, Chapel Hill; We analyze “natural experiments” on eBay where sellers offer identical products with and without charity donations. Charity-linked products are more likely to sell and attract higher prices. These benefits accrue primarily to sellers without extensive eBay histories, suggesting that consumers view charity as a signal of seller quality and a substitute for reputation. We do not find evidence that bundling products with charitable contributions is directly profitable.

Search Terms: social responsibility , corporate philanthropy , reputation

Paper is Available: View/Download

The doGkins Delusion: Deceived Disciples

As of this morning, my best efforts to cleanse this post of any reference that could lead back to the author of the original post which had more misrepresentations clearly intended to persuade by inciting fear , clearly selected because they are virtually certain to be undetected by the average lay person who cannot recall any chemistry more advanced than introductory university-level chemistry and/or does not have exceptional critical thinking skills to compensate for lack of subject matter expertise.

But even that wasfortuitous; the new title is much pithier and if you accept a premise that a very charismatic Brit who  makes you chuckle when he talks (the one that doesn’t use a bliss symbol board–that I know of) as an acceptable theorum upon which to build a less intuitive proof, then you will understand at least two dimensions to the reference. (Although I wrote this article almost 24 hours ago as a response to a specific article, the author has contacted me and requested that I withdraw any references to the article.  Since my objection is not personally about the author, I have done so.  I feel the need to point out, though, that even if the author did not intend for anyone to see those thoughts–someone did.  And that it would have been a great example for others to see that it we don’t have to let our pride deprive us of any improvements that suggestion might have had or how testing our opinion against the suggestion before rejecting it could reinforce our beliefs in new ways.)

That said, the original post was an egregious violation of the principle that scientists should be scholarly and transcend their biases when they present information.  And since theory is not accepted (or, “become theORum”  if you went to the University of Pop Science Superstar on Audiobook) merely because you were shown enough cases that you accept it intuitively, but only when confirmatory studies demonstrate that the empirical data support accepting a theory as fact with less than 5% chance that cosmic alignment caused enough abnormal data that supports  a theory that is actually false based on application of the proposed theory in a manner that at least one other subject matter expert deems has minimized threats to internal validity and omits no other possible moderating, modulating, or confounding factor but also that at least one methodologist deems was accurately and completely translated into actionable research tasks using a research design, the results of which were  analyzed using the most appropriate statistical test and the emergent trivial conclusion derived via rigorous logical deductive (or inductive, in the case of Grounded Theory and other qualitative designs).

The author’s Masters degree attests that, at some point, she knew this.  For the public disservice that she could have caused with her misrepresentation, the same “false advertising” charges that few people hesitate to accuse corporations of should apply to her. During my three-month Foundations for Doctoral Studies (AMDS 8008) course required of all incoming PhD students at Walden University, we were all required to read On Being a Scientist: Responsible Conduct in Research, discuss what it meant to us, and respond to each others’ interpretations.  During my first three-month course on Research Theory and Design (RSCH 8100), we present our understanding of the very issue of Researcher Bias and Objectivity and respond to our peers’.  (Interested readers can follow the RSCH 8100, Winter 2010 course discussion on Researcher Bias and Objectivity). Some readers have indicated that they were reticent to voice their opinion because they got bogged down trying to follow the science in the examples I used.  I want to emphasize that I object to the principle knowingly misleading people to support your position.  Here’s a simple example that doesn’t involve science; pretend, for a moment, that the blog article I am objecting to is “Kids–Santa does exist! Your parents lied to you!”

The author of this hypothetical article claims that she has definitive proof that Santa Claus does exist, and that parents who tell their children otherwise are lying, perhaps to deprive their children of gifts.  The author claims that, on numerous occasions, during mid- to late-December, she has personally seen Santa at malls across the continent; further, she actually sat on his knee and has photographic proof–seeing is believing (unless you call people who believe in God delusional even though your deluge of grander beliefs  constantly reminds you that things cannot exist unless they prove it to you when you demand it–if only someone else could hear that same voice of reason).  To appear scholarly and unbiased, she even proactively admits the possibility that she Pictureshopped® the Santa into a photo of her sitting on a piano bench and so offers to take you to Square One Mall next December and show you (possibly knowing that most people will not take her up on the offer anyway).

Everything that the author mentioned is true and can be independently verified.  Likewise, the original post did list true hazards and effects of the compounds, all of which can be independently confirmed from authoritative sources, such as the CRC Handbook of Chemistry and Physics.  The problem is not that her facts are not true; the problem is that they are applied to situations that are not representative of eating a Big Mac. It would be analogous to claiming that “If you’re not careful and drink too much bottled water, you could cause a severe electrolyte imbalance resulting in death by cardiac arrest.”

The facts are true.  But there are two wrongs in that statement that don’t make a right: (1) The amount of water most people would have to drink in order to cause a severe enough electrolyte imbalance to cause cardiac arrest.  Britten (2008) describes the case of one man’s death attributed to drinking 10 L of water in one third of a day.  (2) The statement implies that the danger applies at levels close enough to average water consumption that readers might accidentally drink nearly 10 litres of water if they don’t pay special attention to their drinking. (Please see my own blog post this very problem, at http://robincheung.info/mbalog/2010/03/16/whos-afraid-of-vitamin-e/ )

I support the use of persuasive writing to make recommendations to the public informed by expertise that the public generally does not have; however, in presenting the opinion, the author should still take care to represent supporting evidence in the most relevant contexts, evaluate (but not omit) relevant evidence that opposes the recommendation, and ensure that the assumptions made and circumstances that the recommendation applies to are clear. Maybe the author completely forgot about internal and external validity from Masters studies.   Regardless, this post is does a disservice to science and reflects poorly on respectable nutritionists.  The lay public already views academics and theory as irrelevant in “the real world.”  When scientists deliberately use theory inappropriately, this problem only worsens.  When you’re caught in a lie–it hurts credibility in everything you say in the future and have said in the past. Although there I object to the majority of the assertions in the article, I will consider only two examples.  Although author has since removed these references, I feel they are still excellent examples for two reasons: they were actually used as I describe–these are not contrived hypothetical didactic tools only relevant to convey a point, and since the author only sidestepped the issue and removed the references entirely, if I do not discuss them, then nobody will have gained from the experience:

“Sodium citrate” Years before even being eligible for a Masters programme, students competent in cell physiology, organic chemistry, or even Grade 12 chemistry would know that “citrate” is simply the conjugate base of the carboxylic acid, “citric acid,” which we all know for the fruit after which it is named.   With a pKa of 3.13 Citric acid is a “weak acid” (a formal chemistry term that implies that, in solution, it predominantly does not dissociate into citrate ion and give up a proton; it is not a subjective evaluation of whether it is dangerous or not–a concentrated solution of weak acid can therefore be described as very strong). Of course the MSDS sheets on pure citric acid would list personal protective gear and possible skin, eye, and mucosal irritations; if you took the citric acid out of any citrus fruit, concentrated it, and then splashed it into your face and eyes, it would do exactly the same thing. Wikipedia isn’t acceptable as an academic reference, but it is a good starting point for background on a topic that’s new to us: http://en.wikipedia.org/wiki/Trisodium_citrate

“Calcium Carbonate” We usally call calcium carbonate “chalk,” or occasionally, “Tums.” But those monikers probably conveyed too much sense of being neutral or helpful and would , so either she forgot to mention it or intentionally omitted it. Perhaps  the most egregious affront to science in her blog post is the claim that “calcium carbonate” is “aka. carbonic acid, carbonate salt.” The reason why this alternate name (which isn’t an accepted one to anyone but herself) sounds awkward is because it is a stretch.  It distracts us from the fact that calcium carbonate that makes up about 10% of our bones and associate it with “carbonic acid.”  We hear about carbonic acid because it is a major component of soft drinks (it is formed when carbon dioxide is dissolved in water) and acid rain.  It is absolutely true that calcium carbonate is a salt of carbonic acid.  But the salt of an acid does not inherit physical or chemical properties of the acid itself. The reasoning for this is immediately-obvious if you understand the theory behind what an acid is and what is responsible for the corrosive properties of many acids; perhaps the most prevalent definition of an acid (which definition is most appropriate really depends on the specific compounds that are being discussed, in what reactions, and for what purpose), the Br0nsted-Lowry definition, describes the propensity of an acid to lose protons (a proton can also be thought of as a hydrogen nucleus), whilst bases accept them. The salt of an acid, however, is the result of an acid-base reaction whereby the acidic proton has been replaced by another moiety; in the case of calcium carbonate, the acidic proton was replaced by a calcium ion.  To imply that calcium carbonate still behaves as an acid no longer makes sense. (I avoid generalizing that all salts are no longer acidic because there are common acids, such as sulphuric acid or phosphoric acid, that are “polyprotic acids.”  In the case of phosphoric acid, there are three acidic protons, and forming a salt by replacing only one of these still leaves two remaining protons to contribute acidic character.) It is also absolutely true that table salt (sodium chloride) is a salt of hydrochloric acid, such as the predominant acid our stomachs secrete.  Hydrochloric acid is used to make PVC plastic, for sewage treatment, and as an industrial-strength rust-remover.  It is the same kind of stretch to claim that to consume table salt is to consume the sodium salt of a chemical used to make plastic, a sewage treatment agent, or industrial rust-remover. http://en.wikipedia.org/wiki/Hydrochloric_acid

The public needs protection against this kind of abuse more than from any junk food.

Lies, D*mn Lies, and Statistics: Statistics is actually your friend, when not misused.

This post is a response to an article, posted on her F*cebook Wall by a friend, Stacey Burkett: http://timesofindia.indiatimes.com/Life/Relationships/Man-Woman/Women-are-most-attractive-at-

F-test Fisher-Snedecor distribution compares elements of variation in data: the basis of ANOVA, one of the most common statistical tests.

31/articleshow/6187549.cms.  When my Wall comment exceeded the length of most of even some of my longer blog posts, I decided that I should actually make it an actual blog post.

This post is also relevant to the recent move by Statistics Canada to eliminate the long-form census, issued to 20% of the population every five years, that must be completed under penalty of law and replace it with an entirely-voluntary one; the same principles apply, although I will dedicate another blog post to the Statistics Canada census issue to illustrate the principles in another application as well as to respond more precisely to specific issues.

Surveys and statistics are used to describe all of us, all the time.  Used by marketing researchers who want to define and characterize target markets and psychologists who want to determine the impact of certain personality traits on job performance, surveys can characterize a target population of interest, with known precision, without requiring a census (“census” is a formal survey design term that refers to measuring every member of a population, rather than a “sample,” which is a smaller subset of the population, selected such that the results from it can be generalized to represent the entire population in cases where it is impractical or impossible to conduct a census.  But it is not only population size that limits our ability to conduct a meaningful census; the simple fact that not every individual that is relevant to your survey will be alive at the same time can make a true census impossible.)

LIES, DAMN LIES, AND STATISTICS

“Lies, damn lies, and statistics,” is a reference to the abuse of statistics to support a position.  I feel that this cliché has, itself, been abused and resulted in the unnecessary malignment of statistics, which is actually an extremely powerful tool not only to characterize populations or phenomena, but to predict events, with known confidence (such as the application of probit models and logistic regression that take categorical or numeric predictor variables, such as age, income level, and preferences, that describe a customer segment and calculate the probability that customers will purchase a new product).  I am even more dismayed at the cynicism that has come to surround statistics; whilst the cliché describes the intentional and unintentional abuse of statistics out of context or inappropriately with intent to influence rather than inform, most people–even those who have taken an introductory statistics course in university (perhaps especially those, since most people are thoroughly confused and intimidated by the subject after an introductory course)–do not have sufficient understanding of the theoretical bases for statistical techniques to see the power in them.  Our world is not a deterministic place; even the most reliable process will occasionally yield unexpected results.  Thus, it is vitally important that we can quantify the likelihood that an observation is truly a characteristic result of a phenomenon and state how confident we are that a given observation was not the result of random chance.

Binary outcomes can be modeled with the probit model

The rest of this post pertains to the article that my friend, Stacey, posted to Facebook: http://timesofindia.indiatimes.com/Life/Relationships/Man-Woman/Women-are-most-attractive-at-31/articleshow/6187549.cms The article claims that a 2,000 man and woman survey administered by QVC, an American shopping channel, established that “females in their early thirties are seen as more attractive than younger girls as they are more confident and stylish.”  Although the article is clearly intended as a lighthearted attempt to console their aging customers by presenting findings that run contrarily to what most people would expect, published statistics have a way of turning up supporting an opinion that they do not legitimately apply to.  With respect to the QVC survey, I believe that the results should always be accompanied with a disclaimer outlining the specific population for which the findings can be considered valid; else, surely such a finding would eventually be applied to justify discrimination or disadvantage individuals unfairly.

Having just completed RSCH 8200 in my third straight quarter of research design coursework, survey design–specifically, internal and external validity, and reliability–is quite fresh in my mind. Especially now that the “Long-form Census” is so prominent in the news, I thought a quick run-down on sampling and survey design would help us all put the discussions we hear in the news into perspective–I found that many of the arguments presented by “experts” in the news are not compelling to someone that is trained in statistics because they often will take an extreme position which is not necessarily relevant, such as implying to the public that there are not methods to quantify how relevant certain findings are to the general public or how consistently people will give a response.

Critical thinkers who think ahead but lack discipline in their problem solving would, by now, be wanting to ask, “How do you quantify attractiveness? What is considered “attractive” versus “unattractive”? Is a sample of 2,000 adequate to establish this?

In order to have any meaning at all (which is the origin of that saying, “There are lies, d*mn lies, and statistics,” which much maligns statistics, which is actually the best quantitative tool we have to evaluate results and estimate confidence in them), it is important to keep the following considerations in mind (all other aspects of the design being done “by the book”):

EXTERNAL VALIDITY / GENERALIZABILITY:
External validity is a characteristic that quantifies how generalizable the findings are. Do they apply only to native residents of Toronto? all of Ontario? Eastern Canada? To evaluate this in a social sciences setting, we use our understanding of the underlying theory to inform our design (how attractive a person is may be affected by any or all of physiology, culture, how cosmopolitan was someone’s hometown, any specific trauamatic or pleasurable experiences a person had, etc.)

SAMPLING STRATEGY
The choice of sampling strategy used (random sampling, stratified sampling, purposeful non-probability sampling, etc.) is important to match the population the sample is to be representative of. Most people intuitively know that a sample must be random in order to be representative. But being random is not the only important consideration; if the population QVC wishes to characterize comprises 80% women and 20% men, then the 2,000-subject sample should comprise 1,600 women and 400 men–a 50/50 mix would not be representative of their customer base, if gender contributes to aesthetic preference.  Similarly, if men and women of different ages tend to have different preferences, the random sample should also comprise similar age proportions to the population of interest.

SAMPLING SIZE
Whenever poll results are presented in the news, particularly during governmental elections, we are used to hearing “Poll is accurate to within 3.1% 19 times out of 20.” This means that the sample was adequate to be 95% confident that the results from the poll’s sample are within 3.1% of the true results of the entire populaton. In order to do this, we need to consider what kind of statistical analyses will be done (this determines what measures will be relevant), the size of the effect being studied (phenomena with stronger effects generally need smaller samples to be confident of their effects), and the desired confidence (in most cases, 95%). Software such as G*Power 3 (http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/) can be used to calculate the sample size that is required to attain a given margin of error for a given effect size and statistical test. In doctoral research, dissertation committees and Internal Review Boards will generally require justification of proposed sample size in order to assess and minimize the burden on test subjects.

INTERNAL VALIDITY:
There are several facets to Internal Validity, but in general, they all pertain to ascertaining “how well does this survey actually measure the underlying construct that I intend to measure?” For example, face validity assesses how good a question is as a proxy to what you actually want to know: “What was your last grade completed?” is less valid than “How many years of school did you complete?” if you want to compare amount of education to job performance without regard to at what level the schooling was–people who have skipped a grade would have completed a higher grade level for the same number of years of schooling. Content validity describes how completely the survey describes the contributing factors to a phenomenon; in the above example, a survey that records only the number of years of schooling, without regard to at what levels, would lack content validity in describing the relationship between job performance and education completed, because 8 years of education between Grade 1 and Grade 8 would not have the same results as 8 years of education between Grade 9 and completing a Bachelors degree.

RECOMMENDATION: ALWAYS ACCOMPANY RESULTS WITH EXPLICIT DISCLAIMER

While it is clear that QVC intended this survey as a lighthearted consolation to its aging customers by presenting results that seem to run contrary to what most people might guess, statistics have a way of turning up supporting controversial opinions that are not valid for the sample used.  In order to minimize harm to individuals and mitigate the malignment of statistics, QVC should accompany its results with an explicit explanation of the population that its findings can be legitimately applied to.