Sotto Voce.

"Qui plume a, guerre a." — Voltaire

The Statistical Anomaly

“Figures often beguile me, particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force: ‘There are three kinds of lies: lies, damned lies, and statistics.'” — Mark Twain*

Many years ago, I was helping a client prepare a report on primary education that made use of statistical data collected by the federal government. In the report was a table showing the math proficiency of elementary school students from first to fifth grades for a particular year. The table showed that for each grade level, the level of proficiency was less than the lower grade; i.e., first graders were more proficient at math than second graders, second graders were more proficient than third graders, and so on.

The caption for the table read, “Students become less proficient in math as they progress through elementary school.”

I pointed out to my client that the table did not, in fact, show that. The table presented a snapshot of a five groups of students in a single year, not a progression of a single group of students through five years. In fact, I said, someone could use the same data to argue the opposite, that every year we’re getting better at teaching math to elementary students.** Unfortunately, I lost the argument; because it sounded more urgent, the original caption stayed. (I didn’t work for that client much longer after that, anyway.)

We live surrounded by data. We have more numbers to crunch than ever, and we can crunch them faster than ever. But we seem to be constantly befuddled by what to do with the results. We pick a statistic and then hang everything else on it regardless of whether it can support the weight. Consider:

  • Until recently, charity watchdogs ranked nonprofits by the amount of money they spent on overhead. Oops.
  • The value of a piece of scientific research is still largely determined by the “impact factor” of the journal in which it is published, which is just an indication of how many times its articles were cited elsewhere. Oops.

The obsession with minimizing overhead expenses drove nonprofits to starve themselves of things like decent computers and software that could help them manage their operations more efficiently, supplies and administrative support for the program people in the field, outreach and communications to help boost charitable donations, and even qualified staff with the experience to lead the organization.

Or else, to keep their vital back offices intact but still below the magic no-no threshold of 10 percent of total revenue, they chased after mega-gift donors and forgot about cultivating their loyal bases, mass-mailed and robo-called everyone in the phone book instead of developing sustainable fundraising strategies, brought in board members for their wallets and Rolodexes instead of their abilities and vision, and sold the naming rights to everything from the entire building down to the urinals.

All to meet that arbitrary number that the watchdogs had deemed the magic number.

Nobody in science or medicine — two fundamentally data-driven enterprises — really believes that impact factors mean anything more than what they actually represent: one publisher’s “hey, betcha didn’t know” trivia about citations. But nonetheless both fields are in thrall to them. Because as goes the impact factor, so goes the grant funding and the job offers and the tenure.

Why do we become so captivated by the One True Statistic? Why do we put more meaning into a single interesting number than it can possibly really represent? Why are we so often bad at using quantitative data to make qualitative decisions?

_____
* = Turns out Disraeli never said that. But I bet he would have wished that he had.
** = It’s like the old story about the student who outsmarts his statistics professor by arguing that if it takes one ship five days to cross the ocean, then obviously it should take five ships only one day to cross it, ipso facto all statistics are hogwash. The correct response to the snarky student, of course, is to point out that if each ship leaves one day apart, then yes, the first ship will arrive in Marseilles one day after the last one leaves New York. But in those stories, the teacher is never better at being a wise ass than the student.


Categorised as: Life the Universe and Everything

Comments are disabled on this post


3 Comments

  1. Bill M says:

    You make some good points. I wonder when the percentage promoters will be put out of business.
    Remember 200% of Nothing?

  2. TCWriter says:

    Like the post, though unfortunately it appears to exceed the “5% bullet point” threshold. Shame, that.

    My wife is the ED of a nonprofit. I understand the point of watchdog groups aimed at ferreting out nonprofits which raise money only to pay themselves, but it did get pretty oppressive.

    They depended on grants to buy new PCs every few years, and the last time they had a whole MS Exchange/Windoze system foisted on them.

    It saved them the captial cost, but the cost of keeping it running was astonishing, and it didn’t do its job as well as less-expensive, less-complex systems.

    Yet the watchdog group was happy…

  3. sottovoce says:

    Another example of this mentality that hits close to home is the use of typing speed as a metric of editorial quality.

    Seriously?

    If you give me a deadline of Tuesday at 5 for a 2,500-word article on, I don’t know, the latest developments in paper towel absorbency, then by God I’ll be handing you a 2,500-word inside scoop on what’s hot at the Brawny labs by Tuesday at 4:55. Who in their right mind gives a ram’s damn how fast I typed it?

    The difference between typing 80 wpm and 90 wpm is 10 minutes for every 5,000 words. Get over it.

    The other (and arguably worse) problem with writing and editing tests that use automated word-count tracking is that they subtract for corrections.

    Again, seriously?

    You really honestly expect your writers to not only get it perfect on the first draft, but to never, ever make a typo?

    In that case, you’re not looking for a writer, so please don’t bother wasting my time by calling me.

    (*Deep breath*) Sorry. Like I said, this one hits a little close to home.