“Figures often beguile me, particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force: ‘There are three kinds of lies: lies, damned lies, and statistics.'” — Mark Twain*

Many years ago, I was helping a client prepare a report on primary education that made use of statistical data collected by the federal government. In the report was a table showing the math proficiency of elementary school students from first to fifth grades for a particular year. The table showed that for each grade level, the level of proficiency was less than the lower grade; i.e., first graders were more proficient at math than second graders, second graders were more proficient than third graders, and so on.

The caption for the table read, “Students become less proficient in math as they progress through elementary school.”

I pointed out to my client that the table did not, in fact, show that. The table presented a snapshot of a five groups of students in a single year, not a progression of a single group of students through five years. In fact, I said, someone could use the same data to argue the opposite, that every year we’re getting better at teaching math to elementary students.** Unfortunately, I lost the argument; because it sounded more urgent, the original caption stayed. (I didn’t work for that client much longer after that, anyway.)

We live surrounded by data. We have more numbers to crunch than ever, and we can crunch them faster than ever. But we seem to be constantly befuddled by what to do with the results. We pick a statistic and then hang everything else on it regardless of whether it can support the weight. Consider:

• Until recently, charity watchdogs ranked nonprofits by the amount of money they spent on overhead. Oops.
• The value of a piece of scientific research is still largely determined by the “impact factor” of the journal in which it is published, which is just an indication of how many times its articles were cited elsewhere. Oops.

The obsession with minimizing overhead expenses drove nonprofits to starve themselves of things like decent computers and software that could help them manage their operations more efficiently, supplies and administrative support for the program people in the field, outreach and communications to help boost charitable donations, and even qualified staff with the experience to lead the organization.

Or else, to keep their vital back offices intact but still below the magic no-no threshold of 10 percent of total revenue, they chased after mega-gift donors and forgot about cultivating their loyal bases, mass-mailed and robo-called everyone in the phone book instead of developing sustainable fundraising strategies, brought in board members for their wallets and Rolodexes instead of their abilities and vision, and sold the naming rights to everything from the entire building down to the urinals.

All to meet that arbitrary number that the watchdogs had deemed the magic number.

Nobody in science or medicine — two fundamentally data-driven enterprises — really believes that impact factors mean anything more than what they actually represent: one publisher’s “hey, betcha didn’t know” trivia about citations. But nonetheless both fields are in thrall to them. Because as goes the impact factor, so goes the grant funding and the job offers and the tenure.

Why do we become so captivated by the One True Statistic? Why do we put more meaning into a single interesting number than it can possibly really represent? Why are we so often bad at using quantitative data to make qualitative decisions?

_____
* = Turns out Disraeli never said that. But I bet he would have wished that he had.
** = It’s like the old story about the student who outsmarts his statistics professor by arguing that if it takes one ship five days to cross the ocean, then obviously it should take five ships only one day to cross it, ipso facto all statistics are hogwash. The correct response to the snarky student, of course, is to point out that if each ship leaves one day apart, then yes, the first ship will arrive in Marseilles one day after the last one leaves New York. But in those stories, the teacher is never better at being a wise ass than the student.

Categorised as: Life the Universe and Everything

Comments are disabled on this post