Partiality

I was listening to This American Life and their episode on Urban Legends. There were a lot of things in this episode that really irritated me. Mainly because the reporting presented conclusions, but did so by missing pretty important alternative explanations. Explanations that left their conclusions on less solid footing than they presented.  I’m only going to talk about one of them.

If you’re familiar with TAL, you know that they break their show into “Acts”. An act is a single story that revolves around the central theme that they’ve chosen for that week. The theme for this particular week was urban legends that turn out to be true. Act 3 was called “Sleeper Cell”.  And it was about the urban legend that cell phones are dangerous.

At one point, the person being interviewed, Christopher Ketcham, notes that there are lots of studies on each side of this debate. But he notices that if you draw a line around the source of funding of the studies, you discover that of those studies that are funded by industry, 75% show no harm. But those studies that are not funded by industry have 75% of them showing harm.  The implication is that industry is buying the results that they want.

And let me say right now: maybe that’s true. But the big question that came to my mind was this: who’s funding the studies that aren’t funded by industry? There are really only two possibilities: these studies were funded by individuals (extremely rare) or they were funded by the government. And here’s where Ketcham makes an assumption that I think is false: that government is independent, working for our better interests, with no ulterior motives involved, and that their funding of studies is for pure scientific results only, and is not in any way influenced by an agenda.

I don’t believe this assumption. Maybe my friends on the left do. To which I would say to remember that republicans are part of the government, too. It’s pretty easy to see how they’ve got ulterior motives, isn’t it? Sure, you might say, they’re funded by industry. Ok. But some of them lose to democrats. Who funds the democrats campaigns? By people ardently opposed to industry? If you believe that republicans are beholden to the sources of their funding, why do you then not believe the same thing about democrats?

My point is this: if you assume that the source of funding for a study invalidates the results, then don’t you have to call into question the studies that are funded by the government? Aren’t politicians at least as politically motivated to lie and get the results they want as is industry? Why do we automatically assume ill gotten results when industry funds a study, but automatically assume validity when government funds a study?

And none of this even goes to the heart of the problem: it is patently false to say that the source of funding is sufficient as the *only* means to invalidate a study. If you want to correctly invalidate a study, you have to find fault with one of the following:

• The methodology used to gather the data in the study
• The data
• The conclusions drawn from the data

A critique of the funding source is only valid *AFTER* you’ve found some flaw inside the study. At that point, and only at that point, can you claim that the funders bought the results that they wanted. If your only critique of a study is the funding source, what you’re demonstrating is that you were either too lazy to read the study or too ignorant to be able to critique it on its merits.

Watch me now feign surprise that Ketcham, the guy who found the connection between the funding and the results, was a journalist and not a scientist.

And one more thing. Ketcham’s article on this topic says the following: “Interphone researchers reported in 2008 that after a decade of cell-phone use, the chance of getting a brain tumor—specifically on the side of the head where you use the phone—goes up as much as 40 percent for adults.” Two things: first notice the use of weasel words “as much as”, second it provides no context for what 40% means. That looks like a really big number so it must be a really big risk, right?

Let me show you a completely hypothetical example of how numbers can be used to mislead.  Suppose that in any given year, you have a 1 in 1,000,000 chance of dying prematurely. That’s a 0.0001% chance. If you introduce something else and the risk of dying now is 2 in 1,000,000 the percentage is now 0.0002% chance. However since the number of people who died has doubled, you can say that the risk of the thing you introduced increases your chances of death by 100%. And this is not false. But the risk of death is still only 0.0002% after introducing the change. In absolute terms, the new risk is still incredibly small, only slightly larger than the old risk.

Journalists are prone to this type of reporting.  Numbers like this are really big and create a reason for people to read their story. Note that by this logic, a risk that goes from 0.0001% to 0.001% is a 1000% increased risk. But a 0.001% risk is still an incredibly small risk.  So when you read that the risk of some bad thing increases by some percentage, the thing to ask is this: what was the risk before, and what is the risk after? If they’re both really small numbers then you probably shouldn’t worry about it.