Bad Science

by Ben Goldacre

Cover image

Publisher: Fourth Estate
Copyright: 2008
ISBN: 0-00-724019-8
Format: Trade paperback
Pages: 324

Buy at Powell's Books

Many years ago, when I was still learning how to be a systems administrator, a colleague pointed out that solving computer problems is an exercise of the scientific method. If something is broken and you're trying to fix it, you form a hypothesis for what the problem could be. That hypothesis has testable implications, either by implying a fix or by implying something else that you can measure. You then test your hypothesis by attempting the fix or adding additional reporting to see if the data is consistent with your hypothesis. If it is, you tentatively accept the hypothesis and try that fix, but you stay wary for a while since you could still be wrong. If it isn't, you abandon that hypothesis and come up with another one.

I've never forgotten this. It describes a process that I was unconsciously following, and which just about anyone who tries to fix a computer problem unconsciously follows, but being aware of the process makes the process more effective. It also crystalizes what's wrong with changing things on a system without a testable hypothesis: even if the problem goes away, you're left with little understanding of what happened. It's not only bad problem analysis, it's bad science.

Solving problems in computers is easy. If you do the same thing repeatedly, you repeatedly get the same result. They rarely get mysteriously better or mysteriously worse on their own, and what works with one computer by and large works exactly the same on another computer. Computers make it easy to follow the scientific method; they practically beg you to.

None of that is true of people and medical problems. They get better on their own. They get worse on their own. The exact same treatment for the exact same problem does completely different things in different people. The problem is significantly harder and therefore requires significantly more rigor and care for reliable results and real knowledge. However, that chaos and randomness also provides opportunities for charlatans and frauds, for mistaken patterns, for selective attention to detail, and for endless self-deception.

Bad Science is a book about evidence-based medicine. It is a ground-up explanation of why we do medical studies, how they work, why they work, and how they fail. It's also a trenchant, beautifully-written attack on medical frauds and superstitions and how they're promulgated. Goldacre explains what a double-blind study is, why it works, and how researchers (intentionally or unintentionally) break it. He offers the best explanation of the placebo effect that I've ever read. And he skewers the news media presentation of medical evidence while providing the reader with the tools to understand why nearly all medical reporting in the mainstream press is useless or worse. This is, in short, a book about the scientific method, in all of its difficult complexity, made beautifully readable and applied to the area of scientific research that affects the average person most frequently in their daily life.

Bad Science is also one of the most delightfully entertaining books I've ever read.

Is chlorophyll "high in oxygen"? No. It helps to make oxygen. In sunlight. And it's pretty dark in your bowels: in fact, if there's any light in there at all then something's gone badly wrong. So any chlorophyll you eat will not create oxygen, and even if it did, even if Dr Gillian McKeith PhD stuck a searchlight right up your bum to prove her point, and your salad began photosynthesising, even if she insufflated your guts with carbon dioxide through a tube, to give the chloroplasts something to work with, and by some miracle you really did start to produce oxygen in there, you still wouldn't absorb a significant amount of it through your bowel, because your bowel is adapted to absorb food, while your lungs are optimised to absorb oxygen. You do not have gills in your bowels. Neither, since we've mentioned them, do fish. And while we're talking about it, you probably don't want oxygen inside your abdomen anyway: in keyhole surgery, surgeons have to inflate your abdomen to help them see what they're doing, but they don't use oxygen, because there's methane fart gas in there too, and we don't want anyone catching fire on the inside. There is no oxygen in your bowel.

This was the passage that got me to buy the book. Or:

Classically, cosmetics companies will take highly theoretical, textbookish information about the way that cells work — the components at a molecular level, or the behavior of cells in a glass dish — and then pretend it's the same as the ultimate issue of whether something makes you look nice. "This molecular component," they say, with a flourish, "is crucial for collagen formation." And that will be perfectly true (along with many other amino acids which are used by your body to assemble protein in joints, skin, and everywhere else), but there is no reason to believe that anyone is deficient in it, or that smearing it on your face will make any difference to your appearance. In general, you don't absorb things very well through your skin, because its purpose is to be relatively impermeable. When you sit in a bath of baked beans for charity you do not get fat, nor do you start farting.

Pointing out how to tell what medical information is proven and what is simply asserted is serious, but there's a great deal of complete nonsense available to serve as examples and Goldacre takes full advantage of it.

The first few chapters of the book are a warm-up to lay the groundwork of basic science: what is an experiment, what is a controlled trial, and how does evidence-based medicine work. The ideas Goldacre debunks along the way are warm-up-level nonsense: detox (based largely on a simple bit of chemical fraud), ear candles, an utterly daft British school program called Brain Gym, the real science behind skin cream, and for an extensive discussion of placebos, homeopathy. By that point, the reader understands what a double-blind study is, something of how experimental bias works, and why you can't get a randomized study by assigning patients consecutive numbers and then dividing them into even and odd. Then it's time for the first solid meat of the book: nutritionists, followed closely by the vitamin supplement industry.

Bad Science is more of a toolkit than a debunking of any particular bit of bad medicine, and the examples Goldacre brings into the book are illustrative of more fundamental points rather than the purpose of it. Still, it's startling how many sacred cows he can dispatch as a side effect of teaching how to evaluate medical evidence. Nothing in the initial chapters was much of a surprise and I've been skeptical of nutritionists for a while, but I still vaguely thought vitamin C helps prevent colds (it doesn't), that anti-oxidants are good for you (fruits and vegetables are; consuming anti-oxidants as supplements doesn't appear to do anything useful and may actually be harmful), and that vitamin A helps eyesight (this was intentionally manufactured British propaganda during World War II to hide the existence of radar). Goldacre not only debunks such things but explains how and shows how you can do the same, building the reader's developing toolkit. The quacks become more subtle, going from no trials to manipulated trials or cherry-picked results. Bad Science isn't just a presentation of better medical information: it's an education in how to read the newspaper.

Goldacre saves press coverage of medical science for last, mixed with some excellent chapters on bad statistics and cognitive fallacies. The most memorable idea from this section of the book for me was Goldacre's discussion of natural frequencies and their clear cognitive superiority to percentages for stating risk. The media is in love with relative risk increases stated as percentages, which is about the most deceptive and misleading way that one can present risk. After reading this book, I'm instantly suspicious of any news story about risk that uses percentages. If you get from this that I'm suspicious of nearly every news story about risk, there's a good reason for that.

The last chapter of the book is a look at medical scares and specifically at the MMR vaccine controversy, which sadly isn't as funny as the rest of the book because it's too enraging. It's a great thematic conclusion: it includes bad statistics, faked studies, media malfeasance, fake experts, statistical cherry-picking, and failure of risk analysis. It also drives home the point that this topic is important. It may not matter what people decide to smear on their face. The vitamin supplement makers are mostly just scamming people out of money and not hurting them. But the drop in vaccine uptake is killing people, and the media is helping through fundamental failures in how it covers medical controversy. It's an important but sobering note, and I think Goldacre does a good job acknowledging the emotion on the side of autistic parents without compromising his point about the science.

This is a brilliant book. It's written by a British doctor from the perspective of the British media and medical fads, but I found it translated easily to a US perspective. It's the best explanation of how medical studies are done that I've ever read, and I suspect is better than many medical textbooks (at least in memorability and succinctness, if not in comprehensiveness). It's accessible, straightforward, and unfailingly entertaining, and it's also easily the funniest book that I've read in the past year.

Bad Science is not published in the US, so US residents will need to pay the price for an import from the UK. It's worth every penny. It's the best non-fiction book I've read in the past year.

If you want to get a feel for the material before buying, Ben Goldacre's blog, also named Bad Science, will give you a feel. Also available is a chapter omitted from the trade paperback edition as well as Goldacre's columns for the UK paper The Guardian and other postings. But the book is even better.

Rating: 10 out of 10

Reviewed: 2009-04-06

Last spun 2023-05-13 from thread modified 2023-01-07