BROADCASTS.com

Aquinas famously\xa0said: beware the man of one book. I would add: beware the man of one study.

For example, take medical research. Suppose a certain drug is weakly effective against a certain disease. After a few years, a bunch of different research groups have gotten their hands on it and done all sorts of different studies. In the best case scenario the average study will find the true result \u2013 that it\u2019s weakly effective.

But there will also be random noise caused by inevitable variation and by some of the experiments being better quality than others. In the end, we might expect something looking kind of like a bell curve. The peak will be at \u201cweakly effective\u201d, but there will be a few studies to either side. Something like this:

We see that the peak of the curve is somewhere to the right of neutral \u2013 ie weakly effective \u2013 and that there are about 15 studies that find this correct result.

But there are also about 5 studies that find that the drug is very good, and 5 studies missing the sign entirely and finding that the drug is actively bad. There\u2019s even 1 study finding that the drug is very bad, maybe seriously dangerous.

This is before we get into fraud or statistical malpractice. I\u2019m saying this is what\u2019s going to happen just by normal variation in experimental design. As we increase experimental rigor, the bell curve might get squashed horizontally, but there will still be a bell curve.

In practice it\u2019s worse than this, because this is assuming everyone is investigating exactly the same question.

Suppose that the graph is titled \u201cEffectiveness Of This Drug In Treating Bipolar Disorder\u201d.

But maybe the drug is more effective in bipolar i than in bipolar ii (Depakote, for example)

Or maybe the drug is very effective against bipolar mania, but much less effective against bipolar depression (Depakote again).

Or maybe the drug is a good acute antimanic agent, but very poor at maintenance treatment (let\u2019s stick with Depakote).

If you have a graph titled \u201cEffectiveness Of Depakote In Treating Bipolar Disorder\u201d plotting studies from \u201cVery Bad\u201d to \u201cVery Good\u201d \u2013 and you stick all the studies \u2013 maintenence, manic, depressive, bipolar i, bipolar ii \u2013 on the graph, then you\u2019re going to end running the gamut from \u201cvery bad\u201d to \u201cvery good\u201d even before you factor in noise and even before even before you factor in bias and poor experimental design.

So here\u2019s why you should beware the man of one study.

If you go to your better class of alternative medicine websites, they don\u2019t tell you \u201cStudies are a logocentric phallocentric tool of Western medicine and the Big Pharma conspiracy.\u201d

They tell you \u201cmedical science has proved that this drug is terrible, but ignorant doctors are pushing it on you anyway. Look, here\u2019s a study by a reputable institution proving that the drug is not only ineffective, but harmful.\u201d

And the study will exist, and the authors will be prestigious scientists, and it will probably be about as rigorous and well-done as any other study.

And then a lot of people raised on\xa0the idea\xa0that some things have Evidence and other things have No Evidence think\xa0holy s**t, they\u2019re right!

On the other hand, your doctor isn\u2019t going to a sketchy alternative medicine website. She\u2019s examining the entire literature and extracting careful and well-informed conclusions from\u2026

Haha, just kidding. She\u2019s going to a luncheon at a really nice restaurant sponsored by a pharmaceutical company, which assures her that they would\xa0never\xa0take advantage of such an opportunity to shill their drug, they just want to raise awareness of the latest study. And the latest study shows that their drug is great! Super great! And your doctor nods along, because the authors of the study are prestigious scientists, and it\u2019s about as rigorous and well-done as any other study.