A big red flashing sign saying "This science is completely bogus"

People always ask me (because I study these things for a living), "How can I tell good science from bad? It seems like every time one expert tells you something, another expert will tell you something else"

It actually isn't hard at all to identify the truly bogus science--and you don't need to rely on any technical knowledge or any intuitions about what is plausible and what isn't. Most truly bad science comes with a big red flashing sign the science is bogus. Take for example, the recent pair of studies which allegedly show that pledges of virginity are effective in preventing risky sexual behavior.

The study has been writen up in the NYT. The third paragraph of the article mentions this fact:

The new findings have not been submitted to a journal for publication, an author said. The independent experts who reviewed the study said the findings were unlikely to be published in their present form.

That's really all you need to know. These studies have not been subjected to peer review. The authors, rather than having their work vetted by other experts, went straight to the press with it. You can stop reading the article now. There is no credible expert testimony to be found here. No evidence of anything.

But lets keep reading anyway, just to see how badly knowledge gets transmitted in our society.

The story begins, really, with an article by Peter Bearman at Columbia and and Hannah Brückner at Yale, published in the The Journal of Adolescent Health. Bearmna and Brückner found that STDs were just as common among people who pledge abstinence as those that don't. Why is this? Well, people who take abstinence pledges do delay vaginal intercourse. This is from the B&B article:

Thus, pledgers experience first sex later than others across adolescence, and a significant minority holds out far into young adulthood. By age 25, we estimate that 25% of consistent male pledgers are still virgins, compared with 7% of nonpledgers and 15% of inconsistent pledgers. For females, the corresponding numbers are 21% for consistent pledgers, 6% for nonpledgers, and 10% for inconsistent pledgers. Note that these results are for vaginal intercourse only.

Pledgers also have fewer partners than nonpledgers: the average numbers for males are 2.4 and 1.5 respectively, for females, 2.7 and 1.9. (As always the numbers for male and female partners don't add up, so either men are having sex with some undersampled population of women, or someone is lying.)

The problem seems to be that any benefits of delayed vaginal intercourse and decreased number of partners are completely cancelled out by increased rates of other kinds of sex and decreased rates of condom use. Male pledgers, for instance, were 4 times as likely as nonpledgers to report having anal sex without vaginal sex. They also only used condoms about 30% of the time for their first round of buttfucking, which is a good predictor for how often they will use condoms in the future.

In other words, abstience only education increases the rate of unprotected anal sex, and this cancells out the benefits of delayed vaginal sex with fewer partners. And check this out

In this context, it is important to know that pledgers are less likely than nonpledgers to be tested for STDs, and to have ever seen a doctor because they are worried about an STD.

Also, check out the huge failure rate for pledges of abstinance

Of all respondents, not conditioning on having had sex, the pattern is roughly similar. Specifically, 61% of all pledgers, 90% of all nonpledgers, and 79% of all inconsistent pledgers have sex before marriage or interview date.

When I taught in Alabama, I always had students tell me that abstinence was the only 100% effective method of birth control. This was often accompanied by some preposterous figure for the failure rate of condoms, like 20%. But look, even the best pledgers have a 61% failure rate. No matter how bogus your condom data is, pledges of abstinence work even worse.

Ok, so that is the beginning of the story. A study in a peer reviewed journal showing that abstinence only has no public health benefit. How does the Heritage foundation respond? By showing that the same numbers can be cooked to show something else. They went over the same data used in B&B's article. According the NYT, the found that

adolescents who made virginity pledges were less likely to engage in any form of sexual activity. If those who made promises did become sexually active, their array of sexual behaviors was likely to be more restricted than those of adolescents who did not make a pledge, Dr. Rector's team said.
Those who made pledges were less likely to engage in vaginal intercourse, oral sex, anal sex and sex with a prostitute, and they were less likely to become prostitutes than were adolescents who did not take such a pledge, the Heritage team said.

Wow, how did they make the numbers say something completely different? Well, they just violate the norms for good statistics. Again, the NYT

The team needs to do "a lot of work" on its paper, said David Landry, a senior research associate at the Alan Guttmacher Institute in New York. He said in an interview that it was "a glaring error" to use the result of a statistical test at a 0.10 level of significance when journals generally use a lower and more rigorous level of 0.05.

Also, the researchers decided to use self-reports of STD rates, rather than the laboratory tests of blood samples used by B&B. After all, which is more likely to tell you the truth about an STD, a biological assay or a teenager who has taken an abstinence pledge?

What about the Times role in all of this? Well, I suppose they think they have provided fair and balanced coverage because they both quoted the Heritage foundation and its critics. I got all of my information critical of the study from the Times article. But this is a completely shit model of scientific objectivity. You shouldn't report both sides of an issue when one side hasn't even been subjected to peer review. Effectively, there is no credible other side to this issue. Giving equal time to the Heritage Foundation study is like giving equal time to holocaust deniers, people who believe in UFOs, or creationists.

(For a nice discussion of how journalist "balance" distorts science, see this article in the Columbia Journalism Review.)

Here's what I want to know: Why can't I call up the New York Times, present them with a bunch of unpublished data, and have them write a story about it.

Citation

Bruckner, Hannah, and Peter Bearman. 2005. After the promise: The STD consequences of adolescent virginity pledges. 36 (4):271.