Not wanting to be always the purveyor of bad news, I was looking forward to today’s results from the Pro2000 microbicide studies. After hopeful results in an earlier trial, I’d convinced myself the gel would prevent HIV. But it doesn’t.

It’s very depressing news from a huge, well designed and well managed study of over 9,000 women across four countries. I’m fond of saying that if you torture the statistics enough they will confess to anything (as we saw in the recent vaccine trial in northern Thailand.) But however badly you twist the arms of this study, they’re not going to scream success. For the record:

* If you exclude the women who got pregnant or stayed in the trial more than the planned period of one year, there were 130 new infections among those who used the microbicide gel, and 123 in those who used the identical-feeling placebo. Calculated as new infections per 100 woman-years of exposure, that’s 4.5 for the gel and 4.3 for the placebo, making the microbicide 5% more risky. The statisticians are 95% sure that the true effect of the microbicide is somewhere between decreasing risk by18 % or increasing it by 34 %. In other words, we can’t make any claims at all that the product works.

* If you don’t exclude people who got pregnant and look at everyone in the trial for as long as they were enrolled, you get 145 new infections with the microbicide versus 143 without: 4.6 new infections per 100 years of exposure in both. No difference. None. The true effect using this analysis is somewhere between reducing the risk of HIV infection by 21% and increasing it by 26%.

* The researchers also looked at whether women who used the gel consistently were less likely to get infected than those who didn’t. They weren’t.

It is hard to measure consistent use of microbicides in these studies, but researchers compared three sources of information: what all participants reported during study visits, whether all participants brought back used applicators, and what a sub-sample of women recorded in their detailed sex diaries. All point in the same direction: around 90% of women used the gel most of the time. That in itself might be counted a triumph compared to some earlier studies. It means that if we could find something that actually works, women would be quite likely to want to use it. All eyes will now be on the CAPRISA study which is testing a vaginal gel that has antiretorvirals embedded in it.

For my own part, I’m feeling somewhat sheepish as well as disappointed. When the results of an earlier, smaller trial of Pro2000 microbicide showed that it reduced infection by 30%, I was dismissive, bordering on rude, about the tyranny of the statisticians who said the results were “not significant”, and that we needed more research before acting.

I remain impatient with scientists who want to delay any action until we have perfect data. In the field of public health we are often obliged to do the best we can with what we have; as long as policy-makers are prepared to change their approach as the data improve we can save valuable time and lives. But in this case, the caution was well placed. Mea culpa.

Once again, it is worth drawing attention to the most basic fact in this research: 30 years into the epidemic, in a population that had safe sex counselling up the wazoo as well as universal access to condoms and other services such as STI treatment, more than four in 100 women are still getting infected with HIV. Pro2000 may not work, but just urging people to use condoms doesn’t, either. We need to keep looking for something that does.

The product in question is Pro2000 gel, and the results of the first large trial on more than 3,000 women were reported yesterday at the Conference on Retroviruses and Opportunistic Infections. CROI is all a scientific conference should be (and all the biannual AIDS Circus is not), and you can see and hear every presentation online. The results of the Pro2000 study show that women using the gel were 30% less likely to become infected with HIV than women using a placebo, and a third less likely than women using nothing at all. The reason the researchers are not screaming about it more joyfully is that the results are “not statistically significant”. Meaning, in this particular case, that we can only be between 90 and 94% sure that the difference in infection rates were really the results of the gel, and not the results of pure chance.

This is just silly. If I told you that there was a 94% chance that the red car was a third more likely to crash than the blue car, which would you drive? Yet we’ve managed to establish a norm in the scientific community that only differences that have a 95% probability of not being due to chance can be trusted. For nerds, that means a “p value” of five percent or less is sacrosanct: (p <0.05) has become a talisman of good science. I’m not the first to remark that things can be significant without being statistically significant — economist Tim Hartford wrote a column on statistical significance and Guinness in the FT only last week.

Someone at the conference remarked that “none of us in this audience worship at the alter of the p value of point oh five” but in fact, many of us do. Another thing that researchers at CROI have been bending over backwards to do is to prove that people on ARVs don’t have more risky sex than people not on ARVs. (Aside: this completely misses the point about “behavioural disinhibition” — jargon for “Oh look! HIV won’t kill me! Let’s party!”. What matters is not so much what infected people do once they are on meds, what matters is what uninfected people do because they no longer see any visible connection between unprotected sex and death. Still, people feel the need to show that ARVs don’t make you screw more.) So when a group working in Uganda showed that people on ARVs were 70% more likely to have an extramarital partner than people not on ARVs, they were happy to worship at the alter of the p value of point oh five. In this case, the p value was 0.09 — in other words there was a greater than 90% chance that the differences were real, but researchers were able to say there were “no differences”. We worship from the underside of the alter, too. A larger study looking at ARVs, risky sex and HIV transmission found that unprotected sex was “significantly lower” in those on ARVs. In fact, 17% of those on ARVs reported unprotected sex compared with 19% of those not on ARVs. The difference may have been statistically significant, yes, but does it meet the most important test of significance, the “So What?” test? Almost certainly not.

Epi-rant over. The microbicide trial (and the fact that there is very low transmission from people on antiretrovirals to their partners in the two ARV studies I’ve just ranted about) wasn’t the only good news at CROI today. Giving monkeys antiretrovirals before exposing them to SHIV rectally worked pretty well, too, which bodes well for PrEP in humans. Disappointingly, though, it worked best when the drugs were given between a week and a day before exposure — ARVs taken just a couple of hours before exposure didn’t have much effect. Bang goes my dream of earning millions with an Ecstasy/ Viagra/ Tenofovir combination pill for big nights out. Maybe I’ll just have to settle down and get a real job.

Thanks to Roger for prodding me to spend my day at a virtual conference…