A month ago, the media got very excited about an HIV vaccine. Study results, released in Thailand with a maximum of fuss and a minimum of detail, showed that the two-step vaccine might protect about a third of the people who get the shots against HIV. Then the doom-mongers weighed in: without more information, we might be overestimating the effects of the jabs. So, is the syringe half full or half empty?

Now more details have been released, we’re still looking at half a glass. Hurrah! said some reports. The vaccine really is protective. Boo hoo, said others. Unless you torture the statistics, they don’t confess to much of an impact. I finally got around to combing through the full report of the trial in the New England Journal of Medicine. Both the optimists and the pessimists are right. It really depends on what your hopes and expectations were. If you are a basic scientist (as most of the people involved in the study were) you’d be pretty thrilled by the results, because they show that vaccines might one day work. If you are a public health boffin such as myself, you’d be pretty disappointed, because the study suggests that that this vaccine doesn’t work for the people who really need it — a point much underplayed in the official reports.

In their full paper the research team reported three sets of results for this study among young men and women in Northern Thailand; only the most optimistic of these was reported to the press in the initial release of results a month ago. Keith Alcorn of Aidsmap has produced a typically sound and balanced summary of the paper if you want more details. But here’s my more opinionated take.

Analysis 1: Real World.
This is technically known as an “intention to treat analysis”. The final analysis includes everyone who was enrolled in the study, regardless of whether or not they followed all the procedures correctly. This is the analysis which is most interesting to public health boffins, because it comes closest to showing how things might happen in the messy reality of life, where people forget to show up for appointments, get given the wrong dose by mistake, etc. In this analysis, people in the vaccinated group were 26.4% less likely than those in the unvaccinated group to get infected with HIV. There was a 92% probability that this difference was not due simply to chance; that means that the difference would not be considered “significant” by those who cleave to the mystical figure of 95% to dictate what is or is not worth considering. Taking into account random differences between the people assigned to the vaccine and the placebo groups, the researchers were 95% sure that the real effect of the vaccine did something between making you 48 percent less likely to contract HIV, and making you four percent more likely to get infected.

Analysis 2: Ideal World.
Known as the “per protocol analysis”, this looks only at the people who got all their shots on time, in the right doses. This is more or less the human equivalent of doing things in lab conditions, and is the sort of analysis that is most useful for basic scientists. Only three quarters of all the study subjects qualified. That in itself is worrying to people like me; if we can’t deliver four doses of vaccine to a quarter of the participants in an incredibly well-organised, well-funded study with hugely well-motivated study staff, how the hell are we going to do it in the real world? More worrying to the basic scientists, I would have thought, is the fact that in this sub-population of people who did everything exactly comme il faut, the vaccine did not have a more pronounced effect (26.2%, with an 16% chance that the effect was due to chance). Because numbers were smaller there was an even wider range that might have reflected the “true” outcome, from increasing infections by 13.3 percent to cutting them by 51.9 percent.

Analysis 3: Tidied-up World
Not a common convention, the “modified intention to treat analysis” essentially reflected the real world with the messiest bits knocked off. In this analysis, the researchers included everyone in the study, except the seven people who it turned out were already infected before their first jab. These people were missed in the initial screening test because they were still in the “window period” during which a person has the virus, but not yet the antibodies which cause a test to show up positive. They were discovered because they had turned positive by their last jab; the team then went back and used a (much more expensive) test for the virus itself on the original screening sample and found that they had already been infected. From a basic science point of view, it makes perfect sense to excluse these people from the analysis; obviously, a vaccine can’t protect people who are already infected. From a public health point of view, it’s debatable whether we should tidy up the data like this. If we put huge national vaccine programmes in place, we’re going to be vaccinating people who are in the window period, especially in the early years, and in groups at highest risk. I’d say we want to take that into account when estimating the potential effect of a vaccine. it was this “tidy” analysis that hit the headlines a month ago, and gave the study its only “significant” result — a 31.2% reduction in HIV infection, with a 96% probability that the effect was not a statistical fluke. This time, we could be 95% sure that the vaccine didn’t make things worse, that it reduced infection by at least 1.1%, and perhaps by as much as 51.2%.

As a public health nerd, I’m most interested in the Real World Analysis. But I’m even more interested in something that’s buried down at the bottom of Table 2.

Click to enlarge

Here, the research team looks at the effect of the vaccine on people with different levels of risk behaviour. In a shockingly poor piece of paper writing/ editing, it is not actually possible to tell from the Methods section of this paper how the different levels of risk are defined. But the definition for high risk does seem to include at least some of the usual suspects: needle sharing during drug injection, same sex partners for men, commercial sex etc. And what Table 2 shows is that the vaccine makes no difference at all for those at highest risk. It might cut infection rates by nearly half in that group, or it might increase the chance of getting HIV by nearly three quarters. The best-guess estimate is that it cuts infection rates by under 4% among the people who are most likely to be exposed to the virus. Four percent is as good (or bad) as nothing.

The researchers point out that the study was not designed to look at these differences, but call the results “intriguing”. To an immunologist, they must be. Perhaps the immunity conferred by the vaccine is not strong enough to withstand the repeated assaults suffered by someone who shares needles daily or turns tricks three times a week — I have no idea. But to public health workers, it is not intriguing, it is devastating. If a vaccine doesn’t work for the people who need it most, what’s the point? It depends on costs, of course. But would we really develop something that we could give to people who have a very low probability of exposure, while leaving those who are likely to be at risk for HIV unprotected?

It’s a false dichotomy, of course. This trial is a triumph for basic science, because it gives us something positive to work with. It is very far from being a triumph for public health, and it is not helpful that in the early rush of euphoria it was presented as such. I’d even be wary of the language used by the authors of the NEJM paper: in their headline result, they reported quite wrongly that the study showed that “there was a trend toward the prevention of HIV-1 infection among the vaccine recipients”. A trend is something that develops over time. If anything, the data suggest that the effect of the vaccine was weakened over time, so the trend was away from protection, not towards it. But I’m splitting hairs. With vaccines, the basic science has to be right before we even think about the public health questions. This study will send the immunologists back to the drawing board. They need to figure out how we have a possibly succesful vaccine that makes no difference to viral loads in those who do get infected. They need to understand why people who are most exposed to HIV are least likely to be protected. They need to parse out the mechanism by which these two sets of shots, each of which has failed on its own, might be working together. If (and it is still only an if) they can do all of that, then develop something that really does work, the public health nerds can start worrying about how to deliver it, and to whom.

Note the rueful way the Canadian researchers lament the absence of US participation in the North American Opiate Medicate Initiative. In their excellent paper in the New England Journal of Medicine researchers from Vancouver and Montreal thank “the many U.S. scientists who contributed to the early design discussions but ultimately were unable to participate in the trial” because of what they ellipitcally call “financial and logistic barriers”.

This trial was being planned at the same time that the Traditional Values Coalition, defender of all that is Right and Good in America, were sticking the Republican Rottweilers on the National Institutes for Health for funding studies of sexual and drug-taking behaviour. No surprise, then, that US scientists had to drop out of the study. There is no reason in the world to believe that heroin prescription wouldn’t work as well in the US as it does in Canada, the UK or any other country at reducing consumption of street heroin, keeping people in treatment and cutting crime among that hard core of users that have tried and failed to get off smack by using methadone or just saying no. But in the current climate (yes, even with the Obama administration in occupation) there’s really not much point in doing studies in the States — no amount of evidence will lead to a policy change. As Virginia Berridge points out in an interesting editorial in the same issue of NEJM, drug policy is more a matter of history and culture than it is of science. America, founded on puritanism, has always been less tolerant of opiates than the Brits, who used them to fuel an unequal trade with China and some properly great literature.

One finding that surprised the Canadian researchers: while most people in the study obviously knew if they were taking methadone (orally) or heroin (injected) a small number of users were randomly assigned to inject hydromorphone instead of heroin. Neither they nor the study staff knew who was getting the real thing and who was getting the semi-sythetic cough suppressant. Amazingly, not one of the people shooting up cough medicine for a year could tell they weren’t taking smack. As the researchers pointed out in slightly mealy-mouthed research-speak, “the benefits of injectable opiod maintenance might be achievable without the emotional and regulatory barriers often presented by heroin maintenance”. Meaning that we might get away with prescribing drugs to help chronic users stabilise their lives if we could just stay out of the headlines. The “SMACKING UP YOUR TAXES TO SUPPORT JUNKIES!” type headlines.

A finding that didn’t surprise the Canadian researchers: people who were injecting drugs, even on prescription, were much more likely to OD than people on methadone — mostly because the heroin doesn’t mix so well with some of the other drugs they had been taking (crack cocaine use didn’t change for any of the study groups in Canada, although it fell in all groups in the UK). BUT, as the researchers point out, all but one of the overdoses happened in the study clinic, where staff were able to administer nalaxone and provide other support so that users got through the overdose ok. If they’d been out shooting up street smack, the chances are they wouldn’t have been so lucky. Which is one more reason to support supervised injecting facilities such as Vancouver’s impressive Insite.

One thing the Canadian researchers didn’t report was the relative cost of the different approaches. The UK study reported that heroin maintenance cost about £15,000 per person per year, about a third of the cost of a year in jail. But it took a report on the BBC to tell us that we could put three people on methadone for a year for the same amount. The question is: how many of them would still be on treatment at the end of the year?

(The Beeb story has an interesting video interview of one of the users of the programme, but sadly no embed code).