The New England Journal of Medicine (NEJM) recently reported the results of a large-scale nationwide trial in Israel of the Pfizer covid vaccine, BNT162b2. The study concludes that the vaccine has high effectiveness against covid. The NEJM has a reputation of reporting accurate data, but the packaging and presentation of that data can produce bias in conclusions. Careful examination of the data suggests that the benefits of vaccination are statistically significant but small in magnitude.
The trial followed 596,618 subjects who received all doses of the vaccine for up to forty-two days (six weeks). The same number of unvaccinated subjects were matched as controls. The study design was sound. All subjects were PCR negative at the time of entry. End points were conversion of PCR test to positive (cases), cases plus symptoms consistent with covid (symptomatic cases), symptomatic cases who were hospitalized (hospitalized), hospitalized subjects who developed severe disease (severe hospitalized), and severe hospitalized subjects who died (deaths). Note that each end point is derived from the previous end points.
The highlight of the article is a figure illustrating the Kaplan-Meier event curves for each endpoint. Figure 1 illustrates the raw incidence curve for hospitalization, which will illustrate the salient features of all the data.
Figure 1: Cumulative Hospitalization Event Curve
Source: Data are from Noa Dagan, Noam Barda, Eldad Kepten, Oren Miron, Shay Perchik, Mark A. Katz, Miguel A. Hernán, Marc Lipsitch, Ben Reis, and Ran D. Balicer, “BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Mass Vaccination Setting,” NEJM (2021): 1–12. Blue curve represents the unvaccinated control group. Orange curve represents the vaccinated treatment group.
Both curves increase over time. There is little difference between the treatment and control groups for weeks 1 and 2 due to the time period necessary to acquire immunity from the vaccine. The efficacy of the vaccine is due to flattening of the curve earlier for the treatment group compared to the control group. This results in a lower plateau for the treatment group. Efficacy is calculated as an odds ratio and represents the divergence between the two curves over a time interval. Efficacy from the time of vaccination was 46 percent for conversion of the PCR test, 57 percent for symptomatic covid illness, 74 percent for hospitalization, 62 percent for severe hospitalization, and 72 percent for death. The efficacy increased from week 0 to week 3 as immunity developed and the treatment curve flattened. The most favorable efficacy was from week 3 to week 4 and was 60 percent for conversion of the PCR test, 66 percent for symptomatic illness, 78 percent for hospitalization, 80 percent for severe illness, and 84 percent for death. These efficacy figures are misleading, however, as the absolute risks to the control group were low.
The appearance of the curves indicates the statistical significance of the data. However, the presentation of the figure amplifies the differences between uncommon events. The alternative presentation of the data would be the survival plots. Figure 2 illustrates the Kaplan-Meier survival plots for PCR conversion which has the greatest number and percentage of events to maximize the differences between the curves. The minimum value for the y axis must be 0.75 in order to perceive any difference between the curves. If the y axis minimum were set to 0, as is the usual procedure, the curves could not be distinguished without an increase in graphical resolution.
Figure 2: Kaplan-Meier Survival Plot for Conversion of PCR
Source: Data are from Dagan et al., “BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Mass Vaccination Setting.” The blue curve represents the unvaccinated control group. The orange curve represents the vaccinated treatment group.
For uncommon events, the number to treat in order to prevent an event can be a more useful metric of efficacy than odds ratios. For the entire study group of 596,618 vaccinated subjects, the number to vaccinate in order to prevent a single event was 364 for PCR conversion, 490 for symptomatic case, 4,004 for hospitalization, 5,014 for hospitalization with severe disease, and 25,940 for death. These are very high numbers to treat. While claims of efficacy may be statistically significant, the utility of vaccination at this time is very debatable. At a cost of $20 per jab and 2 jabs/vaccination, the cost of preventing a death would be over $1 million, and the cost of preventing a hospitalization would be about $160,000 without considering the costs of any side effects.
The NEJM article spends a lot of space discussing efficacy between weeks 3 and 4. The article concludes: “[T]he estimated benefit increases in magnitude as time passes.” Although efficacy, whether calculated by odds ratio or by number to treat, does improve from weeks 0 to 3 due to the time required to flatten the vaccinated event curves (time to acquire immunity), efficacy does not continue to improve beyond week 3 due to flattening of the unvaccinated event curves. The most likely explanation for the flattening of the unvaccinated event curves is the declining incidence of new cases and prevalence of active cases at the current time. This decline in cases is not due to vaccination, as it is worldwide and occurring in locations without vaccination programs. This declining utility of infection control measures is also true for masks, handwashing, isolation, and social distancing. The transmission of virus in the absence of any control measures declines with the prevalence of the virus.
Although large-scale vaccination programs may make sense in the Southern Hemisphere due to the approaching winter, following the science in the Northern Hemisphere would lead to policies of voluntary vaccination programs timed three weeks prior to the next expected outbreak of respiratory viruses. For full immunity to be achieved by next October 1, large-scale vaccination should take place around September 1–7 rather than the current time. Longer term follow-up data will be necessary before we know whether vaccination at the current time will protect against the next outbreak.
Conclusions
The Israel covid vaccine program reported by NEJM demonstrates a benefit to vaccination. However, the benefit is small. Vaccination is unlikely to be cost effective in terms of preventing future healthcare costs. The benefit of vaccination is decreasing over time due to declining prevalence of the covid-19 virus. Current government policy for the Northern Hemisphere should be planning how to vaccinate those who volunteer for vaccination during a window from September 1 to 7 rather than forcing people to vaccinate now.
