A Critique of "Global estimates of lives and life-years saved by COVID-19 vaccination during 2020-2024"

Methodological Limitations Hamper Reliability of Estimates; Ignoring Adverse Events and Vaccine-Related Deaths Renders them Meaningless.

RE: PREPRINT: "Global estimates of lives and life-years saved by COVID-19 vaccination during 2020-2024".

The study, authored with the intent of quantifying the global benefits of COVID-19 vaccination, is ambitious in scope and detail. It estimates lives saved and life-years preserved by vaccination campaigns during a tumultuous and evolving pandemic. While the authors attempt to provide a comprehensive evaluation, the study suffers from several methodological weaknesses, reliance on overly simplistic assumptions, and failure to address key biases, including the Lyons-Weiler/Fenton bias (aka, case counting window bias). These flaws undermine the robustness of its conclusions and call into question the validity of its estimated benefits.

A major limitation of the study lies in its assumption that vaccination and prior infection are globally independent. This assumption disregards the targeted nature of many vaccination campaigns, which prioritized high-risk populations that may also have been more likely to encounter the virus. Such simplifications overlook critical heterogeneities in population-level exposure and immunity development, leading to inflated estimates of lives saved. Furthermore, the assumption that nearly all individuals would have been infected in the absence of vaccination during the Omicron period exaggerates the counterfactual burden of the virus. It ignores variation in exposure risks, natural immunity, and behavioral adaptations that reduce infection rates.

The exclusion of vaccine adverse effects represents another significant oversight. While the authors emphasize lives and life-years saved, they fail to account for the potential harms associated with vaccination, especially in younger populations. These groups contributed minimally to the estimated benefits, according to the study's own data, yet are included in the overall benefit calculations without considering potential adverse events. Incorporating such risks, particularly when evaluating quality-adjusted life years (QALYs) rather than raw life-years, would provide a more accurate and balanced assessment of the trade-offs involved.

The study’s population stratifications—limited to age and residence status—neglect other important factors such as comorbidities, socioeconomic disparities, and regional healthcare variations. This homogenization erases meaningful differences within populations, undermining the reliability of the results. Additionally, the life expectancy calculations for elderly individuals, particularly those in long-term care facilities, rely on overly reductive assumptions about remaining life-years. The fixed adjustment factor (f=0.5) for reduced life expectancy among COVID-19 fatalities is unsubstantiated, failing to account for wide variations in health status and background mortality risks within these groups.

A particularly glaring omission is the failure to account for the Lyons-Weiler/Fenton bias. This case counting window bias stems from inconsistent timing in classifying cases as vaccinated or unvaccinated, due to the entrenched practices of not counting a vaccinated person as vaccinated until 8 weeks following the receipt of the second dose. This is problematic especially when vaccinated individuals contract COVID-19, are hospitalized or die shortly after receiving the vaccine. In earlier studies that found disease enhancement, the onset of new infection was much shorter than 5 or 8 weeks after the first dose.

The bias causes cases to be misclassified as unvaccinated, or, more often, left out of the math altogether, inflating infection fatality rates (IFR) for the unvaccinated group and exaggerating vaccine efficacy (in RCT) or effectiveness (in retrospective studies; VE) against mortality. Correcting for this bias is essential for producing reliable estimates, particularly when comparing the pre-Omicron and Omicron periods. The absence of any mention or adjustment for this well-documented bias significantly undermines the validity of the study's estimates and conclusions.

While the sensitivity analyses are extensive, they are insufficiently robust to address the breadth of uncertainty in key parameters. The chosen ranges for VE (40-85% pre-Omicron and 30-70% during Omicron) and IFR adjustments fail to encompass the full variability observed in real-world data. For example, vaccine performance against newer variants, waning immunity, and differences across vaccine types are not adequately modeled. This omission leads to overconfidence in the estimates and obscures the complexity of vaccination outcomes across diverse contexts.

Furthermore, the study lacks calibration or validation against empirical data. While comparisons to other modeling efforts are provided, there is no attempt to validate the estimates against real-world metrics such as excess mortality trends or observed vaccine effectiveness in various regions. Without such validation, the study remains purely theoretical and risks overestimating the benefits of vaccination due to unexamined assumptions and modeling biases.

Despite these criticisms, the study has notable strengths that deserve acknowledgment. Its stratification of populations by age and residence status is an important step toward understanding differential impacts of vaccination, even if the implementation is flawed. Additionally, the effort to estimate life-years saved—while controversial—represents a meaningful attempt to quantify broader benefits beyond mortality reduction. These elements, however, require refinement and greater contextual sensitivity to achieve their full potential.

To address its limitations, the study would benefit from several key improvements. First, it should explicitly incorporate adverse effects of vaccines for all ages groups, by integrating these into QALY estimates. Second, the Lyons-Weiler/Fenton bias must be addressed through adjustment of effectiveness and efficacy estimates to reflect proper classification of vaccinated and unvaccinated cases, with clear adjustments in the modeling framework. Third, sensitivity analyses should be expanded to include a wider range of plausible parameter values and explore multi-way interactions to capture the full scope of uncertainty. Finally, the study must validate its findings against empirical data from diverse regions to ground its theoretical models in observed realities.

---

Summary of Key Criticisms and Recommendations

This study provides a broad analysis of the global benefits of COVID-19 vaccination but is fundamentally weakened by methodological oversights, particularly the reliance on oversimplified assumptions about vaccination and infection independence, the exclusion of adverse events and deaths from vaccines, and the failure to address the Lyons-Weiler/Fenton bias. Its sensitivity analyses are insufficiently comprehensive, and its findings lack external validation, reducing their credibility. By incorporating a more exacting approach that uses all of our knowledge, addressing case counting biases, and validating its results against empirical data, the study could evolve into a more robust and reliable assessment of vaccination impacts.

While the critique highlights these flaws, it also underscores areas where the study makes meaningful contributions, such as its attempts to stratify populations and quantify life-years saved. Acknowledging these strengths while providing actionable recommendations ensures that this critique is both constructive and fair. Moving forward, greater attention to empirical grounding and methodological transparency will be essential for advancing the discourse on the benefits of COVID-19 vaccination.