Vaccination and Transmission: Overstated Impacts and Unexplored Variables – A Critical Review of Methodological Gaps and Data Limitations

A new preprint tries to conclude that the vaccine prevented transmission, particularly with dose 1 and dose 3. They ignore massive factors, like natural immunity. Here is my thorough analysis.

A preprint is available of a study that sought to estimate the impact of SARS-CoV-2 vaccination on virus transmission in the UK during 2021 by employing a Bayesian hierarchical model. Its primary focus was assessing how one, two, and three vaccination doses influenced the time-varying reproduction number (Rt), reflecting the virus’s transmissibility at different pandemic stages.

Though the study is among the first to analyze vaccine efficacy against transmission at the population level, several factors challenge its conclusions. The context of lockdowns, easing restrictions, and the Delta variant complicates interpretation. Moreover, the authors' methodological choices, such as assuming constant vaccine efficacy over time and excluding key demographic variations, raise concerns about the robustness and applicability of their findings.

This critique focuses on these methodological shortcomings. The goal is to provide a clearer understanding of the study’s ability to inform public health policies while highlighting where improvements could enhance the accuracy and relevance of future research on vaccination and transmission dynamics.

Methodological Critique

The methodology used in this study is central to the validity of its conclusions. While applying a Bayesian hierarchical model is appropriate for estimating transmission effects across multiple regions, several critical methodological choices undermine the reliability of the results. Lack of formal model selection, constant vaccine efficacy assumptions, and omission of sociodemographic factors weaken the analysis.

Lack of Formal Model Selection

The study did not implement a formal model selection process, such as using the Akaike Information Criterion (AIC) or Bayesian Information Criterion (BIC), to compare competing models and ensure the best fit for the data. Modeling complex systems like viral transmission, where numerous variables influence outcomes, demands formal, objective, and transparent model selection. Without this step, the chosen model may overfit or underfit the data, leading to potentially misleading conclusions. Overfitting could make the vaccine appear more effective than it is by capturing noise rather than meaningful patterns, while underfitting may miss important transmission dynamics. Additionally, without comparing alternative models, the authors cannot be confident that their choice best represents the underlying epidemiological processes at play.

Assumption of Constant Vaccine Efficacy

Another major limitation is the assumption that vaccine efficacy remains constant over time. Evidence shows that immunity conferred by COVID-19 vaccines wanes, particularly against infection and transmission, as demonstrated in multiple studies. Ignoring this dynamic can significantly distort the results by overestimating vaccine effectiveness in the later stages of the study period. Incorporating a time-varying vaccine efficacy model, which accounts for waning immunity and booster doses, would have provided a more realistic understanding of the vaccine’s role in transmission reduction. The static model used in this study likely fails to capture important changes in vaccine effectiveness over time, especially when booster doses were rolled out or when immunity began to wane.

Sociodemographic and Geographic Variability

The study also lacks consideration for sociodemographic factors that influence transmission dynamics and vaccine uptake. Age, population density, income levels, and occupation-based exposure risks are crucial in determining transmission and vaccine efficacy. By failing to account for these factors, the study risks generalizing its results across diverse populations where vaccine uptake, exposure risk, and health outcomes vary widely. Although the model incorporates Lower Tier Local Authority (LTLA)-level data to capture regional variability, this level of aggregation does not reflect the full spectrum of localized demographic differences. For example, urban and rural areas may have vastly different transmission dynamics that are not captured at the LTLA level.

Bayesian Hierarchical Model Assumptions

The choice of a Bayesian hierarchical model requires careful specification of prior distributions, yet the study does not provide sufficient transparency about these priors or how they were chosen. The results of Bayesian models can be sensitive to the choice of priors, especially in cases where data is sparse or highly variable. If the priors are not well-calibrated or are too rigid, they can skew the results and lead to biased estimates of vaccine efficacy. More detail on how the priors were selected and whether sensitivity analyses were conducted to test their robustness would strengthen the study’s credibility.

Absence of Sensitivity Analyses

In addition to the lack of formal model selection, the study does not include sufficient sensitivity analyses to test the robustness of its conclusions under different assumptions. For example, varying the assumptions around vaccine efficacy, transmission rates, or regional variation could reveal how sensitive the model is to these key parameters. Sensitivity analyses, such as leave-one-out cross-validation, are fundamental in models with hierarchical structures, as they can help identify whether the model’s results are overly dependent on specific assumptions or data points. By not conducting these tests, the study leaves the possibility that its findings are susceptible to unexamined variables or assumptions.

While the study uses a sophisticated modeling approach, these methodological shortcomings—lack of formal model selection, assumption of static vaccine efficacy, failure to account for sociodemographic variability, insufficient discussion of prior assumptions, and absence of sensitivity analyses—limit the confidence that can be placed in its conclusions. Addressing these issues would make the study’s findings more robust and applicable to broader contexts.

Impact of Changing Control Measures

The context in which the vaccines were deployed in the UK during 2021 is critical for interpreting the results of this study. The interaction between vaccination, ongoing control measures, and changes in transmission dynamics plays a major role in shaping the outcomes observed. This section critiques how the study accounts for the changing public health measures and their potential effects on the perceived vaccine efficacy.

Interaction with National Lockdowns

During the study period, the UK experienced a phased easing of restrictions. The national lockdown, in place at the beginning of 2021, was gradually relaxed between March and July. These lockdowns and restrictions significantly impacted viral transmission, reducing the number of contacts between individuals and, consequently, the opportunities for the virus to spread (Institute for Government) (GOV.UK).

Impact on Vaccine Effectiveness

Additive Effect of Lockdowns: When strict lockdowns are in place, transmission is suppressed through non-pharmaceutical interventions (NPIs), such as social distancing and stay-at-home orders. This suppression can magnify the apparent effectiveness of vaccines because, during these periods, viral spread is already controlled to some extent. The additional effect of vaccines on transmission is likely more visible in this context than it would be without lockdowns.

Confounding with Easing Restrictions: As restrictions were lifted, particularly from April to July 2021, transmission rates would have naturally increased due to higher levels of social interaction, regardless of vaccination efforts ​(House of Commons Library)​(GOV.UK). The study’s model has not fully captured the complexities of this transition, leading to an overestimate of vaccine effectiveness during the later part of the study period.

Delta Variant Dominance

The Delta variant emerged in the UK in early 2021 and became the dominant strain by mid-year, coinciding with the widespread administration of second doses of vaccines. Delta’s higher transmissibility altered the transmission dynamics significantly, reducing the efficacy of the vaccines compared to their performance against earlier variants ​(GOV.UK​, Wikipedia).

Impact on Vaccine Doses

Second Dose Efficacy: The study finds little impact of the second dose on transmission, likely due to the timing of Delta’s rise. However, this result might be misleading if the model does not account for how much Delta’s increased transmission dampened the impact of the second dose. The failure to explicitly model the differences in variant-specific vaccine effectiveness could skew the results. It is understood that antibody-dependent enhancement likely helped Delta spread throughout Europe.

Shift in Transmission Reflecting Failed Vaccine Matching: With Delta’s higher baseline transmission rate, the model may have missed the drop in vaccine efficacy reflected in the increased transmission caused by the shift in the biology of the Delta variant. This would make it difficult to discern the true vaccine impact during Delta’s dominance.

Behavioral Changes Post-Vaccination

As vaccination coverage increased, individuals may have changed their behavior, assuming they were less likely to contract or transmit the virus. This risk compensation phenomenon—where people take more risks because they feel protected—could lead to increased exposure to the virus, diluting the observed effectiveness of the vaccine​(GOV.UK)​(Wikipedia).

Impact on Transmission

Increased Social Interactions: Once vaccinated, individuals may have resumed higher-risk activities (e.g., meeting indoors, traveling), leading to more transmission opportunities. The study does not fully account for this behavioral shift, which could confound its estimates of vaccine efficacy.

Underestimation of Indirect Transmission: Behavioral changes post-vaccination could also increase the risk of transmission to unvaccinated individuals, which is not explicitly considered in the study. This indirect effect could further complicate the interpretation of vaccine effectiveness, especially in areas with low vaccine uptake.

Insufficient Granularity of Non-Pharmaceutical Interventions (NPIs)

While the study acknowledges the presence of NPIs like lockdowns and social distancing, it does not differentiate between the varying intensities of these measures at different points in time or across regions. This could obscure the full picture of how these interventions interacted with vaccination efforts.

Impact on Transmission

Region-Specific NPIs: Different regions of the UK experienced different levels of restrictions and timing for reopening. A more granular analysis that captures these regional differences in NPI intensity would provide a clearer understanding of how vaccines interacted with other control measures to reduce transmission.

Effect of NPIs on R₀: NPIs have a direct effect on reducing the reproduction number (R₀). Without fully accounting for the changing levels of NPI stringency, the study risks attributing reductions in transmission solely to vaccines when, in fact, they may have been influenced by ongoing public health measures.

The study does not adequately capture the complex interplay between vaccine deployment, lockdown measures, and the emergence of the Delta variant. The easing of restrictions and behavioral shifts post-vaccination likely influenced transmission dynamics in ways that the model does not fully account for. If better incorporated, these factors would offer a more accurate understanding of how vaccines contributed to reducing transmission during this period.

Data Limitations

The study’s conclusions are significantly impacted by the limitations of the data used. These limitations range from the exclusion of key variants like Omicron, to gaps in geographical granularity, as well as underreporting and delayed reporting of cases. Each of these factors raises questions about the robustness of the model and the reliability of the results.

Exclusion of Omicron Variant: One of the most notable data limitations is the exclusion of the Omicron variant, which emerged at the end of 2021. Omicron’s higher transmissibility and partial evasion of vaccine-induced immunity would have significantly altered transmission dynamics had it been included. By focusing only on wild-type, Alpha, and Delta variants, the study fails to capture the full spectrum of how vaccination impacted transmission in the later stages of the pandemic.

Impact on Generalizability

Inability to Apply Findings to Later Stages: Omicron became the dominant strain in many countries, including the UK, shortly after the study period. Since Omicron spreads more easily and appears to evade some vaccine protections, the results of the study are not fully applicable to understanding vaccine effectiveness in more recent waves of the pandemic.

Variants with Greater Immune Evasion: The study’s exclusion of a variant with significant immune escape capabilities—such as Omicron—means that it does not reflect the full scope of challenges that vaccines face in reducing transmission, particularly as the pandemic evolves.

Limited Variant Analysis

The study only considers three variants—wild-type, Alpha, and Delta—despite other variants circulating during the study period, such as Beta and Gamma, though less prevalent. By not analyzing these other variants, the study misses an opportunity to assess how vaccines performed against a broader array of viral mutations.

Impact on Vaccine Efficacy Estimates

Missed Opportunity to Assess Broader Effectiveness: Including more variants, especially those with different transmissibility or immune evasion characteristics, would have provided a more comprehensive understanding of vaccine performance across different strains of the virus.

Underestimation of the Full Scope of Vaccine Challenges: Focusing solely on three variants underplays the challenges vaccines faced as newer variants with immune escape mechanisms emerged.

Geographic and Temporal Data Gaps

The aggregation of data at the Lower Tier Local Authority (LTLA) level introduces limitations in the study’s ability to capture localized transmission dynamics. Regional differences in behavior, population density, and healthcare access likely contributed to transmission dynamics in ways that were not fully addressed by this level of data aggregation.

Impact on Transmission Estimates

Loss of Granularity: LTLAs are relatively large geographic units, and aggregating data at this level may obscure more localized transmission spikes, particularly in densely populated areas or areas with higher non-compliance with public health measures.

Temporal Reporting Delays: COVID-19 data, particularly case numbers and reproduction rate (Rt) estimates, were often subject to reporting delays. These delays can distort the model’s estimates of transmission, especially during periods of rapid transmission or variant emergence.

Case Underreporting and Data Quality Issues

The study assumes the accuracy of the Rt estimates and case reporting. Yet, it is well-known that underreporting of cases—due to limited testing capacity, access issues, or delays—could skew transmission estimates. This is particularly relevant early in the pandemic when testing availability was limited or inconsistent across regions.

S-Gene Drop-Out from UK’s PCR Tests: It is known that the then-called “UK variant” had a mutation that caused 1 of 3 primer sets to fail. This mutation occurred in a sequence targeted by a primer on the RNA sequence encoding the spike protein. Mathematically, this led to an instantaneous drop in the sensitivity of the PCR test by 50%. This problem was unrecognized for 8 months. The model does not address this source of variation in “transmissibility”, which was actually variation due to failure of a control measure (testing+isolation (see Wikipedia on B.1.1.7)

Impact on Reproduction Number Estimates

Underestimation of Transmission: If cases were underreported in some regions, the study may underestimate the reproduction number (Rt) during key periods. This underestimation could make the vaccine appear more effective in controlling transmission.

Delayed Data Incorporation: Using retrospective case reports or sero-surveillance data introduces time lags that may not have been fully accounted for in the model, leading to discrepancies between the estimated and actual transmission rates.

Representativeness of the Data

The study does not explicitly address whether the data used is fully representative of all socioeconomic and ethnic groups in the UK. Vaccination rates and exposure risks vary significantly between different demographic groups, and failure to account for these disparities could introduce bias. Unaccounted for or hidden hetergeneity in populations being studied is a well-known problem (See Wikipedia (Simpsons paradox).

Impact on Study Findings

Potential Bias in Vaccine Uptake and Exposure Risk: Different socioeconomic and ethnic groups may have had varying levels of access to vaccines and differing exposure risks (e.g., essential workers and multi-generational households). If these factors were not sufficiently accounted for, the study’s findings may not accurately reflect vaccine effectiveness across all populations.

Time Lag in Data Collection

The model is based on time-varying reproduction numbers, but the speed at which data is collected and processed varies significantly during a fast-moving pandemic. Delays in reporting cases, deaths, or hospitalizations can distort the estimated impact of vaccination, especially during periods of variant emergence or rapid transmission shifts.

Impact on Real-Time Transmission Understanding

Missed Localized Outbreaks: Due to reporting delays, localized outbreaks may not have been captured in real-time, leading to transmission estimates that are lower than the reality in certain areas or periods.

Lack of Adjustment for Delays: If the model does not adjust adequately for reporting delays, it may fail to capture the true dynamics of transmission during critical periods, such as the Delta surge.

The data limitations in this study—including the exclusion of the Omicron variant, limited variant analysis, geographic and temporal aggregation, potential underreporting, and representativeness issues—affect the accuracy of the conclusions. These limitations suggest that the study may not fully capture the true complexity of vaccine effectiveness, especially in light of evolving viral variants and localized transmission dynamics. Addressing these data issues in future studies will be essential for producing more accurate and generalizable results.

Interpretation of Results

As discussed in earlier sections, the study’s interpretation of vaccine effectiveness against SARS-CoV-2 transmission is influenced by the limitations of its methodology and data. This section critiques how the authors have interpreted their findings and explores alternative explanations for the observed transmission trends while addressing key uncertainties.

Overestimation of Vaccine Impact

The study concludes that the vaccine significantly reduced transmission, particularly with the first and third doses. However, this conclusion may overestimate the vaccine's effectiveness due to several factors that were not fully addressed in the model.

Confounding Factors

Easing of Restrictions: The study attributes reductions in transmission primarily to vaccination, but the easing of lockdowns and changes in public behavior (e.g., increased social interactions post-vaccination) were likely significant contributors to the transmission dynamics during this period. Without fully adjusting for these changes, the model may have overstated the direct impact of vaccines​(GOV.UK)​(House of Commons Library).

Simultaneous NPIs: Non-pharmaceutical interventions (NPIs) such as social distancing, travel restrictions, and mask mandates (the social signaling to compliance tool) were gradually relaxed alongside vaccination efforts. This overlap makes it difficult to isolate the specific impact of vaccination on transmission, and the model may have inadvertently attributed reductions caused by these measures to the vaccines alone​ (House of Commons Library)​(Wikipedia).

Unaccounted Waning of Efficacy: The assumption of constant vaccine efficacy over time does not reflect real-world evidence of waning immunity. If the model had included waning immunity, it likely would have shown that vaccines were less effective at reducing transmission during the later stages of the study period, particularly after the second dose ​(House of Commons Library).

Counterfactual Scenario Limitations (Null Hypothesis)

The study also presents a counterfactual scenario estimating the reproduction number (Rt) in the absence of vaccination. While useful for understanding the potential impact of vaccination, this counterfactual is limited by several large and untested assumptions.

Effect of Policy Based on Assumption of Transmission Efficacy on Human Behavior

The UK government, like governments worldwide, insisted (despite zero evidence) that the vaccines prevented transmission and were the best way to protect yourself and your loved ones. This is known to at best, unknowable at the time, and given all available data except the study being critiqued, simply false. The false confidence of vaccinated persons caused many people to believe they could not become infected or transmit the virus. This represents another massive shift in factors that affect viral transmission dynamics.

Uncertainty in Credible Intervals

The wide credible intervals for the first and third doses indicate significant uncertainty. While the point estimates suggest a 39.3% reduction in transmission for the first dose and a 48.69% reduction for the third dose, the credible intervals for both are pretty broad.

Confidence in Estimates

Wide Intervals Indicating Uncertainty: The credible intervals for these estimates (e.g., 26.64% - 48.07% for dose 1 and 27.97% - 71.30% for dose 3) suggest that the actual effect could be considerably lower or higher than the point estimates indicate. This uncertainty raises questions about how confident we can be in the study’s conclusions and whether the observed reductions in transmission were indeed due to vaccination, as opposed to other factors like behavioral changes or ongoing NPIs ​(Institute for Government)​(GOV.UK).

Implications for Policy: Policymakers may overestimate the certainty of these findings if they focus solely on the point estimates without considering the wide range of possible outcomes indicated by the credible intervals. This could lead to overly optimistic expectations of vaccine impact, especially when dealing with future variants or waves of the virus.

Alternative Explanations for Transmission Reductions

The study largely attributes the reduction in transmission to vaccination, but there are alternative explanations that could have contributed to the observed trends.

Natural Immunity

Impact of Prior Infections: Natural immunity from previous infections likely played a role in reducing transmission, particularly as the pandemic progressed and more individuals were exposed to the virus. The study does not sufficiently address the contribution of natural immunity, which could have amplified the observed reduction in transmission independent of vaccination​ (GOV.UK).

Adoption of Treatment Protocols: Dexamethasone, an anti-inflammatory steroid, and others were approved and widely used in the UK for treating severe COVID-19 cases based on robust clinical trial evidence. Protocols for early intervention were actively disregarded, but this does not mean that significant percentages of the population did not attempt to self-medicate against the official narrative.

Blended Immunity Effects: Individuals who were previously infected and then vaccinated (sometimes called “hybrid immunity”) may have stronger protection, which could lead to lower or, in the case of ADE, higher transmission rates. This effect is not explicitly modeled, and the study may attribute some of the benefits of this blended immunity solely to vaccination.

Localized Variations in Behavior

Regional Differences: The study aggregates data across the UK, but transmission dynamics and behavior likely varied significantly between regions. Areas with higher vaccine uptake and compliance with NPIs may have driven national averages down. At the same time, other areas could have experienced higher transmission due to lower adherence to public health measures​ (House of Commons Library). These regional differences are not fully explored in the study, leading to possible overgeneralization of results.

The usefulness of the study for providing insights into the potential impact of vaccination on transmission is severely limited, as its interpretation of results is affected by several limitations. The likely overestimation of vaccine effectiveness, the limitations of the counterfactual scenario, and the high degree of uncertainty in the estimates suggest that the study’s conclusions should be interpreted with caution. Furthermore, alternative explanations for transmission reductions—such as natural immunity, regional behavior differences, and the influence of ongoing NPIs—should be considered when applying these findings to public health strategies.

Summary of Key Limitations

Methodological Concerns. The lack of formal model selection, assumption of constant vaccine efficacy, and absence of sensitivity analyses limit the reliability of the study’s results. While sophisticated, the chosen Bayesian hierarchical model does not fully capture key factors like time-varying efficacy, behavioral shifts, or regional heterogeneity.

Data Limitations. The exclusion of critical data points, such as the Beta, Gamma, and Omicron variants and more granular regional data, reduces the generalizability of the results. Additionally, underreporting, delayed data, and limited variant analysis raise concerns about the study's accuracy.

Confounding Effects. The easing of lockdowns, ongoing non-pharmaceutical interventions, and the emergence of more transmissible variants like Delta confound the interpretation of vaccine efficacy. Without fully accounting for these overlapping factors, the study may overestimate the impact of vaccines on transmission.

Wide Uncertainty: The credible intervals for vaccine efficacy estimates suggest a high degree of uncertainty. This means that while the point estimates indicate moderate-to-high effectiveness, the actual impact could vary significantly, and the conclusions should be taken cautiously.

Recommendations for Future Research

To improve the accuracy and applicability of future research on vaccine efficacy and transmission, the following steps should be considered:

1. Dynamic Vaccine Efficacy Models: Incorporating time-varying efficacy that accounts for waning immunity and booster shots would provide a more realistic picture of long-term vaccine effectiveness. All shifting variables that could increase or decrease transmission, including natural immunity, should be included.

2. Granular Data Analysis: More detailed data at the local level (e.g., below LTLA), as well as an expanded analysis of emerging variants like Omicron, will better capture regional and variant-specific transmission dynamics.

3. Model Selection and Sensitivity Analyses: Employing formal model selection methods and conducting sensitivity analyses would ensure that the model is robust and the results are not unduly influenced by specific assumptions.

4. Consideration of Sociodemographic Factors: Future models should incorporate sociodemographic variables, such as age, income, and occupation, to understand how different population groups experience varying levels of vaccine effectiveness and transmission risk.

Real-World Application

Due to its limitations, the study cannot be relied upon to help guide public health strategies. It cannot inform on the indirect effects of vaccination on transmission. Policymakers should consider these limitations when considering future policies on combining vaccination efforts with non-pharmaceutical interventions. Direct evidence of prevention of transmission—early—and monitoring any loss of efficacy is the only means by which transmission should be studied.

In sum, the study’s findings should be interpreted cautiously due to the methodological and data-related constraints. Future research should aim to address these limitations to provide a more precise and more accurate assessment of vaccine effectiveness in close-to-real-world scenarios.