The Decay of Actual Knowledge Generation from Biomedical Research Science from 2008 to 2022 is Horrifying

Biomedical research methodology is being dumbed down, making it impossible for research scientists to render results that lead to knowledge and understanding.

UPDATE(9/12/2022): IN THE COMMENTS, A READER CONFUSES MY SOUNDING THE ALARM ON THE DUMBING DOWN OF SCIENCE AS ‘BOMBAST’, AND TRIES TO SHOW THAT MY RESULTS WERE FALSE BY DOING HIS OWN PUBMED SEARCHES. I NEGLECTED TO REPORT CERTAIN DETAILS OF MY SEARCH THAT HE DID NOT HAVE, INCLUDING (1.) THAT I USED NO QUOTATION MARKS AROUND THE SEARCH TERMS, AND (2.) THAT I RESTRICTED MY SEARCH TO RANDOMIZED CLINICAL TRIALS. HE ACKNOWLEDGED THAT MY RESULTS PROVED REPRODUCIBLE AFTER ALL. HIS RESULTS ALSO SHOWED THAT A MINORITY OF - AVERAGING 3-5% - OF STUDIES THAT INVOLVED MULTIPLE REGRESSION LIKELY REPORTED AN INTERACTION TERM.

FOR THOSE WHO ARE INVOLVED IN BIOSTATISTICS AND EPIDEMIOLOGY, I ALSO OFFER THIS POWERFUL CITATION THAT SHOWS PRECISELY THE EFFECT OF IGNORING THE INTERACTION TERM THAT I HAVE BEEN WARNING ABOUT SINCE 2015 (HERE, FOR EXAMPLE, IS AN ARTICLE IN 2018). THE PROBLEM IS THAT THE BETA COEFFICIENT ESTIMATES BECOME BIASED IF THE INTERACTION IS SIGNIFICANT AND NOT STUDIED OR NOT REPORTED. THIS IS HOW YOU MAKE SOMETHING LIKE A VACCINE LOOK SAFE WHEN IT IS NOT. I STAND BY MY RESULT AS BEING MORE INCLUSIVE AND MORE FULLY REPRESENTATIVE THAN MORE RESTRICTED RESULTS FOUND WITH QUOTATIONS, AND THERE ARE CLEAR EXAMPLES OF THIS IN THE VACCINE SAFETY STUDIES. REGARDLESS, THE VATCHEVA ET AL., CITATION FROM 2015 PROVIDES A GOOD REFERENCE FOR THE PROBLEM.

HOW TO SPOT THE PROBLEM: WATCH FOR VARIABLES THAT ARE ‘ADJUSTED FOR’ WITH NO CONSIDERATION OF INTERACTIONS OF THOSE VARIABLES WITH THE MAIN EFFECT. CALL OUT SUCH STUDIES WHEN YOU SEE THEM; UNLESS THE AUTHORS REPORT THE INTERACTION TERM, THE EXONERATION OF THE VACCINE BEING QUESTIONED AND THE ADVERSE EVENTS BEING STUDIED IS IN SERIOUS DOUBT.

Vatcheva KP, Lee M, McCormick JB, Rahbar MH. The Effect of Ignoring Statistical Interactions in Regression Analyses Conducted in Epidemiologic Studies: An Example with Survival Analysis Using Cox Proportional Hazards Regression Model. Epidemiology (Sunnyvale). 2015 Feb;6(1):216. doi: 10.4172/2161-1165.1000216. Epub 2015 Jan 15. PMID: 27347436; PMCID: PMC4918637.

---

The public can no longer assume that the biomedical and clinical research community is up to the task of ensuring that their research studies produce knowledge about reality because methodology is being dumbed down. Here’s the proof.

When scientists perform a regression analysis, they seek to understand whether a relationship exists between a dependent variable (Y) and potential or known independent variables {X1, X2, X3…Xn}. Having data on the right independent variables is key, and sometimes a specific relationship is of highest interest. However, variation associated with other factors (X2, etc) might impact the relationship between X1 and Y.

We typically use regression analysis to study such problems. In classical and modern regression analysis, the specific details of the relationship between X1 and Y - and questions on whether X2 or others impacts that relationship can be studied using multiple regression.

In multiple regression, one can also specific interactions among any of the independent variables X2… Xn and study the impact of including not just, say X2 on the relationship between X1 and Y, but also the nature of any interaction between X1 and X2 and their JOINT impact on Y.

Take, for example, fluoride (X1), aluminum (X2), and intelligence (Y).

In a multiple linear regression, we might compare the models

intelligence (Y) = b1x1 + i + e (model 1)

intelligence (Y) = b2x2 + i + e (model 2)

intelligence (Y) = b1x1 + b2x2 + i + e (model 3)

where i is the model intercept (where the model hits Y at all x’s = 0), and e is the measurement error of the model (aka residuals).

If one sees that model 1 yields a significant relationship between aluminum exposure and intelligence, and that model 2 also yields a significant relationship between fluoride exposure and intelligence, it’s possible that model 3 either has no significance for aluminum, or for fluoride, or for both. How is that possible?

That can occur when the model, is not fully specified. One can include the interaction term

intelligence = b1x1 + b2x2 + b3x1x2 + i + e

and then find a significant p-value for b3, and no significance for b1, b2 or both and then we know that the partial model was misleading.

One would then conclude that aluminum and fluoride have a synergistic toxic effect.

And that’s a powerful tool for generating understanding.

So, given this powerful tool, you'd expect that graduate schools and schools of medicine around the world would encourage graduate-level training in statistical analysis that would empower researchers to flourish and do well at teasing apart these relationships.

At least that’s what I’d expect.

To find out, I searched Pubmed for research articles that used the term “multiple regression” and then I searched for research articles that used the terms “multiple regression” and “interaction”. I then calculated for each year the percentage of studies published in Pubmed that used multiple regression and considered the interaction of variables.

Hoping the use of the study of interaction terms would have increased over time, I plotted the results by year. I was devastated when I saw the result.

Since 2008, there has been a steady loss in the percentage of studies that have used multiple regression and that have used or considered the interaction term from above 10% in 2008 to less than 1% in 2022.

The estimated loss has been 0.69% per year. In 2023, there is expected to be NO study of interactions. This means the expenditure on research by Congress has become a calamitous waste.

Biomedical and clinical research of all types is being dumbed down via the adoption of incomplete model specification. This is horrific because it means that the “knowledge” base we all rely upon to understand complex relationships, such as possible interactions between cancer treatments, interactions between lifestyle and diet, and interactions between and among drugs… is decaying. Interactions are just no longer being studied.

It’s easy to choose a pet X1 and manipulate the p-value via model selection by including any of {X2…Xn} to make a significant regression parameter lose significance - as if the addition of the second parameter’s effect then “explains” the causality of the first parameter better. This “parameter shopping” is akin to p-hacking, and we’ve seen it in vaccine studies over the years.

Everyone who reads this will be at various stages of understanding these issues. We have a Biostats class at IPAK-EDU which will be offered again in the Spring of 2023 if we have enough students. Doing multiple regression with interaction terms is fairly straightforward; all major medical research Universities have biostatistics departments that should be helping their clinicians design studies in a manner that fosters the study of interaction in an appropriate manner, such as being sufficiently powered (large enough to study interaction terms), and using objective model selection criteria.

It’s my earnest hope that this message will reach mainstream academic researchers so they may become inspired to do better and to make the expenditure of their time and resources more worthwhile.

I doubt it will, so perhaps I should call for funding for our research @ IPAK, where we do use interaction terms.

Nevertheless, please share this with anyone you know who does statistical analysis, or who performs research of any kind involved human subjects who might help turn the curve upwards in terms of actual knowledge discovery. Or, click on the image below to sign up for our classes in Analytics @ IPAK-EDU and get your brain on!