Models Gone Wild
Post #1100
This post is not about popular runway models who got caught on video doing outlandish things. Instead, it is about the creation of data models in the attempt to estimate a reality on the ground. Like modelers for climate change, modelers for epidemics are notorious for presenting a worst-case scenario.
Having a worst-case scenario is important but more important is the plausibility on that worst-case scenario. Two ways to be dishonest when creating models are by cherry-picking raw data and then data snooping after you input the raw data. All relevant data is supposed to go into creation of a model, rather just a small part of it.
It doesn’t necessarily mean that every observed data value, everywhere, needs to be entered into the model. If there is an honest way to aggregate much of it, that may be good enough. But it does mean dealing with data extremes honestly. With an infection fatality rate (an IFR), noting the highest value is not as important as the lowest one.
That’s because many things can “increase” (estimate is higher than reality) an IFR estimate. Some examples may be to withhold vital medicine from patients, force dangerous drugs into their system, hold patients inside wards and therefore out of sunlight for long enough (several weeks) to deplete them of vitamin D, etc.
But there are relatively few ways to “decrease” an IFR estimate (estimate is “lower than reality”). The only ways to “decrease” an IFR estimate which come to my mind are both mathematical:
accidentally fail to take note when infected people die on your watch
accidentally overestimate the spread of infection
The first mistake is hard to believe in. It is hard to believe that a doctor would continue saying that the patient is alive once a motionless body becomes cold to the touch and there is no evidence of any brain activity or heart activity. Almost all doctors would “not miss detecting it” if death comes to a known infected patient.
This assumes a prospective study (watching people over time), which is the favored way to investigate in medical science. Trying cross-sectional analysis (snapshot of all deaths, estimating how many were COVID) or retrospective analysis (case-control studies; looking backward in time) are not ideal, and should be avoided when possible.
This leaves just one plausible way to underestimate an IFR: overestimate the number of infections. But early evidence proved that that was not the case. In fact, gross underestimation of infection was the problem:
That leaves approximately zero reasons why an observed IFR estimate may be lower than reality. And it means that the lowest IFR value found holds the most weight, — i.e., it is most likely to be true to the reality on the ground — whereas higher estimates are more likely confounded by extraneous factors like administering poor medical care.
By late-January 2021, the UK had ample data on people tracked over time. Analyzing Alpha variant (“SGTF”) COVID separately, they tracked 65,000 wild-type (Wuhan-1; non-SGTF) COVID infections for 28 days and recorded the deaths. They found an IFR of 0.1% (65 total deaths), which is similar to the flu:
But at Imperial College London, modelers utilized an evidence-contradicting IFR of 0.9%. When observed deaths in the UK get adjusted upward in order to account for any deaths that might have occurred more than 28 days after verified infection, you can determine the range of plausibility for wild-type COVID IFR:
[click to enlarge]
In this simulation, 160 million random IFR values were created and tested to see if they were capable of re-creating the estimated total of 73 deaths from 65,000 wild-type COVID infections. To my knowledge, this is the largest-ever sample size created for estimating a COVID IFR by way of computer simulation (takes almost 20 minutes).
The posterior distribution is the set of all values which hit the mark and therefore successfully created those 73 deaths from those 65,000 wild-type COVID infections. At almost 2,500 values, it was enough to create a 99.98% Credible Interval of the IFR. Out of 160 million attempts, it was those 2,500 values which could explain UK deaths.
When modeling epidemics, modelers should be able to explain — or at least to “explain away” — very low numbers on fatality, such as the low (“flu-like”) numbers from the UK. By ignoring low numbers and cherry-picking raw data, you can make things seem more deadly than they are.
You can also tweak models after the fact — e.g., data snooping — where you adjust parameter assumptions until you find a result that you like. One way to go wrong doing this is by manipulating effect modifiers, such as age, until you tweak them so that they are just right — producing the worst outcome possible from your raw data.
And a big tell would occur if the modeler does not share the guts of the model (the code), showing all of the assumptions that went into the published results. Such a lack of transparency indicates the possibility that behind-the-scenes tweaking was done. To my knowledge, Imperial College London has never published code for the model.
Reference
[ Worldwide underestimate of COVID infections was 16-fold (IFR estimates were up to 16 times higher than they should have been) ] — Vallée A. Underestimation of the number of COVID-19 cases, an epidemiological threat. Epidemiol Infect. 2022 Nov 8;150:e191. doi: 10.1017/S0950268822001728. PMID: 36345838; PMCID: PMC9978169. https://pmc.ncbi.nlm.nih.gov/articles/PMC9978169/
[ “Time-to-death” probability for COVID; using the 95% upper bound of both the mean and the SD of the lognormal model which had fit the actual deaths best (89% of all COVID deaths occur by Day 28) ] — Linton NM, Kobayashi T, Yang Y, et al. Incubation Period and Other Epidemiological Characteristics of 2019 Novel Coronavirus Infections with Right Truncation: A Statistical Analysis of Publicly Available Case Data. Journal of Clinical Medicine. 2020 Feb;9(2). DOI: 10.3390/jcm9020538. PMID: 32079150; PMCID: PMC7074197. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7074197/
[ 65 deaths by Day 28 from 65,000 wild-type COVID infections by 19 Jan 2021; adjusted up to 73 by dividing by 0.89 ] — UK Technincal Briefing #5. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/959426/Variant_of_Concern_VOC_202012_01_Technical_Briefing_5.pdf
[ Imperial College model using an evidence-contradicting IFR of 0.9% ] — Imperial College COVID-19 Response Team. Report 9: Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand. https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf
[ Later COVID model which requires an IFR of about 0.9% again, just like the Imperial College one did ] — Two Years of U.S. COVID-19 Vaccines Have Prevented Millions of Hospitalizations and Deaths. https://doi.org/10.26099/whsf-fp90
Code to independently confirm my findings (just copy & paste it into R):
# Bayes Estimation of Wuhan-1 IFR (73 adjusted deaths from 65000 cases)
# Draw 160 million times from a uniform prior distribution
set.seed(123)
n_draw <- 160000000
prior_rate <- runif(n_draw, 0, 1)
# Define the generative model as a binomial experiment
gen_model <- function(rate) {
deaths <- rbinom(1, size = 65000, prob = rate)
deaths
}
# Simulate the data
deaths <- rep(NA, n_draw)
for(i in 1:n_draw) {
deaths[i] <- gen_model(prior_rate[i])
}
# Filter for those IFR values that resulted in data actually observed
post_ifr <- prior_rate[deaths == 73]
# Count the IFR values that led to the data (73 deaths from 65000 cases)
length(post_ifr)
# Convert IFR to percentage
wuhan1.ifr.percent <- 100*post_ifr
#get 5-Number Summary
summary(wuhan1.ifr.percent)
# get 99.98% Credible Interval (0.01th %-ile, 99.99th %-ile)
quantile(wuhan1.ifr.percent, c(0.0001, 0.9999))
#create histogram to view distribution of values
hist(wuhan1.ifr.percent, main="Plausible IFRs for Wild-type (Wuhan-1) COVID",
xlab="IFR (%)", xlim=c(0,0.9), breaks=seq(0,0.9,0.005),
xaxp=c(x1=0,x2=0.9,n=9), col=rainbow(16) )
# bold vertical dashed lines at 99.98% confidence bounds
abline(v=c(quantile(wuhan1.ifr.percent, 0.0001),
quantile(wuhan1.ifr.percent,0.9999)), lty=c(5,5) )