Conclusive Data
Post #1060
The minimal clinically important difference (MCID) is a concept defined as “the smallest difference in a score that patients perceive as beneficial and that would lead, in the absence of harmful side effects or excessive costs, to adaptations in care management”. —Annals of Physical and Rehabilitation Medicine, 2012
After investigating the terrible policy decisions made during COVID, Subcommittee Chairman Wenstrup issued a statement defining how things must be treated differently in the United States when moving forward from the COVID debacle:
In short, responses to pandemics must be guided by conclusive data. But two questions immediate arise:
What constitutes conclusive data?
Which, if any, COVID measures were actually backed by conclusive data?
In science, there is some measurement error that only allows you to give a range of estimates on the effect of something. In the real-world, there is also something called the Minimum Clinically Important Difference (MCID). You might be able to find a difference within your measurement error, achieving statistical significance.
But statistical significance is not good enough for “real-world” policies. To have conclusive data on a measure or protocol means having so much good data that your minimum estimate is beyond the MCID. The MCID is patient-centered, it depends on what it is that people want out of a measure or protocol.
It is not chosen by what it is that “doctors” think is best for people. It is chosen by what it is that “people” think is best for themselves. In economics, when people are offered an amount of money today versus an amount in one month’s time, they want 33% more money ($20 vs. $15) if they have to wait one month in order to receive it:
In other words, their Minimum Financially Important Difference is a gain of 33%. But what about health gains? Do people value health gains when they are even smaller than a 33% improvement from baseline? Yes, people value health gains — and therefore would become willing to use protocols — if they get at least 17% better:
In this example, people needed to see at least a 17% improvement from baseline in the WOMAC arthritis index, before they were willing to admit that they had improved. A similar threshold level of improvement was found in those with COPD who underwent pulmonary rehabilitation (PR), applied to a 6-minute walk test (6MWT):
If you make 350 meters in the walk test for your baseline measurement, and then you improve by 50 meters (you make it to 400 meters before exhaustion), then you’ve improved by 14.3% from baseline. Taking both health improvement estimates into account, a good cutoff for affirmable health gains appears to be 15% improvement.
If a measure or protocol has to improve results by a minimum of 15%, then the lower bound of a confidence interval around the effect measure must be above 15% improvement. When put in terms of relative risk (RR), that means an upper bound that is below 0.85 (for risk reductions) or a lower bound above 1.15 for improvement:
The cutoffs here use 0.75 (25% reduced risk) and 1.33 (33% improvement) but the concept shows that only the confidence intervals in A and F meet the condition of providing “conclusive data.” Only they lie entirely beyond the MCID that is important to people. An example of conclusive data comes from Triple Therapy for ulcers:
As you can see, the top group taking triple antibiotics to kill H. pylori bacteria had gastic ulcers that remained “cured” for 5 years into the future (for 60 months). Not many medicines can “cure” such a high proportion of people for 5 years. Compared to conventional, acid-reducing, therapy (A) — the Triple Therapy is conclusively better.
What about the official Standard of Care measures for COVID? Did they have “conclusive data” supporting their inclusion?
Social Distancing
Social distancing never had any good data behind it, let alone any conclusive data.
Masks
Masks did not have confidence intervals which decisively showed a 15% improvement from baseline (upper bound <0.85). They didn’t even have confidence intervals lying entirely on one side of the line of no difference (RR=1.0):
Paxlovid
Paxlovid (nirmatrelvir & ritonavir) did not “consistently” have confidence intervals lying entirely on the side of at least a 15% benefit from baseline (upper bound <0.85). There was only a low certainty of the evidence (low CoE) and, for all-cause mortality in the hospitalized, the upper bound of the interval even crossed the line of no difference:
Remdesivir
Remdesivir did not have confidence intervals lying entirely on the side of at least a 15% benefit from baseline (lower bound >1.15). The lower bound of the interval even crossed the line of no difference:
Molnupiravir
Molnupiravir did not have confidence intervals lying entirely on the side of at least a 15% benefit from baseline (upper bound <0.85). And given the low mortality in the studied groups, 1,000 people have to get exposed to molnupiravir (and possible adverse effects) before one person benefits:
Baricitinib
Baricitinib did not have confidence intervals lying entirely on the side of at least a 15% benefit from baseline (upper bound <0.85). Though there was enough data where there was a moderate certainty of evidence (moderate CoE):
Tocilizumab
Tocilizumab did not have confidence intervals lying entirely on the side of at least a 15% benefit from baseline (upper bound <0.85):
These Standard of Care measures did not have “conclusive data” backing them.
Reference
[conclusive data required] — https://oversight.house.gov/release/wenstrup-releases-statement-following-dr-faucis-two-day-testimony/
[minimum 33% benefit required for money] — https://econweb.ucsd.edu/~v2crawford/BDTIntertemporalSlides12.pdf
[minimum 17% benefit required for osteoarthritis] — https://www.sciencedirect.com/topics/medicine-and-dentistry/minimal-clinically-important-difference
[minimum 17% benefit required for COPD] — Houchen-Wolloff L, Evans RA. Unravelling the mystery of the 'minimum important difference' using practical outcome measures in chronic respiratory disease. Chron Respir Dis. 2019 Jan-Dec;16:1479973118816491. doi: 10.1177/1479973118816491. PMID: 30789024; PMCID: PMC6323555.
[superiority/inferiority graph] — https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-017-2024-2
[Triple Therapy for ulcers had “conclusive data”] — https://onlinelibrary.wiley.com/doi/full/10.1046/j.1365-2036.16.s2.24.x
[Infectious Diseases Society of America page] — https://www.idsociety.org/practice-guideline/covid-19-guideline-treatment-and-management/#null