Vaccination 2 Ways

The Failure of Complex Systems

“Complex systems are systems where the collective behavior of their parts entails emergence of properties that can hardly, if not at all, be inferred from properties of the parts.”

Complex systems have many moving parts and variables, changing as components interact, creating intended and unintended consequences. They are systems “that are more than the sum of their parts.” Can we agree that healthcare and the public health effort we call the COVID vaccination effort are complex systems?

To better understand how complex systems fail, we should consider the work of Dr. Richard Cook, an anesthesiologist, and Professor of Healthcare System Safety, Royal Institute of Technology (Sweden). He wrote a seminal paper on “How Complex Systems Fail,” with several salient guides to our analysis.

• “…evaluations based on such reasoning as ‘root cause’ do not reflect a technical understanding of the nature of failure but rather the social, cultural need to blame specific, localized forces or events for outcomes.” – Complex systems are intrinsically hazardous, so multiple layers of protection are built into the system. Each safeguard works in isolation but, when combined, may have unintended consequences – the sum is greater than the parts. It is rare, if at all, that there is one cause of failure – failure is a link of many “necessarily insufficient” reasons.
• “Knowledge of the outcome makes it seem that events leading to the outcome should have appeared more salient to practitioners at the time than was actually the case.” – Hindsight is always 20-20. The discussion of our initial medical responses to COVID in the Northeast, the institution of lockdowns, masking, and social distancing reflect hindsight bias - “outcome knowledge poisons the ability of after-accident observers to recreate the view of practitioners before the accident of those same factors.”
• Humans are “the adaptable element of complex systems” – while participants may restructure “the system in order to reduce exposure of vulnerable parts to failure,” we can just as easily facilitate failure.
• Making systems “safer” paradoxically introduces new forms of failure – especially new technologies, such as an mRNA vaccine, or online vaccination scheduling. Change creates “opportunities for new, low frequency but high consequence failures. …These new forms of failure are difficult to see before the fact; attention is paid mostly to the putative beneficial characteristics of the changes.

These concepts are the grains of salt that need to be applied to the discussions of the pandemic and future preparedness.

The Science of Applying Science

John List, the author of The Voltage Effect, is an economist whose work focuses on scaling small research ideas into big policy. His research provides analytic strength in understanding the process of taking a research idea, the COVID vaccines, from clinical trials to national and global rollouts. Dr. List points to five stumbling blocks along the way.

• False Positives – A research study that appears to demonstrate a positive result, but the result is incorrect for various reasons.

"The ones that get very sick and go to the hospital are the ones that don't take their vaccine. But it's still their choice, and if you take the vaccine, you are protected."

-President Trump

The vaccines are effective, perhaps not in completely immunizing us from COVID, but certainly in lessening the risk of hospitalization and death.

• Know your audience – “…in the scientific marketplace, researcher incentives dictate a subject pool choice that is more likely to find larger treatment effects than a random sample would support.”

         The individuals chosen to participate in the clinical trials were selected to be representative of Americans based upon gender, age, and ethnicity. But, as with many clinical trials, they differed from the much larger target population. I would argue a key difference was motivation. Certainly, some trial participants signed on for altruistic, help our brothers and sisters reasons, but most were motivated by fear, desiring the protection from a virus, burning its way through nursing home populations.

       The American public varied considerably in its sense of personal fear; based on the extent of infection in their local area and assessment of their risk. The fearful concerns of the 75-year old in an epicenter, say New Jersey, are quite different than those of a 20 something in Montana. We erred in believing that everyone was equally as motivated to be vaccinated as the trial participants.      

• Is it the chef or the ingredients? – What makes a meal successful?  The ingredients or those involved in cooking? What in the process must be maintained, what is non-negotiable to achieve the result? The vaccination rollout was not dependent upon one individual chef, but there were non-negotiable ingredients. For example, the 15-minute observation period after immunization and the ability to provide emergency care. That was non-negotiable and meant that mass immunization required space and, in many instances, shelter from inclement weather. What works for a neatly scheduled 100 patients in a lab facility does not necessarily work for 1,000 patients in a gymnasium. Scheduling immunization was also non-negotiable; remember the early rollout when pent-up demand met incapable scheduling systems, listening to stories of people spending futile hours getting an appointment only to have the system crash? Of course, the greatest non-negotiable was the need for everyone to be vaccinated to achieve “herd immunity.” Compliance with medical advice has been a problem for decades.
• Spillovers – economists call them externalities; we know them as unintended consequences. Several are particularly salient for vaccination. The participants in the clinical trials and early beneficiaries of vaccination – when it became available – were highly motivated to get protected and they influenced those around them. We had not considered those that were more fearful of the cure than the disease. The hesitant, who are social creatures, like the fearful, are influenced by friends and family. Friends and families constitute networks of individuals. In some instances, we referred to them as pandemic pods; in other instances, as pockets of vaccine hesitancy or anti-vax sentiment. These social forces, amplified through social media, changed the equilibrium from rejoicing at the vaccines’ arrival to the fight to increase participation in vaccination.         
• Cost Trap – To scale vaccinations, their cost had to decrease as we grew in scale. The taxpayer picked up the tab, so we have little to say about the cost. The lockdowns evidenced the rising costs associated with increasing scale. The price of quarantining travelers or positive cases is relatively small, but scale that quarantining to societal levels. Also, there was the Great Barrington Declaration, which suggested that the cost was too great. (Of course, this does reflect a significant degree of hindsight bias)   

Dr. List mentions ways we might improve scaling research into policy.

• Marginal thinking – looking to capture the “low-hanging fruit,” determining how much more effort and cost is needed to persuade a few more people to be vaccinated. We missed a significant opportunity here, treating the vaccine-hesitant as anti-vaxxers – they were not. Black Americans were among the most hesitant, but it was only late in the game that efforts were explicitly directed to them. In New York, public service announcements addressed to this community didn’t air until the past few months. We went after the adamantly opposed when we should have gone for the hesitant.
• Quitting is for winners – it is essential to know when to stop when further attempts are futile, not wasting our fiscal, physical, and mental resources. Again, we made decisions that may yet haunt us - enabling virtue-signaling on both sides of getting vaccinated, allowing a public health decision to be politicized in culture wars. We also chose mandates, which moved many but left many with a confirmation of a government they feared more than COVID.        

Dr. List offers what “science” shows are critical factors in successfully scaling an idea into policy. Dr. Cook reminds us that these complex systems are based on our behavior - they will never be perfect will often let us down, although rarely in the same way. We need resilient systems, not foolproof ones – after all, “Nothing is foolproof to a sufficiently talented fool."