When Science Fails: Surrogate endpoints and wrong conclusions

 

Galileo Galilei and Anthony Fauci are linked to each other by a chain of events that started at the beginning of modern science, during the 17th century. But the Science that Fauci claimed to represent is very different from that of Galileo. While Galileo studied simple linear systems, modern science attempts to study complex, multi-parameter systems, where the rigid Galilean method just cannot work. The problem is that, while it is obvious that we can measure only what we can measure, that's not necessarily what we want, or need, to measure. Tests based on "surrogate endpoints" may well be the best we can do in medicine and other fields, but we should understand that the results are not, and cannot be, a source of absolute scientific truth.

The Scientific Method

Galileo Galilei is correctly remembered as the father of modern science because he invented what we call today the "scientific method," sometimes still called the "Galilean method." It is supposed to be the basis of modern science; the feature that makes it able to be called "Science" with a capital first letter, as we were told over and over during the Covid pandemic. But what is really this scientific method that's supposed to lead us to the truth? 

Galileo's paradigmatic idea was an experiment about the speed of falling objects. It is said that he took two solid metal balls of different weights and dropped them from the top of the Pisa Tower. He then noted that they arrived at the ground at about the same time. That allowed him to lampoon an ancient authority such as Aristotle for having said that heavier objects fall faster than lighter ones (*). There followed an avalanche of insults to Aristotle that continues to this day. Even Bertrand Russel fell into the trap of poking fun at Aristotle, accused of having said that women have fewer teeth than men. Too bad that he never said anything like that.

It may well be that Galileo was not the first to perform this experiment, and it is not even clear that he actually performed it, but that's a detail. The point is that the result was evident, clear-cut, and irrefutable. Later, Newton started from this result to arrive to the assumption that the same force that acted on an apple falling from a tree in his garden was acting on the Moon and the planets. From then on, science was supposed to be largely based on laboratory experiments or, anyway, experiments performed in tightly controlled conditions. It was a major change of paradigm: the basis of the scientific method as we understood it today.

The Pisa Tower experiment succeeded in separating the two parameters that affect a falling body: the force of gravity and the air drag. That was relatively easy, but what about systems that have many parameters affecting each other? Here, let me start with the case of health care, which is supposed to be a scientific field, but where the problem of separating the parameters is nearly impossible to overcome.

The surrogate endpoint in medicine

How can you apply the scientific method in medicine? Dropping a sick person and a healthy one from the top of the Pisa Tower won't help you so much. The problem is the large number of parameters that affect the nebulous entity called "health" and the fact that they all strongly interact with each other. 

So, imagine you were sick, and then you feel much better. Why exactly? Was it because you took some pills? Or would you have recovered anyway? And can you say that you wouldn't have recovered faster hadn't you taken the pill? A lot of quackery in medicine arises from these basic uncertainties: how do you determine what is the specific cause of a certain effect? In other words, is a certain medical treatment really curing people, or is it just their imagination that makes them think so?

Medical researchers have worked hard at developing reliable methods for drug testing, and you probably know that the "gold standard" in medicine is the "Randomized Controlled Test" (RCT). The idea of RCTs is that you test a drug or a treatment by keeping all the parameters constant except one: taking or not taking the drug. It is designed to avoid the effect called "placebo" (the patient gets better because she believes that the drug works, even though she is not receiving it) and the one called "nocebo" (the patient gets worse because he believes that the drug is harmful, even though he is not receiving it). 

An RCT involves a complex procedure that starts with separating the patients into two similar groups, making sure that none of them knows to which group she belongs (the test is "blinded"). Then, the members of one of the two groups are given the drug, say, in the form of a pill. The others are given a sugar pill (the "placebo"). After a certain time, it is possible to examine if the treatment group did better than the control group. There are statistical methods used to determine whether the observed differences are significant or not. Then, if they are, and if you did everything well, you know if the treatment is effective, or does nothing, or maybe it causes bad effects.  

For limited purposes, the RCT approach works, but it has enormous problems. A correctly performed RCT is expensive and complex, its results are often uncertain and, sometimes, turn out to be plain wrong. Do you remember the case of "Thalidomide"? It was tested, found to work as a tranquilizer, and approved for general use in the 1960s in Europe. It was later discovered that it had teratogenic effects on fetuses, and some 10.000 babies in Europe were born without arms and legs before the drug was removed from the market. Tests on animals would have shown the problem, but they were not performed or were not performed correctly. 

Of course, the rules have been considerably tightened after the Thalidomide disaster and, nowadays, testing on animals is required before a new drug is tested on humans. But let's note, in passing, that in the case of the mRNA Covid vaccines, tests on animals were performed in parallel (and not before) testing on humans. This procedure exposed volunteers to risks that normally would not be considered acceptable with drug testing. Fortunately, it does not appear that mRNA vaccines have teratogenic effects. 

Even assuming that the tests are complete, and performed according to the rules, there is another gigantic problem with RCT: What do you measure during the test?  Ideally, drugs are aimed at improving people's health, but how do you quantify "health"? There are definitions of health in terms of indices called QALY (quality-adjusted life years) or QoL (quality of life). But both are difficult to measure and, if you want long-term data, you have to wait for a long time. So, in practice, "surrogate endpoints" are used in drug testing.  

A surrogate endpoint aims at defining measurable parameters that approximate the true endpoint -- a patient's health. A typical surrogate endpoint is, for instance, blood pressure as an indicator of cardiovascular health. The problem is that a surrogate endpoint is not necessarily related to a person's health and that you always face the possibility of negative effects. In the case of drugs used to treat hypertension, negative effects exist and are well known, but it is normally believed that the positive effects of the drug on the patient's health overcome the negative ones. But that's not always the case. A recent example is how, in 2008, the drug bevacizumab was approved in the US by FDA for the treatment of breast cancer on the basis of surrogate endpoint testing. It was withdrawn in 2011, when it was discovered that it was toxic and that it didn't lead to improvements in cancer progression (you can read the whole story in "Malignant" by Vinayak Prasad).  

Consider now another basic problem. Not only the number of parameters affecting people's health are many, but they strongly interact with each other, as is typical of complex systems. The problem may take the form called "polydrug use," and it especially affects old people who accumulate drugs on their bedstands, just like old cars accumulate dents on their bodies. An RCT test that evaluates one drug is already expensive and lengthy; evaluating all the possible combinations of several drugs is a nightmare. If you have two drugs, A and B, you have to go through at least three tests: A alone, B alone, and the combination of A+B. If you have three drugs, you have seven tests to do (A, B, C, AB, BC, AC and ABC). And the numbers grow rapidly. In practice, nobody knows the effects of these multiple drug uses, and, likely, nobody ever will. But a common observation is that when the elderly reduce the number of medicines they take, their health immediately improves (this effect is not validated by RCTs, but that does not mean it is not true. I noted it for my mother-in-law who died at 101). 

The case of Face Masks 

Some medical interventions have specific problems that make RCTs especially difficult. An example is that of face masks to prevent the spreading of an airborne pathogen. Evidently, there is no way to perform a blind test with face masks, but the real problem is what to use as a surrogate end-point. At the beginning of the Covid pandemic, several studies were performed using cameras to detect liquid droplets emitted by people breathing or sneezing with or without face masks. That was a typical "Galilean," laboratory approach, but what does it demonstrate? Assuming that you can determine if and how much a mask reduces the emission of droplets, is this relevant in terms of stopping the transmission of an airborne pathogen? As a surrogate endpoint, droplets are at best poor, at worst misleading.  

A much better endpoint is the PCR (polymerase chain reaction) test that can directly detect an infection. But even here, there are many problems. As an example, consider an often touted study performed in Pakistan that claimed to have demonstrated the effectiveness of face masks. Let's assume that the results of the study are statistically significant (really?) and that nobody tampered with the data (and we can never be sure of that in such a heavily politicized matter). Then, the best you can say is that if you live in a village in Pakistan, if there is a Covid wave ongoing, if the PCR tests are reliable, if the people who wore masks behave exactly like those who don't, and if random noise didn't affect the study too much, then by wearing a mask you can delay being infected for some time, and maybe even avoid infection altogether. Does the same result apply to you if you live in New York? Maybe. Is it valid for different conditions of viral diffusion and epidemic intensity? Almost certainly not. Does it ensure that you don't suffer adverse effects from wearing face masks? Duh! Would that make you healthier in the long run? We have no idea.

The Pakistan study is just one example of a series of studies on face masks that were found to be ill-conceived, poorly performed, inconclusive, or useless in a recent rigorous review published in the Cochrane Network. The final result is that no one has been able to detect a significant effect of face masks on the diffusion of an airborne disease, although we cannot say that the effect is actually zero. 

The confusion about face masks reached stellar levels during the COVID-19 pandemic. In 2020, Tony Fauci, director of the NIAID, first advised against wearing masks, then he reversed his position and publicly declared that face masks are effective and even that two masks are better than just one. Additionally, he declared that the effectiveness of masks is "science" and, therefore, cannot be doubted. But, nowadays, Fauci has reversed his position, at least in terms of mask effectiveness at the population level. He still maintains that they can be useful for an individual "who religiously wears a mask." Now, imagine an RCT dedicated to demonstrating the different results of "religiously" and "non-religiously" wearing a mask. So much for science as a pillar of certainty. 

Surrogate endpoints everywhere

Medicine is a field that may be defined as "science" since it is based (or should be based) on data and measurements. But you see how difficult it is to apply the scientific method to it. Other fields of science suffer from similar problems. Climate science, ecosystem science, biological evolution, economics, management, policies, and others are cases in which you cannot reproduce the main features of the system in a laboratory and, at the same time, involve a large number of parameters interacting with each other in a non-linear manner. You could say, for instance, that the purpose of politics is to improve people's well-being. But how could that be measured? In general, it is believed that the Gross Domestic Product (GDP) is a measure of the well-being of the economy and, hence, of all citizens. Then, it is concluded that economic growth is always good, and that it should be stimulated by all possible interventions. But is it true? GDP growth is another kind of surrogate endpoint used simply because we know how to measure it. But people's well-being is something that we don't know how to measure. 

Is a non-Galilean science possible? We have to start considering this possibility without turning to discard the need for good data and good measurements. But, for complex systems, we have to move away from the rigid Galilean method and use dynamic models. We are moving in that direction, but we still have to learn a lot about how to use models and, incidentally, the Covid19 pandemic showed us how models can be misused and lead to various kinds of disasters. But we need to move on, and I'll discuss this matter in detail in an upcoming post. 

______________________________________________

(*) Aristotle's "Physics" (Book VIII, chapter 10) where he discusses the relationship between the weight of an object and its speed of fall:

"Heavier things fall more quickly than lighter ones if they are not hindered, and this is natural, since they have a greater tendency towards the place that is natural to them. For the whole expanse that surrounds the earth is full of air, and all heavy things are borne up by the air because they are surrounded and pressed upon by it. But the air is not able to support a weight equal to itself, and therefore the heavier bodies, as having a greater proportion of weight, press more strongly upon and sink more quickly through the air than do the lighter bodies."

  

The Rise of the Key Opinion Leaders: the End of Politics as we Know it?

 

In ancient Japan, a "kagemusha" (shadow warrior) was an impersonator who took the aspect and the role of the actual leader. It was simply a decoy to be used in battle but, in our times, the problem for leaders is not so much to avoid bullets but to avoid the much more powerful propaganda techniques that may destroy them. The result is the rise of a new kind of kagemusha, the KOLs (Key Opinion Leaders). The KOLs do not impersonate the true leaders, but express their ideas and plans in public, taking the blame for the failures and the mistakes that may result. The real leaders, instead, remain in the shadows. The KOLs operate in many fields, not just in politics. For instance, they are popular in science. But, their presence in politics is becoming more and more evident. 

I met a top-level KOL (key opinion leader) for the first time in 2005 when I was organizing a conference on energy. The Tuscan Regional Government was sponsoring the conference, and they wanted a high-profile speaker. So, they insisted on inviting Jeremy Rifkin, the author of "The Hydrogen Based Economy" (2001). I disagreed, but they were those who paid for the conference, so we had to invite him. Rifkin wanted $10,000 as a fee, a first-class plane ticket, and VIP treatment. He got all that in exchange for a talk of about 45 minutes, in which he said nothing new or especially interesting. He took no more than a few questions, giving vague answers, then he disappeared, leaving for another conference. He didn't even stay for the speakers' dinner.   

That was at a time when the KOLs were still relatively rare -- I think the acronym didn't even exist. But, over the years, the term is becoming common, even though the term "influencer" remains more frequent. Normally, the term KOL is applied to fashion and performing arts. You see in the image a typical example.

There also exists a line of KOLs operating in science. Carl Sagan (1934–1996) was an early incarnation of the KOL scientist. He started his career as a first-class scientist, but he was among the first to suffer from what was later called the "Sagan Effect." It describes how scientists who move into popularization soon become "celebrity scientists" and tend to neglect or abandon real scientific research. Emil Kirkegaard calls them "the Kardashians of Science." Another example is Neil deGrasse Tyson.  (*) 

Both Sagan and Tyson did a good service to science with their popularization efforts, but that is not always the case. The KOLs may well distort or falsify the results of scientific research, depending on who are their sponsors. It is, simply, corruption. The scientist who takes a few steps into the media, soon discovers that there is good money to be made there. Much more money than what an average scientist can even dream of. Then, they discover that the more time they can dedicate to the media, the more money they can make. Soon, the (former) scientist starts to operate in the "pieceworker mode." They move from one conference to another, from an interview to another, trying to cram as many of them per day as possible. 

How much are the KOLs of science paid? Of course, that depends on rank. I already told you about the fee that Jeremy Rifkin mustered in 2005, today it must be much more for top-level KOLs. I could give you the names of scientists who ask fees of the order of Eur 20,000-30,000 for participating in a conference, and they are not the top stars in their fields. In Italy, the fee of a middle-level virologist during the pandemic, Ilaria Capua, was reported to be $2000 for ten minutes of a TV interview. More famous virologists surely make more. But of course, much of the money provided by the industry for the KOLs is shrouded in consultancies, teaching fees, honoraria, and various perks (conferences in fashionable resort places, for instance). There are also plain salaries that can be very high. As an example, for years, Tony Fauci, director of the National Institute of Allergy and Infectious Diseases, was the highest-paid government employee in the United States with more than $400,000 per year. And he surely had additional sources of income. 

As you can imagine, the corruption problem is especially bad in medicine, where a lot of money can be made by promoting specific products. On this, you may read the book by Peter C. Goetsche "Deadly Medicine and Organized Crimes." (2013). Goetsche is a somewhat controversial figure for his radical stance on several subjects, but his description of the behavior of the KOLs in medicine is both stunning and realistic. Formerly, they may have been good scientists but, at some moment, they switched to the dark side, being paid to promote the products that the industry sells. They are in for the money, there is little else that matters to them.

Now, stop for a moment and think of what all this means. Imagine yourself interviewed on TV: you are paid $200 per minute (could be much more) for just a few minutes. What are you going to say? Chances are that what you'll say will be a concise, clear, and unequivocal version of what you are expected to say by those who pay you. No time for subtleties, no time for describing alternatives, and no time for mentioning uncertainties. If you have more time, say, you give a talk at a scientific meeting, you are also paid much more. Then, will you want to say anything that displeases your sponsors?

All that is bad enough by itself. But science is not independent of politics, as you know, and that may well make things worse. During the COVID crisis that started in 2020, the public was suddenly exposed to a complex and difficult topic that they had never seen before: they were told about statistical data, such things as mortality, lethality, herd immunity, and much more. In the great confusion, they tended to rely on familiar figures that looked trustworthy. Tony Fauci was by far the most visible of them in the US, followed at some distance by figures such as Rochelle Walensky and Francis Collins. They justified and promoted whatever the government thought was a good idea to do. The government, in turn, acted following the advice of these and other KOLs, who were promoting the products of the pharmaceutical industry. Science, KOLs, and money became an unholy mix that created immense damage. But so goes the world. 

The KOLs may now be spilling into politics. The first actor to become a high-rank politician was Ronald Reagan, but he was far from being a "puppet president", although he profited from his experience as an actor to manage his public image. In recent times, though, we are seeing actors becoming frontmen for figures who remain backstage. A good example is Vladimir Zelensky, president of Ukraine. Independently of what you think of what's happening in Ukraine, Zelensky is clearly a modern kagemusha: an actor playing the role of the president. The way he dresses, the short beard, the posture, all are part of a character that could have starred in a movie, except that the war in Ukraine is all too real. Because of the dangers involved in the current situation, it is understandable that the Ukrainian powers that be much prefer a kagemusha as president rather than to appear themselves on stage. In the picture, you see also the French President, Emmanuel Macron, who may have tried to copy Zelensky's warlike style. 

So far, Zelensky remains a relatively isolated case. But it is possible that the KOL fashion will spread to other countries and other leaders. As an example, I can cite Mr. Matteo Salvini in Italy. As the leader of the League, he became deputy prime minister in 2018, and he is still a member of parliament. Salvini is popularly known in Italy as "Captain Nutella," owing to his penchant to present a public image of himself while eating junk food. He was never an actor, but he started his career as a participant in a TV game show, and he does not have much more than that in his professional CV (**). Incidentally, I have the impression that Zelensky took Salvini as a model; the same beard, the same style of dressing in sweatshirts, the same populist rhetoric. Not the Nutella, though. 

I would not be surprised if Salvini, or some other equally shallow kagemusha-style character, will soon take the job of the prime minister of Italy. This winter we are going to see a serious crisis in which many Italians will find themselves without heating at home and with no fuel, no electric power, and no jobs. At that point, the Nutella will hit the fan, as they say. I don't envy the person who will find him or herself in the role of the prime minister at that moment. The job could become as dangerous in Italy as it is in Ukraine now and, as you know, in Italy we already had a case of a prime minister hanged upside-down. Surely, during the coming hard times, the really powerful people will prefer to take a low-profile role. 

So, are we going to have KOLs as leaders everywhere?  The trend is surely visible. But, if you are worried about the end of politics, I can reassure you. In my personal experience, and I have personally known several high-ranking politicians in Italy, they are not actors playing the role of the evil character. They are really evil! And, KOLs or no KOLs, they'll continue ruling us. 

 

(*) On a personal note, my career as KOL was nipped in the bud when I was invited to speak about nuclear energy in a debate on a national channel in 2010. It was a time when the Italian government had big plans for new nuclear plants. The people who had invited me had noted that I was involved in peak oil studies and, from that, they must have deduced that, since I was against fossil fuels, I had to be favorable to nuclear energy. During the debate, I mentioned the problems of the availability of mineral uranium, and I mentioned the "uranium peak." Immediately, they cut me off. Just like that: I disappeared from the screen and the debate went on without me. And they never invited me again. Had I been a little smarter, I could have made some money by becoming a nuclear KOL, but so it goes.

(**) The populist image that Salvini proposes to his constituency doesn't mean that he is dumb. Not at all. On the contrary, he is a smart guy, perfectly able to catch opportunities when they appear. I think he would be better than many others as Prime Minister in Italy.