It doesn’t just matter if a treatment works. We also need to know how well it works

In my last article, I set out the case for randomised controlled trials, or RCTs, as the surest, most rigorous way to conclusively establish whether or not a proposed new drug works, ie that any benefit is due to the drug and that the benefits exceed any harm.

Well-conducted, large RCTs can tell us a lot more than just a “yes” or “no” answer to the question “Does it work?”. Equally relevant, if also not more important, is the question on how well it works, or the size of the (beneficial) effect.

This effect can be measured in many ways.

Depending on the outcome of primary interest, it could be lives saved, deaths prevented, serious events averted, or quicker time to full recovery. Knowing the size of the effect is important in clinical decisions and armed with the right information, an intelligent patient might well make a reasoned, and perfectly reasonable, trade-off between financial cost, anticipated side effects, and alternative or no treatment.

To take a simple example: If a drug reduced the duration of illness from Covid by a day but cost a great deal of money, then it is a perfectly valid choice to suffer the symptoms for an extra day and use the money not spent on something you really value. A holiday, perhaps, or a good meal.

So, how do we measure the effect of a treatment that works? Perhaps of greater importance, how do we communicate this knowledge to both doctors and patients to aid better and more rational decision making?

Enter the world of NNTs — the number needed to treat

The easiest way to understand trial metrics is to take the simplest situation. And since coronavirus is all the rage, I shall take a recent topical example.

The Oxford recovery trial of dexamethasone in severe Covid generated the following raw data.

If you follow the logic of the analysis, the concept of what I refer to as “trial metrics” becomes clear. At heart, the essential idea is relatively simple, but it leads to a powerful idea in terms of understanding the real worth of a new treatment.

Under the null hypothesis that dexamethasone had no effect on mortality risk, we would have expected, purely by random probability, to have observed 120 deaths in the 324 dexamethasone-treated patients, and 252 in the 683 standard care treated patients.

But what we in fact observed was a large enough departure from that expected result for us to question the null hypothesis. If dexamethasone indeed made no difference, then it is unlikely that we would have seen such a big difference by chance alone. This is what is meant when we say that the result is “statistically significant” . As it happens, the p-value is in fact 0.0004.

But we can go further than that. The box below sets out three other simple metrics that can be calculated.

The “event rate” is simply the percentage of patients in each arm of the trial in which the event of interest — death, in this case — was observed. That is 28 percent in the dexamethasone arm and 40.7 percent in the usual care group.

The relative rate reduction, or RRR, is the relative difference of the two event rates. For the dexamethasone group, the event rate was 31 percent lower than in the usual care group.

The absolute rate reduction, or ARR, is the arithmetic difference between the two event rates: one minus the other. The event rate in the dexamethasone group is 12.7 percentage points lower than the rate in the usual care group.

And now comes the final and most useful metric.: the number needed to treat, or NNT. It is simply the inverse of the ARR.

In our case:

What does the NNT tell us, and how does it help?

Remember that though dexamethasone was shown to reduce the risk of death by a statistically significant amount, it did not eliminate that risk. Despite receiving dexamethasone, 28 percent of patients died. To say that dexamethasone is an effective drug is true — the trial showed that fairly convincingly — but it might give the impression that it prevents every death.

Plainly, it does not. Indeed, it cannot.

The NNT of 8 tells us that if dexamethasone is used in eight patients, then one additional death is prevented. Another way of putting that is to say, “Seven out of eight patients treated with dexamethasone will not benefit from it”.

It is the only drug in all the many clinical trials so far to confer a definite and clear mortality advantage. The size of that effect is a substantial one-third reduction in the risk of dying but even with that effect, seven out of eight patients will not see a benefit attributable to dexamethasone.

To visualise the concept, look at this picture.

Each of the 100 squares represents a patient with Covid needing ventilation — the clinical criterion for recruitment in this trial. The colours denote the outcomes after treatment with dexamethasone.

The 60 green squares represent those who would have survived anyway, even if they had received only standard care. To be even more explicit, even if they had not received dexamethasone.

The 28 red squares represent the patients who died despite receiving dexamethasone, which is, after all, no guarantee of survival. The final 12 orange squares represent those whose deaths were prevented. And because this was a truly randomised trial, we can safely attribute their survival to dexamethasone. Of all the 100 patients represented in this picture, it is only these 12 who benefited from the treatment.

Of course, we can never know in advance which eight patients out of the original 100 would benefit, and so we would advise giving dexamethasone to everyone.

So, a “one-third reduction” in the risk of death translates into an NNT, or number needed to treat, of 8 to prevent one additional death.

An NNT of 8 is actually surprisingly good value. In many chronic diseases like heart disease, we identify risk factors such as hypertension, or a raised blood cholesterol, and we are prescribed pills to swallow every day because we are told that they prevent either death or serious and life-changing events like a stroke, or a heart attack.

Let’s consider the benefit to be gained from treating high blood cholesterol with a statin. This is often recommended because we believe on the basis of large trials that it is effective in reducing the risk of one or more of these serious events.

Now look at these facts:

The number of persons without a history of heart disease who would need to be treated (the NNT) with a statin every day for five years:

• To prevent one heart attack is 104;

• To prevent one stroke is 154

The number of persons with previous heart disease who would need to be treated with a statin every day for five years:

• To prevent one new heart attack is 39;

• To prevent one stroke is 125;

• To prevent one death is 83.

And this is where effectively communicating the real benefit of modern drugs is so helpful to make rational decisions. If a doctor is advising a patient with known heart disease about the benefits of taking a statin, she could perfectly truthfully say, “Statin therapy has been shown to be effective in reducing the risk of death”.

On the other hand, if she knew the numbers behind the RCTs, she could more helpfully say, “If 83 patients with your clinical profile and background risk took a statin every day for five years, one person’s death would be prevented”.

I invite you to ponder this question: Would patients decide differently if the results of RCTs were presented in terms of NNTs?

***

The media must be free and fair, uninfluenced by corporate or state interests. That's why you, the public, need to pay to keep news free. Support independent media by subscribing to Newslaundry today.