The design of this study confuses me. Amputation-free survival is an appropriate efficacy endpoint for CLI trials. It helps us to answer the question as to whether our treatment has been effective at keeping our patient alive longer and avoiding an amputation, two things that are very important in the CLI population. However, the question these authors should have been asking on the back of their previous study was “is paclitaxel toxic to the body or to the leg?”. This would mean defining the primary endpoint as either mortality or major amputation. Both of these were evaluated as secondary endpoints and found no association with PCB. Unfortunately, because the authors failed to register their protocol on PROSPERO before the stage of data extraction, as they have done in previous systematic reviews, we will never know whether they designed their study a priori or whether they have trawled data to find a statistically significant endpoint.
The most concerning aspect of this study are that almost half the randomized controlled trials which were included were from unpublished data. There are several problems with including data which has not been peer-reviewed. The first is that event numbers frequently change when the data are finalized and cleaned prior to the publication. That is an issue when a few data anomalies can change the results of a meta-analysis dramatically when you’re looking at a low frequency endpoint like amputation-free survival at 6 to 12 months. We also do not know whether those studies provide a high quality of evidence, and it is impossible to determine the risk of bias within the meta-analysis. A meta-analysis is of value only if the risk of bias is low or moderate. Finally, it has not had review of its statistical methods to determine whether they are rigorous and appropriate. The authors of this meta-analysis claim they included those studies to avoid publication bias, but all three unpublished studies are in the public sphere and are being reviewed with a view toward publication. Would it not have been appropriate to wait for those finalized data sets and determine the quality of evidence?
Unfortunately, while the authors included three unpublished studies, they completely overlooked the most important unpublished study. Keeping aside what I just said about unpublished studies, if you choose to include them, you must include all of them. The 5-year results of IN.PACT DEEP were presented in August at AMP: The CLI Meeting, which fell inside the period of the systematic review. The 12-month results were published previously, so we know those have been peer-reviewed and are of high quality. The authors included 12-month results, so we know it fulfilled their inclusion criteria. As it has 5-year follow up, if we are to believe the long-term toxicity explanation of their 2018 meta-analysis, it is just this study that should be included, yet it was omitted, with the authors claiming that they were “unaware of its existence.” This is one of the strangest things I’ve ever come across in the world of meta-analysis and it defies plausible explanation.
I took steps to repeat their meta-analysis, using exactly the same data for amputation-free survival, making just one change. I updated the IN.PACT results to include the numbers presented in August. This changed the findings dramatically. There was no significant difference in amputation-free survival between the PCB and uncoated balloon arms (RR = 1.08; 95% CI, 0.89-1.31), illustrating how a single omission can completely change the findings and alter the conclusion. Keep in mind that researchers have demonstrated reduced reinterventions with this technology, so we know it has the potential to benefit our patients. Claiming there’s a safety concern based on incomplete data has the potential to harm patients through restriction of the use of the drug-coated device.
I do not believe the findings are sound. However, if we are to believe that there is no safety signal in claudicants at 12 months, based on those same authors’ 2018 meta-analysis, then finding a safety signal at this short-term timepoint in patients with CLI would make no sense. Yes, it would be surprising and difficult to explain.
Unfortunately, the implications for paclitaxel-coated devices are dire. They were already under a safety cloud and this will add further weight to the voices of those who do not believe in this technology. This is why we have to take such great care to perform studies like this properly, leaving no stone unturned. We have a great responsibility as thought leaders and scientific researchers. It is not good enough to perform an important meta-analysis such as this and simply say that you were unaware of the most important study that exists.
We have a similar meta-analysis in press with the Journal of Endovascular Therapy. Ours also examined mortality after the use of paclitaxel-coated devices in patients treated for CLI. We kept the methodology very similar to the 2018 meta-analysis to answer the question about whether there is a mortality safety signal seen in this group of patients. I am hopeful that those results will add value to the ongoing debate.
Ramon L. Varcoe, MBBS, MS, FRACS, PhD
Director of Operating Theatres
Director of the Vascular Institute
Prince of Wales Hospital, Sydney
Associate Professor, University of New South Wales
Disclosures: Varcoe reports he consults for Abbott, Boston, Medtronic, Intact Vascular, Intervene and Surmodics.