Attikon University General Hospital, Medical School, National and Kapodistrian University of Athens, Greece; Erasme University Hospital, Université Libre de Bruxelles, Brussels, Belgium
aHepatogastroenterology Unit, Second Department of Internal Medicine – Propaedeutic, Research Institute and Diabetes Center, Attikon University General Hospital, Medical School, National and Kapodistrian University of Athens, Greece (Georgios Tziatzios, Konstantinos Triantafyllou); bDepartment of Gastroenterology Hepatopancreatology and Digestive Oncology, Erasme University Hospital, Université Libre de Bruxelles, Brussels, Belgium (Paraskevas Gkolfakis)
In a recently published meta-analysis, the authors reported a significantly higher pneumonia risk among cirrhotic patients exposed to proton pump inhibitors (PPIs) . A second read, however, raises concerns calling for cautious interpretation of the results.
Defining a clinically well-focused and scientifically relevant question, together with the application of strict inclusion/exclusion criteria, is of paramount importance when pursuing high-quality meta-analyses . In this case, the authors evaluated observational studies of different designs, definitions and populations; none of those studies was initially conducted to explore the primary outcome of the meta-analysis. Trying to mathematically harmonize data from irrelevant studies increases the risk of heterogeneity and introduces bias. Moreover, one would expect an adjustment for major cofounders, such as the severity of the underlying liver disease, the dose and duration of PPI administration, and the presence or not of significant comorbidities. However, such an adjustment would be the interest of a meta-regression analysis that is difficult to perform when putting together observational studies. In our opinion, performing a systematic review without a meta-analysis would be more reasonable, since the retrieved data did not fulfill the necessary criteria to calculate a summarized measure effect . Besides, heterogeneity is an issue that must always be anticipated before conducting any analysis. Numerical estimation of heterogeneity may be statistically possible but remains imperfect, since it depends on several parameters . Even if statistical tests fail to demonstrate significant heterogeneity, the quality of the findings may still be undermined, and the authors should a priori present a rigorous sensitivity analysis to investigate it .
Significant publication bias was also evident in this study. Publication bias is one of the most powerful sources of bias, appearing when a considerable amount of data has been missed or overlooked . Performing a broad and expert-assisted search across many databases (including also the so-called “gray zone”) is considered mandatory to prevent the omission of references and minimize its possibility. Arguably, the consequences of publication bias have been studied in relation to meta-analyses of randomized controlled trials, while its effect when observational studies are included is less clear . Still, it could be even more important, because the incidence of adverse events may be estimated erroneously, leading to imprecise associations between clinical variables . Instead of simply listing this as a study limitation, the authors should have tried to deal with it effectively, given that this may impact the effect sizes.
Lastly, the authors report as statistically significant the finding that PPI use is associated with greater pneumonia risk (risk ratio 1.36, 95% confidence interval 1.00-1.85). However, as witnessed both by the result itself (P=0.05) and by the corresponding forest plot, where the diamond shape touches the line of no effect, this is incorrect. There is a trend towards a link between PPI use and pneumonia development, but it fails to reach significance. This gives a totally different perspective, raising at the same time concerns about the results’ credibility, magnitude and precision. Whenever a mathematical combination of extracted data is sufficiently justified, implementation of sound statistical methodology, according to established guidance , is imperative.
In conclusion, meta-analysis is nothing more than a sophisticated statistical tool allowing us to approach research areas where additional knowledge is warranted and “traditional” trials cannot serve. As their numbers increase exponentially, the need for a reliable guide to assist clinicians and journal reviewers is pertinent . Although the majority of meta-analyses are indeed flawed beyond redemption, when they incorporate the fundamental guideline principles and meet rigorous standards (Fig. 1) they can be immensely useful .
Figure 1 Principal issues of systematic reviews and meta-analysis
1. Wongtrakul W, Charoenngam N, Ungprasert P. Use of proton pump inhibitors is associated with a higher risk of pneumonia in cirrhotic patients:a systematic review and meta-analysis. Ann Gastroenterol 2020;33:277-284.
2. Møller MH, Ioannidis JPA, Darmon M. Are systematic reviews and meta-analyses still useful research?We are not sure. Intensive Care Med 2018;44:518-520.
3. Higgins JPT GSe. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. [updated March 2011]. The Cochrane Collaboration;2011:Available from:http://handbook.cochrane.org [Accessed 21 May 2020].
4. Ioannidis JP, Patsopoulos NA, Evangelou E. Uncertainty in heterogeneity estimates in meta-analyses.
5. Dubben HH, Beck-Bornholdt HP. Systematic review of publication bias in studies on publication bias.
6. Golder S, Loke YK, Wright K, Norman G. Reporting of adverse events in published and unpublished studies of health care interventions:a systematic review. PLoS Med 2016;13:e1002127.
7. Wang CH, Li CH, Hsieh R, et al. Proton pump inhibitors therapy and the risk of pneumonia:a systematic review and meta-analysis of randomized controlled trials and observational studies. Expert Opin Drug Saf 2019;18:163-172.
8. Murad MH, Montori VM, Ioannidis JP, et al. How to read a systematic review and meta-analysis and apply the results to patient care:users'guides to the medical literature.
9. Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q 2016;94:485-514.