In a recently published meta-analysis, the authors reported a significantly higher pneumonia risk among cirrhotic patients exposed to proton pump inhibitors (PPIs) [1]. A second read, however, raises concerns calling for cautious interpretation of the results.
Defining a clinically well-focused and scientifically relevant question, together with the application of strict inclusion/exclusion criteria, is of paramount importance when pursuing high-quality meta-analyses [2]. In this case, the authors evaluated observational studies of different designs, definitions and populations; none of those studies was initially conducted to explore the primary outcome of the meta-analysis. Trying to mathematically harmonize data from irrelevant studies increases the risk of heterogeneity and introduces bias. Moreover, one would expect an adjustment for major cofounders, such as the severity of the underlying liver disease, the dose and duration of PPI administration, and the presence or not of significant comorbidities. However, such an adjustment would be the interest of a meta-regression analysis that is difficult to perform when putting together observational studies. In our opinion, performing a systematic review without a meta-analysis would be more reasonable, since the retrieved data did not fulfill the necessary criteria to calculate a summarized measure effect [3]. Besides, heterogeneity is an issue that must always be anticipated before conducting any analysis. Numerical estimation of heterogeneity may be statistically possible but remains imperfect, since it depends on several parameters [4]. Even if statistical tests fail to demonstrate significant heterogeneity, the quality of the findings may still be undermined, and the authors should a priori present a rigorous sensitivity analysis to investigate it [3].
Significant publication bias was also evident in this study. Publication bias is one of the most powerful sources of bias, appearing when a considerable amount of data has been missed or overlooked [5]. Performing a broad and expert-assisted search across many databases (including also the so-called “gray zone”) is considered mandatory to prevent the omission of references and minimize its possibility. Arguably, the consequences of publication bias have been studied in relation to meta-analyses of randomized controlled trials, while its effect when observational studies are included is less clear [6]. Still, it could be even more important, because the incidence of adverse events may be estimated erroneously, leading to imprecise associations between clinical variables [7]. Instead of simply listing this as a study limitation, the authors should have tried to deal with it effectively, given that this may impact the effect sizes.
Lastly, the authors report as statistically significant the finding that PPI use is associated with greater pneumonia risk (risk ratio 1.36, 95% confidence interval 1.00-1.85). However, as witnessed both by the result itself (P=0.05) and by the corresponding forest plot, where the diamond shape touches the line of no effect, this is incorrect. There is a trend towards a link between PPI use and pneumonia development, but it fails to reach significance. This gives a totally different perspective, raising at the same time concerns about the results’ credibility, magnitude and precision. Whenever a mathematical combination of extracted data is sufficiently justified, implementation of sound statistical methodology, according to established guidance [3], is imperative.
In conclusion, meta-analysis is nothing more than a sophisticated statistical tool allowing us to approach research areas where additional knowledge is warranted and “traditional” trials cannot serve. As their numbers increase exponentially, the need for a reliable guide to assist clinicians and journal reviewers is pertinent [8]. Although the majority of meta-analyses are indeed flawed beyond redemption, when they incorporate the fundamental guideline principles and meet rigorous standards (Fig. 1) they can be immensely useful [9].
Figure 1 Principal issues of systematic reviews and meta-analysis