Research seminar: Heiko Rachinger (University of the Balearic Islands)

Paper: Spurious Precision in Meta-Analysis – Some New Results

Authors: Zuzana Irsova (Charles University), Pedro R. D. Bom (University of Deusto), Tomas Havranek (Charles University), and Heiko Rachinger (University of the Balearic Islands)

Abstract: Meta-analysis upweights studies reporting lower standard errors and hence more precision. But in empirical practice, notably in observational research, precision is not given to the researcher. Precision must be estimated, and thus can be p-hacked to achieve statistical significance. Simulations show that a modest dose of spurious precision creates a grave problem for inverse-variance weighting and bias-correction methods based on the funnel plot. Selection models fail to solve the problem, and the simple mean can dominate sophisticated estimators. Cures to publication bias become worse than the disease. We introduce a novel meta-analysis approach that surmounts spuriousness. The proposed method – meta-analysis instrumental variable estimator (MAIVE) – is a two-step IV estimator in which the inverse sample size serves as the instrument in the first step. The method allows for the instrument to be weak.  We apply the method to two datasets – Kvarven et al. (2019)’s widely used comparison of replication results and meta-analysis estimates of 15 psychology studies and a large dataset of 613 meta-analyses in economics, including 209,766 estimates, compiled by Chris Doucouliagos. In these applications, we, first, show that the instrument in the first stage regression typically is rather strong. Second, in around 70% of the cases the proposed method appears to outperform the alternative standard methods. We also discuss three tests for spurious precision and show their efficacy in a short simulation exercise. Finally, we provide an R code that makes the method operational.  

The seminar will take place on Tuesday, December 5, in Room 105 at 14:00.

If you need a discussion slot with Dr. Raichinger, please write to Prof. Tomas Havranek; he will try to find a suitable time on Tuesday.