Toggle Main Menu Toggle Search

Open Access padlockePrints

Simulation-based study design accuracy weights are not generalisable and can still lead to biased meta-analytic inference: Comments on Christie et al. (2019)

Lookup NU author(s): Dr Gavin StewartORCiD

Downloads


Licence

This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).


Abstract

© 2022 The Authors. Journal of Applied Ecology published by John Wiley & Sons Ltd on behalf of British Ecological Society. Variable study quality is a challenge for all the empirical sciences, but perhaps particularly for disciplines such as ecology where experimentation is frequently hampered by system complexity, scale and resourcing. The resulting heterogeneity, and the necessity of subsequently combining the results of different study designs, is a fundamental issue for evidence synthesis. We welcome the recognition of this issue by Christie et al. (2019) and their attempt to provide a generic approach to study quality assessment and meta-analytic weighting through an extensive simulation study. However, we have reservations about the true generality and usefulness of their derived study ‘accuracy weights’. First, the simulations of Christie et al. rely on a single approach to effect size calculation, resulting in the odd conclusion that before-after control-impact (BACI) designs are superior to randomised controlled trials (RCTs), which are normally considered the gold standard for causal inference. Second, the so-called ‘study quality’ scores have long been criticised in the epidemiological literature for failing to accurately summarise individual, study-specific drivers of bias and have been shown to be likely to retain bias and increase variance relative to meta-regression approaches that explicitly model such drivers. Synthesis and applications. We suggest that ecological meta-analysts spend more time critically, and transparently, appraising actual studies before synthesis, rather than relying on generic weights or weighting formulas to solve assumed issues; sensitivity analyses and hierarchical meta-regression are likely to be key tools in this work.


Publication metadata

Author(s): Pescott OL, Stewart GB

Publication type: Article

Publication status: Published

Journal: Journal of Applied Ecology

Year: 2022

Volume: 59

Issue: 5

Pages: 1187-1190

Print publication date: 01/05/2022

Online publication date: 22/03/2022

Acceptance date: 20/10/2020

Date deposited: 20/04/2022

ISSN (print): 0021-8901

ISSN (electronic): 1365-2664

Publisher: Wiley-Blackwell Publishing Ltd.

URL: https://doi.org/10.1111/1365-2664.14153

DOI: 10.1111/1365-2664.14153


Altmetrics

Altmetrics provided by Altmetric


Share