John Nunnery, Professor of Higher Education at Old Dominion University, just published:
Nunnery, J. A. (2026). Workflow-induced evaluation targets in automation-assisted screening: Implications for cumulative evidence in research synthesis. Research Synthesis Methods. Advance online publication. https://doi.org/10.1017/rsm.2026.10116
Evaluations of automation-assisted abstract screening report recall, precision, and workload reduction, and those results are increasingly pooled across studies. Pooling assumes the studies estimate the same quantity. Often they do not. The evaluation target in a screening study is induced by its workflow: how records are represented, how the model infers, how decisions are governed, how performance is evaluated, and which reference-decision set stands as ground truth. Two studies can report the same metric and estimate different quantities, and standard heterogeneity statistics cannot separate variation in method effectiveness from variation in the estimand. Comparison is warranted only when targets align before pooling. The article gives a framework for specifying targets at the study level and checking configuration at synthesis. It is open access.
Research Synthesis Methods is published by Cambridge University Press. In the 2025 Journal Citation Reports it has a Journal Impact Factor of 8.0 and ranks 3rd of 67 journals in Mathematical & Computational Biology by impact factor and 2nd of 67 by Journal Citation Indicator (98th percentile). It is a Q1 journal in both of its categories.
