Can Statistical Methods Reliably Detect Fraudulent Data? Examining the Utility of P-Value Analyses, Extreme Effect Sizes, GRIM, and GRIMMER

Abstract

Data fraud occurs when one creates fake data (i.e., fabrication) or alters real data (i.e., falsification), often to support a desired research hypothesis. It is detrimental to science and occurs frequently, making it a pressing concern. Fortunately, there exist several statistical tools to detect it. Extant research, however, is largely inconsistent regarding which tools work well,and no research examines how well they differentiate fraudulent articles (containing fake data) from legitimate controls. The present thesis investigated how well four popular methods to detectdata fraud differentiated retracted psychology articles from legitimate controls. I included themethod of extreme effect sizes, p-value analysis, GRIM, and GRIMMER. Extreme effect sizesperformed quite well: standardized effect sizes for retracted articles were noticeably larger than controls. The other methods performed at chance levels or worse. I contend that the method ofextreme effect sizes could provide valuable information during investigations of potentiallyfraudulent studies.