Selectivity and Specificity of Automated Decisions Based on p-value Calculations: A Study Design

AHRD 2019 poster proposing a study design to test how well the R package statcheck detects p-value reporting errors in HRD journals.
Presentation & Talk
Poster
Published

February 1, 2019

Keywords

statcheck, p-value, sensitivity and specificity, research quality, human resource development

Overview

statcheck is an R package that scans articles for reported test statistics and recomputes their p-values, so that inconsistencies can be found in a large body of articles without checking by hand. It has been used widely in psychology and has also been criticized for missing tests and raising false alarms.

This poster, with David Passmore and Rose Baker, asks whether statcheck has a useful quality-control role for human resource development (HRD) authors and editors, and lays out a study design to answer the question. It reports a design, not results.

Key points

  • The poster reviews what statcheck does, how it has been used, and the controversy over its accuracy.
  • Proposed design: assemble a census or sample of articles from AHRD journals and manually tally the t, F, r, chi-square, and z statistics used in null hypothesis tests.
  • Record each author’s decision about rejecting the null hypothesis, then run statcheck on the same statistics.
  • Compare the two in a truth table to estimate the sensitivity and specificity of statcheck.
  • An alternative design replaces the author’s decision with p-values recomputed independently of statcheck.

Event

2019 Academy of Human Resource Development International Research Conference in the Americas, Louisville, Kentucky, February 2019 (poster session).

Citation

Passmore, D., Chae, C., & Baker, R. M. (2019, February). Selectivity and specificity of automated decisions based on p-value calculations: A study design. Poster presented at the 2019 Academy of Human Resource Development International Research Conference in the Americas, Louisville, Kentucky.