TY - JOUR

T1 - The problem of measurement bias in comparing selected subgroups

AU - Mendoza, Jorge L.

AU - Lee, Seunghoo

AU - Fife, Dustin

N1 - Publisher Copyright:
© 2020 The British Psychological Society

PY - 2020

Y1 - 2020

N2 - Estimates of subgroup differences are routinely used as part of a comprehensive validation system, and these estimates serve a critical role, including evaluating adverse impact. Unfortunately, under direct range restriction, a selected mean ((Formula presented.)) is a biased estimator of the population mean (Formula presented.) as well as the selected true score mean (Formula presented.). This is due partly to measurement bias. This bias, as we show, is a factor of the selection ratio, the reliability of the measure, and the variance of the distribution. This measurement bias renders a subgroup comparison questionable when the subgroups have different selection ratios. The selected subgroup comparison is further complicated by the fact that the subgroup variances will be unequal in most situations where the selection ratios are not equal. We address these problems and present a corrected estimate of the mean difference, as well as an estimate of Cohen’s d* that estimates the true score difference between two selected populations, (Formula presented.). In addition, we show that the measurement bias is not present under indirect range restriction. Thus, the observed selected mean (Formula presented.) is an unbiased estimator of selected true score mean (Formula presented.). However, it is not an unbiased estimator of the population mean (Formula presented.). These results have important implications for selection research, particularly when validating instruments.

AB - Estimates of subgroup differences are routinely used as part of a comprehensive validation system, and these estimates serve a critical role, including evaluating adverse impact. Unfortunately, under direct range restriction, a selected mean ((Formula presented.)) is a biased estimator of the population mean (Formula presented.) as well as the selected true score mean (Formula presented.). This is due partly to measurement bias. This bias, as we show, is a factor of the selection ratio, the reliability of the measure, and the variance of the distribution. This measurement bias renders a subgroup comparison questionable when the subgroups have different selection ratios. The selected subgroup comparison is further complicated by the fact that the subgroup variances will be unequal in most situations where the selection ratios are not equal. We address these problems and present a corrected estimate of the mean difference, as well as an estimate of Cohen’s d* that estimates the true score difference between two selected populations, (Formula presented.). In addition, we show that the measurement bias is not present under indirect range restriction. Thus, the observed selected mean (Formula presented.) is an unbiased estimator of selected true score mean (Formula presented.). However, it is not an unbiased estimator of the population mean (Formula presented.). These results have important implications for selection research, particularly when validating instruments.

UR - http://www.scopus.com/inward/record.url?scp=85088786548&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85088786548&partnerID=8YFLogxK

U2 - 10.1111/bmsp.12215

DO - 10.1111/bmsp.12215

M3 - Article

AN - SCOPUS:85088786548

JO - British Journal of Statistical Psychology

JF - British Journal of Statistical Psychology

SN - 0007-1102

ER -