The problem of measurement bias in comparing selected subgroups

Estimates of subgroup differences are routinely used as part of a comprehensive validation system, and these estimates serve a critical role, including evaluating adverse impact. Unfortunately, under direct range restriction, a selected mean ((Formula presented.)) is a biased estimator of the population mean (Formula presented.) as well as the selected true score mean (Formula presented.). This is due partly to measurement bias. This bias, as we show, is a factor of the selection ratio, the reliability of the measure, and the variance of the distribution. This measurement bias renders a subgroup comparison questionable when the subgroups have different selection ratios. The selected subgroup comparison is further complicated by the fact that the subgroup variances will be unequal in most situations where the selection ratios are not equal. We address these problems and present a corrected estimate of the mean difference, as well as an estimate of Cohen’s d* that estimates the true score difference between two selected populations, (Formula presented.). In addition, we show that the measurement bias is not present under indirect range restriction. Thus, the observed selected mean (Formula presented.) is an unbiased estimator of selected true score mean (Formula presented.). However, it is not an unbiased estimator of the population mean (Formula presented.). These results have important implications for selection research, particularly when validating instruments.