The moderating effects of population characteristics: a potential biasing factor when employing non-random samples to conduct experimental research

Objectives: Due to time and financial limitations, most randomized controlled trials (RCTs) are conducted employing non-random sampling techniques. Although valuable, when the unique characteristics of a non-random sample unknowingly interact with the treatment, the results of the RCT could become biased. Nevertheless, the amount of bias remains unexamined. Methods: The current study evaluated if non-random sampling techniques could bias the slope coefficients of an RCT when an interaction exists between the treatment and a characteristic in the population using two simulation analyses. Results: The results suggested that the sampling distributions of slope coefficients from an RCT — across random specifications — expand drastically when (1) an interaction between the treatment and a characteristic in the population exists and (2) the non-random sample has unique scores on that characteristic. Conclusions: Considering these findings, four recommendations are made for scholars currently or intending to conduct a RCT employing non-random sampling techniques.