Self-Selection and Treatment Assignment in Field Experiments

Abstract

Self-selection and unbalanced treatment assignment are two major concerns in experimental evaluations as they compromise the validity of a study at the external and the internal margin. In this paper we present evidence on the selection of partner institutions into participation and balanced treatment assignment in field experiments. We answer two questions: (1) do stakeholders that choose to participate in a field experiment differ from the population of interest, and (2) does pre-treatment balancedness on observable characteristics translate to lower bias and increased power in a real-world setting, and which method should best be used if that is the case? To this end, we conducted a recruitment experiment, inviting stakeholders to participate in field experiments with their institutions and varying the salience of the research topic and the stakes of participation. We combine this experimental data with a rich set of administrative data on institution and municipality characteristics to identify a possible self-selection bias. Moreover, we compare pure randomization, matching, and re-randomization to a new treatment assignment method – the minimum mean squared error treatment assignment (min MSE) proposed by Schneider and Schlather (2017). We find no evidence for a self-selection bias on observable characteristics and estabhlish that balancedness as achieved by the minMSE method reduces bias of treatment effect estimation by 33% compared to pure randomization. The minMSE method performs best in increasing pre-treatment balancedness of observable characteristics compared to pure randomization, matching, and simple re-randomization.