When is ATE Enough? Rules of Thumb and Decision Analysis in Evaluating Training Programs

This paper explores the relationship between the theory and practice of program evaluation as it relates to training programs. In practice programs are evaluated by mean-variance comparisons of the empirical distributions of the outcome of interest for the treatment and control programs. Typically, earnings are compared through the average treatment effect (ATE) and its standard error. In theory, programs should be evaluated as decision problems using social welfare functions and posterior predictive distributions for outcomes of interest. This paper considers three issues. First, under what conditions do the two approaches coincide? I.e., when should a program be evaluated based purely on the average treatment effect and its standard error? Second, under more restrictive parametric and functional form assumptions, the paper develops intuitive mean-variance tests for program evaluation that are consistent with the underlying decision problem. Third, these concepts are applied to the GAIN and JTPA data sets.