Let’s Take the Con Out of Randomized Control Trials in Development - Pritchett, Lant (Center for International Development, Harvard University, 2021)
The enthusiasm for the potential of RCTs in development rests in part on the assumption that the use of the rigorous evidence that emerges f rom an RCT (or from a small set of studies identified as rigorous in a “systematic” review) leads to the adoption of more effective policies , programs or projects. However, the supposed benefits of using rigoro us evidence for “evidence based” policy making depend critically on th e extent to which there is external validity. If estimates of causal i mpact or treatment effects that have internal validity (are unbiased) in one context (where the relevant “context” could be country, region, implementing organization, complementary policies, initial conditions , etc.) cannot be applied to another context then applying evidence th at is rigorous in one context may actually reduce predictive accuracy in other contexts relative to simple evidence from that context—even i f that evidence is biased (Pritchett and Sandefur 2015). Using empiric al estimates from a large number of developing countries of the differ ence in student learning in public and private schools (just as one po tential policy application) I show that commonly made assumptions abou t external validity are, in the face of the actual observed heterogene ity across contexts, both logically incoherent and empirically unhelpf ul. Logically incoherent, in that it is impossible to reconcile genera l claims about external validity of rigorous estimates of causal impac t and the heterogeneity of the raw facts about differentials. Empirica lly unhelpful in that using a single (or small set) of rigorous estima tes to apply to all other actually leads to a larger root mean square error of prediction of the “true” causal impact across contexts than j ust using the estimates from non-experimental data from each country. In the data about private and public schools, under plausible assumpti ons, an exclusive reliance on the rigorous evidence has RMSE three tim es worse than using the biased OLS result from each context. In making policy decisions one needs to rely on an understanding of the relevan t phenomena that encompasses all of the available evidence.