The White House and Experts Explore Low-Cost Randomized Controlled Trials

Recent­ly I had the oppor­tu­ni­ty to par­tic­i­pate in a con­fer­ence on low-cost ran­dom­ized con­trolled tri­als (RCTs) host­ed by the White House Office of Sci­ence and Tech­nol­o­gy Pol­i­cy and the Coali­tion for Evi­dence-Based Pol­i­cy (CEBP). The con­fer­ence pro­vid­ed an oppor­tu­ni­ty for par­tic­i­pants rep­re­sent­ing gov­ern­ment, non­prof­its, acad­e­mia and phil­an­thropy to explore how to use these types of stud­ies to improve social spend­ing and to hear from the win­ners and final­ists in CEBP’s low-cost RCT com­pe­ti­tion. 

Well-con­duct­ed RCTs pro­duce high­ly cred­i­ble evi­dence about the effec­tive­ness of pro­grams, prac­tices and oth­er inter­ven­tions, yet tra­di­tion­al­ly, RCTs can be quite expen­sive. The win­ning projects show how the cost of these tri­als can be reduced through strate­gic use of exist­ing, reli­able data. The Annie E. Casey Foun­da­tion and the John and Susan Arnold Foun­da­tion sup­port­ed the contest.

The con­fer­ence ses­sions high­light­ed ben­e­fits of low-cost RCTs and also raised some impor­tant ques­tions. Some of the ben­e­fits par­tic­u­lar­ly rel­e­vant to our work at the Casey Foun­da­tion include:

• Con­duct­ing stud­ies at low­er cost max­i­mizes phil­an­thropic invest­ments in evaluation.
• Dis­ag­gre­gat­ing data can deter­mine what inter­ven­tions and poli­cies work best for com­mu­ni­ties of col­or. By using large datasets, many low-cost RCTs allow for data to be bro­ken down by race. The Foundation’s Race for Results report, which includes an oppor­tu­ni­ty index by race and eth­nic­i­ty, under­scores the impor­tance of pin­point­ing such effec­tive approaches. 
• Encour­ag­ing the use of admin­is­tra­tive records low­ers cost and makes max­i­mum use of this exist­ing data. The Casey Foun­da­tion is sup­port­ing and encour­ag­ing the use of inte­grat­ed data sys­tems (IDS) that allow states and coun­ties to link infor­ma­tion from mul­ti­ple pub­lic sys­tems’ data­bas­es to allow for under­stand­ing of how ser­vices impact indi­vid­u­als as well as poten­tial sav­ings and ben­e­fits for gov­ern­ment. For exam­ple, a study that draws upon an IDS might use data from a hous­ing pro­gram and the pub­lic health sys­tem to deter­mine if the pro­gram improves res­i­dents’ health outcomes. 

The meet­ing also raised ques­tions: How can the appro­pri­ate use of low-cost RCTs become more com­mon­place? And, where do low-cost RCTs fit into the evi­dence-build­ing toolbox?

In the inau­gur­al year of the com­pe­ti­tion, CEBP received more than 50 appli­ca­tions, which indi­cates that there are pro­grams and poli­cies across fed­er­al, state and local gov­ern­ments in a vari­ety of fields ripe for a low-cost RCT. While there are researchers with deep expe­ri­ence con­duct­ing RCTs, con­fer­ence speak­ers called for more oppor­tu­ni­ties for younger researchers to be steeped in this new­er approach. As Jim Shel­ton of the U.S. Depart­ment of Edu­ca­tion not­ed, the con­fer­ence atten­dees were the ​“choir.” For this approach to gain even more trac­tion in influ­enc­ing social spend­ing, skep­tics will need to be part of the con­ver­sa­tion about the strengths and lim­i­ta­tions of low-cost RCTs. 

The low-cost RCT has many dif­fer­ent uses, as the com­pe­ti­tion win­ners and final­ists showed: a repli­ca­tion study of a proven pro­gram, a focused study to deter­mine whether a more exten­sive eval­u­a­tion is war­rant­ed and a test of a pol­i­cy ques­tion. Though the con­fer­ence had a laser focus on low-cost RCTs, I was lis­ten­ing to the pre­sen­ta­tions won­der­ing how this par­tic­u­lar type of study fits into a broad­er con­text of evi­dence that gov­ern­ment, non­prof­its and phil­an­thropic orga­ni­za­tions can draw upon to deter­mine how best to achieve orga­ni­za­tion­al goals.