Recently I had the opportunity to participate in a conference on low-cost randomized controlled trials (RCTs) hosted by the White House Office of Science and Technology Policy and the Coalition for Evidence-Based Policy (CEBP). The conference provided an opportunity for participants representing government, nonprofits, academia and philanthropy to explore how to use these types of studies to improve social spending and to hear from the winners and finalists in CEBP’s low-cost RCT competition.
Well-conducted RCTs produce highly credible evidence about the effectiveness of programs, practices and other interventions, yet traditionally, RCTs can be quite expensive. The winning projects show how the cost of these trials can be reduced through strategic use of existing, reliable data. The Annie E. Casey Foundation and the John and Susan Arnold Foundation supported the contest.
The conference sessions highlighted benefits of low-cost RCTs and also raised some important questions. Some of the benefits particularly relevant to our work at the Casey Foundation include:
- Conducting studies at lower cost maximizes philanthropic investments in evaluation.
- Disaggregating data can determine what interventions and policies work best for communities of color. By using large data sets, many low-cost RCTs allow for data to be broken down by race. The Foundation’s Race for Results report, which includes an opportunity index by race and ethnicity, underscores the importance of pinpointing such effective approaches.
- Encouraging the use of administrative records lowers cost and makes maximum use of this existing data. The Casey Foundation is supporting and encouraging the use of integrated data systems (IDS) that allow states and counties to link information from multiple public systems’ databases to allow for understanding of how services impact individuals as well as potential savings and benefits for government. For example, a study that draws upon an IDS might use data from a housing program and the public health system to determine if the program improves residents’ health outcomes.
The meeting also raised questions: How can the appropriate use of low-cost RCTs become more commonplace? And, where do low-cost RCTs fit into the evidence-building toolbox?
In the inaugural year of the competition, CEBP received more than 50 applications, which indicates that there are programs and policies across federal, state and local governments in a variety of fields ripe for a low-cost RCT. While there are researchers with deep experience conducting RCTs, conference speakers called for more opportunities for younger researchers to be steeped in this newer approach. As Jim Shelton of the U.S. Department of Education noted, the conference attendees were the “choir.” For this approach to gain even more traction in influencing social spending, skeptics will need to be part of the conversation about the strengths and limitations of low-cost RCTs.
The low-cost RCT has many different uses, as the competition winners and finalists showed: a replication study of a proven program, a focused study to determine whether a more extensive evaluation is warranted and a test of a policy question. Though the conference had a laser focus on low-cost RCTs, I was listening to the presentations wondering how this particular type of study fits into a broader context of evidence that government, nonprofits and philanthropic organizations can draw upon to determine how best to achieve organizational goals.