Evaluating Programs in Complex Systems: New Approaches
Increasingly, human service programs that serve children and families are embedded in complex systems. Creating opportunity for parents and children together means that child-care centers might, for example, be in housing projects whose residences are connected to college programs. Community organizations increasingly provide parenting skills education, workforce development programs and have mental health counselors on site. The goal: program participants experience seamless delivery of multiple services under the umbrella of a single program. But integrating approaches that traditionally have been separate requires a new approach to evaluation as well.
Traditionally, program evaluation relies on isolating intervention effects, determining core components and procedures that can be standardized to a significant degree. But complex systems that provide individualized protocols to meet shifting participant needs challenge these methodologies at a time when interest in identifying evidence-based practices, programs and policies continues to grow — and with it, demand for rigorous program evaluation. As it invests in building the evidence base for two-generation approaches, the Casey Foundation is examining how evaluation methodology can be adapted.
“What happens when traditional evaluation methods are not adequate for providing evidence?” asks T’Pring Westbrook, a senior associate with the Foundation’s Research and Evaluation team. “The emerging theory is that it is not the isolation of effects, but the synergy of multiple effects that leads to significant change. The development of new evaluation methodology is in the very early stages, but some important themes are emerging.”
According to Westbrook, those themes include the following:
- Despite the customization and complexity inherent in a service delivery approach, it’s still critical to define and explicitly articulate the “intervention.”
- The flexibility of the intervention makes it even more important to develop a logic model and/or a comprehensive theory of change.
- Multi-method evaluation designs that draw from implementation science, appropriate qualitative protocols, quantitative approaches and sophisticated statistical analyses are increasingly yielding rich information.
- New approaches to evaluation require an investment in complementary new approaches to measurement.
Casey grantees and others will come to together to discuss these important issues during a panel discussion at 1:45 p.m. Friday, Nov. 10, at the American Evaluation Fall 2017 Research Conference: From Learning to Action. The panel, “New Approaches to Evaluating Complex Social Service Systems,” will highlight evaluation efforts across many areas of social service — integrated education, financial and health services for parents and children and place-based initiatives — and the challenges of evaluating programs that really consist of a mosaic of interventions, partners and activities.
Panelists include consultant Allison Holmes; Kathleen Dwyer, senior social science research analyst at the Administration for Children and Families, U.S. Department of Health and Human Services; Margaret Sullivan, senior researcher at Mathematica Policy Research; and Susan J. Popkin, director, Neighborhoods and Youth Development at the Urban Institute.
Learn more about the panel discussion on complex system evaluation
