Advancing Equity in Evaluation Policy
As policymakers and practitioners make choices about how to serve young people, evaluation plays an increasingly important role, shining a light on critical issues related to equity and outcomes. But how do we know whether evaluation processes themselves are equitable? And how can federal agencies and programs be held accountable for equity in those processes?
With support from the Annie E. Casey Foundation and the William T. Grant Foundation, the Forum for Youth Investment recently published “What’s the Role of Equity in Evaluation Policy?,” a new brief that explores how agencies that serve youth use evaluation policies to address equity-related issues — and how those policies can be replicated or improved.
What is evaluation policy, and why does it matter?
Federal evaluation policy outlines principles to guide the research that an agency conducts or pays for. With Congress’ passage of the Foundations for Evidence-Based Policymaking Act, requiring all government agencies to adopt an evaluation policy, evaluation policy will now play an expanding role in policymaking.
Effective policymaking relies on evidence, and evidence relies on research. But the framing of research questions — and the selection of research methods — can promote (or hinder) equity. For example, a research question can advance equity by anticipating that different groups of people in a study may be affected by an intervention in different ways, and tailor the question across groups. Attention to equity in evaluation policy can influence who gets to help define research questions and interpret findings, and whether practitioners and young people themselves have a role.
As a starting point to unpack how federal evaluation policy addresses equity, “What’s the Role of Equity in Evaluation Policy?” looks at how agencies address equity within five common high-level principles that agencies already use: rigor, relevance, independence, transparency and ethics.
How are agencies addressing equity now?
The brief uses agency examples to show how the existing principles have room to meaningfully incorporate equity and how some agencies already describe the principles in ways that clearly relate to and align with equity. For example:
- Rigor. Many agency policies recognize rigor across an array of methodologies, not just a single gold-standard methodology. Allowing for multiple evaluation methods is an equity issue because it affects the ability to understand and address the needs of diverse populations, programs, settings and audiences for research.
- Relevance. What is an evaluation for? Whose interests does an evaluation address? Policies that embrace the communities or populations being studied as critical in determining evaluation priorities and questions promote equity by inviting diverse voices to inform planning and processes. This greater diversity can in turn help to ensure that subsequent policy decisions serve as many people as possible — including marginalized groups.
- Independence. Independence does not mean dismissing the voices of people and communities involved in an evaluation but rather, as some agency policies acknowledge, striking a balance so that research can equitably address the questions, needs and concerns of the people whose outcomes are being studied.
- Transparency. As part of a commitment to transparency, many agencies commit to publicly sharing their findings, regardless of whether these findings are positive or negative. The Centers for Disease Control offer an example of how agencies can go further through policy that mandates equitable evaluation products and processes.
- Ethics. Agencies are advised to respect the basic rights and dignity of study participants when conducting evaluations — for instance, maintaining their privacy and prioritizing their well-being. Ethics are crucial to equity and prevent evaluation efforts from causing harm, particularly to communities affected by structural racism and historical inequities.
However, the paper’s analysis reveals that, troublingly, agency policies lack explicit references to equity. Combined with a scarcity of references to similar concepts like fairness, cultural competence and diversity, this absence makes it hard to tell how agencies approach equity, whether equity is a priority and whether communities are equipped to hold agencies accountable for using equitable approaches in evaluation.
Recommendations for promoting equity at the federal level
The paper maintains that there should be no doubt for agencies that equity matters in evaluation and that agencies are accountable for equity.
With the appointment of evaluation officers across government agencies and the increased role of evaluation policy in policymaking, the federal government has an opportunity to promote equity systemically and on a broad scale. In its brief, the Forum for Youth Investment makes the following recommendations for federal government leaders as they approach evaluation policy:
- Strike a tone that prioritizes equity. Draft evaluation policy using language that promotes equity as an important value.
- Set minimum expectations. Make equity one of the evaluation policy’s principles, or include it specifically within other principles. Be clear about the ways equity is promoted in the evaluation policies and set benchmarks to hold the agency accountable.
- Establish key practices. Write an operational guide to articulate evaluation policy, show how principles can be put into practice and codify standards for everyone in the organization. Agencies like USAID have used operational guides alongside evaluation policies to clarify equity expectations.
“As evidence increasingly influences which services and programs are provided for young people across the nation, federal agencies have an important role to ensure that evaluation policies are designed and implemented to advance equity,” said Ilene Berman, a senior associate with the Casey Foundation’s Evidence-Based Practice Group. “This brief provides valuable guidance toward that goal.”
Learn more about embedding equity principles in social science research
