Advancing Equity in Evaluation Policy

Posted April 11, 2020, By the Annie E. Casey Foundation

Children in the community

As pol­i­cy­mak­ers and prac­ti­tion­ers make choic­es about how to serve young peo­ple, eval­u­a­tion plays an increas­ing­ly impor­tant role, shin­ing a light on crit­i­cal issues relat­ed to equi­ty and out­comes. But how do we know whether eval­u­a­tion process­es them­selves are equi­table? And how can fed­er­al agen­cies and pro­grams be held account­able for equi­ty in those processes?

With sup­port from the Annie E. Casey Foun­da­tion and the William T. Grant Foun­da­tion, the Forum for Youth Invest­ment recent­ly pub­lished What’s the Role of Equi­ty in Eval­u­a­tion Pol­i­cy?,” a new brief that explores how agen­cies that serve youth use eval­u­a­tion poli­cies to address equi­ty-relat­ed issues — and how those poli­cies can be repli­cat­ed or improved.

What is eval­u­a­tion pol­i­cy, and why does it matter?

Fed­er­al eval­u­a­tion pol­i­cy out­lines prin­ci­ples to guide the research that an agency con­ducts or pays for. With Con­gress’ pas­sage of the Foun­da­tions for Evi­dence-Based Pol­i­cy­mak­ing Act, requir­ing all gov­ern­ment agen­cies to adopt an eval­u­a­tion pol­i­cy, eval­u­a­tion pol­i­cy will now play an expand­ing role in policymaking.

Effec­tive pol­i­cy­mak­ing relies on evi­dence, and evi­dence relies on research. But the fram­ing of research ques­tions — and the selec­tion of research meth­ods — can pro­mote (or hin­der) equi­ty. For exam­ple, a research ques­tion can advance equi­ty by antic­i­pat­ing that dif­fer­ent groups of peo­ple in a study may be affect­ed by an inter­ven­tion in dif­fer­ent ways, and tai­lor the ques­tion across groups. Atten­tion to equi­ty in eval­u­a­tion pol­i­cy can influ­ence who gets to help define research ques­tions and inter­pret find­ings, and whether prac­ti­tion­ers and young peo­ple them­selves have a role.

As a start­ing point to unpack how fed­er­al eval­u­a­tion pol­i­cy address­es equi­ty, What’s the Role of Equi­ty in Eval­u­a­tion Pol­i­cy?” looks at how agen­cies address equi­ty with­in five com­mon high-lev­el prin­ci­ples that agen­cies already use: rig­or, rel­e­vance, inde­pen­dence, trans­paren­cy and ethics.

How are agen­cies address­ing equi­ty now?

The brief uses agency exam­ples to show how the exist­ing prin­ci­ples have room to mean­ing­ful­ly incor­po­rate equi­ty and how some agen­cies already describe the prin­ci­ples in ways that clear­ly relate to and align with equi­ty. For example:

  1. Rig­or. Many agency poli­cies rec­og­nize rig­or across an array of method­olo­gies, not just a sin­gle gold-stan­dard method­ol­o­gy. Allow­ing for mul­ti­ple eval­u­a­tion meth­ods is an equi­ty issue because it affects the abil­i­ty to under­stand and address the needs of diverse pop­u­la­tions, pro­grams, set­tings and audi­ences for research.
     
  2. Rel­e­vance. What is an eval­u­a­tion for? Whose inter­ests does an eval­u­a­tion address? Poli­cies that embrace the com­mu­ni­ties or pop­u­la­tions being stud­ied as crit­i­cal in deter­min­ing eval­u­a­tion pri­or­i­ties and ques­tions pro­mote equi­ty by invit­ing diverse voic­es to inform plan­ning and process­es. This greater diver­si­ty can in turn help to ensure that sub­se­quent pol­i­cy deci­sions serve as many peo­ple as pos­si­ble — includ­ing mar­gin­al­ized groups.
     
  3. Inde­pen­dence. Inde­pen­dence does not mean dis­miss­ing the voic­es of peo­ple and com­mu­ni­ties involved in an eval­u­a­tion but rather, as some agency poli­cies acknowl­edge, strik­ing a bal­ance so that research can equi­tably address the ques­tions, needs and con­cerns of the peo­ple whose out­comes are being studied.
     
  4. Trans­paren­cy. As part of a com­mit­ment to trans­paren­cy, many agen­cies com­mit to pub­licly shar­ing their find­ings, regard­less of whether these find­ings are pos­i­tive or neg­a­tive. The Cen­ters for Dis­ease Con­trol offer an exam­ple of how agen­cies can go fur­ther through pol­i­cy that man­dates equi­table eval­u­a­tion prod­ucts and processes.
     
  5. Ethics. Agen­cies are advised to respect the basic rights and dig­ni­ty of study par­tic­i­pants when con­duct­ing eval­u­a­tions — for instance, main­tain­ing their pri­va­cy and pri­or­i­tiz­ing their well-being. Ethics are cru­cial to equi­ty and pre­vent eval­u­a­tion efforts from caus­ing harm, par­tic­u­lar­ly to com­mu­ni­ties affect­ed by struc­tur­al racism and his­tor­i­cal inequities.

How­ev­er, the paper’s analy­sis reveals that, trou­bling­ly, agency poli­cies lack explic­it ref­er­ences to equi­ty. Com­bined with a scarci­ty of ref­er­ences to sim­i­lar con­cepts like fair­ness, cul­tur­al com­pe­tence and diver­si­ty, this absence makes it hard to tell how agen­cies approach equi­ty, whether equi­ty is a pri­or­i­ty and whether com­mu­ni­ties are equipped to hold agen­cies account­able for using equi­table approach­es in evaluation.

Rec­om­men­da­tions for pro­mot­ing equi­ty at the fed­er­al level

The paper main­tains that there should be no doubt for agen­cies that equi­ty mat­ters in eval­u­a­tion and that agen­cies are account­able for equity.

With the appoint­ment of eval­u­a­tion offi­cers across gov­ern­ment agen­cies and the increased role of eval­u­a­tion pol­i­cy in pol­i­cy­mak­ing, the fed­er­al gov­ern­ment has an oppor­tu­ni­ty to pro­mote equi­ty sys­tem­i­cal­ly and on a broad scale. In its brief, the Forum for Youth Invest­ment makes the fol­low­ing rec­om­men­da­tions for fed­er­al gov­ern­ment lead­ers as they approach eval­u­a­tion policy:

  • Strike a tone that pri­or­i­tizes equi­ty. Draft eval­u­a­tion pol­i­cy using lan­guage that pro­motes equi­ty as an impor­tant value.
  • Set min­i­mum expec­ta­tions. Make equi­ty one of the eval­u­a­tion policy’s prin­ci­ples, or include it specif­i­cal­ly with­in oth­er prin­ci­ples. Be clear about the ways equi­ty is pro­mot­ed in the eval­u­a­tion poli­cies and set bench­marks to hold the agency accountable.
  • Estab­lish key prac­tices. Write an oper­a­tional guide to artic­u­late eval­u­a­tion pol­i­cy, show how prin­ci­ples can be put into prac­tice and cod­i­fy stan­dards for every­one in the orga­ni­za­tion. Agen­cies like USAID have used oper­a­tional guides along­side eval­u­a­tion poli­cies to clar­i­fy equi­ty expectations.

As evi­dence increas­ing­ly influ­ences which ser­vices and pro­grams are pro­vid­ed for young peo­ple across the nation, fed­er­al agen­cies have an impor­tant role to ensure that eval­u­a­tion poli­cies are designed and imple­ment­ed to advance equi­ty,” said Ilene Berman, a senior asso­ciate with the Casey Foundation’s Evi­dence-Based Prac­tice Group. This brief pro­vides valu­able guid­ance toward that goal.”

Learn more about embed­ding equi­ty prin­ci­ples in social sci­ence research

Popular Posts

View all blog posts   |   Browse Topics