PPN Home > Issue Briefs > Evidence Based Practices

Sign up for PPN updates by email

PPN Issue Briefs

What is an Evidence-Based Practice?


by Teryn Mattox (PPN content manager) and M. Rebecca Kilburn (PPN director)


Facing a growing emphasis on accountability for achieving results in the area of children and family services, decisionmakers such as funders and service providers are increasingly making programmatic choices based on the best research evidence. But what is the best available research evidence? And how can decisionmakers without research training make sense of that evidence? Woman using laptop to do research

Programs shown to be effective through rigorous research are known as evidence-based practices (EBPs). EBPs have garnered a new prominence in the field of child and family services in the past decade. Federal funding streams have increasingly been tied to research evidence, particularly around programming related to children and youth. [1]  Concurrent with these trends, the field of child and family studies has become savvier in designing and publishing research studies, with the goal of establishing research evidence in support of programs and practices that improve outcomes for children and their families. However, the research evidence supporting these programs can be of varied quality. This can be problematic, because there is no single set of standards against which EBPs are evaluated in the field of child and family services.

This brief is intended to help PPN visitors understand some of the variation in how the field of child and family services defines "best available research evidence." We describe the history of EBPs and review varying definitions of EBPs, comparing the PPN criteria with those from other organizations.


The Emergence of EBPs


The concept of EBPs first came into use in medicine. The term became commonplace in the medical field starting in the 1990s, where an often-cited definition of EBP was the "conscientious, explicit and judicious use of current best evidence in making decisions about the care of individuals". [2]  This is a reasonable definition, but it raises the question of what constitutes "current best evidence"—the criteria for which have evolved through expert consensus, tradition, theories, and the incentive for pharmaceutical and other companies to demonstrate the effectiveness of their products. Today, research evidence in the medical field has become quite rigorous and plentiful [3-6], so "current best evidence" is of a very high quality. In medicine, it is currently generally accepted that a well-implemented randomized controlled trial is the ideal means by which to establish that an intervention is effective and to minimize the biases that might render a study's conclusions invalid. [7]

Since its emergence from the field of medicine, the EBP concept has been adopted in other fields.

In medicine, evidence-based practice has been defined as the "conscientious, explicit and judicious use of current best evidence in making decisions about the care of individuals" [2]


Evidence-Based Practice in Child and Family Services


In the field of child and family services, a generally accepted standard of what constitutes "current best evidence" has not been established. This may be due in part to the fact that evidence is more difficult to establish in this setting, where ethical and practical constraints often preclude random assignment of individuals to different interventions. Additionally, specifying what exactly the intervention is can be more difficult in this field, given that complex human interactions and relationships, rather than a drug or specific medical procedure, may be the driving force behind a program or practice's effectiveness. This also makes it difficult to isolate the effects of any intervention. Beyond these difficulties, more significant resource constraints exist in the field, where there is less of a profit motive driving organizations to invest in costly efficacy trials. [8-10]  Nevertheless, in the past decade, child and family services, ranging from child welfare to teen substance abuse prevention, have begun to actively participate in the evidence-based practice movement, with varying degrees of success. [9] [11-13]

Given the challenge of conducting randomized trials in child and family services, what constitutes "current best evidence" will necessarily be different than it is in the medical field. Thus, the child and family services field has been somewhat flexible in defining what should constitute an evidence base. This flexibility is reflected in an increasing number of definitions of what constitutes an EBP, leading to some ambiguity and debate around the meaning of "best evidence" in this context. [14-15]  At the same time, the EBP movement in medicine has influenced the field of child and family services by setting a high standard for the quality of research evidence and, in that way, encouraging more rigorous research. [16]

Along with the proliferation of EBP definitions has come, unsurprisingly, a proliferation of programs and practices being labeled, or labeling themselves, as "evidence-based." With policymakers, funders, and others relying on an evidence-based designation for decisionmaking, this self-labeling and the lack of standardized evidence criteria can be problematic. This is particularly true in the era of the Internet, when a quick search for "evidence-based practice for children and family" yields tens of millions of results with highly variable quality. There is thus a need for this significant quantity of information to be filtered for quality in order for the EBP designation to have any utility. [16]

PPN and other resources that list EBPs in child and family services have evolved along with the proliferation of EBP information in the field. The What Works Clearinghouse, for example, is a federally sanctioned database of evidence-based programs in the field of education. [17]  Other resources have been developed by professional organizations, states, and other groups. [18]  These resources provide clear evidence rating criteria against which programs can be evaluated for rigor. These resources are often operated by research entities that assess the quality of the evidence and present the information in a way that is useful to decisionmakers.

This brief provides an overview of EBP resources that provide information about effective programs that improve outcomes for children and their families. A complement to the EBP resources that focus on evidence related to effective programs are systematic review projects, such as the Campbell Collaboration (www.campbellcollaboration.org) and the Cochrane Collaboration (www.cochrane.org). The reviews conducted by these projects generally pool data from research on intervention strategies rather than examining one specific program at a time. Examples of the types of topics that have been the subjects of systematic reviews include "Social Skills Training For Children with Learning Disabilities" [19-20] and "Later School Start Times for Supporting the Education, Health and Well-being of High School Students". [20-21]

Among child and family services-focused EBP resources that review specific programs, there are some significant differences in how the programs are evaluated. Some of these differences reflect the quantity and quality of research in the specific topic area of the EBP resource. For example, some subfields, such as infant foster care, have few randomized trials and few studies with long-term follow-up, necessitating a different standard for the "current best evidence" compared with home visiting, where there are many more research studies that use randomized trials and other more-rigorous research methods. Some EBP resources, such as PPN, handle this issue by having multiple tiers of evidence. That is, these resources include studies meeting less-rigorous criteria but place such studies in a lower-criteria category (such as "promising" instead of "proven," in the case of PPN). This is intended to maximize the "current best evidence" information accessible to users while also making clear by the categorization that the evidence is viewed more cautiously.

One of the goals of this brief is to familiarize PPN users with the rigorous EBP resources that include evidence related to U.S. child and family services. With a better understanding of the similarities and differences between these EBP resources, users can make better use of the full range of high-quality evidence that is available. The EBP resources discussed in this article are included in the box below. All of the resources cited use evidence criteria that would meet at least the lowest tier of evidence, the "promising" rating, on the PPN site.

Evidence-Based Practice Resources on Child and Family Services


Promising Practices Network (PPN) on Children, Families and Communities
PPN is a group of individuals and organizations who are dedicated to providing quality evidence-based information about what works to improve the lives of children, families, and communities.

What Works Clearinghouse
An initiative of the U.S. Department of Education's Institute of Education Sciences, the What Works Clearinghouse was created in 2002 to be a central and trusted source of scientific evidence for what works in education.

Home Visiting Evidence of Effectiveness (HomVEE)
The Department of Health and Human Services launched HomVEE to conduct a thorough and transparent review of the home visiting research literature and provide an assessment of the evidence of effectiveness for home visiting programs models that target families with pregnant women and children from birth to age 5.

Office of Juvenile Justice and Delinquency Prevention's (OJJDP's) Model Programs Guide
The OJJDP Model Programs Guide is designed to assist practitioners and communities in implementing evidence-based prevention and intervention programs that can make a difference in the lives of children and communities.

California Evidence Based Clearinghouse for Child Welfare (CEBC)
CEBC provides child welfare professionals with easy access to vital information about selected child welfare-related programs.

Substance Abuse and Mental Health Services Administration's (SAMHSA's) National Registry of Evidence Based Programs and Practices (NREPP)
NREPP is a searchable online registry of more than 250 interventions supporting mental health promotion, substance abuse prevention, and mental health and substance abuse treatment.

Blueprints for Violence Prevention (Blueprints)
The Blueprints mission is to identify truly outstanding violence and drug prevention programs that meet a high scientific standard of effectiveness.

ChildTrends' Lifecourse Interventions to Nurture Kids Successfully (LINKS)
LINKS summarizes evaluations of out-of-school time programs that attempt to enhance children's development.

Coalition for Evidence-Based Policy (CEBP)
CEBP seeks to increase government effectiveness through the use of rigorous evidence about what works in social interventions.

EBP resources vary according to the evidence required to accept a program or practice. For instance, some resources require a program or practice to offer replication materials or have demonstrated that replication is feasible. They can also differ on whether they admit practices or programs with evaluations that have any weaknesses. For example, some federal EBP resources, such as the What Works Clearinghouse and HomVEE, use a more rigid formula than other groups. [22]  This exactitude is beneficial as a means of ensuring consistency, but in certain cases may also lead to rejection of high-quality studies on technicalities. [23]

While the specifics may differ, substantial similarities remain across EBP resources in the requirements for a program to be labeled "evidence-based," "proven," a "model program," etc. In Table 1, we identify the requirements for the top tier of evidence for each of the major EBP resources on a number of dimensions:

  • Topical focus — All EBP resources restrict the programs that they include to those that target a certain set of outcomes or target groups. PPN, for example, includes only programs affecting children and families, while OJJDP includes only juvenile justice or delinquency programs.
  • Research design — A program evaluation's design affects the legitimacy of its results. Randomized controlled trials are generally accepted as the research design that best minimizes threats to the validity of a study's findings, but other designs may be acceptable, including quasi-experimental designs, nonrandomized controlled trials, prospective or retrospective cohort studies, cross-sectional studies, case control studies, case series and registries, and case reports. [7]
  • Statistical significance — Statistical significance is a measure of how certain evaluators are that the effects observed for the group that received the treatment are different from those for a group that did not receive the treatment. The threshold for statistical significance is one area where most EBP definitions are in agreement.
  • Practical significance — It is possible for a measured outcome to be statistically significant but too small in size to actually matter in practical terms. For example, a preschool program may be shown to increase by one letter the number of letters a child can identify. While this may be statistically significant, knowing one more letter of the alphabet may not be practically significant given the costs of the program. Practical significance can be subjective, so "effect sizes," a statistical measure of the size of a program impact, are typically used.
  • Attrition — Attrition occurs when study participants drop out of the research study over time. They might drop out of the treatment group, the nontreatment group, or both. Attrition can compromise a research study—for example, by changing the composition of the participants so that the individuals remaining in the study no longer represent the population that the intervention aims to affect.
  • Quality of outcome measures — Evaluated programs monitor changes in outcome measures to establish that the program is effective. Outcomes can be measured in various ways, such as child hospitalizations, teenage substance use, or family functioning. Some EBP resources require that the outcome measures used have been established in the academic literature as valid measures (that is, the measures assess what they say they do) and exhibit other desirable properties, such as consistency across repeated measurement. Additionally, some resources require that measures reflect a change in some outcome measure of real value rather than just participation in the intervention.
  • Publication or authorship requirements — Some EBP resources require that program evaluation findings be published in a peer-reviewed journal. Others require that the evaluation be independently conducted by individuals other than the program developers.
  • Replication — An initial program evaluation shows that a program is effective in that particular context, but a successful replication of the program shows that the program can be effective in other contexts. Some EBP resources require that evaluation findings have been replicated in order to achieve the highest tier of evidence.

As can be seen in Table 1, the EBP resources vary in the criteria used to make a "highest-tier" evidence designation. Additionally, in making their judgments, some resources rely exclusively on very detailed and explicit requirements for each of the criteria, while others also employ expert judgment from a review panel.

PPN has reviewed the major EBP resources that provide information on child and family services and programs and identified those that utilize criteria that meet at least the "promising" standard. Links to these resources can be found in the Other Reviewed Programs section of the PPN website. This enables PPN users to access additional sources of EBP information on children and families.


Table 1:  Criteria for Highest-Tier Evidence Designation for EBP Resources on Child and Family Services
(click to enlarge image)


NOTE: This table includes only explicit evidence criteria listed on the site for the highest evidence designation.

Conclusion

Strong research evidence on what works to improve outcomes in the field of child and family services has proliferated in the past decade and a half along with EBP resources to help policymakers and others evaluate the burgeoning research. The growth in the quantity of research and EBP resources has resulted in a multitude of definitions regarding what qualifies as "evidence." While there is debate on what constitutes an EBP, definitions have a number of evidence criteria in common. Decisionmakers can use resources such as PPN and others discussed in this document to identify EBPs. PPN supports this process by providing a list of Other Reviewed Programs along with programs reviewed in-house so that users can find "current best evidence" on programs.

About PPN

The Promising Practices Network (PPN) is a user-friendly website that provides evidence-based information on child, youth, and family policy (see www.promisingpractices.net). The target audience includes policymakers, practitioners, funders, the media, the general public, and others who may or may not have research training. PPN adds value in its ability to screen research evidence and synthesize complex information to make it available in a way that is easy to understand.


References


1.   Jennings, Edward T., and Jeremy L. Hall, Evidence-Based Practice and the Use of Information in State Agency Decision-Making, Washington D.C. : Institute for Federalism and Intergovernmental Relations, 2009.

2.   Sackett, David L., William M. C. Rosenberg, J. A. Muir Gray, R. Brian Haynes, and W. Scott Richardson, "Evidence-Based Medicine: What It Is and What It Isn't," British Medical Journal, Vol. 312, No. 7023, January 1996, pp. 71-72.

3.   Evidence-Based Medicine Working Group. "Evidence-Based Medicine: A New Approach to Teaching the Practice of Medicine," Journal of the American Medical Association, Vol. 268, No. 17, 1992, pp. 2420-2425.

4.   Claridge, Jeffrey A., and Timothy C. Fabian, "History and Development of Evidence-Based Medicine," World Journal of Surgery, Vol. 24, No. 5, March 2005, pp. 547-553.

5.   Guyatt, Gordon H., Andrew D. Oxman, Regina Kunz, Gunn E. Vist, Yngve Falck-Ytter, and Holger J. Schünemann, "What Is 'Quality of Evidence' and Why Is It Important to Clinicians?" British Medical Journal, 2008, pp. 336-995.

6.   Montori, Victor M., and Gordon H. Guyatt, "Progress in Evidence-Based Medicine," Journal of the American Medical Association, Vol. 200, No. 15, 2008, pp. 1814-1816.

7.   Steinberg, Earl P., and Bryan R. Luce, "Evidence Based? Caveat Emptor!" Health Affairs, Vol. 24, No. 1, 2005, pp. 80-92.

8.   Dirkx, John M. "Studying the Complicated Matter of What Works: Evidence-Based Research and the Problem of Practice," Adult Education Practice, Vol. 56, No. 4, August 2006, pp. 273-290.

9.   Rosen, Aaron, "Evidence-Based Social Work Practice: Challenges and Promise," Social Work Research, Vol. 27, No. 4, December 2003, pp. 197-208.

10.   Rubin, Aaron, and Danielle Parrish, "Views of Evidence-Based Practice Among Faculty in Master of Social Work Programs: A National Survey," Research on Social Work Practice, Vol. 17, No, 1, January 2007, pp. 110-122.

11.   Buysse, Virginia, and Patricia W. Wesley, eds., Evidence-Based Practice in the Early Childhood Field, Washington, D.C.: Zero to Three, 2006a.

12.   Buysse, Virginia, and Patricia W. Wesley, "Evidence-Based Practice: How Did It Emerge and What Does It Mean for the Early Childhood Field?" Zero to Three Journal, Vol. 27, No. 2, November 2006b, pp. 50-55.

13.   Chaffin, Mark, and Bill Friedrich, "Evidence-Based Treatments in Child Abuse and Neglect," Children and Youth Services Review, Vol. 26, No. 11, November 2004, pp. 1097-1113.

14.   Mattox, Teryn, and M. Rebecca Kilburn, "Understanding Evidence-Based Information for the Early Childhood Field: Tips from RAND's Promising Practices Network," Zero to Three Journal, Vol. 32, No. 4, 2012, pp. 4-9.

15.   Mullen, Edrward J., and Dabid L. Streiner, "The Evidence for and Against Evidence Based Practice," Brief Treatment and Crisis Intervention, Vol. 4, No. 2, Summer 2004, pp. 111-121.

16.   Johnson, M., and M. J. Austin, Evidence-Based Practice in the Social Services, Berkeley: University of California, Berkeley, School of Social Welfare, 2005.

17.   U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse, website. [As of February 25, 2013: http://ies.ed.gov/ncee/wwc/]

18.   National Early Childhood Technical Assistance Center, "Evidence-Based Practice," website. [As of February 25, 2013: http://www.nectac.org/topics/evbased/evbased.asp]

19.   Schwartz, Jamie, Chad Nye, and Lucy Funderburk, "Social Skills Training For Children with Learning Disabilities,", The Campbell Collaboration Library of Systematic Reviews, website, December 2010. [As of February 26, 2013: http://campbellcollaboration.org/lib/project/129/]

20.   The Campbell Library, The Campbell Collaboration Library of Systematic Reviews, website. [As of February 25, 2013: http://www.campbellcollaboration.org/library.php]

21.   Davison, Colleen M., Lisa Newton, Rob Brown, John Freeman, Lee-Anne Ufholz, and J. David Smith, "Systematic Review Protocol: Later School Start Times for Supporting the Education, Health and Well-Being of High School Students," The Campbell Collaboration Library of Systematic Reviews, website, December 3, 2012. [As of February 26, 2013: http://campbellcollaboration.org/lib/project/230/]

22.   U.S. Department of Health and Human Services, Home Visiting Evidence of Effectiveness, "DHHS Criteria for Evidence-Based Program Models, website. [As of February 25, 2013: http://homvee.acf.hhs.gov/document.aspx?rid=4&sid=19&mid=6#go1]

23.   Schonfeld, Alan H. "What Doesn't Work: The Challenge and Failure of the What Works Clearinghouse to Conduct Meaningful Reviews of Studies of Mathematics Curricula," Educational Researcher, Vol. 35, No. 2, March 2006, pp. 13-21.


About this Issue Brief


This document was produced by the Promising Practices Network (PPN) on Children, Families and Communities and is published online as part of PPN's Issue Brief series. This Issue Brief is available at the following URL:
http://www.promisingpractices.net/briefs/briefs_evidence_based_practices.asp

The Promising Practices Network (http://www.promisingpractices.net) is operated by the RAND Corporation (http://www.rand.org).

Copyright © 2013 Promising Practices Network