from the Child Welfare Special Issue Improving the Use and Usefulness of Research Evidence

by Kimberly DuMont and Christine James-Brown

This special issue of Child Welfare focuses on ways to improve the use of research within child welfare in ways that benefit the most vulnerable children and families. By research we mean a type of evidence that comes from applying systematic methods and analyses to investigate a question that is defined before the work begins. This includes descriptive studies, evaluations, syntheses, and cost-benefit studies.

The focus on constructive uses of high-quality research is timely. Limited resources require a need to focus on programs and services that have a strong chance of meeting the needs of children, youth and their families, and we suspect this will require integrating research evidence and the experience and expertise that already existing within child welfare and related child and family serving organizations. In addition, “private and public funders as well as policy makers [are] applying increased pressure to collect and analyze data, incorporate research knowledge, and build evidence of effectiveness” (Yates, Nix, Coldiron, & Williams). In fact, as you will see in the introduction to this special issue, the Children’s Bureau, the federal agency best positioned to influence policy and federal funding for child welfare organizations, has significantly increased its emphasis on using research evidence in the delivery of child welfare services.

This special issue reflects the views of both the Child Welfare League of America (CWLA) and the William T. Grant Foundation (WTG) that research has the potential to meaningfully inform child welfare policy and practice. We also believe research agendas that are informed by decision-makers would better serve child welfare organizations. It’s a two-way street.

The notion of improving the use of research to promote improved outcomes in child welfare may seem abstract and less important than other issues facing a child or family. However, strengthening connections between the research and decisionmaking relevant to practice and policy has practical value. For example, in this issue, Nunno describes the importance of appropriate uses of restraints and the role research can play to guide the identification and application of alternative approaches to minimize additional trauma to youth. With such high stakes, and increasing external pressures, it is time for child welfare organizations to call on research in meaningful ways.

Research also has the potential to sharpen decision-makers’ understanding of issues facing their organizations and children, youth, and families; generate evidence about the effectiveness of programs, policies, and practices; and guide improvement (Littell & Shlonsky, 2010). It can inform strategies for allocating resources in tight fiscal climates and guide the selection of programs that promote child safety (DuMont, 2014). Research-based risk assessments can minimize disruptions to a child’s living situation or schooling and conserve resources. Research-informed protocols can help train workers and instruct the principles they use in practice. Research can also be used strategically to justify existing positions or argue for new approaches. Thus, different uses of research can affect the short- and long-term outcomes for children, youth, and families.

CWLA and WTG share a concern that the potential of research often goes unrealized and that in the long run, we miss opportunities to do right by children. Indeed, in this issue, the empirical studies by Palinkas and colleagues and Wulczyn and colleagues reveal low rates of research use within the child welfare system. They do not assign blame, but instead report on lower research use relative to other service areas (Palinkas et al.) and throughout various levels of the system (Wulczyn et al.). These findings align with prior research (Aarons & Palinkas, 2007; Wang, Saldana, Brown, & Chamberlain, 2010), again underscoring the perennial challenge of integrating research in child welfare. Even when research is neatly packaged into validated screening tools and risk assessments, it is often overlooked ( Johnson et al., 2008).

Not an Easy Path

Given the complex system required to support children and families, it is not surprising that research use within child welfare is uneven and even lacking. An array of services is required to promote the well-being of children, youth, and families. This includes child welfare specific services as well as related child and family strengthening services and supports. Services are required to provide in-home family preservation, residential care, physical and mental health care, assistance with housing, child care, and employment and financial benefits. In addition, the organizations delivering these services are complex and involve different organizational structures, governing bodies, and decision-making processes. Furthermore, decisionmakers have different needs for research and different capacities to access, make sense of, apply, and inform research. These realities make it difficult to know where research sits within the child welfare system and how to integrate it into the day-to-day work.

Several of the articles included in this issue discuss barriers on the production side of research that hinder the use of research evidence. Maciolek notes limitations in the relevance of bodies of evidence for the decisions required by local leaders. Researchers can overlook questions that are salient to policy and practice and miss opportunities to generate research about the costs and resources required to implement a program. Others noted that even when an organization’s leadership was supportive of using research, individual employees had difficulty understanding the research and its related terminology (e.g., evidence-based, research-informed, tiered evidence; Yates et al.).

Other barriers to using research were more ingrained in the practice organizations. Some child welfare agencies lacked the commitment from leadership and an organizational culture to trust and use research evidence (Wulczyn et al.). In other situations child welfare organizations were under-resourced and lacked the technical infrastructure to access research and to perform analyses on their own data (Macy et al.). Agencies also reported a lack of knowledge about how to integrate research-based concepts into the work of an organization and struggled to create ongoing opportunities for information sharing and understanding. Some of these barriers are perennial and have been documented in prior work in child welfare (Killos et al; Palinkas et al.; Wulczyn et al.).

Funding also poses challenges to research use. As noted by Mackie and colleagues, even with the best interest of the child at the center of their planning, leaders are still bound by complex factors when determining the funding sources for their programs and services. Priorities might shift as a result of specific grant requirements; the state of the greater economy; budget reduction; media attention about child fatalities; or class action lawsuits ( Jordan & Connelly, 2016). These competing demands make it difficult to hire committed research staff, develop the research savvy of others, and implement and maintain the integrity of research-based programs and services. Resources are more likely to be available if the leadership that views research in a favorable light, as is illustrated by Killos and colleagues’ review of New Jersey’s efforts.

Improving the use of research evidence will require an acknowledgement of these challenges as well as intentional and innovative thinking, resources, and new alliances. Ultimately, our hope is that organizations, researchers, and other stakeholders find ways to routinize productive uses of research evidence, develop pathways for the questions and needs of practitioners and policy-makers to shape research agendas, and to document these journeys and impacts.

Promising Approaches for Improving the Use of Research Evidence

So how do we move forward to meet the increasing pressure on child welfare organizations to implement research-based programs and services? One goal of this issue was to advance the conversation about strategies for improving research use. And many of the articles in this issue extend our understanding of research use in child welfare by richly describing ways of operating and relationships between the research and practice communities that may help us do better by children and youth. The case examples come from practitioners and researchers and offer concrete ways for moving forward. Two ideas stood out: cultivating trusting relationships among collaborators and building an organization that values and has the capacity to use research.

Cultivate Trusting Relationships between the Research Communities and Practice Communities

Several articles stressed the importance of bringing together researchers and policy-makers and practitioners to work from the perspective of a shared goal: developing and implementing research projects that will result in programs that improve outcomes for children (Maciolek).

Trusting relationships are associated with research use (Asen & Gurke, 2014; Palinkas, Short, & Wong, 2015; Finnigan & Daly, 2012; Larson and Langworthy). These relationships develop overtime and require sustained opportunities for the researchers and agency and program leadership to engage about the needs of the various partners (Landsman and Rosenwald; Larson & Langworthy; Nunno et al.; Yates et al.). In keeping with these goals, several articles provided rich descriptions of how focused attention and informal activities help cultivate healthy relationships among researchers, intermediaries, and decision-makers. (Fisher et al.; Yates et al.)

Yates and colleagues, for example, recommend building trust and value for research by providing opportunities for staff to “[vocalize] feelings of anxiety, confusion, and disempowerment.” Larson and Langworthy’s explored the benefits from listening to the voices within child welfare agencies and conducting cross-system collaborations to produce the most useful evidence possible. And Fisher and colleagues offered ways to overcome tensions between evaluators and decision-makers. Collectively the articles illustrated strategies and challenges faced during efforts to strengthen ties, build on assets, and leverage expertise. Repeatedly we witnessed the importance of a long-term view and persistence in efforts to forge trusting relationships among collaborators.

Metz and Bartley provided empirical evidence on the value of cocreation in fostering a shared language and goals. Importantly several teams also provided concrete examples of some of the activities that embody co-creation among evidence-based program developers, academic researchers, private sector providers, and public agencies. Activities included the co-creation of presentations, co-constructing logic models, jointly developing desk-guides, and regularly scheduled structured conversations. These activities allowed decision-makers, supervisors, and frontline workers to deepen their understanding of evidence-based practices and be swayed about the relevance and utility of various models for their local clientele. These documents and tools were then integrated with institutional practices. Thus, joint work helps establish shared objectives among the stakeholder groups who share a commitment to improving the lives of youth, and these activities need to happen at the project’s outset (Maciolek). In addition, formal and informal pathways for knowledge sharing and joint problem-solving are also critical for research use and organizational learning.

These observations are consistent with findings from empirical studies on the use of research evidence in education and mental health. These studies consistently find that understanding and use of research evidence improves when decision-makers are provided with structured opportunities to engage with research evidence, debate it, and question its applicability to the local setting (DuMont, 2015; Honig et al., 2014; McDonnell & Weatherford).

Build the Social Climate, Technical Infrastructure, and Staff Capacity to Support Routine Uses of Research Evidence

To institutionalize meaningful uses of research evidence, organizations need to establish the foundations that help socialize individuals to those practices and set them on a path that eventually becomes self-reinforcing (Colyvas & Powell, 2009).

As noted above, one obstacle to the routine use of research in child welfare organizations has been limited knowledge about how to integrate practice-based knowledge and experience into evidencebased knowledge. But the articles in this issue suggest several promising strategies, from accessing research (Martin, Walsh, & Reutz), to altering processes within child welfare organizations (Landsman & Rosenwald; Metz & Bartley; Palinkas et al.), to partnering with researchers and universities (Fisher, Spangler, Huebner; Larson & Langworthy; Nunno et al.).

Organizations working to promote the welfare of children need a strong foundation to support the integration of research and existing policies, practices, and routines. Yates and colleagues and Metz and Bartley both described the role of a leadership team to model buy-in and help create connections between the research evidence and the agencies’ missions. This requires leadership that promotes activities, relationships, and incentives that lead to routine and productive uses of research. For example, leadership can help to allocate staff, staff time, and data structures that support the engagement and understanding of research (see Farrell & Coburn, 2016).

Staff training offers another opportunity to enhance the research appreciation and savvy of child welfare workers. This might include training or wrap-around sessions or fellowship programs that help develop the skills and knowledge about the research process. For example, New Jersey recently made a concerted effort to build a culture that values research (Killos et al.). These efforts required vision and buy-in from the leadership and a financial investment.

Other articles illustrated the critical role administrative or organizational data can play when applying evidence-based programs (Macy, Ermentrout, Redmond, Rizo, & Pollock; Nunno et al.; Yates et al.). Data can be used to verify the need for the evidence-based programs, monitor implementation, and inform improvement. Time and again, these articles demonstrated the importance of followthrough and sustained support. Once an evidence-based program is selected and integrated with program operations, it takes time and much iteration to hone and improve its effectiveness. These are stories of sustained and carefully monitored uses of research evidence.

A Collective Agenda for Using Research to Promote Child Welfare

History has shown that it is at this juncture—when the barriers to using research are made visible—that interest wanes and actions falter. We hope this special issue will be a marker of something different and usher in a new era of activities and research studies to improve the usefulness and use of research evidence in child welfare. This issue offers empirical evidence and examples from research and practice about ways to structure relationships between individuals and organizations and within organizations to facilitate the use of research in ways that benefit children, youth, and families. What happens next is up to the many stakeholders that are concerned with serving children, youth, and families. Let’s move forward together in the best ways we know how.

For Child Welfare Agencies

• Integrate research with clinical and practice expertise and embed knowledge from bodies of research evidence and research-informed tools, programs, and practices within the organizational cultures. Take advantage of the resources that have already been developed to assist decision-makers with the process of selecting appropriately fitting evidence-based programs (Martin et al.; Fisher et al.) and integrating research evidence into their organizations (Metz & Bartley; Palinkas et al.; Yates et al.). Although most examples focused on the use of research that informed a packaged program, there are other uses of research evidence For example, Nunno and colleagues described a more conceptual use of research in which the partnerships looked for ways to incorporate knowledge from research into everyday practice to improve the experience for children in residential treatment settings.

• Be creative about identifying the resources necessary to fund these efforts. The New York City example described by Killos and colleagues demonstrates how resources and contracts can be renegotiated and reallocated—without adding funds—to deliver evidence-based services. By integrating the use of research into the contract process, the agency’s efforts to institutionalize research use withstood the shift of an administration change. This is a notable achievement, and efforts are currently underway to assess whether the integrated efforts yielded better outcomes for children, youth, and families. This example makes it clear that such approaches will require forethought and planning about how to leverage and repurpose existing processes and relationships.

• Develop partnerships with universities and other research entities to expand opportunities for shared learning and project development. These opportunities might include establishing ongoing partnerships, doing program development work, developing data-sharing agreements, and constructing joint research agendas.

• Leverage intermediary organizations that are well positioned to bridge gaps between researchers and public agencies and private providers (Killos et al.). This will afford structured opportunities for practitioners and policy-makers to identify and communicate research questions that will drive improved outcomes for children.

• Seize opportunities to use research for immediate impact on policy and practice. In this issue, Mackie and colleagues revisit the idea of a policy window. Consistent with this notion, the authors encourage organizational leaders to view leadership transitions, times of crisis, and budget processes as good moments to use research to inform decision-making. Researchers and intermediaries also need mechanisms for anticipating the policy windows and accumulating and facilitating access to relevant research.

For the Research Community

• Involve practitioners and policy-makers in your research endeavors and jointly construct agendas that meet both your academic needs and the needs of the local context. Larson and Langworthy provide a thoughtful discussion of the recursive learning that can occur when both partners listen.

• Extend what we already know about how child welfare decisionmakers and their organizations make sense of research alongside of other types of evidence. Designs might adopt a rigorous mixed-method approach that both investigates the process of integrating and embeds research in the routines and tools of child organizations, while also sharing examples of successful and unsuccessful attempts to negotiate this challenge. The examples are critical because they reveal the underlying processes and structures that are needed to move forward.

• Systematically study efforts to improve the use of research. We need to know what is required to create organizational cultures that routinely consider relevant research evidence. This will require identifying, creating, and testing possible strategies that extend the ideas discussed in this issue. Importantly, we also need to know whether the thoughtful consideration and use of research-informed tools, evidence-based practices, and knowledge learned from bodies of research make a difference in the quality of services delivered and youth outcomes.

 

The William T. Grant Foundation is committed to supporting efforts to strengthen connections between research and practice and research and policy. In 2009, the Foundation launched an initiative to better understand the use of research evidence in policy and practice as it relates to youth. In 2015 the Foundation shifted its focus to studies that identify, create and test the conditions that improve the production and use of high-quality research in ways that benefit young people.

Researchers are needed to pursue these lines of inquiry within child welfare. Other funders are also needed to extend support to these important questions.

For Membership Organizations, Coalitions, Funders, Advocacy Organizations, and other Key Intermediaries

• Offer training opportunities—for example, encouraging and supporting schools of social work to prepare future leaders to be facile in accessing and critically evaluating research.

• Offer opportunities for researchers and practitioners and policies to engage in meaningful and sustained conversation. This might take the form of ongoing, brokered learning communities or sustained conversations among different groups about a body of research, its implications for child welfare, and gaps in the knowledge base.

 

The Children’s Bureau can continue and expand its efforts to provide discretionary funds that will promote and support the development and use of research, especially as it relates to addressing challenges of building evidence groups of people that have had negative experiences with program evaluation and research.

Almost from its founding, CWLA has promoted and supported best practice in child welfare and has made efforts to advance the use of research evidence. With a membership base that includes public sector organizations, private providers, schools of social work, researchers, training entities, and advocates bound by a commitment to improving outcomes for children and families who are vulnerable, CWLA is in an ideal position to facilitate this type of relationship development and collaboration. CWLA’s National Blueprint for Excellence in Child Welfare includes core principles to guide the thinking and decision-making of organizations and individuals committed to child safety, permanency, and well-being. One of these principles is quality improvement; within that principle is a standard requiring that organizations collaborate with universities and other entities conducting research, and those universities and other researchers should disseminate research findings widely to contribute to research and practice applications. CWLA will develop guidelines and tools to support organizations in achieving these standards.

Closing Thoughts

This issue of Child Welfare highlights the potential role for research in child welfare, yet child welfare organizations, their leaders, and their staff have yet to fully embrace research evidence as a tool in their efforts to do better work (Palinkas et al.; Wulczyn et al.). This collection of articles highlights the importance of intentional alliances and processes for realizing that potential. While there is value in attending to the accessibility and rigor of research, investments are needed from the research, policy, and practice communities and key intermediary organizations that broker these relationships to cultivate long-term collaborations. Structured opportunities and processes also need to be established to help inform research agendas in ways that are responsive to decision-makers’ local needs and to find ways to make existing bodies of research work for local contexts. This may require shifts in the culture of an organization, and the examples provided suggest this is possible.

We hope this special issue encourages interest in both the process and the value of integrating research in child welfare organizations and realizing the potential for research to benefit children, youth, and families. We encourage those working within the area of child welfare to leverage the experiences, strategies, and research from other fields, such as education and health, to advance the use of research and help child welfare improve outcomes for children. Please join us in actively pursuing strategies to do better.

Kimberly DuMont
Senior Program Officer
William T. Grant Foundation

Christine James-Brown
President & CEO
Child Welfare League of America

 

References

Aarons, G.A., & Palinkas, L.A. (2007). Implementation of evidence-based practice in child welfare: Service provider perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 34, 411–419.

Colyvas, J.A., & Powell, W.W. (2009). Measures, metrics, and myopia: the challenges and ramifications of sustaining academic entrepreneurship. Advances in the Study of Entrepreneurship, Innovation, and Economic Growth, 19, 79–111.

DuMont, K. (2014). Realizing the potential of research in child welfare. New York: William T. Grant Foundation.

DuMont, K. (2015). Leveraging knowledge: Taking stock of the William T. Grant Foundation’s use of research evidence grants portfolio. New York: William T. Grant Foundation.

Farrell, C.C., & Coburn, C.E. Absorptive capacity: a conceptual framework for understanding district central office learning. Journal of Educational Change, November 2016, 1–25.

Fisher, E., Spangler, J., & Huebner, R. Tensions and opportunities: Building meaningful partnerships between child welfare decision-makers and evaluators. Child Welfare, 94(3), 91-111.

Honig, M.I., Venkateswaran, P.M., & Twitchell, J.M. (2014). Leaders’ use of research for fundamental change in school district central offices: Processes and challenges. In K.S. Finnigan and A.J. Daly (Eds.), Using research evidence in education: From the schoolhouse door to Capitol Hill (pp. 33–52). Heidelberg: Springer.

Johnson, M.A., Stone, S., Lou, C., Vu, C.M., Ling, J., Mizrahi, P., & Austin, M.J. (2008). Family assessment in child welfare services. Journal of Evidence-based Social Work, 5(1–2), 57–90.

Jordan, E., & Connelly, D.D. (2016). An introduction to child welfare funding and how states use it. Bethesda, MD: Child Trends.

Killos, L., White, C.R., Pecora, P., Maher, E., O’Brien, K., Danielson, D., DuMont, K., Wulczyn, F., Samuels, B., & Anderson, C. Strategies to improve the use and usefulness of research in child welfare. Child Welfare, 94(2), 33-55.

Landsman, M., & Rosenwald, M. Strategies for strengthening the utility of research in supportive housing–child welfare partnerships. Child Welfare, 94(3), 135-150.

Larson, A., & Langworthy, S.E. Learning to listen and listening to learn: Recursive information flow to build relationships and improve practice. Child Welfare, 94(3), 59-84.

Littell, J. H., & Shlonsky, A. (2010). Toward evidence-informed policy and practice in child welfare. Research on Social Work Practice, 20(6), 723–725. Maciolek, S. Engaging the child Welfare community in examining the use of research evidence. Child Welfare, 94(3), 167-184.

Macy, R.J., Ermentrout, D.M., Redmond, P.H., Rizo, C.F., & Pollock, M.D. From novel to empirical: Developing community-based programs into research-ready programs. Child Welfare, 94(3), 151-166.

Martin, J., Walsh, C.R., & Reutz, J.R. Selecting an evidence-based practice in child welfare: Challenges and steps to identifying a good fit. Child Welfare, 94(2), 107-114.

McDonnell, L. & Weatherford, S. (2014). Research evidence and the common core state standards. In K.S. Finnigan and A.J. Daly (Eds.), Using research evidence in education: From the schoolhouse door to Capitol Hill (pp. 69–89). Heidelberg: Springer.

Metz, A., & Bartley, L. Co-creating the conditions to sustain the use of research evidence in public child welfare. Child Welfare, 94(2), 115-139.

Nunno, M.A., Smith, E.G., Martin, W.R., & Butcher, S. Benefits of embedding research into practice: An agency-university collaboration. Child Welfare, 94(3), 113-133.

Palinkas, L.A., Short, C., & Wong, M (2015). Research-Practice Policy Partnerships for Implementation of Evidence-Based Practices in Child Welfare and Child Mental Health. New York: William T. Grant Foundation.

Palinkas, L., Wu, Q., Fuentes, D., Finno-Velasquez, M., Holloway, I.W., Garcia, A., & Chamberlain, P. Innovation and the use of research evidence in youth-serving systems: A mixed-methods study. Child Welfare, 94(2), 57-85.

Wang, W., Saldana, L., Brown, C. H., & Chamberlain, P. (2010). Factors that influenced county system leaders to implement an evidence-based program: A baseline survey within a randomized controlled trial. Implementation Science, 5, 72.

Wulczyn, F., Alpert, L., Monahan-Price, K., Huhr, S., Palinkas, L.A., & Pinsoneault, L. Research evidence use in the child welfare system. Child Welfare, 94(2), 141-165.

Yates, M., Nix, J., Coldiron, J.S., & Williams, L. From a provider’s perspective: Integrating evidence-based practice into the culture of a social service organization. Child Welfare, 94(2), 87-106