June 25, 2003
Pursuant to HHS’s request for comments regarding the improvement of the Adoption and Foster Care Analysis and Reporting System (AFCARS), CWLA submitted the following to the U.S. Children’s Bureau.
CWLA AFCARS COMMENTS
The specific strengths of the AFCARS
AFCARS has been part of a major movement into information technology in the child welfare field. AFCARS provides a national picture of children in foster care and adoption that was not available ten years ago. It has provided a starting place for standardizing some data elements in child welfare, and has played a part in improving the quality of child welfare data. AFCARS reinforces the importance of data to managers, workers and the field, and has been key to the federal push toward utilizing data for outcome measurement (which also presents challenges as described below).
The existence of AFCARS as a common database for foster and adoptive programs nationwide, and the progress that has been made over the last decade, is an accomplishment that warrants continued federal funding to support, improve and maintain AFCARS.
The specific weaknesses of the AFCARS or suggestions for areas of improvement, including ideas about how the suggested improvement could be made and how the Federal government could facilitate the changes
AFCARS has provided an important picture of children in care nationwide, but the data system continues to present challenges. CWLA and the National Working Group to Improve Child Welfare Data have extensively discussed AFCARS as part of a larger discussion to improve data quality and outcome measurement. The National Working Group consists of representatives from state child welfare agencies, with guest participation from the Children’s Bureau, federal contractors, national organizations and the research community. With these many perspectives involved in the conversation, CWLA and the National Working Group have identified major areas for improvement related to AFCARS. In short, AFCARS’ weaknesses and areas for improvement include:
- An ongoing process for state input – there is no formalized process to allow this at the present time.
- Increased resources to support ongoing maintenance and improvements of AFCARS – a meaningful and coordinated set of different resources is currently lacking.
- Adequate data for accurate outcome measurement for program improvement – important information is currently missing.
- Coordination and/or consolidation of the many federal systems, regulations and reviews relevant to child welfare programs (including AFCARS) – states feel multiple and competing pressures from the various simultaneous federal efforts.
- Flexibility to make changes to the system, without legislative intervention – long timeframes for making changes to AFCARS weakens the system.
- Improvements of data quality (the reliability, validity, relevance and appropriate application of the AFCARS data) – there are serious issues in this arena and states have provided empirical evidence of the wide variation in counting placements and reporting child abuse in foster care, as well as anecdotal evidence about other definitional discrepancies.
CWLA and the National Working Group to Improve Child Welfare Data offer the following recommendations for improvement in these areas.
An Ongoing Process for State Input
- The Children’s Bureau should form an ongoing Advisory Group to guide AFCARS, similar to the NCANDS State Advisory Group.This group should be:
- Charged with guiding long-term plans for improvement and assisting in the development of new or modified data definitions and policy guidance, prior to official release. The formation of this group puts in place a process for decision-making that actively incorporates the experience of states and other stakeholders. This is important for both short-term and long-term improvements, including increasing the reliability and validity of the data.
- Composed of state child welfare agencies, researchers, and other stakeholders. The states, however, should have the deciding “vote” in order to ensure a clear reflection of the majority of states’ views on matters under consideration; other participants should serve as ex-officio members. Participation should be voluntary.
- Funded, supported, and coordinated with other meetings in order to promote full participation in the group. Funding and coordination with other meetings (such as NCANDS meetings) will allow consistent participation of states, particularly in this time of budget cuts and extreme travel restrictions.
- Coordinated with the NCANDS State Advisory Group. While AFCARS and NCANDS are two independent federal data reporting systems, they contain information about many of the same children served by the child welfare system. The two advisory groups can coordinate their efforts by overlapping parts of their meetings to discuss common issues and challenges. The meetings should not conflict, as some states may have the same representative serving on both groups.
Increased Resources to Support Ongoing Maintenance and Improvements of AFCARS
- The Children’s Bureau should provide additional resources for states to support improvement of AFCARS. The Children’s Bureau and NRC-ITCW offer technical support through on-site technical assistance and the annual National Data Conference. Building on this, more states would benefit from the following:
- Devote a day at the annual national data conference to AFCARS. This will allow all states to receive the same information simultaneously, without having to choose among competing sessions. It will also enhance dialogue around issues and solutions regarding AFCARS, and would serve as a time for states to participate in an Advisory Group. Continued coordination with the NCANDS meeting and data conference improves the ability of states to participate.
- Provide individual technical assistance opportunities during the data conference. Ideally this would entail a team of technical assistance resource persons. Individual sessions allow states to better apply the rules, regulations, and technical points to their own state data systems and policies.
- Provide funds for accommodations during the annual national data conference. States currently benefit from the coordination of the NCANDS meeting and Data Conference, but this is not enough for some states with extreme travel restrictions. Enabling attendance will promote further improvement to the data quality, which will ultimately help improve information about outcomes for children.
- The Children’s Bureau should provide an accessible, comprehensive and easily understandable resource for definitions and guidance regarding AFCARS. The Children’s Bureau currently provides information online through a Child Welfare Policy Manual and other resources. In addition, NRC-ITCW has compiled an AFCARS binder that brings together current regulations, policy directions, and other information in one document. Building on these existing resources, the National Working Group recommends that the Children’s Bureau:
- Organize materials by topic, extensively index, and cross-reference the binder, and make it available on the Internet.Arranging information by topic (e.g. each element, overall AFCARS population, specific cross-cutting topics, etc.) will improve the states’ ability to understand instructions and thus prepare more comparable AFCARS files. For instance, all definitions, policies, technical points, special guidance, or other information regarding trial home visits should be placed in one location, limiting the need for users to locate the information in different parts of the manual. Extensive indexing and cross-referencing will further assist users in finding related information.
- Continue to provide definitional clarity. Progress has been made in clarifying definitions, but states continue to seek definitional clarity for the purpose of federal reporting. Providing operational definitions in some areas would increase the comparability among states’ data. For example, it may be useful to provide 1) a timeframe for counting hospitalization as a placement change that approximates “short-term” for the purpose of national comparison and 2) further guidance as to the inclusion of juvenile justice populations in AFCARS. Scenarios would be useful in clarifying some issues. The proposed AFCARS advisory group would be able to assist in identifying areas for more clarity in AFCARS definitions and provide feedback as definitions and guidance are drafted.
Adequate Data for Accurate Outcome Measurement for Program Improvement
AFCARS was not originally designed to support outcome measurement and thus has not been able to provide adequate data for the federal outcome measures and standards. The following recommendations will help mitigate these problems.
- The Children’s Bureau should make several technical changes to AFCARS. For the short-term, maintain the current AFCARS data model, allowing time for more stability, while long-term improvements are conceptualized and tested. Make several modifications to improve the current outcome measures, giving states enough time and resources to accomplish this.
- Add a new data element or set of elements to AFCARS: population type to distinguish child welfare, juvenile justice, and mental health and mr/dd populations. The proposed AFCARS advisory group could assist in the development of clear definitions for the population types.
- Require states to encrypt childID’s in the same way for each data submission. That is, the childID’s should be encrypted the same across time periods and be the same for both NCANDS and AFCARS. This will permit a ‘cross walk’ between the systems to ensure the validity of outcome measures that rely on both systems (e.g., child maltreatment in foster care). It will also permit the identification of entry cohort groups for future longitudinal analysis.
- Consider the benefits of annual submissions rather than semi-annual data submissions. This may simplify the process and allow a faster turn around of federal data and related products. This might eliminate the need for the federal creation of annual files, which some states have been unable to duplicate. There would, however, need to be additional data clean-up activities.
- The Children’s Bureau should work in partnership with state child welfare agencies, researchers and other stakeholders to identify clear outcome measures for the future and ways in which AFCARS would need to be modified to support them. An inclusive process of discussion and collaboration is vital to creating the best system, as demonstrated through the NCANDS.
- Consider recommendations relevant to outcome measures that CWLA and the National Working Group have submitted to the Children’s Bureau under separate cover on June 20, 2003.
- Identify and test modifications to AFCARS that would allow entry-cohort longitudinal analysis, in addition to point-in-time and exit-cohort analysis. All three perspectives are important to performance measurement. Test the modification to AFCARS in some states prior to national implementation. Ensure that all states are included in the discussion about proposed changes, and that special considerations be made for states that may be unable to comply with the changes.
- Fund the changes for an improved AFCARS and improved outcome measurement system. State child welfare agencies have invested much in using data to improve their programs. They are committed to continuing this effort, but face harsh fiscal realities. The National Working Group emphasizes the importance of significant funding to help states implement changes, and discourages the federal government from making un-funded mandates.
Coordination and/or Consolidation of the Many Federal Systems, Regulations and Reviews Relevant to Child Welfare Programs (including AFCARS)
Significant progress has been made over the last ten years with data and information systems in child welfare, with support and incentive from the federal government. At this time, however, state child welfare agencies face competing priorities as they try to meet a range of federal requirements and provide services to children and families with extremely tight budgets.
When AFCARS was first proposed, many states did not possess the computer systems to generate data about children in placement. The federal government responded by providing assistance in the development of Statewide Automated Child Welfare Systems (SACWIS), and today all states report to the federal system, either through their SACWIS or legacy systems. Since then, the Adoption and Safe Families Act (ASFA) has outlined new requirements, the Child and Family Service Reviews began with an emphasis on outcome measurement and standards, and the National Youth in Transition Database is underway. In addition, all states participate in the National Child Abuse and Neglect Data System (NCANDS). Finally, states must meet IV-E requirements in order to retain funding. With this background in mind, CWLA offers the following recommendation:
- The Children’s Bureau should take steps to align and/or consolidate the requirements, policies, definitions and reviews associated with the following:
- National Child Abuse and Neglect Data System (NCANDS)
- Adoption and Foster Care Analysis and Reporting System (AFCARS)
- National Youth in Transition Database (NYTD)
- Statewide Automated Child Welfare Information System (SACWIS)
- Child and Family Services Review (CFSR) and outcome measures
- Adoption and Safe Families Act (ASFA)
- Title IV-E requirements and audits
Flexibility to Make Changes to the System, Without Legislative Intervention
AFCARS has the potential to improve vastly if more flexibility is provided, similar to the NCANDS model. While changes should not be made lightly, the flexibility to make changes without legislative intervention offers a more efficient approach to improving data quality, meeting changing needs and requirements, and providing both a national picture and adequate data for program improvement. The proposed Advisory Group offers an avenue for guiding such changes.
- The Children’s Bureau and other stakeholders should examine the types of options that are available in this arena including, but not limited to, the introduction of legislation to give the Children’s Bureau the flexibility it needs to help AFCARS grow and change.
Improvement of Data Quality
While the quality of AFCARS data has improved dramatically over the last ten years, there is much work to be done. With different laws, policies and definitions in the states, as well as differences in many counties, it is a challenge to provide a fully reliable national data source. Yet, we have begun to learn more clearly how states are similar and different through national surveys completed by the National Working Group to Improve Child Welfare Data (NWG), the ongoing work of the National Child Abuse and Neglect Data System (NCANDS), and the various reviews completed by the federal government (e.g. AFCARS and CFSR reviews). Since data quality – the reliability, validity, relevance and appropriate application of the AFCARS data – continues to be an issue, it appears as a theme throughout the recommendations.
- The Children’s Bureau should work in partnership with state child welfare agencies to continue to improve the quality of the data, including the following:
- The proposed Advisory Group, with states as primary advisors, will provide an ongoing forum to confront data issues and recommend changes to address them. States have already begun to identify specific elements that require better definition.
- Funding state participation in AFCARS meetings will enable more state representatives to offer their knowledge and experience and learn more about other states, thus enriching the conversation.
- States will be able to prepare more comparable AFCARS files when more definitional clarity is reached, all instructions are organized by topic, and states are able to hear the same information simultaneously during one day devoted to AFCARS.
- The flexibility to make changes without legislative intervention, and as a part of an ongoing process, will allow for more efficient implementation of proposed changes.
- Data elements currently in the AFCARS that could be deleted and any elements that should be added;States are concerned that any changes to AFCARS will be costly to implement in these difficult fiscal times. With this in mind, no deletions are recommended at this time.There are many areas for which data can be added or improved. Funding to support changes would be extremely important. With funding, additional new elements and changes to the system may provide a better data reporting system. The proposed AFCARS Advisory Group will be able to propose such changes, guide the final decisions, and assist in the development of definitions and implementation of changes. All existing AFCARS data elements must be re-examined for definitional relevance and consistency.Some new elements to consider include the following:Population Type – A new element or set of elements to distinguish population type (to identify child welfare, juvenile justice, and mental health and mr/dd populations) would improve AFCARS, as the populations served vary from state to state. This element would enhance the data set and provide better information for the outcome measures. The proposed AFCARS advisory group could assist in the development of clear definitions for the population types and offer suggestions for implementation.Multiple Birth Indicator – A new element to identify twins, triplets or other multiple births would indicate that multiple records with identical information are legitimate.
Sibling Group Indicator – A new element to identify sibling groups would assist in better describing the child welfare population and the characteristics/outcomes associated with sibling groups.
Discharge Reason for Previous Removal – Consider adding a new data element to AFCARS to provide the discharge reason (e.g. reunified, adopted, guardianship) if the child was already in care. This corresponds to Element #20, Discharge Date for Previous Removal. Although the information may be retained in the current Discharge Reason field, it will be overwritten when the child is discharge again.
Placement History – The current AFCARS submission includes only the details about the current or latest placement setting. A history of previous placements would be advantageous in showing the progress a child was making toward the case plan goal (e.g., going from the most to least restrictive placement), and for the development of future outcome measures. It would also show the types and duration of all placement settings, providing important links to outcomes. Furthermore, it would shed light on particular foster children, such as runaways, showing the types of settings they run from and how long they remain gone. This may prove to be challenging to implement and should be thoroughly examined and guided by the proposed AFCARS Advisory Group.
Missing Children – There needs to be a way to capture children for whom the existing placement settings do not apply. The way they should be reflected in AFCARS must be determined with consultation by the proposed AFCARS Advisory Group.
Adoption Disruption – There needs to be data that shows that an adoption has disrupted or dissolved, with clear definitions for the use of each term.
Some revisions to consider include the following:
Current Placement Setting – The current list of choices should be modified to be more descriptive about group, residential and institutional settings. More specific categories may include shelter, group home, residential treatment facility, independent living, hospital, juvenile detention center, juvenile facility, or other institution. These options need to be discussed and developed with the states, with the development of clear definitions for their use and mapping instructions.
Type of Disability – The current list of choices needs to be re-examined and changed with the input from the states and other stakeholders.
Discharge Reason – Expand the list to specify transfers to the juvenile justice agency or mental health agency. This would provide important information relevant to the link between juvenile justice, mental health and child welfare.
Was Child Previously Adopted? – Need better definition. Consider limiting it to adoption by the reporting state agency.
Adoptive Parents – The current choices to not include same sex parents and consideration should be given to changing the structure in a way that allows different parental configurations.
Marital status – Consider addition of the category ‘separated.’ Many couples are married but do not live together, and this has a profound impact on the child and case planning.
Most Recent Case Plan Goal – There needs to be a category called “Independent Living” as a case plan goal The use of the current category ’emancipation’ for youth receiving Independent Living services is inadequate. In most states, your who are approaching the age of majority cannot access Independent Living services unless they sign an agreement to remain a ward of the state. There needs to be a distinction between those who are emancipated from the system and those who elect to stay in the system and receive Independent Living services.
- Strategies to improve data quality for AFCARS, including the use of incentives.More resources and support plus positive incentives are more likely to improve the quality of the data in AFCARS; CWLA and the National Working Group do not support the use of penalties.Positive incentives to improve data quality include the following:
- More analysis and distribution of AFCARS data (widespread availability of the data is a positive incentive to improve it)
- Financial incentive (bonuses) for timeliness, accuracy and appropriate service delivery
- Increase funding for AFCARS to help states maintain their systems and address data quality issues proactively.
- Your experience with and use of:
- Demographic and other information on foster children, adopted children, foster parents, adoptive parents, birth parents, child descriptors such as disability status, prior adoption status, foster care information on current and previous foster care episodes and discharge; CWLA routinely prepares and analyzes data from AFCARS to present on the National Data Analysis System web site and provides some analysis to states upon request. CWLA also provides county-based and state-based analysis to states about such items as placement rates, placement trends, racial and ethnic patterns, and other data that states request.CWLA also has worked closely with the states to shed light on the similarities and differences in the data, to improve data quality and interpretation of the data. Two of the federal outcome measures – the number of placement changes and child maltreatment in foster care – were carefully analyzed to show the variations in state reporting.While some states do use AFCARS data, many states have reported that they do not use their AFCARS submission for internal purposes. They tend to rely on other reports generated from their computer systems.
- Financial information;The source of financial support for children in placement is used by CWLA to describe the current placement population.
- How the data files are structured and how data are submitted.File structure:With significant financial support for states and ongoing input from the proposed AFCARS Advisory Group, modify the data model to support entry cohort analysis in addition to point-in-time and exit cohort analysis. The creation of a relational database should be considered. State resources are scarce and database changes would be costly for states, but with financial support a better system could be developed to support more comprehensive data analysis to better inform program improvement. This major change would also better support the outcome measures used in the Child & Family Service Reviews.Data submission process:Alternative submission methods should be explored to simplify the data submission process, as some states have encountered problems using the Direct Connect Software. Specifically, consider allowing states to send AFCARS files to ACF using File Transfer Protocol (FTP) and a secured server. This approach will better handle large files and should be secure, given that AFCARS’s Child ID’s are encrypted and states follow a coding structure for reporting.Additional areas for improvement include the following:
- Annual files should be automatically provided to the state, without the need of making a request. Alternatively, a state should only have to make one request to receive the annual files every year and any time each is updated.
- Syntax to generate state profiles should be accessible to all states at no cost.