Storytellers:
Steve Kelban,
Executive Director, Andrus Family Fund
Sabena T. Leake
Program Officer, Andrus Family Fund
Patti Chamberlain,
Executive Director, Oregon Social Learning Center

Background
From its inception in 2000, the Andrus Family Fund (AFF) has incorporated William Bridges’ Transition Framework (TF) into all the projects it funds, guided by the firm belief that an understanding of Transitions will improve the chances of these projects’ long-term success.

The framework works on the principle that any external change a person goes through needs to be accompanied by an internal change process. S/he first has to give up old ways or ideas (Endings), move through an in-between phase (the Neutral Zone), before s/he is finally ready to adopt a new way of life (a New Beginning).

Bridges’ TF was initially used to help for-profit corporations deal with large-scale change. No one had tried to apply the TF to the non-profit world, with communities in need of reconciliation or with youth moving out of the foster care system to independent living. However, AFF staff’s faith in the framework led them to require their grantees to incorporate it in their programs. They decided that they would try to quantify the usefulness of the TF in their grantees’ programs only after there were sufficient grantees applying it in their work and producing data that could then be analyzed.

Two years later, that time had arrived.

In 2002, during the Foster Care Grantees Yearly Learning Exchange, various grantees raised the question of the framework’s effectiveness, asking for tools to concretely measure and articulate the usefulness of Transitions in helping their youth move from foster care to independence.

With this mandate, in December 2002, AFF Executive Director Steven Kelban and Program Officer Sabena Leake asked Patti Chamberlain from the Oregon Social Learning Center (OSLC) if she would be willing to head a project to measure the effectiveness of the Transitions Framework in the foster care program.
The OSLC has two arms: one involved with direct social services and the other, a research arm that is also well respected in child welfare circles. Patti was heavily involved with both arms of the OSLC and the year before, AFF had awarded her a one-year grant to implement Transition-enhanced services for a select group of young foster women at the center — the Women in Transition group — and then compare this group’s progress with a control group receiving ‘regular’ non-TF-enhanced support, to evaluate the effectiveness of the TF in foster care. Since Patti was already involved in an evaluation of the TF, she was a natural choice to head the new project.

In April 2003, the AFF Board approved a two-year grant to the OSLC to look for ways to evaluate the effectiveness of the TF on foster youth and Patti got to work immediately. The following month, she and the AFF staff met with four Foster Care to Independence grantees¹ who had been invited to be part of a ‘Measures Working Group’ (facilitated by Patti) to discuss possible ways in which to measure the impact of the TF on foster youth.

Developing a Theoretical Model of Change
From the group’s discussions, the following hypothesis was agreed on: The more a youth understands the TF, the greater the likelihood that s/he will successfully be able to live on their own, independent of the foster care system.

To prove this, the first question the group had to tackle was: How do you define a ‘successful’ transition from foster care to independence?

In trying to answer this question, the group looked at what existing grantees were doing to measure the success of their youth’s transition to independence. During a break-out session at the 2002 AFF Foster Care Learning Exchange, a few grantees (including Patti) arrived at three measures of success: Safety, Independence, and Productivity². However, these ‘distal’ outcomes could only be measured in the long-term; the Measures Working Group also needed more immediate indicators of a successful transition.

To develop a list of more ‘proximal’ or short-term outcomes, the Working Group looked to Berkshire Farms, an AFF grantee that had developed a very comprehensive model consisting of 14 criteria (such as increased self-esteem, increased social skills, etc.) to measure the development progress of youth in their program. This 14-point model had been jointly developed by staff and youth, and reflected not so much the hard skills needed for independent living but the more intangible, psychosocial skills needed for the internal wellbeing of the youth.

The Measures Working Group noticed that Berkshire’s 14 criteria could be grouped into five broad areas of development: Social Skills, Social Supports, Internal Resources, Relationships, and Planning/Decision-Making. Success in these five areas, they decided, would imply initial success at transitioning to independence. The group then defined specific indicators of success within each of these five areas to help flesh out what each construct meant. (See Appendix 1 for the list of indicators.)

After the meeting, Patti created a model she called the Theoretical Model of Change, a pictorial representation of the group’s hypothesis. (See Appendix 2) As she explained it, certain personal characteristics of foster youth can affect their proximal and distal outcomes. A youth with a more severe history of abuse/victimization, for example, would be expected to have poorer proximal and distal outcomes, and so on.³ But independent of personal characteristics, the implementation of the TF in foster care programs should also have an effect on a youth’s proximal outcomes. This in turn should have an effect on distal outcomes. The greater a youth’s understanding of the TF, it was hypothesized, the more successful his/her proximal outcomes should be.

Measuring Understanding
Kelly and Sabena asked all the foster care grantees to forward whatever tools they were currently using to measure the successful transition of their foster youth. An overwhelming number of grantees responded sending survey tools, questionnaires, etc., that they had developed or used to measure various aspects of the transition to independence.

Among the many tools received from grantees, was a survey questionnaire developed by Dr. Robert Arendt, the Research Director at the Buckeye Ranch, to evaluate how well their foster care youth “got” the concepts behind the TF. This survey had 10 questions and did not specifically quiz the youth on the terms used in the TF; rather it assessed their thoughts and behavior as it related to any change or transition they were going through. The survey contained statements like “It is o.k. to feel uncomfortable learning a new way to deal with an old problem” and “You feel there are periods of your life where you had to end an old way of behaving and begin a new way of behaving.” Youth were asked to select a response out of a five-point scale from “Strongly Agree” to “Strongly Disagree.”

Using this questionnaire as a starting point, Patti developed a measure called the Youth Experience of Transitions Questionnaire (YET). The YET contained 16 statements with which foster youth had to indicate their level of agreement or disagreement. Each item was scored on a 5-point ‘Agree-Disagree’ response scale with 3 being a “Not Sure.”

Some of the items were reverse-worded to assess a single concept from multiple perspectives and increase the accuracy of the results. For example, item 9 — “I have learned a lot about myself by reflecting about my life” — was the opposite of item 15 — “I think it is a waste of time to sit around and think about the past.”
The draft YET went through three rounds of verification and modification with staff and youth, before it was finally considered both accurate and reliable. The final version of the YET questionnaire (see Appendix 2) was approved by the Working Group and is now a mandatory requirement for all new AFF grantees. In their award letters, new grantees are asked to administer the YET to their youth both before and after they run their TF-enhanced programs.

Existing grantees are not required to administer the YET since it wasn’t part of their original grant agreement. However, many volunteered to do so and have provided the data they have gathered to Patti for her analysis.

Measuring Success
While the effort to find a measure of understanding was ongoing, the Working Group was also trying to choose a tool to measure the success of a youth’s transition to independent living. The Working Group could have decided to develop and validate a range of specific psychometric tools that would measure each and every individual indicator of success they had earlier identified (see Appendix 1). Instead, the Group decided to look for a general outcome measure with good psychometric characteristics (reliability and validity) that had already been developed and in wide use in child welfare. This approach had a number of advantages. Firstly, it would be less of a burden — in terms of both cost and time involved — for grantees to apply in their programs. And secondly, it would be generally applicable across all of AFF’s grantees.

The group evaluated the Child Behavior Checklist (CBCL), the most widely used measure in the United States to assess child development. After researching if the tool measured the characteristics the Group had previously identified and if it was extensively used with the child welfare population, the checklist was recommended by the Working Group.

All new foster care grantees will be required to administer the CBCL to their youth at the end of their TF-enhanced training. Similar to the YET, existing grantees will not be required to administer the CBCL but they can volunteer to test this outcome measurement tool in their programs if they wish to do so.

Next Step
Parallel to the YET is the SET — the Staff Experience of Transition Questionnaire — that needs to be developed to measure how well grantees’ staff members understand and appreciate the TF. Since staff have direct contact with foster youth during their preparation for the transition to independent living, their knowledge of the concepts behind the TF can help in their counseling and mentoring of the youth. The hypothesis is that the better the staff’s understanding of the TF, the better the youth’s success at transitioning. To test this hypothesis, the SET will be developed in the months to come.

AFF staff believe that by using the YET and the CBCL (and eventually, the SET as well), the Foundation and its grantees will soon be able to answer the question: “Does the incorporation of the Transition Framework improve the chances for youth who leave the foster care system to lead a safe and productive life?”

Appendix 1: Indicators of Success

Appendix 2: Theoretical Model of Change

^ Top

1. The grantees were the Oregon Social Learning Center, the Independent Living Research Center (in New York City), First Concern (in New Jersey), and Berkshire Farms Center and Services for Youth (in upstate New York).
2. Safety included access to housing and freedom from substance abuse. Productivity meant stable employment and a continuing investment in education. Independence meant freedom from state dependence and acquiring self-advocacy skills.
3. These relationships have been validated by extensive research in adolescent development.