It’s not often that you hear nonprofit executives praise a performance evaluation system imposed by government funders.  But, that is exactly the reaction – with a few significant caveats – from providers after two years’ experience with the Administration for Children’s Services’ new Scorecard.

“They have done a better job here than I have seen anybody else do in trying to get this right,” says Jim Purcell, CEO of the Council of Family and Child Caring Agencies (COFCCA).

“I think they have done a great job,” says Bill Baccaglini, CEO of The New York Foundling.

That said, providers still have significant concerns regarding demands imposed by Scorecard on already over stretched agency staff, the fairness of scoring and, most important of all, whether inadequate funding and resultant higher than acceptable caseloads actually allow agencies to address programmatic deficiencies identified via the system.

Click here to see the FY09 Family Based Scorecards

Click here to see the FY09 Residential Scorecards

Beyond EQUIP

ACS has a long history of attempting to assess the performance and quality of nonprofit agencies with contracts to provide foster care and other child welfare services.   

In 2000, the agency launched EQUIP -- the Evaluation and Quality Improvement Protocol – which gathered information on contract agencies’ “outcomes”,  “process” and “quality”.  The system combined scoring from case record readings with statistical analyses of the time to reunite children with their families or place them in adoptive homes, rates of re-entry into foster care and the frequency of substantiated reports of abuse or neglect for children in care.   Results were rolled up into a single numerical score on a scale of 1-100 which were categorized as Excellent (85 or higher), Very Good (80.00-84.99), High Satisfactory (74.00-79.99), Low Satisfactory (70.00 to 73.99), Needs Improvement (65.00 to 65.99) and Unsatisfactory (below 64.99).

Over the next seven years, ACS would use EQUIP to drive decisions relating to rates of reimbursement –a “performance-based rate” with bonuses for higher performing agencies – and contracting, i.e. which agencies to place on review status or terminate contracts.

“EQUIP was pretty advanced for its time,” says Valerie Russo, Deputy Commissioner of the Office of Quality Assurance at ACS.  However, she explains, ACS Commissioner John Mattingly wanted a system which would be better designed to guide agencies in their own efforts to improve practice and outcomes.  The answer would be Scorecard.

Scorecard

Scorecard is EQUIP on steroids – and with a post-doctoral degree.

Like EQUIP, Scorecard is based on the findings of case record reviews – now known as PAMS (Provider Agency Assessment Measurement System) -- and outcome data culled from the extensive and cumbersome Child Care Review System (CCRS).  

Unlike EQUIP, Scorecard looks at a significantly broader range of indicators which are then grouped into the key practice areas of Safety, Permanency and Well Being.  Home-based programs also include ratings for Foster Parent Support.

In the area of Safety, providers are rated on such factors as the appropriateness of safety assessments, the frequency and quality of casework contacts with children and foster parents, child safety during parent-child visits, sleeping arrangements in the foster home, foster parent training, frequency of reports of founded abuse and neglect reports to the Office of Special Investigations (OSI), the numbers of children missing or absent from care (AWOLs) and the timeliness of corrective action plans and foster home certifications.

The practice area of Permanency -- efforts to safely reunite a child with their family, finalize their adoption, or prepare them for a safe and successful life as an independent adult – includes even more Scorecard indicators.  There are five factors involving parent and child visitation, seven relating to Service Plan Reviews, four associated with Preparing Youth for Adulthood…and more.   

Ratings for Well Being are based on more than 20 different factors including sociotherapist contacts, visits with siblings, medical and mental health services, education, step-ups to higher levels of care and lateral moves from one placement to another.

In each practice area, individual indicators are scored in different ways and carry varying weights towards determining the final score.  

If it sounds complicated, that’s because it is.  The FY2010 Methodology manual is a mere 61 pages – not quite as long as healthcare reform legislation, but impressive nonetheless.  “It is so broad and so deep that it just makes your head spin,” says Mary Jane Dessables, Senior Program and Policy Analyst at COFCCA.

Collaboration  

Surprisingly, however, a system this complex and intensive has drawn praise from provider agencies rather than calls for revolt.   In part, that is because the system was designed with considerable input from agencies themselves.

“I think it was very collegial,” says Bill Baccaglini.  

Dessables describes the process as “transparent and collaborative”.   At one point, she recalls, ACS invited agency representatives to use an allocation of push pins to assign appropriate weights to the various proposed Scorecard indicators.  “You could put all the push pins on one indicator or spread them around,” says Dessables.  “It was a way for agencies to say how important they felt these different activities were in serving children and families.”

As a result, providers seem more inclined to believe that Scorecard – unlike many other government evaluation efforts – actually measures the right things.  

“The Scorecard is great,” says Jerry McCaffery, CEO of MercyFirst.  “I would be hard pressed to say that anything in there shouldn’t be there.”

So, How Did We Do?

Unfortunately, Scorecard results appear, at least at first glance, to be less than stellar.   

Unlike EQUIP which ranked each agency with a single numerical score, Scorecard gives separate letter grades to agencies in the areas of Safety, Permanency, Well Being and, in the case of home based programs, Foster Parent Support.   A quick look at the Home Based program results for 2009 reveals lots of “Bs” and “Cs” (36% and 48% of all grades, respectively) with only a smattering of “A”s, “D”s and “F”s, (6%, 7% and 3%, respectively).

In fact, say both ACS and providers, Scorecard is not designed to offer up easy marks.  It takes a score of 95 to get an “A” in the Safety practice area and a 92 or better to “ace” Permanency, Well Being or Foster Parent Support.  “B”s only start at 85 with “C”s running from 75 to 84.  “D”s go from 65 to 74 and you get an “F” for 65 or below.

“On safety, which has to be our first responsibility, it was a very conscious decision to set the threshold for getting an ‘A’ at 95%,” says ACS’ Valerie Russo.

“It’s hard to argue with that,” acknowledges MercyFirst’s Jerry McCaffery while noting that achieving these outcome levels can be extremely difficult based on the ways Scorecard awards or deducts points for problems with only a handful of cases.  

“It only takes a very small number of OSI reports to seriously impact your scores,” says COFCCA’s Mary Jane Dessables.  “And, while all OSI reports are important, many are actually outside the control of the agencies and many do not indicate that there was any imminent risk of serious physical harm to the child.”

“It is like setting a high bar on what already is a high bar,”  says McCaffery. At the same time, however, he notes that most objective observers interpret letter grades based on their own experience in school.  Scorecard grading, he argues, is much tougher than that; it’s harder to get “A”s and “B”s.   “That’s difficult to explain to board members, donors and the general public,” he says.

On a system-wide basis, the average score for Home Based programs in all practice areas was “C”.  In fact, the system as a whole fell just short of earning “B”s -- within one point in both Safety (84.14) and Permanency (84.34) and within two points for Well Being (83.27).

Residential programs came in a little higher on their Scorecard with “B” in Safety (87.17) and Well Being (86.32) and a “C” in Permanency (84.45).   ACS emphasizes, however, that the two Scorecards are different and comparisons are not appropriate.

Performance Improvement

Scorecard is more than just the scores itself, stresses ACS.   It is a broader performance monitoring and feedback system designed to assist agencies in improving the quality of their practice.

With EQUIP, explains Russo, providers got their scores and data long after the fact and in a format that wasn’t particularly user friendly.  

“In Scorecard, we work with the agencies to make their data available to them as we get it,” says Susan Fojas, Associate Commissioner for Performance Measurement, Monitoring and Improvement.   

Agencies receive regular quarterly reports with updates on a variety of outcomes data.  “We see how we are doing and can catch data and methodology errors early,” says Lee Pardee, Director of Policy and Practice Implementation at SCO Family of Services.

Staff from the Office of Agency Program Assistance (APA) work with agencies on areas of concern and plans for improvement.  “The meetings they schedule with programs utilizing PAMS data offer us an opportunity to examine our practice and develop corrective actions that we think may be useful,” says Diane Krasnoff, Assistant Executive Director at SCO.

 “By the time the Scorecard comes out at the end of the year, we already have been working with agencies on these issues,” says Fojas.  

“It is like being in a class where there is a lot of homework and a lot of quizzes during the year, not just a final exam,” says Valarie Russo.  “You should have a sense of what your grade is and your teacher is working with you on those areas where you are not doing so well.”

The shift to individual grades in each of the practice areas also highlights specific agency weaknesses which previously might have been overshadowed in a single, unified grade.   

“This allows you to drill down,” says Baccaglini.

Scorecard’s “complementary data”, which provides outcome data comparisons for agencies on population subgroups, is a particularly valuable innovation.  “Dividing the complementary measures scores into categories of children by level of difficulty, less than or more than two years in care and age groupings was ingenious,” says SCO’s Diane Krasnoff.  “This results in fairer and more relevant comparisons among agencies for these measurements.”

Concerns

Despite the positive feedback, agencies do have a number of concerns about the Scorecard system.  “The two biggest objections agencies have is when they are being held responsible for things they can’t control or a score is designed in a way where there are only a few variables or where a small number of cases can have a disproportionate impact,” says Mary Jane Dessables.

“Agencies can be penalized for decisions which have been approved and signed off on by ACS staff,” says Jim Purcell.  “For example, agreeing in a Family Team Conference that keeping a child in care for another six months is the right thing to do doesn’t absolve you from the fact that the child stayed in care longer.  You lose points.”

Providers also take hits to their grades for children placed outside their home communities or problems in kinship foster homes, even though those placement decisions were made by ACS.

The practice area of Foster Parent Recruitment and Support is one sore point where providers argue that low ratings may be driven by failure to meet foster parent recruitment or retention targets which are no longer consistent with actual bed needs within given community districts.   “A reader might interpret a low mark to mean that we did not treat our foster parents well,” said one executive director.

“We also continue to have serious reservations about the use of extremely small numbers to derive large conclusions about the quality of an agency’s work, especially in the case record audit (PAMS) section of Scorecard,” says Richard Altman, CEO of Jewish Child Care Association.   Altman cited sampling of an item like ‘sibling visits’, where “the randomly collected cases may yield a very small proportion of actual relevant cases (where there are siblings and the agency has case planning), but the information from those cases is then generalized to the entire sample and ultimately the entire program.   Circumstances in a very few case situations can lead to enormous changes in an agency’s score both in the PAMS component and Scorecard in general.  This is very problematic and can lead to seriously erroneous conclusions about the case practice of an agency.”

“We start to get into some dangerous territory when you get sample sizes that are so small,” says Bill Baccaglini.  “ACS needs to be cautious about how they use the results.”

Agencies also argue that a single Scorecard rating scale for all residential programs is inherently unfair.  “We are concerned that grouping so many different kinds of congregate care programs together, and then comparing their performance, is a classic example of comparing apples to oranges,” says JCCA’s Altman.  “An agency with only group homes or one with a few SILP apartments is completely unlike one with a large Residential Treatment Center.  Besides creating misleading comparison ‘rankings’ this grouping renders the data fairly useless for quality improvement… ACS should measure congregate programs with similar programs (e.g., SILP with SILP, RTC with RTC), not against all residential programs in general.”  

 At the same time, however, many agencies give ACS high marks for flexibility and responsiveness in making adjustments to Scorecard based on experience and input from providers.  

“ACS heard our complaints that timely finalization of adoptions were less and less within our control due to serious delays in Family Court,” says SCO’s Lee Pardee.  “In response, they added two components to the score that are fully within our control: photo listing/adoptive placements and timely filing of TPRs after goal changes.”

Bearing the Burden

Scorecard, with its foundation of PAMS case record reviews and intensive data analyses, itself requires a major commitment of resources on the part of nonprofit provider agencies.

“You need to have somebody to manage the process,” says COFCCA’s Mary Jane Dessables. “Agencies that don’t have really good QI or Compliance Departments are going to run into problems because there are so many things to keep track of.”

“We spend an inordinate amount of time training our workers to document the way ACS wants things documented,” says one agency executive. “Compliance is king. All agencies have some systems in place -- and need to have them -- to ensure that we meet their requirements.”

Some agencies are now rigorously training staff to fill out progress notes using a template that ensures PAMS case readers will be able to easily find all required casework activities. “They put ‘x’ in the first paragraph, ‘y’ in the second paragraph and ‘z’ in the third paragraph,” says Dessables.  “It’s very structured and all the elements are there.”

The downside, say agency executives, is that case records no longer tell the story of a child and family.  “You used to be able to pick up a case record and get a feel for who this child was and who the parent was,” said MercyFirst’s Jerry McCaffery.  “That’s not true anymore.”

“Unfortunately we end up spending too much time and effort on this rather than with the children and families we serve,” says another agency executive.

Slicing and dicing Scorecard data also takes time and manpower.  “The massive amount of data that ACS sends us for the various scores is both good and bad,” says SCO’s Lee Pardee.  “It is a lot of work to check all the data and there are quite a few oddities that show up.”

The Ultimate Challenge

For many providers, however, the ultimate frustration with Scorecard is the near impossibility of addressing newly identified areas of concern without the necessary resources.

“By all measures, our caseloads are still too high,” says Jim Purcell.  “At the end of the day, we are asking workers who the State thinks can handle 12 cases to handle 18 or 20 and to do all these things right every time -- not miss a casework contact, file all the papers on time, write their permanency plan for the court, sit in court, and when it gets adjourned come back and do it all again.”

“This is like that carnival game where as soon as you hit a mole in one place, he pops up someplace else,” says one executive director.  “There is no way we can be on top of everything at once.  We’re always dealing with the most problematic issue.  Then something we were doing well begins to unravel.”

“We are putting this enormous pressure on frontline caseworkers to do too many things that probably humanly can’t be done and then we are judging them on it.  That is pretty tough,” says Purcell.

ACS declined to comment on this

 issue.

An Achievement in Progress

Regardless of these larger concerns, Scorecard appears to represent a significant achievement in the development of human service performance evaluation systems.  “It is clearly the most robust monitoring system I have seen government develop,” says Jim Purcell.

“It is not perfect in any way,” says ACS’ Valerie Russo.  “We know that this is an iterative process. But, we are proud of the work we have done and how far it has come.”  

---

Add this page to your favorite Social Bookmarking websites

COMMENTS

B

i

u

Quote

Code

List

List item

URL

NAME *
EMAIL_VERIFICATIONS_REPLIES
URL
CODE   
ChronoComments by Joomla Professional Solutions

SUBMIT_COMMENT