Understanding and Influencing Your State Report Card
From our conversations with school leaders, we know that state report cards can be challenging to tackle, especially as the school year comes to a close. We wanted to better understand how report card scores fluctuate over time, so we did some digging into trends over the last few years using the District of Columbia’s Performance Management Framework (PMF) as an example.
While this analysis focuses on D.C. specifically, the lessons learned can be applied to school report cards in other parts of the country. Let’s start by examining a commonly heard sentiment:
I want to improve our PMF score.
Great! But then again, so does everybody else, yet we know PMF scores do decline. Let’s try again:
I want to strategically improve our PMF score.
Better! For this post, we’re going to focus on incremental growth that can be sustained for three years or more, which is what you should generally be targeting (There are times – looming charter renewals, for example – when you need to sacrifice sustainability for a one-off bump in score. But that’s a topic for another day). Developing a plan to strategically improve your PMF score should be rooted in data and focused on PMF metrics, rather than overall school improvement. This is not the time to strategize around increasing new enrollment, improving teacher retention, etc. Your plan should answer the questions
What is likely achievable given data trends
Which measure provides the highest leverage based upon the scoring system.
Shoot for the Moon, but Pack a Parachute.
Big, Hairy, Audacious Goals around student academic growth, attendance, etc. are a great way to set a vision and motivate your staff. But – especially when your school is judged by these metrics – you also need to be keenly aware of the typical annual growth seen by schools.
We looked at the year-over-year change in each PK-12 PMF measure over a 3-year period and plotted the frequency of the increase or decrease (see image below). The size of the bubble represents the volume of schools in the specified range. In the first of these examples, we see the one year change in Math Achievement (PARCC 3+) ranged from -10% to +15%, though it was most heavily concentrated between +/-5%. For In-Seat Attendance (ISA), most schools experienced a change of only 1 percentage point in either direction. While there are definitely instances of schools making double-digit gains (or losses) in a given year, those are clearly the outliers.
In general, we found that a range of 2 to 4 percentage points is the largest sustainable annual gain for any given PMF measure – and often, even less. (As a rule of thumb, expect no more than 1% gain for measures in the Environment category. Elsewhere, 5% represents the high end of what’s attainable.) It is important to use this information to create both motivating and realistic goals.
Not all gains are treated equally under the PMF
Remember that every measure on the PMF carries a point value, and some measures are worth far more points than others (hello, MGP!). As a result, gaining 5% on one measure might barely eke out 1 point on the PMF, while the same 5% gain on a different measure could earn almost 4 times as many points. (Increasing the ELA 3+ Achievement outcome on the HS PMF by 5% produces a gain of 1.1 points under the 2019 business rules. Increasing the ELA Growth measure on the PK-8 PMF by the same 5% produces a gain of 3.6 points. (This ignores the relative difficulty of gaining 5% in these two categories since we’re only interested in how those gains are realized as points.))
The chart below illustrates these tradeoffs. Effort is defined as 1 standard deviation of year-over-year gain, while Gain is defined as the point increase for a change with effect size 0.4. The horizontal axis shows how difficult a measure is to influence – i.e. the effort required for sustained point gain. The vertical axis shows how valuable a measure is to the PMF – i.e. how many points can be earned for the same amount of gain. Notice, for example, how In-Seat Attendance carries a lot of payoff for every percentage point gained. But remember that each percentage point of attendance is hard to earn. Conversely, AP/IB enrollment is easy to influence, since theoretically a school could simply increase the number of students in AP classes and see who performs well. If it works out, it has one of the highest PMF weights. Of course, arbitrarily increasing
In the long term, the goal is sustained growth so that should be your driving force. However, if you’re torn between two equally important metrics, the difference in payoff can serve as a good tiebreaker.
(Effort was defined as 1 standard deviation of year-over-year gain (all of the metrics are conveniently symmetric about zero so losses were ignored). Gain was defined as the point gain for a change with effect size 0.4. The dashed lines represent mean values.)
Eating the Elephant
Our best advice around the PMF is to stop thinking of it as a single score. Instead, treat it for what it is: over a dozen separate measures, each of which captures a different aspect of a school’s health. As the saying goes: How do you eat an elephant? One bite at a time.
The key is to select one or two focal areas where your school can reasonably sustain growth over time. Here are some bonus questions that can help guide the process:
Is there a measure that is closely tied to your school’s mission?
Has one particular measure been trending downward faster (or upward slower) than the others?
If you have clearly-defined competitors, is there a particular measure where you underperform them?
Is there a measure that is dangerously close to the floor value (where you stop earning PMF points)?
No matter which question guides you or what school report card you are accountable to, make sure to take a deliberate approach when choosing which measures to go after for improvement. Create strategies that influence measures with higher weights, a higher chance for sustained growth, and therefore stronger chances of impacting your overall PMF score. Regardless of what your current report card score is, you have the opportunity to make long term changes.
[This post contributed by Laura Lund, DC Regional Data Manager and Adam Labay, Senior SIS & Database Specialist.]