Data Doesn’t Live in a Vacuum

ICYMI – here’s a great article on data from DAC Chair and public school advocate Jim Earley. When you hear that 50% of kids are failing or public schools are failing, know that you’re getting a load of…you can pick the word. Data is meant to be a tool to inform instruction and next steps, not to be used as a weapon again kids, schools, teachers, or districts.

Data Doesn’t Live in a Vacuum

Context matters. Data is a valuable tool — if it’s used appropriately. However, when data is taken out of context, or facts are presented in the absence of other relevant data, the result is that we make bad decisions. For example, we can look at a hypothetical company’s profit and loss statement that shows they lost money in the last quarter. On it’s own, one might consider the company to be failing. But what if that same company’s bottom line doesn’t show that they decided to defer profits for a couple of quarters to make significant investments that would deliver dramatic profit down road? What if, quarter over quarter, net losses dropped dramatically, and year over year, these losses are lower than predicted? Is that company still a failure? Should we divest? Of course, if we only evaluated the company’s performance based on the current quarter, a loss might make us reconsider investment. However, when we look at data comparatively, and in a broader context, the story changes and our decision to divest might be premature.

Similarly, when we consistently hear from pubic school opponents in Jeffco that, “50% of 3rd graders are not meeting expectations in ELA (English Language Arts),” the first reaction might be to think that something is really wrong. However, if we look at data in a vacuum, we aren’t getting a true picture of what is going on in our schools, or how teachers and school districts are moving the needle to improve learning outcomes. Nor does it tell us other important parts of the story that put the data into appropriate context.

For starters, it’s true that 45.4% of 3rd graders met or exceeded expectations on the 2016–17 CMAS tests based on cut scores in Jeffco. What is really going on? Are schools really failing? And if we’re to believe public school opponents, are teachers really not effective, and are district leaders really failing to improve academic performance? Based on that single data point, it might be hard to believe that academic achievement is growing. But it is — this is when context matters.

First, 3rd grade ELA scores the previous year show that 42.3% of students met or exceeded the same expectations. That’s a 3.1% increase in percentage of students, year over year. For a district with 86,000 kids and 157 schools, moving the needle that much is quite a feat. The problem, however, is that CMAS only measures learning as a single snapshot in time, typically in the spring. It doesn’t show where students were at the start of the year. For this we can use other bodies of evidence, like the DIBELS assessments to measure beginning of year and end of year growth in reading. This is administered to most students from Kindergarten to Grade 3. This accounts for roughly 23,000 students in Jeffco. At the start of the year, 4,270 students were reading significantly below grade level; by the end of the year, 52% of these students had demonstrated improvement in their reading ability; 32% of these students had demonstrated reading at grade level or above. Now, when we put this into context with CMAS scores, it adds to the body of evidence of how much learning is going on.

DIBELS Benchmark Testing By Grade (K-3) Comparing Beginning of Year with End of Year

Testing By Grade (K-3) Comparing Beginning of Year with End of Year

What public school opponents never mention is that this only the third year that CMAS has been administered. This assessment is intentionally much more rigorous than previous assessments. For example, take a look at the following sample questions from the 3rd grade TCAP assessment, and compare with a question from the 3rd grade CMAS assessment (below). The level of ability required to answer the CMAS question requires students to not only comprehend the text, and summarize the main points, it also requires students to compare and contrast key aspects of the text and provide evidence in essay form.

Educators expected a drop in scores. They did. However, the data shows that score are growing compared to that baseline year three years ago. This demonstrates that not only are students learning to reiterate facts, they’re learning to do much more including analyze, assess, compare, contrast and defend their work through evidence in the texts. That’s tremendous level of skill for a third grader to demonstrate.

3rd Grade Sample TCAP Question

3rd Grade ELA Sample Question

 

Another facet to understanding how big this is, compare Jeffco’s scores with other districts across the state. 3rd grade ELA scores for Douglas County School District (DCSD) show that 49.7% of students met or exceeded the standard. At first glance, that number is larger than the 45.4% in Jeffco. However, DCSD scores dropped 2.1% year over year. There are two other important reasons for comparing Jeffco with DCSD. The first is that DSCD has a substantially lower Free and Reduced Lunch (FRL — a measure of poverty) rate compared to Jeffco: 12% for DSCD compared to 30% for Jeffco. A wealth of research shows that poverty is the largest, most significant factor in academic achievement. By virtue of that, we should expect DSCD’s percentage of 3rd graders meeting and exceeding expectations to be much higher.

The second reason for comparing against DSCD is the result of a decade-long history of reforms that have drastically impacted kids and families, teachers, administrators, and the community. At one time, DCSD was one of the highest performing districts in the state; they no longer can claim that accolade. In fact, they currently under-perform because of policies that have diminished the value of teachers and school administrators. So much so, that morale increased teacher and administration attrition rates, with several staff members going public with their dissent with district policies.

We’ve also heard from public school opponents that 4th grade scores are roughly the same as 3rd grade. Ah, but this is where context matters even more. 51.8% of 4th graders met or exceeded expectations in ELA. Note that these were last year’s 3rd graders. That means that we saw an increase of nearly 10% of students who didn’t meet expectations the previous year now achieving at grade level or above.

Of course, there are tremendous challenges that have to be addressed by Jeffco. Achievement gaps between different demographic groups remain large and persistent. For example, there is a 24% gap between White and Hispanic students, and 26% gap between non-FRL and FRL students, indicating that socioeconomic obstacles that affect our communities are also manifesting themselves in academic achievement. Until these issues are addressed by communities, achievement gaps will continue.

What we’ve demonstrated is that relying on a single point of data as evidence of a school’s or district’s performance not only fails to show the full story of how they’re doing. Data doesn’t live in a vacuum. It’s a tool that is intended to be used to compare and to diagnose, and to identify and measure areas for improvement. However, when data is used inappropriately and out of context, particularly for political agendas, it becomes a cynical instrument that harms kids and families, teachers and administrators, schools and districts, and the community as a whole. It’s deceitful, reckless, and irresponsible. Context matters.

Note: All CMAS data cited in this article can be found at: https://www.cde.state.co.us/assessment/2017cmasdistschoveralresultsmathela