"State education officials have refused comment on the cause of the idea's demise, but speculation has been that it had been difficult to square the idea with reality."

—Martin Cothran
Senior Policy Analyst
P. O. Box 22100, Lexington, KY  40522
Phone: 859-255-5400

Special to the Herald-Leader
 
Martin Cothran 
The Family Foundation
About 700 words 
First Serial Rights
© The Family Foundation 

Playing with the Numbers

The State School Board is considering a plan that would inflate state test scores.  But there are a few things we should get straight first.

SEVERAL YEARS AGO, state education officials changed the name of the state KIRIS tests to CATS, apparently figuring that the association with Kentucky basketball would help the credibility of the tests.

The name change in the test was designed to respond to critics of the test who had questioned its accuracy and academic focus.  The changes, however, were mostly cosmetic.

Now the State School Board is considering a much more significant change: this one having to do with the way the tests are graded.

In some cases, under the proposed plan, scores on the tests could be inflated as high as 400-700 percent.

In Kentucky’s testing system, there are four different grades or rankings for students and schools: Novice, Apprentice, Proficient and Distinguished—Novice being the lowest grade and Distinguished being the highest.  The idea was to get as many schools as possible into the Proficient category by 2014.

Under the new scoring system, many students would move from the Apprentice category (the second worst ranking) into the Novice category—a move down.  But huge numbers of students would move into the Proficient and Distinguished categories—up, rather than down, resulting, in many cases, in huge score increases.

In basketball terms, this amounts to moving the 3-point line closer to the basket.

If you move the 3-point line closer to the basket, scores go up, since many shots that before the change would have only counted for two will now count for three points.  It is a way of increasing the scores without the players having to play any better than they did before.

Here is what happens to some of the scores under the new scoring system, if you compare 1998 KIRIS scores to what the 2000 CATS scores would have been had they new scoring methodology been applied:

Under the new plan, no one has to do anything better.  But the players and the teams will look like they are.

Against this objection, the defenders of the Kentucky Education Reforms have an answer: We have to change the scoring system, since the old one is not accurate.

John Kenneth Galbraith once said that you should never try to defeat your opponent if he can be counted upon to defeat himself.  And self-defeat is a skill with which education officials in Kentucky have, on occasion, demonstrated a remarkable facility.

To say that a change in the scoring method is needed because of a lack of scoring accuracy in the past is to admit that what critics have said all along is true: that the tests have not accurately reflected student and school achievement.

It could be argued—and has been—that it is because of the changes that were necessary in moving from the KIRIS to the CATS tests that makes the scoring changes necessary.  But that doesn’t explain the dramatic inflation that also results from applying the new methods to the actual 2000 tests—after the changes took place.  Moreover, the inaccuracy apparently goes back much further than the change in tests in 2000.  One expert interviewed by the School Board in its April meeting speculated that the scoring inaccuracies could go as far back as 1993!

If the inflated scores that result from applying the new scoring system really are accurate, then the level of inaccuracy of past tests must have been great indeed.

The tests have had a checkered history, and have been plagued by controversy over their accuracy since their inception.  But even in the most heated moments, the State School Board has never backed down from its position that the tests were an accurate representation of student achievement.

But the argument for the new scoring plan essentially makes the opposite assumption: that the tests really weren’t accurate after all.

The implications of that assumption are far-reaching, and include questions such as: Have the millions of dollars of taxpayer money that has gone to schools gone the right schools?  Why are some scores being dramatically increases, and some (such as math scores) lowered?  Why are middle school scores under the new scoring system showing parity with high school scores?  Has the supposed “middle school problem” been illusory all along?

These are the kinds of questions that ought be answered before we change the testing system yet again.

Martin Cothran is senior policy analyst with The Family Foundation of Kentucky, a nonprofit educational organization dealing with public policy issues affecting families.