Why reading tests don’t actually test reading skills

Here’s a quick question: Do reading tests test reading skills? You may think that’s a question on the order of “Who’s buried in Grant’s tomb?” But if you said yes, the real answer may surprise you.

Pretty Young School Girl Reading A Book

Reading tests–of the standardized variety that kids in public schools take–are certainly supposed to test reading skills. The tests that have come into use in the past few years, which are aligned to the Common Core State Standards, are aiming to test skills like “finding the main idea and key details,” and “making logical inferences from text.”

But what if you can’t understand the reading passages, because they assume a lot of knowledge and vocabulary you don’t have? Would you be able to demonstrate your skill at finding the main idea or making inferences if you couldn’t understand what you’d just read?

That’s the problem that many kids–and especially those from low-income, less educated families–confront when they take reading tests. And that’s a huge reason why test scores for that segment of the student population have remained stubbornly abysmal.

This is a truth that some people–although not the majority of those involved in education–have long acknowledged. Cognitive psychologist Daniel Willingham has declared that reading tests are really just “knowledge tests in disguise.”

Now we have some good, hard evidence to back up that assertion with respect to the two most widely used Common Core-aligned reading tests. These are the tests designed by two consortia, one called Smarter Balanced and the other called PARCC. DC students take PARCC in 3rd through 8th grade, and again in high school, usually in 10th grade.

The evidence comes in the form of an analysis by Ruth Wattenberg, a member of the DC State Education, who studied released questions from the two tests and wrote up her findings for a new-ish organization called Knowledge Matters. That organization is making the case that if we want to teach kids to understand what they read–and prepare them for high school, college, and life–we need to teach them history, science, and the arts beginning in the early elementary grades.

Right now, most elementary school students–and especially those from low-income families, who are the least likely to pick up knowledge about these subjects at home–are getting a steady diet of math and reading. “Reading” usually means a lot of work on foundational skills like phonics–which is important. But it also means hours of practicing supposedly abstract reading comprehension skills like “finding the main idea,” with the theory being that once you’ve perfected that skill on simple texts, you’ll be able to apply it to find the main idea of any text, no matter how complex.

That theory–as many experiments have demonstrated–is actually wrong. While some practice with reading comprehension skills can be helpful, the main determinant of whether you can understand what you’re reading is whether you have the relevant background knowledge and vocabulary. And the best way to expand your background knowledge and vocabulary is to learn about things like history, science, and the arts.

The people who design the PARCC and Smarter Balanced tests actually understand this. Since there’s no way of knowing which students have studied what, the idea is to cover a wide variety of topics, to balance out whatever unfair advantages in background knowledge some kids might have. If one group of kids has studied snowy owls, for example, and there’s a passage on snowy owls on the test, they might ace that. But then the next passage might be on ancient China, which they know nothing about–but some other kids might have studied that.

Of course, these days, students in many elementary schools are unlikely to have studied much of anything aside from reading and math skills. So the wide variety of topics on reading tests privileges the kids who have acquired a fair amount of knowledge from home. And those kids tend to be the wealthier ones.

Wattenberg looked primarily at 3rd-grade questions about “informational text” passages, most of which had something to do with geography or science. She found that even these 3rd-grade passages assumed a lot of knowledge and vocabulary that many eight-year-olds would be unfamiliar with–concepts like the names and general locations of continents and units of measurement like kilograms, as well as vocabulary like “rodent, “intestine,” “crystallizes,” and “Aurora Borealis.”

From interacting with low-income high school students and speaking with some of their teachers, I know there are many high school students who are unfamiliar with these words and concepts. I can only imagine the confusion of a low-income 3rd-grader when confronted by them.

Here’s an example of just one paragraph from one of the reading passages Wattenberg analyzed, with the words she identified as challenging replaced by a nonsense word, blunk. See if you can make sense of it:

In one of the most blunk places in the world, the blunk blunk, a people have survived over a blunk of [sic] years. They are the blunk. For the blunk, the blunk is a place blunk with life. Depending on how far north they live, the blunk find everything from blunk blunk and blunk blunk to blunk blunk. The blunk have blunk themselves to the various blunk they blunk. At one time they were considered to be among the healthiest people in the world. This is no longer the case; the blunk blunk has changed blunk over the past blunk. The arrival of blunk and modern blunk resulted in big changes to the blunk diet and blunk blunk blunk.

Here’s the passage with the words included, for the sake of comparison:

In one of the most remote places in the world, the Canadian Arctic, a people have survived over a thousand of years. [sic] They are the Inuit. For the Inuit, the Arctic is a place teeming with life. Depending on how far north they live, the Inuit find everything from Caribou herds and polar bears to beluga whales. The Inuit have adapted themselves to the various regions they inhabit. At one time they were considered to be among the healthiest people in the world. This is no longer the case; the Inuit lifestyle has changed dramatically over the past decades. The arrival of southerners and modern technology resulted in big changes to the Inuit diet and way of life.

The point is not that we should lower our expectations for what 3rd graders can read and understand. The point is that it’s unrealistic to expect kids to meet those expectations if we haven’t given them—and their teachers—the tools that will enable them to do so.

Wattenberg’s analysis is well worth reading in its entirety. You can find it here.


  1. Sarah says

    It’s great to see another post from you and you are so right that it’s “well worth reading” Ruth Wattenberg’s study/analysis in Knowledge Matters.. What a breath of fresh air! Many thanks for the link.

Leave a Reply

Your email address will not be published. Required fields are marked *