Follow us:

Education Lab Blog

Education Lab is a project to spark meaningful conversations about education solutions in the Pacific Northwest.

December 4, 2013 at 6:00 AM

Ice cream doesn’t cause drowning and other warnings about interpreting data

Amid the national debate over how best to improve our nation’s public schools, data  from scientific studies often are used (and misused) to bolster one argument or discredit another – about the effectiveness of charter schools, say, or the value of standardized testing.

But how is an educator, policymaker or parent supposed to sort out credible evidence from the hype?

The science journal Nature recently published a list of 20 concepts that non-scientists should understand about scientific research.

Many of the concepts make good sense for evaluating education research, including this biggie that bears repeating often.

Correlation does not imply causation:  “It is tempting to assume that one pattern causes another,” according to the Nature article.  “However, the correlation might be coincidental, or it might be a result of both patterns being caused by a third factor — a ‘confounding’ or ‘lurking’ variable.”

A classic example is the observation that when ice cream sales go up, drownings increase. Does ice cream cause drowning? No, the lurking variable is a hot summer day, which boosts ice cream sales and swimming.

That may have been what Jack Buckley, the guy in charge of research at the U.S. Department of Education, was warning about this week in a Washington Post story  about new international testing data.

“People like to take international results like this and focus on high performers and pick out areas of policy that support the policies that they support,” he said. “I never expect tests like these to tell us what works in education. That’s like taking a thermometer to explain why it’s cold outside.”

Strong correlations are useful, however, in telling scientists where to dig deeper.

The last concept on the Nature list directly addresses educational research:

Extreme measurements may mislead: “Any collation of measures (the effectiveness of a given school, say) will show variability owing to differences in innate ability (teacher competence), plus sampling (children might by chance be an atypical sample with complications), plus bias (the school might be in an area where people are unusually unhealthy), plus measurement error (outcomes might be measured in different ways for different schools). However, the resulting variation is typically interpreted only as differences in innate ability, ignoring the other sources.”

In other words, figuring out how much of a student’s test score to attribute to the teacher’s competence rather than to other factors such as genetics, home life, prior schooling and the random error built into any complex measurement is difficult work and easily can go astray.

Comments | Topics: Research, U. S. Department of Education, Washington Post


No personal attacks or insults, no hate speech, no profanity. Please keep the conversation civil and help us moderate this thread by reporting any abuse. See our Commenting FAQ.

The opinions expressed in reader comments are those of the author only, and do not reflect the opinions of The Seattle Times.

The Seattle Times

The door is closed, but it's not locked.

Take a minute to subscribe and continue to enjoy The Seattle Times for as little as 99 cents a week.

Subscription options ►

Already a subscriber?

We've got good news for you. Unlimited content access is included with most subscriptions.

Subscriber login ►
The Seattle Times

To keep reading, you need a subscription upgrade.

We hope you have enjoyed your complimentary access. For unlimited access, please upgrade your digital subscription.

Call customer service at 1.800.542.0820 for assistance with your upgrade or questions about your subscriber status.

The Seattle Times

To keep reading, you need a subscription.

We hope you have enjoyed your complimentary access. Subscribe now for unlimited access!

Subscription options ►

Already a subscriber?

We've got good news for you. Unlimited content access is included with most subscriptions.

Activate Subscriber Account ►