Amid the national debate over how best to improve our nation’s public schools, data from scientific studies often are used (and misused) to bolster one argument or discredit another – about the effectiveness of charter schools, say, or the value of standardized testing.
But how is an educator, policymaker or parent supposed to sort out credible evidence from the hype?
The science journal Nature recently published a list of 20 concepts that non-scientists should understand about scientific research.
Many of the concepts make good sense for evaluating education research, including this biggie that bears repeating often.
Correlation does not imply causation: “It is tempting to assume that one pattern causes another,” according to the Nature article. “However, the correlation might be coincidental, or it might be a result of both patterns being caused by a third factor — a ‘confounding’ or ‘lurking’ variable.”
A classic example is the observation that when ice cream sales go up, drownings increase. Does ice cream cause drowning? No, the lurking variable is a hot summer day, which boosts ice cream sales and swimming.
That may have been what Jack Buckley, the guy in charge of research at the U.S. Department of Education, was warning about this week in a Washington Post story about new international testing data.
“People like to take international results like this and focus on high performers and pick out areas of policy that support the policies that they support,” he said. “I never expect tests like these to tell us what works in education. That’s like taking a thermometer to explain why it’s cold outside.”
Strong correlations are useful, however, in telling scientists where to dig deeper.
The last concept on the Nature list directly addresses educational research:
Extreme measurements may mislead: “Any collation of measures (the effectiveness of a given school, say) will show variability owing to differences in innate ability (teacher competence), plus sampling (children might by chance be an atypical sample with complications), plus bias (the school might be in an area where people are unusually unhealthy), plus measurement error (outcomes might be measured in different ways for different schools). However, the resulting variation is typically interpreted only as differences in innate ability, ignoring the other sources.”
In other words, figuring out how much of a student’s test score to attribute to the teacher’s competence rather than to other factors such as genetics, home life, prior schooling and the random error built into any complex measurement is difficult work and easily can go astray.