Ok I said I’d do this and here it is. The National Assessment of Educational Progress (NAEP) employs “scientific sampling” of students to take various standardized tests. You can go to the NAEP data explorer at
http://nces.ed.gov/nationsreportcard/naepdata/" onclick="window.open(this.href);return false; (click on “NDE Main”) and break things down by demographics.
For this post I looked at the 2009 8th Grade Math test. Massachusetts students averaged 299 while Texas students averaged 287. The difference is “statistically significant.” However, the idea that the overall average suggests that Massachusetts has a better educational system is an illusion that disappears when demographics are taken into account.
I’ll just take three major factors into account to the extent that I can. They are race, socioeconomic status, and parental education level. Race is self-explanatory. Socioeconomic status is addressed through looking at whether or not students are eligible for the school lunch program (poor vs. not poor). Parental education level reported by NAEP ranges from parents who did not finish high school to parents who graduated college. It is the highest level achieved by at least one parent.
So here is how Massachusetts and Texas compare when you compare apples and apples by breaking it down according to those factors into race/socioeconomic status/parental education groups (for groups such that sample sizes were large enough for NAEP to report averages):
Poor Hispanic children of parents who didn’t finish high school: Massachusetts 261, Texas 272
Poor Black children of high school graduates: Massachusetts 260, Texas 263
Poor Hispanic children of high school graduates: Massachusetts 270, Texas 274
Poor Black children of parents with some education after high school: Massachusetts 272, Texas 275
Poor Hispanic children of parents with some education after high school:
Massachusetts 269, Texas 284
Poor White children of college graduates: Massachusetts 297, Texas 293
Poor Black children of college graduates: Massachusetts 280, Texas 271
Poor Hispanic children of college graduates: Massachusetts 267, Texas 279
Not poor White children of high school graduates: Massachusetts 292, Texas 295
Not poor White children of parents with some education after high school: Massachusetts 295, Texas 300
Not poor White children of college graduates: Massachusetts 315, Texas 311
Not poor Asian or Pacific Islander children of college graduates: Massachusetts 334, Texas 325
Texas students in eight of the 12 groups for which comparisons are available had the higher sample averages. There was only one “statistically significant” difference between States when things were compared group by group as indicated. That one, in bold above, had poor Hispanic children of parents with some education after college in Texas scoring higher than those in Massachusetts.
I might add that this is just scratching the surface in terms of adjusting for demographics. I picked some major factors but there are many other factors. Plus the precision of what I can do is limited by the way NAEP categorizes things. For instance: Categorizing students as being eligible for the school lunch program or not does not fully capture differences in socioeconomic status. Billionaires go into the same group as middle class people. Parental education doesn’t fully capture that factor either. Children with two parents holding doctorates go into the same category as children who have one parent holding a bachelors degree and the other parent without a college degree. So on and so forth.
The fact that one can eliminate the difference between Texas and Massachusetts with such a crude, preliminary demographic adjustment demonstrates how fallacious the idea of judging educational systems without taking student demographics into account is.
But it is done ALL the time. My belief is that it’s done because of “political correctness.” I think there is an aversion to being perceived as “blaming” the students when school systems don’t perform well in terms of test results. But if demographics are taken into account and standardized test scores are used as the measure, the perception of those red States in gerneral and the South in particular as having inferior educational systems is seen as substantially exaggerated and perhaps downright false.