State Demographics-Adjusted Standardized Test Score Rankings

[JohnStOnge]

My gut (sorry no links for intuition) tells me that John started with a solution in mind and then went looking for statistics that supported the desired solution. A classic mistake that goes against true scientific and/or statistical analysis. I'll give him credit for doing the legwork but that doesn't mean I have to buy the results.

Your gut is incorrect. See my next post. I never do that. The results drive my beliefs rather than visa versa. At least in terms of what the levels and associations are. I do have beliefs about what causes the levels and associations. And those beliefs could never be proven or disproven. It's the nature of the situation.

[JohnStOnge]

So you found a website that agrees with your hypothesis and you linked to it here. Nice. If it's on the internet it must be true.

The link I posted takes you to a query page on the US Department of Education National Assessment of Educational Progress (NAEP) web site. Click on "Main NDE" and you will get a window asking you to agree to terms. Agree to the terms and it you'll be taken to the query page.

The NAEP tests "scientific" samples of US students to test how well they are doing in various areas. What that means is that you can relate the results to the populations represented. For instance, if you do a query that shows among the results an average score for White students such that at least one parent graduated college and who are not eligible for the school lunch program you are looking at an average based on a "scientific" probability sample of such students nationally. That means the average you are looking at is an unbiased estimate of what the average would be if every student in that group in the country took the test. You can, in other words, legitimately extrapolate the results to the national population being described. It's a VERY good information source.

You can use the query to break things down as you want to see if I'm misrepresenting anything. You should be able to readily select the 2013 8th grade mathematics test. Then you can select "National" for the sample. Then you can go to the "select variables" tab and select School Lunch Program Eligibility, Parental Education Level, and Race/Ethnicity. If you just want to see the results to the nearest whole number you can skip the "Edit Reports" tab and go to the "Build Reports" tab. The report I've been taking numbers from, for the most part, is the "Cross-Tabulated Report 1."

Do the query. Generate the report. Look at it yourself. I'm not misrepresenting anything.

[kalm]

So you found a website that agrees with your hypothesis and you linked to it here. Nice. If it's on the internet it must be true.

The link I posted takes you to a query page on the US Department of Education National Assessment of Educational Progress (NAEP) web site. Click on "Main NDE" and you will get a window asking you to agree to terms. Agree to the terms and it you'll be taken to the query page.

The NAEP tests "scientific" samples of US students to test how well they are doing in various areas. What that means is that you can relate the results to the populations represented. For instance, if you do a query that shows among the results an average score for White students such that at least one parent graduated college and who are not eligible for the school lunch program you are looking at an average based on a "scientific" probability sample of such students nationally. That means the average you are looking at is an unbiased estimate of what the average would be if every student in that group in the country took the test. You can, in other words, legitimately extrapolate the results to the national population being described. It's a VERY good information source.

You can use the query to break things down as you want to see if I'm misrepresenting anything. You should be able to readily select the 2013 8th grade mathematics test. Then you can select "National" for the sample. Then you can go to the "select variables" tab and select School Lunch Program Eligibility, Parental Education Level, and Race/Ethnicity. If you just want to see the results to the nearest whole number you can skip the "Edit Reports" tab and go to the "Build Reports" tab. The report I've been taking numbers from, for the most part, is the "Cross-Tabulated Report 1."

Do the query. Generate the report. Look at it yourself. I'm not misrepresenting anything.

IF you are correct, what would you do with the information?

[BlueHen86]

The link I posted takes you to a query page on the US Department of Education National Assessment of Educational Progress (NAEP) web site. Click on "Main NDE" and you will get a window asking you to agree to terms. Agree to the terms and it you'll be taken to the query page.

The NAEP tests "scientific" samples of US students to test how well they are doing in various areas. What that means is that you can relate the results to the populations represented. For instance, if you do a query that shows among the results an average score for White students such that at least one parent graduated college and who are not eligible for the school lunch program you are looking at an average based on a "scientific" probability sample of such students nationally. That means the average you are looking at is an unbiased estimate of what the average would be if every student in that group in the country took the test. You can, in other words, legitimately extrapolate the results to the national population being described. It's a VERY good information source.

You can use the query to break things down as you want to see if I'm misrepresenting anything. You should be able to readily select the 2013 8th grade mathematics test. Then you can select "National" for the sample. Then you can go to the "select variables" tab and select School Lunch Program Eligibility, Parental Education Level, and Race/Ethnicity. If you just want to see the results to the nearest whole number you can skip the "Edit Reports" tab and go to the "Build Reports" tab. The report I've been taking numbers from, for the most part, is the "Cross-Tabulated Report 1."

Do the query. Generate the report. Look at it yourself. I'm not misrepresenting anything.

IF you are correct, what would you do with the information?

He would discuss it in a series of hour long videos on YouTube.

[JohnStOnge]

IF you are correct, what would you do with the information?

I'd take all demographic factors that are associated with variation in test scores, including race, into account instead of ignoring race because it's not "politically correct" to consider it. If I put out rankings of school system units at any level...schools within a district, district within a State, States within the Country, etc....I would include rankings adjusted for demographics of student populations.

Perhaps MOST importantly I would make sure that ANY system for rating teachers takes the nature of the students he or she is serving into account in some way. If it's too politically "hot" to fully consider demographics, I would do something like have baseline testing of each student at the beginning of each school year. Then any assessment of teacher performance based on standardized testing at the end of the school year would be adjusted to account for where that student was as measured by the baseline test at the beginning of the school year.

If you look at the raw average NAEP 8th grade math scores I've been using Massachusetts is ranked #1. Do you know what would happen if you took the people who run that State's education system and put them in Mississippi with the same money they have per student in Massachusetts, etc., and told them to run the Mississippi school system? They'd almost certainly end up near the bottom in the Standardized Test score rankings because they'd be dealing with a much more difficult situation in Mississippi in terms of demographics than they have to deal with in Massachusetts.

It's just not right that we continue to piss all over educators who have the most difficult problems to deal with because we don't want to admit that demographics matter.

[BlueHen86]

IF you are correct, what would you do with the information?

I'd take all demographic factors that are associated with variation in test scores, including race, into account instead of ignoring race because it's not "politically correct" to consider it. If I put out rankings of school system units at any level...schools within a district, district within a State, States within the Country, etc....I would include rankings adjusted for demographics of student populations.

Perhaps MOST importantly I would make sure that ANY system for rating teachers takes the nature of the students he or she is serving into account in some way. If it's too politically "hot" to fully consider demographics, I would do something like have baseline testing of each student at the beginning of each school year. Then any assessment of teacher performance based on standardized testing at the end of the school year would be adjusted to account for where that student was as measured by the baseline test at the beginning of the school year.

If you look at the raw average NAEP 8th grade math scores I've been using Massachusetts is ranked #1. Do you know what would happen if you took the people who run that State's education system and put them in Mississippi with the same money they have per student in Massachusetts, etc., and told them to run the Mississippi school system? They'd almost certainly end up near the bottom in the Standardized Test score rankings because they'd be dealing with a much more difficult situation in Mississippi in terms of demographics than they have to deal with in Massachusetts.

It's just not right that we continue to piss all over educators who have the most difficult problems to deal with because we don't want to admit that demographics matter.

Admit it, this is going on YouTube. White people are smarter than blacks because they have more alien DNA. 16 hours of poorly produced video should be enough time to cover it.

[JohnStOnge]

Admit it, this is going on YouTube. White people are smarter than blacks because they have more alien DNA. 16 hours of poorly produced video should be enough time to cover it.

[YoUDeeMan]

Why is everyone killing the messenger?

The information is right there...if you doubt it, run your own tests.

[kalm]

Why is everyone killing the messenger?

The information is right there...if you doubt it, run your own tests.

It's boring and JSO is pretty much admitting what's apparent. Educational problems are mostly socio-economic problems.

[biobengal]

Show us the model and equations you used. Peer review, etc.

No peer review but the model is SCORE = 238.662211718951 + 0.268281205096343 x (% population Asian only + % population Non Hispanic White) + 0.920976742418761 x (% population with at least a Bachelors degree).

That model was used to "predict" what one would expect a state average score would be. Then the predicted score was subtracted from the actual average score to yield a departure from the expected score.

So, for instance, the actual score for Texas is 288.20. The predicted score for Texas based on how things work nationally for the referenced demographics is 275.95. So Texas "overachieved" by 12.25. That's how it works.

I gave it a go for science scores for 8th graders and included % of each race, % bachelors and median income. Top model explained 75% of the variation and included %white, %bachelors and %black. %black explained just 4% of this variation (using partial rsq). Thus, %white and %bachelors were most important... who knew? Deviation from predictions were calculated:

Better than expected (rank to right):

South Dakota-5
Georgia-4
Virginia-3
North Dakota-2
Texas-1

Lower than expected:

Rhode Island-50
California-49
Alabama-48
West Virginia-47
Illinois-46

Some agreement with John's work... but our models did not agree on the importance of the different factors, including %bachelors degrees. I also examined science scores. I used AIC model selection, not sure what John used. Yet, so many problems with this analysis: 1) science scores different from math, presumably different from other subjects, 2) snap shot of a single year; these scores (I presume) can wiggle a bit, 3) factors are hopelessly correlated, making the selection of the different factors difficult and 4) not enough computing power to do a real analysis. Fun stuff; however, not sure if I agree with your "theory". Still, respectable job with limited data.

[kalm]

No peer review but the model is SCORE = 238.662211718951 + 0.268281205096343 x (% population Asian only + % population Non Hispanic White) + 0.920976742418761 x (% population with at least a Bachelors degree).

That model was used to "predict" what one would expect a state average score would be. Then the predicted score was subtracted from the actual average score to yield a departure from the expected score.

So, for instance, the actual score for Texas is 288.20. The predicted score for Texas based on how things work nationally for the referenced demographics is 275.95. So Texas "overachieved" by 12.25. That's how it works.

I gave it a go for science scores for 8th graders and included % of each race, % bachelors and median income. Top model explained 75% of the variation and included %white, %bachelors and %black. %black explained just 4% of this variation (using partial rsq). Thus, %white and %bachelors were most important... who knew? Deviation from predictions were calculated:

Better than expected (rank to right):

South Dakota-5
Georgia-4
Virginia-3
North Dakota-2
Texas-1

Lower than expected:

Rhode Island-50
California-49
Alabama-48
West Virginia-47
Illinois-46

Some agreement with John's work... but our models did not agree on the importance of the different factors, including %bachelors degrees. I also examined science scores. I used AIC model selection, not sure what John used. Yet, so many problems with this analysis: 1) science scores different from math, presumably different from other subjects, 2) snap shot of a single year; these scores (I presume) can wiggle a bit, 3) factors are hopelessly correlated, making the selection of the different factors difficult and 4) not enough computing power to do a real analysis. Fun stuff; however, not sure if I agree with your "theory". Still, respectable job with limited data.

Still boring.

[JohnStOnge]

Some agreement with John's work... but our models did not agree on the importance of the different factors, including %bachelors degrees. I also examined science scores. I used AIC model selection, not sure what John used. Yet, so many problems with this analysis: 1) science scores different from math, presumably different from other subjects, 2) snap shot of a single year; these scores (I presume) can wiggle a bit, 3) factors are hopelessly correlated, making the selection of the different factors difficult and 4) not enough computing power to do a real analysis. Fun stuff; however, not sure if I agree with your "theory". Still, respectable job with limited data.

I just did an ordinary least squares regression using the "linest" function in Excel (yes I know people say Excel makes errors but I went for about a year checking results I got with linest against results I got from a regression calculator at http://www.wessa.net/rwasp_multipleregression.wasp" onclick="window.open(this.href);return false; they were always the same so I finally stopped). I wasn't going to use interaction terms or terms raised to powers so there were only seven possible models. Among those models I just picked the one producing the highest r2 value such that all coefficients were "significant" at α = 0.05. The results for the individual regressions are below.

%People below poverty level is SES
%People with at least bachelors degree is EDU
%People who are non Hispanic White or Asian only is RACE

Results significant at α = 0.10 underlined.

SES r2 = 0.519

EDU r2 = 0.441

RACE r2 = 0.268

SES + EDU r2 = 0.563

SES + RACE r2 = 0.593

EDU + RACE r2 = 0.695

SES + EDU + RACE r2 = 0.707

The reference I use for the purpose recommends using Variance Inflation Factor to decide whether or not the model has combinations of variables that are too close to colinear. And it gives you a quick and dirty way to do it. What you do is take each independent variable in a multiple regression and regress it against the other independent variables. Then if you don't get a high r2 value you don't have to eliminate any variables. The reference actually allows you to get a r2 value of 0.9 before it says you have to remove variables to eliminate getting too close to colinearity. In practice I don't go that high.

But none of the four multiple variable models above are characterized by any colinearity problem; at least according to that reference (which is at my office so I can't cite it now). The model I ended up picking is [EDU] + [RACE} and the r2 value you get between the two independent variables is 0.021 so, at least according to the reference I use, near colinearity is not an issue at all.

The "theory" is simply that demographics matter and that if you are fair about ranking State education systems by looking at standardized test scores you have to adjust for demographics. And doing something to consider demographics changes things. As I said what I did is crude. It looks at the "background" in which State education systems exist but it does not involve direct assessments of the student populations. As is just about always the case there are violations of the assumptions of linear regression. I didn't analyze residuals or anything (though I don't think that matters in terms of the bottom line because the significance level of each coefficent in the final model is < 0.00000011; as a practical matter there's no doubt that both of those variables are strongly associated with variation in test scores).

I was surprised that SES was not "significant" when both EDU and RACE were in the model because I know from looking at NAEP data that SES is a factor when parental education level and race are taken into account. But I'd decided ahead of time on how I was going to get the model I was going to use so I stuck to that. That's actually important in statistics because all of the probability statements involve what the probability of things happening were before you started. If you go back and change what you planned to do because you didn't get the result you expected or wanted you are fudging on the probabilities. As I've said many times I think a person can to a large extent end up with a model saying what they wanted said because we have computers and can do things like generate a bunch of models then select the "best" one among them. We can keep playing around ad infintim with adding terms, raising terms to powers, creating interaction terms, transforming variables, etc. to increase the chances of getting what we want. That's why I like to keep it simple and build a few models based on what I think will happen ahead of time.

It would be better to be able to adjust the NAEP data for the actual demographics of the students who took the test. But I have not been able to figure out how to do that because they do not provide any information on the numbers of students in each group via that calculator.

Nevertheless I think the results are consistent with the "theory" that demographics matter and school systems should not be evaluated based on standardized test scores without taking them into account. And by that I mean ALL demographic factors that are seen be characterized by "significant" associations with test scores. RACE is such a factor. If you spend some time using that query I linked breaking things down and you are honest about it you will say that RACE is as important a factor as there is and it doesn't go away when you "control" for other factors.

I know that people argue that it's not RACE itself but rather something else environmental associated with race. Like culture, which would be very hard to quantify, is often cited. If people making that argument are correct then RACE is really a proxy for culture and maybe some other things. But as a matter of math the variable RACE is still very important and I don't think people help things by trying to avoid saying so. Whatever is going on with it it's not easy to "fix" so it's going to impact the standardized test scores of students in the charge of a given teacher or system.