Teaching to the Test...what it means...what TN is doing about it...

It's been really busy in my project with the beginning of school, especially with all the "Expect More, Achieve More" campaign and the scores going down as a result. My apologies.
Recently, there was a lot of movement in Los Angeles around releasing INDIVIDUAL TEACHER value-added scores in the newspaper. See information on that HERE and how it drew support from Sec. of Education HERE. It may be a while before that happens in TN, if it ever happens, but it's been interesting to see all the different voices coming out of this conversation


In thinking about this, and as I go out and meet with a lot of people, there's usually someone who ends up asking, "Isn't this all about teaching to the test?" (especially with the new standards coming online in TN). Since news travels fast, I ended up trading a few e-mails with Dr. Stone from the Education Consumer Foundation, an organization that's very supportive of value-added data and testing as a way to hold educational leaders accountable.
I thought the exchange was interesting...I asked whether or not there was research that backed up value-added scores and how reliable they are, assuming you have some students who do not test well.  If you're interested, read on...
-----------------------
There are some kids who are not good test takers. Moreover, on any given day, some students will have a bad day. This kind of problem exists with batting averages, passes completed, and all kinds of human phenomena that are measured, reported, and never questioned.
But teacher effect scores are averages of repeated measurements of all students taught by a teacher over a 3-year period. Teacher performance quality is judged by how teachers compare to other teachers. No teacher is likely to have all of the poor test takers and ones having bad days over a period of years.


And if all of these precautions were not enough, there are statistical corrections in the data to prevent spurious outcomes and the results are interpreted conservatively, i.e., only the extremely effective or extremely ineffective are identified as such.


The problem with the question is that it accepts the notion that there exists some [unspecified] indicator of "teacher effectiveness" that is stable and accurately measured, and to which TVAAS teacher effect data can be compared. In truth, there are lots of intuitively appealing ways to assess what teachers do--like extended classroom observations--or to look at how students respond to instruction--like individual interviews with students or very long and thorough performance assessments--but all are fallible and most are far more fallible than standardized tests!


What you need to know is that the tests that are used are fallible but they are given repeatedly such that the false negative scores are canceled out by false positive scores. In the real world, the best estimate of the true quantity of anything is the average of repeated fallible measurements.


Teaching to the test may or may not be a problem--depending on how the testing is done. The problem with "teaching to the test" comes about when a test is constructed to sample a broader domain of knowledge or skill and teachers teach only the specifics of the sample, not the broader domain. Student performance on the sample might look good but the scores are misleading. But when the test items are written fresh every year or they are randomly drawn from some huge pool of items (TCAP does both), the chances of a teacher succeeding through teaching to the test are lowered substantially.
Broadly speaking, we need to get used to the idea that if we are going to get serious about holding teachers and students accountable, we are going to have to get serious about how we do testing. Just like we externally audit things like banks and corporations, we need to audit how testing is being done. Students have been known to cheat. Teachers have been known to behave unethically--doing things like teaching to the test. When the outcome is important and there are potential conflicts of interest, sensible people pay attention.
----------------------
Interestingly, the state of TN was just awarded money from the federal government to work on developing new tests... SEE THAT HERE.  So, if we're making better tests, then that might help with the "teaching to the test" critique. 
----------------------
By the way, the new Teacher Evaluation Advisory Committee has offered its initial recommendations for the new evaluation system HERE.