Wednesday, April 27, 2011

Student Performance on NJ State Tests: A Poor Measure of Teacher Effectiveness

Graphic from this Site
I. The State Test

In 2010, 86,000 11th graders in New Jersey wrote responses to a "persuasive" prompt as part of the high stakes state assessment. Student who do not pass the assessment cannot graduate. This was the prompt:

Writing Situation
Several teenagers in the neighborhood are suing a local fast food restaurant, claiming that their poor health resulted from consuming the restaurant's food. The lawsuit has forced the owners to close the restaurant. This has caused a controversy in the community.
You decide to write an article for your school newspaper expressing your opinion on the teens' lawsuit against the restaurant.
Directions for Writing
Write an article either supporting or opposing the teens' lawsuit. Use reasons, facts, examples, and other evidence to support your position.

These 86,000 students were not actually allowed to conduct any research or gather facts, examples or other evidence to support their fictitious positions, as no resources nor external research may be used, nor did their response actually have to ascribe to the genre specified (newspaper article).  Rather students had to invent reasons, facts, examples, and other evidence that supported being for or against the teens' lawsuit and then put these "ideas" into paragraphs (hopefully 5) and do so in 60 minutes.

As I read the prompt and thought of so many students composing responses, I wondered what students had learned about writing and persuasion?
  • Did they learn that making up reasons, facts, examples, and other evidence is acceptable practice when writing to persuade?
  • Did they learn that the five-paragraph response penned in an hour is the same thing as composing a newspaper article?
  • Did they learn that all persuasive positions result in either being for or against something?
  • Did they learn that one takes a definitive position after being presented with only a situation, sans details or other texts?
  • Did they learn that research does not require any searching?
  • Did they learn that persuasion does not require actual facts, truthful statistics, reasoned examples?
  • Did they learn that writing is a task you do without careful and honest thought?
  • Did they learn that audience doesn't actually matter to a writer?
Although I don't write a lot of persuasive texts, I cannot recall an instance when I resorted to making up facts, reasons, examples, or other types of evidence to prove being for or against something.  Anywhere I have worked, such an approach would be classified as academic dishonesty, not a strategy for writing. Is this an apt measure that would allow you to feel confident that your child had developed the requisite skills, dispositions, and habits of mind to be able to and want to continue learning after high school? I would be disappointed if any of my child's teachers taught my son to write based on this limited sense of composing.

Now to be sure, this one task is not the whole of the two-day assessment.  Students also read two different texts, answer multiple choice and open ended response questions, and write another response, this time in 30 minutes to different prompt.  None of the work students do during this test would one characterize as examples of authentic work, the type one would expect do outside of a testing situation.  For example, when was the last time you tried to identify the "best" central meaning of a text you were reading? Or spent time identifying a specific literary term with a sentence from the text you were reading? Is being able to select the statement from a field of four that includes personification really an important indicator of "language arts literacy" prowess?

II. Inside English Teachers' Classes
A few months after students took the HSPA, I was visiting a high school English teacher's class.  This is the narrative I recorded after visiting:

Students in Jillian Honoria's English class are performing monologues based on their reading of Fyodor Dostoevsky’s Crime and Punishment on the day I visit. Once seated I quickly noticed the life-size doll propped on a rolling desk chair.  Dressed in a black suit, white shirt, and scruffy boots, this class-made version of Raskolnikov, the protagonist from Crime and Punishment, sported wild black hair topped with a hat.  During the class period, the students would address him as they performed monologues written from the perspective of other characters from the text.  Dr. Honoria began class by having students compose a 10-minute journal entry.Within seconds, the room was silent, save the noise of everyone in the class writing. After some discussion about the texts students had written, the student performances began. The students’ performances demonstrated their significant understanding of the text.  Dr. Honoria directed students to choose a character other than the protagonist, compose a written monologue that would extend the narrative in some manner, and then perform the monologue in class.  One student stepped into the character of Sonya, Raskolnikov’s love, and set the scene for her monologue in Siberia, years after the novel’s close.  Seated on the tile floor of the classroom at Raskolnikov’s feet, with a basket of knitting, wearing fingerless gloves, she delivered a powerful six-minute monologue that highlighted her remembered relationship with Raskolnikov, her faith in God, and how each informed her understanding of love. It is this newly made knowledge she conveyed that I found so compelling.  Her performance was preceded by a young man who assumed the character of Alyona, the pawnborker who visits Raskolnikov from her place in hell, who is more insulted by his disrespect of the money he stole, than being murdered by him...
Or considered this synopsis of class work: 

In another humanities' course team taught by Ms. Carmen and Ms. Joseph, I watch as eleventh grade students present ten-minute presentations intended to persuade their peers to vote for their global citizenship project.  The students had determined five potential global projects and students working in teams created film-based multimedia projects. Students researched global problems that they believed they could adequately spend the year helping to solve, determined a project, created a film that would not only explain the project, but also potentially persuade an audience to adopt their project, and then embedded that film into a ten-minute presentation and presented in front of their peers, teachers, principal, assistant principals, and myself (Director of Curriculum).
III. Teacher Evaluation

In New Jersey, Governor Christie and Acting Department of Education Commissioner Chris Cerf want to base at least 50% of teacher evaluation with how well students do on tests, such as the HSPA. NJDOE reported:
Governor Christie today proposed and sent to the legislature a package of bills that gets at the root of the problems in New Jersey’s public education system by reforming the tenure system to demand results for New Jersey’s children in the classroom and reward the best and brightest teachers
The Governor believes that basing at least 50% of a teacher's evaluation on his/her students' standardized test performance, such as the HSPA is an apt indicator of successful teaching.  Keep in mind that currently NJ does not have any state-issued assessments that can be used to evaluate any teachers who teach K-2, teachers who do not teach mathematics or language arts in grades 3 - 8, or science in grades 4 and 8. At the high school level the state only has assessments that could be used to evaluate biology, algebra, and 11th grade mathematics and English teachers.

No one else.

Already there are gross examples of abusive and frankly idiotic tests being used to measure teacher effectiveness. One example comes from Colorado where 6-year-olds were subjected to an art test. 

On exam day in Sabina Trombetta's Colorado Springs first-grade art class, the 6-year-olds were shown a slide of Picasso's "Weeping Woman," a 1937 cubist portrait of the artist's lover, Dora Maar, with tears streaming down her face. It is painted in vibrant -- almost neon -- greens, bluish purples, and yellows. Explaining the painting, Picasso once said, "Women are suffering machines."
The test asked the first-graders to look at "Weeping Woman" and "write three colors Picasso used to show feeling or emotion." (Acceptable answers: blue, green, purple, and yellow.) Another question asked, "In each box below, draw three different shapes that Picasso used to show feeling or emotion." (Acceptable drawings: triangles, ovals, and rectangles.) A separate section of the exam asked students to write a full paragraph about a Matisse painting.
In a 29-year-career as an educator and as a working artist, I cannot think of anyone I have met who would subject children to such a test as a means to measure students' understanding of a state's fine arts standards.

IV. Effectiveness

To be sure, understanding teacher and administrator effectiveness is complex and needs to be determined locally.  It is a mistake to think student performance on a single measure is a reliable indicator of a teacher's capacity to teach. Further, we have additional concerns when the test content is so disconnected from the century we live in. Using the results from that same out of date state test to measure teachers like Dr. Honoria, Ms. Carmen and Ms. Joseph  who are engaging students in aesthetic and complex learning is foolish at best.


  1. Mary Ann. Thank you for so clearly explaining what is wrong with standardized testing as a measurement of learning and teacher effectiveness. I have had to deal with a very similar persuasive writing question on a test and struggled with the ethics of telling students to make up statistics and facts in order to jump through the hoop.

  2. You are very welcome, Heidi. Academic dishonesty is something teachers, especially English, work hard to help students understand and the test asks them to commit it. Go figure.

  3. @CRS, thanks. I work with very fine teachers and am married to one. They inspire this.

  4. What a pleasure to read such a thought provoking post on what is often cast as such a black and white issue. So often we claim that testing and teaching ti the test is not the best way forward for education, but rarely explain why. That is exactly what you have done here.

    I think it is the inauthenticity of these tasks that makes them so dangerous. I love the questions you raised. Thanks.

  5. @Jabiz. Thanks so much for your comments. I agree that a danger lies in a very narrow understanding of "literacy" not literacies. I don't think we could ever adequately measure (especially for a graduation requirement)students' capacity outside of projects that develop over time and include NCTE's 21st century standards, not common core.

  6. What struck me as incredibly artificial was the first line of the question. Perhaps it's a cultural difference between Australia and the US, but it would be extremely difficult to find 11th graders in Australia who were familiar with what a lawsuit actually is and the ramifications and involvements of all the parties concerned.

    Sure - it's passed into the popular vernacular, but is that what the question is asking? Is teaching a good understanding of the legal process and lawsuits for redress now part of the American curriculum in high school? If not, how do the students come up with an adequate response without the ability to research. And in 5 paragraphs??

    Perhaps the deeper question is: "Who tests the writers of these tests?"

    - Ian

  7. Ian, I thought similarly. Even though there are TV shows that include law, watching that and understanding lawsuits is quite different and obviously require different skill sets and prior knowledge. Further, there is no assurance that the evaluators (for hire people at an hourly wage) know enough to be able to read the texts well. The entire thing is farcical.

    The process of placing students in positions where they must make up facts, statistics and reasons in an effort to "persuade" two anonymous evaluators is wrong. Often English teachers need to discuss the value and importance of citing sources when writing and for state testing, they need to set that aside and direct students to do so too.