Let's Not Confuse Assessing Writing A Response to a Prompt with Assessing Meaningful Learning: Are NJ Taxpayers Being Duped?

In New Jersey, every public school student from grades 3 through 8 is assessed using the NJ-ASK in reading, writing, and mathematics.  The public is told to believe that these state measures adequately  represent and correctly measure important student LEARNING.  In this post, I want to test those assumptions by sharing a seventh grade student's returned essay written in response to a "speculative" prompt on the grade 7 NJ-ASK test. The response was selected by the New Jersey Department of Education (NJDOE) as an exemplar of a failing score (3). The NJDOE publishes a collection of student writing that they believe exemplify writing across a score of performance form inadequate command of writing to superior command of writing (scale of 1 to 6) using the RHSM rubric. A score of 3 means that the student did not demonstrate command of written language. A 3 means the student showed only partial command.

This is the prompt the student responded to.  S/he had 25 minutes to read the prompt and write a response during the test.

Writing Task:

Imagine you have the chance to meet a famous person in history. You will get to spend the entire day with this person. Whom would you choose to meet, and how would you spend your day?

Write a story about meeting a famous person in history and how you spent the day.

This is the student's written response as written.

If I had the chance to meet one famous person in history, it would be Walt Disney. Walt Disney created Disneyland and Disney World, huge theme parks that tons of people have been to. Many people go back to visit the parks again! These parks make kids and adults happy. He created all the characters…that’s amazing. That would be why I would want to meet him.

            How would I spend my day with Walt Disney? That is a good question. First I would go to breakfast with him and ask him many questions about how he created Disney Land/World, and about how he created the characters. Then off we would go to Disneyland and sneak a few rides in! After walking around a while we would take a break and have lunch. I would then take the time to ask him more personal questions, about himself. Like, when did you start making the characters, or, when did you first start drawing?

             Then we would walk around a bit and let our food digest. After that…more rides! Also I would discover how they work everything and how long it took to make everything.

 By that time dinner rolls around. We go eat at one of the restaurants there and he tells me about how he made everything and what does it feel like to be making families happy and having a huge business.

 Finally, we would check out any shows and he would show me the room where everything gets created. After that we say our farewells and I head home.

             That would be who I would choose as my famous person, and what we would do in one day.

 

I want to make a few comments about what the student wrote, the nature of the assessment, how the evaluators confuse issues of prior knowledge with composition skill, and finally ask if this type of assessment actually measures important learning.

What the Student Wrote
The student who penned this response in 25 minutes has command of English language.  Not only does s/he demonstrate command, but the writing has moments of sophistication as evidenced by syntax, phrasing, compositional risks, and humor. If I taught this student, I have no doubt I would consider him/her very able. To have partial command, the following characteristics would be present in the student's writing (this is from the states scoring rubric) :

• may lack opening and /or closing
• would have some lapses or flaws in organization
• may lack some transitions between ideas
• would have repetitious details
• would have several unelaborated details
• would have errors/patterns of usage errors
• would have little variety in syntax
• wold have some sentence errors
• would have patterns of mechanical errors 

This response has an opening and closing, is organized sequentially, has transitions that allow the reader to move from one moment in the day to the next, has significant syntactical sophistication, and few mechanical errors.  There is a "vagueness" about the nature of work done by Disney. This student (not his/her writing) received a partially proficient score on this writing task on the NJ ASK. Although educators are told not to place students into remedial courses based on the NJ ASK, it is routinely done.  It is possible that this student would be placed in a remedial course in order for his writing to be "fixed."

What Does this Assessment Measure?
This assessment measures the degree to which a student can render a multi-paragraph response to a specific prompt using pen and paper and no other tools. The student is not required to develop a topic, frame a topic, or use any resources to assist him or her in elaborating upon details.  Although the students are directed to revise and edit, the time frame provided (25 minutes) renders this process null.  It is hard to revisit a work within the same time frame it is first created.  Therefore, the presence of first drafts as finished products is more usual than not in NJDOE testing. I have written about ten books and at least a dozen juried articles and none have been written without research. I routinely check information when I blog. I always have a search engine open in another window.  This is not unusual. And yet, NJDOE purposefully requires students to not make use of any tools other than a pen and paper in 2011.  If the tests didn't hurt kids and cost taxpayers megabucks this might be an amusing anecdote. But, it is not.

Was the Assessment Measured Correctly? What Role Did Prior Knowledge Play in the Writing?

So, if the concerns are not compositional, why didn't the student write with more details? As I indicated, one might consider there to be vagueness with the text. But this "problem" is not a compositional issue, but rather suggests an absence on the part of the student to know usable information about Disney's actual work. As such, the vagueness only surfaces when the student references the work done by Disney. Had the student access to information about Disney, the absence of details would likely go away.  Do we really want to measure how much information a student has about an historical figure in his/her head and is that information any indicator of compositional power? Stating that the writer only has partial command of English language is wrong.

Students' written work on state assessments is evaluated by scoring companies who spend about 90 seconds scoring a work. These results are then tabulated and aggregated and the pass/fail scores make front page news in our local and state paper, declaring the winners and the losers. Judgments based on these limited assessments and faulty scoring become truths we use to justify the goodness and poorness of a school, a teacher and a principal. The issues concerning the scoring process has been written about.  Todd Farley's account of scoring state assessments is an eye opener. His book, Making the Grade: My Misadventures in the Standardized Testing Industry describes his career in the test scoring industry. He raises serious questions about the validity of the test scoring process and the scorers.

What We Need Instead
 
Learning is situated, not an autonomous activity. NJDOE, like other states, privileges a belief that information (like Walt Disney's work habits) is stored in the individual's head and the assessment's purpose is to make visible that knowledge via a test.  The more required "information" a learner can pull from his/her head and represent using the tools provided (paper and pen) the more proficient the learner is said to be.

In contrast this century requires a more informed perspective about learning and assessing. As James Gee explains, a sociocultural perspective on learning:

looks at knowledge and learning in terms of a relationship between an individual with both a mind and a body and an environment in which the individual thinks, feels, acts, and interacts.

An apt assessment for this century does not posit the learner being only a "head" and the assessment a means to extract specified information from that head. A more appropriate assessment would be:

1. authentic (a text [written, multimedia, etc.] the learner or learners had actually developed, revised, edited and published for real purposes), 
2. of this century (the learner(s)made use of a variety of tools as necessary including other human beings and composed text(s) using full array of materials, methods and technologies), 
3. privileged audience by including the potential for reader responses and further responses from the author(s), 
4. situated the learner(s) as author(s) and co-assessor(s) whose performance would be informed by their environment.

This quality of "assessment" would potentially be worthy of our time and money. Assessing learners is important work and requires far more powerful and authentic measures than what NJ currently mandates.