30 SOUTHERN BRANCH.
tion for a systematic acquisition of vocabulary. The grammar and composition work is equivalent to that comprised in part f of Thomas's Practical German Grammar.. A number of easy texts are read and short poems memorized....
iii
Table of Contents
Chapter 1: Introduction ........................................................................................................1
Background, Significance, Purpose Statement, and Study Setting...
1
AN ANALYSIS OF AUTOMATIVE ESSAY SCORING PROGRAMS AND THEIR
POTENTIAL IMPACT ON DIRECT WRITING ASSESSMENT SCORES IN UTAH
Chapter 1
Introduction - Nature of the Problem
Every English teacher knows the pain of issuing a writing assignment. For...
2
Opponents say it “diminishes the role of teachers and warps students’ notions of good
writing” (Grimes, & Warschauer, 2010). The Conference on College Composition and
Communication stated:
Writing-to-a-machine violates the essentially...
3
will also investigate the amount of revisions students perform using the programs, and whether
or not revising with these programs translates into better writing scores on the DWA.
The research will be conducted on schools in a single district....
5
Literature Review
Teaching writing has always been a challenge in education. It is an immensely complex
task the human brain takes on; invariably it is on the upper levels of Bloom’s Taxonomy. The
complexity of the process is reflected in the...
6
to take some of the grading burden, teachers could more easily “assign more writing, and so
students could get the practice they needed to develop as writers—practice that was not possible
in most classrooms because of the burden it placed on...
7
writing must be significant. It must be significant enough to continue looking for other answers,
or continue down the very troubling path of not assigning a sufficient amount of writing for
student, thereby perpetuating the cycle of producing...
8
produce a poorly written paper and still achieve high test scores if they use qualities the AES
system has (sometimes bewilderingly) identified as good writing.
In essence, to write a bad essay and get a good score, the student would have...
9
critical thinking skills and enhance their awareness of the differences between
human and machine readers. Therefore, we are not concerned about spoofing,
provided students do not use a ‘cheat sheet’ and a spoofing vigilant reader scans
the...
10
writing at this institution is not valued as human communication—and this
in turn reduces the validity of the assessment (CCCC Executive Committee,
2004).
They also have concerns with companies not communicating their algorithms with their...
11
One of those that focus on primarily on scoring is Project Essay Grader (PEG). This was
the program developed by Ellis Page in the 60’s. PEG uses sample essays to generate its basis
for what makes up a “good essay” given the topic. The...
12
as students write on various topics. This issue is not limited to PEG; it is found throughout the
AES systems. A question for future studies would be: Are these variations in algorithms
detectable by writers?
As mentioned previously, a strength...
13
Burstein, 2006). Interestingly, despite the high correlation of AES to human scores, and the
newer software, in 2006 the AWA switched to using IntelliMetric, a program developed by
another company (Dikli, 2006; Grimes, Warschauer,...
14
(Rudner). It also reports a high correlation between IEA and human scored essays (Grimes,
Warschauer, 2008; Valenti, 2003) IEA asserts that, “Over many diverse topics, the IEA scores
agreed with human experts as accurately as expert scores...
15
Experiments
There has been a multiplicity of research done with AES systems. Most of the research
in the past focused on how accurately AES systems can grade an essay when compared with
humans. With a few exceptions, the research seems to...
16
In another study, IntelliMetric’s accuracy when compared with human scores was
debated. When the study, performed in Texas, indicated that the IntelliMetric scores did not
correlate with human scores, the researchers developed the hypothesis...
17
similarly was sentence structure. In the framework of this study, this is not surprising; the hard
rules of grammar are the same, regardless of culture or other human factors.
AES System Perceptions
In a study asking whether MyAccess! users...
18
teachers concerning how MyAccess! scores. The same report notes that in response to the
statement, “MyAccess! gives fair and accurate scores” student responses averaged 3.5 on a scale
of 5. Teacher responses were 2.8 (2010).
In another...