GRE e-rater

It turns out that the ETS has a whole research division, including researchers in natural language processing who come up with stuff like e-rater and other machine essay graders, and that they publish about these systems.

According to the latest system description paper:

The feature set used with e-rater V.2 include measures of grammar, usage, mechanics, style, organization, development, lexical complexity, and prompt-specific vocabulary usage.

E-rater is part of Criterion, a web-based service that provides students with instant scoring and feedback on their submitted essays. Criterion has a number of writing analysis tools whose output form the feature vector used by e-rater. The score is a simple weighted average of the feature values.

One noteworthy detail is that in determining the parameters to use for this model, e-rater ecshews exclusively statistical machine learning (optimization) approaches in favor of allowing judgmental control, for reasons of control (to avoid unintentional skew and other undesirable statistical effects) and transparency (to make the system easier to understand and explain).

It would be interesting to see how straightforward it is to game e-rater, given the above information and access to the implementation in Criterion.

Follow me on Twitter for stuff far more interesting than what I blog.

  • Hey it is really a very nice and interesting news for me to go at least an average at home and than have to go for more practise and think for more good score. Thanks.

  • Hey it is really a very nice and interesting news for me to go at least an average at home and than have to go for more practise and think for more good score. Thanks.

  • AWfail3

    agree totally and that what are using for the computer based AW section !
    one person + one e-rater

    if (scores are near)
    {done;}
    else
    {call another person;}

  • I would imagine after a while students will figure out how to game the e rater. This is exactly what happens to the CAT based test where students has figured out how to crack the system, and ETS is rolling out a new version of GRE in August 2011. Students who want to go to graduate school in the next 2-3 years should probably take the GRE as soon as possible because the new one is supposed to be harder

  • Facebook User

    If there are any students who cannot afford a GRE prep
    course, we have a FREE GRE video course (with over 500 videos) that covers
    everything you need to know on test day: http://www.greenlighttestprep.com/