Assessing the assessors: GenAI and human ratings of second language writing using analytic and holistic rubrics

Information from the abstract

With the evolution of generative artificial intelligence (GenAI) technology, there has been a surge in research exploring its potential for automated essay scoring (AES). However, few studies have investigated the AES capabilities of multiple GenAI models within language classrooms nor have studies compared GenAI performances when scoring different types of rating scales. Therefore, this study aims to compare the analytic and holistic ratings of GenAI models (e.g., ChatGPT, Claude, Gemini, Copilot, DeepSeek) and nine experienced teachers on 104 students’ opinion paragraphs in a Thai EFL classroom context. GenAI models scored each paragraph following structured, rubric-aligned prompting. Data were analysed using Spearman’s rank-order correlation and many-facet Rasch measurement. Findings revealed that overall, experienced teachers demonstrated relatively less extreme severity estimates, better-fitting ratings, and broadly satisfactory score differentiation across rubrics. GenAI models showed better alignment with human ratings for holistic scoring and language than for task completion and content, and organisation. While approaching human rating behaviours, GenAI models’ rating performances varied across scoring rounds, rubrics, and traits. The findings support supervised application of GenAI models as a supplementary scoring assistant in classroom assessment, with teacher oversight remaining essential. This study offers implications for validating and integrating GenAI-assisted AES into L2 writing classrooms.

Why this record is monitored

This record has an Impact Signal of 79/100 based on recency, source, collaboration, and bibliographic signals. It prioritizes monitoring and is not a judgment of research quality.

Related topics: EFL/ESL Teaching and Learning · Student Assessment and Feedback · Second Language Acquisition and Learning