Rating scale impact on EFL essay marking: A mixed-method study

Loading...
Thumbnail Image

Date

2007

Authors

Barkaoui, Khaled

Journal Title

Journal ISSN

Volume Title

Publisher

Elsevier

Abstract

Educators often have to choose among different types of rating scales to assess second-language (L2) writing performance. There is little research, however, on how different rating scales affect rater performance. This study employed a mixed-method approach to investigate the effects of two different rating scales on EFL essay scores, rating processes, and raters’ perceptions. Four EFL teachers in Tunisia rated a set of 24 EFL essays silently and two subsets of four essays while thinking aloud using a holistic scale and then a multiple-trait rating scale. The essay scores were analyzed using G-theory while the think-aloud protocols were coded in terms of Cuming, Kantor, and Power's (Cumming, A., Kantor, R., & Powers, D. (2002). Decision making while rating ESL/EFL writing tasks: A descriptive framework. Modern Language Journal, 86 (1), 67–96.) rater decision-making scheme. The holistic scale resulted in higher inter-rater agreement. Raters employed similar processes with both rating scales. Raters were the main source of variability in terms of scores and decision-making behavior. These findings have implications for writing assessment practices and for further research.

Description

Keywords

EFL essay rating, Holistic rating, Multiple-trait rating, Think-aloud protocols, Generalizability theory, Mixed-method research, Writing assessment

Citation

Barkaoui, K. (2007). Rating scale impact on EFL essay marking: A mixed-method study. Assessing Writing, 12, 86-107