Article Preview
TopIntroduction
Automated writing evaluation (AWE), also known as automated writing feedback (AWF), is a computerized feedback system that uses natural language processing (NLP) to analyze and provide feedback on student writing regarding grammar, style, and content. These systems aim to enhance writing proficiency by delivering immediate and customized feedback on language errors applicable across various writing tasks. Recently, there has been a significant surge in interest concerning the pedagogical aspects of AWE feedback within English as a foreign language (EFL) writing research. The direction of this research has markedly shifted from considering AWE as an alternative to human evaluators to examining how AWE-generated feedback can enhance the quality of EFL writing (Ranalli et al., 2017; Hibert, 2019).
The potential of AI-powered AWE and automated text scoring (ATS) systems in enhancing writing in English as a second language has been explored in numerous studies (Chen & Cheng, 2008; Grimes & Warschauer, 2010; Alikaniotis et al., 2016, Tang & Wu, 2017; Zhang, 2021). The literature yielded several key areas encompassing various aspects of AWE use in EFL classroom settings. First, despite their effectiveness in improving writing skills and providing quick and measurable feedback, AWE tools exhibit varying levels of effectiveness. User attitudes towards these tools are also mixed, with an appreciation for their convenience and immediacy but criticism for their inability to provide nuanced feedback. In addition, while AWE systems emphasize the role of revision and practice, the constant submission of redrafts does not necessarily lead to improved writing skills. Therefore, the necessity for human intervention and implementation strategies persists, reinforcing that these tools can supplement the writing activity without replacing the role of instructor feedback.
TopBackground
Effectiveness and Limitations of AWE
Research indicates a clear potential for AWE tools to enhance the writing process and provide valuable feedback (Dikli, 2006). These tools, underpinned by theoretical foundations, offer opportunities for deliberate practice and holistic feedback on student writing, with or without instructor support. AWE potential for EFLs is reflected in studies such as Lee (2020), in which students using AWE feedback over an extended period reported improved writing competence. Despite its potential to enhance the writing process, AWE tools exhibit varying levels of effectiveness and have limitations that need to be considered (Hibert, 2019). While competent in detecting lexical errors, AWE feedback may not accurately evaluate university student essays, such as those offered by Pigai (http://en.pigai.org/; an AWE system popular in China that uses NLP to assess and provides feedback on English language essays; Gao, 2021).
The effectiveness of AWE can vary significantly depending on student proficiency levels. Xu and Zhang (2021) found that lower-level students benefited more from Pigai AWE feedback than their more proficient peers. On the contrary, Huang and Renandya (2020) found that integrating AWE did not result in improved drafts for less proficient students, indicating that more targeted approaches may be needed based on the varying needs of different learner profiles. Studies underscore the importance of these tools to provide more targeted, specific feedback that addresses individual student needs (El-Ebyary & Windeatt, 2010; Chapelle et al., 2015; Koltovskaia, 2020). Wang, Chen, and Cheng (2006) noted that while AWE was beneficial in improving certain formal aspects of writing, it fell short in identifying more complex issues of coherence and idea development. Considering their potential advantages and limitations, this highlights a critical balance in the perceived effectiveness of such tools.