An Item Response Theory Approach to Enhance Peer Assessment Effectiveness in Massive Open Online Courses

Introduction

E-learning technologies have been evolving and expanding at high rates, so Massive Open Online Courses (MOOCs) and Open Educational Resources (OERs) are being rapidly integrated into educational processes by organizations and institutions around the world (West-Pavlov, 2018). The Internet, and in general, digital access to information, has been recognized as the main tool supporting development (Gillwald et al., 2019): that is why several efforts are made, also by international agencies, to promote the availability of network connections in developing countries (Siles, 2020). Suppose the availability and spread of numerous education/training opportunities could help surmount the abovementioned barrier. In that case, Technology Enhanced Learning (TEL), and Networked Learning in particular, reveal themselves as significant for development beyond their intrinsic educational and pedagogical advantages. Networked education is gaining further confirmation in the current turn of time, while an infectious disease has been spreading and keeps reappearing, forcing long-lasting modifications in the protocols of the worldwide educational systems. Students, and people in general, have to minimize contacts (in presence), and of course, teaching and learning activities have to go on. MOOCs and Open Education are effective and flexible resources to educate, train, and empower populations previously denied actual access to education or online education. MOOCs have been developing for several years to provide learning content for a huge worldwide audience (de Freitas et al., 2015). However, it is still quite frequent that one can enroll in a MOOC and attend the lectures free of charge. On the other hand, easy and inexpensive enrollment is one of the factors significantly producing MOOC dropout rates, together with difficulties in maintaining motivation and engagement.

With respect to motivation and engagement, MOOCs have the same problems as other typologies of distance learning, just made more severe by the extended number of learners. A strong didactic strategy involving the extensive use of assessment, particularly formative assessment (Bloom et al., 1971), can be (part of) a solution to such problems. In contrast, the assessment option for MOOCs is known to be limited (Admiraal et al., 2015).

In particular, peer assessment (PA) is available as a powerful strategy to support educational activities and foster learners’ success (also when a huge number of learners is implied) (Alcarria et al., 2018). By PA, learners can be exposed to different cognitive experiences: on the one hand, they are requested to perform a task (e.g., answering an open-ended question); on the other hand, they are requested to assess other learners’ works, being then involved in cognitive activities of a higher level than just answering (Bloom, 1956).

Moreover, a significant aspect of PA is that it can be delivered at a distance. PA has been already introduced in MOOCs as a learning strategy (Sun et al., 2015). However, the reliability of the assessments, and in general, the applicability of this strategy, are still under discussion by scholars (Alcarria et al., 2018). In practice, there is a question of reliability about the final grade computed by the PA framework if it is based exclusively on peers’ marking work. Furthermore, it concerns the reliability of students’ grading ability, which is clearly, if not completely, dependent on peers’ proficiency in the subject matter of the task to grade. So, studying new PA models suitable to be applied to a MOOC and enhancing the MOOC learning setting is a worthwhile research activity. In particular, one of the directions is how to have a PA system able first to manage, in a computationally feasible way, the big amount of data coming from a PA session in a MOOC. Second, to use such data to maintain reliable student models to use them for automated grading of the tasks/artifacts produced by the learners. Having the above in mind and aiming to study the best configuration of a PA system operating in a MOOC, in this paper, we consider the integration in the PA model of the well-established formal methods described in the item response theory (IRT).