Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis - Mohd Noh, Muhamad Firdaus; Mohd Matore, Mohd Effendi Ewan (Frontiers in Psychology, 2022)
Evaluating candidates’ answers in speaking skill is difficult and rarely explored. This task is challenging and able to bring inconsistency in the rating quality among raters especially in speaking assessment. Severe raters will bring more harm than good to the results that candidates receive. Many Faceted Rasch Measurement (MFRM) was used to explore the differences of teachers’ rating severity based on their rating experience, training experience and teaching experience. The research uses a quantitative approach and a survey method to enlist 164 English teachers who teach lower secondary school pupils, who were chosen through a multistage clustered sampling procedure. All the facets involving teachers, candidates, items and domains were calibrated using MFRM. Every teacher scored six candidates' responses in a speaking test consisting of three question items, and they were evaluated across three domains namely vocabulary, grammar and communicative. Results highlight that the rating quality was different in terms of teachers’ rating experience and teaching experience. However, training experience did not bring any difference to teachers’ rating quality on speaking test. The evidence from this study suggests that the two main factors of teaching and rating experience must be considered when appointing raters for the speaking test. The quality of training must be improved in order to produce a rater with good professional judgement. Raters need to be supplied with answer samples with varied level of candidates’ performance for them to practice before becoming a good rater. Further research might explore any other rater bias that may impact the psychological well-being for certain groups of students.