Exploring the Validity of Comparative Judgement: Do Judges Attend to Construct-Irrelevant Features? - Chambers, Lucy; Cunningham, Euan (Frontiers in Education, 2022)
When completing a comparative judgement (CJ) exercise, judges are asked to make holistic decisions about the quality of the work they are comparing. A key consideration is the validity of the expert judgements. This paper will detail a study where an aspect of validity, whether or not judges are attending to construct-irrelevant features, was investigated. There are a number of potentially construct-irrelevant features indicated in the assessment literature, we focused on four: appearance, handwriting, Spelling Punctuation and Grammar (SPaG), and missing response vs. incorrect answer. This study explored this through an empirical experiment supplemented by judge observation and survey. The study was conducted within an awarding organisation. The particular context was within a programme of work trialling a new method of maintaining exam standards involving the comparative judgement of candidates’ examination responses from the same subject from two different years. The judgements in this context are cognitively demanding and there is a possibility that judges may attend to superficial features of the responses they are comparing. It is therefore important to understand how the CJ decisions are made and what they are or are not based on, so that we can have confidence in the judgements and know that any uses of them are valid.