Scaling and Disagreements: Bias, Noise, and Ambiguity - Uma, Alexandra; Almanea, Dina; Poesio, Massimo (Frontiers in Artificial Intelligence, 2022)
Crowdsourced data are often rife with disagreement, whether because of genuine item ambiguity, overlapping labels, subjectivity or annotator error. Hence, a variety of methods have been developed for learning from data containing disagreement. One of the observations emerging from this work is that different methods appear to work best depending on characteristics of the6dataset such as the level of noise. In this paper, we investigate the use of an approach developed to estimate noise, temperature scaling, in learning from data containing disagreements. We find that temperature scaling works with data in which the disagreements are the result of label overlap, but not with data in which the disagreements are due to annotator bias, as in, e.g., subjective tasks such as labelling an item as offensive or not. We also find that disagreements due to ambiguity do not fit perfectly in either category.