MeaningBERT: assessing meaning preservation between sentences - Beauchemin, David; Saggion, Horacio; Khoury, Richard (Frontiers in Artificial Intelligence, 2023)
In the field of automatic text simplification, assessing whether or not the meaning of the original text has been preserved during simplification is of paramount importance. Metrics relying on n-gram overlap assessment may struggle to deal with simplifications which replace complex phrases with their simpler paraphrases. Current evaluation metrics for meaning preservation based on large language models (LLM) such as BertScore in machine translation or QuestEval in summarization have been proposed. However, none has a strong correlation with human judgement of meaning preservation. Moreover, such metrics have not been assessed in the context of text simplification research. In this paper, we present a meta-evaluation of several metrics we apply to measure content similarity in text simplification. We also show that the metrics are unable to pass two trivial, inexpensive content preservation tests. Another contribution of this work is MeaningBERT 1 , a new trainable metric designed to assess meaning preservation between two sentences in text simplification, showing how it correlates with human judgment.To demonstrate its quality and versatility, we will also present a compilation of datasets used to assess meaning preservation and benchmark our work against a large selection of popular metrics in the literature applied to these datasets.