<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v2.3 20070202//EN" "journalpublishing.dtd">
<article xml:lang="EN" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="research-article">
<front>
<journal-meta>
<journal-id journal-id-type="publisher-id">Front. Psychol.</journal-id>
<journal-title>Frontiers in Psychology</journal-title>
<abbrev-journal-title abbrev-type="pubmed">Front. Psychol.</abbrev-journal-title>
<issn pub-type="epub">1664-1078</issn>
<publisher>
<publisher-name>Frontiers Media S.A.</publisher-name>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="doi">10.3389/fpsyg.2023.1076379</article-id>
<article-categories>
<subj-group subj-group-type="heading">
<subject>Psychology</subject>
<subj-group>
<subject>Original Research</subject>
</subj-group>
</subj-group>
</article-categories>
<title-group>
<article-title>Is machine translation a dim technology for its users? An eye tracking study</article-title>
</title-group>
<contrib-group>
<contrib contrib-type="author" corresp="yes">
<name><surname>Kasper&#x00117;</surname> <given-names>Ramun&#x00117;</given-names></name>
<xref ref-type="aff" rid="aff1"><sup>1</sup></xref>
<xref ref-type="corresp" rid="c001"><sup>&#x0002A;</sup></xref>
<uri xlink:href="http://loop.frontiersin.org/people/2062303/overview"/>
</contrib>
<contrib contrib-type="author">
<name><surname>Motiej&#x0016B;nien&#x00117;</surname> <given-names>Jurgita</given-names></name>
<xref ref-type="aff" rid="aff1"><sup>1</sup></xref>
<uri xlink:href="http://loop.frontiersin.org/people/2183346/overview"/>
</contrib>
<contrib contrib-type="author">
<name><surname>Patasien&#x00117;</surname> <given-names>Irena</given-names></name>
<xref ref-type="aff" rid="aff2"><sup>2</sup></xref>
<uri xlink:href="http://loop.frontiersin.org/people/2110598/overview"/>
</contrib>
<contrib contrib-type="author">
<name><surname>Pata&#x00161;ius</surname> <given-names>Martynas</given-names></name>
<xref ref-type="aff" rid="aff2"><sup>2</sup></xref>
<uri xlink:href="http://loop.frontiersin.org/people/2110436/overview"/>
</contrib>
<contrib contrib-type="author">
<name><surname>Horba&#x0010D;auskien&#x00117;</surname> <given-names>Jolita</given-names></name>
<xref ref-type="aff" rid="aff1"><sup>1</sup></xref>
<uri xlink:href="http://loop.frontiersin.org/people/1076633/overview"/>
</contrib>
</contrib-group>
<aff id="aff1"><sup>1</sup><institution>Faculty of Social Sciences, Arts and Humanities, Kaunas University of Technology</institution>, <addr-line>Kaunas</addr-line>, <country>Lithuania</country></aff>
<aff id="aff2"><sup>2</sup><institution>Faculty of Informatics, Kaunas University of Technology</institution>, <addr-line>Kaunas</addr-line>, <country>Lithuania</country></aff>
<author-notes>
<fn fn-type="edited-by"><p>Edited by: Marijan Palmovic, University of Zagreb, Croatia</p></fn>
<fn fn-type="edited-by"><p>Reviewed by: Lieve Macken, Ghent University, Belgium; Laura Kamandulyte Merfeldiene, Vytautas Magnus University, Lithuania</p></fn>
<corresp id="c001">&#x0002A;Correspondence: Ramun&#x00117; Kasper&#x00117; &#x02709; <email>ramune.kaspere&#x00040;ktu.lt</email></corresp>
<fn fn-type="other" id="fn001"><p>This article was submitted to Language Sciences, a section of the journal Frontiers in Psychology</p></fn></author-notes>
<pub-date pub-type="epub">
<day>06</day>
<month>02</month>
<year>2023</year>
</pub-date>
<pub-date pub-type="collection">
<year>2023</year>
</pub-date>
<volume>14</volume>
<elocation-id>1076379</elocation-id>
<history>
<date date-type="received">
<day>21</day>
<month>10</month>
<year>2022</year>
</date>
<date date-type="accepted">
<day>12</day>
<month>01</month>
<year>2023</year>
</date>
</history>
<permissions>
<copyright-statement>Copyright &#x000A9; 2023 Kasper&#x00117;, Motiej&#x0016B;nien&#x00117;, Patasien&#x00117;, Pata&#x00161;ius and Horba&#x0010D;auskien&#x00117;.</copyright-statement>
<copyright-year>2023</copyright-year>
<copyright-holder>Kasper&#x00117;, Motiej&#x0016B;nien&#x00117;, Patasien&#x00117;, Pata&#x00161;ius and Horba&#x0010D;auskien&#x00117;</copyright-holder>
<license xlink:href="http://creativecommons.org/licenses/by/4.0/"><p>This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.</p></license></permissions>
<abstract>
<p>State-of-the-art research shows that the impact of language technologies on public awareness and attitudes toward using machine translation has been changing. As machine translation acceptability is considered to be a multilayered concept, this paper employs criteria of usability, satisfaction and quality as components of acceptability measurement. The study seeks to determine whether there are any differences in the machine-translation acceptability between professional users, i.e., translators and language editors, and non-professional users, i.e., ordinary users of machine translation who use it for non-professional everyday purposes. The main research questions whether non-professional users process raw machine translation output in the same way as professional users and whether there is a difference in the processing of raw machine-translated output between users with different levels of machine-translated text acceptability are analyzed. The results of an eye tracking experiment, measuring fixation time, dwell time and glance count, indicate a difference between professional and non-professional users&#x00027; cognitive processing and acceptability of machine translation output: translators and language editors spend more time overall reading the machine-translated texts, possibly because of their deeper critical awareness as well as professional attitude toward the text. In terms of acceptability overall, professional translators critically assess machine translation on all components of which confirms the findings of previous similar research. However, the study draws attention to non-professional users&#x00027; lower awareness regarding machine translation quality. The study was conducted within a research project that received funding from the Research Council of Lithuania (LMTLT, agreement No S-MOD-21-2), seeking to explore and evaluate the impact on society of machine translation technological solutions.</p></abstract>
<kwd-group>
<kwd>machine translation</kwd>
<kwd>acceptability</kwd>
<kwd>usability</kwd>
<kwd>quality</kwd>
<kwd>satisfaction</kwd>
<kwd>end-users</kwd>
<kwd>professional translators</kwd>
</kwd-group>
<contract-sponsor id="cn001">Lietuvos Mokslo Taryba<named-content content-type="fundref-id">10.13039/501100004504</named-content></contract-sponsor>
<counts>
<fig-count count="14"/>
<table-count count="0"/>
<equation-count count="0"/>
<ref-count count="41"/>
<page-count count="14"/>
<word-count count="8357"/>
</counts>
</article-meta>
</front>
<body>
<sec sec-type="intro" id="s1">
<title>1. Introduction</title>
<p>Neural machine translation is more and more frequently used in the translation and localization market. Following the AI Index Report, artificial intelligence has allowed improving machine translation in certain language pairs almost to human quality (Perrault et al., <xref ref-type="bibr" rid="B27">2019</xref>). According to some scores, &#x0201C;[t]he fastest improvement was for Chinese-to-English, followed by English-to-German and Russian-to-English&#x0201D; (Perrault et al., <xref ref-type="bibr" rid="B27">2019</xref>). However, the performance varies between different language pairs and that depends on language pair popularity, which &#x0201C;defines how much investment goes into data acquisition&#x0201D; (Perrault et al., <xref ref-type="bibr" rid="B27">2019</xref>).</p>
<p>For these reasons, researchers and research administrators have recently been paying attention to the effects that artificial intelligence and developed technologies bring about on the translation industry, translator&#x00027;s profession, career and daily tasks, as well as training and skills needed, but also in terms of the perceptions within society. In this perspective, some important research papers have been published in the past few years where translation scholars have concluded that, for example, artificial intelligence-powered machine translation and other language related technologies have fundamentally changed public awareness and attitudes toward multilingual communication (Vieira et al., <xref ref-type="bibr" rid="B40">2021</xref>). Such technologies are now increasingly being used to overcome language barriers not only in situations of personal use but also in high-risk environments, such as health care systems, courts, police and so on. The availability and impact of machine translation accessibility and impact on society, including the importance of full participation of various social groups in communication processes, are being analyzed and evaluated (Vieira et al., <xref ref-type="bibr" rid="B40">2021</xref>). On the other hand, public awareness of the capabilities as well as the quality of machine translation is identified as insufficient (Kasper&#x00117; and Motiej&#x0016B;nien&#x00117;, <xref ref-type="bibr" rid="B19">2021</xref>).</p>
<p>Although machine translation is breaking down language barriers, and its accuracy and efficiency are getting closer to human-level translation, human effort is needed to reduce the negative impact of machine translation in society (Hoi, <xref ref-type="bibr" rid="B14">2020</xref>). The communication processes supported by machine translation can be of high quality if the process participants are aware of the quality shortcomings (Yasuoka and Bjorn, <xref ref-type="bibr" rid="B41">2011</xref>). Studies have also found that machine translation can help reduce the exclusion of ethnic minorities in a wide variety of fields (Taylor et al., <xref ref-type="bibr" rid="B36">2015</xref>).</p>
<p>There is a plethora of research on the quality of machine translation and use of post editing (see Ueffing, <xref ref-type="bibr" rid="B37">2018</xref>; Ortega et al., <xref ref-type="bibr" rid="B26">2019</xref>; Vardaro et al., <xref ref-type="bibr" rid="B38">2019</xref>; Nurminen and Koponen, <xref ref-type="bibr" rid="B25">2020</xref>; Rossi and Carr&#x000E9;, <xref ref-type="bibr" rid="B31">2022</xref>, to mention but a few). The benefits of machine translation post editing in different language pairs have been acknowledged in multiple studies employing a diversity of research designs (see Carl et al., <xref ref-type="bibr" rid="B1">2011</xref>, <xref ref-type="bibr" rid="B2">2015</xref>; Moorkens, <xref ref-type="bibr" rid="B22">2018</xref>; Stasimioti and Sosoni, <xref ref-type="bibr" rid="B34">2021</xref>). Studies have also addressed the issue of machine translation acceptability (see Castilho, <xref ref-type="bibr" rid="B3">2016</xref>; Castilho and O&#x00027;Brien, <xref ref-type="bibr" rid="B6">2018</xref>; Rivera-Trigueros, <xref ref-type="bibr" rid="B29">2021</xref>; Taivalkoski-Shilov et al., <xref ref-type="bibr" rid="B35">2022</xref>). However, the attitudes and perceptions of translation students, novice translators, professional translators and posteditors have been mainly taken into the focus, possibly due to a somewhat easier access to respondents and more convenient research design (see Moorkens and O&#x00027;Brien, <xref ref-type="bibr" rid="B23">2015</xref>; Rossi and Chevrot, <xref ref-type="bibr" rid="B32">2019</xref>; Ferreira et al., <xref ref-type="bibr" rid="B11">2021</xref>). The acceptability of machine-translated content by non-professional users has not been extensively studied. The ordinary users&#x00027; perspective is important because of the variety of purposes for which they take machine translation for granted and use it daily (Kasper&#x00117; and Motiej&#x0016B;nien&#x00117;, <xref ref-type="bibr" rid="B19">2021</xref>).</p>
<p>The study<xref ref-type="fn" rid="fn0001"><sup>1</sup></xref> seeks to investigate the acceptability of raw machine translation texts in Lithuanian, a low-resource language. In this paper, we report the results of an eye tracking experiment with professional translators and non-professional users of machine translation with the focus on acceptability. The inter-group and intra-group comparisons of raw machine-translated text acceptability are made. The research questions are as follows: do non-professional users process raw machine translation output in the same way as professional users? Is there a difference in the processing of raw machine-translated output among non-professional users with different levels of acceptability of machine-translated text? Is there a difference between professional and non-professional users&#x00027; comprehension of the raw machine-translated output?</p></sec>
<sec id="s2">
<title>2. Literature overview</title>
<p>Machine translation acceptability is a multilayered concept. Criteria of usability, satisfaction and quality have been indicated to be the components of acceptability. Castilho and O&#x00027;Brien (<xref ref-type="bibr" rid="B6">2018</xref>) define acceptability as machine translation output quality in terms of correctness, cohesion and coherence from the reader&#x00027;s perspective. Even if the text contains errors, it does not mean that it is considered unacceptable. If the needs of the readers are satisfied, the text has served its mission (Castilho and O&#x00027;Brien, <xref ref-type="bibr" rid="B4">2016</xref>, <xref ref-type="bibr" rid="B6">2018</xref>). In order to measure acceptability, Castilho (<xref ref-type="bibr" rid="B3">2016</xref>) defines the three criteria. Usability is related to efficiency and effectiveness of the text and may be measured by exerted cognitive effort; satisfaction, which is understood as a user&#x00027;s positive attitude toward the translated text, may be measured through web surveys, post-task satisfaction questionnaires or moderators&#x00027; ratings; and quality is defined by fluency, adequacy, syntax and grammar, and style in translated content or as text easeability, readability, etc. (Castilho, <xref ref-type="bibr" rid="B3">2016</xref>). For the purposes of this research, acceptability is understood as a notion combining satisfaction, usability and quality as assumed by the ordinary readers of the text who have no linguistic background or related, e.g., translator, training.</p>
<p>Research employing eye tracking methodology is common in Translation Studies (Carl et al., <xref ref-type="bibr" rid="B1">2011</xref>; Castilho, <xref ref-type="bibr" rid="B3">2016</xref>; Daems et al., <xref ref-type="bibr" rid="B8">2017</xref>; Moorkens, <xref ref-type="bibr" rid="B22">2018</xref>; Vardaro et al., <xref ref-type="bibr" rid="B38">2019</xref>; Ferreira et al., <xref ref-type="bibr" rid="B11">2021</xref>; Stasimioti and Sosoni, <xref ref-type="bibr" rid="B34">2021</xref>). Among the existing body of scientific literature on the acceptability criteria of machine translation, of particular mention are those published papers that employ eye tracking experiments. Since acceptability is a vague notion representing quite subjective understanding and judgement, eye tracking studies present relevant insights into the readers&#x00027; cognitive processing of the (machine-translated) text they are reading. The research reveals that the required cognitive load is generally to a greater or lesser extent higher in cases where machine translation is provided in comparison with human-translated or post-edited text.</p>
<p>Jakobsen and Jensen (<xref ref-type="bibr" rid="B16">2008</xref>) report the results of a translation process study, focusing on the differences between the reading of a text with the aim of understanding its meaning and reading the same text (or a very similar text) with the expectation of having to translate it next. The authors recorded eye movements of six translation students and six professional translators who were asked to perform four tasks at the speed at which they normally work, namely read a text for comprehension, read a text in preparation for translating it later on, read a text while performing its oral translation and read a text while typing a written translation. The researchers compared task duration, total number of fixations, total gaze time and average duration of individual fixations for each task and found out that the purpose of reading had a clear effect on eye movements and gaze duration. Overall, the increases in the number of fixations from the first to the last task of the experiment were statistically significant (Jakobsen and Jensen, <xref ref-type="bibr" rid="B16">2008</xref>).</p>
<p>In a study by Guerberof Arenas et al. (<xref ref-type="bibr" rid="B13">2021</xref>), researching the effect of different translation modalities on users through an eye tracking experiment, 79 end users&#x00027; (Japanese, German, Spanish, English) experiences with published translated, machine-translated and published English versions were compared. The authors focused on the number of successful tasks performed by end users, the time necessary for performing successful tasks in different translation modalities, the satisfaction level of end users in relation to different translation modalities and the amount of cognitive effort necessary for carrying out tasks in different translation modalities (Guerberof Arenas et al., <xref ref-type="bibr" rid="B13">2021</xref>). They measured usability, i.e., effectiveness (by asking participants to perform some tasks), efficiency (by measuring the time to complete the tasks and by measuring cognitive effort using an eye tracker) and satisfaction. The authors came to the conclusion that the effectiveness variable was not found to be significantly different when the subjects read the published translated version, a machine-translated version and the published English version of the text although efficiency and satisfaction were significantly different, especially for less experienced participants. The results of the eye tracking experiment revealed that end users&#x00027; cognitive load was higher for machine-translated and human translated versions than for the English original. The findings also indicated that the language and the translation modality played a significant role in the usability, regardless of whether end users finished the given tasks and even if they were unaware that MT was used (Guerberof Arenas et al., <xref ref-type="bibr" rid="B13">2021</xref>).</p>
<p>In a study by Hu et al. (<xref ref-type="bibr" rid="B15">2020</xref>), an eye tracking experiment involving 66 Chinese participants with low proficiency in English who also had to fill in questionnaires on comprehension testing and attitudes showed that the quality of raw machine-translated output was considered somewhat lower, but almost as good as that of a post-edited machine-translated output, although the research design involved non-professional post-editing of machine-translated text.</p>
<p>Some earlier user-centered studies where raw machine translation was analyzed <italic>via</italic> eye tracking, screen recording experiments and post-task questionnaires determined a lower usability of machine-translated instructions in comparison with post-edited output (Castilho et al., <xref ref-type="bibr" rid="B5">2014</xref>; Doherty and O&#x00027;Brien, <xref ref-type="bibr" rid="B10">2014</xref>; Doherty, <xref ref-type="bibr" rid="B9">2016</xref>).</p>
<p>In a study of non-professional users where acceptability of a machine-translated text from English into Lithuanian was tested, an eye tracking experiment revealed that the cognitive processing was greater, i.e., required a longer gaze time and fixation count, on machine translation errors in comparison with correct segments of text (Kasperavi&#x0010D;ien&#x00117; et al., <xref ref-type="bibr" rid="B18">2020</xref>). The machine-translated segments with errors required more attention and cognitive effort from the readers, but the results regarding overall acceptability of the raw machine-translated text obtained <italic>via</italic> a post-task survey did not correlate with the readers&#x00027; gaze time spent on segments with errors.</p>
<p>Literary texts have also received some attention with regard to the differences between human and machine translations from English into Dutch as perceived by end users. Colman et al. (<xref ref-type="bibr" rid="B7">2021</xref>) employed eye tracking to analyze end users&#x00027; reading process and determine the extent to which machine translation impacts the reading process. An increased number of eye fixations and increased gaze duration while reading machine translation segments was found in comparison with human translation (Colman et al., <xref ref-type="bibr" rid="B7">2021</xref>).</p>
<p>Although scarce, there is some research, based on research designs employing methodologies other than eye tracking, determining how the acceptability of machine-translated texts in various languages is perceived by non-professionals or low proficiency future professionals. The broad public uses machine translation for many reasons and purposes and they may not fully understand or consider how machine translation really works and what quality it generates. In a study of 400 surveyed participants, acceptability of the text that had been machine translated from English to Lithuanian was found to be affected by such factors as age and education. The less educated and senior participants were more prone to consider machine translation reliable and satisfactory (Kasper&#x00117; et al., <xref ref-type="bibr" rid="B17">2021</xref>).</p>
<p>In a study by Rossetti et al. (<xref ref-type="bibr" rid="B30">2020</xref>), 61 participants were surveyed in order to get insight into the &#x0201C;impact of machine translation and postediting awareness&#x0201D; on comprehension and trust. The participants were asked to read and evaluate crisis messages in English and Italian using ratings and open-ended questions on comprehensibility and trust. The authors found insignificant differences in the end users&#x00027; comprehension and trust between raw machine-translated and post-edited text (Rossetti et al., <xref ref-type="bibr" rid="B30">2020</xref>). However, users with low proficiency of English were more positive toward raw machine-translated text in terms of its comprehension and trust (Rossetti et al., <xref ref-type="bibr" rid="B30">2020</xref>).</p>
<p>In another study with translation agencies, professional translators and clients/users of professional translation, the level of user awareness of machine translation was studied through surveys (Garc&#x000ED;a, <xref ref-type="bibr" rid="B12">2010</xref>). Acceptability and evaluation of machine translation from Chinese into English was at the focus. The researcher found out that &#x0003C;5% of professional translators considered the quality of machine translation very high. The translation agencies expressed a very similar view on machine translation to that of the translators. The clients/users of professional translations (about 30%) who were aware of and requested machine translation had an intermediate or positive assessment of the quality of machine translation (Garc&#x000ED;a, <xref ref-type="bibr" rid="B12">2010</xref>).</p>
<p>As the amount of content to be translated is growing, there is a demand to cut the cost of translation orders, which leads to a growing need for research and testing how translators work with machine translation (Moorkens and O&#x00027;Brien, <xref ref-type="bibr" rid="B23">2015</xref>) and the newly-arising need to learn how the end users are aware of, perceive, use and accept machine-translated content.</p></sec>
<sec sec-type="materials and methods" id="s3">
<title>3. Materials and methods</title>
<p>Machine translation quality overall can be assessed in various ways: by applying automatic quality estimation metrics, by carrying out an error analysis by professionals/experts, employing cognitive experimental methods with human experts or professionals or semi-experts or non-experts, determining acceptability of the output of non-experts/non-professionals/amateur users, <italic>via</italic> qualitative methods, etc. Recently, cognitive experimental methods for machine translation quality assessment have been increasingly employed, e.g., eye tracking, key logging, screen recording, post-performance (retrospective) interviews, think-aloud protocols, etc. In an eye tracking experiment, fixation count and time, gaze time, saccades, pupil dilation, and other variables can be measured, although researchers have determined that, for example, pupil dilation may not adequately reflect cognitive effort involved or provide valid and reliable data. To test the validity of the data, cognitive translation researchers have employed complementary methods, including other experimental methods, interviews or surveys. Translation research studies employing eye tracking have mostly relied on post-performance or retrospective interviews/surveys, and the number of subjects involved in an eye tracking experiment for translation research varies between 2 and 84 (per language). The most common eye movement measures taken into account and described in translation research are fixation time and fixation count (Kasper&#x00117; and Motiej&#x0016B;nien&#x00117;, <xref ref-type="bibr" rid="B19">2021</xref>).</p>
<sec>
<title>3.1. Experiment</title>
<p>For the current study, we used an eye tracking experiment along with a questionnaire in order to ensure the validity of results obtained. Before the experiment, a larger-scale population survey was conducted to find out the purposes, typical circumstances of machine translation use and systems employed non-professional users (Kasper&#x00117; et al., <xref ref-type="bibr" rid="B17">2021</xref>). In the survey, the respondents were asked to indicate a machine translation tool that they used most often. The reported results of the survey revealed that the absolute majority of the respondents indicated that they used Google Translate as the tool for machine translation (Kasper&#x00117; et al., <xref ref-type="bibr" rid="B17">2021</xref>). We, therefore, also employed it for the machine translation of the text in the research design of this particular study. Google Translate has over 500 million users per month and over 140 billion words are translated per day (Schuster et al., <xref ref-type="bibr" rid="B33">2016</xref>; Hu et al., <xref ref-type="bibr" rid="B15">2020</xref>). The text chosen for a reading task in the experiment was a recipe of a dish. The motivation behind selecting the text of a recipe for this experiment lies in the findings of the above-mentioned study where the respondents indicated various reasons for using machine translation in their everyday activities, one of the most common being household purposes (Kasper&#x00117; et al., <xref ref-type="bibr" rid="B17">2021</xref>). The text of a recipe, originally in English, was machine translated using Google Translate to Lithuanian. The translated excerpt given to the subjects as a reading task contained 371 words and was arranged on three slides 13&#x02013;15 lines each.</p>
<p>In the machine-translated excerpt, we selected areas of interest with errors and areas of interest without errors. According to scientific literature, the perceptual span in western languages is about 13&#x02013;15 characters to the right of the center of vision, and 3&#x02013;4 to the left (McConkie and Rayner, <xref ref-type="bibr" rid="B21">1975</xref>; Rayner, <xref ref-type="bibr" rid="B28">1998</xref>). Therefore, all our selected areas of interest (both with and without errors) included 18&#x02013;20 characters. In the raw translated text prepared for the experiment, 12 distinct errors were selected as areas of interest. Another 12 areas of interest without errors were selected as control. To identify the errors, we used the Multidimensional Quality Metrics, which is a typology of errors developed for assessment of the quality of human translated, machine translated and post-edited texts. This system covers more than 100 error types and can be adapted to all languages (Lommel et al., <xref ref-type="bibr" rid="B20">2014</xref>). Within this classification, the following main types of errors are as indicated: terminology; accuracy (for example, addition, mistranslation, omission, untranslated text, etc.); linguistic conventions (also called fluency in the previous versions of the taxonomy, related to errors in grammar, punctuation, spelling, unintelligible text, etc.), design and markup (errors related to visual presentation of a translated text, such as text formatting, layout); locale conventions (errors related to locale-specific content); style (errors related to inappropriate organizational or language style); and audience appropriateness (for example, errors related to culture-specific reference) (MQM Commitee, <xref ref-type="bibr" rid="B24">2022</xref>). The 12 identified errors fell into two 2 different categories of errors, namely accuracy and linguistic conventions. Accuracy errors were those of mistranslation, untranslated text, omission, and addition. Errors that fell within the linguistic conventions category were those of an incorrect word form (ending) resulting in inappropriate agreement between the words in a phrase.</p>
<p>Eye tracking was performed using a commercial non-invasive eye tracking device SensoMotoric Instruments GmbH Scientific RED-B.6-1524-6150133939 and SMI BeGaze 3.7.42 software for data analysis. For each area of interest (AOI), several eye movement measures were taken into consideration: fixation time (total time of fixations that happened in the AOI), dwell time (total time of fixations and saccades that happened in the AOI), and glance count (the number of times when the gaze entered the AOI).</p>
</sec>
<sec>
<title>3.2. Research participants</title>
<p>In total, there were 30 subjects in the experiment: 11 professional translators, language editors and revisers and 19 non-professional users of machine translation, who were of different educational backgrounds, age, occupation. All subjects were native speakers of Lithuanian. Among the non-professional users, 13 had a university degree and 6 had secondary education. The subjects gave consent to participate in the experiment on a voluntary basis. They were informed that the text they were reading was a machine translation. The subjects were also told that they would have to answer questions about the text afterwards filling in a post-task questionnaire. There were 4 reading comprehension questions, all related to the errors in the text, including 2 true/false questions and 2 open questions. The post-task questionnaire also had 9 statements, 3 per each component of acceptability (i.e., quality, usability and satisfaction). The statements could be assessed by the subjects on a 5-point Likert scale, where 1-completely disagree, 2-somewhat disagree, 3-neither agree nor disagree, 4-somewhat agree, and 5-completely agree. In total, in this part of the questionnaire, the subjects of the experiment could accumulate a maximum of 45 points: 15 for quality, 15 for usability and 15 for satisfaction. The questions and the statements provided to the subjects in a post-task questionnaire were presented in their native, i.e., Lithuanian, language.</p>
</sec>
<sec>
<title>3.3. Data analysis</title>
<p>IBM SPSS Statistics 27 was used for descriptive and relationship analysis. Descriptive statistics were calculated for quantitative nominal and ordinal data. The relationships between data were investigated using column plots and box plots. Although the convenience sample was used, limiting the usefulness of hypothesis testing, several non-parametric tests (one-sample Kolmogorov-Smirnov test, independent-samples Mann-Whitney <italic>U</italic>-test, independent-samples Moses test of extreme reaction) with a significance level of 0.05 were used to explore what hypotheses would be more promising for further research.</p></sec></sec>
<sec sec-type="results" id="s4">
<title>4. Results</title>
<p>The findings of our study demonstrate that the average fixation time on the areas of interest with errors of both groups of the subjects was longer than on the areas of interest without errors, which confirms findings of other studies that errors attract more readers&#x00027; attention and require more cognitive effort than correct text (see <xref ref-type="fig" rid="F1">Figure 1</xref>). The average fixation time on the areas of interest with errors (in percentage from total time of the trial) was 12.6 vs. 11.7% for professional and non-professional users of machine translation, respectively. On the other hand, professionals also demonstrated a longer average fixation time on areas of interest without errors than non-professionals, i.e., 11.8 vs. 10.4%. The longer average fixation time on both types of areas of interest within the professionals&#x00027; cohort might be interpreted that professional translators and language specialists who work with texts on a daily basis have different skills and a more pronounced critical look at any text. Such a hypothesis would still have to be tested on a broader scale experiment.</p>
<fig id="F1" position="float">
<label>Figure 1</label>
<caption><p>Average percentage of fixation time on all areas of interest with errors and without errors in the groups of professional and non-professional users of machine translation.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fpsyg-14-1076379-g0001.tif"/>
</fig>
<p>Independent-samples Mann-Whitney <italic>U-</italic>test would indicate that the hypothesis that the fixation time of professionals and non-professionals for AOIs with errors has the same distribution (more precisely, the hypothesis that the probability of fixation time being higher for a random professional than for random non-professional is 0.5) could not be rejected (<italic>p</italic> &#x0003D; 0.792). Still, the independent-samples Moses test of extreme reaction suggests that, while hypothesis about the distributions having the same range could not be rejected, the value of <italic>p</italic> is much closer to the level of significance (<italic>p</italic> &#x0003D; 0.079). Similar (although weaker) relationship holds for AOIs without errors (<italic>p</italic> &#x0003D; 0.670 and <italic>p</italic> &#x0003D; 0.180).</p>
<p>As <xref ref-type="fig" rid="F2">Figure 2</xref> shows, while the median of dwell time for AOIs with errors was very similar for professionals (17,851 ms) and non-professionals (18,722 ms), the spread of it was clearly different. That might be assumed to be rather surprising, for, intuitively, one might suppose that professionals are going to be more like each other than non-professionals.</p>
<fig id="F2" position="float">
<label>Figure 2</label>
<caption><p>A simple boxplot of dwell time on AOIs with errors in the groups of professional and non-professional users of machine translation.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fpsyg-14-1076379-g0002.tif"/>
</fig>
<p>Different eye movement measures, including the fixation time, were also compared in the groups of the subjects who scored high and low in the post-task survey for the questions demonstrating quality and usability of the text and the users&#x00027; satisfaction with the text.</p>
<p>On all components of acceptability (see <xref ref-type="fig" rid="F3">Figure 3</xref> for quality, <xref ref-type="fig" rid="F4">Figure 4</xref> for usability, and <xref ref-type="fig" rid="F5">Figure 5</xref> for satisfaction), non-professional users scored higher than professionals. The average total quality scores were 6.2632 for non-professionals and 5.3636 for professionals (median 6 vs. 5, respectively). The average total usability scores were 6.6842, i.e., slightly better, for non-professionals compared with professionals, i.e., 6.0000 (median 7 vs. 6, respectively). In terms of the average total satisfaction scores, the non-professionals&#x00027; scores were much more increased compared with professionals, i.e., 5.7895 vs. 3.5455 (median 6 vs. 3), respectively. This suggests that non-professional users were more positive toward the machine-translated text than professional users, perhaps because professional users are more aware of features of good translation and are able to notice when they are not present.</p>
<fig id="F3" position="float">
<label>Figure 3</label>
<caption><p>Total quality scores in the groups of professional and non-professional users of machine translation.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fpsyg-14-1076379-g0003.tif"/>
</fig>
<fig id="F4" position="float">
<label>Figure 4</label>
<caption><p>Total usability scores in the groups of professional and non-professional users of machine translation.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fpsyg-14-1076379-g0004.tif"/>
</fig>
<fig id="F5" position="float">
<label>Figure 5</label>
<caption><p>Total satisfaction scores in the groups of professional and non-professional users of machine translation.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fpsyg-14-1076379-g0005.tif"/>
</fig>
<p>Independent-samples Mann-Whitney <italic>U</italic>-test would also indicate that the hypothesis that the total satisfaction score of professionals and non-professionals has the same distribution (more precisely, the hypothesis that the probability of this score being higher for a random professional than for a random non-professional is 0.5) can be rejected (<italic>p</italic> &#x0003C; 0.001). On the other hand, the same test does not suggest rejecting the hypotheses that the total usability score and the total quality score of professionals and non-professionals have the same distributions (<italic>p</italic> &#x0003D; 0.427 and <italic>p</italic> &#x0003D; 0.381).</p>
<p>One-sample Kolmogorov-Smirnov test suggests that the hypotheses of total quality score, total usability score and total satisfaction score having normal distribution could be rejected (<italic>p</italic> &#x0003D; 0.020, <italic>p</italic> &#x0003D; 0.017, <italic>p</italic> &#x0003C; 0.001), while the hypothesis that their sum has a normal distribution could not be rejected (<italic>p</italic> &#x0003D; 0.200).</p>
<p>The subjects from the group of non-professional users who thought that the quality was low (having scores lower than average; there were 16 such subjects out of 21) demonstrated a longer average fixation time both for AOIs with errors and without errors (12.2 vs. 10.7%, respectively) than those subjects who thought that the quality was high (10.4 vs. 9.4%, respectively) (see <xref ref-type="fig" rid="F6">Figure 6</xref>).</p>
<fig id="F6" position="float">
<label>Figure 6</label>
<caption><p>Average percentage of fixation time of non-professional users who rated quality of the raw machine-translated text higher and lower than the average.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fpsyg-14-1076379-g0006.tif"/>
</fig>
<p>The same pattern was observed for the usability and satisfaction components. The subjects who thought that the text was barely usable (having scores lower than average; there were 16 such subjects out of 21) showed a longer fixation time result that those who thought that the text was usable (12.0 vs. 11.0% and 10.6 vs. 9.6%, respectively) (see <xref ref-type="fig" rid="F7">Figure 7</xref>).</p>
<fig id="F7" position="float">
<label>Figure 7</label>
<caption><p>Average percentage of fixation time of non-professional users who rated usability of the raw machine-translated text high and low.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fpsyg-14-1076379-g0007.tif"/>
</fig>
<p>The non-professional users who were less satisfied with the text (value lower than average; there were 17 such subjects out of 21) demonstrated a longer average fixation time result in comparison with those who were more satisfied with the text (12.0 vs. 10.6%, respectively) (see <xref ref-type="fig" rid="F8">Figure 8</xref>).</p>
<fig id="F8" position="float">
<label>Figure 8</label>
<caption><p>Average percentage of fixation time of non-professional users who were more and less satisfied with the raw machine-translated text.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fpsyg-14-1076379-g0008.tif"/>
</fig>
<p>All professional translators, language editors and revisers who read the raw machine-translated text provided to them in the experiment thought that the text quality was low, and they scored low on the questions of satisfaction in the post-task questionnaire on acceptability components. Only in terms of usability, the subjects of the professional translators&#x00027; group were divided into those who thought that the text was usable to some extent (usability higher than average) and those who thought that the text was not usable. The results for average percentage of fixation time of the two groups of professional translators - low scorers and high scorers for usability statements&#x02014;are shown in <xref ref-type="fig" rid="F9">Figure 9</xref>. The subjects in the group of low scorers for the usability statements demonstrated a shorter average fixation time compared with those who scored higher, 12.4 vs. 14.3% for AOIs with errors and 11.7 vs. 12.7% for AOIs without errors, which also raises questions for further research, discussion and implications.</p>
<fig id="F9" position="float">
<label>Figure 9</label>
<caption><p>Average percentage of fixation time of professional translators who scored low and high regarding the usability with the raw machine-translated text in the post-task questionnaire.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fpsyg-14-1076379-g0009.tif"/>
</fig>
<p>As <xref ref-type="fig" rid="F10">Figure 10</xref> shows, total satisfaction scores for non-professionals who looked at AOIs with errors for a shorter period of time than average varied greatly. The higher limit of those scores decreased for non-professionals who looked at such AOIs longer, while the lower limit tended to stay the same. On the other hand, the satisfaction scores for the professionals tended to stay the same, as for non-professionals who paid more attention to the AOIs with errors.</p>
<fig id="F10" position="float">
<label>Figure 10</label>
<caption><p>A scatter plot of fixation times for AOIs with errors and total satisfaction scores.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fpsyg-14-1076379-g0010.tif"/>
</fig>
<p>However, the independent-samples Mann-Whitney <italic>U</italic>-test would indicate that the hypothesis that the total satisfaction score of professionals and non-professionals has the same distribution (that the probability of this score being higher for a random professional than for a random non-professional is 0.5) cannot be rejected (<italic>p</italic> &#x0003D; 0.157).</p>
<p>Besides, the subjects&#x00027; text comprehension was measured <italic>via</italic> a post-task reading comprehension questionnaire, consisting of 4 questions, i.e., 2 true/false questions and 2 open questions. <xref ref-type="fig" rid="F11">Figure 11</xref> demonstrates how the subjects scored in both groups. The professionals scored better in text comprehension compared with non-professional users (median 2 vs. 3, respectively) (see <xref ref-type="fig" rid="F11">Figure 11</xref>).</p>
<fig id="F11" position="float">
<label>Figure 11</label>
<caption><p>A simple box plot of text comprehension results in the groups of professionals and non-professional users.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fpsyg-14-1076379-g0011.tif"/>
</fig>
<p><xref ref-type="fig" rid="F12">Figure 12</xref> shows how fixation times for AOIs with errors correlate with the number of correctly answered questions. It may be seen that the pattern differs between professionals and non-professionals, with professionals having higher spread for more correct answers and non-professionals having higher spread for average number of correct answers. It is also interesting that the median dwell time was mostly the same for non-professionals giving different numbers of correct answers, while the median dwell times for professionals giving the highest and the lowest numbers of correct answers are lower than for professionals who gave a medium number of correct answers. Furthermore, both professionals and non-professionals who gave no correct answers (there were two such professionals and two such non-professionals) had low dwell times (with the maximum lower than the medians of every other group).</p>
<fig id="F12" position="float">
<label>Figure 12</label>
<caption><p>A simple box plot of fixation times for AOIs with errors by the number of correctly answered questions.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fpsyg-14-1076379-g0012.tif"/>
</fig>
<p><xref ref-type="fig" rid="F13">Figure 13</xref> shows how glance counts for AOIs with errors correlate with the number of correctly answered questions. The differences between professionals and non-professionals may be observed, with professionals having higher spread and non-professionals having lower spread for the higher number of correct answers. Professionals tended to reach higher glance counts (for each number of correct answers, professionals tended to have a higher median glance count, with the exception of the group of no correct answers, which might have been an outlier). Furthermore, non-professionals who gave no correct answers had high glances counts (with median higher than the medians of every other group of non-professionals). As they also had low dwell times, this might indicate that the respondents who gave no correct answers were relatively inattentive.</p>
<fig id="F13" position="float">
<label>Figure 13</label>
<caption><p>A simple box plot of glances counts for AOIs with errors by the number of correctly answered questions.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fpsyg-14-1076379-g0013.tif"/>
</fig>
<p><xref ref-type="fig" rid="F14">Figure 14</xref> shows how dwell times for AOIs with errors correlate with the number of correctly answered questions. The pattern again differs between professionals and non-professionals, with professionals having a higher spread for more correct answers and non-professionals having a higher spread for the average number of correct answers. Furthermore, both professionals and non-professionals who gave no correct answers had low fixation times (with the maximum lower than the medians of every other group), which may imply that less attention and effort while reading results in lower comprehension. Of course, such a finding needs to be tested and proven in a better targeted study, as in this particular case there might have been other factors like the text type, topic, tiredness, general absence of interest, etc. that influenced the results.</p>
<fig id="F14" position="float">
<label>Figure 14</label>
<caption><p>A simple box plot of dwell times for AOIs with errors by the number of correctly answered questions.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fpsyg-14-1076379-g0014.tif"/>
</fig></sec>
<sec sec-type="discussion" id="s5">
<title>5. Discussion</title>
<p>Previous studies focusing solely on machine translation acceptability are few. Even fewer studies apply eye tracking to test machine translation acceptability. They mainly focus on the experiments with professional translators and/or translation students. To the best of our knowledge, there are no reported studies where machine translation acceptability by non-professional users was tested <italic>via</italic> an eye tracking experiment. No such research testing acceptability of machine-translated text into Lithuanian has been conducted so far. Lithuanian, like many other smaller languages, is considered underresourced. It is also a morphologically rich synthetic language. Consequently, machine translation quality is less adequate than in other languages where investment into data acquisition and machine translation development is more substantial. Therefore, the views of Lithuanian language speakers, or smaller language speakers overall, toward machine translation might be diverse and involve many more risks or unexpected threats, if the output is used without critical awareness and judgment. For these reasons, comparisons between our results and previous research are only partial or indirect.</p>
<p>This study revolved around three research questions. The first question was related to comparison between professional and non-professional users&#x00027; processing of raw machine translation output. The most obvious finding to emerge from this study is that there is a difference in the machine translation output cognitive processing and acceptability between professional and non-professional users. In comparison with non-professional users, professional users of machine translation, i.e., translators and language editors, spend more time overall reading the machine-translated texts, most probably because of their deeper critical awareness as well as proficient attitude toward the text. They also demonstrate a longer average fixation time and a greater average glance count on the machine translation errors. In terms of acceptability overall, professional users critically assess machine translation on all components of acceptability. This might possibly be explained by an assumption that professionals have less tolerance toward insufficient quality of machine translation, know how to prepare texts for publishable quality and see mistakes, inaccuracies and style issues in a text almost instantaneously. On the other hand, even if the text contains errors, it might still be usable.</p>
<p>The results obtained in this study seem to be to some extent consistent with the findings obtained in previous studies. Garc&#x000ED;a (<xref ref-type="bibr" rid="B12">2010</xref>) who investigated the level of user awareness of machine translation among professional translators and clients or users of translation found out that only a small proportion of professionals considered the quality of machine translation very high, which is not surprising since at the time machine translation had lower quality than the neural machine translation now. However, in the same study, the clients/users of translations demonstrated more positive assessment of the quality of machine translation compared to that of professional users (Garc&#x000ED;a, <xref ref-type="bibr" rid="B12">2010</xref>). Our findings are also in line with the implications revealed by Vieira (<xref ref-type="bibr" rid="B39">2020</xref>) who concluded that there is a clear divide between the perceptions of professionals and non-professionals toward machine translation and its capabilities. In his study, Vieira acknowledged that the public coverage of machine translation veers more toward positive attitudes rather than negative. In our study, non-professional users&#x02014;end-users with no linguistic background&#x02014;also had more positive attitudes toward machine translation quality, usability and satisfaction compared with the professional translators&#x00027; attitudes toward the text. However, in principle, our results may also be indirectly considered to be in agreement with those obtained in a study by Hu et al. (<xref ref-type="bibr" rid="B15">2020</xref>) where subjects with low proficiency in English considered a raw machine-translated output quality lower than the post-edited text, i.e., one containing no errors. Although Hu et al.&#x00027;s and our studies have different designs and purposes, it may be inferred that even non-professionals who may be expected to be ignorant of or care less about mistakes in the text are generally aware of drawbacks and notice them.</p>
<p>Some of our study results may also be to some extent comparable with those obtained in the investigation by Colman et al. (<xref ref-type="bibr" rid="B7">2021</xref>) where an increased number of eye fixations and increased gaze duration while reading machine translation segments were found in comparison with human translation (Colman et al., <xref ref-type="bibr" rid="B7">2021</xref>), which may imply that less naturalistic and possibly erroneous text segments require more cognitive load. In our study, all respondents (both professionals and non-professionals) demonstrated increased values of all tested eye movement variables on areas of interest with errors compared with areas of interest without errors.</p>
<p>Other noteworthy findings to emerge from this study relate to the question whether there is a difference in the processing of raw machine-translated output between non-professional users with different levels of acceptability of machine-translated text. The overall acceptability of machine-translated text was found to be higher for those non-professional users who spent less time/effort on areas of interest with errors, which might be an indication that the participants who did not notice or were more positive or tolerant about the mistakes were more positive about the machine-translated text in general. The text comprehension results revealed that the subjects in the group of professional translators who scored low on the comprehension questions, demonstrated a greater number of glance counts, which may imply that the professional background may influence the level of comprehension. These findings indirectly support the more positive attitudes toward raw machine-translated text in terms of its comprehension and trust by users with lower proficiency of language as reported by Rossetti et al. (<xref ref-type="bibr" rid="B30">2020</xref>).</p>
<p>However, with a relatively small sample size, caution must be applied while interpreting the results within the group of non-professional users of machine translation as the findings might be diverse depending on the subject&#x00027;s background, level of education, experience, language proficiency and other variables.</p></sec>
<sec sec-type="conclusions" id="s6">
<title>6. Conclusions</title>
<p>The study was aimed at determining the acceptability of raw machine translation texts in Lithuanian, a low-resource language. An eye tracking experiment measuring acceptability <italic>via</italic> the comparison between professional and non-professional users of machine translation and <italic>via</italic> the comparisons between the respondents who assessed the quality, satisfaction with and usability of the text differently (either lower or higher than average) revealed some insightful findings. There is a difference in the machine translation output cognitive processing and acceptability between professional and non-professional users. The professional users scored better in text comprehension compared with non-professional users. One of the possible reasons for that might be the experience of professional translators in dealing with badly written (perhaps also machine-translated) text. Professional users critically assess machine translation on all components of acceptability. Non-professional users&#x02014;end-users with no linguistic background&#x02014;have more positive attitudes toward machine translation quality, usability and satisfaction, which may imply possible risks if machine translation is used without critical awareness, judgement and revision. The lower professional users&#x00027; satisfaction with the text, and overall acceptability, may suggest that they are likely to have higher expectations for the translated text.</p>
<p>The major general implication of these findings is the lower awareness of non-professional users regarding the machine translation output drawbacks and imperfections, which may result in a variety of misunderstandings that might go unnoticed and ignored, as well as risks and threats with undefined consequences.</p>
<p>The major limitation of this study is the small and uneven sample sizes of professional and non-professional users of machine translation. More equal sample sizes of different groups would help establishing a greater degree of accuracy on this matter. Besides, the differences within the non-professionals&#x00027; group should be taken into consideration, as the results may be affected by various individual characteristics of subjects. Therefore, larger controlled trials could be focused more on the differences in educational backgrounds and language proficiency of subjects as well as the provided stimulus text variety or task description to give more definitive evidence regarding acceptability of machine translation.</p>
<p>A further limitation concerns imperfections of eye tracking equipment. To some extent they have been mitigated, but those mitigations can also be a cause of further limitations (for example, padding AOIs by about 1 character to all sides is a common way to mitigate imprecision of eye tracking leading to failures to notice the subject looking at the AOI, but it can result in including cases when the subject is looking at the area near the AOI).</p>
<p>Notwithstanding these limitations, the study provides a possibility to understand more deeply the readers&#x00027; cognitive processing and the level of acceptability they exhibit toward machine-translated texts. Overall, the results of the study demonstrate diversified and contrasting views of the population and call for raising public awareness and machine translation literacy improvement.</p></sec>
<sec sec-type="data-availability" id="s7">
<title>Data availability statement</title>
<p>The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.</p></sec>
<sec sec-type="ethics-statement" id="s8">
<title>Ethics statement</title>
<p>The studies involving human participants were reviewed and approved by Research Ethics Commission of Kaunas University of Technology. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.</p></sec>
<sec sec-type="author-contributions" id="s9">
<title>Author contributions</title>
<p>RK and JM: conceptualization. RK, JM, IP, and MP: methodology. RK, JM, IP, MP, and JH: investigation and writing&#x02014;review and editing. IP and MP: data curation, visualization. RK, JM, and JH: writing&#x02014;original draft preparation. All authors have read and agreed to the published version of the manuscript.</p></sec>
</body>
<back>
<sec sec-type="funding-information" id="s10">
<title>Funding</title>
<p>This research had received funding from the Research Council of Lithuania (LMTLT, agreement No S-MOD-21-2).</p>
</sec>
<sec sec-type="COI-statement" id="conf1">
<title>Conflict of interest</title>
<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.</p>
</sec>
<sec sec-type="disclaimer" id="s11">
<title>Publisher&#x00027;s note</title>
<p>All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.</p>
</sec>
<fn-group>
<fn id="fn0001"><p><sup>1</sup>Approval to conduct this study was obtained from the Research Ethics Committee of Kaunas University of Technology (No. M6-2021-04 as of 2021-06-16).</p></fn>
</fn-group>
<ref-list>
<title>References</title>
<ref id="B1">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Carl</surname> <given-names>M.</given-names></name> <name><surname>Dragsted</surname> <given-names>B.</given-names></name> <name><surname>Elming</surname> <given-names>J.</given-names></name> <name><surname>Hardt</surname> <given-names>D.</given-names></name> <name><surname>Lykke Jakobsen</surname> <given-names>A.</given-names></name></person-group> (<year>2011</year>). <article-title>&#x0201C;The process of post-editing: a pilot study,&#x0201D;</article-title> in <source>Copenhagen Studies in Language</source> (<publisher-loc>Frederiksberg</publisher-loc>), <fpage>131</fpage>&#x02013;<lpage>142</lpage>.</citation>
</ref>
<ref id="B2">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Carl</surname> <given-names>M.</given-names></name> <name><surname>Gutermuth</surname> <given-names>S.</given-names></name> <name><surname>Hansen-Schirra</surname> <given-names>S.</given-names></name></person-group> (<year>2015</year>). <article-title>&#x0201C;Chapter post-editing machine translation: efficiency, strategies, and revision processes in professional translation settings,&#x0201D;</article-title> in <source>Psycholinguistic and Cognitive Inquiries Into Translation and Interpreting</source> (<publisher-loc>Amsterdam</publisher-loc>: <publisher-name>John Benjamins Publishing Company</publisher-name>), <fpage>145</fpage>&#x02013;<lpage>174</lpage>.</citation>
</ref>
<ref id="B3">
<citation citation-type="thesis"><person-group person-group-type="author"><name><surname>Castilho</surname> <given-names>S.</given-names></name></person-group> (<year>2016</year>). <source>Measuring acceptability of machine translated enterprise content</source> (Ph.D. thesis). <publisher-loc>Dublin</publisher-loc>: <publisher-name>Dublin City University</publisher-name>.</citation>
</ref>
<ref id="B4">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Castilho</surname> <given-names>S.</given-names></name> <name><surname>O&#x00027;Brien</surname> <given-names>S.</given-names></name></person-group> (<year>2016</year>). <article-title>&#x0201C;Evaluating the impact of light post-editing on usability,&#x0201D;</article-title> in <source>Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC&#x00027;16)</source> (<publisher-loc>Portoroz</publisher-loc>: <publisher-name>European Language Resources Association, ELRA</publisher-name>), <fpage>310</fpage>&#x02013;<lpage>316</lpage>.</citation>
</ref>
<ref id="B5">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Castilho</surname> <given-names>S.</given-names></name> <name><surname>O&#x00027;Brien</surname> <given-names>S.</given-names></name> <name><surname>Alves</surname> <given-names>F.</given-names></name> <name><surname>O&#x00027;Brien</surname> <given-names>M.</given-names></name></person-group> (<year>2014</year>). <article-title>&#x0201C;Does post-editing increase usability? a study with Brazilian Portuguese as target language,&#x0201D;</article-title> in <source>Proceedings of the 17th Annual conference of the European Association for Machine Translation</source> (<publisher-loc>Dubrovnik</publisher-loc>: <publisher-name>European Association for Machine Translation</publisher-name>), <fpage>183</fpage>&#x02013;<lpage>190</lpage>.</citation>
</ref>
<ref id="B6">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Castilho</surname> <given-names>S.</given-names></name> <name><surname>O&#x00027;Brien</surname> <given-names>S.</given-names></name></person-group> (<year>2018</year>). <article-title>&#x0201C;Acceptability of machine-translated content: a multi-language evaluation by translators and end-users,&#x0201D;</article-title> in <source>Linguistica Antverpiensia, New Series - Themes in Translation Studies, Vol. 16</source> (Antwerp).</citation>
</ref>
<ref id="B7">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Colman</surname> <given-names>T.</given-names></name> <name><surname>Fonteyne</surname> <given-names>M.</given-names></name> <name><surname>Daems</surname> <given-names>J.</given-names></name> <name><surname>Macken</surname> <given-names>L.</given-names></name></person-group> (<year>2021</year>). <article-title>&#x0201C;It&#x00027;s all in the eyes: an eye tracking experiment to assess the readability of machine translated literature,&#x0201D;</article-title> in <source>31st Meeting of Computational Linguistics in The Netherlands (CLIN 31), Abstracts</source> (<publisher-loc>Ghent</publisher-loc>).</citation>
</ref>
<ref id="B8">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Daems</surname> <given-names>J.</given-names></name> <name><surname>Vandepitte</surname> <given-names>S.</given-names></name> <name><surname>Hartsuiker</surname> <given-names>R. J.</given-names></name> <name><surname>Macken</surname> <given-names>L.</given-names></name></person-group> (<year>2017</year>). <article-title>Identifying the machine translation error types with the greatest impact on post-editing effort</article-title>. <source>Front. Psychol</source>. <volume>8</volume>, <fpage>01282</fpage>. <pub-id pub-id-type="doi">10.3389/fpsyg.2017.01282</pub-id><pub-id pub-id-type="pmid">28824482</pub-id></citation></ref>
<ref id="B9">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Doherty</surname> <given-names>S.</given-names></name></person-group> (<year>2016</year>). <article-title>Translations| the impact of translation technologies on the process and product of translation</article-title>. <source>Int. J. Commun</source>. <volume>10</volume>, <fpage>947</fpage>&#x02013;<lpage>969</lpage>.</citation>
</ref>
<ref id="B10">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Doherty</surname> <given-names>S.</given-names></name> <name><surname>O&#x00027;Brien</surname> <given-names>S.</given-names></name></person-group> (<year>2014</year>). <article-title>Assessing the usability of raw machine translated output: a user-centered study using eye tracking</article-title>. <source>Int. J. Hum. Comput. Interact</source>. <volume>30</volume>, <fpage>40</fpage>&#x02013;<lpage>51</lpage>. <pub-id pub-id-type="doi">10.1080/10447318.2013.802199</pub-id></citation>
</ref>
<ref id="B11">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Fereira</surname> <given-names>A</given-names></name> <name><surname>Gries</surname> <given-names>S. T.</given-names></name> <name><surname>Schwieter</surname> <given-names>J. W.</given-names></name></person-group> (<year>2021</year>). <article-title>&#x0201C;Assessing indicators of cognitive effort in professional translators: A study on language dominance and directionality,&#x0201D;</article-title> in <source>Translation, interpreting, cognition: The way out of the box</source>, ed Tra&#x00026;Co Group (<publisher-loc>Berlin</publisher-loc>: <publisher-name>Language Science Press</publisher-name>), <fpage>115</fpage>&#x02013;<lpage>143</lpage>. <pub-id pub-id-type="doi">10.5281/zenodo.4545041</pub-id></citation>
</ref>
<ref id="B12">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Garc&#x000ED;a</surname> <given-names>I.</given-names></name></person-group> (<year>2010</year>). <article-title>Is machine translation ready yet?</article-title> <source>Target</source> <volume>22</volume>, <fpage>7</fpage>&#x02013;<lpage>21</lpage>. <pub-id pub-id-type="doi">10.1075/target.22.1.02gar</pub-id></citation>
</ref>
<ref id="B13">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Guerberof Arenas</surname> <given-names>A.</given-names></name> <name><surname>Moorkens</surname> <given-names>J.</given-names></name> <name><surname>O&#x00027;Brien</surname> <given-names>S.</given-names></name></person-group> (<year>2021</year>). <article-title>The impact of translation modality on user experience: an eye-tracking study of the microsoft word user interface</article-title>. <source>Mach. Transl</source>. <volume>35</volume>, <fpage>205</fpage>&#x02013;<lpage>237</lpage>. <pub-id pub-id-type="doi">10.1007/s10590-021-09267-z</pub-id><pub-id pub-id-type="pmid">34776636</pub-id></citation></ref>
<ref id="B14">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hoi</surname> <given-names>H. T.</given-names></name></person-group> (<year>2020</year>). <article-title>Machine translation and its impact in our modern society</article-title>. <source>Int. J. Sci. Technol. Res</source>. <volume>9</volume>, <fpage>1918</fpage>&#x02013;<lpage>1921</lpage>.</citation>
</ref>
<ref id="B15">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hu</surname> <given-names>K.</given-names></name> <name><surname>O&#x00027;Brien</surname> <given-names>S.</given-names></name> <name><surname>Kenny</surname> <given-names>D.</given-names></name></person-group> (<year>2020</year>). <article-title>A reception study of machine translated subtitles for MOOCs</article-title>. <source>Perspectives</source> <volume>28</volume>, <fpage>521</fpage>&#x02013;<lpage>538</lpage>. <pub-id pub-id-type="doi">10.1080/0907676X.2019.1595069</pub-id></citation>
</ref>
<ref id="B16">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Jakobsen</surname> <given-names>A. L.</given-names></name> <name><surname>Jensen</surname> <given-names>K. T. H.</given-names></name></person-group> (<year>2008</year>). <article-title>Eye movement behaviour across four different types of reading task</article-title>. <source>Copenhagen Stud. Lang</source>. <volume>36</volume>, <fpage>103</fpage>&#x02013;<lpage>124</lpage>.</citation>
</ref>
<ref id="B17">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kasper&#x00117;</surname> <given-names>R.</given-names></name> <name><surname>Horba&#x0010D;auskien&#x00117;</surname> <given-names>J.</given-names></name> <name><surname>Motiejunien&#x00117;</surname> <given-names>J.</given-names></name> <name><surname>Liubinien&#x00117;</surname> <given-names>V.</given-names></name> <name><surname>Pata&#x00161;ien&#x00117;</surname> <given-names>I.</given-names></name> <name><surname>Pata&#x00161;ius</surname> <given-names>M.</given-names></name></person-group> (<year>2021</year>). <article-title>Towards sustainable use of machine translation: usability and perceived quality from the end-user perspective</article-title>. <source>Sustainability</source> <volume>13</volume>, <fpage>3430</fpage>. <pub-id pub-id-type="doi">10.3390/su132313430</pub-id></citation>
</ref>
<ref id="B18">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kasperavi&#x0010D;ien&#x00117;</surname> <given-names>R.</given-names></name> <name><surname>Motiej&#x0016B;nien&#x00117;</surname> <given-names>J.</given-names></name> <name><surname>Pata&#x00161;ien&#x00117;</surname> <given-names>I.</given-names></name></person-group> (<year>2020</year>). <article-title>Quality assessment of machine translation output</article-title>. <source>Texto Livre</source> <volume>13</volume>, <fpage>271</fpage>&#x02013;<lpage>285</lpage>. <pub-id pub-id-type="doi">10.35699/1983-3652.2020.24399</pub-id></citation>
</ref>
<ref id="B19">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Kasper&#x00117;</surname> <given-names>R.</given-names></name> <name><surname>Motiej&#x0016B;nien&#x00117;</surname> <given-names>L.</given-names></name></person-group> (<year>2021</year>). <article-title>&#x0201C;Eye-tracking experiments in human acceptability of machine translation to study societal impacts,&#x0201D;</article-title> in <source>Sustainable Multilingualism 2021: The 6th International Conference</source>, eds A. Dauk&#x00161;aite-Kolpakoviene and &#x0017D;. Tama&#x00161;auskaite (June 4&#x02013;5, 2021, Kaunas, Lithuania): book of abstracts (<publisher-loc>Kaunas</publisher-loc>: <publisher-name>Vytautas Magnus University</publisher-name>), <fpage>114</fpage>.</citation>
</ref>
<ref id="B20">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Lommel</surname> <given-names>A.</given-names></name> <name><surname>Uszkoreit</surname> <given-names>H.</given-names></name> <name><surname>Burchardt</surname> <given-names>A.</given-names></name></person-group> (<year>2014</year>). <article-title>Multidimensional quality metrics (MQM): a framework for declaring and describing translation quality metrics</article-title>. <source>Tradum&#x000E0;tica tecnol. trad</source>. <volume>12</volume>, <fpage>455</fpage>. <pub-id pub-id-type="doi">10.5565/rev/tradumatica.77</pub-id></citation>
</ref>
<ref id="B21">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>McConkie</surname> <given-names>G. W.</given-names></name> <name><surname>Rayner</surname> <given-names>K.</given-names></name></person-group> (<year>1975</year>). <article-title>The span of the effective stimulus during a fixation in reading</article-title>. <source>Percept. Psychophys</source>. <volume>17</volume>, <fpage>578</fpage>&#x02013;<lpage>586</lpage>. <pub-id pub-id-type="doi">10.3758/BF03203972</pub-id><pub-id pub-id-type="pmid">28709924</pub-id></citation></ref>
<ref id="B22">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Moorkens</surname> <given-names>J.</given-names></name></person-group> (<year>2018</year>). <article-title>&#x0201C;Chapter Eye-Tracking as a Measure of Cognitive Effort for Post-Editing of Machine Translation,&#x0201D;</article-title> in <source>Eye Tracking and Multidisciplinary Studies on Translation</source> (<publisher-loc>Amsterdam</publisher-loc>: <publisher-name>John Benjamins Publishing Company</publisher-name>), <fpage>55</fpage>&#x02013;<lpage>69</lpage>.</citation>
</ref>
<ref id="B23">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Moorkens</surname> <given-names>J.</given-names></name> <name><surname>O&#x00027;Brien</surname> <given-names>S.</given-names></name></person-group> (<year>2015</year>). <article-title>&#x0201C;Post-editing evaluations: trade-offs between novice and professional participants,&#x0201D;</article-title> in <source>Proceedings of the 18th Annual Conference of the European Association for Machine Translation</source> (<publisher-loc>Antalya</publisher-loc>), <fpage>75</fpage>&#x02013;<lpage>81</lpage>.</citation>
</ref>
<ref id="B24">
<citation citation-type="web"><person-group person-group-type="author"><collab>MQM Commitee.</collab></person-group> (<year>2022</year>). <source>MQM (Multidimensional Quality Metrics): what is MQM?</source> Available online at: <ext-link ext-link-type="uri" xlink:href="https://themqm.org/">https://themqm.org/</ext-link> (accessed October 10, 2022).</citation>
</ref>
<ref id="B25">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Nurminen</surname> <given-names>M.</given-names></name> <name><surname>Koponen</surname> <given-names>M.</given-names></name></person-group> (<year>2020</year>). <article-title>Machine translation and fair access to information</article-title>. <source>Transl. Spaces</source> <volume>9</volume>, <fpage>150</fpage>&#x02013;<lpage>169</lpage>. <pub-id pub-id-type="doi">10.1075/ts.00025.nur</pub-id></citation>
</ref>
<ref id="B26">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Ortega</surname> <given-names>J.</given-names></name> <name><surname>S&#x000E1;nchez-Mart&#x000ED;nez</surname> <given-names>F.</given-names></name> <name><surname>Turchi</surname> <given-names>M.</given-names></name> <name><surname>Negri</surname> <given-names>M.</given-names></name></person-group> (<year>2019</year>). <article-title>&#x0201C;Improving translations by combining fuzzy-match repair with automatic post-editing,&#x0201D;</article-title> in <source>Proceedings of Machine Translation Summit XVII: Research Track</source> (<publisher-loc>Dublin</publisher-loc>: <publisher-name>European Association for Machine Translation</publisher-name>), <fpage>256</fpage>&#x02013;<lpage>266</lpage>.</citation>
</ref>
<ref id="B27">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Perrault</surname> <given-names>R.</given-names></name> <name><surname>Shoham</surname> <given-names>Y.</given-names></name> <name><surname>Brynjolfsson</surname> <given-names>E.</given-names></name> <name><surname>Clark</surname> <given-names>J.</given-names></name> <name><surname>Etchemendy</surname> <given-names>J.</given-names></name> <name><surname>Grosz</surname> <given-names>B.</given-names></name> <etal/></person-group>. (<year>2019</year>). <source>The ai index 2019 annual report</source>. <publisher-loc>Technical report, Stanford</publisher-loc>: <publisher-name>AI Index Steering Committee, Human-Centered AI Institute, Stanford University</publisher-name>.</citation>
</ref>
<ref id="B28">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Rayner</surname> <given-names>K.</given-names></name></person-group> (<year>1998</year>). <article-title>Eye movements in reading and information processing: 20 years of research</article-title>. <source>Psychol. Bull</source>. <volume>124</volume>, <fpage>372</fpage>&#x02013;<lpage>422</lpage>. <pub-id pub-id-type="doi">10.1037/0033-2909.124.3.372</pub-id><pub-id pub-id-type="pmid">9849112</pub-id></citation></ref>
<ref id="B29">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Rivera-Trigueros</surname> <given-names>I.</given-names></name></person-group> (<year>2021</year>). <article-title>Machine translation systems and quality assessment: a systematic review</article-title>. <source>Lang. Resour. Eval</source>. <pub-id pub-id-type="doi">10.1007/s10579-021-09537-5</pub-id><pub-id pub-id-type="pmid">34597937</pub-id></citation></ref>
<ref id="B30">
<citation citation-type="web"><person-group person-group-type="author"><name><surname>Rossetti</surname> <given-names>A.</given-names></name> <name><surname>O&#x00027;Brien</surname> <given-names>S.</given-names></name> <name><surname>Cadwell</surname> <given-names>P.</given-names></name></person-group> (<year>2020</year>). <article-title>&#x0201C;Comprehension and trust in crises: investigating the impact of machine translation and post-editing,&#x0201D;</article-title> in <source>Proceedings of the 22nd Annual Conference of the European Association for Machine Translation</source> (<publisher-loc>Lisboa</publisher-loc>: <publisher-name>European Association for Machine Translation</publisher-name>), <fpage>9</fpage>&#x02013;<lpage>18</lpage>. Available online at: <ext-link ext-link-type="uri" xlink:href="https://aclanthology.org/2020.eamt-1.2">https://aclanthology.org/2020.eamt-1.2</ext-link></citation>
</ref>
<ref id="B31">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Rossi</surname> <given-names>C.</given-names></name> <name><surname>Carr&#x000E9;</surname> <given-names>A.</given-names></name></person-group> (<year>2022</year>). <article-title>&#x0201C;How to choose a suitable neural machine translation solution: Evaluation of MT quality,&#x0201D;</article-title> in <source>Machine Translation for Everyone: Empowering Users in the Age of Artificial Intelligence</source>, ed D. Kenny (<publisher-loc>Berlin</publisher-loc>: <publisher-name>Language Science Press</publisher-name>), <fpage>51</fpage>&#x02013;<lpage>79</lpage>. <pub-id pub-id-type="doi">10.5281/zenodo.6759978</pub-id></citation>
</ref>
<ref id="B32">
<citation citation-type="web"><person-group person-group-type="author"><name><surname>Rossi</surname> <given-names>C.</given-names></name> <name><surname>Chevrot</surname> <given-names>J.-P.</given-names></name></person-group> (<year>2019</year>). <article-title>Uses and perceptions of machine translation at the european commission</article-title>. <source>J. Special. Transl</source>. <volume>31</volume>, <fpage>177</fpage>&#x02013;<lpage>200</lpage>. Available online at: <ext-link ext-link-type="uri" xlink:href="https://shs.hal.science/halshs-01893120/file/Rossi_and_Chevrot_article8.pdf">https://shs.hal.science/halshs-01893120/file/Rossi_and_Chevrot_article8.pdf</ext-link></citation>
</ref>
<ref id="B33">
<citation citation-type="web"><person-group person-group-type="author"><name><surname>Schuster</surname> <given-names>M.</given-names></name> <name><surname>Johnson</surname> <given-names>M.</given-names></name> <name><surname>Thorat</surname> <given-names>N.</given-names></name></person-group> (<year>2016</year>). <source>Zero-shot Translation With Google&#x00027;s Multilingual Neural Machine Translation System. Google AI blog</source>. Available online at: <ext-link ext-link-type="uri" xlink:href="https://ai.googleblog.com/2016/11/zero-shot-translation-with-googles.html">https://ai.googleblog.com/2016/11/zero-shot-translation-with-googles.html</ext-link></citation>
</ref>
<ref id="B34">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Stasimioti</surname> <given-names>M.</given-names></name> <name><surname>Sosoni</surname> <given-names>V.</given-names></name></person-group> (<year>2021</year>). <article-title>&#x0201C;Chapter Investigating post-editing: a mixed-methods study with experienced and novice translators in the English-Greek language pair, &#x02018;</article-title> <source>Translation, Interpreting, Cognition: The Way Out of the Box</source> (<publisher-loc>Berlin</publisher-loc>: <publisher-name>Language Science Press</publisher-name>).</citation>
</ref>
<ref id="B35">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Taivalkoski-Shilov</surname> <given-names>K.</given-names></name> <name><surname>Toral</surname> <given-names>A.</given-names></name> <name><surname>Hadley</surname> <given-names>J. L.</given-names></name> <name><surname>Teixeira</surname> <given-names>C. S. C.</given-names></name></person-group> editors (<year>2022</year>). <article-title>&#x0201C;Using technologies for creative-text translation,&#x0201D;</article-title> in <source>Routledge Advances in Translation and Interpreting Studies</source> (<publisher-loc>London</publisher-loc>: <publisher-name>Routledge</publisher-name>).</citation>
</ref>
<ref id="B36">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Taylor</surname> <given-names>R. M.</given-names></name> <name><surname>Crichton</surname> <given-names>N.</given-names></name> <name><surname>Moult</surname> <given-names>B.</given-names></name> <name><surname>Gibson</surname> <given-names>F.</given-names></name></person-group> (<year>2015</year>). <article-title>A prospective observational study of machine translation software to overcome the challenge of including ethnic diversity in healthcare research</article-title>. <source>Nurs. Open</source> <volume>2</volume>, <fpage>14</fpage>&#x02013;<lpage>23</lpage>. <pub-id pub-id-type="doi">10.1002/nop2.13</pub-id><pub-id pub-id-type="pmid">27708797</pub-id></citation></ref>
<ref id="B37">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Ueffing</surname> <given-names>N.</given-names></name></person-group> (<year>2018</year>). <article-title>&#x0201C;Automatic post-editing and machine translation quality estimation at eBay,&#x0201D;</article-title> in <source>Proceedings of the AMTA 2018 Workshop on Translation Quality Estimation and Automatic Post-Editing</source> (<publisher-loc>Boston, MA</publisher-loc>: <publisher-name>Association for Machine Translation in the Americas</publisher-name>), <fpage>1</fpage>&#x02013;<lpage>34</lpage>.</citation>
</ref>
<ref id="B38">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Vardaro</surname> <given-names>J.</given-names></name> <name><surname>Schaeffer</surname> <given-names>M.</given-names></name> <name><surname>Hansen-Schirra</surname> <given-names>S.</given-names></name></person-group> (<year>2019</year>). <article-title>Translation quality and error recognition in professional neural machine translation post-editing</article-title>. <source>Informatics</source> <volume>6</volume>, <fpage>41</fpage>. <pub-id pub-id-type="doi">10.3390/informatics6030041</pub-id></citation>
</ref>
<ref id="B39">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Vieira</surname> <given-names>L. N.</given-names></name></person-group> (<year>2020</year>). <article-title>Machine translation in the news</article-title>. <source>Transl. Spaces</source> <volume>9</volume>, <fpage>98</fpage>&#x02013;<lpage>122</lpage>. <pub-id pub-id-type="doi">10.1075/ts.00023.nun</pub-id></citation>
</ref>
<ref id="B40">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Vieira</surname> <given-names>L. N.</given-names></name> <name><surname>O&#x00027;Hagan</surname> <given-names>M.</given-names></name> <name><surname>O&#x00027;Sullivan</surname> <given-names>C.</given-names></name></person-group> (<year>2021</year>). <article-title>Understanding the societal impacts of machine translation: a critical review of the literature on medical and legal use cases</article-title>. <source>Inf. Commun. Soc</source>. <volume>24</volume>, <fpage>1515</fpage>&#x02013;<lpage>1532</lpage>. <pub-id pub-id-type="doi">10.1080/1369118X.2020.1776370</pub-id></citation>
</ref>
<ref id="B41">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Yasuoka</surname> <given-names>M.</given-names></name> <name><surname>Bjorn</surname> <given-names>P.</given-names></name></person-group> (<year>2011</year>). <article-title>&#x0201C;Machine translation effect on communication: what makes it difficult to communicate through machine translation?&#x0201D;</article-title> in <source>2011 Second International Conference on Culture and Computing</source> (<publisher-loc>Kyoto</publisher-loc>: <publisher-name>IEEE</publisher-name>).</citation>
</ref>
</ref-list>
</back>
</article> 