Numerous research domains have used eye tracking. In marketing, its presence is apparent in consumer experience research, where viewers’ eyes were examined on various purchase stages (Duerrschmid and Danner, 2018; Ishibashi et al., 2019). In health care, it is acknowledged for its diagnostic (Sun et al., 2022), therapeutic (Harezlak and Kasprowski, 2018), and interactive properties (Tscholl et al., 2020). In education, examining learners’ experience with learning materials is facilitated by such technology (Conley et al., 2020; Serrano-Mamolar et al., 2023; Susac et al., 2023). Throughout the years, eye tracking is regarded as a tool for examining reading engagement (Child et al., 2020; Liu and Yu, 2022), comprehension (Abundis-Gutiérrez et al., 2018; Mézière et al., 2023), and atypical patterns like dyslexia or ADHD (Klein et al., 2019).

The ever-evolving technological landscape popularized the use of multimodal texts enabling learners to absorb information regardless of their learning styles (Bearne, 2012; Jancsary et al., 2016; Smith, 2012). Readers need to interact with multiple modes of information. The development of frameworks relative to multimodal literacies has taken up space in this field (Serafini, 2015; Shin et al., 2020). This area demands scoping examinations, constructing appropriate methodologies and optimizing its function. Eye-tracking provides perspectives on how readers can interact efficiently with multimodal materials (Holmqvist et al., 2011a, 2011b). Contrarily, the researchers find the tension posed by the utility of multimodal materials in literacy, believing that such practices cause strain to the limit of cognitive processing of the learners, thus limiting its potency (Mayer and Moreno, 2010). The insights gained from eye-tracking studies may guide the development of effective multimodal learning resources (Alemdag and Cagiltay, 2018), enabling researchers to analyze how long readers focus on particular elements of multimodal text (van der Sluis et al., 2018; Armfield, 2011; Schmidt-Weigand et al., 2010), which often demands transitions between different types of representations and careful consideration of information presented in graphs, as is the case with the Programme for International Student Assessment (PISA) items (Susac et al., 2018; Mason et al., 2015). Mason et al. (2022), showed how visual attention patterns can be used to inform the development of instructional video content. To illustrate the potential of eye-tracking in refining multimodal materials, consider its application in multimedia learning. Mayer and Fiorella (2021) discuss how students’ eye movements in multimedia lessons can reveal whether they are efficiently coordinating attention between explanatory diagrams and narration, highlighting areas where learners struggle to connect visual and verbal information, prompting educators to revise the layout or sequence of these materials for enhanced learning (Wiegand et al., 2017). Thus, this mini review validates the body of research on eye tracking in multimodal reading to identify the arguments that drive this expanding field. It reveals gaps in research while providing contrasting perspectives that influence current understandings. The review does not only contextualize current knowledge but also prognosticates on future implications.

The application of eye tracking technology in reading is anchored on Just and Carpenter’s (1980) “eye-mind hypothesis,” positing a connection between gaze location and cognitive processing. Whoever started the eye tracking movement during reading remains unsettled. However, the experiments of Louis Emile Javal in 1879 concluded that reading is a nonlinear process since readers’ eyes exhibit a series of quick movements dubbed by brief moments of stillness on certain parts of the text (Płużyczka, 2018). Moving to the early 20th century, Edmund Huey invented the first yet intrusive eye tracker to understand reading behaviors (Walczyk et al., 2014). Following this, Buswell’s seminal work in the 1920s discovered that eye movements are not smooth but composed of saccades and fixations (Wade, 2020), concepts that were previously identified by Javal but were not named during his time. Since then several researchers demonstrated how saccadic movements correlated with cognitive processing during reading (Rayner, 1978; Taylor, 1965).

The late 20th and early 21st centuries yielded profound insights into how readers interact with the texts through eye tracking technology within conventional reading environments. For example, skilled readers have shorter fixations and longer saccades as opposed to struggling readers (Boland, 2004; Weger and Inhoff, 2006). Shifting the focus to how engaged readers interact with texts, they display longer fixations on meaningful parts, while disengaged readers demonstrate quick eye movement patterns, shorter fixations, and frequent regressions—which signify comprehension difficulty (Holmqvist et al., 2011a, 2011b; Rayner and Pollatsek, 2006). Additionally, dyslexics display atypical eye movement patterns, finally shedding light on their word recognition and processing speed difficulties (Hyönä and Olson, 1995; Jones et al., 2008). Learners with ADHD, on the other hand, have erratic fixations with frequent saccadic movements (Karatekin and Asarnow, 1998), thereby clarifying why reading is a predicament for them. On a practical note, eye tracking technology paved to the development of targeted interventions like improving text readability (Goldberg and Wichansky, 2003) or integrating assistive technologies in the teaching practice (van Gog and Scheiter, 2010).

A multimodal text is a combination of more than one of the “modes,” pertaining to the method of communication being used: spatial, linguistic, visual, gestural, and audio, creating meaning far beyond the capacity of any single mode to do so (Moses and Reid, 2021; Sutrisno et al., 2023; Jewitt, 2013; Forceville, 2011). For example, with digital presentation, it can be read and seen coupled with animations and voice narration (Kress, 2010). Second is interactivity where many of the multimodal texts, especially those digital forms let users engage with the content through clicks, links, or swipes that determine how and what they navigate through in the text (Serafini, 2014). Third characteristic of the multimodal texts is non-linearity. Hypertexts and websites give readers liberty to navigate content through different pathways.

Kress and van Leeuwen (2006) explain that multimodal literacy relies on a reading operation where the reader decodes and syncretizes these diverse semiotic sources (Serafini, 2011; Forceville, 2010) unlike traditional print sources. For example, websites and new forms of digital media, such as digital comics strips or an infographics allow users to click on links to take them where they choose to within the content, thereby actively assuming meaning-making agency (Jewitt, 2013; Bezemer and Kress, 2008). In the case of an instructional video, they interpret the visual demonstrations as well as the auditory instructions (Serafini, 2014). For these reasons, it is necessary to delve into the literature about multimodal text reader interaction. To begin with, it widens to print literacy and encompasses various ways different people communicate within media environments (Cope and Kalantzis, 2009). With the growing incorporation of technology in educational systems, understanding how readers interact with multimodal texts can inform teaching practices so that students will be better prepared in the contemporary world (Walsh, 2010; Jewitt, 2009).

As noted by Shin (2023), “we live in a multimodal world,” and this becomes evident with the integration of various modes of communication to convey complex information. Eye tracking, according to Holsanova (2014), is a potentially useful tool for gathering accurate visual data about how readers engage with multimodal texts.

The area of eye tracking in the field of literacy remains significant, enticing developers and explorers to continually locate substantial materials towards its optimized implementation. As demonstrated by the GARY (Schiavo et al., 2015) and ZuCo (Hollenstein et al., 2018) applications, one can actually readily witness a variety of eye tracking-specific literacy-promoting methods. As Holsanova (2014) firmly thought, the technology’s current contribution to literacy cannot be dismissed. The visions of Just and Carpenter in 1980 relative to the eye-mind hypothesis paved the way to these innovations, integrating the nuances in different formats throughout its development. On the other hand, its sparsity in its field stands observable despite its prevalence in studies.

Extensive research on eye tracking in traditional settings has produced insightful findings, particularly regarding its impact on multimodal texts (Gegenfurtner et al., 2011; Hoffman, 2016; Huth et al., 2024; Lee and Révész, 2018; Loewen and Inceoglu, 2016). Additionally, the extent of the demographics of the participants includes dyslexics (Hyönä and Olson, 1995; Jones et al., 2008) and children with ADHD (Karatekin and Asarnow, 1998). Focusing on eye movements (Indrarathne and Kormos, 2017; Lee and Jung, 2021), including the dilation of the pupils (Brunyé et al., 2019), and length of gaze on images and texts (Koć-Januchta et al., 2017), this area of study has demonstrated interesting insights especially to helping readers. While present advancements in eye tracking have integrated technological innovations (Krebs et al., 2021; Gu et al., 2022), the speedy development of technology enhancements demands continual findings.

Another gap observed in this field was the lack of studies on long term effects of eye tracking such how repeated exposure to multimodal texts influences memory consolidation and cognitive development, seeking recommendations on pragmatic actions. Pjesivac et al. (2021) and Borkin et al. (2015) concurringly shared vital insights on visual cues but nevertheless maintained a narrow approach on recognition and retention stages. While Coskun and Cagiltay (2022) posited similar findings on the utility of visual cues towards enhanced information recall, the postulations remained unlinked with long term considerations. Even in the area of including audio with text elements, studies regarding this concern seemed lacking. Conklin et al. (2020), Kruger (2012), and Liu et al. (2011) demonstrated the benefits of audio for reading focus, but their studies only covered voice-assisted reading. The field’s exclusive practices, which make it difficult for the discipline to provide long-term considerations in eye tracking research, are implied in this gap.

Studies on eye tracking in a cross-disciplinary lens demonstrated another demand to assist this field gain prominence. Technology was heavily considered in the studies visited (Krebs et al., 2021; Gu et al., 2022; Schiavo et al., 2015; Hollenstein et al., 2018) especially augmented and virtual reality integrations, but current literacy partnerships with psychology seemed ignored. Two of the closest cross-disciplinary findings were the studies concerning participants with ADHD (Karatekin and Asarnow, 1998), and dyslexia (Hyönä and Olson, 1995; Jones et al., 2008), which currently are demanding fresh explorations considering their dates of publication.

The following suggestions are seen to be advantageous to the field of eye tracking research about its application in reading multimodal texts.

There is a need in integrating more innovative applications to eye tracking activities and in reading multimodal texts. Considering the fast pace of the technological landscape in education, rearing today’s children with the most advanced gadgets, latest developments towards multimodal literacy, especially the gamified ones are critical.

The narrow participant demographics in these studies call for expanding research to include mature readers and diverse needs. Similarly, the spectrum in ADHD has developed in recent years, demanding specific interventions and approaches for a particular spectrum type. Moreover, studies concerning children with socio-emotional learning needs should also be given attention in the field of eye tracking research, and how literacy can be addressed among these children especially in reading multimodal texts.

Challenges in AR and VR technologies linked with multimodal text literacy development demand explorations for eye tracking research. Institutions can explore feasible means to alleviate the costs and help communities to enjoy such innovations. Moreover, workshops and trainings may be arranged for utilizing such advancements toward eye tracking research. One may also observe and examine the suggestions of Mayer and Moreno (2010) in decreasing the cognitive processing load that multimedia materials place among learners.

These recommendations may not be exhaustive considering that this is a mini-review. Longer term consolidation of study results through systematic reviews and meta-analyses can provide greater insights into unexplored impact of eye-tracking on literacy development and multimodal reading in the digital era.

This paper synthesizes eye tracking studies to explore how readers interact with multimodal texts. Findings reveal that multimodal reading requires cognitive flexibility since readers navigate interplays of content. Key strategies like prioritizing information based on its perceived importance and confirming text with visual aids are essential for comprehension. However, research gaps persist, particularly in understanding how varied populations, such as those with reading difficulties or non-native language backgrounds, engage with multimodal texts in naturalistic settings. Future research should focus on these areas to enhance instructional designs and embrace the digital evolution of literacy practices.