We first describe those approaches that can be generally applied to image-text pairs regardless of the domain and then discuss news values central to analysing and understanding news articles. The discussion follows a chronological order signifying the evolution of treatments of image-text relations over the years.

Multimodal machine learning: In computational sciences, multimodal learning is a well-established research area whereby meaningful information is extracted from two or more modalities and combined to achieve a specific task. The aim is to reduce the semantic gap between more directly accessible features and semantic interpretations (Smeulders et al., 2000) so that the derived semantics can serve as an effective bridge between contributions made in quite different presentation modalities, such as text and image. Bridging the semantic gap is expected to lead to better performance (often involving prediction) in tasks using multiple modalities instead of just one, e.g., sentiment prediction (Poria et al., 2017) from videos using sequences of image frames and audio. With the recent groundbreaking advances in machine learning and the availability of large multimodal datasets, representation learning models can be trained automatically (Ngiam et al., 2011; Chen et al., 2020; Jia et al., 2021; Radford et al., 2021), and their meaningfulness assessed via standard prediction testing benchmarks. A large body of work exists (Baltrusaitis et al., 2019) targeting areas spanning learning algorithms, the fusion of multiple modalities, evaluation metrics and applications that go beyond engineering to domains like medicine and arts. Some noticeable areas and applications with image and text as modalities that have made significant progress in the last decade are image captioning (Kiros et al., 2014; Karpathy and Fei-Fei, 2015; Hossain et al., 2019), cross-modal retrieval (Socher et al., 2014; Xu et al., 2015; Zhen et al., 2019), and text to image generation (Mansimov et al., 2016; Qiao et al., 2019; Ramesh et al., 2021).

Beyond descriptive and literal relations: In the case of most vision and language model learning and benchmark tasks, the surrounding text is always semantically related at a descriptive and literal level to the image. However, this assumption hardly holds for examples commonly found in online news or advertisements. In such media, the presentation modalities can be weakly linked or abstractly related to one another so as to emotionally engage and influence the reader. Social media offer another example where posts are mostly multimodal, with users uploading billions of photographs¹ with opinions every day on sites such as Facebook and Twitter. Consequently, some researchers (Chen et al., 2013) have investigated Twitter data to distinguish between visually relevant and visually-irrelevant tweets and have built models to predict the two categories. Taking this one step further, Vempala and Preotiuc-Pietro (2019) take inspiration from Marsh and White (2003) and have studied image-text tweets for modeling relations that capture which modality adds information (or not) to the other. There is also work on estimating the semantic correlation (Zhang et al., 2008; Xue et al., 2015) and concept level relatedness between modalities (Yanai and Barnard, 2005) (e.g., between local image regions and words), but researchers have only recently (Chinnappa et al., 2019) started combining inter-modal relationships with computational modeling to investigate relations at a deeper level.

Zhang et al. (2018) focus on investigating non-literal relations between visual and textual persuasion for the automatic analysis of advertisements. They divide the relations into parallel vs. non-parallel, e.g., when the image and text convey the same message independently (but not necessarily exhibiting literal overlap) and when the meaning is ambiguous or incoherent if the image and text are combined, respectively. However, one difficulty with communication science taxonomies is that their level of detail sometimes makes it difficult to assign a particular class to an image-text pair, especially for an untrained analyst. Recently, to address such issues, both Kruk et al. (2019) and Otto et al. (2019b) have extensively combined image-text taxonomies from semiotics with metrics from computational science and proposed interpretable and computable categories for image-text relations. Inspired from semiotics, Kruk et al. (2019) focused on investigating Instagram posts from three perspectives: author intent, contextual or literal meaning overlap, and signified meanings (meaning multiplication) between the image and text. Their contextual and signified relations are inspired by the semiotically-inflected proposals of Kloepfer (1976) and Marsh and White (2003), respectively. The authors report noticeable gains in multimodal intent classification when the interpretation of image and text diverges. Alikhani et al. (2020) introduce a new dataset and study different types of cross-modal coherence relations (such as subjective, story, meta) between image-text pairs. Most recently, Utescher and Zarrieß (2021) studied types of referential relations between image and text in long text documents such as Wikipedia articles and posed a new research direction for vision and language modeling frameworks. Similarly, Sosea et al. (2021) show the use case of modeling image-text relationships (unrelated, similar, complementary) to improve multimodal disaster tweet classification.

Image-text computational metrics: Whereas the approaches above generally attempt to provide classification in terms of text-image relations directly, it has also been proposed that more widely applicable and accurate characterizations might be gained by drawing on more robust metrics derived from the text, image, and their inter-relations from which more specific text-image relations can be derived. Henning and Ewerth (2018), for example, propose two metrics to characterize image-text relations: cross-modal mutual information (CMI) and semantic correlation (SC). For a given image-text pair, CMI (with values ranging between 0 and 1) measures the number of shared real-world objects, entities and concepts, while SC (ranging between -1 and 1) measures how much meaning is shared between the two modalities. In contrast to the common assumptions (Parekh et al., 2021) made in several works including image captioning and image-text synthesis where image and text are always semantically related, the semantic correlation here can also be negative, which means relations where the overall meaning is incoherent can also be captured. Similarly, Otto et al. (2019a) propose a further metric called the abstractness level (ABS), which measures whether the image is an abstraction of the text or vice versa.

Otto et al. (2019b) also propose a taxonomy of eight image-text relations derived from Barthes (1977) and Martinec and Salway (2005), mapping the three metrics SC, CMI and STATUS to distinctive image-text classes. While CMI and SC measure the information and meaning overlap, STATUS, as introduced in Section 2.1.1, measures the relative importance of an image and text. In addition to the three positively correlated categories termed as complementary, illustration and anchorage, there are three negatively correlated (contrasting, bad illustration, bad anchorage), and two image-text relations with no information overlap (interdependent, uncorrelated), which may be deployed in domains such as advertisements and social media to catch the user's attention.

Computational news values and analysis: Recent efforts toward estimating and extracting news values have focused almost exclusively on the text. Potts et al. (2015) investigates linguistic techniques such as tagged lemma frequencies, collocation, part-of-speech tags, and semantic tagging to extract news values. The study used a 36-million-word corpus of news reporting on Hurricane Katrina to explore the usefulness of computer-based methods. Bednarek et al. (2021) conduct a similar empirical study of news articles concerning the Australian national holiday “Australia Day” in the Australian press. In addition, di Buono et al. (2017) and Piotrkowicz et al. (2017) focused on developing fully automatic methods for extracting and classifying news values from headline text. di Buono et al. (2017) relied on news value specific statistical features and Piotrkowicz et al. (2017) additionally experimented with word embeddings and emotion labels to train machine learning algorithms such as Support Vector Machines (SVN: Cortes and Vapnik, 1995) and Convolutional Neural Networks (CNN: LeCun et al., 1998). Similarly, Belyaeva et al. (2018) address the problem of estimating news values like frequency, threshold, and proximity by applying various text mining methods.

For news media research in general, Motta et al. (2020) describe a framework of newsworthy aspects (called news angles) and a data schema to bridge the gap between formal journalism literature and computational capabilities (named entity recognition, temporal reasoning, opinion mining, and statistical analysis). News angles, in addition to signifying the newsworthiness of an event, also provide a predefined theme/structure to report the event. Similarly, some very recent works address the use of technology toward supporting journalistic practices using semi-automated news discovery tools (Diakopoulos et al., 2021), discuss responsible media technology use for personalized media experiences and news recommendation with automatic media content creation (Trattner et al., 2021), and propose the development of AI-augmented software to support citizen involvement in local journalism (Tessem et al., 2022). All these works position the use of recent breakthroughs in image recognition, natural language processing and other AI technologies for automated or semi-automated news discovery, content creation, and improving user interaction and accessibility. In addition, problems of bias in automated systems, fake news, privacy, news literacy and societal challenges are also discussed and considered important challenges in the ever-changing landscape of technology and information accessibility. However, the role of news angles and news in general in the context of modalities other than text has yet to be explored or discussed.

As briefly described before, Author Intent represents the author's main purpose behind the news article. Recently, Kruk et al. (2019) propose an intent taxonomy for Instagram posts centered around the presentation of self (Hogan, 2010; Mahoney et al., 2016). They propose eight categories, which are: advocative, promotive, exhibitionist, expressive, informative, entertainment, provocative:discrimination, and provocative:controversial. While independent journalists' and authors' intent can be placed under some of these categories, we are more interested in intent arising as a combination of author, the idea of the article and the publisher. Thus, the categories that strictly represent presentation of self (exhibitionist, expressive) are in general less significant intents for news. Categories under provocation, on the other hand, are related to hate-speech and propaganda, which is certainly an interesting area on its own right in computational analytics.

Building on this, we propose a more structured two-level taxonomy for capturing author intent. Informative, Entertaining and Persuasive are the top-level categories. Informative means that the author's main purpose is to inform the reader about news via unbiased reporting of facts and information. The articles under this category are descriptive and expository focused on details, facts, eye-witness accounts and opinions/arguments of others (such as public figures or people in general) and linguistically distinguishable as aligning along some of the distinct dimensions of registerial variation described in detail by Biber (1988). Entertaining articles, in contrast, refer to satirical or spoof pieces that often take a humorous jab at organizations or people that hold power. Finally, Persuasive refers to an author supporting or refuting a policy, person, organization or broadly an idea in the article via personal opinions, interpretations, point of view or judgements. Not all the intent categories are mutually exclusive, and so an article can have more than one author intent. These diverse properties naturally suggest that corresponding computational language models differentiate among the text types and registers represented.

Developing the account further, Informative is then divided into two categories:

Entertaining news articles rely heavily on irony and humor, and it is almost impossible to mistake them as serious news. This is different from fake news, which intentionally misleads people into believing that the story is real (Golbeck et al., 2018). The articles here mimic real news but still cue the reader that they should not be taken seriously. Examples of this category are news articles from satire websites like thespoof.com, thecivilian.co.nz and thedailymash.co.uk. Two examples from such websites are shown in Figure 4, where highlighted text shows informal language that is not used in actual news articles. Interestingly, the images used in these examples are neutral and do not reflect the entertaining intent of the article.

Persuasive articles are written with the purpose to persuade and convince the readers to consider or accept author's opinion, position and point of view. Opinion pieces and articles fall under this category. It is also a possibility that an article holds no position and so only falls under the informative category. If not, persuasive is divided into two opposing sub-categories: Promote and Demote, defined with respect to the core theme or idea of the article. Promote refers to when an author aligns with or supports a policy, person or organization and, in so doing, aims to persuade the reader to do the same. Demote is the converse of this. We show two examples of such news articles in Figure 5, where article (a) is optimistic about the Internet for opinion journalism, while article (b) is about press culture in the US and the lack of gender diversity. Additionally, use of harsh language or tonality, attacks toward someone, presence of controversy and propaganda are all possibilities within such articles. These establish important targets for further research but are not yet explicitly categorized under our taxonomy.

Our proposed taxonomy of cross-modal relations (last column) is inspired from multiple taxonomies. Table 1 shows a comparison of taxonomies proposed in the computational science literature that draw on semiotics. The top three rows in the table represent relations where the image and text are coherent and belong together, and relations in the next three rows are where the image and text differ in meaning and are incoherent. While the top three relations can be represented in both Kruk et al. (2019) and Vempala and Preotiuc-Pietro (2019), the differences between relations based on modality's importance cannot be established uniquely. Moreover, Vempala and Preotiuc-Pietro (2019) cannot represent any of the relations where either image and text are incoherent or related on an abstract level (row 7 in Table 1) without sharing any information. Whereas, Kruk et al. (2019) can represent the incoherent relations, all ambiguously under one combination of its contextual and semiotic classes. As our relation taxonomy is inspired from Otto et al. (2019b), all the relations in the leftmost column can be uniquely represented by our taxonomy (rightmost column). In addition, we have extensions of CMI and SC to explicitly model the presence of additional information and fine-grained relations, respectively.

We suggest additions and modifications for object- and semantic-level relations along with news-centric attributes for richer news analysis. We define these relations building on measures established in previous work. However, while relations in other taxonomies can be broken down (sometimes not distinctively) at conceptual, semantic and relevance levels, our framework is more fine-grained. Furthermore, our framework can be applied to any image-text pair, while at the same time offering relevant relations and attributes specifically for news analysis.

There are then two types of high-level cross-modal relations in the framework, Objective relations and Semantic relations, which are explained in detail in following two Sections 3.2.1 and 3.2.2.

In this section, we modify and expand the news values defined by Caple (2013), Caple and Bednarek (2016), and Caple et al. (2020), as they define news values for both image and text in a discursive and content-specific manner, which is interesting and relevant from a multimodal news analytics perspective. We derive new sub-classes for some of the news values to make them more concrete, discrete or meaningful. In our framework, any news value may have the same or different sub-categories for the image and text, which adds another dimension for comparing both modalities. Having said that, where needed both modalities can be used to estimate a single multimodal news value with respect to the news article (explained in each news value). We discuss each news value and its realizations in each modality, define scope and list visual and verbal cues essential to identify them in image and text. In previous work, news values are indicated by their presence or absence, which we also follow for most news values. We leave out the news value Aesthetics for two reasons: first, it is only defined for the image, and second, it is a subjective-only and not concretely defined news value. This makes categorizing an image into non-aesthetic vs. aesthetic from only the perspective content a difficult, and probably inappropriate task. In the following sub-sections, we list the news values (see Figure 9) in our framework and their descriptions from the content-only perspective, which can differ from the user-dependent news values covered in subjective interpretation in Section 3.4.

We provide five examples of news articles that are categorized with the respective Proximity, Timeliness, and Eliteness categories in Figure 10; five examples of news articles for which Sentiment, Superlativeness, and Impact news values are identified in Figure 11; and five examples of news articles that are categorized with Personalization, Novelty, and Consonance news values in Figure 12. Each sample is matched with a corresponding news value and its sub-category and marked with the modality (I: Image, T: Text) that provides the essential cues.

Recipients perceives information in the news differently depending on their background, knowledge and experience. This source of variation is far from arbitrary, however. With multiple presentation modalities such as image and text, different kinds of objective and semantic relations can be established between modalities from the available content and information, as explained in detail above. But, in addition, the differing sets of traits, such as age, gender, education, reader's location, language, domains of interest, religion, click history, etc., attributable to each recipient (reader) induce further changes in relations when contrasted with the content-based perspective. This change in objective and semantic relations, as well as some news values, is called Subjective Interpretation. One use case for this kind of analysis is reader-specific studies based on demographics and how the information composed using presentation modalities varies from the intended purpose. From a media studies perspective, given the cross-modal relations extracted from a news publisher's corpus and its reader base characteristics, one can analyse links between the content used for news vs the reader demographics. For instance, if certain image-text compositions are more prevalent for one type of readers' demographics or characteristics than the other.

The analysis can be further refined based on certain news values that are highly reader-centric and exhibit a different value from the reader's perspective. All news values except timeliness and superlativeness are constructed in the view of the target audience. For example, consider a news article about Germany winning a football match against France. The content would be mostly about how well Germany played, won the game, and construct the story with certain statistics about the game. Even though the article has a positive connotation, a French team supporter living in either Germany or France would probably perceive it differently from German football fans or even perceive it negatively. Here, we describe how news values may be varied from a recipient's perspective:

From the computational perspective, we have three kinds of resources here, first is cross-modal relations or news values based on content, second are reader characteristics, and lastly the reader-centric news values. Treating two as inputs and one as the output in a predictive task, we can consider these problems under user modeling to predict readers' preferences, readers' characteristics or improvise information dissemination (language and image usage).

In this section, we discuss some illustrative articles from different news media channels. We particularly pay attention to author intent, cross-modal relations and news values in order to show the necessity of adequately reflecting the sometimes quite complex interplay between these different aspects. In the previous sections, we presented examples of each aspect with pairs of image-short text (headline or caption). Here, we extend the focus to include an article's components, including the headline, image captions and some parts of the main text. Structural analysis of this kind has also been performed and shown specifically for news values by Caple and Bednarek (2017), enabling the comparison and interaction of news values across an entire article. Now, we show how it is also beneficial to perform analyses that incorporate our broader set of perspectives that go beyond news values alone.

In Figure 13, we present four articles for comparison, all related to COVID-19 and taken from top-read daily news media channels in the United Kingdom. In the top row (Figure 13A), we see two articles from dailymail.co.uk (a) and (b); bottom left an article from the guardian.com (Figure 13C); and bottom right an article from the mirror.co.uk (Figure 13D). The articles in each column are drawn from the same dates in March and September 2022, respectively. The left column articles are about the “rise in weekly COVID-19 cases by 50%”, and the right column articles are about “WHO's announcement or statement about the end of the COVID-19 pandemic”. In the following, we discuss differences and similarities across the three media channels drawing on the perspectives defined in our analytic framework.

Starting with the two Daily Mail articles, a common observation is the use of images and text to provide both a supporting and a contrasting viewpoint with respect to the headline. In the left article,² several graph images (three with black background, one shown in Figure 13A) are used without direct references in the text and no captions (Uncorrelated OO and no extension). Here, the text comes to the same conclusion (Equal STATUS and there is no modification) evident in the image (high-positive SC). These three images and the supporting texts followed by the author's own opinions (sometimes referring to theories behind the increase without citing sources) promote (persuasive author intent) the main theme of “rise of infections”. Conversely, three more graph images are included in the article to provide a different viewpoint on “covid being less deadly than the flu” and “covid impact on rise in the waitlist of routine treatments”. One of these images (shown in Figure 13A) has a caption which refers (Image STATUS) to the image (has a common entity “NHS”, partial OO), summarizes the graph (high-positive SC), and adds more entities (Text extends Image). The image, on the other hand, amplifies (amplification) the “one in nine people waiting” messaging by showing the difference in people on the waitlist now vs. pre-covid times. Interestingly, this image caption is not discussed and referenced anywhere in the main article but instead in a subordinated text fragment embedded within the article, which reflects the complex structure used by the Daily Mail. In addition, another image (not shown in Figure 13A), which also has a caption, exhibits similar image-text relations, presenting in detail opinions from the author and quoted excerpts from cited experts (informative, reports facts, and opinions). In summary, the majority of the article has image-text pairs that exhibit different relations as in the beginning of the article, and promotes (persuasive author intent) the “living with covid” perspective, in contrast to the headline (demotes, persuasive author intent). This demonstrates clearly how important it is that any analysis is capable both of differentiating finer structural units between which relations can hold and of allowing those relations to be divergent.

The article also constructs distinct news values of superlativeness (large numbers, rise of cases and waitlist), impact (infection rise, deaths, routine treatments), proximity (national-UK and international-European), timeliness (recent, ongoing, and old), sentiment (positive and negative), and eliteness (NHS, other organizations, experts). As with the image-text relations, here there is variation over the extent of the article of news values as well as between the image and text. For instance, the main article text constructs the news value of timeliness as ongoing, whereas images explicitly construct recent and old by showing years as far back as 2007. Similarly, superlativeness (from thousands of cases to millions of people on the waiting list), impact and sentiment (from negative-rise in cases to positive-covid less deadly than flu to again negative-rise in wait list) are constructed in different ways across the extent of the article.

In a similar manner, the article³ on the right, first, primarily reports (informative, facts and opinions) regarding the statement given by WHO and, later, provides a contrasting viewpoint on the long period lockdowns in China and their impact on residents. Two out of three images (one shown on the left of Figure 13B) are stock photographs (partial OO and low-positive SC and no modification) with captions providing the main message (Text STATUS) and additional entities (Text extends Image). A notable difference from the other Daily Mail article is the absence of the connection to the rest of the main article text, except for the news value of eliteness via images. This could be because the first article is written by a health and science editor for an in-depth analysis. The third image (shown on right in Figure 13B) depicts the situation in the caption (high OO and high-positive SC) and exhibits the same relations as the previous two image-caption pairs. In addition, the caption here exhibits the different news value of timeliness (June) in comparison to the lockdown news (from September) from China. Furthermore, similar to the other article, the later text on the lockdown in China consists of the author's claims and opinions without concrete source references or quotes from experts. In summary, several distinct news values are constructed “simultaneously” in the article: eliteness via both image and text (Director-General and WHO), novelty (lowest figures) via text, international proximity (U.S., England, China, World) in text, positive high impact (lowest figures, end of pandemic) and negative medium impact (resident's condition in a Chinese city), and lastly sentiment (from positive-lowest cases worldwide to negative-hunger, forced quarantines in China).

In contrast to the Daily Mail's article on the left, The Guardian⁴ article (also written by a science editor) covers the news on the rise in cases and its impact on people aged 55 and above. The article focuses on facts from government reports and figures, scientific reports and opinions of five different experts (informative, facts, and opinions). Moreover, the structure of the text (paragraphs instead of sentences) and the use of images is quite different from the Daily Mail's article. For example, the left image is not referenced in the main text, but the image-caption pair is relevant for the construction of news values of personalization and impact, resulting in high-positive SC but no modification with the caption and some of the text snippets in the main text. In addition, the image-caption pair has a partial OO and both extend in terms of new information, with the text as the main message (Text STATUS) and constructing news value of local proximity (Dedworth, Berkshire), timeliness as ongoing and future, and eliteness (NHS, NHS personnel). Conversely, the second graph image from gov.uk acts as a placeholder image (in this case, of COVID-19 patients in hospitals) and is not referenced in the main text; it is then only weakly linked (low-positive SC) to show the rise in cases and hospitalizations (quality modification, not matching the figures). Such image-text relations consequently suggest a sub-optimal combination of image and text in terms of information dissemination. In addition, several other news values are constructed in the rest of the text: sentiment as mostly negative, superlativeness with large numbers, high negative impact for the nation's elderly, local and national proximity (Berkshire and England), timeliness as recent, ongoing, and future, and eliteness (institutes, experts, government, and NHS).

Lastly, the Daily Mirror⁵ article shows the most contrasting use of images with respect to the article and the main text. In three (two shown in Figure 13D) out of five images in the article, the news value of personalization (focus on laypersons) is constructed. The article's main text stands in contrast to the Daily Mail article in that most of the focus is on disseminating the WHO's announcement and information from documents shared by the WHO (informative, facts, and opinions). Moreover, the rest of the text only points to vaccine strength, decreasing the number of cases, and the alert levels in the United Kingdom. In contrast, two image-caption pairs (one shown on the left in Figure 13D) exhibit negative SC and quality modification (image showing people wearing masks vs. mask mandate ending in February in text). In terms of objective relations, there is partial OO and Image extends Text. This could be because of either the use of older dated image or to signify that people are still cautious in public places, and so serves to persuade the reader to be cautious (persuasive, promote the “cautious perspective” or demote the “end of covid” message). In addition, the news value of superlativeness is constructed with multiple people in the image to emphasize this cautiousness aspect. This is a notable difference between the Daily Mail and the Daily Mirror articles: both are informative and persuasive, but compose their articles via different usages of text, image-caption pairs and news values. While the Daily Mail relies on expanding proximity (to China's condition) and high-positive SC image-text pairs, the Daily Mirror shows negative SC image-text pairs and news value of personalization to contrast with the central theme of the article.

Author intent: Previous work related to author intent tackles the problem of identifying news (reporting of factual information) vs. opinion (Krüger et al., 2017), the persuasive effect of news editorials (Baff et al., 2020) and argumentation strategies in news (Khatib et al., 2017). These earlier approaches rely on engineered lexical features which are not robust against changes in topics and not generalisable to unseen (during training) news publishers. Recently, Alhindi et al. (2020) combine argumentation features and computational language models like BERT⁶ (Devlin et al., 2019) to identify news vs. opinion. They show promising performance in comparison to previous approaches across datasets and publishers. For multimodal author intent detection, Kruk et al. (2019) benchmark several large multimodal model features to classify author intent from Instagram posts. The recent work suggests that a combination of unique lexical and multimodal features can be used to classify author intent classes as described in our framework.

Cross-modal relations: There is a wide body of work on vision-language modeling (Gan et al., 2022), where the goal is to learn visual and multimodal features from a large-scale corpus of image-text pairs from the Web. These models often have deep network backbones like BERT (Devlin et al., 2019) and are trained with loss functions such as masked language modeling, masked image region prediction and image-text matching. Recent work on image-text matching and retrieval (Cao et al., 2022) has explicitly focused on refining cross-attention mechanism and local alignment to get better retrieval performance. Consequently, these models focus more toward aligning literal (objective relations) relations than abstract and high-level semantic relations (semantic correlation STATUS, modification). Recent work (Xu et al., 2022) suggests that weakly aligned (e.g., low-positive SC) and unaligned (e.g., zero, negative SC) pre-training can be beneficial and is still understudied. On the contrary, recent work such as Henning and Ewerth (2018) and Otto et al. (2020) have tailored multimodal neural networks in order to predict relationship metrics like CMI, SC, and STATUS. Müller-Budack et al. (2021b) have taken a different approach of predicting image-text consistency in news over named entities like persons, locations and events, using several supervised pre-trained model features for face recognition, geo-location estimation and event detection.

News values: For all the news values except sentiment, parts of speech tagging (Chiche and Yitagesu, 2022) and named entity recognition and linking (Luo et al., 2015) can identify words and phrases for each news value. With the identified words, one can either create a rule-based classifier to classify the news values or train a data-driven classifier. In addition, temporal reasoning and extracting temporal and causal relations can be beneficial to identify the impact and novelty of the event in news (Caselli and Vossen, 2017). Both lexical and neural network-based classifiers are well explored for sentiment and can be used to detect sentiment in news (Mello et al., 2022). Moreover, convolutional neural networks (CNN) and multimodal neural networks can also be applied for visual (Ortis et al., 2020) and multimodal (Soleymani et al., 2017) sentiment analysis. Similarly, topic modeling can be performed using recent state-of-the-art approaches like BERTopic (Grootendorst, 2022), which combines word embeddings with TF-IDF (Term Frequency-Inverse Document Frequency) to create topic representations. Recently, some works have explored deep networks for image understanding in combination with text-based topic detection for news videos (Li et al., 2017) and Twitter posts (Zhang et al., 2019). For images, while there is ongoing research for tasks related to some news values, others, like stereotype identification and novelty detection, are still yet to be explored. For other news values, such as identifying location, visual cultural symbols and identifying objects for more context, open vocabulary multimodal models like CLIP (Contrastive Language Image Pre-training) (Radford et al., 2021) can be used to detect the presence of things of interest. Some news values require face recognition (Zhu et al., 2021) and facial analysis (Li and Deng, 2022), which can be efficiently done with the use of CNN-based face detection and facial expression prediction. For identifying the impact of events in images, deep networks can be applied for a disaster impact assessment on aerial imagery (Gupta et al., 2021) and image-text pairs of tweets (Rizk et al., 2019). Lastly, there are a few works where a BERT-based model has been used for detecting racist (Fokkens et al., 2018), gender (Chiril et al., 2021) and immigrant (Sánchez-Junquera et al., 2021) stereotypes in news, tweets and political debates respectively. As each news value is a separate task, an ensemble or a rule-based model using different features and labels extracted from the discussed models could also be explored in a multi-task and multimodal learning setting.

As the computational modeling of cross-modal relations is an active research area (see Section 2.2), both the objective and the semantic relations are interesting. For example, they can serve as the basis for creating novel datasets for news analysis for multimodal learning tasks to foster further research in the area. Recently, there have been attempts to create datasets for non-literal relations between image and text in advertisements [e.g., Zhang et al. (2018)], tweets [e.g., Vempala and Preotiuc-Pietro (2019)], Instagram posts (Kruk et al., 2019), and across domains (Otto et al., 2019b). However, only Otto et al. (2019b) decouple relations at conceptual and meaning level motivated from semiotics and none of them are in the news domain. Recent work by Alikhani et al. (2020), Sosea et al. (2021), and Utescher and Zarrieß (2021) have posed new research directions and interesting applications of modeling a variety of image-text relations. From the perspective of news values, recent work on automatic extraction of news values from news text (di Buono et al., 2017; Piotrkowicz et al., 2017; Belyaeva et al., 2018) can be extended to both images and text with novel datasets and multimodal modeling contributions. The challenge of detecting news values in both image and text can be at the level of just predicting the type and class of news value to detecting spans of text and regions in the image signifying the news value. From the user modeling perspective, there has been much work on news recommendation based on Twitter user profiles (Abel et al., 2011, 2013), user graphs (Wu et al., 2021), click patterns and behaviors (Wu et al., 2019, 2020) and multimodality (Wu et al., 2022b). Wu et al. (2022a) present a detailed survey of personalized news recommendation and organize relevant literature by the type of news content, properties (like a publisher), dynamic information (like popularity, click-through rate) and user features (like user networks). Our framework includes the recipients' perspectives in the news process and provides a combined view with news values where contributions can be made from both the content and user modeling for further research in the area.

Modeling semiotic relations at the conceptual and meaning level for news can be important for news-specific downstream tasks like the detection of out-of-context images (Aneja et al., 2021; Luo et al., 2021) or fake news (Giachanou et al., 2020; Singh and Sharma, 2021). Both applications can benefit from predicting additional attributes relevant to news dissemination when coupled with news values from a multimodal perspective. For instance, Springstein et al. (2021) present an application where cross-modal consistency between text and photo is checked for entity types persons, locations and events. Additional attributes like news values can be added as consistency parameters to enrich existing models. Accurate identification of news values in text and images can also help estimate the newsworthiness of informal news on social media. As mentioned before, the news recommendation system is an interesting application that can be enriched with predictable news values from the content combined with available user traits and click patterns. News retrieval (Tahmasebzadeh et al., 2021) can be enriched based on image-text relations and news values. For instance, news can be retrieved and ranked higher of a particular event where images are relevant (high objective overlap and semantic correlation) to the news text and includes personal stories or eyewitness accounts (personalization).

From the social science perspective, some of the models and applications discussed above enable news analysis, user studies and news comparisons. For instance, the automatic identification of news values in text and image can aid in studying narrative differences between mainstream news and news on social/alternative media. In terms of news comparison and supporting discourse analysis, Pollak et al. (2011) use text mining to find contrasting patterns between coverage of “2007 Kenyan elections and post-election crisis” in local (Kenyan) and Western (British and US) newspapers. As presented in the framework, fine-grained relations between image and text coupled with news values can provide a multimodal perspective of image and text usage in important events (like an election crisis) across news publishing platforms. Recently, O'Halloran et al. (2021) have introduced a platform called Multimodal Analysis Platform (MAP) for searching, storing and analysing multimodal content (text, images, and videos) in online social and news media. However, the tool focuses on extracting information from individual modalities and so far does not integrate this as would be necessary to operationalize meaningful search and descriptive analysis of meanings emerging from combinations of modalities. The integration of analytical capabilities of image-text relations and news values would therefore be an important additions for such a tool in order to perform analyses going beyond the descriptive aspects.

In this paper, we have reviewed multimodal semiotics and computational science literature to derive a framework of image-text relations, news values, and author intent for the support of richer multimodal news analysis. The presented framework covers the entire spectrum of the news process, from news production to news consumption. Taking inspiration from semiotics as well as recent work in the fields of multimedia and multimodal machine learning on relations between image and text, we outlined a framework based on objective information (objective relations), meaning formation (semantic relations) and relative relevance (STATUS) between the two presentation modalities. We also introduced a new relation called modification as a sub-category of semantic relations better to capture the role of images in news dissemination. Further, for news-specific analysis, we introduced the aspect of author intent to identify the author's primary purpose behind an article and news values. While there have been a few attempts at automatically detecting some news values from text and particularly news headlines, image and text together have yet to be considered in computational research, so this is an upcoming specific research challenge. Finally, we included subjective interpretation, which captures the end user's and reader's subjectivity toward presentation modalities and news value aspects. Relevant examples for each part of the framework and comprehensive comparisons with existing work were also presented. Finally, we concluded the paper with possible research directions, applications and relevance of the work from a social science perspective.