The advent of large language models (LLMs) has enabled significant progress in the field of natural language processing (NLP) and has helped provide promising results for various tasks (Brown et al., 2020). Many types of LLM have been proposed to solve both language-specific and domain-specific tasks (Lewis et al., 2019; Chung et al., 2022; Touvron et al., 2023). However, LLMs primarily favored well-resourced language with large updated training corpora, which may lead to hallucination problems, especially in lower-resource languages, in which the training corpora are not abundantly available (Ji et al., 2023). Extracting knowledge from these low-resource languages is not only beneficial as it helps include more available data. It could also provide deeper insight into the model's behavior across linguistic variations. To mitigate the hallucination problem, researchers have explored augmenting LLMs with external structured data sources, such as knowledge graphs (Guu et al., 2020; Asai et al., 2021; Mialon et al., 2023). Integrating structured information graphs with the LMs has been one of the common approaches (Yao et al., 2019; Kang et al., 2022), as graphs can be constructed in a domain-specific fashion, such as finance (Yang et al., 2018; Elhammadi et al., 2020).

Event graphs, which store event information from unstructured plain texts that describe “who, when, where, what, why” and “how” of the action, can provide a simplified version of a more generalized knowledge graph (Xiang and Wang, 2019; Li et al., 2022). Focusing on event extraction is particularly promising for enhancing NLP in low-resource settings because it involves parsing relationships within the narrow scope of particular events, thus requiring less extensive linguistic understanding for the model. Although close-domain event extraction, which follows specific domain schema, may provide better results in downstream retrieval tasks (Chambers et al., 2014; Björne and Salakoski, 2018; Han et al., 2018), this specialization often results in complex annotation systems that can be cumbersome and domain-restrictive, especially for low-resource languages. Moreover, while the use of additional syntactic information for extraction tasks has been studied in English (Fader et al., 2011; Wang C. et al., 2023; Wang Z. et al., 2023), it remains under-explored in low-resource languages.

In this work, we propose a methodology that streamlines the process for open-domain event extraction for corporate documents written in Thai and demonstrates its utility in a downstream task. Our guideline aims to make structured information extraction more accessible, by reducing the complexity of the annotation process. We also utilize Universal Dependencies [UD; Nivre et al., 2016] during the pre-training step to help the extraction model better understand the structural information of the sentences.

The main contributions of this work are as follows:

The rest of this paper is organized as follows. Section 2 analyzes previous work. Section 3 describes our methodology. Section 4 reports on our experiments. Section 5 provides a discussion of the results. Section 6 elaborates on the application of the event graphs. Section 7 concludes with a summary. By simplifying the initial extraction process, our method could allow for a more straightforward transition into an extraction task for other types of relations, such as, part-of or causal relations, which often require a deeper understanding of the interconnectedness of entities beyond their basic semantic relationships.

In this section, the background of the paper is explained along with literature reviews, outlining the previous work on event extraction, and Universal Dependencies. First, the definition and prior works of event extraction are explained. Second, the Universal Dependencies are described including the definition and its advantages.

This section outlines the annotation process and the event extraction models used in this work.

In this section, we compare the annotation time between event annotation using our proposed guideline and the traditional UD annotation. The annotated data was then used in a comparative study between different approaches to event extraction tasks.

Compared to the baseline UD tagging, event annotation following our guideline is substantially faster. The decrease can be attributed not only to the fewer number of relations but also to the less complex annotation scheme that the annotators need to process. Annotating using our proposed guideline mostly follows the semantic structure of the sentence, eliminating the need to recognize minor syntactic relations like “case,” or “disclose.” The more complicated relations between clauses like “acl,” “advcl,” “csubj,” or “xcomp” are also omitted. In addition, event annotation treats multiple-word phrases as single units, eliminating the need to understand the intraterm connection. As a result, when developing the data for structural information extraction models, starting from semantic relations similar to the proposed event extraction could be more practical and time-efficient, especially for languages with no pre-existing structural training data.

From the subsequent span-based classification result, the model using language-specific pretraining outperforms the multilingual one in both entity and relation extraction, likely attributed to both language-specific and task-specific fine-tuning. Previous work has reported that using multilingual BERT performs substantially worse for low-resource languages, like Thai, as it does not benefit from cross-lingual transfer (Wu and Dredze, 2020) and shows that monolingual BERT-based models perform even worse for NER, POS, DEP tagging. In our case, we show that fine-tuning using task- and language-specific data offers an option to improve upon the monolingual BERT-based models.

When comparing the models in different settings, although the generative model with multilingual pretraining outperforms most of the span-based ones, it still lags behind the monolingual SpERT on the relation extraction task. This discrepancy is likely because the entity recognition task can benefit from the encoder-decoder architecture used in this work. A similar observation has also been previously reported (Wu et al., 2023). Nevertheless, specific downstream tasks must be taken into account when selecting candidate baseline models, as other types, such as masked LMs, could be computationally cheaper for domain-specific training.

In contrast to entity extraction, the relation extraction task could benefit more from the span-based two-step classification architecture. While SpERT inherently approaches relation extraction as a direct classification task, the generative-based method necessitates the simultaneous learning of relation generation with the identification of the entities of interest.

Lastly, when UD is included during the pre- training stage, the generative model outperforms in both tasks. Using UD information allows the model to learn the syntactic structure of the language, potentially aiding in the semantic inference of the subsequent relation extraction.

This result motivates the use of UD in conjunction with a more simplified event annotation framework when developing models for structure extraction, especially for low-resource languages. Although UD annotation is substantially more time-consuming, our work shows that including such information is likely beneficial to the subsequent semantic-related tasks.

After obtaining the list of event attributes from the event extraction model, these sets of structured event information can be adopted to enhance other downstream tasks. In this section, we demonstrate the application of the extracted event graph to improve node classification in the retail banking product domain. Additionally, we explore the potential of transforming our event graph into a more generic knowledge graph where the types of relations are not constrained to only those present in our event annotation guideline.

The event graph in this experiment was constructed from the list of event triplets extracted using the UD-pretrained model from a set of 6,024 internal documents written in Thai, describing the details of financial products and services. This results in 69,801 nodes and 168,964 relations. Out of the total entity nodes, 500 nodes were selected and labeled into one of the 15 categories: “Process,” “Debit,” “Credit,” “Loan,” “Service,” “Promotion,” “System,” “Right,” “Fee,” “Insurance,” “Document,” “Contact,” “Account,” “Statement,” and “RewardPoint.” These nodes were selected such that the resulting 500-node sub-graphs were sufficiently connected (no disconnected graphs), and the numbers of each label were balanced. The averaged F1-score of 5-fold cross-validation of these 500-node sub-graph was then used to assess the performance of the model.

In the baseline model, only the text embedding derived from a pre-trained Thai language model, Wangchanberta (Lowphansirikul et al., 2021), was used. For our model, the node embedding derived from the event graph using Hash-GNN (Tan et al., 2020) was concatenated with the original text embedding as an additional feature.

Table 3 shows the averaged F1 scores of the model using text embedding or text+node embedding as features. The result shows an ~2 percentage point improvement (77.71% from 75.87%) when the model uses node embedding in conjunction with text embedding. This improvement underscores the significance of the relational information provided by our event graph using the simple Hash-GNN. To achieve further improvement, one could employ more advanced (though computationally more expensive) node embedding techniques, namely, GCN (Kipf and Welling, 2017) or GAN (Veličković et al., 2018). In addition to the improved performance, our node classification approach adaptable to other domains and can assist organizations in processing large textual data. A similar technique could be employed to categorize entity names present in internal documents, by labeling small subset samples and then using a classification model with the extracted event graph to incorporate contextual information.

Moreover, our extracted event graph can also be merged and reformatted to construct a more generic knowledge graph. Briefly, the procedure involves finding a pair of triplets such that the head entity of one pair is the same as the tail entity of the other pair. For example, the sentence “A criminal, previously exorenated, stole a car” would be converted into {subj, rel, obj} = {A criminal, stole, a car}. By merging the triplets afterward, the model is allowed to be trained under the constraint of recognizing only seven predefined relation types, yet allowing the extracted triplets to be rearranged to cover more generalized relations. Such a generalized knowledge graph can then be applied to assist in other domain-specific or language-specific information retrieval tasks, such as question answering on knowledge graphs (KGQA; Khongcharoen et al., 2022), or KG-enhanced LLMs (Pan et al., 2023).

In this paper, we introduced a streamlined event annotation framework that allows for substantially faster labeling over the baseline UD tagging. We propose that initiating the development of data for structural information extraction models with simple semantic relations, akin to event extraction, proves more practical, particularly for languages with no pre-existing structural training data.

Language-specific pretraining helps achieve better performance over the multilingual counterparts in both entity and relation extraction tasks. Notably, we underscored the importance of fine-tuning using task- and language-specific data to improve upon monolingual BERT-based models.

Under different settings, while the generative model with multilingual pretraining generally performs well, the span-based two-step classification architecture of SpERT shows a particular advantage for relation extraction tasks. The integration of UD information during the pre-training stage further improved the performance in both tasks, indicating a potential synergistic relationship between syntactic structure understanding and subsequent semantic inference.

Moreover, we leveraged the structured event information obtained from the event extraction model to improve node classification in the retail banking product domain. We also proposed a simple method for converting our event graph into a more generic knowledge graph that expands beyond our event relation types.

In conclusion, our research underscores the value of semantic-based event extraction, language-specific pretraining, and the integration of syntactic structure understanding through UD for improved performance in structural information extraction tasks. The methods we propose are not only efficient but also versatile, with potential applications in other domains, especially for developing similar structural training data for low-resource languages.

The data supporting the conclusions of this article will be made available by the authors, upon reasonable request.

CS: Formal analysis, Investigation, Methodology, Validation, Writing – original draft. AT: Data curation, Formal analysis, Writing – original draft. TA: Data curation, Investigation, Writing – original draft. TC: Supervision, Writing – original draft, Writing – review & editing. ST: Conceptualization, Formal analysis, Supervision, Writing – original draft, Writing – review & editing. PB: Investigation, Supervision, Validation, Writing – original draft, Writing – review & editing.