Parties are the central actors in representative democracies and electoral competition as they perform important democratic functions such as aggregating and articulating citizens' interests. To understand electoral processes or estimate party responsiveness, the identification of party positions is a crucial concern in electoral and party research—in fact, it has become a subdiscipline in its own right (Laver, 2001). There is a wide range of methodological approaches and types of data to determine party positions: they can be estimated with the help of expert or mass surveys, legislative voting behavior, media analyses, or based on texts (for a detailed discussion, see Mair, 2001). Within the text-based methods, a further distinction can be made between quantitative and qualitative analysis methods. In addition, text analyses can draw on various types of text data, e.g., parliamentary or candidate speeches (Lauderdale and Herzog, 2016; Atzpodien, 2020) as well as policy papers or coalition agreements (Benoit et al., 2005; Gross and Krauss, 2021). More frequently, however, election or party manifestos¹ are relied upon to estimate parties' policy positions (Slapin and Proksch, 2008; Volkens et al., 2013; Bräuninger et al., 2020).

Election manifestos are so well suited for an analysis of party positions as they are a fairly reliable and readily available data source. However, their analysis is also accompanied by challenges—one of the biggest being cross-national comparison because of the multitude of languages (Lucas et al., 2015, p. 255). Especially if the manifestos will be coded manually, either native-language coders/analysts have to be hired or the texts have to be translated.² When it comes to translation, there are two options: professional translators or machine translation (MT). The latter method and its pros and cons will be examined in this article to identify opportunities and obstacles for the estimation of party and policy positions based on automatically translated manifestos. The article primarily concerns the MT tool DeepL and based on manifestos in four European languages it evaluates its translation quality using back translation and Wordfish analyses. The main results of this evaluation seem to confirm the company's promise of excellent quality: first, the back translation shows that more than 90% of the checked sentences are reproduced word-for-word or at least synonymously. Second, the Wordfish analysis shows that the positioning of the automatic translation on the left-right scale is very close to that of the original text indicating that the translation did not change the tone, content, and political implications.

The remaining article is structured as follows: first, I discuss current developments and state-of-the-art MT methods to outline their benefits and shortcomings compared to human translation. Afterwards, I exemplify this by presenting a step-by-step guide to implementing MT with the commercial artificial intelligence (AI) tool DeepL³ for large-scale projects. The added value of this section is that an evaluation of the translations is carried out to assess their quality as an attempt to open the black box of the AI machine translation algorithm. The subsequent conclusion summarizes the results and evaluates them against the backdrop of existing limitations.

In translation studies, machine translation is defined as “the automatic conversion of text from one natural language to another” and it is the “process in which the interlingual conversion of text is carried out by a machine, even if the proper functioning of that machine relies on human labor before, during or after run time” (Kenny, 2019b, p. 305–306). Even though machine translation has been used for several decades, the technical developments of the past years have led to a great surge of innovation, for example, the development of end-to-end neural machine translation (NMT), which has become the state-of-the-art method (Tan et al., 2020, p. 5). Neural machine translation is what is meant when we generally talk about translation with the help of AI or deep learning. It is neither the aim nor within the scope of this article to trace the development stages and technical aspects of MT,⁴ but rather to discuss its advantages for political science and especially party research. Nevertheless, it is worth briefly outlining its simplified functionality: essentially, NMT works in the interaction of source to target language, in that the neural network predicts the translation word by word using word embeddings, i.e., the spatial representation of words that exhibit the semantic relationships among them. With increasing amounts of training data, the network “learns” and, thus, improves the quality of the translations. However, for commercial MT algorithms it is difficult to assess robustness and interpretability because these models are essentially black boxes as we know neither their training data nor coding. From Tan et al. (2020, p. 16), it can be concluded that “noisy inputs” i.e., erroneous spelling or incorrect usage of words, are a particular problem to the translation's robustness. Yet, it can be assumed that manifestos and other official documents published by political parties are likely to be almost free of errors, as they are usually well-curated and edited. This is one reason why, for translated text data, manifestos are superior to e.g., parliamentary speeches.

Machine translation has some decisive advantages and disadvantages compared to “traditional” translation by humans. First, one of the most severe disadvantages is undoubtedly the fact that AI has no feeling for language and does not register sensitive language usage—especially if context is lacking. Examples of this are irony and sarcasm as well as idioms or (political) expressions established at a certain point in time, such as the NATO's 2% defense investment guideline, which in German is commonly referred to as the “Zwei-Prozent-Ziel” (literal translation: “two-percent-target”). As it is currently discussed, the term is well-known among policy-makers, the media, and the informed public, but 20 years from now that might not be the case—making the term meaningless. Related to this point is a second drawback of MT, namely its handling of language change or evolution. That languages change over time is a normal and inherently unproblematic process. For MT, however, semantic change in particular can be a difficulty because the usage and connotation of certain terms change. In English, for example, the use of the word “gay” has changed at least three times (Lalor and Rendle-Short, 2007, p. 148; Shi and Lei, 2020, p. 35). Accordingly, the expression was first used with a positive connotation as a synonym for “jolly” or “happy”, this changed to the neutral meaning “homosexual”, and more recently to a negatively connoted expression for “boring” or “lame”. A similar development can be observed with regard to the “N-word” and the linguistic representation of Black people in general (Washington, 2023). Even though MT's handling of such cases has not yet been systematically evaluated in the literature, it can be assumed that incorrect or offending translations may occur—especially if the corpus spans several decades, i.e., potentially across semantic changes. As human translation still marks the gold standard, it can be expected that professionally trained translators better capture such language use because of their context knowledge. Bizzoni and Lapshinova-Koltunski (2021), however, find that translations from different translators are stylistically quite heterogeneous. This implies that the translation quality is highly dependent on the individual and their language and content knowledge. However, further research that investigates the comparability of different human translators is needed. Alternatively, computational linguistics research discusses the possibility of combining MT with human translation in order to use “the best of both worlds” (Li et al., 2023, p. 9511; Peña Aguilar, 2023). In this approach, human translations are fed into the training dataset of the MT algorithm to improve its quality. Accordingly, this approach requires access to the MT algorithm, which—at least for commercial software—is usually not available. Lastly, there may also be quite mundane obstacles when using machine translation, e.g., the preferred MT does not offer all languages or is not available in a country. The opacity of AI tools described above also prevents outsiders from assessing how well which tool has been trained with which inputs and, above all, languages and language combinations. Since, however, these obstacles cannot be eliminated by the researcher(s), they must at least be considered as a limitation in a critical reflection.

On the other hand, one extremely significant advantage of MT compared to human translation is its resource efficiency. This is to be understood in two respects: for one, machine translation requires significantly less time, and for another, financial resources. This fact makes MT particularly interesting for research projects that have to get by without generous funding and/or a large team and must be completed within a foreseeable schedule. Another advantage of machine translation lies in the deep-learning approach of neural translation tools as with every translation, i.e., data input, they are continuously learning and improving. Consequently, MT algorithms draw on a constantly growing body of knowledge, whereas typically only one translator is hired per document, meaning that translation quality depends on that person.

In summary, when deciding between machine and human translation, the potentially poorer or more error-prone translation quality must be weighed against the significant time and money savings of real-time translation. For large corpora, human translation is just not feasible because of the sheer amount of text. In addition, the type of text to be translated and the time frame in which it was created are also decisive. For example, irony and sarcasm, which can be hard to grasp for a MT algorithm, play a much greater role in political speeches than in press releases and party manifestos. If the time period in which the texts were created spans several decades, it should also be reflected to what extent language change could influence the results. Moreover, Reber (2019, p. 118) points out that the “choice of method”, i.e., full-text translation vs. translation of individual words/expressions, must be considered because MT algorithms rely on the context in which a word is used. He concludes that the translation of entire documents is more accurate and should, thus, be the first choice. Accordingly, a blanket recommendation for one or the other does not make sense; however, especially given further development, training, and evaluation of MT, it is expected that this method will become more and more established in (political science) research.

Unsurprisingly, machine translation has already been applied in political and communication science research and has also been evaluated (i.e., Lucas et al., 2015; de Vries et al., 2018; Düpont and Rachuj, 2022). Furthermore, Lucas et al. (2018) provided the R package translateR, which enables translation with API integration. All of these applications have used Google Translate for the translation.⁵ This article, in contrast, hereafter presents the application of the commercial AI DeepL and discusses the advantages of this MT tool. For this purpose, I will report on my experiences with translating current European manifestos. In addition to manifestos, there are many other possible applications of MT in party research, such as party statutes, social media posts, or speeches. However, the above-mentioned considerations and potential shortcomings must be assessed for different types of textual data. Before presenting the procedure, first, some brief explanations about DeepL.

DeepL is a translation AI by the same-named German company, which has been available since 2017 for an initial seven (exclusively European) languages. At the time of writing (July 2023), DeepL offers translations for 31 languages, which, according to the company's statements, significantly exceed the quality of other machine translations. This quality is said to have been evaluated both by “scientific tests” and “external professional translators” (DeepL, 2022). However, these claims are difficult to verify because the sources of the tests and evaluations are neither published, provided, nor cited on the company's homepage. Yet, several technology magazines and media outlets generally confirm these statements (i.e., Coldewey and Lardinois, 2017; Wyndham, 2021). Additionally, first academic assessments seem to confirm the good translation quality. Hidalgo-Ternero (2020, p. 170) compares Google Translate and DeepL translations of Spanish idioms and concludes: “[O]verall, DeepL slightly outperforms Google Translate […][as] the global results exhibit an accuracy rate of 70% […] for Google Translate and 78% […] for DeepL.” Also focusing on the Spanish-English language combination, Peña Aguilar (2023) finds that DeepL outperforms both Bing and Google Translate. Lastly, in his comparison of Google Translate and DeepL, Reber (2019, p. 117) concludes that both tools perform equally well for full-text translation. For my research, I chose DeepL partly because of this said translation quality. More importantly however, it was decisive that DeepL—in contrast e.g., to Google Translate or Amazon Translate—can process plain text files (txt format), which are the standard file format especially in scientific quantitative text analysis.

DeepL offers multiple usage options tailored to different needs. For one, there are both free and paid subscriptions; for another, DeepL features simple text translation in the web or app translator, translation of entire files and API integration. The DeepL API can also be integrated into R using the deeplr-package (Zumbach and Bauer, 2021), which is a good alternative to text or file translation, in particular when the text to be translated is not exceedingly long. Up to 500,000 characters per month (~280 standard pages) can be translated free of charge; in addition to a monthly fee, API Pro is billed according to actual consumption. So before deciding for or against API translation, it should be calculated how many characters the text to be translated comprises. In the following, I will concentrate exclusively on file translation which is offered for six file formats (docx, htm, html, pdf, pptx, txt), but as this guideline aims to make the translated manifestos accessible for (automated) text analysis, which requires machine-readable data, only text files (txt) are considered here. The code provided in the Supplementary material, however, also includes API translation (see Reber, 2019 for an exemplary application of the DeepL API).

Summarizing, again no general and universally valid recommendation can be made with regard to the selection of the MT tool. Rather, the decisive factors are the volume of the text data to be translated, the file format in which it is available, and the analyses that are to be performed following translation. For the aforementioned reasons, however, this article focuses only on DeepL.

The overall aim of this article was to discuss the benefits and pitfalls of machine translation, particularly using DeepL, for political science and party research. I present the process of automatically translating a large set of party manifestos with its difficulties and workarounds using R and DeepL. For large-scale cross-national studies, multilingual text data are a challenge that can be met using MT. The step-by-step guide presented here will assist other scholars with such projects, and the commercial translation AI DeepL offers a solution by providing high-quality translations for 31 languages. The most tedious challenge within the workflow is the pre-processing of files, i.e., the text extraction, as manifestos exhibit an increasingly sophisticated layout and have grown in length. In general, it can be stated that the more complex the layout, i.e, the typesetting, of the source file, the more difficult the text extraction. To provide further added value to the technical implementation, in a second step I tried to open the black box of the MT algorithm a bit to evaluate the quality of the machine-translated texts. After all, translation quality and thus the tool's scientific trustworthiness is of central importance, especially for the analysis of party positions and rhetoric. Therefore, I performed an exemplary evaluation of the DeepL translations both by re-translating a set of manifestos of German parties and by determining the relative position on the left-right scale of German, Estonian, Polish and Italian manifestos using Wordfish analyses. The results of this evaluation indicate that DeepL offers high-quality translations, which do not significantly change the content and the positioning. The back translation revealed that DeepL re-translates 90% of the sentences either word-for-word or at least synonymously with an average error rate of 3.4 errors per 80 sentences. The subsequent Wordfish analysis showed for all language combinations that the relative positions of original and translation are very close and that there is no systematic right-left bias. Nevertheless, it should not be forgotten that most MT algorithms are black boxes that cannot be fully opened by the evaluation carried out here. Both future research and further development of the algorithms would be necessary to make MT an integral part of social science research.

In contrast to the approach presented here, there are recent developments to process and analyze multilingual corpora in their original language. New transformer models such as BERT (Devlin et al., 2019) or multilingual sentence embeddings (Licht, 2023) were enabled by advances in computational linguistics in the field of large language models (LLMs) and provide valid and reliable results. Which of the two approaches should be chosen in each individual case depends centrally on how the text data are to be processed in the further course of the research project. The central advantage of translated text data is that, on the one hand, the entire corpus can be examined in one analysis, e.g., a machine learning algorithm, and, on the other hand, that the results can be inspected by one researcher and further examined, e.g., with manual coding. This paper contributes to the validation of analyses using translated text data, as the main findings show that MT—and particularly DeepL—produces reliable and trustworthy results. The importance of this contribution is also underlined by the fact that, according to de Vries et al. (2018, p. 418), many authors either simply assume that MT provides reliable results or do not consider this issue at all.

While this article concentrates on party manifestos and policy positions, MT can be valuable for other types of textual data and analyses as well. In fact, the amount and availability of (political) textual data such as press releases, speeches, newspaper articles, and social media posts is growing every day and MT is one way to make this data accessible for automated comparative research. The added value of machine translation in general and DeepL in particular is that the entire text, including its framing and rhetoric, is made accessible for analysis. This allows for broader research perspectives that cannot be covered, for example, by existing coding schemes such as the Manifesto Project.

Concluding, all of this leaves us with the scientific-ethical question of whether such a further “algorithmization” of (social science) research is desirable and a trend worth supporting. Particularly in light of open science initiatives, this development must be critically questioned. This applies all the more to commercial software and algorithms like DeepL. In his opinion commentary, Arthur Spirling (2023, p. 413) therefore claims: “The rush to involve such artificial-intelligence (AI) models in research is a problem. Their use threatens hard-won progress on research ethics and the reproducibility of results. Instead, researchers need to collaborate to develop open-source LLMs that are transparent and not dependent on a corporation's favors.” Consequently, it would be extremely desirable for companies to disclose their algorithms or for open-source MT tools to be developed that provide high-quality translations for a variety of language combinations. For such tools, it would then also be possible to combine human and machine translation—an approach that promises significant quality improvements of up to 28% (Li et al., 2023, p. 9512). Until such tools are available, however, I argue in this article that commercial AI tools such as DeepL are a good alternative. After all, MT tools can also contribute to making science more accessible and inclusive by enabling the analysis of countries that are not typically the focus of interest. Furthermore, machine translation enables scientists without large financial resources to participate in and contribute to the research on political parties.

The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.

JP: Conceptualization, Formal analysis, Methodology, Visualization, Writing – original draft, Writing – review & editing.