Depression is also referred to as major depressive disorder (MDD). It is a widespread mental illness that impacts a significant percentage of people globally. According to the statistics of the World Health Organization, approximately 332 million people in the world have depression (Herrman et al., 2019). Depression is characterized by symptoms such as persistent sadness, loss of interest in daily activities, irritability, hopelessness, changes in appetite or weight, and low self-worth. In more severe cases, it may also result in suicidal thoughts or actions.

Diagnosing depression is inherently complex, typically requiring the expertise of trained mental health professionals. While early detection is essential for effective intervention, access to timely diagnosis remains limited due to high costs and inadequate access to mental health care, particularly in rural and underserved areas (Semrau et al., 2019). Artificial intelligence (A.I.) has shown promise in recent years in various medical applications (Uppal et al., 2023; Zafar et al., 2023; Wang et al., 2025; Zhao et al., 2023), including mental health assessment. However, the development of AI-based depression detection systems is hindered by several challenges, chief among them being the shortage of clinically validated data for model training.

Most datasets that are accessible to the general public for depression detection are relatively small and exhibit severe class imbalance. For instance, the Extended Distress Analysis Interview Corpus (E-DAIC) includes only 275 participants, of whom just 66 (24%) are diagnosed with MDD. Similarly, the Chinese Multimodal Depression Corpus (CMDC) and the Multimodal Open Dataset for Mental-health Analytics (MODMA) contain merely 78 and 52 participants, respectively, with fewer than half representing depressed individuals. The potential of AI models to generalize across various populations and therapeutic scenarios is severely limited by these dataset size and representativeness constraints.

Among various methods of detecting depression, the investigation of linguistic features in patient speech has been regarded as especially beneficial. Text features offer a more abundant level of interpretability compared to auditory or visual modalities. Linguistic markers, particularly those with absolutist terms such as “NEVER” indicated by phrases such as “I NEVER want to wake up” (Adam-Troian et al., 2022), and desperation or fatigue expressions, more commonly represented by “I am so tired all the time,” are found to correlate with depressive symptoms. Conventional clinical instruments, including the Beck Depression Inventory (Beck et al., 1996), Self-Rating Depression Scale (Zung, 1965), and Patient Health Questionnaire (PHQ; Spitzer et al., 1999), rely strongly on patient feedback in verbal or written format, thereby highlighting the central role that textual information plays in depression evaluation.

In spite of these strengths, the variation in patients' language abilities, emotional expressions, and cultural contexts poses significant obstacles to the construction of representative textual datasets. Without sufficient diversity and sample populations, artificial intelligence systems are unable to successfully pick up on subtle cues of indicators of depression and might not be as generalizable to real-life situations. It is necessary to overcome these limitations to fully harness the potential of artificial intelligence-based depression intervention systems.

To resolve limitations related to insufficient training data, we recommend using a transfer learning approach based upon deep pre-trained language models to strengthen performance in minimal, clinically annotated datasets. Specifically, our model utilizes two big language models (LLMs) to take advantage of their pre-trained knowledge and two different neural network designs, namely a one-dimensional convolutional neural network (1DCNN) and a fully connected neural network typified by dense layers combined with a dropout mechanism for automatic depression detection. By fine-tuning the model using diagnostic labels and harnessing the linguistic knowledge encoded in the LLMs, the proposed architecture enhances the system's ability to identify depressive symptoms from textual input.

The following is a summary of this research's main contributions.

This paper's remaining sections are arranged as follows: Section 2 provides an overview of previous studies pertinent to this research; Section 3 describes the workflow, dataset, data processing steps, and model structure; Section 4 outlines the experiment and evaluation setups; Section 5 presents the experiment configurations and evaluation criteria; and finally, Section 6 concludes the study with final remarks.

Speech text provides valuable insights for depression evaluation, which motivates researchers to develop automatic depression detection methods using textual data. Among the deep learning approaches, 1DCNN and variants of Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTM) and BiLSTM, have been widely employed for these tasks.

Convolutional neural networks, initially proposed for image processing (LeCun et al., 1989), were utilized for one-dimensional data in the mid-2010s, tackling difficulties in natural language processing, time-series analysis, and signal processing. An early milestone in this direction was the work of Kim et al., who applied 1DCNNs to text classification in 2014 (Kim, 2014).

Complementing CNNs, Long Short-Term Memory (LSTM) networks, proposed by Hochreiter and Schmidhuber (1997), addressed the vanishing gradient issue of traditional RNNs, allowing the effective learning of long- and short-term correlations. Building on LSTM, Graves and Schmidhuber (2005) proposed BiLSTM, which combines forward and backward LSTMs to capture bidirectional dependencies, making it suitable for sequential data processing tasks, e.g., speech recognition, automatic translation, and classifications.

In 2023, Wani et al. explored depression screening using Word2Vec (Church, 2017) and TF-IDF (Baena-García et al., 2011) features combined with CNN and LSTM models (Ahmad Wani et al., 2023). They achieved accuracies of 99.01% with Word2Vec (CNN + LSTM) on data sourced from Facebook, Twitter, and YouTube. However, one of the study's limitations is its dependence on non-clinically diagnosed data, raising concerns about its applicability in clinical settings.

To address limitations in data and feature extraction, transfer learning techniques have emerged as a promising approach across various research domains and have shown effectiveness in a range of language-related applications.

The BERT model (Vaswani et al., 2017), pre-trained on massive corpora such as BooksCorpus (Zhu et al., 2015) and English Wikipedia, has become an essential building block of transfer learning techniques. Such proficiency in learning knowledge from long and uninterrupted texts through document-level training has been key to its applicability in tasks that need deep contextual understanding. For instance, Milintsevich et al. (2023) applied the RoBERTa (Liu et al., 2019) model, an adaptation of BERT, for depression prediction based on clinical transcripts from the DAIC dataset. According to their study, depression detection revealed a macro-F1 score of 73.9 within a binary setting, citing the need for clinically validated datasets.

Zhang and Guo (2024) presented the MDSD-FGPL algorithm, integrating BERT and T5 encoders in fine-grained prompt learning, through a research project carried out in 2024. The multi-tier detection approach yielded an F1-score of 0.8276 for binary classification, hence pointing out the advantages of encoder integration. Hadikhah Mozhdehi and Eftekhari Moghadam (2023) applied Emotional BERT toward tasks of emotion recognition on the Wang and MELD datasets through another investigation, further proving the effectiveness of transfer learning in deriving contextual knowledge from minimal datasets.

The T5 model, described by Raffel et al. (2020), has been widely applied to a variety of text-centered analytical tasks (Bao et al., 2024). The model's flexibility is further demonstrated by the Sensory-T5 version presented by Zhao et al. (2025), which incorporates sensory information to enhance emotion classification accuracy. This methodology has led to significant precision and F1 score improvement on various datasets.

Inspired by a similar effort, Lu et al. (2025) also fine-tuned T5 for their ABSA and achieved remarkable improvement through their application of data augmentation techniques and implicit rationale-driven information management. Chawla et al. (2024) also identified emotional dimensions that influence the negotiation process, particularly with regard to outcome predictions such as satisfaction and perceptions of partners. Incorporating the CaSiNo dataset with emoticons, Linguistic Inquiry and Word Count, and T5 models, they conducted a comparison whereby T5-Reddit performed better than other models in detecting subtle emotional expressions.

As impressive as the breakthroughs achieved through leveraging pre-trained models have been for various text analytical applications, most available works tend to be based upon a single model developed for a specific architecture or a single configuration of tasks. Even though these models have delivered impressive performances, overreliance upon a solitary model has the potential to compromise generalizability, especially when applied in cases like detecting clinical depression, where access to annotated knowledge is sparse and contextual centrality matters.

Considering these limitations, the current work proposes initiating an innovative dual-language model integration. A methodological strategy that combines benefits of two sizable pre-trained language models: BERT and T5. The models have complementary characteristics in structural design and learning methods. BERT is an encoder-based model that is particularly designed to handle bidirectional contexts. The use of contextual features in a sentence reinforces its ability to capture subtle semantic relationships and complex syntactic relationships. On the other hand, T5 is a text-to-text transformer model based on a combined encoder-decoder model that is ideally suited to generate and reconstruct text content, thus demonstrating a better ability to generalize and abstract across diverse tasks. By combining the models, the current approach leverages the accuracy of BERT in producing contextual embeddings (Devlin et al., 2019) and the versatility of T5 in capturing task-agnostic patterns through its unified text-to-text framework (Raffel et al., 2020), ultimately leading to more accurate and efficient detection of depression from transcribed speech.

The initial step involves pre-processing raw text transcripts, which is integrated into the proposed framework. As illustrated in Figure 1, the framework takes advantage of the collaborative strengths of transformer-based pre-trained large language models, specifically BERT and T5, both of which are highly reputed for their efficiencies in natural language processing. These models possess the capability to learn subtle linguistic patterns, grammatical structures, and other language attributes (Rodríguez-Ibánez et al., 2023), and are assumed to capture transferable knowledge.

To effectively utilize this pre-trained knowledge for our task, both models are fine-tuned using pre-processed data tailored to depression prediction. The processed text is fed into the fine-tuned BERT and T5 encoders, generating two distinct sets of embeddings. These embeddings are subsequently passed through parallel 1DCNN and Dense branches to extract task-relevant features and produce preliminary predictions.

The resulting predictions are then passed to a fusion module that performs a logical “AND” operation for the final output.

The experiment's hardware is a workstation with an AMD Ryzen 9 CPU and 64 GB RAM. The software environment is the Windows 11 operating system, Jupyter IDE, and Python 3.11. The models were trained using a CPU.

The experimental results are assessed using the following metrics: accuracy, precision, recall, and F1 score. These metrics offer a thorough comprehension of the model's functionality. The equations for these metrics are defined as follows:

where TP, TN, FP, and FN represent the counts of true positives, true negatives, false positives, and false negatives in the predictions, respectively. Additionally, the study involves plotting the Receiver Operating Characteristic (ROC) curves for the learning models for comparison purposes. The ROC curve depicts the true positive rate (TPR) vs. the false positive rate (FPR) at various threshold settings. Furthermore, the area under the curve (AUC) was calculated. An increased AUC signifies improved overall performance.

In this section, we examine and compare the traditional text feature extraction algorithms and transfer learning-based text embeddings using both conventional ML and deep learning models. The study evaluates six conventional machine learning models–Logistic Regression (LR), Support Vector Machine (SVM), Random Forest (RF), Naive Bayes (NB), K-Nearest Neighbor (KNN), and Decision Tree (DT)–alongside five deep learning models designed for one-dimensional text data analysis. The deep learning models include a fully connected dense neural network, a 1DCNN, a recurrent neural network (RNN), a gated recurrent unit (GRU), an LSTM, and a bidirectional long short-term memory network (BiLSTM).

This study presents a dual-stream transfer learning framework for text-based depression detection, leveraging transformer-based large language models.

Transfer learning, particularly through BERT and T5, offers substantial advantages over non-TL approaches–especially in tasks requiring nuanced interpretation of linguistic patterns. Although models such as RF and Dense perform competitively in the non-TL setting, they are outperformed by TL-based models in terms of recall and F1. This highlights the strength of contextual embeddings in capturing subtle depressive cues.

When T5 embeddings were processed through a 1DCNN architecture, performance improved significantly. The T5 + 1DCNN model achieved an accuracy of 88.5%, an F1 of 87.7%, and an AUC of 95.5%–a substantial increase over standalone T5, which showed weaker performance (accuracy: 64.1%, F1: 63.1%, AUC: 65.6%). These results suggest that convolutional layers effectively complement T5 embeddings by extracting local sequential patterns that improve representation quality.

The integration of BERT and T5 embeddings through parallel Dense and 1DCNN branches led to further performance improvements. The resulting fusion model, which combines all four branches, achieved the best results across all evaluation metrics, including accuracy (91.3%), F1 score (90.0%), and AUC (95.6%). Its high precision (95.2%) indicates strong reliability in minimizing false positive predictions. Although high recall is critical for depression screening, the proposed model balances sensitivity with improved precision to reduce false alarms in decision-support use. It is important to emphasize that the combination of BERT and T5 is not intended to assert theoretical optimality of a specific model pairing. Instead, this design aims to reduce reliance on a single pre-training bias by leveraging heterogeneous semantic encoders. Such an agreement oriented fusion strategy enhances robustness when modeling subtle and implicitly expressed linguistic markers of depression.

Compared to the best-performing non-TL model (RF + TF-IDF, accuracy: 88.3%), the proposed fusion method improved accuracy by 3.0%, demonstrating its superior effectiveness in real-world, low-resource clinical datasets.

In summary, transfer learning with pre-trained LLMs significantly enhances the capability of automated depression detection systems. By combining BERT and T5 within a dual-stream architecture, this study demonstrates a robust approach that outperforms both traditional and deep learning baselines. Importantly, the proposed framework has potential clinical relevance: it may support early-stage depression screening, reduce the burden on clinicians by providing automated pre-assessment tools, and enable scalable integration into telehealth and digital platforms for mental health monitoring.

Looking ahead, future research could explore the generalizability of this method to other mental health conditions, its extension to multimodal settings (e.g., combining text, audio, and video), and its deployment in clinical environments for scalable and early-stage mental health assessment.