Figure 1 presents a structural diagram of the TCR sequence recognition algorithm. In summary, TCR sequences associated with cancer and those not associated with cancer are initially encoded into a matrix using amino acid biochemical features as model inputs. This matrix is subsequently processed in the convolutional layer using a multi-scale convolutional kernel to extract features. A max pooling layer is employed to encode the feature set of amino acid fragments of varying lengths before applying a multi-head self-attention mechanism to assign differential attentional weights. The resulting attention-weighted encoding matrices are interconnected along the channel dimension, producing an attention-weighted matrix that contains key molecules of different lengths. This weighted pattern matrix is then further processed using bidirectional long- and short-term memory networks, which focus on the correlations between these key patterns. Finally, a self-attention mechanism is introduced to assign varying attention weights, followed by the application of a linear classifier for binary classification.

The model architecture and final hyperparameter configuration, including convolutional kernel dimensions, pooling strategies, and fully connected layer specifications, are detailed in Table 1 . The process of parameter tuning, which involves a systematic evaluation of alternative dropout rates and learning rates, along with the associated performance metrics, is thoroughly documented in Supplementary Table 2 .

The experiments were executed on a high-performance computing platform operating Ubuntu 20.04, featuring an Intel^® Xeon^® Platinum 8470Q processor with 20 virtual CPUs, 90GB of RAM, and an NVIDIA virtual GPU with 48GB of memory. The software environment consisted of Python 3.8 and PyTorch 1.10.0 with CUDA 11.3 for acceleration, supplemented by standard scientific computing libraries. For model development, 30,000 cancer-associated CDR3 sequences and approximately 60,000 non-cancer sequences were encoded, assigning binary labels (1 for cancer and 0 for non-cancer). The dataset was divided using stratified sampling, with 80% designated for training and 20% for validation. To ensure robust performance evaluation, fivefold cross-validation was employed across all experiments. The training process utilized the Adam optimizer with a learning rate of 0.001 and cross-entropy loss for error computation. To mitigate overfitting, dropout was applied with a probability of 0.5 during training. The model was trained for a maximum of 1,000 epochs, with an early stopping criterion activated if the validation loss did not improve for 20 consecutive epochs.

This study utilized six metrics to assess model performance: accuracy (ACC), sensitivity (SEN), specificity (SPE), area under the receiver operating characteristic curve (AUC), F1-score, and Matthews Correlation Coefficient (MCC). Each metric offers unique insights into the classifier’s capabilities (Equations 11–17):

where TP, TN, FP, and FN represent true-positive, true-negative, false-positive, and false-negative predictions, respectively.

Due to the inability to directly utilize raw amino acid sequences for model training, this study employed a biochemical feature-based encoding strategy to convert these sequences into numerical form. Focusing on the functional characteristics of antigen-binding sites within the CDR3 region, 553 biochemical feature indicators of amino acids were selected from the AAindex database for principal component analysis (PCA). Through dimensionality reduction, a 20 × 20 amino acid feature matrix was derived, and the top 15 principal components, which collectively accounted for over 95% of the cumulative variance, were chosen to construct a standardized 20 × 15 AAindex coding matrix. For CDR3 sequences shorter than 20 amino acids, a zero-padding strategy was applied to encode them uniformly into a 20 × 15 matrix structure.

In order to assess the performance of DeepCaTCR, we conducted a comparative analysis with leading caTCR recognition models, namely, DeepLION and BertTCR. This evaluation utilized a consistent encoding scheme, training dataset, learning rate, and batch size across all models. Through fivefold cross-validation, DeepCaTCR demonstrated superior performance in antigen-specific TCR recognition, achieving an ACC of 0.807 ± 0.003 and AUC of 0.863 ± 0.003, as presented in Table 2 . Notably, DeepCaTCR outperformed both DeepLION (ACC: 0.801, AUC: 0.854) and BertTCR (ACC: 0.760, AUC: 0.790), achieving the highest ACC and AUC values. The sensitivity of DeepCaTCR (0.586) was 28% higher than that of BertTCR (0.457), while its specificity (0.918) remained the highest among all models evaluated. Furthermore, the F1-score (0.669) and MCC (0.548) exceeded those of the competing models.

We conducted a detailed analysis of the performance metrics for each fold and performed paired t-tests to assess statistical significance, comparing each model against DeepCaTCR. The findings indicated that BertTCR was significantly outperformed by DeepCaTCR across all metrics (p < 0.0001), with the exception of specificity ( Figure 2 ). DeepLION demonstrated significantly lower performance than DeepCaTCR in terms of ACC, AUC, and MCC (p = 0.01–0.02). We posited that the suboptimal performance of BertTCR could be attributed to its limited number of filters, which was initially set at six. To investigate this hypothesis, we increased the number of filters in BertTCR to nine, resulting in a significant enhancement in model performance (p < 0.008, Supplementary Figure S1 ).

To further substantiate the contributions of the core components within DeepCaTCR, we performed ablation studies focusing on BiLSTM and MHSA. We developed variant models, namely, DeepCaTCR-noBiLSTM, DeepCaTCR-noMHSA, and DeepCaTCR-noBiLSTM-noMHSA. We subjected these ablation variants to the same experimental conditions as the baseline DeepCaTCR model, encompassing input data preprocessing, shared embedding layer parameters, output layer architecture, loss function, optimizer configuration, and train/test splits. The sole alteration involved the exclusion of specific model components.

The comprehensive DeepCaTCR model demonstrated superior performance across several metrics, achieving the highest accuracy (0.807), AUC (0.863), specificity (0.918), F1-score (0.669), and MCC (0.548). This underscores the synergistic advantages of integrating BiLSTM and MHSA. Upon the exclusion of BiLSTM (DeepCaTCR-noBiLSTM), there were notable declines in performance metrics: ACC decreased by 1.5%, AUC by 1.7%, SPE by 2.5%, F1 by 1.8%, and MCC by 5.1% ( Table 2 ). Interestingly, SEN exhibited a slight improvement (0.593 compared to 0.586), which may be attributed to the reduced complexity of the model influencing class-specific predictions. The removal of MHSA (DeepCaTCR-noMHSA) resulted in smaller yet consistent reductions in ACC (0.9%), AUC (1.2%), SPE (2.2%), F1 (0.4%), and MCC (2.9%). Similar to the removal of BiLSTM, SEN improved (0.602 compared to 0.586), suggesting that attention mechanisms may trade off some sensitivity for specificity. The most pronounced degradation in performance metrics (ACC: −2.0%, AUC: −2.7%, F1: −4.2%, MCC: −7.1%) underscores the complementary roles of BiLSTM and MHSA in feature extraction and context modeling. Notably, SEN experienced a sharp decline (0.560 compared to 0.586), indicating that the combined use of BiLSTM and MHSA enhances recall for positive samples.

While DeepCaTCR exhibits strong capabilities in recognizing cancer-associated sequences, its effectiveness in clinically distinguishing between cancer patients and healthy individuals requires further validation through independent experiments. It is important to note that accurately evaluating the overall immune status presents substantial technical challenges. This difficulty arises because antigen-specific TCRs constitute only a small fraction of an individual’s TCR repertoire, typically less than 0.1%, and there is a considerable background noise (43). To address this issue, the study employed the iSMART antigen-specific clustering technique to extract representative sequences from each database. This approach enabled the quantification of an individual’s tumor immune response by calculating the mean cancer probability, or cancer score, of these characteristic sequences.

DeepCaTCR demonstrated robust discriminatory power across diverse sample types and clinical scenarios, as quantified by the mean cancer score of antigen-specific TCR clusters. TILs exhibited significantly higher cancer scores than PBMCs from healthy donors (p < 5e−07, Wilcoxon rank-sum test, Figure 3A ). There was near-perfect discrimination (AUC > 0.95) for all cancer types ( Figure 3B , Supplementary Table S3 ), with primary lung cancer (AUC = 1), pancreatic cancer (AUC = 0.998), and melanoma (AUC = 0.994) showing high specificity (SPE > 0.96) and sensitivity (SEN = 1.0). Untreated cancer patients had significantly higher PBMC cancer scores than healthy controls ( Figure 3C , p < 0.0007), with ovarian cancer (AUC = 0.997) and pancreatic cancer (AUC = 0.989) having the leading performance ( Figure 3D ). Treated patients ( Figure 3E ) displayed reduced cancer scores versus untreated cohorts, likely due to the therapy-induced depletion of tumor-reactive T cells. Despite lower scores, the most model-maintained AUC > 0.81 ( Supplementary Table S3 ) was for refractory cancers (glioblastoma: AUC = 0.814; bladder cancer: AUC = 0.83; CRC: AUC = 0.919), although lung cancer discrimination declined (AUC = 0.667), potentially reflecting prolonged T-cell exhaustion.

Notably, DeepCaTCR maintained high specificity in non-cancer contexts. Evaluation of virus-infected and healthy cohorts ( Figures 3F, G , Supplementary Table 4 ) revealed consistently strong performance metrics: YFV (AUC = 0.992, SPE = 1.0), GVHD (AUC = 0.984, SPE = 0.933), and HCMV (AUC = 0.956, SPE = 0.899). Healthy donors (AUC = 0.978, SPE = 0.955) and additional healthy samples (AUC = 0.947, SPE = 0.89) further confirmed the model’s ability to distinguish cancer-associated TCRs from benign immune responses. Preliminary observations suggested that, despite several elevated cancer scores, the scores within the healthy cohort remain relatively low. Subsequent validation using additional cohorts comprising both healthy and virus-infected individuals demonstrated that the cancer scores fall within anticipated ranges ( Figure 3F ), indicating that the initial findings may be attributable to the characteristics of the study population rather than methodological flaws.

Further analysis of the correlation between cancer scores and demographic variables such as age and gender yielded a Spearman’s correlation coefficient of R = 0.033 (p = 0.66, Figure 3H ), while a comparison by gender using the Wilcoxon test resulted in a p-value of 0.58 ( Figure 3I ), indicating no significant association. Additionally, we examined the correlation between cancer scores and TCR counts. In the initial healthy cohort (n = 176), a marginal correlation was observed (R = 0.15, p = 0.044, Supplementary Figure S2A ). In the subsequent validation cohort (n = 82), a significant but stronger correlation was found (R = −0.5, p = 2.1e−06, Supplementary Figure S2B ). However, in the combined analysis (n = 258), no correlation was detected (R = 0.021, p = 0.74, Supplementary Figure S2C ). The results suggest that cancer scores are generally stable within healthy populations, unaffected by age or gender, and not clearly associated with TCR counts. Elevated scores may reflect population-specific characteristics or weak biological factors.

Building on the exceptional recognition performance of peripheral blood samples from early-stage breast cancer patients demonstrated in the previous study (AUC = 0.955), this research further validated the generalizability of DeepCaTCR for the early diagnosis of multiple cancer types. The model’s capability to differentiate between tumor stages was systematically evaluated by collecting PBMC samples from patients with early (stage I–II) and advanced (stage III–IV) primary treatment. Additionally, independent healthy samples were collected as controls.

As illustrated in Figure 4A , the median cancer score for all early-stage cancers was significantly elevated compared to that of the healthy control group, as determined by the Wilcoxon test (p < 0.05, applicable across all cancer types). Furthermore, Kendall’s tau coefficient demonstrated a positive correlation between cancer scores and disease progression in both ovarian cancer (τ = 0.629, p = 0.0047) and pancreatic cancer (τ = 0.359, p = 0.094). DeepCaTCR achieved high AUCs for different stage cancers (stage I lung: 0.998; stage I RCC: 0.947; stage II pancreatic: 0.934, Figure 4B , Supplementary Table S5 ), with specificity consistently >86% across types.

In the context of early-stage lung cancer identification, DeepCaTCR demonstrated superior performance relative to all evaluated benchmarks, as detailed in Table 3 . DeepCaTCR achieved an AUC of 0.998 ( Figure 4C ), surpassing DeepCAT’s AUC of 0.912, while maintaining a balanced sensitivity of 100% and specificity of 98.3%. In contrast, DeepLION2 and BertTCR exhibited lower specificity at comparable sensitivity levels, with AUCs of 0.69 and 0.85, respectively. The MCC for DeepCaTCR was 0.945, compared to 0.719 for DeepCAT, highlighting its balanced classification performance.

Addressing the limitations inherent in the average scoring strategy for capturing the dynamic characteristics of the TCR repertoire, this study introduces a variance-based repertoire scoring method that markedly enhances the detection performance for early-stage cancers. While average scoring can indicate the overall tumor relevance of the TCR repertoire, it struggles to effectively characterize the heterogeneous features of TCR cancer score distribution during clonal amplification. Consequently, the variance scoring system developed in this study successfully captures the dynamic features of TCR clonal amplification during the early immune response by quantifying the degree of dispersion in cancer score distribution. As illustrated in Figure 5A , the distribution of cancer scores among early-stage patients with RCC, OV, PC, and lung cancer exhibited a more pronounced trend of intergroup segregation following the implementation of the variance scoring strategy (p < 0.001). This development facilitated the identification of more discriminative features for the subsequent classification model.

The variance-based DeepCaTCR model demonstrates consistent enhancements in the AUC relative to baseline methodologies ( Table 4 , Figures 5B–F ). In the context of identifying pancreatic cancer patients, the AUC increased from 0.935 (mean scorer) to 0.972, representing an improvement of ΔAUC = +0.037, while specificity rose from 0.862 to 0.966. For ovarian cancer patient identification, the AUC increased from 0.852 to 0.931, maintaining high specificity (0.776 compared to 0.638 for the average classifier). In the multi-cancer identification task, the unified model achieved an AUC of 0.967, effectively balancing sensitivity (0.969) and specificity (0.897), thereby underscoring its applicability across various cancer types. These findings suggest that variance-based scoring can reduce the false-positive rate, as evidenced by the increased specificity for RCC (0.914 compared to 0.879 with the average scoring model) while preserving sensitivity, which is crucial for early detection.

At clinically actionable specificity thresholds, variance scores demonstrated exceptional performance ( Table 5 ). Specifically, at a specificity level exceeding 98%, variance scores achieved a sensitivity of 62.5%, compared to 53.1% for mean scores, thereby significantly surpassing the performance of DeepCAT (9.4%) and BertTCR (0%). When the specificity threshold was set above 95%, the sensitivity of variance scores increased to 81.3%, in contrast to 75% for the mean score method, indicating their reliability in low-prevalence screening scenarios. Furthermore, the variance-based scoring method exhibited greater robustness in terms of AUC stability, as evidenced by a narrower 95% confidence interval (0.934–0.999) compared to that of the mean-based scoring method (0.895–0.986). The efficacy of the variance-based scoring method may be attributed to its capacity to quantify TCR clonal diversity during the early stages of tumor development, a characteristic that is not captured by mean-based methods.

To elucidate the biological relevance of TCR sequences predicted by DeepCaTCR, key motifs, their functional significance, and overlap with known cancer-associated TCRs were analyzed ( Figure 6 ). DeepCaTCR identified key amino acid motifs in TCR sequences from the validation set and assigned importance scores to each motif ( Figure 6A ). Visualization results show that larger and darker residues correspond to higher importance scores, indicating that these sequence patterns may play a key role in antigen recognition. Among TCRs with high prediction confidence (score > 0.95), certain motifs were highly recurrent ( Figure 6B ). The most frequent motifs included “CSAR” (140 occurrences), “CASP” (45 occurrences), and “PG” (35 occurrences). These motifs may represent conserved structural or functional elements in cancer-associated TCRs. A heatmap analysis revealed that DeepCaTCR-identified motifs are enriched in the McPAS-TCR (45) database ( Figure 6C ). “ASS” (24,730 occurrences) and “ASSL” (5,759 occurrences) were among the most frequent motifs in McPAS-TCR, aligning with their high frequency in DeepCaTCR predictions. Other motifs like “AG” (4,831 occurrences) and “EA” (3,997 occurrences) further validated the biological relevance of DeepCaTCR’s predictions.

We initially conducted a search for the top 22 high-scoring TCRs (score > 0.98) across three major databases [TCRdb (46), VDJdb (47), and McPAS-TCR (45)] but did not identify any exact matches among cancer-associated TCRs. This outcome is likely attributable to the exceptionally high diversity of TCR sequences. Considering the significant heterogeneity across cancer types, the absence of these 22 TCRs in existing databases is biologically plausible. To evaluate potential partial matches, we applied various mismatch tolerance criteria tailored to each database’s functionalities. In TCRdb, we recorded near-matches with up to two amino acid mismatches, as provided by the database. VDJdb allowed for the extraction of similar sequences with an Informativeness score of 8 or higher, indicating high-confidence hits. For McPAS-TCR, we conducted local searches using Python scripts, identifying sequences with up to four mismatches, although no hits were found with two or fewer mismatches.

A detailed breakdown is provided in Figure 6D , where blue blocks denote partial database matches, such as “CSVEDRRRTGYTEAFF” with five matches in TCRdb and one in McPAS-TCR. The analysis revealed that 11 TCRs (50%) exhibited partial matches in at least one database. For instance, the TCR sequence “CASSSGLAVPCNEQFF” demonstrated four matches in TCRdb and two high-confidence matches in VDJdb, in addition to seven hits in McPAS-TCR. Another TCR, “CSAHPGGLAGAEQYF”, was found to have two matches in TCRdb and seven in McPAS-TCR. Conversely, 11 TCRs (50%) did not exhibit matches in any of the databases under the specified criteria, exemplified by sequences such as “CSAPRDSLRRADEQYF” and “CSARPRGPLAAEAFF”, suggesting that these may represent previously uncharacterized cancer-reactive TCRs.