We denote X = {x₁,x₂,…,xi,_{xi + 1},…,_xn} as a feature dataset, where n is sample number (l≪n) and Y_l = ^{{y₁,y₂,…,y_l}T} ∈ ℝl×^M is a label set about datasetXl={xi}i=1l. Xu={xj}j=l+1n is an unlabeled feature dataset, where x_i is d dimensions (x_i ∈ ^Rd) vector of the i-th sample. We compute LWM x^i about x_i:

where the k nearest neighbors of x_i are arranged in Ks(x_i), and the Euclidean distance algorithm is used to find these neighbors. We design an undirected weight graph G = (X,D), where D ∈ ℝn×ⁿ is weight matrix, D_ji = D_ij≥0, and the element is measured as follows:

where τ is a changeable parameter in Gaussian kernel function. If the distance between x_i and x_j is smaller, D_ij is higher, and vice versa. Therefore, the clustering problem is changed into a graph problem in this paper.

Since both SSPCA and SA-SSCCM methods require that the training data and test data meet the IID assumption, this paper reasonably combines the SSPCA method with a multi-model adaptation learning method (i.e., MA-PCA). This proposed method not only improves the robustness on noises/outliers but also solves the problems of insufficient label data and noisy data affecting the performance of the model and the different distribution of training data and test data. We therefore propose the following basic formula of MA-PCA:

where Ω_M(W,γ) is the multi-model adaptation term, and Ω_B(W,v_m(x_j)) is used for reducing the negative impact of noises/outliers. We have the following function:

where λ_s,λ,C are balance parameters that can be adjusted to avoid overfitting during model training. The details about the other parameters are provided in Dan et al. (2021).

In our domain adaptation learning, given is {Wis}i=1q as a multiple-source model set, where q is source model number, and Wis is the i-th source model. Each source model is obtained by learning the specified public dataset. This paper expects the classification results of the target model to be consistent with those of the source domain models. In other words, this paper will learn the target classifier f(x) = ^WTx in the whole sample space, regularize the weight parameters of different source domain models to control its complexity, and make each target instance be close to the source domain models. According to this criterion, it can be realized by introducing the multi-source domain model adaptation regularization function on the target domain. The formula is described as follows:

where γ = ^{[γ₁,…,γ_q]T}, and γ_i is a weight of the i-th source model. We constraint that the sum of γ_i is 1 (i.e., ∑i=1qγi=1). It is better to explore the contribution among the source models. The divergence matrix of the target domain is S = ^XXT. η ∈ R⁺ is a balance parameter to control the contribution of ||γ||22. This parameter can be changed to avoid overfitting on multiple-source models. In addition, in the second equation in (4), we employ the sampling frequency as a weight to access the real distribution for the target domain.

Remark 1: The divergence matrix S is important for connecting the source classifiers and the target classifier. It will promote the learning of the target classification model to the real distribution direction of the target domain, thereby improving the generalization performance of adaptive learning, which is essentially different from other domain adaptation regularization terms based on DAL models (Bottou and Vapnik, 1992; Duan et al., 2012a). In order to better fit the model idea, this paper refers to formula (4) as the construction of the regularization term for divergence-constrained multi-model adaptation.

We expect better model adaptation performance for EEG-based emotion recognition by combining SSPCA with scatter-constrained multi-source classifier model. Therefore, a unified framework MA-PCA is obtained to learn W,v_m(x_j),γ by combining formulas (3) and (4). The optimization problem of MA-PCA can be described as follows:

where γ = ^{[γ₁,γ₂,…γ_q]T}, γ ∈ ℝq^×1,γ^T_1q = 1, and q are the number of source domain models. When β = 0, MA-PCA degenerated to SSPCA. When β > 0, β is used as a balance parameter. When γ_i is constant 1, it indicates that there is only one single-source domain, and its distribution is the same as the target domain. At this time, MA-PCA approximates the SA-SSCCM method. When 0≤γ_i < 1, MA-PCA is a multi-model adaptation learning method based on the possibility clustering assumption, and the distribution of source domain and target domain can be identical or non-identical.

Firstly, fixing v_m(x_j) and γ to solve W: for ease of calculation, the following formula (5) is transformed into matrix form. W can be written as W=∑i=1nxi⁢αi. Under the Representation Theorem, the formula (5) exists in reproducing kernel Hilbert space, and the kernel of W can be rewritten as follows: W=∑i=1nK⁢(xi,x)⁢αi (Belkin et al., 2006). Therefore, formula (5) is mapped into a finite dimensional space of the optimization α_i and can be reformulated as follows:

where the details about J,L,V^ can be found in Dan et al. (2021).

By solving the derivation of (6) w.r.t. α and letting it be equal to 0 (i.e., ∂⁡Q(α)/∂⁡α = 0), we obtain the following:

where

and

Finally, the solution of W is Kα.

Then, we fix W and γ to solve v_m(x_j). The optimal problem of the objective function in (5) can be rewritten as follows:

By solving the derivation of Eq. (5) w.r.t. v_m(x_j) and letting it be equal to zero (i.e., ∂⁡Q(v_m(x_j))/∂⁡v_m(x_j) = 0), then

we can get:

Since x_j means any one instance, the general presentation of v_m(x_j) is as follows:

We define Ai=tr[(W-Wis)TS(W-Wis)](i=1,…,q) which is corresponding to the adaptation of the i-th source model to the target domain. Given W and v_m(x_j), the objective function (5) can be rewritten in (11) as follows:

where A = ^{(A₁,…,A_q)T}. The optimal estimation of γ becomes the optimal weight division problem of multiple model adaptation learning with scatter constraints. Theoretically, if η = 0, the optimal γ_i will be 1; otherwise, if A_i = min_{j = 1,…,q}⁡A_j, γ_i is 0; if η→ + ∞, the optimal γ_i will tend to be the same weight 1/q. Thus, the following theorem is obtained:

Theorem 1 (Karasuyama and Mamitsuka, 2013). The following equation is the optimal solution of (11):

where ρ=(η+∑i=1qAi)⁢/⁢ζ, ζ = |{i|ρ−A_i > 0,i = 1,2,…,q}|.

Theorem 1 presents that there are ζnon-zero entries in the optimal γ. According to the target domain, this optimization can select source domains automatically with a different γ. If γis bigger, there is a higher similarity between the source domain and the target domain. Since the optimal γcould be calculated via the optimal ζ, Karasuyama and Mamitsuka, 2013) presented a special method to obtain the optimal ζ. This algorithm effectiveness can be proved under the given right amount of source domains.

Next, the algorithm adopts a coordinate descent strategy to solve (11), which is close to the method in Geng et al. (2012). In each iteration cycle, when other entries are fixed, we select two items to update, and γ^T_1q = 1 must be satisfied at the end of each iteration. Suppose that in an iteration cycle the i-th and j-th entries are selected, the following iterative formula can therefore be obtained:

We iteratively traverse all paired entries in γ and optimize any two entries in γ by (12) until the optimization function (5) converges. Intuitively, the updating criteria in (12) tends to be that the larger the value to γ_i, the smaller the A_i. Since A_i measures the distribution distance between the i-th source model and the target domain, the smaller the A_i, the higher the correlation between the i-th source domain and the target domain.

Remark 1: After obtaining the optimal solution of W and V, the label matrix of the samples in the target domain can also be obtained, and the influence of noise has been effectively suppressed. The label and the membership of out of sample can also be calculated by W and V. Finally, the performance of W relies on the learned graph G = XD and the multiple-source domain adaptation model.

Since the objective function (5) is a multi-objective optimization function, it is difficult to guarantee the global optimal solution. It is worthy to note that the algorithm in this paper adopts an alternate iterative strategy for optimizing. Since the objective function of a single optimization variable [i.e., equations (6), (8), and (11)] is convex and in closed form, the iterative optimal solution can be obtained. Therefore, this work only deduces the asymptotic convergence of the algorithm based on the iterative target value of the objective function. The derivation process is shown in (13) as follows:

where ^Witr,^vitr,^γitr are the optimal solutions at the itr-th iteration. ε is a very small constant. The objective function will converge to a local optimum. The derivation process proves that the iterative target value of the algorithm shows a downward trend. When the value drops to a certain threshold (at least greater than ε), we stop the iterative. Finally, the objective function will converge to the local target value of each single optimization variable.

According to Lan et al. (2018) and Zhong et al. (2020), there exist certain significant differences between SEED and DEAP since they can be generated by different subjects, sessions, EEG devices, experimental schemes, emotional stimuli, etc., Detailed information about these two datasets can be viewed in Lan et al. (2018). In the following experiments, we adopt differential entropy (Lan et al., 2018; Zhong et al., 2020) as the data feature in emotion recognition, which has also been widely used in the preceding literatures (Shi et al., 2015; Zheng et al., 2015, 2016; Chai et al., 2016, 2017; Lan et al., 2018; Zhong et al., 2020) for domain adaptation emotion recognition.

We will systematically compare our method with such state-of-the-arts as SSPCA (Dan et al., 2021), a baseline without domain adaptation, FastDAM (Duan et al., 2012b), Multi-KT (Tommasi et al., 2014) with l₂-norm constraint on p, A-SVM (Yang et al., 2007), and DSM (Duan et al., 2012a). Since existing deep domain adaptation frameworks have achieved many inspiring results on emotion recognition as well as visual recognition, we also additionally present comparisons with several deep (CNN-based) domain adaptation methods with deep features: DAN (Long et al., 2015) and Reverse Grad (Ganin and Lempitsky, 2015).

It was noted that, in the DA schema, automatic parameter tuning is not possible for source classifiers using cross-validation due to the reason that training and test data are from different data distributions. Therefore, all methods evaluated in this paper on the dataset are empirically searched for the parameter space to optimize parameter settings to obtain the best results for each method. The parameters of all methods are adjusted to obtain the best results, except for the specially specified parameters.

For the method SSPCA without domain adaptation, this experiment will fuse the decision values of all classifiers obtained from the independent training of labeled samples in each source domain and target domain. DSM, Multi-KT, A-SVM, and FastDAM are domain adaptation methods. For A-SVM, this experiment also fuses the decision values of all basic classifiers, and each classifier is learned from a labeled sample in a source domain.¹

There are some hyper-parameters in the objective function (5) that need to be determined. First of all, this experiment sets the most important parameters (e.g., γ_i) as optimization variables in the iterative optimization process, and only a few crucial parameters that are proposed in MA-PCA need to be pre-defined (e.g., λ,λ_s,C,η,β), and considering that parameter determination is an open problem in the field of machine learning, we have determined parameters empirically in a past work. Since the exponent of γ_i plays the role of avoiding trivial solutions in the process of optimizing γ_i, as proved in Hou et al. (2017), the larger the exponent of γ_i is, the closer all weight values are to be same. In order to reflect the differences among different source domains, this experiment will set the index of γ_i to be 2 by experience. The validity of this decision will be verified in the experimental results in the following section. The hyperparameter λ,λ_s,C,η,β is adjusted within the range of {10⁻⁴,10⁻³,…,10³,10⁴}. Finally, we search the nearest neighbor number k from the set {35,10,15,17} to construct the nearest neighbor graph in MA-PCA (also SSPCA) and obtain the first-ranked recognition accuracy from the optimal parameter configuration.

For non-linear learning methods MA-PCA, FastDAM, and multi-KT, Gaussian kernel K_i,j = exp⁡(−σ^{||x_i−x_j||2}) is default kernel function, σ = 1/d, and d is the feature dimension. In FastDAM, γi=exp⁡(-δ⁢D⁢i⁢s⁢t⁢(Xis,X))∑iexp⁡(-δ⁢D⁢i⁢s⁢t⁢(Xis,X)) (i = 1,…,S) is the weight value of each source domain, δ = 100. For the benchmark method SSPCA, the target domain samples are directly mapped to the source domain without any domain adaptation, and the decision values of all classifiers obtained from the independent training of labeled samples from each source domain and target domain are equally fused.

Note that different subjects even from the same dataset still have different EEG feature distributions due to individual characteristics. We therefore practice the so-called leave-one-out cross-validation strategy conducted also in Lan et al. (2018) to evaluate the emotion recognition performance of MA-PCA—that is, one subject remained to be the target domain, and the others from the dataset are constructed as multiple sources. In this multi-source scenario, we follow the same setting as Tao et al. (2021) to evaluate our method compared with other state-of-the-arts on SEED and DEAP, respectively.

It is more challenging on emotion recognition when across datasets, with the differences in acquisition pathways, subjects’ characteristics, and behaviors. The previous experiments show the performance comparison of MA-PCA with other DA methods within the dataset (i.e., across subjects). This subsection further evaluates the robust effectiveness of MA-PCA when adapting across datasets. In this experimental scenario, multiple different protocols were constructed using different EEG devices and emotional stimuli by sampling the training and testing datasets separately. Therefore, six experimental settings, namely, DEAP → session I, DEAP → session II, DEAP → session III, session I → DEAP, session II → DEAP, and session III → DEAP, were set up to demonstrate that MA-PCA has robust effectiveness on emotion recognition with cross-dataset. For simplicity of expression, session I, session II, and session III in SEED are coded as SI, SII, and SIII, respectively, [for detailed experimental setup information, see the literature Tao et al. (2021)].