The traditional model for linear mixing and separation of signals is that the source signals S = [ s 1 , s 2 , … , s m ] ⊤ and noise N = [ n 1 , n 2 , … , n m ] ⊤ are input into the channel. In this case, the noise is predominantly additive white noise, and the effect of the channel on the mixed signal is abstracted as the matrix A. Thus, the instantaneous mixing model is X = AS + N (in this paper, the noise signal is considered part of the channel’s influence, which is incorporated into A), where X represents the mixed signal observed by the receiver from M sources. Therefore, by estimating the channel matrix, i.e., the separation matrix W, the separation of the mixed signal can be achieved. After passing through the separation matrix, the instantaneous mixing model becomes Y = [ y 1 , y 2 , … , y m ] ⊤ = W X = W A S , where W = A ^⊤, thus enabling the separation of the mixed signal (Cardoso, 1999).

Traditional linear mixing models are suitable for open water environments with high signal-to-noise ratios and low interference. However, in complex maritime environments, signal propagation for USVs is affected by various nonlinear factors, such as multipath effects, Doppler shifts, sea waves, weather changes, and electromagnetic interference between vessels. Therefore, in intelligent USV navigation scenarios, blind source separation often involves nonlinear mixing problems. The source continuous signals S(t) = [s ₁ (t),s ₂ (t),…,s_n (t)] are mixed through a nonlinear system and arrive at the receiver. The resulting received mixed continuous signals are denoted as X(t) = [x ₁ (t),x ₂ (t),…,x_m (t)], and the nonlinear mixing model can be expressed as Equation (1):

where S(t) represents the vector form of n source signals, X(t) denotes the m-dimensional mixed signals, f() is a nonlinear vector function indicating that each observed signal x_i (t) is the result of a nonlinear transformation applied to all source signals, and I() represents the mutual information among the signals.

This paper addresses the limitations of conventional BSS algorithms in processing nonlinear mixing models, including restrictive assumptions on nonlinear mixing, insufficient handling of correlations and dynamic variations among source signals, and high sensitivity to noise and interference. To overcome these issues, a TSeq-GAN network is proposed for the separation of mixed AIS signals under a nonlinear mixing model. The flowchart of the proposed BSS algorithm is presented in Figure 1 . Upon receiving the raw AIS mixed signal with added noise, the signal is normalized to render it suitable for neural network input. The generator then combines the processed signal with random noise and attempts to generate a counterfeit signal resembling the true AIS source signal. Since AIS signals are modulated using GMSK, the GMSKmodulated signal is considered most similar to the source signal; hence, it is used as the real signal, which, together with the generator-produced counterfeit signal, is fed into the discriminator. The discriminator determines whether the generated signal is genuine or counterfeit. Through this adversarial process, the generator continuously refines its network parameters to produce signals that closely approximate the real source signals, while the discriminator simultaneously improves its ability to distinguish between genuine and counterfeit signals. When the discriminator can no longer differentiate between them, the source signal is effectively obtained (Goodfellow et al., 2020).

The traditional GAN generator consists of a multilayer perceptron (MLP) architecture with fully connected neurons, meaning each neuron is connected to all neurons in the preceding layer. This simple fully connected structure limits the MLP’s ability to effectively capture spatial structures or temporal dependencies in sequential data. Therefore, this work introduces CNN and LSTM networks into the generator of the conventional GAN framework to optimize the MLP architecture. The TSeq-GAN network structure is illustrated in Figure 2 . This novel architecture can automatically capture multidimensional temporal and spatial features from the raw input during end-to-end training. These features not only contain information about the source signals but also implicitly represent noise characteristics, thereby naturally achieving a denoising effect during the separation process.In the TSeq-GAN network, the CNN extracts local features of the signal via convolutional kernels, learning features across multiple scales, which facilitates the recognition of AIS signals under low signal-to-noise ratio conditions. Furthermore, the translation invariance and local connectivity of convolution operations help suppress Gaussian and impulse noise. The LSTM, a neural network specialized for sequential data processing, captures temporal correlations between preceding and succeeding signal frames (Hochreiter and Schmidhuber, 1997), compensating for CNN’s lack of temporal memory. More importantly, in the presence of time-slot collisions, the LSTM enhances signal structure modeling and reduces the risk of pseudo-separation.The GAN framework approximates the distributional characteristics of the signals rather than merely fitting point-to-point mappings. Therefore, the proposed TSeq-GAN separation algorithm relies on the CNN front-end to filter out high-frequency noise, the LSTM to model contextual dependencies, and adversarial training in GAN to eliminate pseudo-noise outputs, thereby effectively countering noise interference.

Firstly, the mixed AIS signals are transformed into a matrix format, which, on one hand, enhances the network’s ability to process complex, variable, and noisy signals, and on the other hand, fully leverages the CNN’s local feature extraction, the LSTM’s temporal sequence modeling, and the GAN’s adversarial training capabilities, particularly when dealing with highly nonlinear and multimodal signals. Figure 3 illustrates the internal structure of the generator. The generator receives as input two components: a random noise vector Z ( Z ∈ ℝ ( d s ) ) and a mixed signal matrix S ( S ∈ ℝ ( N * N ) ) .The matrix S is flattened into a vector S ∈ ℝ ( N * N ) , which is then concatenated with the random noise vector S, resulting in the combined input, as shown in Equation (2) :

The combined input is fed into the generator, where random noise is introduced to enhance the diversity of the generated signals, while the mixed signal provides useful features to guide the generator’s learning. The combined signal I is expanded into a three-dimensional vector I_3D and input into the CNN layer. The CNN layer consists of four 1D convolutional neural network layers. The convolution operation in the first layer is given by the following Equation (3) :

the output size of the convolution operation is given by X ( l ) ∈ ℝ N × C ( l ) × T ( l ) , where C ( l ) represents the number of channels and T ( l ) is the temporal length, such that X C N N ∈ ℝ N × C 128 × T . In the first convolutional layer, the input channel is set to 1, the output channel is 16, and the kernel size is 1 ∗ 3 to extract basic low-level local temporal features. In the subsequent layers, the input channels are sequentially 16, 32, and 64, and the output channels are 32, 64, and 128, respectively. All convolutional layers use a kernel size of 1 ∗ 3. These three convolutional layers progressively increase the abstraction level of the features, capturing higher-level signal patterns. After each convolutional layer, LeakyReLU, as shown as Equation (4) is used as the activation function, with the following formula for LeakyReLU:

this activation function introduces non-linearity while retaining negative value information. After feature extraction, to adapt to the LSTM layer, X_CNN is transposed to a dimension of N × T × 128, as shown in equation: X ( L S T M − i n ) = X C N N ⊤ ∈ ℝ N × T × 128 .

The input X is fed into an LSTM layer with 256 hidden units. The LSTM layer is used to process the high-dimensional features extracted by the convolutional layers, leveraging its memory mechanism to capture long-term dependencies. The two stacked LSTM layers provide powerful temporal modeling capabilities. The temporal data processed by the LSTM is given by Equation (5):

where h t ∈ ℝ 256 represents the hidden state used to capture temporal features, c_t represents the current cell state, h t − 1 represents the hidden state from the previous time step, and c t − 1 represents the cell state from the previous time step. The output of the LSTM layer is the hidden state at the final time step, h final = h T ∈ ℝ N × 256 ,which represents the global feature of the entire sequence, where h_T is the hidden state at the last time step.

The output of the LSTM layer is passed through two fully connected layers. The first layer maps the high-dimensional features, while the second layer outputs signal data that matches the dimensionality of the real source signal. Each fully connected layer uses the Tanh activation function to constrain the output within the range of [−1,1], simulating the characteristics of the real signal. The Equation (6) for the Tanh function is:

the Tanh activation function normalizes the amplitude of the generated signal, facilitating the comparison with the actual target signal for loss computation. The output h_final from the LSTM layer is mapped through the fully connected layer as Equation (7):

where w ₁ and b ₁ are the weight matrix and bias term of the first fully connected layer, and w ₂ and b ₂ are the weight matrix and bias term of the second fully connected layer, The signal network feature extraction is shown in Figure 4 .

The discriminator adopts a straightforward architecture: real and generated signals are both input and then passed through three fully connected layers that progressively reduce the feature dimensionality. The first layer projects the input into a 256-dimensional feature space; the second layer further compresses it to 128 dimensions; and the third layer outputs a single probability, representing the confidence that the input is a genuine AIS signal.In this work, we use the AIS physical-layer Gaussian Minimum Shift Keying (GMSK) waveform as the “real” signal in the discriminator alongside the generator’s synthetic outputs. This design is justified by two factors. First, the AIS protocol uniformly employs 9.6 kbps GMSK modulation at the physical layer, so using the native GMSK waveform as the ground truth guarantees that our evaluation metrics align exactly with the operational AIS standard. Second, GMSK’s continuous-phase nature—characterized by a constant envelope, high spectral efficiency, and robustness to multipath—ensures that the baseband waveform faithfully preserves the signal’s critical time- and frequency-domain features. Consequently, our chosen metrics can accurately quantify deviations between the separated output and the standard modulated signal.For nonlinear representation, each hidden layer uses a LeakyReLU activation, and the final output layer applies a Sigmoid function to yield a [−1,1] probability for binary classification.

Finally, we employ Mean Squared Error (MSE), Signal-to-Interference Ratio (SIR), and continuous Mutual Information (cMI) as objective metrics to assess the quality of the separated signals.

The loss function of TSeq-GAN consists of both generator and discriminator losses. The generator loss comprises adversarial loss, mean squared error loss, interference suppression loss, and statistical correlation loss. The adversarial loss is the conventional GAN loss function that drives the adversarial game between the generator and the discriminator, thereby improving the quality of the generated model. The discriminator’s output, confined within the range [0,1], represents an evaluation of the “realness” of the input samples; hence, the cross-entropy loss function is employed.

To meet the demands of multi-source sensing in scenarios with high traffic density and suboptimal communication conditions for USVs, the generated samples must closely approximate the distribution of real samples. Therefore, this paper implements a multidimensional constraint mechanism within the generator by jointly introducing mean squared error loss, interference suppression loss, statistical correlation loss, and adversarial loss. The overall loss is then backpropagated to update the generator’s training parameters. The integration of these multidimensional constraints optimizes the generator’s performance from multiple perspectives, thereby enabling high-quality BSS of AIS signals.The network adversarial loss in this algorithm is defined as Equation (8):

Here, z denotes the noise distribution, G(z) represents the sample generated by the generator G upon receiving z, and D (G(z)) is the discriminator D’s output for G(z), which is a probability value ranging from 0 to 1. The generator aims to minimize the discriminator’s ability to distinguish the generated samples, driving the output of the generated samples as close to 1 as possible. The mean squared error (MSE) loss is defined as Equation (9):

where y_i represents the GMSK signal. The interference suppression loss is defined as Equation (10):

where∥·∥₂ represents the Euclidean norm, and ϵ is set to a small constant to prevent division by zero. The statistical correlation loss is defined as Equation (11):

where, p ( y i , G ( z ) ) is the joint probability density function of y_i and G(z), while p(y_i ) and p(G(z)) are their respective marginal probability density functions. In the simulation program, the mutual-inforegression function from scikit-learn is used to estimate the mutual information between continuous variables. Different loss components in the loss function may have varying numerical scales; if one loss term has a significantly larger magnitude, it could lead the model to overly optimize that specific objective during convergence, thereby neglecting other objectives and causing convergence difficulties or instability during training. To meet the requirements of multi-objective tasks and ensure training stability, optimal weight parameters for each loss term have been determined through parameter tuning experiments. The overall loss calculation for the generator network is detailed as follows as Equation (12):

Since the discriminator’s objective is to distinguish real samples from generated samples as accurately as possible, this study employs the cross-entropy loss function to compute the loss for real samples L_real as shown as equation (14), and generated samples L_fake as shown as equation (15), separately, and then sums them to obtain the total discriminator loss L_D as Equation (13):

This algorithm was trained for 1000 epochs on a computer equipped with a 2.40 GHz Intel I5-9300H CPU, GTX 1650 GPU, and 8GB of RAM, requiring approximately 86.1667 minutes. During the initialization and preparation stages, the Adam optimizer was used to update the parameters of both the generator and the discriminator, with an initial learning rate set to 0.0002. The training loop consisted of multiple epochs, and within each epoch, both the discriminator and the generator were trained in batches of 128.

In the experiments, specific weight parameters were assigned to each component of the generator loss. The weight λ ₁ for the MSE loss was set to 0.4 to ensure that the generator prioritizes the accuracy of signal generation, making the generated signal as similar as possible to the target signal. The weight λ ₂ for the interference suppression loss was set to 0.01 to prevent the loss function from overemphasizing SIR, thereby avoiding the excessive optimization of SIR at the expense of other loss components. The weight λ ₃ for the statistical correlation loss was set to 0.5, which helps the generated signal capture more signal structure and features rather than merely minimizing the MSE. Finally, the weight λ ₄ for the adversarial loss was set to 0.05 to balance the training process, ensuring that the generator does not merely produce signals that satisfy the discriminator without being close to the actual source signals.

Because fixed datasets contain only a limited set of message formats, coding schemes, and channel conditions, models trained on them tend to overfit to these “seen” samples. Although the AIS message format and modulation are standardized, the VHF channel is subject to multidimensional variation—multipath fading, noise, interference, antenna orientation, etc.—that changes dynamically in both space and time. Consequently, a static dataset cannot fully represent the true transmission environment. Moreover, the GAN-based separation network proposed here is prone to mode collapse if the training data lack diversity; in contrast, a dataset with richer variability forces the generator to “cover” a wider distribution of real signals and reduces the risk of memorizing specific examples, yielding a more robust mapping.

To this end, our AIS signal corpus is generated via a stochastic process that introduces maximal diversity in message content, temporal structure, channel conditions, and noise statistics. Each signal instance begins with a binary payload produced by a pseudo-random number generator (PRNG), which is then encoded and modulated using Gaussian Minimum Shift Keying (GMSK). The modulated symbols are concatenated with a fixed control sequence to create distinct message payload structures. To emulate realistic transmission impairments, we inject additive white Gaussian noise (AWGN) at randomly selected SNR levels and simulate channel mixing via randomly generated noise matrices. As a result, the dataset exhibits inherent randomness at every generation and mixing stage, ensuring that each simulation run produces novel signal realizations that satisfy the prerequisites of blind source separation.To evaluate robustness, we benchmark the separation performance under a channel SNR of 5 dB. Furthermore, to assess the network’s behavior in dense maritime traffic scenarios, we conduct experiments with varying numbers of simultaneously mixed signals, thereby validating the algorithm’s ability to disentangle AIS streams under different vessel-density conditions.

The baseband signal, after undergoing HDLC bit stuffing, synchronization-sequence insertion with buffering, and NRZI encoding, is then passed through a Gaussian low-pass filter to impart Gaussian characteristics. The time-domain impulse response of the Gaussian low-pass filter as Equation (16):

where σ represents the standard deviation of the Gaussian function, which determines the extent of the pulse response’s spread. After passing through the Gaussian low-pass filter, the signal is modulated using Minimum Shift Keying (MSK) to generate the AIS signal. The time-domain impulse response of the Gaussian low-pass filter as Equation (17):

where f_c is the carrier frequency and ϕ(t) is the instantaneous phase of the signal. The carrier frequency band for AIS signal modulation are 161.975MHz (international standard channel) and 162.025MHz (some countries have expanded their channels), and in this paper, the carrier frequency is set to 161.975MHz. Therefore, the GMSK signal can be represented as Equation (18):

where the modulation index h is used to characterize the modulation intensity of the signal, b(t) represents the baseband signal, and the pulse shaping function h_g (t) is used to filter the baseband signal, thereby limiting the signal bandwidth and controlling its signal characteristics.

To simulate the unpredictable Gaussian white noise interference in real channel environments, the signal data is processed through a Gaussian noise network for noise addition. Figure 5 illustrates the baseband AIS signal, Figure 6 shows the GMSK-modulated signal, and Figure 7 the schematic illustration of the modulated AIS signal mixture (when there are four signal sources). Figure 8 depicts the schematic illustration of the AIS signal mixture after noise interference (four signals combined).

The comparative experiments are conducted in scenarios where the channel conditions for USV navigation are poor, in order to verify the separation performance of different algorithms for source signals. In this study, the FastICA, JADE, SIC, Higher-order statistics algorithms, AGAN, and TSeq-GAN networks are sequentially applied to separate blind signals with 4, 6, 8, and 10 mixed source signals. Meanwhile, MSE, SIR, and cMI are used as evaluation metrics for the performance of the algorithms.

Figure 9A illustrates the changes in MSE for mixed signals separated by six different algorithms after mixing with varying numbers of source signals. Overall, the error between the mixed signals and the source signals increases as the number of source signals rises. The TSeq-GAN network proposed in this study shows MSE values of 0.1488, 0.1627, 0.1876, and 0.2321 for 4, 6, 8, and 10 mixed source signals, respectively, with a reduction of at least 35.5252% in MSE compared to the mixed signals. Figure 9B shows the changes in SIR for mixed signals separated by the six algorithms. Generally, the coherence of the separated signals decreases as the number of mixed signals increases. The TSeq-GAN network proposed in this study achieves coherence values of 9.3806, 8.0132, 7.8426, and 7.0178 for 4, 6, 8, and 10 mixed source signals, respectively, with an improvement of at least 1.2287 times in the signal coherence. Figure 9C illustrates the changes in cMI for mixed signals separated by the six algorithms. Overall, the similarity between the separated signals and the source signals decreases as the number of mixed signals increases. The TSeq-GAN network proposed in this study achieves cMI values of 2.1801, 1.6323, 1.4221, and 1.2073 for 4, 6, 8, and 10 mixed source signals, respectively, with a 3.5084-fold increase in continuous mutual information. Therefore, based on the robustness reflected by MSE, SIR, and cMI evaluation metrics, the TSeq-GAN network outperforms the other algorithms in signal separation.

The experimental results show that the TSeq-GAN network continuously adjusts the model through endto-end joint optimization, rather than relying solely on analytical derivation or local optimization. Unlike other algorithms in the comparative experiments, the CNN and LSTM structures within the TSeq-GAN network effectively capture local temporal dependencies and long-term relationships in the random signals. The adversarial mechanism between the generator and discriminator allows for adaptive model adjustment. This network structure and joint optimization mechanism enable better extraction of features from random signal sequences, making the algorithm highly robust and generalized.

In this section, we conduct an in-depth investigation into the influence of each module within the TSeqGAN network on the performance of the separation algorithm. The experiments are carried out under a SNR of 5 dB. By selectively removing different layer structures in the TSeq-GAN network, we verify that each module employed in this work contributes significantly to enhancing the network’s overall performance.

The experimental data are divided into three parts for verification. The first part involves a source signal mix of 3–4 signals, representing a scenario where the USV operates in a open and sparse water environment. The experimental results are shown in Table 1 . The second part involves a source signal mix of 5–6 signals, representing a scenario where the USV operates in a low-traffic-density water environment. The experimental results are shown in Table 2 . The third part involves a source signal mix of 7–8 signals, representing a scenario where the USV operates in a medium-traffic-density water environment. The experimental results are shown in Table 3 . The fourth part involves a source signal mix of 9–10 signals, representing a scenario where the USV operates in a crowded water environment. The experimental results are shown in Table 4 . MSE is used to measure the error between the generated signal and the true signal, SIR is used to assess the strength of the separated source signals, and cMI is used to evaluate the similarity between the extracted signals and the true signals.

In the open water environment, this network reduces MSE by at least 28.1267%, improves SIR by at least 15.1262%, and enhances cMI by at least 8.9196 compared to the other three networks. In lower traffic density water environments, the network reduces MSE by at least 19.8408%, increases SIR by at least 12.1445%, and improves cMI by at least 1.752% compared to the other three networks. In moderate traffic density water environments, the TSeq-GAN network outperforms the others by reducing MSE by at least 9.841%, increasing SIR by at least 13.1603%, and improving cMI by at least 4.1104%. Similarly, when the USV operates in crowded water areas, the network shows significant improvements in MSE, SIR, and cMI compared to the other three networks, with MSE decreasing by at least 27.9242%, SIR increasing by at least 10.0328%, and cMI improving by at least 16.962%.

As demonstrated by some of the experiments, the multi-level feature structure of the TSeq-GAN network proposed in this study enables the network to simultaneously handle both short-term and long-term dependencies of signals, while adaptively learning to generate separation results that align with the true distribution of the signals. This multi-level feature extraction and generation capability makes the model more powerful in BSS tasks compared to other independent network structures. Furthermore, through adversarial training, the TSeq-GAN network can effectively learn the separation characteristics of signals even in high-noise environments, complex interference, and nonlinear signal mixtures, ensuring that the model possesses good robustness and generalization capabilities. In contrast, other independent network structures are relatively weaker in handling complex noise and interference within the signals.

Due to the significant efficiency differences among various network architectures, this section presents a time performance comparison experiment between the TSeq-GAN network and other networks, such as the Attentional GAN, when the number of mixed sources is 4, as shown in Table 5 .

Although the TSeq-GAN network is not the most optimal in terms of computational efficiency, its signal extraction accuracy improves by up to 70.7662%, the quality of generated signals improves by up to 1.2287 times, and the similarity with the source signals increases by up to 3.5084 times. Considering the complex scenarios that USVs encounter during autonomous navigation, which are subject to varying degrees of noise, interference, and multipath effects, the accuracy of AIS signal separation directly influences the USV’s ability to assess the positions of surrounding vessels. Therefore, while timeliness is important for real-time signal processing, in BSS tasks in this scenario, if the accuracy of the algorithm cannot be guaranteed, the results of real-time processing will be meaningless.

Figures 10A, B illustrate the variations in MSE, SIR, and cMI, respectively, as the loss weights are adjusted. In this experiment, a global sensitivity analysis method is employed. Based on the correspondence between the magnitudes of the different loss functions and the overall performance, parameters λ ₁ and λ ₄ are adjusted simultaneously to observe changes in the overall model performance, while parameters λ ₂ and λ ₃ are adjusted concurrently to assess the overall performance variations. Consequently, a total of four sets of comparative experiments are conducted: in one scenario, with λ ₁ and λ ₄ held constant, one group increases λ ₂ and λ ₃ to 0.05 and 0.7, respectively, while another group increases them to 0.1 and 0.9; in another scenario, with λ ₂ and λ ₃ fixed, one group decreases λ ₁ and λ ₄ to 0.05 and 0.005, respectively, and another group increases them to 0.1 and 0.01, respectively.