A feedforward model, such as a N-layer fully connected DNN, maps an input vector v ₀ to an output vector v _N = f _DNN (v ₀) through repetitive steps of the form is given by v n = α n W n v n − 1 + b n , n = 1,2 , … . N , (1) where v _n−1 is the output of (n − 1)th layer, v _n is the output of nth layer, W _n is the weight matrix, b _n is the bias vector of nth layer, and α _n is the activation layer [21]. The parameters W _n and b _n of the layer are represented by θ n = W n , b n . (2)

The network is capable of approximating nonlinear functions through the use of the activation function α _n , which establishes nonlinear relationships between the layers. The rectified linear unit (ReLU) is a frequently used activation function in modern ANNs. It acts separately on each of its input vector elements by maintaining the positive values and equating negative to zero [35], i.e., z = α _Relu (k) with z i = m a x 0 , k i , (3) where z and k are vectors whose ith elements are represented by z _i and k _i , respectively. The ReLU function has a continuous gradient when compared to other well-known activation functions like the hyperbolic tangent and sigmoid, which makes training computationally less expensive and prevents the effect of vanishing gradients. This impact happens for activation functions with asymptotic behavior because the gradient can shrink and slow the learning algorithm’s convergence.

The softmax activation function is frequently used in the last (decision) layer of an ANN, and the elements z _i of the output z = softmax k) are provided by z i = e x p k i ∑ j e x p k j . (4)

By labeling the training data, the neural network training may be performed under supervision. This establishes a pairing between the intended output vector v ̂ N and input vector v ₀. As a result, the training goal is to minimize the cost function C ( W n , b n ) as a function of the weight matrix, W _n , and bias vector, b _n , of all N layers. The cost function over the set of training inputs M between the DNN output, v _N , and intended output, v ̂ N , is given by C W n , b n = 1 | M | ∑ i = 1 M L v 0 , v ̂ N . (5)

In Eq. 5, L (z, k) represents the loss function and |M| is the training example number. Cross-entropy is the loss function that we used in this work, represented as L k , z = − ∑ i k i ⁡ log z i . (6)

In modern deep learning, error backpropagation allows for efficient computation of the gradient [33]. Adam optimizer, a cutting-edge algorithm with improved convergence, dynamically changes the learning rate η [36]. In this work, training process optimization is carried out using the Adam algorithm, and PyTorch and Keras [37] were used to create all of the numerical results.

An AE is a concept in which data are first encoded into a compressed form and subsequently decoded to revert it to its original shape. This technique, employed in unsupervised learning, involves acquiring a condensed representation of input data through the utilization of an NN. This acquired representation can then be effectively applied to tasks such as reducing noise, compressing data, or extracting features. The autoencoder is composed of three primary components: the encoder section, a code, and the decoder section. The encoder element takes the input data and compresses it into a lower-dimensional form referred to as the encoded data or code. This compressed representation is usually presented as a vector of numerical values, serving as a compact summary of the input data. This implies that the encoder’s function is to compress the data. Within the encoder, one or more hidden layers exist that apply nonlinear transformations to the data, generating the encoded version. The structure of an autoencoder is depicted in Figure 1. Subsequently, the code is input into the decoder phase, which effectively reconstructs the initial data, namely, the input data, from the dimensionally reduced code. This decoder is comprised of one or more concealed layers that undertake the task of transforming the condensed representation back into the original data form. The objective of the decoder’s output is to closely mirror the input data. The primary aim is to restore the original data with minimal loss of information from the compressed representation.

A loss function is employed to minimize disparities between the input and reconstructed data, often using metrics like mean squared error, during the training process of the AE. The weights of both the encoder and decoder are updated through the utilization of backpropagation and optimization algorithms like stochastic gradient descent. Figure 2 illustrates the sequential progression of the autoencoder. Beginning with the input data, the autoencoder undertakes the task of learning how to extract the most pivotal features and subsequently encapsulate them in a streamlined manner. This is achieved by implementing a bottleneck layer within the network’s core, housing fewer neurons in comparison to the input and output layers. Through this method of data compression, the autoencoder gains the ability to understand the process of discarding irrelevant information while concentrating on the most significant and critical features. Upon the successful training of the autoencoder, the code generated by the encoder can be harnessed for various purposes, such as data compression or feature extraction. For instance, if the autoencoder has been trained on images, the compact representation of the image, i.e., the code produced by the encoder, can be applied for tasks like image retrieval or classification.

Autoencoders possess the remarkable capacity to acquire valuable data representations devoid of the necessity for labeled data, which stands out as a significant advantage. This attribute renders them particularly advantageous for tasks where acquiring labeled data is resource-intensive or challenging. Moreover, autoencoders exhibit proficiency in handling high-dimensional data, a domain often fraught with challenges for other categories of machine learning algorithms. Nonetheless, grappling with overfitting remains one of the principal hurdles associated with autoencoders. This occurs when the autoencoder predominantly learns the intricacies of the training data features rather than cultivating valuable, generalizable features suitable for new data instances. To counteract the menace of overfitting, regularization approaches such as dropout techniques and weight decay methods can be harnessed. These strategies serve to maintain the autoencoder’s focus on salient and broadly applicable features, curbing the inclination toward excessive adaptation to training data specifics.

First, we consider the effects of nonlinear phase noise in single-mode optic fiber models. The nonlinear Schrödinger equation (NLSE) is used for modeling the propagation of signals in an optical fiber employing distributed amplification as follows [39]: ∂ k d , t ∂ d = i γ ‖ k d , t ‖ 2 − i β 2 2 ∂ 2 x d , t ∂ d 2 − p d , t . (7)

Here, k (d, t) is the signal, which is transmitted, t is the time coordinate, and d is the distance coordinate; the nonlinearity parameter is represented by γ, β ₂ is the group velocity dispersion (GVD) coefficient, and the Gaussian noise is represented by p (d, t). The right side of the equation has two terms. The equation’s first term shows the Kerr effect, which induces a shift in the phase that is proportional to the signal’s power and leads to a substantial distortion in optical fiber systems. The second term shows dispersion. As the channel likelihood function in [Eq. 7] is unknown, a simple dispersionless channel is considered by ignoring β ₂ in [Eq. 7]. The equation of the model based on recursion is given as follows [40]: k i + 1 = k i e j L o γ | k | 2 / S + p i + 1 , 0 ≤ i ≤ S . (8)

Here, k ₀ = k is the input to the channel consisting of complex values, r = k _S is the output of the channel, p _i+1 ∼ CN(0, P _N /S), fiber link length is represented by L _o , γ is the nonlinearity parameter, and P _N is the noise power. Ideal distributed amplification is assumed by the model and S → ∞.

To obtain numerical results, the fiber model in [Eq. 8] is assumed to have an optical fiber length of L _o = 100 km, a nonlinearity coefficient of γ = 1.27, and a noise power of P _N = −21.3 dBm. The model is iterated S = 50 times for simulation, providing a good approximation of the true asymptotic channel PDF [41].

Using PyTorch’s random number generator, the dataset gets generated inside the network on the fly. The training dataset size is 1.10⁴. The number of Epochs = 120, and the batch size is varied during the training. The size of the validation dataset is 1.10⁵. The cross-entropy loss function and Adam optimizer [36] in PyTorch are used for training the AE separately for various values of P _in . The AE structure parameters for M = 16 are summarized in Table 1.

In Figure 4, the 16-point constellation that has been learned under different P _in for nonlinearity-based fiber optic channels is shown. The points in the learned constellation are equally spaced and pentagonal-shaped at −11 dBm, as shown in Figure 4A. In highly nonlinear regimes, the constellations appear to be random. The constellation points with high energy have varying radii. As a result, after propagation through the nonlinear fiber channel, these points do not overlap with other points. Additionally, the farther the point is from others, the larger its radius.

In a nonlinear channel, when the length of the optical fiber increases, the nonlinearity of the fiber increases as well, leading to signal quality degradation, which can limit the achievable transmission distance and data rate. For obtaining different numerical results, consider the optic fiber model in [Eq. 8] with a nonlinearity coefficient of γ = 1.27, noise power of P _N = −21.3 dBm, and input power of P _in = 2 dBm. Table 1 shows the autoencoder parameters used for different lengths. For training the model, the number of Epochs = 120, and the batch size is varied during the training. In the initial iteration, the batch size is kept small to obtain a working solution. The size of the batch increases as the training nears completion. If the size of the batch is kept small, there will be no misclassifications, and the training will not improve. If the batch size is large, there are higher chances of error in the batch; hence, there will be a reason for the training to keep on improving. The results of training loss curves are shown for different fiber lengths in Figure 5. The loss function called cross-entropy and the optimizer known as Adam [36] are used during the training in PyTorch.

The effect of nonlinearity for different optical fiber lengths, L _o , is shown in Figure 6. The effect of the nonlinear channel on L _o = 5,000 km is shown in Figure 6A. The received constellation is affected by phase noise or phase rotation. The right side of the figure shows the decision regions formed on the basis of the received constellation. When the length is decreased to L _o = 100 km, the nonlinear channel has less impact on the received symbols as compared to L _o = 5,000 km. In other words, the received constellation has less phase noise, and decision regions can easily segregate the received symbols, as shown in Figure 6B. Length is reduced further to L _o = 50 km and L _o = 15 km, as shown in Figure 6C and 6D. The received signal is less affected by the phase noise, and more accurate information is obtained at the receiver side with a reduced symbol error rate. This effect can also be seen in the decision boundaries formed on the received constellation. Therefore, when the fiber length is increased, nonlinearity will affect the received constellation by increasing the phase noise. In other words, received information will be degraded, and the symbol error rate will increase, making it difficult for the receiver to recover the signal information. The symbol error rate on the validation dataset for different optical fibers is shown in Figure 7.

We consider the effects of chromatic dispersion-based channels on the fiber length. First, a discrete Fourier transform (DFT) is used to convert the original signal into the frequency domain. 16-QAM is used as input. It is fed into the channel, and chromatic dispersion is applied to it. To shift to the time domain (TD), the inverse discrete Fourier transform (IDFT) of the signal is taken. The chromatic dispersion in the TD is given as h L o , t = c j D λ 2 L o e j π c D λ 2 L o t 2 . (9)

In the frequency domain, chromatic dispersion will be H L o , ω = e − j β 2 L o 2 ω 2 = e j D λ 2 L o 4 π c ω 2 , (10) where λ is the light’s wavelength, D is the dispersion coefficient of the optical fiber, β ₂ is the group velocity dispersion coefficient, ω is the angular frequency, c is the speed of light, and L _o is the optical fiber length.

To obtain the numerical results, consider D = 17 ps/nm − km, c = 3 × 10⁸ m/s, and P _in = 2 dBm. Table 2 shows the parameters of the NN-based receiver used for the chromatic dispersion-based channel. The model is trained using Keras. During training, the number of Epochs = 50. The loss function known as sparse categorical cross-entropy and optimizer known as Adam [36] are used during the training.

Chromatic dispersion broadens the signal due to which ISI occurs for different optical fiber lengths, L _o , as shown in Figure 8. The effect of chromatic dispersion when L _o = 100 km is shown in Figure 8A. The received 16-QAM constellation is spread out, and the symbol error rate will increase. When fiber length decreases, the effect of chromatic dispersion on the signal is reduced, as shown for L _o = 75 km in Figure 8B. The signal constellation is less spread out, and ISI is lower. Therefore, the symbol error rate will reduce as the fiber length is reduced. Figures 8C and 8D represent the effect of chromatic dispersion when the propagation distance is further reduced to L _o = 50 km and L _o = 20 km, respectively. It can be seen that the received constellation is slightly affected by the effect of chromatic dispersion. Hence, the spreading of the signal constellation is reduced further, which results in a reduction of the symbol error rate, as compared to L _o = 100 km and L _o = 75 km. To conclude, ISI is reduced by decreasing the fiber length in a chromatic dispersion-based channel, resulting in a reduction of the symbol error rate.

In this technique, dispersion is applied to the signal through half of the distance ‘k’, then nonlinearity acts on the middle of the distance, and finally, dispersion acts again on the remaining half of the distance. The operation is shown as [43] X l + k , t = exp k 2 D exp N l + N l + k 2 exp k 2 D X l , t . (12)

Several iterations have to be performed by considering an amplified optical communication system. For each span of the amplifier, dispersion and nonlinearity are mitigated, which leads to high complexity.

To simplify iterative and symmetric SSFM, an assumption is made that nonlinearity acts only at the beginning of every amplifier. So, for each span, only one iteration is performed. Due to this, the complexity of the system is reduced significantly for the mitigation of nonlinear phase noise [44]. However, the performance of noniterative asymmetric SSFM is 2–3 dB poor compared to iterative SSFM. SSFM has a high computational cost of approximately 105 multiplications per symbol per channel. On the other side, it is a quite powerful simulation technique for mitigating both dispersion and nonlinearity [45].

For our numerical simulations, we consider iterative and symmetric split-step Fourier method under different fiber lengths. Consider β ₂ = −20 × 10⁻²⁴ s ²/km, α = 0.2 dB/km, γ = 1.27 W/km, h = 1 km, and P _in = 1 dBm. Here, β ₂ represents the GVD coefficient, nonlinearity coefficient is represented by γ, h is the step size, P _in is the input power, and α is the attenuation. The neural network-based receiver parameters are shown in Table 3. The model is trained using Keras with number of Epochs being 120. A loss function known as sparse categorical cross-entropy is used. Adam optimizer [36] is used for training of the model. The results of different optical fiber lengths, L _o , used in the SSFM-based channel with 16-QAM input, and their decision regions formed are shown in Figure 9.

Figure 9A shows the effect of the SSFM-based channel for L _o = 100 km. A small portion of the received symbols are intermixed with each other; hence, the symbol error rate will be high. For Figure 9B, the received constellations for L _o = 50 km are forming a boundary with each other, and there is little interference between the received symbols. In Figure 9C, the length is reduced to L _o = 25 km. For this length, the received constellation symbols do not interfere with each other and are some distance apart. Figure 9D shows that for L _o = 5 km, the symbol error rate is minimum compared to other lengths, and received symbols are far apart from each other. Therefore, for short propagation distances, the channel effect on the received constellation is small and the symbol error rate is reduced.