Training and validating a Recurrent Neural Network (RNN) model often require a lot of signals and corresponding labels. Manually labeling or interpreting seismic data could be extremely time-consuming and highly subjective. In addition, inaccurate manual interpretation, including mislabeled and unlabeled faults, may mislead the learning process (Pham et al., 2019; Wu et al., 2019; Wu et al., 2020; Alkhalifah et al., 2021; Zhang et al., 2022). To avoid these problems, we create synthetic seismic signals and corresponding labels based on the convolution model for training and validating our RNN model.

According to the convolution model, we know that seismic data are formed of a seismic wavelet and reflectivity: S t = ∑ i = − ∞ ∞ w i r t − i = w t * r t (1) where w t is the wavelet, r t is the reflectivity and S t is the synthetic seismic signal, and t means time.

A ricker wavelet is a good representation of seismic wavelets, so we chose it to generate our signals: R t = 1 − 2 π f t 2 e − π f t 2 (2) where R t is the ricker wavelet and f is the frequency of the ricker wavelet.

The reflectivity is decided by the lithology of the stratum. We can first calculate the impedance sequence Z by density ρ and velocity v of the stratum. Thus, we can calculate reflectivity from Z : r i = Z i + 1 − Z i Z i + 1 + Z i (3) Z i = ρ i v i (4) where Z i is the impedance, r i is the reflectivity, ρ i and v i are the density and velocity of the i -th layer of the stratum.

So, our workflow of creating synthetic seismic signals, see Figure 1, is as follows: we first establish a stratum of randomly distributed sandstone and mudstone, and then we calculate the impedance sequence Z using ρ and v of the sandstone and mudstone. At last, we obtain the synthetic seismic signal using Eq. 1, and the label is the corresponding stratum.

The sampling interval of the Ricker wavelet and impedance sequence is 1 ms, so the sampling interval of the synthetic seismic signal is 1 ms too.

Our self-supervised pre-training is based on a masked autoencoder, which performs well in NLP and computer vision. A masked autoencoder masks random parts from the input signal and reconstructs the missing part. It has an encoder that maps the input signal to a latent representation and a decoder that reconstructs the original signal from the latent representation. The workflow is shown in Figure 2.

As mentioned, a masked autoencoder masks random parts from the input signal and reconstructs them. It is important to choose how to mask random parts. In this paper, we design a method for the seismic signal with reference to Masked Language Model (MLM) in BERT (Devlin et al., 2019). It is as follows.

We randomly choose a part of the signal, and we replace the chosen part with (1) zeros 80% of the time, (2) a random part of this signal 10% of the time, and (3) keep unchanged 10% of the time.

In target reconstruction task design we follow the MLM in BERT (Devlin et al., 2019) closely. According to the ablation over different masking strategies in BERT, the best strategy of masking rates of zero masking, random masking, and no masking are 80%, 10%, and 10%, respectively. The purpose of no masking is to reduce the mismatch between pre-training and fine-tuning as the signal will not be masked during the fine-tuning stage. The mixed strategy of random masking and zero masking will help the model learn better.

Another point of the masked autoencoder is the masking ratio. It is different between different signals, when the signals are highly semantic and information-dense, the masking ratio need could be small (e.g., 10%), but on the contrary, when the signals are low in semantic, it needs to be high (e.g., 75%). For low semantic signals, a missing part can be recovered from neighboring parts without high-level understanding of signals. A high masking ratio can create a challenging self-supervisory task that requires holistic understanding beyond low-level signal statistics. We tried different masking ratios to find out the best option: about 20%.

Masked autoencoders have an encoder and a decoder.

Encoder: An encoder maps the input signal to a latent representation. We know that a seismic signal is a sequence signal, and the RNN model is a kind of sequential model, that is good at extracting information from a sequence signal. So, our encoder is based on Bi-LSTM, a kind of bidirectional RNN. We use a bidirectional RNN because the wavelet has a width, which makes one point of the seismic signal is affected by the above and below points.

Decoder: A decoder reconstructs the original signal from the latent representation. It plays a different role in reconstructing different signals. When the decoder’s output is of a lower semantic level such as pixels, the decoder needs to be designed carefully. This is in contrast to language, where the decoder predicts missing words that contain rich semantic information. So, our decoder is based on Bi-LSTM and MLP. In addition, the decoder is only used during pre-training to perform the signal reconstruction task. Therefore, it is more lightweight than the encoder to make sure effectiveness of the encoder and reduce computation.

The overview of pre-training is shown in Figure 2. The N means the number of layers of the encoder’s Bi-LSTM, and the M means the number of layers of the decoder’s Bi-LSTM. The details of the pre-training model are shown in Table 2.

We pre-train our model using an Adam optimizer (Kingma and Ba, 2017) with β 1 = . 9 , β 2 = . 999 , and a batch size of 1,600. We use a step learning rate decay, see Figure 3. We pre-trained for 200 epochs at 4 masking ratios: .1, .2, .3, and .4. To get a qualitative sense of our reconstruction task, see Figure 4. Our loss function computes the Mean Squared Error (MSE) between the reconstructed and original seismic signals. We compute the loss only on masked parts, similar to BERT (Devlin et al., 2019) and MEA (He et al., 2021), so the model output on unmasked parts is qualitatively worse. So, we overlay the output with the unmasked parts to improve visual quality.

We can fine-tune our pre-trained model to downstream tasks, and, a new decoder for the downstream task is necessary. In this paper, the downstream task is a classification task.

Downstream task: We choose a simple task as our downstream task, so we can pay more attention to the effect of self-supervised pre-training. In the downstream task, our model predicts the sandstone content from the input seismic signal. This is a simple task but also meaningful, we can get an overall understanding of a stratum from the prediction results which can be used as a reference for further work.

The output of our downstream task is the percentage of the Sandstone content, which is suitable for regression. However, we still choose classification because a regression task is harder than a classification task. As mentioned above, we need a simple task to validate our method. In addition, we can calculate the accuracy which can intuitively show the performance of the model.

Our model predicts the sandstone content from the input seismic signal and its decoder is a simple MLP, see Figure 5. The details of the fine-tuning model are shown in Table 3.

We fine-tune our model using again an Adam optimizer (Kingma and Ba, 2017) with β 1 = . 9 , β 2 = . 999 , a batch size of 2,048 and apply a weight decay of .001. We use a step learning rate decay, see Figure 6.

In fine-tuning we try several pre-training models with different masking ratios to find out the best choice of masking ratio. We train our model 300 epochs and record the losses during training, the losses of the last 50 epochs are shown in Figures 7, 8. We find that there is an abrupt behavior for the training and validation losses between epochs 270 and 280. We think this is caused by the an excessively high learning rate for that stage of the training. So, when the learning rate decreases, the problem disappears.

After training, we evaluate the accuracy of the models on the test synthetic dataset, see Figure 9.