The Multi-Scale Feature Extractor (MSFE) is a critical component for processing remote sensing images at various scales. These multi-scale inputs are denoted as { X ( s ) } s = 1 S , where each X ( s ) represents an image at scale s . For each scale s , the MSFE utilizes a shared Vision Transformer (ViT) backbone, allowing the model to process different scale inputs while maintaining computational efficiency. The shared architecture enables the extraction of scale-invariant features, which are crucial in remote sensing tasks due to the diverse resolutions present in such images.

The feature extraction process for each scale s can be formalized as follows. Given an input image X ( s ) , the MSFE processes it through a feature extractor ϕ x , which is parameterized by θ x ( s ) for each scale. The result is a fixed-dimensional vector z ( s ) , which encodes the scale-specific information (Formula 2): z s = ϕ x X s ; θ x s (2)

This embedding z ( s ) contains the key features extracted from each image at its corresponding scale.

The backbone of the MSFE is based on a shared transformer architecture, inspired by Vision Transformers (ViTs). This shared transformer consists of multiple stages, as illustrated in Figure (a). At each stage, the input image is progressively reduced in resolution through a Patch Embedding layer, while the number of feature channels is increased. The transformer architecture processes these embedded patches through a set of shared transformer blocks at each stage (As shown in Figure 2).

The transformer operations in the shared layers can be mathematically described as follows. For a given input sequence X patch , the self-attention mechanism computes a weighted sum of all positions (Formula 3): Z attn = Softmax Q K T d k V (3) where Q , K , and V represent the queries, keys, and values, which are linear transformations of the input patches. After the attention calculation, a feed-forward network (FFN) is applied to each position in the sequence (Formula 4): Z ′ = FFN Z attn (4)

This process is repeated for N transformer blocks at each stage, progressively refining the features as the resolution decreases, but the number of channels increases.

In addition to the shared transformer architecture, the MSFE incorporates a multi-scale attention mechanism, as depicted in Figure (b). The attention mechanism operates over windowed patches of the image. The pooling operation first divides the image into default windows. These windows are projected into a latent space, where the spatial relationships between the patches are captured by a spatial transform mechanism.

The attention mechanism for a window is given by Formula 5: Z w = Softmax Q w K w T d k V w (5)

The spatial transformation adjusts the window positions and allows the model to integrate information from multiple scales, ensuring that features from different regions of the image are processed appropriately.

Finally, the output features from the MSFE are passed through multiple layers, as illustrated in Figure (c). These layers include layer normalization, multi-head attention, and feed-forward layers. The final output Z l is the representation used for task-specific outputs, such as classification, segmentation, or detection.

The final output Z l is computed through repeated applications of the following operations (Formula 6): Z l = FFN Norm X + VSA X (6)

Here, VSA refers to the Vision Self-Attention module, and FFN represents the feed-forward network. The normalized outputs are added to the input via a residual connection to stabilize training. The output Z l is passed to a task-specific classification head for downstream tasks (As shown in Figure 3).

The MSFE leverages a shared transformer architecture across multiple scales to efficiently capture features at different levels of resolution. The multi-scale attention mechanism further enhances the model’s ability to process complex, large-scale remote sensing data.

The Adaptive Consistency Module (ACM) is a key component of the Adaptive Multi-Scale Consistent Network (AMSCN), designed to ensure consistency across features extracted from multiple scales. In remote sensing or vision tasks where images can be captured at different resolutions, it becomes crucial to align feature representations across these scales. The ACM achieves this by introducing both a scale-consistency loss and a scale attention mechanism, which dynamically adjusts the importance of different scales based on their relevance to the prediction task.

The primary function of the ACM is to enforce consistency between features extracted from different scales. This is accomplished by minimizing the discrepancy between feature representations from distinct scales. To achieve this, the ACM introduces a scale-consistency loss, denoted as L scale , which encourages features from different scales to be similar while maintaining the ability to differentiate scale-specific information when necessary.

Given the feature representations z ( s ) and z ( s ′ ) for two different scales s and s ′ , the scale-consistency loss is defined as the mean squared error (MSE) between these features (Formula 7): L scale = 1 S S − 1 ∑ s ≠ s ′ z s − z s ′ 2 (7) where S denotes the total number of scales. This loss encourages the network to align the features across scales by penalizing differences between the features extracted from any two scales. The normalization factor 1 S ( S − 1 ) ensures that the scale-consistency loss is independent of the number of scales.

This formulation promotes the learning of robust, scale-invariant features while still allowing the model to capture unique scale-specific information as needed for particular tasks.

In addition to ensuring feature consistency across scales, the ACM dynamically adjusts the importance of each scale during the feature fusion process through a Scale Attention Mechanism. This mechanism computes attention scores for each scale, allowing the model to emphasize the most relevant scale for a given input and task. The scale attention score α ( s ) for scale s is computed using the following softmax formulation (Formula 8): α s = exp W a z s ∑ s ′ = 1 S ⁡ exp W a z s ′ (8) where W a is a learnable weight matrix applied to the feature representation z ( s ) of scale s . This weight matrix transforms the features into a score space, which is then normalized using the softmax function to obtain the attention weights α ( s ) . These attention weights determine the contribution of each scale to the final feature representation.

Once the attention scores α ( s ) are computed for each scale, the final scale-consistent feature representation is obtained by taking a weighted sum of the scale-specific features (Formula 9): z final = ∑ s = 1 S α s z s (9)

Here, z ( s ) represents the feature vector corresponding to scale s , and α ( s ) is the attention score computed for that scale. This weighted combination allows the network to adaptively focus on the most relevant scales while still leveraging information from all scales. By dynamically adjusting the importance of each scale based on the input, the ACM ensures that the final feature representation z final is both robust and flexible, capturing important multi-scale patterns.

To improve the robustness of the ACM, additional regularization terms can be introduced to further align the features across scales while preserving discriminative power. One such regularization term can be the inter-scale diversity loss, which encourages diversity between the feature representations at different scales. This can be defined as Formula 10: L div = 1 S S − 1 ∑ s ≠ s ′ 1 − z s ⋅ z s ′ ‖ z s ‖ ‖ z s ′ ‖ (10)

This term ensures that while the features from different scales are aligned, they still maintain a level of diversity, which is crucial for capturing unique scale-specific information. By combining the scale-consistency loss L scale with the inter-scale diversity loss L div , the model can achieve a balanced representation that is both consistent and diverse across scales.

The Cross-Modality Fusion Layer (CMFL) is a crucial component of the Adaptive Multi-Scale Consistent Network (AMSCN) that integrates scale-consistent visual features with contextual information from associated textual data. In applications such as remote sensing, visual data (e.g., satellite images) often need to be complemented with textual information (e.g., crop reports, weather conditions, or geographic descriptions). The CMFL is designed to perform this cross-modal fusion effectively, using a transformer-based approach to align and merge the information from these two modalities.

The textual data, denoted as T , is first processed by a transformer-based text encoder ϕ t ( T ; θ t ) , parameterized by θ t . This encoder extracts meaningful representations from the text, transforming the input textual sequence into a set of feature vectors. The output of the text encoder is a sequence of textual features h t ( j ) , where j indexes the tokens in the textual sequence. Formally, this process can be written as Formula 11: h t = ϕ t T ; θ t (11) where h t = h t ( 1 ) , h t ( 2 ) , … , h t ( | T | ) and | T | is the length of the textual sequence.

The CMFL employs a cross-attention mechanism to align the visual features, extracted by the visual backbone, with the contextual information from the textual data. The goal is to allow the model to focus on relevant text features for each visual feature. The visual features, denoted as z final ( i ) , are the scale-consistent features obtained from the Multi-Scale Feature Extractor (MSFE). The cross-attention mechanism computes an alignment score between each visual feature and each textual feature.

Let q i and k j represent the query vector for the i -th visual feature and the key vector for the j -th textual feature, respectively. These are computed as follows (Formula 12): q i = W q z final i , k j = W k h t j (12)

Here, W q and W k are learnable weight matrices used to project the visual and textual features into a shared latent space. The cross-attention score β j ( i ) between the i -th visual feature and the j -th textual feature is then computed using the dot product followed by a softmax normalization (Formula 13): β j i = exp q i ⊤ k j ∑ j ′ = 1 | T | ⁡ exp q i ⊤ k j ′ (13)

This attention score represents the relevance of the j -th textual feature for the i -th visual feature, allowing the model to attend to the most relevant parts of the text for each visual feature.

Once the cross-attention scores β j ( i ) are computed, the final fused feature for the i -th visual feature is obtained by taking a weighted sum of the value vectors corresponding to the textual features. The value vector v j for each textual feature is computed as Formula 14: v j = W v h t j (14) where W v is another learnable weight matrix. The final fused feature h fused ( i ) for the i -th visual feature is then obtained by summing the value vectors weighted by the cross-attention scores (Formula 15): h fused i = ∑ j = 1 | T | β j i v j (15)

This fusion process ensures that each visual feature is enhanced by the relevant textual information, resulting in a more contextually informed representation.

After the cross-modality fusion, the fused features h fused ( i ) are passed through a final prediction layer to generate the output for the task at hand. For example, in the context of predicting the probability of double-cropped soybeans in a target area, a binary classification layer can be applied, resulting in a predicted probability y ̂ .

The overall training objective for the AMSCN involves minimizing a combined loss function, which consists of three main components: Prediction loss ( L pred ) : This is the binary cross-entropy loss for the prediction task, which penalizes incorrect predictions of the target label. - Scale-consistency loss ( L scale ) : This loss ensures that features from different scales are aligned and consistent. - Cross-modality alignment loss ( L cross ) : This loss encourages effective alignment between the visual and textual features during the cross-attention fusion process.

The total loss function is expressed as Formula 16: L AMSCN = L pred + λ L scale + γ L cross (16) where λ and γ are hyperparameters that control the relative contributions of the scale-consistency and cross-modality alignment losses, respectively. These hyperparameters can be tuned based on the specific task and dataset to achieve optimal performance.

The AMSCN (Attention-based Multi-Scale Convolutional Network) employs a hybrid learning strategy that leverages both supervised learning and self-supervised learning (SSL) to maximize the effective use of labeled and unlabeled data. This combination allows the model to excel in scenarios where labeled data is limited, which is often the case in remote sensing applications. By utilizing this dual approach, the model can improve its generalization and robustness across varying geographical and climatic conditions, crucial for tasks like identifying suitable regions for double-cropping soybeans.

The supervised component of this hybrid strategy is guided by the binary cross-entropy loss function, denoted as Formula 17: L pred = − 1 N ∑ i = 1 N y i ⁡ log y ̂ i + 1 − y i log 1 − y ̂ i , (17) where y i represents the ground truth label indicating whether a region is suitable for double-cropping, and y ̂ i is the model’s prediction for the i -th sample in the dataset D , which consists of N labeled samples. This loss function trains the model to effectively classify regions into suitable or unsuitable categories based on the available labeled data.

In contrast, the self-supervised learning (SSL) component uses a masked image modeling (MIM) strategy inspired by the Masked Autoencoder (MAE) framework. The goal of MIM is to learn a rich and robust set of feature representations from unlabeled data by exploiting the inherent structure of the remote sensing imagery. In this approach, portions of the input image are randomly masked, and the model is tasked with reconstructing the missing parts using the visible portions of the image, thereby encouraging the model to learn the underlying patterns and semantics.

The self-supervised loss function, L SSL , is formulated as (Formula 18): L SSL = 1 M ∑ i = 1 M X masked i − X ̂ reconstructed i 2 , (18) where X masked ( i ) represents the masked version of the i -th input image, X ̂ reconstructed ( i ) denotes the corresponding reconstruction produced by the model, and M is the total number of masked samples used for self-supervised learning. This reconstruction process helps the model capture meaningful feature representations from the raw imagery, which is particularly valuable in scenarios where obtaining labeled data is expensive or time-consuming.

By integrating both supervised and self-supervised objectives into the training process, the AMSCN effectively learns from a mix of labeled and unlabeled data. The overall loss function for training the model can thus be expressed as a weighted sum of the two components (Formula 19): L total = γ L pred + δ L SSL , (19) where γ and δ are hyperparameters that balance the contributions of the supervised and self-supervised losses during training. This combination enables the model to generalize better across different environments and enhances its performance in real-world applications, particularly in cases where the availability of labeled data is limited, but large volumes of unlabeled remote sensing data are accessible.

To improve the resilience of the AMSCN (Attention-based Multi-Scale Convolutional Network) against the noise and uncertainties often present in remote sensing data, we incorporate several robust optimization techniques into the training process. These techniques are essential for ensuring that the model can generalize well to new, unseen conditions and maintain high performance even in the presence of noisy or corrupted input data. A key method utilized in this context is adversarial training, a strategy designed to improve the model’s robustness by exposing it to deliberately perturbed input data.

Adversarial training operates by introducing adversarial noise, denoted as n , into the original input images. These perturbed inputs are referred to as adversarial examples and are generated by adding the noise n to the original input images X , yielding adversarial inputs X adv = X + n . The perturbation n is carefully crafted to maximize the model’s prediction error, typically by following the gradient of the model’s loss with respect to the input data. This adversarial noise is often subtle enough to be imperceptible to human observers but can significantly impact the model’s predictions.

Formally, the adversarial training objective can be expressed as Formula 20: L adv = 1 N ∑ i = 1 N L pred f θ X i adv , T i , y i , (20) where L pred is the binary cross-entropy loss function used for the main classification task, f θ denotes the model with parameters θ , X i adv represents the adversarial input for the i -th sample, T i is the associated temporal data or additional features (such as climatic or geographical information), and y i is the ground truth label for the i -th sample in the dataset. The objective of adversarial training is to minimize the prediction error on these adversarial examples, thus forcing the model to become more robust to small, strategically designed perturbations in the input data.

The adversarial noise n is typically generated by maximizing the loss function with respect to the input, using a method such as the Fast Gradient Sign Method (FGSM), which computes n as follows (Formula 21): n = ϵ ⋅ sign ∇ X L pred f θ X , T , y , (21) where ϵ is a small scalar controlling the magnitude of the perturbation, ∇ X is the gradient of the loss function with respect to the input image X , and sign ( ⋅ ) denotes the sign of the gradient. This perturbation is added to the input image X to generate the adversarial example X adv , and the model is then trained to correctly classify these perturbed inputs.

By incorporating adversarial training, the AMSCN learns to be less sensitive to small, potentially adversarial changes in the input data, enhancing its robustness and generalization capabilities. This approach is particularly valuable in remote sensing tasks, where data can be subject to various sources of noise, such as sensor errors, atmospheric conditions, and data preprocessing artifacts. The adversarial training process ensures that the model develops feature representations that are more stable and less influenced by these noise sources.

Moreover, the total training loss for the AMSCN can be modified to include both the standard prediction loss and the adversarial loss, leading to an overall objective function defined as (Formula 22): L total = μ L pred + ν L adv , (22) where μ and ν are weighting coefficients that control the relative importance of the standard and adversarial losses during training. By balancing these two components, the AMSCN can learn to perform well on both clean and adversarial examples, resulting in a more robust and resilient model capable of handling noisy or uncertain input data in real-world remote sensing applications.

Another strategic enhancement involves the use of ensemble learning to quantify and reduce prediction uncertainty. We train multiple instances of the AMSCN with varying initializations and hyperparameters, generating an ensemble of models { f θ ( k ) } k = 1 K . The final prediction is obtained by averaging the outputs of the ensemble models (Formula 23): y ̂ ensemble = 1 K ∑ k = 1 K f θ k X , T , (23) where K is the number of models in the ensemble. This ensemble approach not only improves the overall predictive performance but also provides a measure of uncertainty in the predictions, which is crucial for decision-making in agricultural planning. The variance among the predictions from different ensemble members serves as an indicator of uncertainty, allowing stakeholders to assess the confidence in the model’s predictions.

To understand the contribution of each component of the AMSCN, we conduct an ablation study by systematically removing or altering key components of the model. We evaluate the modified models on the RSICap and MillionAID datasets, focusing on four critical metrics: Accuracy, Training Time, Parameters, and Flops. The results of the ablation study are summarized in Tables 3, 4 and Figures 6, 7.

The results of the ablation study, summarized in Tables 3, 4, provide valuable insights into the contributions of each component of the AMSCN model. The removal of the scale consistency mechanism resulted in a significant drop in accuracy and recall, emphasizing the importance of this mechanism for achieving high segmentation performance. For instance, on the RSICap dataset, accuracy dropped from 97.54% to 88.32% when the scale consistency mechanism was omitted, as shown in Table 3. Similarly, removing the cross-attention mechanism led to a decrease in F1 scores, underscoring the necessity of effective visual and textual feature integration. Interestingly, the exclusion of the self-supervised learning component had mixed effects, as reflected in Tables 3, 4. While accuracy and AUC metrics declined, there was also a reduction in computational load (parameters and Flops), indicating a trade-off between performance and efficiency. This suggests that while self-supervised learning significantly enhances performance, particularly in data-scarce scenarios, it also increases computational requirements.

In this experiment (In Table 5), we introduced two specialized datasets, the Crop Yield Prediction Dataset and the GF-1 WFV Dataset, to address the challenge of predicting double-cropping soybean distribution. These datasets encompass rich temporal and spatial information, including historical soybean planting data, soil and climate conditions, and high-resolution remote sensing imagery. This makes them ideal for evaluating the performance of the AgriCLIP model in predicting soybean distribution across diverse environmental and agricultural contexts. Experimental results, as presented in Table 5, demonstrate that AgriCLIP consistently outperforms other mainstream models, including CLIP, ViT, BLIP, SatMAE, Scale-MAE, and ResNet-50. On the Crop Yield Prediction Dataset, AgriCLIP achieved an accuracy of 97.84 percent, a recall of 95.27 percent, an F1 score of 93.72 percent, and an AUC of 95.18 percent. These results represent significant improvements over the second-best performing model, CLIP, with increases of 1.81 percent in accuracy, 9.85 percent in recall, 4.02 percent in F1 score, and 3.42 percent in AUC. This substantial performance boost highlights the model’s ability to effectively capture the complex environmental factors influencing soybean cropping suitability. Similarly, on the GF-1 WFV Dataset, AgriCLIP exhibited superior performance with an accuracy of 98.18 percent, a recall of 94.01 percent, an F1 score of 94.07 percent, and an AUC of 95.34 percent. Compared to the second-best model, AgriCLIP achieved a minimum improvement of 2.05 percent across all metrics. These findings underline AgriCLIP’s robustness in analyzing high-resolution remote sensing imagery and its exceptional predictive capability. While some comparison models, such as CLIP and ViT, demonstrated strengths in isolated metrics, their overall performance lacked the balance and consistency observed in AgriCLIP. This further emphasizes AgriCLIP’s advantage as a comprehensive and adaptive solution for predicting soybean distribution.