CNNs have demonstrated exceptional performance in various image segmentation tasks, showcasing their robust feature representation capabilities. However, despite these strengths, CNN-based methods frequently encounter limitations in modeling long-range relationships. A primary issue is their inefficiency in capturing global context information. Methods that rely on stacking receptive fields necessitate continuous downsampling convolution operations, leading to deeper networks. Training such deep neural networks on small datasets can present significant challenges, including training instability and overfitting. Overfitting is particularly common in deep learning models due to their strong expressive ability relative to traditional models (Zhang et al., 2023). Non-local attention mechanisms have been increasingly utilized in various fields to address challenges related to capturing long-range dependencies and global information (Mei et al., 2020; Huang et al., 2022; Abozeid et al., 2023; Zhou et al., 2023). While these mechanisms can enhance the network’s ability to capture global context, they also introduce considerable computational complexity. This complexity, which is quadratic in relation to the input size, often renders these methods impractical for high-resolution images.

Attention mechanisms were utilized in numerous research that focused on integrating Convolutional Neural Networks. Especially to further enhance the output processing of CNNs. Various visual tasks were implemented with integrated approaches, including video processing (Qi and Zhang, 2023; Sun et al., 2022; Mujtaba et al., 2022), image classification (Dosovitskiy et al., 2021; Liu et al., 2021), and object detection (Benmouna et al., 2023; Wen et al., 2023).

The transformer in natural language processing used transformation tasks (Vaswani et al., 2023). Several natural language processing activities have since shifted to using it. Some natural language processing activities have switched to using ViT. Pre-training on very large datasets is required for ViT (Chen et al., 2023; Misawa et al., 2024). To the State of the Art in the natural image segmentation task, Imagenet replaced the encoder component of the decoding network with a transformer (Doppalapudi, 2023; Xia and Kim, 2023).

Although transformer-based models have demonstrated impressive skills in diverse visual tasks, they have not yet attained acceptable results when compared to traditional CNNs. Transformer designs still exhibit worse performance in visual tasks compared to similarly-sized CNNs, such as EfficientNet (Thakur et al., 2023). The computational cost of transformers based on the mechanism of self-attention is O N 2 C , in contrast to the convolution-based CNNs O N C 2 (Zhou et al., 2024). Therefore, employing the transformer for image-related activities will unavoidably need a substantial amount of GPU resources.

The present cutting-edge architecture in computer vision predominantly depends on complete CNNs, with UNet (Chen et al., 2021) and its variations being notable instances. The current state-of-the-art (SOTA) framework in computer vision primarily relies on full CNNs, with UNet and its variants being prominent examples. UNet (Chen et al., 2021) employs an encoding-decoding network architecture. This architecture utilizes cascaded convolutional layers to extract various levels of visual characteristics. The decoder utilizes skip connections to recycle high-resolution feature maps generated by the encoder, enabling the retrieval of crucial feature information (Petit et al., 2021).

Deep learning, although powerful, often requires extensive training data to effectively enhance model learning. However, challenges arise in scenarios like the ArUco dataset due to limitations in data collection related to factors such as lighting conditions, capture angles, and distances. Moreover, the availability of large, publicly accessible datasets is limited, further complicating model training (Lee et al., 2019). To address these challenges, the development of lightweight deep learning models becomes imperative.

Research in deep learning has demonstrated that supervised training of deep learning models heavily relies on large labeled datasets (Karimi et al., 2020). This requirement poses a significant challenge, especially in scenarios where data collection is constrained. Techniques such as model optimization, pruning, quantization, and knowledge distillation have been explored to create lightweight deep-learning models suitable for mobile terminals (Wang et al., 2022). These approaches aim to reduce the computational burden while maintaining model performance.

A self-attention-based vision transformer (ViT), known as MobileViT, is employed to learn the global representation of images. MobileViT (Mehta and Apple, 2022) stands out as the initial lightweight, general-purpose transformer designed for mobile devices. An approach integrating a transformer with a CNN-based lightweight model was investigated, with a particular focus on assessing the feasibility of this lightweight network model for the challenging task of ArUco marker segmentation.

In this part, the Mini-MobileNet-MobileViT (MoNetViT) network architecture and its principal network components were introduced. The system’s backbone structure follows to the architecture of an encoder and decoder, as represented in Figure 1A. Section 2.4.1 offers a more detailed explanation sub-network of the encoder, while Section 2.4.2 focuses on the exploration sub-network of the decoder. The MobileViT module, a crucial part of the network’s encoder architecture, is introduced in Section 2.4.3. This section covers the architecture of the MobileViT module, its primary calculation process internally, and the comparisons between this module and CNN. Furthermore, the MQA module that is suggested in this study is described in Section 2.4.4. The Globalized Block and the Asymmetrical Globalized Block, as well as the justification for their adoption, are part of this module.

Figure 2A displays the configuration of a standard non-local block (Wang et al., 2018). The non-local block (Wang et al., 2018) first requires the computation of the similarity between all places. This is accomplished by performing matrix multiplication on an input X ∈ ℝ C × H × W . The primary computational procedure in the non-local block can be succinctly described as consisting of the following five steps:

The source feature X is subjected to three 1 × 1 convolutions, denoted as W ϕ , W θ , and W γ , resulting in the transformation of X into ϕ ∈ ℝ C ̂ × H × W , θ ∈ ℝ C ̂ × H × W , and γ ∈ ℝ C ̂ × H × W . The three numbers correspond to the query, key, and value, respectively. They are used to change the total amount of streams from 𝐶 to C ̂ .

Reassessing the Non-local Asymmetric Block

The computational complexity of the global attention block can be described as O C ̂ N 2 = O C ̂ H 2 W 2 . The calculation efficiency is mostly affected by N’s size. To fix this, reduce N to S S ≪ N without altering output size. Zhu et al. (2019) offers the asymmetric non-local block, whose construction is shown in Figure 2B, to tackle this problem.

The asymmetrical pyramid non-local block (APNB) is a modified version of this block that incorporates pyramid pooling within the non-local block in order to decrease computational expenses. One more thing is added after θ and γ : a spatial pyramid pooling function (Lazebnik et al., 2006) to pick out a few good anchor points. When the spatial pyramid pooling modules are P n θ and P n γ , with n denoting the pooling layer’s output size (width or height) and n ⊂ _ 1 3 6 8 as per Zhu et al. (2019), the overall amount of sampling anchor points is S = ∑ n ∈ 1 3 6 8 n 2 = 110 . If we assume that H = 224 and W = 256 , the number of calculations will be reduced by an amount H × W S = 256 × 256 110 ≈ 595 . This adjustment efficiently decreases the value of N to a lower value, S , by selectively sampling a few sample data from θ and γ , instead of utilizing all of the points.

The primary computational procedure in the asymmetrical non-local block entails the subsequent modifications, as demonstrated in Equations 10, 11: (10) ℝ N × C ̂ × ℝ C ̂ × N ︸ E q . 8 → ℝ N × N × ℝ N × C ̂ ︸ E q . 9 → ℝ N × C ̂ . (11) ℝ N × C ̂ × R C ̂ × S ︸ E q . 12 → ℝ N × S × ℝ S × C ̂ ︸ E q . 13 → ℝ N × C ̂ .

The computational technique for the APNB module may be delineated as follows:

Introduce sampling modules P θ and P γ after θ and γ, accordingly, to sample multiple sparse anchor points. These anchor points are designated as θ P ∈ ℝ C ̂ × S and γ P ∈ ℝ C ̂ × S , respectively.

Generating a similarity matrix M P , as demonstrated in Equation 12: (12) M P = ϕ T × θ P , M P ∈ ℝ N × S

M P is normalized. M → P = f M P

f · stands for normalization function.

The attention matrix AP is subsequently computed, as demonstrated in Equation 13: (13) A P = M → P × γ P T , A P ∈ ℝ N × C ̂

The final result is obtained by adding the product of the variables W A and A T to the variable X, and assigning it to the variable Y P . W A is a one-by-one convolution. Y P = W A A p T + X . W A represents a 1 × 1 convolutions.

The APNB discussed earlier operates with a single data stream as input, while the asymmetrical fusion nonlocal block typically utilizes two data sources: the top-level feature map and the lower-level feature map. In contrast, the proposed MQA mechanism extends this approach by incorporating four input sources. As shown in Figure 1A, MQA integrates the fundamental value ( P 1 ), the elevated characteristics P 2 and P 3 , and the edge content ( P e ), allowing for the explicit acquisition of multiple levels of feature representation. By incorporating edge information throughout the semantic segmentation process, the model imposes valuable constraints, enhancing segmentation precision. The cross-entropy loss function further refines this process by measuring the difference between the ground truth ( G T ) and the feature aggregation map ( P f ), ensuring robust alignment of predictions with the actual data.

Commence the acquisition of a parameter map of features X ∈ ℝ C × H × W , and additional feature maps X 1 ∈ ℝ C 1 × H 1 × W 1 , X 2 ∈ ℝ C 2 × H 2 × W 2 , and X 3 ∈ ℝ C 3 × H 3 × W 3 . Five 1 × 1 convolutions, denoted as W ϕ , W θ , W γ , W ϕ 2 , and W ϕ 3 , are applied to transform these input maps into new feature maps: ϕ ∈ ℝ C × H × W , γ ∈ ℝ C × H × W , ϕ 1 ∈ ℝ C × H 1 × W 1 , ϕ 2 ∈ ℝ C × H 2 × W 2 , and ϕ 3 ∈ ℝ C × H 3 × W 3 , as demonstrated in Equation 14: ϕ = W ϕ X , θ = W θ X , γ = W γ X , (14) ϕ 1 = W ϕ 1 X 1 , ϕ 2 = W ϕ 2 X 2 , ϕ 3 = W ϕ 3 X 3

The parameters X , X 1 , X 2 , and X 3 in this experiment correspond to the results P e , P 1 , P 2 , and P3 from MoNetViT. The sample as well as main computation methodologies within the MQA module were equivalent to those in APNB. The selection units P θ , P γ , P ϕ , P ϕ 1 , and P ϕ 2 are utilized to sample multiple sparse anchor points. These anchor points are represented as θ P ∈ ℝ C × S , γ P ∈ ℝ C × S , ϕ P ∈ ℝ C × S , ϕ 1 P ∈ ℝ C × S 1 and ϕ 2 P ∈ ℝ C × S 2 , where S , S 1 , and S 2 denote the number of sampled anchor points. S is less than S 1 , which is less than S 2 , and S 2 is much less than N . Mathematically, this is computed using the following Equation 15: K = P θ θ , V = P γ γ , Q = P ϕ ϕ (15) Q 1 = P ϕ 1 ϕ 1 , Q 2 = P ϕ 2 ϕ 2

The correlation matrix M P for Q and anchoring K is displayed here, as demonstrated in Equation 16: (16) M P = Q T × K

The dimensions of the M P are S × S , where S is significantly smaller than N . Next, the process of normalization is carried out on M P , which enables the calculation of X Q K 1 .

The ultimate result of the initial layer is, as demonstrated in Equation 17: (17) X Q K V 1 = X Q K 1 × V , X Q K V 1 ∈ R C × S

The value of X Q K V 1 is equal to the product of X Q K 1 and V . X Q K V 1 belongs to the set of R C × S . The similarity matrix for levels 2, 3, and 4 is computed using an analogy, as demonstrated in Equation 18: X Q K 2 = Q 1 T × X Q K V 1 X Q K 3 = Q 2 T × X Q K V 2 (18) X Q K 4 = Q 3 T × X Q K V 3

The equation X Q K 2 is equal to the product of Q 1 T and X Q K V 1 . The equation X Q K 3 is equal to the product of Q 2 T and X Q K V 2 . The equation X Q K 4 is equal to the product of Q 3 T and X Q K V 3 .

The ultimate result of levels 2, 3, and 4 is determined in the following manner Equation 19: X Q K V 2 = X Q K 2 × X Q K V 1 , X Q K V 2 ∈ ℝ C × S 1 X Q K V 3 = X Q K 3 × X Q K V 2 , X Q K V 3 ∈ ℝ C × S 2 (19) X Q K V 4 = X Q K 4 × X Q K V 3 , X Q K V 4 ∈ ℝ C × N

The value of X Q K V 2 is equal to the product of X Q K 2 and X Q K V 1 , where X Q K V 2 belongs to the set ℝ C × S 1 . The value of X Q K V 3 is equal to the product of X Q K 3 and X Q K V 2 , where X Q K V 3 belongs to the set ℝ C × S 2 . The value of X Q K V 4 is equal to the product of X Q K 4 and X Q K V 3 , where X Q K V 4 belongs to the set ℝ C × N .

The final result is represented as X Q K V 4 belonging to the set of ℝ C × N . The temporal complexity can be represented as O C S × S + S 1 × S + S 2 × S 1 + S 3 × S 2 , which is significantly lower than O C N 2 in the conventional non-local block.

The function that measures loss is defined like the ones used by Yeung et al. (2022). The loss function comprises two components: L edge and L seg . Equations 20, 21 (Yeung et al., 2022) display the loss function. (20) L edge = − 1 N ∑ i = 1 N y i log y ̂ i + 1 − y i log 1 − y ̂ i

where ground-truth (GT) and the anticipated edge map Pe’s coordinates for each pixel point are represented by ( y i , y ̂ i ).

IoU loss and a conventional cross-entropy loss make up the two components of the L seg loss function. (21) L seg = λ 1 L C E w + λ 2 L I o U w

The NVIDIA Tesla T4 GPU was used to train the model within the PyTorch framework for this project. The model underwent training for 50 epochs using a batch size of 16, with the Adam optimizer and a starting learning rate of 1e-3. A learning rate reduction factor of 0.5 was applied when no improvement was observed for 5 consecutive epochs (patience set to 5). The specific hyperparameters utilized in this investigation are outlined in Table 1. In addition, the experiment aimed to compare the proposed MoNetViT model with several state-of-the-art methods, including TransFuse (Zhang et al., 2021), Inf-Net (Fan et al., 2020), U-Net (Ronneberger et al., 2015), U-Net++ (Kwak and Sung, 2021), Mini-Seg (Kim et al., 2023), and DeepLabV3+ (Asadi Shamsabadi et al., 2022), for multi-class segmentation in ArUco marker identification.

The researchers conducted experiments using freely available datasets for ArUco manual labeling. The dataset employed in this study is an open-source resource that has been fully labeled to indicate various classes of ArUco markers, making it ideal for training models in identification and classification tasks. The dataset preparation involved several pre-processing steps to enhance model generalizability and ensure reproducibility. Images were resized to a resolution of 224 × 224, and pixel values were normalized to the range 0 and 1. Additionally, data augmentation techniques such as random rotation, flipping, and contrast adjustment were applied to improve the model’s ability to generalize across varied conditions. The complete dataset, including all necessary images and labels for training and testing models in ArUco marker identification and classification tasks, can be accessed and downloaded from the following link: https://universe.roboflow.com/loliktry/dataarucomustofa/dataset/5. It was specifically utilized in the multi-class segmentation experiments. Table 2 provides a comprehensive overview of the specific details of this dataset.

Furthermore, the present study utilized identical methodologies as Fan et al. (2020) to assess the performance of the model. Standard criteria, such as accuracy, specificity, sensitivity, and Dice similarity coefficient, comprise the assessment metrics. In addition, it uses several metrics from object recognition evaluation methods, such as the design measure, the enhanced alignment value (Fan et al., 2018), and the mean absolute error.

The segmentation data findings for ArUco Marker on the dataset are displayed in Figure 3, demonstrating that MoNetViT in this study exhibits superior performance compared to other baseline models. U-Net and U-Net++ have low Dice scores and sensitivities, resulting in large unsegmented areas. Inf-Net and Mini-Seg show slight improvements but still lack accurate boundary detection. While TransFuse, a CNN + Transformer model, was evaluated, it is not included in Figure 3 due to its very low Dice score. DeepLabV3+ performs reasonably well but falls short of MoNetViT. Overall, MoNetViT achieves the highest Dice, sensitivity, and specificity scores, along with the lowest MAE, indicating its superior segmentation precision.

The MoNetViT model shows superior performance over other state-of-the-art models, including U-Net, U-Net++, Mini-Seg, Inf-Net, TransFuse, and DeepLabV3+, across key evaluation metrics. As summarized in the Table 3, MoNetViT achieves the highest Dice score (0.9584), sensitivity (0.9424), specificity (0.9424), structural similarity S α 0.9923), and mean edge accuracy E θ mean (0.9381), while also attaining the lowest Mean Absolute Error (MAE) of 0.0077.

This outstanding performance is attributed to the integration of Convolutional Neural Networks (CNNs) with a transformer module, which captures both local and global semantic features. The transformer component is crucial for calculating global semantic relationships, while the CNN module extracts local contextual features, resulting in a more robust feature representation. Additionally, the multi-query attention (MQA) module enriches feature diversity through supervised learning, enhancing overall model performance.

An analysis of false positives (FP) and false negatives (FN) further emphasizes MoNetViT’s robustness. With high sensitivity (0.9424) and specificity (0.9424), MoNetViT significantly reduces both FP and FN compared to other models. For instance, while DeepLabV3+ achieves a Dice score of 0.6351, its sensitivity (0.6392) and specificity (0.9655) indicate a higher FN rate relative to MoNetViT. Similarly, Mini-Seg’s balanced sensitivity (0.6268) and specificity (0.9537) suggest that it is more prone to FP and FN, impacting its reliability. In contrast, MoNetViT’s ability to minimize FP and FN contributes to its high Dice score and overall segmentation accuracy.

Despite Inf-Net having the smallest model size (0.073 MB) and TransFuse having the smallest parameter count (0.038 M), MoNetViT outperforms these models in critical metrics. Its effectiveness stems from model design rather than sheer training data volume, highlighting MoNetViT’s robustness and capacity to generalize effectively on the dataset. Paired T-tests indicated that MoNetViT’s improvements were statistically significant (p < 0.05) compared to all models except U-Net++, Mini-Seg, and DeepLabV3+, suggesting that while MoNetViT is generally superior, these models are still competitive in certain tasks. The addition of FP and FN analysis strengthens these findings, demonstrating that MoNetViT achieves its exceptional performance by addressing key limitations in segmentation errors observed in other models.

Several experiments were conducted to validate the functionality of the Multi Query Attention (MQA) and Fusion Feature Module (FFM), two crucial elements of the MoNetViT. Figure 1A depicts an architecture consisting of three stages. The segmentation performance of the MoNetViT model is considerably improved by the MQA module component, as the findings displayed in Table 4 reveal.

Employing MQA and FFM in conjunction with the baseline improves segmentation performance. Specifically, integrating both MQA and FFM with the baseline resulted in improvements of 1.5 and 1.7% in the Dice coefficient, respectively. The findings indicate that using MQA and FFM enhances the encoder’s and decoder’s clarity, thereby further improving segmentation performance. The analysis shows that combining Backbone, FFM, and MQA results in the best performance across all metrics. This model has the lowest loss (0.0115) and the highest Dice coefficient (0.9731), indicating more accurate segmentation and fewer errors. It also achieves the best sensitivity (0.9686) and specificity (0.9686), accurately identifying both positive and negative cases. Additionally, it has the highest structural alignment ( S α = 0.9951) and precision ( E θ mean = 0.9599), along with the lowest Mean Absolute Error (0.0049). In comparison, other methods like Backbone alone, Backbone+FFM, and Backbone+MQA perform worse in various metrics, highlighting the benefits of using both FFM and MQA together.

Figure 4 provides a detailed overview of the model performance. MoNetViT achieves superior accuracy while maintaining a relatively small parameter count. Specifically, it uses about half the parameters of U-Net (1.014 M vs. 1.953 M) and significantly fewer parameters than U-Net++ (7.783 M) and DeepLabV3+ (13.324 M). While Inf-Net (0.076 M) and Mini-Seg (0.038 M) have smaller parameter counts, and TransFuse uses the fewest (0.019 M), MoNetViT achieves a much higher Dice score (0.9584) compared to these models. This demonstrates that MoNetViT not only optimizes model size and complexity but also outperforms other models, including transformer-based models like TransFuse, in terms of accuracy (Figure 5).

Numerous models for combining multi-scale features use conventional networks for object identification and semantic segmentation in image processing. For example, architectures like Feature Pyramid Network (FPN) and U-Net rely on three primary network paths: bottom-up, top-down, and horizontal connections. These paths allow the integration of high-level semantic information with low-level geometric data. In FPN, the bottom-up path extracts high-level features, while the top-down path applies upsampling to enhance semantic details at higher resolutions. Horizontal connections fuse low-level convolution features with high-level features, resulting in a more detailed representation of semantic information.

Nevertheless, FPN faces challenges due to its complex hierarchical structure. The computation of intermediary layers relies heavily on the higher-level layers, requiring the analysis of preceding layers to be completed before passing information to subsequent layers. This dependency can lead to inefficiencies in computation and integration. The proposed MQA approach addresses these limitations by enabling the simultaneous integration of lower-level map attributes, higher-level attribute maps, and edge attribute maps, streamlining the process and enhancing feature representation.

The current work offers a direct and efficient MQA approach and introduces a cascading multi input computational framework. The system employs a mechanism for attention to iteratively compute and use feature maps of varying sizes, directing the ultimate semantic segmentation process. The proposed MQA has the ability to combine multiple input and multiple scale features, and can be trained end-to-end with ground truth supervision. The proposed technique effectively leverages both low and high-resolution features and integrates a method of attention to successfully accomplish the segmentation job on ArUco markers.

To evaluate the impact of MQA and FFM on MoNetViT’s performance, a series of tests were conducted. The experiments utilized the same network backbone and implementation details to ensure consistency with previous studies. The results, as shown in the radar chart and table, compare the baseline “Backbone,” “Backbone+MQA,” “Backbone+FFM,” and “Backbone+MQA + FFM” configurations. The radar chart illustrates that the region representing “Backbone+MQA + FFM” (in red) is larger than those of other configurations, indicating superior performance across key metrics. Similarly, the table reinforces these findings, showing that “Backbone+MQA + FFM” achieves the highest Dice score (0.9731), sensitivity (0.9686), and specificity (0.9686), along with the lowest MAE (0.0049). These results suggest that integrating both MQA and FFM significantly enhances the backbone’s performance, as the areas with MQA and FFM have a notably larger magnitude than those without these modules.

The contributions of the Multi-Query Attention (MQA) and Feature Fusion Module (FFM) to the segmentation performance were further validated through an ablation study. Table 4 highlights the significant improvements achieved by integrating these modules into the baseline model. Specifically, the Dice coefficient increased from 0.9584 for the baseline to 0.9558 (+1.7%) with MQA alone and 0.9566 (+1.9%) with FFM alone. When both modules were combined, the Dice coefficient reached 0.9731 (+3.4%), demonstrating their synergistic effect. Furthermore, sensitivity and specificity improved from 0.9424 each in the baseline to 0.9686 with the combined MQA and FFM setup, while the Mean Absolute Error (MAE) decreased from 0.0077 to 0.0049. These findings emphasize the critical role of MQA in enhancing multi-scale feature integration and FFM in refining feature clarity, resulting in superior segmentation performance. This robust improvement across key metrics underscores the effectiveness of the proposed MoNetViT architecture in addressing complex segmentation tasks.

MoNetViT’s architecture demonstrates significant advantages through its dual-path encoder, which effectively balances local feature extraction using CNNs and global feature extraction via Transformers. This design allows the model to capture both fine-grained details and long-range dependencies, improving segmentation performance. Additionally, the integration of the Multi-Query Attention (MQA) module enhances multi-scale feature integration, enabling the model to better aggregate features across varying spatial scales. This contributes to improved segmentation accuracy, particularly in complex scenarios. Furthermore, MoNetViT’s lightweight design minimizes computational demands, making it highly efficient for real-time applications without sacrificing performance. Compared to models like DeepLabV3+, which rely on more resource-intensive architectures, MoNetViT achieves a superior balance of accuracy and efficiency, reinforcing its suitability for deployment in resource-constrained environments.

While MoNetViT demonstrates strong performance, scalability to larger datasets may require optimization strategies, and its adaptability to diverse marker types needs further evaluation across different styles and conditions. Future experiments will focus on enhancing scalability and generalizability through transfer learning and broader dataset evaluations.