Marine Decapoda species, which are widely found from the coastal zone to the abyssal zone, play vital roles in ecosystems and in world fisheries (Marquez and Idaszkin, 2021; Joyce et al., 2022; Truchet et al., 2023). In 2020, the total capture production of marine Brachyura crabs was estimated to be 290,000 tons (live weight) (FAO, 2022). However, while Decapoda species are essential for ecosystems and fisheries, certain invasive species could negatively impact the ecosystem, leading to the loss of native biodiversity and economy. For instance, the Europe-native green crab (Carcinus maenas) has been introduced to the Pacific and Atlantic coasts of Canada (Jamieson et al., 1998). This invasion has led to significant ecological and economic impacts, including the displacement of the American lobster (Homarus americanus), resulting in commercial losses of $44 to $114 million per year. Furthermore, juvenile C. maenas crabs outcompete juvenile Dungeness crabs (Metacarcinus magister), resulting in the population decline of the latter and losses of over $6 million per year in the British Columbia decapod fishery (Colautti et al., 2006; Griffin et al., 2023). The expansion of C. maenas is also particularly destructive to the native ecosystems, with Canadian bivalve populations exhibiting 5- to 10-fold decrease in the presence of C. maenas (Tan and Beal, 2015; Ens et al., 2022). In Atlantic Canada, C. maenas has caused 50% to 100% declines in biomass of Eelgrass meadows that many estuarine organisms rely on for food and shelter (Matheson et al., 2016; Ens et al., 2022). Early awareness of non-indigenous species (NIS) is crucial to mitigating these losses in biodiversity and economy. Citizen support has emerged as a valuable tool for early detection, playing an impactful role in monitoring NIS worldwide (Giovos et al., 2019; Johnson et al., 2020). To determine whether certain Decapoda specimens are indigenous or not, the first step is to recognize them. However, the order Decapoda encompasses an immense diversity of crustaceans, with approximately 10,000 species having been described, including crabs, shrimps, and crayfishes (Hobbs, 2001; Creed, 2009). The great biodiversity of Decapoda species poses challenges for the citizens. Without considerable domain knowledge, local citizens may struggle to recognize new NIS. This is especially true when the specimen is uncommon or unfamiliar to them. It takes time for the citizens to identify the specimen, such as by comparing the morphological features of the specimen against the atlas to find out the most similar species. Meanwhile, it is costly to educate the citizens to classify the highly diverse Decapoda species in global oceans. Moreover, even for professionals, the domain knowledge should be kept updated along with exploding knowledge on biodiversity, such as discoveries of novel species. Furthermore, there are huge classification tasks in monitoring the marine Decapoda community, such as in assessing the impacts from pollution, overexploitation, and other anthropogenic pressures. The classification of Decapoda specimens by human is labor-intensive and time-consuming. Hence, an efficient automated tool for assisting professional taxonomic identification of Decapoda specimens is urgently needed to tackle these challenges.

Deep learning (DL) techniques have demonstrated remarkable performance in automated image analysis of specific Decapoda species, such as shrimp classification in food production lines (Liu et al., 2019) and gender classification of the Chinese mitten crab (Eriocheir sinensis) (Chen et al., 2023). However, despite the availability of numerous DL open-source packages and AutoML platforms, there remains a scarcity of professional tools tailored specifically for the image classification of diverse decapods. Importantly, users often have to undertake dataset preparation, model training, parameter optimization, model deployment, and even the entire process themselves to implement the automated image classification. Therefore, in this study, we aim to develop an automated image classifier and propose a user-friendly web application, named DecapodAI, which leverages multi-modal DL techniques to facilitate image classification of marine Decapoda species on a global scale.

Theoretically, every decapod species has the potential to become a NIS if it is introduced outside of its natural habitats. Thus, to identify NIS, a dataset collecting images of diverse decapods found within the global oceans is required, which is crucial for creating a robust automated image classifier to enhance the performance of the DL models. Firstly, to construct the dataset, the images of Decapoda species, as well as their taxonomic annotation, were collected from the World Register of Marine Species (WoRMS, https://marinespecies.org/), as WoRMS aims at providing an authoritative and comprehensive list of names of global marine species. After manually screening out noise images (such as maps), a total of 1,541 images, which cover 553 species belonging to 317 genera from 102 families, remained for model training and performance evaluation. Each image was labeled with taxonomic annotation at the family, genus, and species levels, respectively. The resource links for these images and their taxonomic annotations are included in Appendix S1.

Secondly, the DL model based on CLIP (Contrastive Language–Image Pretraining) was selected as the backbone network, which proposed a progressive learning and achieved in-depth advances in training (Dong et al., 2022). For instance, CLIP achieved competitive accuracy without requiring as many training examples as traditional models, such as the original ResNet-50 (Radford et al., 2021). Despite the successful application of DL methods to the image classification of marine species, such as EchoAI using EffiecientNetV2 for marine echinoderms (Zhou et al., 2023) and FishAI utilizing Vision Transformer for marine fish (Yang et al., 2024), the accuracy for species with limited labeled images should be improved. Specifically, for certain decapod species recorded in WoRMS, fewer than 10 taxonomically labeled images are available. To address this issue and enhance performance for these underrepresented species, the multi-modal deep network CLIP_L14 was employed. Details about CLIP_L14 could be found in the following website: https://huggingface.co/openai/clip-vit-large-patch14. This approach leverages both text and visual information, achieving superior accuracy in few-shot learning scenarios. The introduction of fine-tuning CLIP is illustrated in Figure 1 . CLIP uses contrastive learning to pre-train on large-scale image–text pairs like images and their descriptions to learn cross-modal semantic representations. The image encoder that extracts image features and the text encoder that extracts text features are jointly trained through contrastive loss, making related images and texts closer in the vector space and irrelevant ones farther away. Combined with fine-tuning techniques, the weights of the early layers of the encoder are frozen, and the model parameters are updated using a small learning rate and task loss function on a dataset for a specific task. The main hyperparameters during model training are shown in Appendix S2.

The CLIP model utilizes text encoding to generate text vector [T ₁, T ₂, T ₃…T_N ] and image encoding to produce image vector [I ₁, I ₂, I ₃…I_N ]. Combined with fine-tuning techniques, contrastive loss of matrix is mainly used to train the model. In the testing phase, the most probable labels are chosen for test images.

Thirdly, the CLIP network was trained with 1,060 images at the family, genus, and species levels, respectively, using a batch size of 16, and crafted into the image classifier, termed DecapodAI-Core. DecapodAI was derived from “Decapoda” and “AI.” For each input image, DecapodAI-Core returned recognition results in taxonomic names according to probabilities from the highest to the lowest. Subsequently, the accuracy conventionally measures the proportion of images for which the classification inference matches the expected label. To evaluate the performance of DecapodAI-Core, the average accuracies of the top 1 inference (the ones with the highest probability) were calculated for the test dataset at the family, genus, and species levels, respectively. The detailed accuracy metrics for the test dataset across all families are included in Appendix S3. Similarly, the average accuracies for the inference results with the top 2, 3, 4, and 5 highest probabilities were calculated as previously described (Zhou et al., 2023). Finally, online service is provided openly available at the web application, DecapodAI (http://www.csbio.sjtu.edu.cn/bioinf/DecapodAI/). Once the image is successfully uploaded, it usually takes about seconds for the program to process. The results could be accessed by clicking the hyperlinks when the job is finished, and the results are available for download. While uploading the image, email address could be provided optionally, and the link for the results would also be sent to the email provided. To demonstrate the classification performance of DecapodAI, four images were downloaded from the Global Biodiversity Information Facility (GBIF, https://www.gbif.org) and were used as input images. The resource links for these four images are included in Appendix S4.

The DecapodAI model, trained on the dataset using a fine-tuned CLIP, demonstrates its performance at the family, genus, and species levels through the convergence of the loss function during training, as illustrated in Figure 2 .

DecapodAI achieved average top 1 classification accuracies on the test dataset of 0.717 at the family level, 0.719 at the genus level, and 0.773 at the species level, as shown in Figure 3 . Accuracy improved from top 1 to top 5: at the family level, it increased from 0.717 to 0.902; at the genus level, it increased from 0.719 to 0.914; and at the species level, it increased from 0.773 to 0.939. These variations in accuracy across different taxonomic levels could be attributed to the diversity of text descriptions and dataset quality. In comparison, EfficientNetV2 achieved lower top 1 classification accuracies on the same test dataset: 0.597 at the family level, 0.561 at the genus level, and 0.576 at the species level. These results are notably lower than those obtained with fine-tuned CLIP.

The accuracy for each family is detailed in Appendix S3, which demonstrates that the DecapodAI model using CLIP outperforms the EfficientNetV2 model, particularly on datasets with a small number of images. As for the invasive species C. maenas in the test dataset, the accuracy reached 1.00. In demonstration of DecapodAI, the images from GBIF were used as input. The top 5 inference results at the species level for each image are shown in Figure 4 , along with the probability. Among them, the image of the well-known invasive crab, C. maenas, was correctly classified by DecapodAI. Furthermore, DecapodAI also classified the image of Percnon gibbesi well ( Figure 4 ). P. gibbesi is a native crab species from Eastern Pacific and Tropical Atlantic Ocean (Guillén et al., 2016). Because of its ability to adapt to Mediterranean waters, including its feeding habits, P. gibbesi has spread rapidly and has been proposed to be included in the list of the 100 “Worst Invasives” in the Mediterranean (Puccio et al., 2006; Streftaris and Zenetos, 2006).

In addition, the image of the Asian tiger shrimp Penaeus monodon, a NIS in the western north Atlantic and Gulf of Mexico (Fuller et al., 2014), was misidentified as Brachycarpus biunguiculatus at the top 1 inference level ( Figure 4 ). However, DecapodAI correctly classified it at the top 2 level ( Figure 4 ). For another instance, DecapodAI correctly classified the image of Indo-Pacific hippolytid species Saron marmoratus at the top 2 level ( Figure 4 ). This species is among the ornamental pet shrimps in marine aquarium and has been recorded in the Mediterranean Sea from specimens photographed along the Israeli coastline (Rothman et al., 2013).

The marine NIS are alarmingly impacting and reshaping the biodiversity in oceans globally. There are numerous documentations on invasive Decapoda species, such as the tiger shrimp P. monodon in almost all the Colombian Caribbean Sea (Aguirre-Pabón et al., 2023) and the blue crab Callinectes sapidus in the Mediterranean Sea (Compa et al., 2023). These invasive Decapoda species not only harm indigenous species but also cause substantial disruptions in biodiversity, posing threats to fisheries. Recognizing the importance of early awareness and timely monitoring of NIS for biodiversity conservation, we have developed DecapodAI, a publicly accessible and easy-to-use web application that bridges these gaps by providing automated image classification for Decapoda species on a global scale. This tool allows users around the world to upload images of the specimens of interest and conduct automated classification without any prior knowledge of taxonomy, machine learning, or programming. For instance, a fisherman or scuba diver could use DecapodAI for the identification of the unfamiliar Decapoda specimens found in bycatches or in photographs. The outputs of classification candidates for the input images offered by DecapodAI could also give clues for further survey, such as taxonomic identification by experts or primer design in environmental DNA (eDNA) analysis based on polymerase chain reaction, which is highly sensitive in detecting NIS. Moreover, the public can take advantage of DecapodAI to raise their knowledge of biodiversity and perception of conservation.

It has to be pointed out that the current version of DecapodAI was developed based on only 1,541 images of 553 species. On average, there are less than three images per species. The baseline version of DecapodAI may not match the accuracy of human experts in fine-grained classification. The performance could be improved in the future as WoRMS updates its image database with more species and images. In addition, staying updated with advances in DL techniques could also enhance the performance of DecapodAI.

In summary, an image classifier based on fine-tuning CLIP with image and text information, DecapodAI, was developed for robustly automated classification of Decapoda species at a global scale. This easy-to-use web application can effectively provide community taxonomic references at three different resolutions (family, genus, and species) for marine decapods of interest. DecapodAI can help alleviate the burden of manually analyzing imagery data and enhance the monitoring efforts, including early awareness of NIS. It is expected to promote public participation and contribute to marine biodiversity conservation practices and has promising application prospects in exploring and monitoring the biodiversity of decapods in global oceans.