Image analysis is a crucial component of microscopy research, enabling the extraction of quantitative data from complex visual data sets in an unbiased manner (287). The standard toolbox for image analysis comprises tools for image preprocessing, segmentation, tracking, and quantification. Automating this process ensures unbiased data analysis and simplifies compliance with good practices for data and image acquisition reporting (288).

Image preprocessing seeks to minimise noise, enhance contrast, correct geometric distortions, and improve resolution, thus facilitating the subsequent image analysis steps. Denoising techniques improve the signal-to-noise ratio (289): methods vary from standard Gaussian blur and median filter to more advanced techniques such as 3D block-matching (290), non-local means (291, 292), and wavelet transforms (293). Improvements in contrast and resolution can be achieved with deconvolution techniques, where the information about the point spread function of the microscope is used to remove the signal contribution from out-of-focus planes and surrounding signal sources (294). Resolution and contrast improvements can also be obtained by acquiring the same image with slight changes in the illumination beam (295, 296), with general algorithms considering the noise distribution (150, 151), analysing fluorescence fluctuations with algorithms such as mean shift vector analysis (MSSR, 155), or by deconvolution, like in SUPPOSe (297) or B-SIM and Sparse-SIM for SIM images (298, 299). Image registration corrects image distortions and time drift: this is especially useful when reconstructing a volume in a mosaic (300–302), aligning images in case of sample drift, as needed in intravital microscopy (303–306) or aligning images acquired with different modalities, such as in the case of CLEM (51). Lastly, crosstalk correction improves channel separation, eliminating unwanted spectral bleed-through between channels (4, 174), while spectral unmixing techniques use the characterisation of the emission profile to separate many fluorophores with overlapping emission spectra (241). These approaches are particularly useful in multiplex imaging (239) and when subtracting unwanted autofluorescence contributions (242). Other methods for spectral separation include phasor analysis based on fluorescence spectral data, or on fluorescence lifetime (307). Overall, effective image preprocessing greatly simplifies image segmentation, ultimately improving the extraction of quantitative data from microscopy images.

Segmentation of image data is the process of separating the pixels of the background from the pixels of interest, labelling the objects (e.g. organelles, cells, tissue areas) so that properties such as geometrical descriptors can be measured, or intensity statistics be calculated. Image segmentation is at the basis of most automated analysis workflows (308). Segmentation techniques can be broadly categorised into two main groups: region-based and edge-based methods. Region-based segmentation algorithms group together pixels with similar attributes (e.g., intensity, colour) to form homogeneous regions within an image. These algorithms often use intensity thresholding (309), clustering (310), or watershed (311) to identify meaningful segments. On the other hand, edge-based segmentation focuses on identifying boundaries between objects in an image by detecting abrupt changes in pixel attributes, such as intensity or texture. Common edge detection methods include Canny (312, 313), Sobel (314), and Laplacian of Gaussian (LoG) operators (308). The choice of segmentation technique may vary depending on the specific application, where factors to consider include image complexity, object shape, size, and contrast with respect to the background. Object segmentation can then be followed by object classification according to some measurable property, such as object position or shape factors.

Object detection localises objects or regions of interest in an image. The task typically involves identifying the object to be detected (classification) and determining its position in the image (localisation). Object detection is frequently employed to recognise areas of interest, such as a compartment in a cell or tissue (315), and to identify cells in time-lapse microscopy movies for object tracking (316).

Tracking allows the study of temporal dynamics, such as cellular or subcellular movements. Tracking objects in a movie is a two-step process, where object detection and segmentation are followed by object linking between frames (317, 318). Manual or semi-automated tracking software has dominated the scene in studies of immune cell mobility upon antigen presentation, in cell culture experiments, or in lymph node imaging (319). However, when temporal sampling cannot be done at high frequencies, or the linking process is ambiguous, deep learning techniques may prove helpful in enhancing the effectiveness of classical methods (316, 320).

Recent advances in bioimage analysis are significantly broadening its accessibility, allowing researchers to leverage powerful techniques with reduced reliance on programming expertise and lowered computational resource demands. These developments are driven by a growing trend toward simplified user interfaces (321–326), and standardised analysis protocols for light microscopy (327) and electron microscopy (57). However, while these tools streamline many routine analyses, complex or novel research questions often require more sophisticated approaches. This is where the role of the bioimage analyst remains crucial – bridging the gap between readily available software and advanced techniques, and facilitating the development of custom solutions to address unique research challenges (328).

Open-Source software (OSS) has a key role in bioimage analysis because the access to source code enables any researcher to develop customised workflows, even with a limited knowledge of programming languages, thus guaranteeing more transparency and reproducibility (329).

Image quantification has been made easy in the past decades by many graphical user interfaces (GUI) suites, among which the most renowned include ImageJ or FIJI (330), CellProfiler (323), napari (331), QuPath (332). The key aspect that makes these GUIs widely adopted is the abundance of scripts and plugins, together with the possibility to access the source code and develop custom bioimage analysis solutions, whether implemented as point and click interaction or as a script.

A fundamental role of facilitator for bioimage analysis based on scripting has been covered by development environments such as RStudio ¹ (based on R language), JupyterLab (333, based on Python) or visual programming suite KNIME (334). Because of the richness of the Python environment, in terms of availability of packages, many bioimage analysis solutions have been developed as Jupyter notebooks, especially in the context of machine learning (ML) and deep learning (DL), where code modification might enhance the adaptation to specific image data (118, 335, 336). The availability of complete notebooks where all the steps of a workflow are explained with code comments facilitates the execution by the end user, learning, and reproducibility (335).

The integration between the full control of microscope motorisation (337), image acquisition, real-time (or offline) image analysis, and the possibility to drive a new image acquisition, based on the result of the analysis, constitutes the backbone of what is called feedback microscopy (338), also referred to as smart microscopy (339). The purpose of such integration is to allow the adaptive imaging of the biological sample in the spatial and temporal dimensions (340).

In Figure 2 , we present a possible workflow of feedback-based microscopy to identify infected cells and acquire them at higher resolution, minimising the overall acquisition time. A multichannel fluorescence image, including a nuclear marker and an infection reporter ( Figures 2A–C ), is used to identify all the cells in the field of view ( Figure 2D ) and measure the level of an infection reporter ( Figure 2E ). Using a ML clustering algorithm (e.g. k-means clustering), cells are classified based on their infection state ( Figure 2F ). Then, cells are selected depending on their level of infection ( Figure 2G ), and the microscope is instructed to navigate to the cell position ( Figure 2H ), switch objective ( Figure 2I ), and acquire a higher-resolution image ( Figure 2J ). This type of workflow presents the advantage of scanning larger areas to increase the number of inspected cells and use higher resolution imaging only for the infected ones, which are identified with an unsupervised ML algorithm.

Examples of software tools for feedback-based microscopy include AutoscanJ, for the detection of mitotic cells or chromosomal anomalies, based on the same principle of rescanning cells of interest, previously detected with lower magnification (341) or the real-time drift correction in intravital movies (342). In the field of RT-EM, a similar approach has been developed by SerialEM software (14) and its Python interface pyEM (343). This approach was used in the contests of immunological research to show, with tomographic reconstructions, that plasma cells in patients with multiple myeloma display elongated centrioles (344). Another application has been developed for combining light microscopy and FIB-SEM (336). In single-particle cryo-EM, automatic acquisition is even more important because thousands of images are needed to perform a structural reconstruction (17, 345). In cryo-ET, machine learning approaches were used to fully automate in-situ cryo-ET workflow (346).

While some scripting tools, such as ImageJ macro language, are very popular in the bioimage analysis community (347), the complexity of back-end programming languages (e.g. C or Java) to develop software plugins may hinder the quick implementation of novel ideas. To facilitate the use of feedback microscopy, projects like Pycro-Manager (348) or pymmcore ² have been created to implement translation layers between programming languages (in this case, Python can be used to write scripts rather than Java). On the other hand, the Open-Source Hardware (OSH) movement has allowed the implementation of cheaper solutions for image acquisition compared to proprietary microscopy software. Projects like Micro-Manager (349) to control microscope hardware have revolutionised the field, decoupling the need for commercial licences to operate devices from the mere possession of the equipment. In addition to gaining control of microscopy equipment, the possibility to trigger and modulate the image acquisition with plug-in electronics, for example based on Arduino ³ or Raspberry Pi ⁴ development boards, has widened the possibilities to customise every microscopy platform. However, technology development requires the developers or early adopters to carry the risks of investing resources in technologies that might have limited or delayed benefits. Then, the advantage has to be identified either in the reduced cost of existing open technology or in access to bleeding-edge techniques, which might be rewarded in terms of scientific publications (350).

Finally, Computer-Aided Design (CAD) for machining or 3D printing of microscope components or auxiliary devices has improved the use of resources to run microscopy experiments. Examples include the open optical setup of light-sheet system openSPIM (351), the possibility of fully 3D printing experimental tools or the wide database of open hardware projects developed by the imaging community (for example, by the LIBRE hub ⁵ ). These can include accessories such as syringe injection motors, sample supports, frames for optical filters, enabling components based on electronics (352), or even part of the microscope body, etc. (353). The adoption of OSS or OSH solutions is also strictly dependent on their discoverability, modularity, and the standardisation of the software interface (354).

The revolution of openness in scientific software and hardware is not necessarily in opposition to the business model of microscopy companies (355). In fact, at the request of bioimaging researchers, many companies offer support for the use of open software, such as OMERO (supported by Glencoe ⁶ 356), or have opened part of the software by offering an API to interact with some of the GUI modules, such as Zeiss ⁷ with APEER platform for deep learning (357) or Abberior ⁸ with the possibility to reprogram the hardware configuration (358). In addition to the highly beneficial effect on the broader research community, we believe that companies can also benefit from the openness of both software and hardware. This applies whether resources for science are scarce or research is well funded, because there is always a business model that can be adapted to provide a service for less experienced users (355), and the wide adoption of open imaging solutions by companies enlarges potential customer markets.

The need for openness is even more pressing when AI solutions are implemented, as AI methods are inherently based on probability and as such not prone to reproducibility, and often the general audience employs such methods without a thorough understanding of their applicability and limitations. The second part of this Review aims to clarify some of the concepts and uses of AI for microscopy and immunology.

The concept of artificial intelligence takes root in Leibniz’s characteristica universalis, a common unified language of pure thought in which every language could be translated, and in calculus ratiocinator, a machine capable of replicating that language (359). Computers were meant to take a set of rules and input data and return some output, but could they think autonomously or even generate novel ideas (360)? This question is still guiding research in the AI field. In the 1940s and 1950s, progress in computational capacity motivated people to explore applications in the domain of pattern recognition, where the human brain excelled. In a seminal paper, McCulloch and Pitts (361) proposed the model of a network that took inspiration from the structure of the brain. This network was composed of a single input and output neuron, with an activation state that would contribute to the final output of the network. Later, developments on this original idea extended network complexity to a multi-layer network (362) and developed algorithms to train networks with more than one layer (backpropagation, 363–365).

In today’s technologies, artificial Neural Networks (NN) are built from a collection of nodes (neurons), operating a set of transformations on the input data to learn different representations of it. Nodes can be aggregated in layers and are connected by activation functions (synapses) computing a weighted sum of their input data. When the NN is trained, some connections get stronger, causing them to acquire a higher weight, while others get weaker, thus reducing their weight. So, NN training is essentially a problem of optimisation of parameters (366, 367). Training occurs iteratively: at each cycle, the network’s weights are optimised, and a loss function — an objective measure of training success — is measured (368). This process continues until a stopping point, such as reaching a specified value of the loss function or after a certain number of iterations. A network may perform poorly because of lack of convergence to validation data (underfitting) or lack of generalizability (overfitting). The choice of loss function is an important part of model design (369), along with the definition of layers and their connection types, which are collectively referred to as network architecture. For example, annotated tumour areas in tissue slices are used as ground truth, and the NN predicts which areas could be classified as tumour in the same slice (370). The loss function estimates how precisely the network predicts tumour areas. After the training phase, the network is applied to predict labels (tumour or healthy) on new tissue slices.

Machine learning (ML) is a subfield of AI ( Figure 3A ) that enables systems to learn from data without being explicitly programmed. It focuses on building models or algorithms that can make predictions or decisions by identifying patterns in the data and using them to improve performance over time (378). There are four main types of ML approaches: supervised, unsupervised, self-supervised, and reinforcement learning. Supervised learning is trained on a previously labelled data set that represents bona fide the desired outcome, so both the input and desired output are known. This highlights the importance of preparing a training data set that is most representative of the desired outcome, a simple task in principle but one that should be performed with great care (379). By contrast, unsupervised learning finds a structure from the data itself without any prior labelling information. This is a commonly employed technique when unbiasedly clustering information ( Figure 3B : middle row) and grouping differences in classes. For example, k-means clustering or Principal Component Analysis (PCA, Figure 3B : middle row) belong to this category. Self-supervised learning finds a classification by predicting or completing parts of its input, creating labels automatically from unlabelled data. For example, a self-supervised model has been developed to automatically learn semantic relationships between genomic data and improve tasks such as gene annotation or the role of polymorphisms (380). Lastly, Reinforcement Learning (RL, Figure 3B : bottom row) involves an agent interacting with an environment and learning through trial and error. It’s often used in broad AI tasks, such as AI-assisted game-playing and autonomous driving systems (376). Any ML workflow comprises a set of input data, a model architecture, and one or more loss functions. ML models can range from low-complexity models with few layers of data transformation – shallow learning – to higher complexity models involving many layers and many connections between them – deep learning.

Deep learning (DL) is a subfield of machine learning ( Figure 3A ) where many layers of data representation are connected to create a complex model with multiple levels of abstraction. Even though many foundational concepts and algorithms have been developed in the twentieth century, practical advancements in DL are relatively recent. Three practical steps have contributed to these advancements and to a general renaissance in the field of AI (368). First, the development of small yet significant algorithms enhanced how these deep stacks of layers can be interconnected (366, 367). Second, bigger storage was available to host the growing training data sets. Lastly, cheaper hardware increased computational power. In fact, DL models are based on simple additions and multiplications of big multi-dimensional arrays of data (called “tensors” in mathematics), and these operations can be easily parallelised. The development of powerful graphical units (GPUs), originally designed to improve the gaming experience but with the ability to be programmed for massively parallelised calculations and scientific computing, led to the implementation of GPU-based NNs (381, 382). Today, along with gaming GPUs, researchers can also leverage dedicated GPUs, optimised for DL tasks, and specialised hardware such as tensor processing units (TPUs), with the potential for requiring less computational resources and becoming an integral part of all domains of science, including microscopy. However, practical implementation of these networks still requires programming skills, with Python being the primary development language; popular frameworks include Tensorflow/Keras (368) and Pytorch (383). DL algorithms are affected by hardware bottlenecks in steps that are not hardware accelerated, therefore some attempts have been made to leverage alternative electronics boards like field-programmable gate array (FPGA, 384), which offer the flexibility of reconfigurable circuitry. For applications requiring low power consumption, FPGA have shown to be from 3 to 5 times more efficient than GPU per processed image, for tasks such as image compression (385). The advent of “liquid” and more efficient hardware such as FPGA will dictate the pace by which AI methods are implemented in microscopy and other fields.

To conclude, ML and DL techniques include computational systems capable of learning from data. Shallow ML systems continue to be utilised for their rapid training, minimal computational resource requirements, and low complexity, allowing for a complete understanding of how they operate. In contrast, DL systems require significantly more computing resources but have considerably higher capabilities, thus aiding all areas of microscopy, from experimental design to image acquisition, analysis, and data mining (386). These two types of systems are often used together in algorithms mixing classic programming, traditional ML, and DL according to the task (this permeability is reflected in the dashed lines of Figure 3A ).

Image-based ML methods can preprocess images or segment, detect, and track objects within multidimensional microscopy images. The most used approach is supervised learning, which employs a manually segmented data set to train the model. Traditional, shallow ML techniques for image segmentation include Support Vector Machines (SVM) and Decision Tree classification ( Figure 3B : middle row). Such ML models can yield good results across various applications and are featured in numerous open-source image analysis platforms (ilastik (322), Fiji/ImageJ with WEKA (387) and LABKIT plugins (388), QuPath (332), CellProfiler (323), MIB (389). They are user-friendly, can be used even without programming experience, and demand fewer computational resources compared to DL (388).

On the other hand, DL approaches for image processing have expanded the range of problems that can be solved (366). In the case of image-based methods, they are primarily employing a type of NN called Convolutional Neural Network (CNN, Figure 3B : top row, 359, 379). CNNs use layers to extract hierarchical representations of input data, in a way similar to how we learn information on an object by viewing it from different distances or angles. These layers implement two data processing functions: the convolutional filter (hence the name) and the max pooling function. Convolutional filters act like a magnifying glass that scans an input image while applying different kernels (a small matrix) to the image at each position. These kernels help identify specific features within the image, such as edges or corners. The result of this operation is referred to as a feature map, which highlights where these features exist in the image. Max Pool operations are used after convolutional layers to reduce the spatial size of the data while retaining important information, thereby making the network more efficient for computational analysis. Here, we will survey the landscape of current applications of image-based DL methods to the microscopy modalities that we previously discussed, highlighting, where existing, their applications to immunology or virology ( Table 1 ).

A major theme in DL applications to many microscopy techniques has been finding ways to increase the wealth of extracted information and overcoming the limitations of the specific techniques, such as increasing resolution without sacrificing acquisition speed. In cryo-ET, DL has been used to learn structural information from single-particle cryo-ET analysis (390) or to achieve isotropic resolution without the need for sub-tomogram averaging (391). In single-particle cryo-EM, DL aided particle model-building by creation of intermediate-resolution maps (392) or model building automation (393). In virology, DL with single-particle cryo-EM has been instrumental to the characterization of tegument architecture in human cytomegalovirus (394). In super-resolution, and specifically in localisation microscopy, DL has been used to increase the acquisition speed, by reducing the number of images needed to reconstruct the structures of interest (109), or to help in localising multiple adjacent emitters in 3D, thus improving volumetric reconstructions (110). In immunology, DL with STED has been applied to identify Zika virus reorganization of the endoplasmic reticulum (111). In SIM microscopy, DL can increase resolution and speed (395–397), thus better capturing live-cell events with lower phototoxicity. In confocal and spinning-disk microscopy, DL can increase image resolution (113, 117), reduce optical aberrations (114), and improve FLIM lifetime determination with low photon budget, as in fast live-cell imaging (115). In TIRF, DL helped in improving single molecule FRET (smFRET) by analysing single molecule traces (112). In wide-field microscopy, DL was used to enhance the resolution and optical sectioning capabilities (30, 123, 124), yielding confocal resolution while improving speed. In intravital, microscopy, DL approaches combined with two-photon excitation and adaptive-optics aberration correction have improved subcellular resolution without sacrificing acquisition speed (133). Finally, DL has been applied to light-sheet microscopy for physics-informed deconvolution, i.e. a combination of DL with optical information on the microscope setup (134). In light-sheet and confocal microscopy, DL also provided axial resolution enhancement, by learning from unpaired high-resolution, 2D confocal images and low-resolution 2D images from other planes (113).

Methods to improve resolution, signal or speed often apply to a specific imaging modality and do not translate well to other image modalities. Recent DL models have tried to provide a more general approach, for example when restoring fluorescence images from all imaging modalities (398), improving resolution without additional data acquisition (399), interpolating images between frames (400) or when performing object detection (401).

Major limitations of fluorescence microscopy are photobleaching, phototoxicity and limited speed when acquiring multiple channels. DL methods can overcome these limitations by providing in-silico labelling of transmitted light images (126–129). For example, DL with in-silico labelling of brightfield images has been used to improve the tracking in chemotaxis experiments (131), or to predict the lineage choice of differentiating hematopoietic progenitors (130). Finally, DL has been applied to phase, label-free imaging (125), or to achieve fast, volumetric live-cell microscopy of bioluminescent probes (402). In immunology, DL with optical diffraction tomography has been instrumental for label-free tracking of immunological synapse of CAR-T cells (403).

DL methods are now widely used for preprocessing, segmentation, detection and tracking tasks (379, 404). In image preprocessing, DL models are extensively used to denoise fluorescence images, as indicated by the many examples in the literature (290, 398, 405–414). In immunology, DL denoising was instrumental to improve contrast in Imaging Mass Cytometry, thus helping in characterizing the phenotype of immune populations in human bone marrow samples (415). DL helped in separating channels for filter-free imaging (125, 416, 417), and to assist tracking by improving linking accuracy (316, 320, 418). Also, DL is aiding cell phenotyping from multiplex immunohistochemistry images, for example to characterize tumour microenvironment in lung cancer (121) or pancreatic ductal adenocarcinoma (122). Finally, DL models can assess the quality of fluorescence images and identify artefacts (419). Moreover, DL has been applied to denoise low-dose cryo-TEM images (420).

DL is employed in supervised cell segmentation, such as in the case of general models U-Net (421), StarDist (371), Cellpose (422) and Segment-Anything Model (SAM) models (423, 424), which are available as plugins in many open-source and proprietary image analysis software. Specialised models tackle intracellular organelle segmentation (327, 425), segmentation of extracellular vesicles in TEM (103), HIV-1 virions in TEM (104) or mitochondria in FIB-SEM (426). In immunology and virology, DL has been applied to FIB-SEM images of SARS-CoV2 patient-derived platelets to segment α-granules or mitochondria (105). Ligh-sheet microscopy of lungs together with DL-based analysis allowed the spatial profiling of nanoparticle delivery to alveolar macrophages (135). Image segmentation is a challenging part of image analysis in multiplex imaging, where cells are often densely packed. In this case, DL was employed to perform cell segmentation, thus improving single-cell feature extraction (118, 119), or to perform a detection-based classification, therefore providing phenotypic analysis without segmentation (120).

In detection tasks from fluorescence images, DL helped in recognizing apoptotic cells from intravital multiphoton movies (427), phototoxicity in widefield time-lapse experiments (428) or, together with widefield high-content screening, to detect virus-infected cells and predict if they will follow a lytic or non-lytic infection (429). Furthermore, DL with confocal microscopy helped in classifying expression of TLRs from PBMCs of HIV-positive patients under ART therapy (430). In cryo-ET, CNNs are helping annotation and feature extraction for in situ identification of structures of the molecular components of interest (431), or template matching, i.e. detection of objects with an arbitrary shape, which is the most widely used approach in cryo-ET for particle picking (106). Still in cryo-ET, DL has been applied for finding macromolecules in cellular 3D tomograms (28). DL detection algorithms can also support feedback microscopy in real time (432) by automatically detecting events to guide acquisition, generate feedback, and predict cell fate.

In tracking, CNNs with intravital multiphoton microscopy have been used to accurately measure the position and shape of CD4+ T cells interacting with plasmacytoid dendritic cells in vivo, aiming to study interaction differences in lupus nephritis (132) or to link cell tracks in intravital imaging of leukocytes (433).

One of the limitations of supervised learning is the generation of accurate training data sets, which is, in most cases, a manual task that can be time-consuming and still prone to bias. A possible approach to overcoming this limitation is the use of self-supervised methods. For example, information from the OpenCell database was used to cluster proteins into organelles and individual protein complexes (434). Similarly, in another study DL was used to segment mitochondria, based on a training data set that was generated with DL (435). Such simulation-supervised approach could be, in principle, generalisable to other organelles or even to cellular segmentation in tissues. Self-supervised or weakly supervised models are also employed for cancer prognosis and diagnosis in pathology slides (436).

Overall, image-based DL methods are assisting a wide range of microscopy techniques in tasks from acquisition to image analysis and data extraction ( Table 2 ). Many of the above examples are widely applicable to different types of samples, including those related to immunology or virology.

Data-based ML can analyse, interpret, and learn from data. In this Review, we refer to data-based methods as the ones that can be applied generally to numerical or categorical data without the specific need for data to be generated with imaging techniques. For example, data-based ML is used in microscopy when clustering data extracted from images with the previously described image-based methods (442, 444), or when combining information from microscopy with text data generated with other methodologies, such as genomics or proteomics data (445). These tasks can be approached using traditional, shallow ML or DL.

Techniques using traditional ML include linear regression, logistic regression, and decision trees, such as Random Forest. For example, logistic regression was employed to predict MHC ligand, where a binding model and an antigen processing model were combined, and results were classified according to logistic regression score (446). Random forest was applied to analyse T cell-dendritic cell interaction in a lupus nephritis model (132).

Instead, unsupervised learning techniques are employed when the desired outcome is unknown or input data are not labelled. In this case, unsupervised learning can help clustering data in groups. Notable examples are the K-means, DBSCAN, and an unsupervised version of random forest algorithms (378). Clustering techniques have been used in high-throughput screenings to highlight differences between biological conditions when segmenting and measuring cells (374), or, in immunology, to cluster signatures and perform neighbourhood analysis in multiplex imaging in tissues (206, 439, 447) and single cells (246).

Another set of techniques, called dimensionality reduction, is used to group variables into “super variables”. Techniques falling in this category are PCA, t-SNE, and UMAP (378). Dimensionality reduction has been used to classify cell cycle and disease progression after feature extraction with CNNs (448), to segment touching cells in confocal and two-photon microscopy (449), to group clonal distribution of CD4+ T cells in gut epithelium following Listeria monocytogenes infection (450), or to obtain behavioural signatures of immune cells in intravital inflammation models, guiding the discrimination between pathogenic and non-pathogenic phenotypes (440). Also, dimensionality reduction techniques are essential in multiplex imaging when grouping cell phenotypes (240, 451).

DL for data-based methods can employ different types of architecture depending on the purpose. An Autoencoder Network (AE) is a type of NN that learns to encode input data into a lower-dimensional representation and then decode it back into the original form, thus learning meaningful representations from the data. This can be helpful for tasks like dimensionality reduction or feature extraction. AE ( Figure 3B : top row) has been used to combine low-dimensional representations of scRNA data, generated using large language models, with actual single-cell scRNA-seq data from different species to create a supergene classification that can bridge differences between individual single-cell experiments and different species (452), or to create deep generative models for spatial-omics analysis that can take into account the spatial relationship information (373). A Feed-forward neural network (FFNN, Figure 3B : top row) is another type of NN where information flows only in one direction (forward) through layers of interconnected nodes. FFNNs are commonly applied to regression and classification tasks, as they can learn complex non-linear relationships between inputs and outputs. For example, a feedforward network was used to classify cell tracks in 3D biomimetic gels of immune cells co-cultured with breast cancer cells in organ-on-chip (453).

Recurrent Neural Networks (RNNs) excel at processing sequential data like natural language text, where the context of previous words influences the use of subsequent ones. After dividing data into small chunks – called tokens –, RNNs process them recursively to generate the most likely information based on the previous information. For example, this type of network could predict the cell lineage of hematopoietic cells from brightfield images by extracting time signatures of cells from image features extracted with a CNN (130).

On the other hand, transformer networks weigh the importance of input tokens to construct a connection map without requiring sequential processing (454). They are at the basis of the Natural Language Processing (NLP, Figure 3B : bottom row) chatbots used today, such as ChatGPT, and are also called Large Language Models (LLMs). These tools can serve as a valuable resource aiding researchers in designing microscopy experiments (455) or drafting algorithms to implement the ML techniques described here. LLMs can be applied to data mining in scientific data sets, such as those generated from single-cell omics (as reviewed in 456), and some attempts exist to apply it to bioimage analysis (443, 457). We also foresee that these techniques will be increasingly integrated in microscopy hardware, for AI-assisted sample exploration and acquisition.

Another type of generative network is called Generative Adversarial Network (GAN, Figure 3B : bottom row). This network comprises two networks, one generating new data and the other evaluating its suitability as output. The generating network challenges the evaluating network, thus introducing an element of randomness and “creativity” in the output (458). This broad class of networks can be used to generate models of protein structures (the general AlphaFold (459) and its open version OpenFold (460), or a more specific version for proteins of the immune system (461)) or to reconstruct molecules from cryo-ET tomograms (390, 462). Generative neural networks informed on the features of highly metastatic melanoma by “reverse engineering” a supervised CNN for cell classification. In this example, a CNN is initially trained on patient-derived melanoma xenografts to classify them based on their metastatic capability. Then, a generative neural network is used to create in silico cell images with exaggerated features, which are then used to analyse which features in the CNN are most prevalent (463). This approach is particularly intriguing as it uncovers new quantitative insights within the hidden features of DL, thereby providing information that could potentially lead to the generation of new scientific hypotheses.

In summary, we outlined some applications of data-based ML techniques for microscopy ( Table 2 ). The field is vibrant and complex, and evolving at a fast pace. Shallow ML and DL can be employed to cluster information and combine microscopy data with multi-omics data (29, 441, 442), or to predict molecular biomarkers from pathology images (464). These technologies could aid vaccine design, as outlined in Hederman and Ackerman (465) or to improve antibody design (466). As such, a dialogue between computer scientists, microscopists and immunologists is fundamental.

AI can hugely facilitate the analysis of large image data sets and the characterisation of rare biological events. Because OSS solutions can rely on the contribution and critical judgement of the wide imaging community, they are indispensable for the development of trustable AI applications for bioimage analysis. In this regard, several software tools have been deployed during the past years, both as plugins for the major software GUI such as FIJI (StarDist (371), Cellpose (467), DeepImageJ (468)) or napari (SAM (423)), or as code notebooks (ZeroCostDL4Microscopy (335)).

In the previous sections, we outlined the main DL applications to analyse bioimage data, many of which are distributed as open-source software. Although some of these applications were not specifically developed with immunology in mind, their usage can significantly benefit immunological imaging. For instance, existing code can be adapted for specific biological questions, or their training data can be used to enhance other domain-specific DL models.

Furthermore, publicly available image databases are key in truly open-source AI models (469), as outlined by the Open-Source AI Definition ⁹ . In this sense, data sets specific to the immunological field (319), or for broader imaging purposes, such as the ones hosted on BioImage Archive (470), can constitute a valuable resource of image data to test AI software tools and foster immunological research. These data sets should follow the “Findable, Accessible, Interoperable, Reusable” (FAIR) principles (471).

Finally, the use of LLMs or other AI methods to aid software creation (443) will constitute an essential part of AI deployment and democratisation, as it empowers every scientist, even without programming experience, with the “wisdom of the crowd” provided by AI training data sets that can summarise a vast amount of human knowledge.

The role of AI for instrument control and automation can be delineated in at least two different ways: the first being the support of AI in automating the development of open-source code (443) for feedback microscopy, while the second involves utilising the AI-based methods described above to interpret the image data, thereby revealing information that can be used to redirect the acquisition process.

The use of open-source platforms Micro-Manager (349) or Pycro-Manager (348) for microscope control can be integrated with Python packages such as Scikit-Image ¹⁰ (472) for image segmentation and Scikit-Learn ¹¹ (473) to run ML tasks and feed results back into the acquisition software. As a practical example, DL segmentation methods have been used to identify cells and set the correct acquisition parameters (474) or switch microscopy modality (31) to image immunological synapses. Furthermore, cost-effective open hardware facilitates possible integrations of the acquisition microscope with AI feedback tools, for example to control anaesthesia, temperature, and humidity in intravital imaging (similarly to approaches tested in the clinics, 475, 476), or correcting the state of the optical system by acting on adaptive optics to minimise the loss of signal (238).

The implementation of AI for instrument control is automating the execution of precise and complex hardware tasks, shifting the troublesome duties from the human to the machine. In our view, rather than totally delegating the control of the experiment and the handling of expensive instruments to the automatic agents (as a kind of hardware/software AI-equipped decision maker), the implementation of AI tools should work as advanced technology to help the human researcher in steering the course of the experiment. In this regard, the integration of large language models and feedback microscopy is foreseen as the future evolution of microscopy, where the user doesn’t necessarily need to be highly skilled in all the aspects of microscope acquisition, hardware control, bioimage analysis: a microscopy platform could accept human language instructions and convert these into hardware control operations to provide the requested type of data sets (477).