Investigating the spatiotemporal distribution of particulate organic matter and plankton organisms is essential for enhancing our understanding of the ocean ecology and particle fluxes. Over the years, methodologies used for collecting particle data have evolved from traditional tools such as plankton nets and sediment traps to cutting-edge optical systems including satellites and in-situ cameras. Optical instruments tailored for oceanographic observations, in terms of shadowgraph cameras including but not limited to In-situ Ichthyoplankton Imaging System (IRIIS), Zooplankton Visualization and Imaging System (ZooVIS) and other devices such as the Video Plankton Recorder (VPR), Underwater Vision Profiler (UVP), Lightframe On-sight Key species Investigation (LOKI), Zooglider, Laser In Situ Scattering, Transmissometry and Holography (LISST-Holo) and digital holographic cameras have expanded our insight into ocean particle abundances with their high spatiotemporal coverage (Davis et al., 1992; Gorsky et al., 1992; Cowen and Guigand, 2008; Picheral et al., 2010; Stemmann and Boss, 2012; Bochdansky et al. (2013); Schmid et al., 2016; Choi et al., 2018; Ohman et al., 2019; Giering et al., 2020a; Picheral et al., 2022; Daniels et al., 2023; Takatsuka et al., 2024; Li et al., 2024; Greer et al., 2025). However, given the variety of platforms there has not been a consensus on what constitutes a particle, and what does not, a problem that is exacerbated by the numerical dominance of particles with low optical density (OD) and reflectance (Bochdansky et al., 2022). As a result, a large portion – if not the majority – of aquatic particles have not been accounted for by the optical systems listed above due to the particle-gel continuum (Verdugo et al., 2004; Mari et al., 2017; Bochdansky et al., 2022). In other words, the number of particles per volume is largely dependent on the sensitivity of the instrument, and the threshold settings used in the binarization step from the original grayscale image (Bochdansky et al., 2022). Particle concentrations can therefore only be compared among deployments of the same optical system and identical calibration (e.g., Picheral et al., 2010) and when different instruments are compared the offsets in particle numbers have to be aligned among instrument platforms (Jackson and Checkley, 2011; Zhang et al., 2023).

Prior research has suggested that the choice of analytical methods for processing in-situ image data significantly influences the derived particle information (Schmid et al., 2016; Giering et al., 2020b), which implies a well-designed image analysis protocol can extract more useful information, while a poorly constructed one may hinder the visualization of potential patterns within the same dataset. A variety of particle detection methods have been employed in plankton image analysis, including global binarization (Sieracki, 1998; Davis et al., 2005), Canny Edge detection (Ohman et al., 2019; Orenstein et al., 2020), unsupervised learning segmentation (Cowen and Guigand, 2008; Iyer, 2012; Luo et al., 2018; Cheng et al., 2020b; Durkin et al., 2021; Takatsuka et al., 2024), and other image processing pipelines (Olson and Sosik, 2007; Tsechpenakis et al., 2007; Cheng et al., 2020a; Picheral et al., 2022). However, the literature provides limited detail on these methods. Additionally, the threshold, a primary parameter determining the number and size of particles in the processing algorithm, has often been set arbitrarily. For example, in the Focused Shadowgraph Imaging (FoSI) approach used in this study, which depends largely on light absorbance and to a much lesser extent on scatter, a higher threshold (indicating lower sensitivity) tends to reveal optically dense particles only, whereas lower thresholds with higher sensitivity enable us the visualization of additional particles with low OD (Bochdansky et al., 2022).

Moreover, to study small and low-optical-density particles via FoSI, acquiring particle information from images is complicated by the fact that particles ranging from tens to hundreds of micrometers are often represented by only a few pixels. Global thresholding implemented based on the gray level of image pixels, can produce noisy and incomplete particle edges when dealing with faint particles, which has a larger effect on the reconstruction of smaller particles biasing particle number spectra. While Canny edge detection reduces noise in images, it tends to overestimate edges and overlook smaller particles (Ma et al., 2017; Stephens et al., 2019; Giering et al., 2020b). Misinterpretation of large particles using unsuitable algorithms, especially for those consisting of multiple small fragments, further distorts the particle number spectrum (Bochdansky et al., 2022).

Even with the application of a currently elusive, ideal algorithm in terms of extracting particle images, the challenge increases when attempting to compare data across different optical systems. As a result, these cross-comparisons are exceedingly rare. Establishing robust and standardized criteria is thus crucial for creating objective benchmarks for particle detection. To address this issue, we propose the use of stepped neutral density (ND) filters containing 11 gradients with OD values from 0.04 to 1.0 as a calibration procedure for FoSI. While ND filters have not been widely used in oceanography as a standard reference, OD has been employed in ZooScan to ensure image comparability across various laboratory settings (Gorsky et al., 2010). In photomicrography, one study used ND filters as reference slides to obtain consistent images (Kim et al., 2011).

Here, we first describe an image processing protocol ( Figure 1 ) tailored to the FoSI camera with neutral density filter calibration, preprocessing and quality enhancing algorithms, and the removal of images containing schlieren artifacts using machine learning. Implementing particle reconstructions at various sensitivity settings, we demonstrate the effect thresholds have on particle analysis. While our analysis is mostly geared towards shadowgraph imaging such as FoSI, ISIIS and ZooVIS, much of the image analysis tools provided here can be applied to other platforms as well. The narrative provides much detail and justifications for each individual step in the image analysis pipeline so that it simultaneously serves as an introduction and tutorial for novice users.

FoSI features a straightforward design with an inline configuration of illumination and camera resulting in high sensitivity and a large depth of field compared to non-shadowgraph imaging systems ( Figure 2 ). A red-light LED (625 nm, Cree XLamp) is used as a point source. The light is collimated by a 25.4 mm plano-convex lens (focal length = 150 mm) and travels through 24.4 mm thick sapphire windows that delimit a sample space of exactly 3 cm before it is focused by another plano-convex lens (focal length = 100 mm) into a 25 mm camera lens (Marshall Electronic, V-4325, f/2.5) attached to a board camera (ImagingSource, model DMM 27UR0135-ML, 1/3” sensor). While only the center of the image volume is perfectly in focus, image blur is tolerable at extreme distances of the image volume and the blur is symmetrical so that the edges of particles are rendered accurately after binarization (Watanabe and Nayar, 1998; de Lange et al., 2024). The images are labeled consecutively with the date and time in the format {Image number} + {YY-MM-DD} + {HH-MM-SS}. Sorting the image sequence in chronological order from the beginning to the end is a crucial prerequisite for analysis in MATLAB, as the implemented function dir in the current analysis does not sort the list of data by their labeled image numbers. The captured images are in grayscale and saved as bitmaps. We do not recommend using jpeg or other compressions as we found artefacts associated with these formats after binarization (unpublished results).

FoSI acquires 1280 × 960 pixels images, with a spatial resolution of 11.9 μm per pixel. The depth of field is adjustable depending on the oceanic system FoSI is deployed in. For the data presented in this study it was set to 3 cm, which lead to the total sample volume of ~5.18 milliliters per image before image subtraction. During image analysis, the image data were converted using function im2double() and analyzed as matrices of double-precision arrays rather than image variable (uint8 variable) because they allow the numbers to have decimals in their precision and are not limited to integers and the fixed numeric boundaries of 0 to 255. For instance, during the calculation of reference images, the rolling average of 100 sequential frames using uint8 variables cannot exceed 255, e.g., 100 + 200 equals to 300 numerically but in uint8, it is capped at 255. In other words, any pixel values greater than 255 are truncated to 255, and those less than 0 are set to 0.

Prior to image segmentation, the grayscale images were converted into binary images. A binary image contains only white (value = 1) and black (value = 0) pixels. The regions of interest (ROIs), which contain useful features, are represented by clusters of white pixels, while the black pixels form the background. The corresponding code for algorithms mentioned in the current study was created and executed using MATLAB 2022a (The MathWorks, Inc.).

Large batch analysis was conducted at Old Dominion University High Performance Computing Center (ODU HPC) run by bash scripts. We also compared beam attenuation, derived from light transmission data collected by a transmissometer (WET Labs C-Star, with a path length of 25 centimeters) and fluorescence (Chelsea Aqua 3) with the results analyzed by the current protocol, as well as UVP5 particle spectrum data from the same region and season but in a different year (Kiko et al., 2022).

The method described in this paper presented a detailed image analysis pipeline primarily designed for shadowgraph imaging. In contrast to other existing image analysis algorithms, our method more efficiently extracted information to small and low-optical-density particles from in-situ images. The accurate particle recognition protocol will support studies that require high-resolution data as a solid foundation to further investigate the biological significance of aquatic particles (Verdugo et al., 2004; Giering et al., 2020a; Irby et al., 2024; Huang et al., 2024).

The illumination correction involved the simple flat fielding method devoid of histogram equalization, as any change in image contrast may alter the preceding calibration model. Overcorrection occurred when some pixels in an image are extremely bright (close to 255) or dark (close to 0) (Xu et al., 2010). Piccinini and Bevilacqua, 2018 described a method to avoid overcorrection by maintaining all the pixels with 255 unchanged in the image before and after illumination correction. To resolve this issue in our protocol, we added an artifact mask to exclude those overcorrected pixels. Similarly, illumination correction has been applied to effectively acquire clean particle data in microscopic images (Leong et al., 2003).

Changes in OD are explicitly observed in shadowgraph imaging systems because the configurations enable us to identify this difference. A similar calibration is not feasible with other optical systems such as LISST which are based on scatter, and which lack the capacity to recognize this variation. Threshold calibration offers objective comparison across different camera settings accompanied by an explicit absolute OD value. Having a precise OD value that determines the analytic threshold provides an objective reference (OD) that makes the results comparable. As previously discussed (Bochdansky et al., 2022), the threshold selection process is highly arbitrary when particles are extracted from in-situ images and few researchers aware that varying these thresholds can result in differences of several orders of magnitude in the estimated particle abundance ( Figure 13 ).

Image subtraction is a simple yet critical step in image processing (Oberholzer et al., 1996; Russ, 2006). Gray levels in an image cannot be negative, meaning the image subtraction always results in a positive value. Meanwhile, even if the probability of overlap is low in oligotrophic environments (Conte et al., 2001; Gundersen et al., 2001), absolute subtraction might cause some errors when particles in two consecutive images overlap on the same pixel regions. In cases of high particle concentrations such as estuarine and coastal systems, we recommend filming deionized water images taken either before or after the cast as reference or by physically decreasing the imaging gap between light source and the camera housing to decrease the image volume. In oligotrophic systems such as in the Sargasso Sea and in deep sea environments as shown here, particles are so rare that particle overlaps are a statistically negligible problem.

In shadowgraphs, particles are typically darker than the background, but few exceptions arose during the analysis as we occasionally observed gel-like particles brighter than the background seawater, which was also observed in the presence of schlieren ( Figure 10 ). Most marine particles are known to be of higher OD than the surrounding seawater (Mobley, 2022). However, it is conceivable that gels as well as density discontinuities such as schlieren lead to a lensing effect in which light is concentrated in certain small regions of the image sensor.

In our search for an effective algorithm to analyze millions of small particles from large datasets, we conducted experiments on various edge detection methods, ranging from local thresholding to more complex techniques. Our analysis corroborated results of a previous study on particle detection techniques (Giering et al., 2020b). For example, using Canny detection alone tends to fragment large particles and introduce significant Gibbs ring artifacts. The noises around the large particles are likely to be amplified by Canny detection as wiggly lines. Under the same threshold, Canny detection was found to overestimate particle size, while other algorithms, such as Prewitt and Roberts edge detection (Roberts, 1963; Prewitt, 1970), struggled to detect faint particles ( Supplementary Figure 5 ). Although the Otsu method (Otsu, 1975) automatically determines the threshold based on brightness variation, it had been proved less effective for detecting small, faint particles (Cai et al., 2014; Yuan et al., 2016). In contrast to the algorithms above, our MicroDetect Algorithm was derived from the Canny edge detection algorithm as described by Nayak et al., 2015. To tailor the original algorithm for particle analysis, we eliminated the morphological opening filter, which tends to inflate particle size in our application. Additionally, we removed edges detected by the Canny method after filling in holes to minimize overestimation Figures 7A–E. Although machine learning algorithms are known for their effectiveness in image edge detection (Arteta et al., 2012; Irisson et al., 2022; Panaïotis et al., 2022; Yao et al., 2022), we believe that the MicroDetect Algorithm can be exceptionally practical for image classification following the ROI segmentation, hence decided to set aside the computationally expensive techniques.

The proposed feature-based schlieren detection can be programmed with relatively simple functions. Nevertheless, the applicability of our schlieren detection algorithm is limited to evenly illuminated images. When dealing with image data influenced by severely uneven illumination, the algorithm may fail. Another drawback is attributed to the area filter used to exclude schlieren, which may falsely eliminate large particles with high-aspect ratio, e.g., large medusae, some elongated marine snow particles, diatom chain or marine snow debris (Bochdansky et al., 2016). By contrast, the deep learning-based schlieren detection outperforms in terms of accuracy while the outputs depend largely on data quality during training. Since the training image set requires update when it is applied to a new camera setting, the manual selection and training greatly increase the complexity of the process.

Several studies made intercomparisons of particle parameters measured by different optical instruments. The study of Behrenfeld and Boss (2006) showed a high correlation at upper 200 m North Pacific of Coulter Counter-based particle (~1.5-40μm) data and c_p (r² = 0.77) which compares well with the results presented in the current study ( Supplementary Table 2 ). The spikes in the c_p were not seen in the FoSI data ( Figure 12 ), because they are rare large particles that pass through the light beam of the transmissometer (Briggs et al., 2013; Giering et al., 2020a). Our results resembled a LISST-100 study suggesting c_p is less sensitive to the change of particle size and correlates better with particle abundance than TPV (Boss et al., 2009). Another study inferred particle projected area reflects c_p better than TPV because the particles are imaged as two-dimensional projections (Hill et al., 2011), however, we did not observe this in FoSI data. Our results showed that the correlations ranked with c_p as particle abundance > TPV > particle projected area ( Supplementary Tables 2 , 3 ). Additionally, the statistical results indicate that higher sensitivity levels lead to a stronger correlation between c_p and particle abundance for particles ranging from 22 to 40 μm, with the r² peaking at thresholds between 13 and 19 (see Supplementary Table 1 for the corresponding OD values).

We selected marine snow, Rhizaria and Appendicularia to represent particles containing transparent features that sometimes lead to fragmentation during image analysis. Marine snow particles include aggregates greater than 0.5 millimeter that contain transparent gel matrices. Previous investigations revealed the unique, yet elusive characteristics of marine snow based on optical and fluid mechanical observations (Honjo et al., 1984; Petrik et al., 2013; Markussen et al., 2020; Chajwa et al., 2024; Cael and Guidi, 2024). Chajwa et al. (2024) visualized the mucus comet-tails on sinking marine snow particles with high-resolution Particle Image Velocimetry (PIV) and further inferred that the optically invisible fraction of marine snow visualized via the flow field around the particles contributed greatly to the particle size and their sinking velocity. While not perfect, our MicroDetect Algorithm recognized at least a large portion of the mucus matrix ( Figure 14A ). As the particle size increases, one may assume that sinking velocity of these particles increases according to Stoke’s law. However, it appeared that this is not universally true (Iversen and Lampitt, 2020). Since the gels are neutrally or even positively buoyant (Azetsu-Scott and Passow, 2004; Engel, 2009; Mari et al., 2017; Romanelli et al., 2023), they reduce the particle sinking velocities as clearly demonstrated by Chajwa et al. (2024). This indicates that for the same image data, the improved object detection algorithm will extract more information on the potential sinking velocity of particles. Rhizaria are a group of protists abundant in the mesopelagic zone. Studying them has been challenging owing to their fragile structure and easily being dissolved in fixatives (Matsuzaki et al., 2015; Nakamura et al., 2015; Biard et al., 2015, 2018; Biard and Ohman, 2020; Mars Brisbin et al., 2020). Consequently, researchers use in situ optical devices (Biard et al., 2016, 2018; Nakamura et al., 2017; Biard et al., 2020; Mars Brisbin et al., 2020). The MicroDetect Algorithm detects the endoplasm and spicules of Radiolaria as well as the much more transparent ectoplasm that harbors algae and food vacuoles and is difficult to detect by most photographic systems (Ohtsuka et al., 2015) ( Figure 14B ).

Similarly, Appendicularia and Cnidaria, both broadly distributed zooplankton worldwide (Cairns, 1992; Båmstedt et al., 2005; Capitanio et al., 2008; Kodama et al., 2018; Williams, 2011) exhibit transparent features in their bodies that are not rendered well with conventional image analysis tools but are better outlined using the MicroDetect Algorithm (Figure 14C). However, FoSI cannot visualize the entire transparent structures on Appendicularian houses even at higher sensitivities (Bochdansky et al., 2022). While the MicroDetect Algorithm more accurately recovered their actual sizes, the loss of image details will hinder the taxonomic identification of these plankter. In fact, the ROIs extracted from the original gray scale images will be a better fit than the binary images in terms of identification using machine learning algorithms.

In conclusion, this paper provides a detailed step-by-step workflow for the analysis of oceanographic images from a shadowgraph system. The primary advancement of our method involved the development of the MicroDetect Algorithm to better reconstruct the ocean particles from 24 to 500 μm. Additionally, the novel protocol significantly improved the recognition of gel-like structures in the water column and transparent parts of plankton organisms. This highlights potential new approaches for studies of the oceanic gel phase independent of traditional Alcian Blue staining methods. Finally, we addressed the most vexing problems in image analysis and hope that by the application of neutral density filters as presented here, a standardization of different shadowgraph cameras can be achieved in the future.