We conducted a validation study in a simulated setting in June 2022 at the NICU of the Erasmus MC–Sophia Children’s Hospital. This NICU comprises four open baby sub-units, each with eight beds, and admits approximately 600 neonates annually. During the study period, the clinical staff included 17 neonatologists, nine residents, 91 nurses, 16 nurse trainees, 22 nursing assistants, and nine nurse practitioners.

Before the study, the NICU staff was informed through a clinical lecture and an announcement in the NICU newsletter. The validation took place at an NICU incubator bedspace using a prepared preterm doll lying in an incubator (Figure 1), simulating a ELBW infant weighing approximately 690 g and measuring 35 cm in length (SOMSO Premature Baby, Female, SOMSO MODELLE GmbH, Coburg, Germany). The doll was equipped with a ventilation tube and a feeding tube both secured to the nose and forehead with commonly used tape, and was positioned in a supportive ‘snuggle’ with a rolled-up towel around the legs (Figure 2).

The Clinical Technological Research–Human Research Ethics Committee (KTO-HREC) of Delft University of Technology approved the study (Application number KTO-HREC21005). All collected data were stored anonymously.

The M3D scanner used in this study was in the prototyping stage and not commercially available (Zevenbergen, 2020). The scanner consists of three (eye-safe) 3D cameras (Intel RealSense, Santa Clara, California, USA) assembled in fixed positions within a rigid frame that fits precisely over the transparent cover of a Caleo incubator (Dräger, Lübeck, Germany), positioning the three 3D cameras around the head. (Figure 1).

Obviously, the cameras cannot capture the part of the head touching the matrass. To capture all sides of the head, three separate scan sessions are needed, each with the head in a different position, facing up, right and left. The three scans result in a total of nine overlapping 3D images, covering all areas of the head.

To prevent stress from the head position changes required for the scans, the three sessions are to be synchronized with the protocolized head position changes, which are part of standard NICU standard care. These position changes occur every three to 4 hours, aiming to prevent skull deformations (Kubik et al., 2024; De Koning, 2020). Needed three capture should be done on the same day, during two or maximum three successive routine care moments. If intervals between captures are too long, the head could be significantly modified by growth, creating bias in the measurement. The head position change is integrated with routine care procedures, such as diaper change. By aligning the M3D scan sessions with the routine care moments, the first scan is conducted with the head in the initial position at the start of the routine care. Subsequently, routine care procedures are performed, concluding with the infant in a different head position for the second M3D scan. After three to 4 hours, the infant’s head is repositioned once, providing an opportunity for the third scan.

Pairs of NICU nurses conducted scan sessions using the M3D scanner with a preterm doll (Figure 2) placed in an incubator (Figure 3), specially prepared to simulate a NICU routine care moment. Researchers first demonstrated how to operate the M3D and clarified expectations. Each complete scan session entailed three separate 3D-scans, with the doll’s head position changed between scans. One nurse operated the scanner (Figure 4), while the other assisted in repositioning the head, mirroring routine care practices. After the three scans, the nurse that operated the scanner measured HC using measuring tape, adhering to standard protocol (Figure 5). After completing a session, nurses exchanged tasks and conducted another measurement session following the same procedure.

For each scan session, a comprehensive 3D image of the head was constructed by merging the nine raw 3D images into one. The transition from raw scans to HC/CrV values is illustrated in Figure 6. The merging of all scans was undertaken collaboratively by two researchers: a KTO-student and an Vectory3 employee, who divided the nurses’ scan sessions between them.

HC and CrV measurements were obtained from the 3D image using CraniumPy software (Abdel-Alim, 2023). In the calculation process, a rater manually marks three anatomical landmarks on the 3D head image: the left tragus, right tragus, and nasion (as depicted in Figures 6, 7). Subsequently, the software constructs a virtual plane passing through both tragi and nasion, computing the volume above this plane as CrV. To determine HC, the software automatically identifies the largest section above the ears, aligned parallel to the tragi-nasion plane.

To mitigate potential rater bias, HC and CrV were independently assessed from the 3D scans by three rater groups. The first rater group KTO-V3 comprised a Technical Medicine student and Vectory3 employee lacking clinical background, who evenly divided the acquired M3D scans between them. To reduce interrater variability, the two KTO-V3 raters underwent joint practice sessions for marking tragi and nasion points. The second (RG) and (TS) rater ‘groups’ each comprised a single researcher lacking clinical background. Prior to assessment, all individual raters received informal training in landmark placement techniques.

Accuracy and precision were characterized by the measurement error (ME). As depicted in Figure 8, accuracy and precision were delineated relative to the ground truth (GT), representing the true value of the measured object. The method for establishing GT is explained in the next paragraph.

Accuracy was measured by the mean or median absolute error (MAE) if ME was normally distributed, or the median absolute deviation (MAD) if not. Precision, indicative of spread of the ME, was quantified by the 95% reference interval of the total ME range, encompassing 95% of observed MEs. This interval was preferred over the full ME range to exclude outliers. For non-normally distributed MEs, the Q2.5 – Q97.5 95% reference interval was utilized, while normally distributed MEs were represented by the mean ± 1.96*SD distribution reference interval to cover 95% of MEs. Accuracy and precision were expressed both in measurement units (cm for HC, mL for CrV) and a percentage of the GT value.

The GT values were derived from accurate 3D scans from the doll’s head, obtained using a medical scanner with known high accuracy (3dMD head system, purchased in 2020, 3dMD, Atlanta, USA, with a manufacturer-claimed accuracy of 0.3 mm). These head scans are 360-degree scans. HC and CrV were measured from these scans by the same measuring method as described above, utilizing the CraniumPy calculation script. Four different 3dMD-scans were measured by five raters, resulting in a total of 34 GT measurements. As there were no extreme outliers, we used the mean of these measurements as the GT values for the doll used in this study.

The influence of raters on M3D measurements was assessed by comparing the MEs among the three independent M3D rater groups. To assess the overall accuracy and precision of the M3D HC and CrV measurements, the measurements from all three rater groups were combined and analysed as a single rater group.

For medical measuring instruments, the required accuracy and precision depend on their intended use, with the permissible ME being contingent on the smallest clinical change the instrument should detect (ISO IOfS, 2023). For the M3D scanner, this clinical change is the weekly growth increase in HC and CrV, as this scanner is intended for weekly growth measurements (Villar et al., 2016; Health UDo, 2009; Greer and Olsen, 2013). Figure 9 demonstrates the relationship between weekly measurements and the true weekly growth using a fictive data plot.

To ensure sufficient accuracy for the total population, the clinically allowed ME should be based on the smallest weekly increases. The weekly increases can be expressed in either absolute values (in cm for HC, and mL for CrV) or proportional increases (in %). Table 1 summarises the weekly increase in HC and CrV for preterm infants. We aggregated weekly HC and CrV growth data from multiple sources, including growth reference charts and published studies, to compile this information (Vermeulen et al., 2021; Young et al., 2020; RCPCH, 2019; Dieks et al., 2022; Cole et al., 1998). Detailed tables of the weekly increases are available in the supplementary materials.

We employed descriptive statistics to quantify accuracy and precision, along with MEs, as described above. Normality of measurement distributions and MEs was assessed using Shapiro-Wilk normality tests, supplemented by violin plots for visual representation. Rater group differences were evaluated via one sample T-tests for assumed normal distributions and Wilcoxon rank sum tests for non-normal distributions. Rater effects were examined using either one-way ANOVA, if normality was assumed, or Kruskal–Wallis tests, if assumed non-normal.

To visually depict accuracy and precision, we generated point plots illustrating MEs for each rater group and across all raters.

Statistical analyses were performed using R Statistical Software Version 4.2.1 (R Foundation of Statistical Computing, Vienna, Austria), with significance set at p <0.05 (two-tailed).

NICU nurses conducted a total of fifty-six scan sessions, resulting in successful acquisition of complete head scans in 25 instances (success rate of 44.6%), from which scans HC and CrV could be derived. Figure 10 illustrate a successful head scan along with its corresponding measurement outcomes. In thirty-one scan sessions, HC and CrV measurements were unsuccessful due to various reasons: a) model failure, where the back of the head was incomplete (n = 16); b) capture failure, where not all camera views per capture were saved (n = 13); c) capture failure, where the capture was accidentally overwritten by other participant (n = 1); and d) occlusion of the head, making the selection of reference marks impossible (n = 1).

Statistics for HC measured values and corresponding measurement errors are presented in Table 2. Figure 11 provides a visual interpretation of the distribution of HC measurement errors for each rater. Detailed statistic tables are available in the supplementary materials. Shapiro-Wilk normality tests and violin plots indicate that the measurements are not normally distributed for both individual rater groups and the combined raters. Most M3D measured values are below GT. For all individual rater groups and the combined raters, the median HC measurements deviate significantly from the GT. The Kruskal–Wallis test further confirms that the raters’ influence on median HC across all groups is significant (p= 0.003). Accuracy and precision metrics are presented in Table 2. While the measuring tape has a higher accuracy with a median absolute deviation of 0.2 cm (proportional 0.9% of GT value) compared to M3D’s accuracy (combined raters) of 0.4 cm (1.5% of GT), its precision of 1.6 cm (6.3% of GT) is more than twice as low as M3D’s precision of 0.7 cm (2.9% of GT).

Table 3 presents statistics for CrV measured values and CrV MEs. Figure 12 visualizes the distribution of CrV measurement errors for each rater. Shapiro-Wilk normality tests and violin plots indicate that the measurements are normally distributed for all individual rater groups and the combined raters. Most M3D measured values are below GT. For all individual rater groups and the combined raters, the mean CrV measurements deviate significantly from the GT. The one-way ANOVA further confirms that the raters’ influence on mean CrV across all groups is significant (p= 0.003).

Potentially the most significant clinical advantage of the M3D’s measuring technique, if successfully integrated in NICU care, is adding CrV as longitudinal growth parameter for ELBW infants. Several studies address the clinical added value of 3D volumetric growth data relative to traditional 2D data such as HC (Vermeulen et al., 2021; Burkhardt et al., 2019; Santander et al., 2020; Martini et al., 2018; Meyer-Marcotty et al., 2018; Ifflaender et al., 2013). In relation to this, skull deformations, a relatively common issue in preterm infants, can distort HC measurements, leading to inaccurate assessments. As deformations influence HC, but not skull volume, this also speaks for CrV as more reliable longitudinal growth parameter than HC. Skull deformities such as plagiocephaly can be caused by a prolonged head position, despite protocolized head position changes every three to 4 hours (Kubik et al., 2024; De Koning, 2020). In addition, deformations can also result from continuous positive ear pressure (CPAP) breathing systems, attached with tight straps or hats.

Few other studies have explored feasible techniques for measuring preterm CrV. Dieks et al. (Dieks et al., 2022) conducted longitudinal CrV measurements on 26 preterm infants from 28 up to 40 weeks GA using a handheld 3D camera, suggesting its feasibility for routine neonatal care (Dieks et al., 2022). Similarly, a literature review identified the same handheld 3D camera as suitable for stress-free body size measurements of extremely preterm infants in incubators (Van Gils et al., 2022). However, this method requires eye protectors and a nylon cap on the infant’s head, along with captures from ten different viewing angles, necessitating removal of the incubator’s cover and repositioning the infant, or holding the infant in a parent’s arms (Santander et al., 2021). Vermeulen et al. measured CrV growth using a tabletop 3D scanner, but this approach necessitated placing infants completely outside the incubator (Vermeulen et al., 2021). This would be extremely disruptive and potentially hazardous for preterm ELBW infants due to risks of hypothermia or accidental extubation.

The success rate of scan sessions in our study was 44.6% (25 out of 56 sessions), which we consider too low for clinical use. In comparison, Andrews et al., 2019, in a similar study measuring HC and length of preterm infants with a 3D camera, reported a success rate of 76% (Andrews et al., 2018). One suspected reason for the high failure rate in our study is the lack of immediate feedback (the resulting 3D image) following a scan. Such a ‘preview’ could have aided in assessing scan success, for example, in cases where the view may have been obstructed by artefacts such as endotracheal tubes. Additionally, the cameras’ limited field of view created ‘blind spots’, leading to missing 3D data and hindering the construction of a comprehensive head image.

The precision of the existing measuring tape appears insufficient for weekly HC measurements, as indicated by the data in this study. This conclusion stems from the measuring tape’s ME interval of 1.6 cm (equivalent to 6.3%) significantly exceeds the anticipated weekly HC growth range of 0.4–1.3 cm. Despite the measuring tape demonstrating better accuracy, indicated by its lower median absolute error compared to the M3D scanner, the scanner’s superior precision makes it the preferable option.

However, it is worth noting a potential bias in the M3D scanner’s accuracy due to a systematic error, discussed in detail below. As a limitation in HC measurements, we note that the HC measurements with 3D scans, including the HC GT measurement, measured (the largest section) parallel to the tragi-nasion plane, where the manual measuring tape, as stipulated in the NICU’s protocol for HC measurements, measured at the largest circumference ‘above eyebrows and ears’. This might have caused small bias when comparing measuring tape measurements with GT.

The raters in this study had a significant influence on the accuracy for both HC and CrV measured using the M3D scanner. Because landmarks were manually placed, significant interrater variability was anticipated. However, the plots show that interrater precision for M3D measurements, represented by the spread of M3D ME, hardly differed between raters. In contrast, the intrarater variability was larger for CrV than for HC, indicating that CrV measurements are less precise than HC measurements. This is likely because the calculation script is more sensitive to user input for CrV than for HC. For HC, the exact position of the tragi-nasion plane has a minor influence, because the script ‘searches’ for the largest section above the ears parallel to this plane. In contrast, for CrV the location of landmarks directly affects the volume calculation above the plane. This effect is further enhanced by the small size of the doll’s head and the cubic nature of volume measurements, where small differences in landmark positions result in relatively large CrV differences between raters. Therefore, this technique makes it is harder to accurately measure CrV than HC.

Several studies have used marking of the nasion and tragi for deriving HC and CrV from 3D scans and noted user influence on the measurements (Burkhardt et al., 2019; Ifflaender et al., 2013; Barbero-García et al., 2017; Geil and Smith, 2008; Weinberg et al., 2006). Tareq Abdel et al. developed CraniumPy software (Abdel-Alim et al., 2023) for assessing head morphology and size, validating its intra- and interrater reliability of the method with full 360-degree, highly detailed scans of infants, though not preterm infants. Their study showed that manual marking of landmarks introduces an intra- and interobserver variability of 3%. Ifflaender et al. measured HC and CrV of preterm infants and found high agreement between raters, with mean difference between raters of −0.1 ± 0.74 mm (−0.005% ± 0.2%) for HC and 8.1 ± 6.5 mL (1.5% ± 1.1%) for CrV (Ifflaender et al., 2013).

Our primary limitation for accuracy and precision was the lack of morphological details in the 3D images from the M3D scanner, making it difficult to distinguish anatomical landmarks such as the tragus and nasion. The low resolution of the depth cameras was especially problematic around the ears, complicating the exact location of the tragus. Additionally, fixation tape used to secure the feeding and breathing tubes in the nose, covered the nasion, making it more difficult to locate. The absence of colour in the 3D images further hindered the identification of the anatomical points, as CraniumPy only provided images in grey. The lack of detail in the raw 3D images also made it harder to merge the three captures into an accurate and comprehensive head image.

A suspected systematic technical error may have further influenced inaccuracy. This suspicion arises from the observation that the mean measurements from all individual rater groups were consistently below GT. We believe the cause lies in the complex chain of actions from 3D scans to the final measurements.

Using a doll instead of real patients allowed for the comparison of measured values against accurately defined GT values, which we consider a strength of this study. Most studies assessing 3D body size measuring techniques compare the new method’s performance relative to an existing method (Van Gils et al., 2022). However, obtaining accurate GT values for 3D body size measures such as CrV in real patients is often too complex, with CT or MRI too invasive or expensive. Comparing a novel device only against the traditional ‘gold standard’ technique is debatable: a perfect correlation does not necessarily indicate how close the measured values are to the true GT values. In fact, the new technique’s measurements may be closer to the true values than those obtained with the traditional ‘gold standard’ instrument due to unknown limitations of the latter. A limitation of our study is that it is impossible to define GT with 100% accuracy. The clinical 3dMD head system scanner we used had an accuracy of 0.3 mm, as stated by the manufacturer. Additionally, there is interrater variability in placing landmarks in CraniumPy (Abdel-Alim et al., 2023).

Future studies should focus on improving M3D’s success rate of scans and accuracy and precision. Such a technically improved M3D scanner should first be validated in a simulated setting to confirm accuracy and precision before being validated with real patients. Such study with real patients could collect the first-in-the-world longitudinal preterm CrV data of infants <28 weeks GA. The most effective step to further improve the accuracy and precision of the M3D scanner would be more accurate marking of anatomical points, requiring 3D images with better morphological details and added colour information. Potential systematic errors should be analyzed and addressed under controlled lab conditions. To enhance the success rate of scan sessions, the 3D image should be displayed directly on screen after a scan, allowing users to verify if a capture was successful or needs to be redone. Ideally, the M3D measuring technique should be expanded to include body length in addition to HC and CrV, creating an all-in-one device for stress-free growth monitoring of ELBW infants. In addition, with improved scan quality, the scope of the M3D could be further extended to morphological head analysis, such as monitoring of preterm skull deformations, for preterm infants that cannot be scanned outside their incubator.