Autopsy samples of coronary arteries were acquired from the department of pathology at Klinikum Rechts der Isar in accordance with federal state law and after approval from relatives (ethical approval number: 291/18 S). Coronary arteries were then cut by an experienced pathologist according to the standard AHA classification. OCT imaging of the single segments was then performed as previously described (15): After careful preparation, vessels were wired with a 0.014-inch guidewire over which the OCT catheter (2.7-F, St. Jude Medical, St. Paul, Minnesota) was advanced. An imaging pullback was performed from the distal to the proximal arterial segments while simultaneously flushing the vessel with contrast to improve imaging quality (pullback speed 5 mm/s = 120 frames/s). Afterwards, arteries were cut at 3 mm intervals and stained with haematoxylin-eosin (H.E.) and Movat-Pentachrome. The presence of different atherosclerotic plaque types and plaque components was evaluated by an expert in cardiovascular pathology (MJ) according to the classification by Virmani et al. (20), (see Table 1). Co-registration of OCT frames and histological sections of human autopsy samples was achieved as previously described (21) by first, considering only the proximal, middle or distal part of the OCT pullback for the respective slide and second, visual alignment of anatomical landmarks (calcifications, side branches, lumen contour) within each part of the segment.

OCT pullbacks from the OCT database at the German Heart Center Munich (including all patients undergoing intravascular imaging with clinical indication for OCT during coronary angiography) were screened. OCT imaging was performed according to current recommendations (22) with commercially available OCT systems (Dragonfly DF-OCT-catheter combined with the C7-XRTM imaging system; LightLab Imaging Inc., Westford MA, USA). Pullbacks were assessed for the presence of atherosclerotic plaque features as previously described (23) and included fibrous, lipid and calcified plaques (see Table 2). A total of 222 frames from 51 patients were used for analysis. In case of stented vessels, proximal and distal regions outside the stent were used. For baseline characteristics, (see Table 3).

OCT images were transferred to an offline working station and transformed into 8bit RGB images. Labeling of OCT frames was achieved using the freeware tool LabelMe (24). In each frame, the suspected plaque area was identified. Plaque area was measured based on histopathology and by tracing the leading edge in OCT images using a polygonal line tool, respectively. Plaque area was defined as the presence of lipid accumulation, necrotic core with or without foam cell infiltration and/or calcification (Figure 2). To avoid pitfalls as reported in (7), we included the guidewire artifact into the label “background”.

DeepAD is an ensemble of semantic segmentation networks performing pixelwise classification with respect to atherosclerotic lesions. In addition to histopathology-based OCT annotations, we employed so-called weakly supervised training by using OCT images annotated without histopathology. While histopathology-based annotations represent the gold standard, annotations based on OCT images contain valuable information for learning to segment atherosclerotic lesions, however, remain limited in resolution and diagnosis, hence providing a weak supervisory signal to the training (25). We trained and selected several models in so-called ensembles, an effective method to avoid overfitting on small data sets (26). All the models in the ensemble were created with the same network architecture. The differences in the models were the weight initializations of the networks, training batch sampling as well as optimization parameters. As no random seeds were set when training the models, these parameters were assigned different values at the start of each training round. This is a commonly used strategy to achieve different models as the network training of Deep Learning algorithms is stochastic and typically converges to different local minima (27). At inference time, all model predictions were averaged to obtain the final prediction for a given pixel. For further details on network architecture, hyperparameters, exact ensembling, (see Supplementary Material).

To leverage all the available data, DeepAD was trained several times using “leave one patient out” cross-validation. Specifically, for each training round, all histopathology-annotated OCT images from one patient were held out for testing. The remaining data was divided into a 80 % training and 20 % validation split, with no patient overlap between groups. The model was then trained with the training set and tested on the validation set. For each model in the ensemble, the model weights that performed best on the validation set were then tested on the held-out patient. This process was then repeated until each histopathology-annotated patient had been held out for testing once. The OCT images annotated without histopathology were never included in the test set.

Segmentation describes the task of linking certain regions or pixels within an image to a specific class label (here: lumen, lesion, other). We evaluated the segmentation performance using the intersection over union (IOU, also called Jaccard index), a common metric for evaluation of the quality of segmentation algorithms (28). The IOU is calculated as the overlap between the manual annotation (i.e. the “ground truth”) with the prediction from the algorithm divided by the union (see Figure 2A):

IOU can range from 0 (no overlap between ground truth and prediction) to 1 (complete overlap between ground truth and prediction; Figure 2A).

In contrast to segmentation (i.e. the pixel-wise classification), a-line classification computes a binary output regarding the presence of a certain manually annotated class or label in an a-line (e.g. a lesion). In our work, one a-line was defined as one angular direction originating from the center of the lumen. For each OCT frame, 360 a-lines represented the whole 360 ° vessel circumference. Classification was evaluated on a binary level, with a-line being positive if >3 pixels along the a-line contained atherosclerotic lesion and negative if no pixels along the a-line contained atherosclerotic tissue, as defined in previous works (11). Positive a-line signal (indicating presence of atherosclerotic lesion) was visualized by a green line (arch) in the respective vessel circumference. For results of a-line classification, please see Supplementary Material and Supplementary Figure 1.

In the histopathology data set, a total of 62 lesions from 7 patients were identified according to the classification by Virmani et al. (20) and manually co-registered with OCT (see Table 1 for details regarding the different plaque types) as previously described (21). For the clinical data set, a total of 222 OCT frames were identified as suitable for analysis (see Table 2 for details on the different plaque types). For examples from both datasets (see Figure 3). The histopathology-based annotations as well as the clinical annotations were annotated independently by two clinical and histopathology experts. The estimated interobserver variability for manual annotation of OCT frames with and without histopathology-based annotations was measured as the IOU between two independent expert observers. The median interobserver variability was 0.58 without and 0.71 with histopathology. Thus, the agreement was considerably higher when histopathology-based annotations were available. We also measured the alignment between annotations on the same scans with and without histopathology, being 0.61 across the two annotators. This showed that a difference in results was observed when annotating with or without histopathology.

DeepAD predicted atherosclerotic lesions with a median IOU of 0.68 ± 0.18 across all test patients (Figure 4A). Good and moderate performing examples (Figure 4B) illustrate that the DeepADs is highly accurate in the localization of atherosclerotic lesions across the lumen circumference as well as the difficulty of correctly estimating the extent of this lesion beyond the lumen border (see e.g. Figure 4B, moderate example). Figure 4B (lower row) illustrates a low performing prediction from DeepAD. Here DeepAD falsely predicts a lesion below the lumen which is not supported by the manual histopathology-based annotation. Overall, DeepAD predicts lesions with an IOU of 0.03 lower than the median IOU agreement between two observers, showing close to on par segmentations with expert annotators. To evaluate the importance of using histopathology-based OCT annotations, all OCT images in the histopathology data set were additionally annotated without histopathology by two independent observers. The algorithm was then trained with a dataset of the exact same size, but without histopathology-based annotations. The influence and importance of using histopathology-based annotation in the creation of the DeepAD algorithm was measured as the discrepancy of these models' performance with respect to the histopathology-based annotations. When the algorithm was trained using only annotations without histopathology, the median performance on the test set dropped by more than 0.25 IOU from 0.68 ± 0.18 to 0.42 ± 0.18 when predicting atherosclerotic lesion tissue (Figure 4A). This corresponds to a performance drop of around 33 %. In Figure 4B the predictions from DeepADs good, moderate and low performing cases are also shown for the algorithm trained without histopathology. In this case the segmentation algorithm yields false positive predictions (Figure 4B good performing example) and less extensive segmentation beyond the lumen border (Figure 4B median example). In the low performing example in Figure 4B both algorithms output similar false positive tissue prediction.

Prediction of calcifications in our data set yielded an average IOU = 0.34. Figure 5 shows representative examples of good, intermediate and low performance. In most cases calcification was correctly localized by DeepAD but incompletely predicted (i.e. not all of the calcification was predicted by DeepAD).

To illustrate the use of the DeepAD in real-world clinical practice, atherosclerotic lesion prediction was demonstrated in 11 cases of patients who presented to the German Heart Center and underwent coronary angiography and intravascular imaging with OCT. Performance of the algorithm was retrospectively compared against manual analysis of each pullback by an expert regarding the presence or absence of atherosclerotic lesions as well as correct spatial prediction of the DeepAD on a frame-level. 3D-rendering of the complete pullback allowed quick and intuitive visualization of regions with atherosclerotic lesions in red, (see Figure 6A). A total of 3284 frames were analyzed and high agreement was seen between automated prediction by DeepAD and manual analysis (mean agreement 88%, 2898 of 3284 frames), with good sensitivity, specificity and accuracy (86.8, 82.9 and 85.8%, see Table 4). DeepAD tended to slightly underestimate the presence of atherosclerotic lesions (66 vs. 71% by manual analysis). Representative frames of false-positive predictions are shown in Supplementary Figure 2. For examples on detection of calcification, (see Figure 6B).