The study was divided into three different stages. The first part, i.e., “software design,” was carried out to implement online mass spectra data analysis software. To this end, the imipenem hydrolysis assay conditions were the same as those established by Oviaño et al. (2019), using a collection of 74 genotypically characterized CPE and data analysis with the Clover MS data analysis software (Clover BioSoft, Spain)¹. The results were compared with those obtained with the MBT STAR^®-Carba IVD Kit (Bruker Daltonik) and STAR^®-BL IVD software. The principle of the assay is that imipenem is inactivated due to hydrolysis of the β-lactam ring by bacteria expressing carbapenemase-hydrolyzing enzymes. The hydrolysis reaction modifies the structure of the antibiotic, so disappearance in the native mass peaks (300 and 489 m/z) can be detected by MALDI-TOF. Thus, a positive hydrolysis reaction denotes the presence of a carbapenemase in bacteria. This first part of the study process was accomplished by the Complejo Hospitalario Universitario A Coruña (AC), A Coruña, considered the reference center.

The second part of the study, i.e., “intercenter evaluation,” was designed to standardize the methodology and evaluate the reproducibility among the different centers participating in the survey with the proposed workflow. The same isolates used in the first part of the study were analyzed in the eight participating centers: the Hospital General Universitario Gregorio Marañón (GM), Madrid; the Hospital Universitario Ramón y Cajal (RC), Madrid; the Hospital Universitario Reina Sofía (RS), Córdoba; the Hospital Universitario Virgen Macarena (VM), Seville; the Hospital Universitario Son Espases (SE), Palma; the Hospital Universitario Central de Asturias (CA), Oviedo; the University Hospital Plzeň (PZ), Plzeň; and the Cliniques Universitaires Saint-Luc (SL), Brussels. All reagents were provided by the reference center and solutions were prepared in all centers according to instructions. So, the centers were given a precise protocol with the standardized procedure and an educational video for software management (Supplementary Data Sheet 1 and Supplementary Video 1). Technical support regarding the laboratory procedure and general attention together with software management were available during the study. The results of the MALDI-TOF assay were compared with those obtained by molecular characterization of the isolates.

The last part of the study, i.e., “clinical validation,” was designed to evaluate the proposed methodology and data analysis in real clinical settings. During a 2 months period, all isolates suspected by EUCAST screening criteria (i.e., with a meropenem or ertapenem MIC > 0.125 mg/L) to be carrying a carbapenemase enzyme (maximum 20) were evaluated in parallel in the participating centers using the MALDI-TOF MS assay based on imipenem hydrolysis. The results obtained were compared with those yielded by the reference method used in each clinical laboratory for CPE detection.

A representative collection of 76 non-duplicated clinical isolates (retrospectively obtained from our own collection of isolates belonging to AC) was used in the first and second parts of the study. These included 74 isolates (46 CPE and 28 non-CPE) and 2 control strains. The control strains used were a E. coli ATCC 25922 (negative control) and a PCR-confirmed VIM-producing Citrobacter freundii (positive control). The 74 test species included in the study were 39 Klebsiella pneumoniae, 26 Escherichia coli, 8 Enterobacter cloacae complex, and 1 Klebsiella oxytoca. The isolates were characterized by PCR and sequencing (Oteo et al., 2013, 2016; Dortet et al., 2016; Oviaño et al., 2017). The collection included 46 CPE isolates, being 10 bla_KPC, 3 bla_IMP, 5 bla_NDM, 10 bla_VIM and 18 bla_{OXA–48–like}, and 28 non-CPE isolates: 1 bla_CIT, 1 bla_SHV, 2 bla_CMY, 3 bla_FOX, 1 bla_K–1, 14 bla_CTX–M, and 6 fully susceptible isolates (see Supplementary Table 1 for further details). All non-CPE isolates have carbapenems MIC below the cut-off for screening according to EUCAST criteria, excepting two isolates, one E. coli and 1 K. pneumoniae. Both were tested by PCR and sequencing and no carbapenemase enzyme was found.

In the final part of the study, each center prospectively evaluated (following EUCAST screening guidelines) suspected CPE isolates that emerged in routine testing in clinical microbiology laboratories, i.e., the number of isolates that emerged during the 2 months period or up to 20 isolates. One isolate per patient included. The isolates were derived from different clinical samples. Results were compared with those obtained with the reference method used in each laboratory.

Bacterial isolates were stored at −80°C, in a small vial containing glass cryopearls (Deltalab, Barcelona, Spain). On day 1, isolates were thawed on a blood agar plate by removing one of the pearls from the tube with a sterile loop and rolling it on the plate. After incubation for 18 h, the isolates were subcultured for another 18 h on a blood agar plate for analysis under standard conditions. The samples were incubated in an aerobic atmosphere at 37°C. Briefly, bacteria filling a 1 μl inoculation loop were suspended in 50 μl of solution (10 mM NH₄CO₃, 10 μg/ml ZnCl₂, 0.001% SDS; pH 8) containing 0.5 mg/mL of imipenem (Sigma-Aldrich, Germany) and incubated for 30 min at 37°C with slow agitation (300 rpm) (Oviaño et al., 2019). The samples were then centrifuged at 14,000 rpm, and the supernatant was applied to a MALDI-TOF MS target plate. Each sample was spotted on the plate in duplicate. Once dried, 1 μl of matrix [Matrix IVD HCCA-portioned (Bruker Daltonik) spiked with 1 ppm/μl of reserpine (Sigma-Aldrich, Germany)] was applied to each spot. All runs were performed in the presence of the positive and negative controls treated in the same way as the samples. Because of the instability of imipenem, the antibiotic solution must be prepared just before use, or freezed at −20°C for 1 week or at −70°C for a month. The negative control is useful to evaluate spontaneous hydrolysis of the antibiotic.

The configuration of the equipment was the Research Use Only version as the IVD version does not allow access to the modification of parameters in the FlexControl software. Appropriate calibration was conducted before each run using a mixture of bradykinin [1-5] and [1-7] at 35 μM (Sigma-Aldrich). The mixture was spotted and once dried 1 μl of matrix was added on top of each spot. The spectra were acquired after the mixture was dried.

The mass spectra were obtained using a MALDI Biotyper^® Smart (Bruker Daltonik) system, with Flex Control 3.4 software. Three methods were created for spectral acquisition: MBT_ATB.par, MBT_ATB_AutoX.axe and MBT_ATB.mcl. The operational mass range was between 100 and 1,000 m/z in the linear positive mode. The mass peaks were acquired in 40 shot steps to produce 240 satisfactory shots, and the resolution of the mass peaks selected was higher than 300. The movement of the laser in the spot followed a large spiral.

The parameters considered in the calibration are listed in Table 1.

In the first part of the study, i.e., “software design,” the results were analyzed in parallel, with our in-house imipenem hydrolysis procedure and Clover MS data analysis software (Clover BioSoft, Spain) and compared with those obtained with the MBT STAR^®-Carba IVD Kit (Bruker Daltonik) and the STAR^®-BL IVD software.

Instructions for using Clover MS data analysis software (Clover BioSoft, Spain) in the Carbapenem Hydrolysis Detection Analysis module are provided in Supplementary Data Sheet 1 and Supplementary Video 1. The software automatically performs the baseline subtraction and subsequently calculates the ratio of hydrolysis (RH) of imipenem, based on the ratios of the mass peaks of imipenem (300 m/z) and imipenem complexed with the matrix (489 m/z) and the internal standard, i.e., reserpine (607 m/z) (Figure 1). The mass peak at 300 m/z is usually less intense than the 489 m/z as imipenem tends to stabilize forming an adduct with the matriz (Oviaño and Bou, 2018). The results were normalized according to the controls. Results are displayed in a .pdf report (Supplementary Data Sheet 2).

In the STAR^®-BL IVD software, (a) the cut-off values used were those recommended by the manufacturer. In the Clover MS data analysis software (Clover BioSoft), the RH was calculated in two different ways: (b) by using the intensity of the imipenem mass peaks and (c) by using the area under the curve (AUC) of the peaks. A receiver operating characteristic (ROC) curve was calculated for the three methods, yielding AUC values of (a) 0.977 (0.939–1), (b) 0.994 (0.985–1), and (c) 0.990 (0.976–1), with a 95% confidence interval. As the best results were yielded by the analysis that integrated the mass peaks using intensity (b), the spectra were then treated in this way in the rest of the study. The cut-off values for the in-house developed assay were determined using the ROC curve and the Youden index. The cut-off value established for positivity in the imipenem hydrolysis assay was RH values equal to or above 0.5. Values below 0.2 were considered to indicate negative hydrolysis. A gray or intermediate zone was established between 0.2 and 0.5, which required further testing, including prolonged incubation times, or confirmation by other techniques. The negative and positive controls were each analyzed 20 times in the same run to ensure repeatability. A minimum delta RH value between the positive and negative control of three times the value of the standard deviation was established. The value was established in 0.6. Thus, the positive control should have values ≥0.5, the negative control values <0.2 and the minimum delta RH value should be 0.6.

For comparing which technique yielded best results and if these differences were statistically significant we analyzed the ROC curve. As the AUC for the STAR^®-BL IVD software (a) was 0.977 and this value is out of the 95% confidence interval of the AUC of our developed software (b), (0.985–1) we can conclude that the performance of our developed Clover MS data analysis software (Clover BioSoft) is significantly better when using the intensity of the mass peaks.

Continuous variables are presented as mean (95% confidence interval). According to the cut-off values recommended by the manufacturers, 95.06% of the isolates were correctly classified in (a) the STAR^®-BL IVD software, with 92.00% (79.89–97.41) sensitivity and 100% (86.27–99.71) specificity, whereas using (b) the Clover MS data analysis software (Clover BioSoft), 96.30% of the isolates were correctly classified, with 94.00% (82.46–98.44) sensitivity and 100% (86.27–99.71) specificity. The correlation between the two different procedures and data analysis was calculated with the Spearman correlation coefficient, yielding a ρ value of 0.971.

In the second part of the study, the same isolates used in the first part were analyzed by the eight international centers participating in the study (Supplementary Table 2). Two isolates, i.e., OXA-244-producing K. pneumoniae and OXA-232-producing E. coli although being classified as CPE according to literature, both enzymes have a low carbapenemase activity, even lower than OXA-48 enzyme (Potron et al., 2013; Hoyos-Mallecot et al., 2017; Pitout et al., 2019). Therefore, data on these isolates were not included in the statistical, as were considered outliers. However, the results were consistent with a low level of enzymatic activity. Among the nine centers, one center provided an intermediate result for the OXA-244-producing K. pneumoniae, and three centers provided a negative hydrolysis result. In the case of the OXA-232-producing E. coli, one center provided an intermediate result and five centers provided negative results for imipenem hydrolysis.

Once all of the results were obtained, the ROC curve was recalculated with the data obtained in our center (CHUAC) plus the 8 participating centers, to strengthen the statistical power of the analysis. For statistical calculations, 1296 spectra were analyzed from duplicate measurements of the 72 isolates included in the study and measured in the nine centers. The cut-off value for positivity was finally established for RH values ≥0.4; negative hydrolysis values yielded RH < 0.2, and the gray zone, with intermediate values, was established for values of RH between 0.2 and 0.4. Thus, only the positivity cut-off changed slightly. The value of the positive control was adjusted to the new value. The overall sensitivity of the procedure was 95.5% (92.9–97.1), with 100% (98.5–100) specificity. The positive predictive value was 100% (99.0–100) and the negative value, 93.3% (89.8–95.7).

For statistical calculations, full agreement was considered when the MALDI-TOF assay provided an imipenem positive hydrolysis result for CPE or a negative hydrolysis result for non-CPE isolates. Three types of errors were also considered. Errors were considered minor when the result of the MALDI-TOF assay was in the intermediate category and the isolate was either CPE or non-CPE. Errors were considered major when the result of the MALDI-TOF assay was positive for imipenem hydrolysis for a non-CPE isolate and very major errors were considered when a negative hydrolysis result was obtained by MALDI-TOF for a CPE. The overall agreement rate was 94.9% (92.9–96.4); with minor errors representing 3.5% (2.4–5.3) and very major errors 1.5% (0.8–2.8) of the total. No major errors were found. The detailed results for each center are shown in Table 2. The best results were obtained in GM and in PZ, with 98.6% agreement, and the worst results were obtained in CA, with 88.9% agreement. Minor errors and very major errors ranged between, respectively, 1.4 and 8.3% and 1.4 and 5.6%. Regarding exclusively carbapenemase-producing isolates, the overall agreement rate was 95.5% (92.9–97.1, with minor errors representing 2.0% (1.0–3.9) and very major errors 2.5% (1.4–4.6) of the total.

The rates of agreement regarding the species are shown in Table 3. Values range from 91.6% (87.2–94.5) for E. coli to 100.0% (70.1–100.0) for K. oxytoca. The rates of agreement in relation to the resistance mechanisms are shown in Table 4. The rates of agreement ranged from 87.8% for VIM to 100% for K1. Regarding the accuracy of the MALDI-TOF method for detection of carbapenemases, the best classification rates were yielded for KPC-producing bacteria, with 98.9% of agreement relative to molecular methods.

Two isolates yielded most of the errors in the “intercenter evaluation,” both are VIM-producing E. coli. The first one yielded minor and very major errors in 3 and in 1 out of the 9 centers, respectively. The second one yielded very major errors in 5 out of the 9 centers.

The participating centers GM, RC, RS, PZ, and SL analyzed 20 isolates in the 2-month period; CA analyzed 19 isolates, SE analyzed 18 isolates and VM analyzed 10 isolates (Supplementary Table 3). The 147 isolates included were obtained from different clinical samples, urine (54), colonization samples (51), blood (16), wounds (8), respiratory samples (7), abscesses (5), catheter (4), and biopsies (2). Regarding the reference methods for detection of resistance, GM, CA, and SL used the O.K.N.V immunochromatographic test from CORIS BioConcept (Belgium); RC, PZ, RS, and SE used characterization by PCR and VM by WGS. The species most frequently detected were K. pneumoniae (67), followed at some distance by E. cloacae (Brackmann et al., 2020) and E. coli (Pitout et al., 2019). The carbapenemase enzyme most frequently detected was OXA-48-like (69), followed by VIM (31), KPC (21), NDM (12), and IMP (5). The distribution of the carbapenemase enzymes among the different centers is shown in Table 5.

For statistical calculations, a total of 294 spectra were analyzed for duplicate measurements of 147 isolates. The overall agreement rate relative to the reference method for resistance detection was 92.5% (87.1–95.8), with 4.1% minor errors (1.9–8.6) and 3.4% very major errors (1.5–7.7) (Table 6). Thus, the sensitivity of the MALDI-TOF assay for detecting CPE was 93.9% (88.8–96.7). As no major errors were also found in the clinical validation stage the specificity of the methodology is 100% (70.1–100.0). The best results were obtained in VM and in PZ, with 100% of agreement and the worst results were obtained in SE, with 83.3% agreement. Minor and very major errors ranged between 0 and 10.0% and between 0 and 10%, respectively. Regarding the detection of the different carbapenemases enzymes, the agreements rates ranged from 88.4% for an OXA-48-like-producing isolate to 100% for KPC, NDM, and IMP. Minor and very major errors ranged between 0% (for VIM, KPC, NDM, and IMP) and 5.8% (for OXA-48-like).