Several domain-specific details are fundamental to effective algorithm development:

Effective ML solutions need to design in accommodations to non-ML (e.g., clinical) constraints. Therefore, literature review to inform ML work should extend well beyond ML methods and focus on the clinical use-case itself, without an ML-centric filter. Documentation of use cases and standards of care are published by various agencies, including the World Health Organization (WHO), ministries of health, and non-government organizations (e.g., the Bill and Melinda Gates Foundation, the Global Fund, and the Worldwide Antimalarial Resistance Network).

Below we list some references that are especially relevant to ML researchers designing algorithms for automated malaria diagnosis using Giemsa-stained blood films.

Domain experts are a vital source of guidance and collaboration. They include field experts, i.e. those who work in field clinics or who do field-based research; and subject matter experts, such as WHO personnel and long-time researchers in the space (these groups overlap). The value of their experience and insight to effective algorithm development cannot be overstated.

As an example, our group’s entire ML program for malaria diagnosis has depended absolutely upon expert input from a technical advisory panel, as well as on continued contacts and advice from field clinics. To the degree that our work has succeeded, this expert input has been the key ingredient (along with the closely entwined matter of data collection and curation). We would argue that ML development can only progress toward clinically useful algorithms when domain expertise is somehow integrated into the team (recent examples include Yang et al., 2020; Manescu et al., 2020a, b; Kassim et al., 2021; Poostchi et al., 2018a; Yu et al., 2023; and, for schistosomiasis, Armstrong et al., 2022; Oyibo et al., 2022, 2023).

Connecting with such experts is made easier by two things. First, people (on average) love to talk about their work. Second, field experts are often (again, on average) open to engaging with ML solutions and happy to co-author serious research.

Sources for contacts include: (i) published work, e.g., who is leading and authoring/co-authoring relevant studies; (ii) academic institutions with concentrations of research in the space; (iii) online interest groups, e.g., on LinkedIn; and (iv) non-ML conferences, their attendees, and proceedings, e.g., the American Society of Tropical Medicine and Hygiene.

ML practitioners choose metrics to evaluate model performance by (i) what is customary, familiar, and convenient; (ii) what has been done by previous authors; (iii) what can generate the “state of the art” (SOTA) comparisons required for publication in the ML community; and (iv) what is acceptable to ML reviewers. This creates a closed loop which perpetuates the use of certain metrics without regard to their effectiveness. When entrenched metrics do not assess algorithm performance in a clinically relevant way, it blocks progress toward deployable solutions.

Several commonly-used ML metrics, including object-level ROC curves, AUC, object precision, and F1 score, appear frequently in the ML malaria literature. However, in the malaria context these are flawed measures of performance, for reasons given below. They also do not meet standards of evidence per (WHO, 2021a). They should therefore be avoided when reporting results (though they can be useful intermediate measures for internal algorithm work).

The importance of assessing algorithm performance at the patient level cannot be over-emphasized. The basic unit of clinical care is the patient ² , so the most relevant metrics are defined at the patient level, not the object level. Performance assessed across pooled objects can be a useful intermediate step during ML development, but it is fundamentally unrealistic, because (i) it does not match the clinical task; (ii) it ignores interpatient variability; and (iii) it is dominated by high parasitemia samples. For example, consider four malaria-positive patients, with

Patient 1: 50,000 parasites/µL (p/µL).

Patients 2, 3, 4: each 300 p/µL.

Suppose the algorithm detects almost all parasites in {1}, and misses all parasites in {2,3,4} (a realistic scenario due to interslide variability). Then the object-level sensitivity is 98%, while patient-level sensitivity is 25%.

We have found that two metrics, each defined on a per-patient basis, are particularly useful: false positive rate (FPR) and sensitivity. Each is calculated separately for each patient, using algorithm accuracy on objects within that patient’s sample. These are covered in 3.3 and 3.4, and underpin other metrics related to specificity (3.5), LoD (3.6), and quantitation (3.9).

Interpatient variability (as in Figure 2 ) poses great difficulty for ML, so it must be factored into algorithm evaluation. It is captured by the standard deviations of FPR and sensitivity (cf. 3.3, 3.4), to the degree that the dataset captures interpatient diversity.

A related issue is interclinic variability. For example, clinics can use different stain variants (e.g., Giemsa, Field, and JSB) which yield different color ranges. Even clinics with nominally identical protocols can differ substantially (see e.g., Das et al., 2022 and a detailed example in Torres et al., 2018). Besides variations in presentation, different clinics may produce populations of samples with differently distributed FPRs and sensitivities. Implications of this for tuning algorithms are covered in 3.5.

False Positive Rate (FPR) is the number of distractors mislabeled as parasites per clinically relevant unit of substrate, hereafter cV, e.g., 1 µL of blood (malaria), 10 mL urine (Schistosoma haematobium), 1 gram stool (other NTDs), a specified number of cells in a histological sample, etc; but not “per image tile”, which generally has no clinical relevance (though image tiles can often be translated into the microscopy “Fields of View” used in protocols). Malaria ML papers with some FPR analysis include Linder et al. (2014), Mehanian et al. (2017), Delahunt et al. (2019), and Manescu et al. (2020a). Crucially, FPR is calculated separately for each patient. We denote the vector of FPRs for the population of patients as F.

FPR is not object-level specificity, which is a commonly reported but highly flawed measure in this context (see 3.1).

While FPR can be calculated for any sample, FPRs on positive samples may be erroneously boosted by mis- or unannotated parasites. Thus, the population’s FPR distribution is best characterized using negative samples only.

Interpatient variability makes the standard deviation of FPR, σ( F ), a crucial performance measure. The mean FPR µ( F ) is less relevant because it can be subtracted out, as shown in 3.6 and 3.9. However, since it tends to scale roughly with σ( F ), it can give a hint as to the relative magnitude of σ( F ) (see e.g., Mehanian et al., 2017; Delahunt et al., 2019).

In datasets with insufficient numbers of patients, an FPR calculated over pooled objects has some value as a lower bound on F . In particular, it can be compared to the clinical LoD requirement. For example, a pooled-object FPR of 5,000/µL, vs. a required 100 p/µL LoD (malaria), is a clear sign that work is still needed. Multiple splits of a set of pooled objects does not simulate σ( F ), because each split will include the full patient diversity.

Aside: Samples with high FPRs are sometimes criticized as being due to “poor sample preparation”. However, except for extreme cases this is in the eye of the beholder: human clinicians readily and successfully diagnose “dirty” samples on which ML algorithms fail. Thus, the need to improve sample prep is to large degree a need to accommodate ML methods’ struggles with handling highly variable sample presentations. See Das et al. (2022), and a detailed example in Torres et al. (2018).

Sensitivity (aka recall) is the fraction of positive items in a set that are correctly labeled: Sensitivity = t p t p + f n , where tp = true positives, i.e. positive items labeled correctly, and fn = false negatives, i.e. positive items labeled as negative or missed. The “items” can be parasites (object-level) or malaria-positive patients (patient-level).

Specificity is the fraction of negative items (distractor objects or patients) that are correctly diagnosed as negative:

Specificity = t n t n + f p , where tn = true negatives (negative items in V labeled correctly), and fp = false positives (negative items labeled incorrectly).

Here, LoD roughly means the parasitemia at which the algorithm can consistently (e.g., 95% of cases) distinguish positive and negative cases. Based on the WHO evaluation criteria (WHO, 2016c), the required LoD for malaria microscopy is roughly 100 p/µL, i.e. 1 parasite per 50,000 red blood cells (RBCs) or 80 white blood cells (WBCs). However, expert microscopists routinely achieve LoDs ≈50 p/µL (e.g., Vilela, pers. comm.; Bell, pers. comm.), and the lower LoD is of course clinically desirable. For helminths, LoD is implicitly 1 egg (per 10 mL urine or 1 gram stool) (WHO, 2002, 2021b). Standard Loa loa diagnosis by blood microscopy has an effective LoD > 200 mf/mL when V = 10µL (Mischlinger et al., 2021).

LoD can be directly probed using holdout sets of low parasitemia positive samples. These are not as useful for training anyway, as they supply few parasite objects. However, this is impractical because it’s hard to acquire enough field-prepared positive blood films near the LoD (a work-around is to ablate parasites in the image set of a sample with parasitemia above the LoD, to lower its visible parasitemia).

We can calculate a useful estimate of LoD from F and S as follows:

Denote the putative LoD as L parasites per cV, and suppose that a patient is diagnosed as “positive” when N ≥ T, where N is the number of positively-labeled objects per cV. Note that N = TP + FP in positive patients, and N = FP in negative patients, where TP and FP denote counts per cV, so T P = t p c V V where tp is the number of parasites correctly labeled in V (similarly F P = f p c V V ).

• Make T high enough to ensure to enforce 95% specificity on negative samples as described in (Mehanian et al., 2017) by setting α to 1.65 std devs in Equation 1:

• Then for positive samples the worst case is a very “clean” sample with low FPR, such as the 5^th percentile of samples with FP = µ( F ) − 1.65σ( F ). In this case we must depend mostly on detected parasites to ensure N ≥ T for a positive diagnosis. Suppose for ease that the sample has average sensitivity = µ( S ). Then a sample at LoD has TP = Lµ( S ).

• To diagnose this positive sample correctly (but just barely, i.e. N = T), we need

So the estimated LoD (L per cV) has

• Optionally, +1 can be added to the numerator (i.e. require N = T +1) to prevent unpredictable behavior should both σ( F ) and µ( S ) approach 0:

In our algorithm development, we have found this estimate to be a good (slightly optimistic) proxy for actual LoD when assessing algorithms during development. In particular, it consistently tracked diagnostic accuracy on holdout sets of low parasitemia samples, i.e. lower estimated LoDs mapped to higher accuracy (sensitivity and specificity at the patient level) in holdout sets and field trials. It has the practical advantage that low parasitemia samples are unnecessary, because the vector S can be well characterized by high parasitemia samples. It also allows useful comparison of algorithms, as it directly addresses a key clinical requirement and is anchored to the relevant unit cV.

A more nuanced (and pessimistic) proxy could account for σ(S) by having a denominator = µ( S )−β σ( S ) for some β.

Given a trained algorithm that uses the two hyperparameters C and T, {C, T} can be optimized in a principled way to maximize patient-level sensitivity, subject to the constraint of a fixed target specificity K:

The methods for setting T (Equation 2) and for estimating LoD (Equations 3, 4) both assume that the FPR vector F is gaussian. In our experience this is often not the case. Rather, the FPR distribution may be asymmetrical, with mostly low-FPR samples and a few high-FPR samples. This can be handled by modifying the methods in 3.5 and 3.6 as follows:

Two other methods of calculating T from F may be useful:

For both these methods, the detected objects are assumed to be already classified. If a threshold C on object scores was used, then first T must be calculated for each C, before choosing the best {C,T} pair as in 3.7.

The manual method of choosing {C,T} takes time, but it can yield the best results in a field deployment because it is most closely tailored to the empirical FPR distribution.

Quantitation sometimes has clinical importance. For example, accurate quantitation is needed to monitor for drug-resistant malaria strains by calculating clearance curves (White, 2011; Ashley et al., 2014; WHO, 2016d). For helminths, quantitation targets are typically rough only (e.g., low, medium, high) (WHO, 2002). For Loa loa, a remarkable drug reaction necessitates accurate quantitation at certain high parasitemias only (≈ 20k to 30k worms/mL) (Gardon et al., 1997; D’Ambrosio et al., 2015).

Poisson statistics for rare events give variation in the actual number of parasites in a particular sample with volume V, given a fixed true parasitemia P over the whole sample. The variation is most visible at low parasitemias, e.g., at 100 p/µL, where each RBC has a 1/50,000 chance of containing a parasite in thin film, or each WBC has a 1/80 chance of corresponding to a nearby parasite in thick film.

This variability has two main impacts:

In both cases, automated systems hold a strong advantage because they can scan higher volumes than human technicians, who often by necessity work in a high Poisson error regime (S.I. of Delahunt et al., 2019). Manual microscopy protocols average multiple readers’ estimates (when available) to reduce quantitation error (WHO, 2016b; WWARN, 2023).

When reporting results on datasets of small size, authors should understand how Poisson variability limits their estimates of algorithm performance.

Identification of malaria species is one of the three tasks assessed by the WHO 56 evaluation system (WHO, 2016c). Correct species ID matters clinically because (i) falciparum infections are much more likely to be fatal; and (ii) treatment plans differ by species (CDC, 2023b), since (for example) the hypnozoites of vivax and ovale species require special care.

Because not all species ID errors are equal from a clinical perspective, reported results should preferably include a confusion matrix as in (Delahunt et al., 2019).

Aside: In our experience, it is relatively straightforward to distinguish falciparum vs. non-falciparum on thick film alone (Torres et al., 2018; Vongpromek et al., 2019; Das et al., 2022), also (Kassim et al., 2021), and even mixed species infections that include falciparum can often be identified on thick film by comparing the ring stage and late stage parasite counts (Horning et al., 2021). However, thin films are still typically needed to distinguish between the various non-falciparum species, unless the clinical use case allows geographical priors to be leveraged. A method to distinguish non-falciparum species on thick film would yield clinical benefit by eliminating the need for thin films, due to (i) the ease of thick-only workflows (Carter, pers. comm.; Proux, pers. comm.), and (ii) thin film problems with quality (Long, pers. comm.) and difficulty of species ID at low parasitemias (Lilley, pers. comm.).

Staging parasites (as ring, later trophozoite, schizont, or gametocyte) is not part of the WHO evaluation, and is not generally useful clinically, except as used during species identification or when quantitating asexual forms in non-falciparum species. In falciparum (the main target of quantitation, for drug resistance studies) the difference between ring and gametocyte is glaring.

The methods described here were developed to address the exigencies of field-prepared blood films. They apply equally well to analysis of particular field isolates of any Plasmodium species, since the core issues (inter-sample variability, importance of FP objects, etc) apply to field isolates. The caveats connected to analysis of in vitro cultures (3.4.4) apply to field isolates as well.