Knorr and Ng advocated distance-based outlier detection (DOD) for the first time to soften the limitation of distribution-based methods on data distribution and prior information Knox and Ng (1998). The local distance-based outlier factor (LDOF) is one of the most known variants in distance-based approaches Zhang et al. (2009), which measures the outsiderness degree in scattered real-world datasets. The relative location of one point and its neighbors evaluates the deviation of the patterns, based on which the classical top-n strategy chooses outlier candidates. As the volume of data increases and the form of data diversifies, data streams are becoming popular, spawning many studies on in-stream outlier detection. Angiulli et al. presented three algorithms to detect distance-based outliers in a sliding-window model Angiulli and Fassetti (2010). A novel notion called the one-time outlier query identifies outliers in a targeted window at an arbitrary time. Milos Radovanovic et al., focusing on the effects of high-dimensional datasets, analyzed the relationship between antihubs and outliers taking into account the reverse nearest neighbor, that is, the point neighboring its k-NN Radovanović et al. (2015). Continuous outlier mining employs the sliding-window data structure to reduce time and memory costs, which is flexible in terms of input parameters Kontaki et al. (2016). Scaleable, distributed algorithms have been put forward for substantial data. MapReduce works for distributed tasks: A multi-tactic strategy for DOD is proposed, where data characteristics are considered in data partitioning Cao et al. (2017). The in-memory proximity graph copes with the memory problem of large datasets, analyzing the type of proximity graph for the algorithm Amagata et al. (2022).

MST is an important and widely used data structure in clustering analysis. MST-based clustering finds inconsistent edges and deletes them to form reasonable and meaningful clusters. In the case of the existence of outliers, cutting inconsistent edges can result in isolated points or clusters, which can be utilized for outlier detection.

Jiang et al. proposed a two-phase outlier detection method based on k-means and MST, in which small clusters are selected and deemed outliers Jiang et al. (2001). There are two stages to this method. In the first phase, they used modified k-means clustering by assigning the far point as a new cluster center. In the second phase, an MST is constructed, and the longest edges are cut to find the small clusters, the tree with a few nodes. MST-based spatial outlier detection combines MST-based clustering constructed by the Delaunay triangle irregular net (D-TIN) and density-based outlier detection, performing effectively on the data of soil chemical elements Lin et al. (2008). Previous work also modified the k-means algorithm to construct a spanning tree efficiently Wang et al. (2012). Integrating MST-based clustering and density-based outlier detection improves the quality of detection. Meanwhile, the removal of outliers may lead to enhanced results of MST-based clustering Wang et al. (2013).

From the existing work in outlier detection, it can be concluded that.

• Distance-based models are weak in detecting local outliers. Furthermore, the boundary points in a sparse cluster may be misclassified as outliers.

We propose a novel method inspired by MST to tackle the shortcomings mentioned above.

Spanning tree. Given N n-dimensional data points (vertices) in Euclidean space, the spanning tree is a tree that includes all N vertices without closed loops, in which the number of edges is not greater than N ( N − 1 ) 2 because full connectivity is not required.

Minimum spanning tree (MST). An MST is a spanning tree whose total weight is minimal among all spanning trees, which means that the number of edges in an MST is N − 1. The total weight is the sum of the weight of all edges of the tree. Generally speaking, the weight of an edge in a tree is the Euclidean distance between its two endpoints. Mahalanobis distance or other metrics can also be used as a measure.

Prim’s MST. Among the three traditional algorithms for constructing MST, Prim, Kruskal, and Boruvka, this work employs Prim Medak (2018), whose process can be briefly described as.

• Randomly choose one point in the dataset as the root of the tree;

Euclidean distance (d). Given two endpoints x ₁ and x ₂ of the ith edge e _i of an MST in the n-dimensional Euclidean space, the Euclidean distance between x ₁ and x ₂ is d i = d x 1 , x 2 = ∑ j = 1 n x 1 j − x 2 j 2 . (1)

Threshold of termination (T _t ). A global termination threshold sets the stopping condition of the cluster computation to identify the remaining points as outliers, defined as T t = d ̄ + ∑ i = 1 N − 1 d i − d ̄ 2 , (2) where d ̄ is the average weight of all edges {e _i } in the Prim’s MST constructed of the dataset, computed as d ̄ = ∑ i = 1 N − 1 d i N − 1 , (3) where the numerator accumulates all edges in the Prim’s MST.

Threshold-based Euclidean distance (ted). We put forward a weighted Euclidean distance to replace the traditional Euclidean, computed as t e d x 1 , x 2 = d x 1 , x 2 T t . (4)

T _t is calculated based on all edges from the MST of the entire dataset, which enables the scaled distances to handle different density clusters by reducing the discrepancy of the edge weights.

Mini-MST generation. In this work, the mini-MST generation algorithm starts from a point in the densest cluster and computes the MST using ted. When an edge is supposed to be added to the tree, its weight is first compared to the adaptive exit condition defined below. If the former is greater, the other end of the current edge does not belong to the current cluster. Consequently, the computation of the current mini-MST terminates and a new construction starts.

Mini-edge weight set (MEW). A mini-edge weight set records the weight of the edges added to the mini-MST. Once an edge is added to the MST, its weight enters MEW.

The first value added to MEW, denoted as MEW ₁, defaults to d ₁, the length of the first edge added to mini-MST. The default value performs well on all real-world datasets applied in this work, as Section 5.2 manifests. In exceptional cases, like a significantly high value of d ₁, MEW ₁ can be tuned, such as for the synthetic “Two densities” and “Three clusters” datasets in Appendix, where MEW ₁ was set to 1.

Adaptive exit condition of mini-MST generation (aec). To improve efficiency, we repeatedly compute mini-MSTs and delete the points added to the MST, applying an adaptively updated exit condition that judges whether the mini-MST generation should terminate at the targeted edge e _i : a e c e i = M E W ̄ + ∑ i = 1 M E W M E W i − M E W ̄ 2 , (5) where M E W and M E W ̄ are the size and the average weight of the current MEW that e _i is supposed to enter, respectively.

MST-based outliers. MST-based outliers are the points not added to any generated mini-MSTs. Our method does not require a given number of outliers; Instead, aec and T _t differentiate the different density clusters and outliers. The construction of the mini-MSTs finishes when the weight of the next edge is greater than the threshold, so that the points in this current mini-MST can be regarded as a cluster with the same density. Furthermore, a sliding window is applied to the edge weight denoted by the Euclidean distance. If the mean value of such a window is greater than T _t , the remaining points that are not ready to be added to the tree should be deemed outliers.

In response to traditional MST-clustering-based outlier detection’s weak performance on datasets with different densities, this work applies a novel distance measure scaled by the threshold of algorithm termination for better discrimination of normal points and outliers. Such a threshold could be considered a quasi-measure of noise in the dataset. The second algorithm improvement of this work targets efficiency: Mini-MSTs are built iteratively. Finishing a mini-MST generation in a cluster is followed by the deletion of processed points and a new construction procedure on the remaining points. An adaptive exit condition based on a progressively updated MEW qualifies the termination of the mini-MST building. A traditional MST algorithm, like Prim, is first applied to create an exact MST to find the point in the densest cluster. Subsequently, all edges are sorted in non-decreasing order to ensure that the first edge’s two endpoints are in the densest cluster because the higher the cluster’s density, the shorter the distances between its points. The edges between different density clusters are taken into account.

Figure 1 illustrates a simplified case that embodies four clusters with different densities and six outliers. C ₁ is the densest cluster with the smallest average weight of edges. A Prim’s MST is constructed first to find the point in the densest cluster, as Figure 1A demonstrates. The shortest edge can be identified by sorting the edges in Prim’s MST in non-decreasing order. Let s denote the start point of the shortest edge in C ₁, from which a mini-MST is computed. Like Prim, the shortest edge is repeatedly added to the mini-MST until the next edge’s weight is larger than the exit condition aec (see Eq. (5)). The points in the built mini-MST are labeled normal and removed from the dataset. The above steps are repeated from the point of the next densest cluster, which in this example is C ₂, and the whole procedure ends with the adaptive exit condition being satisfied. The remaining points are considered outliers.

As can be observed from the example in Section 4.1, the proposed method is centered on the iterative computation of mini-MSTs. Similarly to Prim, two arrays, labeled_data and unlabeled_data, are used to record data points added or not added to the tree, initialized by an empty set and all points, respectively. Unlike the traditional exact MST, the MST in our algorithm is constructed according to the data density, and an exit condition is added to obtain a mini-MST for efficiency. The edge weight of the mini-MST is a threshold-based Euclidean distance in place of the conventional Euclidean distance (see Eq. 4). s denotes the start point of the mini-MST. Aside from the MST array used in Prim’s MST, an additional ted_arr records the threshold-based distance between all data points. Algorithm 1 details the mini-MST construction.

Algorithm 1

Mini-minimum spanning tree construction

Require: a set of N data points, R; start point, s; labeled_data; unlabeled_data

Least number. To be noted, the number of points in the cluster falls within a certain range. A cluster containing too few points is considered an outlier cluster. least_number distinguishes normal clusters from outlier clusters: l e a s t _ n u m b e r = R O U N D N n , (6) where the ROUND function finds the closest integer to the parameter; N and n are the size and dimension of the dataset, respectively. If the number of edges of a mini-MST is less than least_number, the points belonging to the tree are marked as outliers.

Since the algorithm starts building MSTs from the densest cluster, it keeps outliers until all mini-MSTs have been generated. Therefore, the threshold of termination T _t (see Eq. 2) can be applied to stop finding normal points. To compare with T _t , a window filled with the weights of the current edge and the following five edges is used: If the mean value of this window is greater than T _t (see Eq. 2), the remaining data points will be treated as outliers.

Algorithm 2 provides the pseudocode of the proposed adaptive mini-MST-based outlier detection, which takes the input of the dataset R with N data points and its corresponding Prim’s MST denoted by the edges. Each edge of the MST consists of a starting point, an endpoint, and an edge weight.

Algorithm 2

Adaptive mini-minimum spanning tree-based outlier detection

Require: dataset R

Ten experiments were conducted on different real-world datasets, as summarized in Table 1, to demonstrate MMOD’s applicability on the benchmark Campos et al. (2016). The datasets will be introduced in detail in Section 5.4, along with the results of the experiments carried out on them.

As a supplement, experiments on five synthetic two-dimensional datasets with different morphologies are added to demonstrate MMOD’s parameter tuning and its availability on manually generated data; plus, the two-dimensional visualization is intuitive and well-readable (see Appendix).

MMOD’s experimental results were compared with eight algorithms from the Python outlier detection package Zhao et al. (2019), including four classical algorithms, k-NN Ramaswamy et al. (2000), LOF Jahanbegloo and Jahanbegloo (2000), angle-based outlier detection (ABOD) Pham and Pagh (2012), and histogram-based outlier score (HBOS) Goldstein and Dengel (2012), as well as four recent algorithms, one class support vector machine (OCSVM) Erfani et al. (2016), lightweight online detector of anomalies (LODA) Pevný (2016), locally selective combination of parallel outlier ensembles (LSCP), and multiple-objective generative adversarial active learning (MOGAAL) Liu et al. (2019) Zhao et al. (2018). LOF and k-NN are classical density-based and distance-based methods, respectively. ABOD is developed for high-dimensional feature space datasets to alleviate the “curse of dimensionality,” an efficient version of which was used in our experiments. HBOS is an unsupervised outlier detection method that computes the outsiderness degree by building histograms. OCSVM is an extension of the support vector algorithm that learns a kernel function called the decision boundary, distinguishing outliers from inliers. LODA is operative for data streams and real-time applications. LSCP, also unsupervised, chooses the competent detectors by using the local region of the data points. The newly presented MOGAAL is based on a generative adversarial active learning neural network.

To generate a fair comparison reference, the k _threshold value for each state-of-the-art method being compared was set to 7, a typical value setting. Literature such as Campos et al. (2016) records the performance of other k _threshold values on most reference methods. The outlier percentage is calculated as the number of outliers divided by the size of the dataset. All experiments were run through Python 3.6.5 on a computer with an Intel^® Core™ 3.2 GHz i5-3470 CPU and 4 GB RAM.

Conventional evaluation metrics precision, recall, and F-measure were applied to analyze and compare the experimental results on real-world datasets. Let m denote the number of correct outliers returned by the detector, n denote the total number of all outliers returned by the detector, and o denote the number of ground-truth outliers. The precision P is the proportion of correct outliers in all outliers identified by the detector: P = m n . (7)

The recall R is the proportion of correct outliers that the detector returns in all ground-truth outliers: R = m o . (8)

The F-measure is the harmonic mean of precision and recall: F = 2 1 P + 1 R = 2 P R P + R . (9)

Applying real-world datasets can demonstrate the effectiveness of the proposed method straightforwardly. Since medical data are one of the most prominent application scenarios of outlier detection, nine widely investigated open-source medical datasets are utilized for experiments. A spam dataset is additionally brought into the experiment as a case for other domain applications. On each real-world dataset, default MMOD parameter settings or formulas defined in Section 3.2 were adopted, such as the MEW’s first added value MEW ₁, the threshold-based Euclidean distance ted, and the exit condition aec, which evidences the broad applicability of MMOD without parameter tuning. The precision, recall, and F-measure values of MMOD’s and the experimental results of the peer methods’ are entirely recorded in Tables 1–3, of which the statistics are plotted in Figures 2–4 for visual comparison.

MMOD has perfect or nearly perfect recalls on most datasets, which should be attributed to its ability to greatly retain possible outliers, enabled by the adaptive exit condition. Such an adaptive termination mechanism also improves the efficiency of the algorithm. LSCP’s recall performance is comparable to MMOD on WDBS, but on the one hand, its recall is extremely worse than MMOD on the other datasets; on the other hand, its precision on WDBS is also significantly inferior to MMOD. Regarding precision, HBOS works well on four datasets, but is overall unstable and particularly poor on the other six. MMOD is optimal in three datasets and at an average level globally.

Two of the advanced aspects of MMOD are that it does not require the number of outliers as input and that it is outlier quantity and proportion insensitive. Such a characteristic was well reflected in the experimental results. The applied datasets include various outlier percentages, such as a small portion of outliers, a large percentage of outliers, and a close proportion of outliers and normal samples. Evidently, most of the peer methods are affected by such setups. For datasets with a high percentage of outliers, such as HeartDisease, Parkinson, Pima, and SpamBase, HBOS has high precision values; however, for cases with a low percentage of outliers, HBOS’s precision almost hits rock bottom. Worse, its recalls are always poor, no matter the outlier percentage. k-NN, LODA, and LSCP are almost the opposite. k-NN and LODA’s precision and recall on datasets with a low percentage of outliers are significantly better than the case with a high percentage of outliers. LSCP’s recall is excellent when the percentage of outliers is low; for the high percentage of outliers, LSCP is almost incapable, not to mention its unsatisfying precision all the time. As a comparison, MMOD’s performance is more consistent regardless of the outlier percentage, without dramatically poor metric values. Its recall is especially consistently splendid, its precision is in the middle of the pack, and its F-measure is relatively robust, all verifying that MMOD, which does not take outlier numbers or percentages as inputs, works insensitively to outlier quantity and proportion.

Which one of recall and precision is more valued during outlier detection is relevant to the application scenario. For medical data, especially disease diagnosis, recall is related to whether cases with real diseases will be missed. The preliminary validation experiments of MMOD’s method suggest its usability on medical data.