A set C is convex if the line segment between any two points in C lies in C, i.e., if ∀x ₁, x ₂ ∈ C and ∀θ ∈ [0, 1], we have θ x 1 + 1 − θ x 2 ∈ C . (1)

A function f : R n → R is convex if the domain of function f (dom f) is a convex set, and if for all x, y ∈ dom f, and ∀θ ∈ [0, 1], we have f θ x + 1 − θ y ≤ θ f x + 1 − θ f y . (2)

As a valuable property of convex functions, strong convexity can significantly speed-up the convergence of first order methods. We say that f : X → R is α-strongly convex if it satisfies the improved subgradient inequality Eq. 3: f x − f y ≤ ▽ f x T x − y − α 2 ∥ x − y ∥ 2 . (3)

A large value of α would lead to a faster convergence rate, since a point far from the optimum will have a large gradient, and thus gradient descent will produce large steps in this case.

As a significant subfield of CA, convex optimization studies the problem of minimizing convex functions over convex sets for mathematical optimization. A convex optimization problem in standard form is written as [40]: m i n i m i z e f 0 x s . t . f i x ≤ 0 , i = 1 , . . . , m h i x = 0 , i = 1 , . . . , p , (4) where the optimization variable is x ∈ R n , the objective function f 0 : R n → R is convex, inequality constraint functions f i : R n → R ( i = 1 , . . . , m ) are convex, and equality constraint functions h i : R n → R ( i = 1 , . . . , p ) are affine [41].

Convex optimization problem shows many beneficial properties. For example, every local minima is a global minima, and if the objective function is strictly convex, then the problem has at most one optimal point. Therefore, if a task can be formulated as a convex optimization problem, then it can be solved efficiently and reliably with effective and rapid optimization and solution, using interior-point methods or other special methods for convex optimization. General convex optimization focuses on problem formulation and modeling, more specifically, it is applied to find bounds on the optimal value, as well as approximate solutions. These solution methods are dependable enough to be embedded in computer-aided design or analysis tools, or even real-time automatic or reactive control systems.

The core design of the Lagrangian duality (or just duality) is to consider the constraints in the convex optimization problem Eq. 4 by constructing an objective function with a weighted sum of the constraint functions [42]. Then the Lagrangian L : R n × R m × R p → R for the problem Eq. 4 is L x , λ , v = f 0 x + ∑ i = 1 m λ i f i x + ∑ i = 1 p v i h i x , (5) with dom L = D × R m × R p , where λ _i is the Lagrange multiplier associated with the ith inequality constraint f _i (x) ≤ 0, and v _i is the Lagrange multiplier associated with the ith equality constraint h _i (x) = 0. In addition, the vectors λ and v are referred to the dual variables or Lagrange multiplier vectors of the problem Eq. 4 [39]. Therefore, the Lagrange dual function (or just dual function) g : R m × R p → R is defined as the minimum value of the Lagrangian over x: for λ ∈ R m , v ∈ R p , g λ , v = inf x ∈ D L x , λ , v = inf x ∈ D f 0 x + ∑ i = 1 m λ i f i x + ∑ i = 1 p v i h i x . (6)

The associated dual problem of convex optimization problems could often produce an interesting interpretation regarding the original problem and lead to an efficient or distributed method for solving it. Therefore, it also reflects theoretical or conceptual advantages of convex optimization and CA [43].

In direct density estimation, abnormal data are defined as samples with a density less than the preset threshold. A probability density function for a continuous random variable is a non-negative Lebesgue-integrable function [47], and satisfies F X x = ∫ − ∞ x f X u d u , (7) where F _X (x) is the cumulative distribution function of X.

Since the multivariate Gaussian distribution model (see Figure 4A) [48] is not capable of describing the situation where the data in the same set conform to multiple different distributions, the mixture of Gaussian (MoG) (see Figure 4B for instance, which is a linear combination of Gaussian distributions) was used to model the general data distribution [49]. Each Gaussian distribution in the MoG is defined as a component, and then the probability density of the target variable x, p(x), is defined in the MoG as: p x = 1 2 π d ∑ k = 1 γ α k 1 det ∑ k exp − 1 2 x − μ k T ∑ k − 1 x − μ k , (8) where d represents the sample dimension, α _k is a mixed coefficient, μ _k and ∑ _k are the mean and covariance matrix of the kth component, and γ is the number of mixed components. The Expectation-Maximization (EM) algorithm [50], a typical algorithm using CA, is adopted to optimize the parameter α _k , μ _k and ∑ _k . The EM algorithm searches for the maximum likelihood estimation of parameters in a probability model that depends on unobservable hidden variables. For samples x ₁, x ₂, ..., x _n , the hidden variables of each sample are assumed to be z ^(j), j ∈ [1, m]. Then, the algorithm finds the lower bound of the likelihood function through Jensen’s inequality [46]. ln ⁡ L θ = ∑ i = 1 n ln ∑ j = 1 m Q i z j p x i , z j ; θ Q i z j ≥ ∑ i = 1 n ∑ j = 1 m Q i z j ln p x i , z j ; θ Q i z j (9)

In Eq. 9, L(θ) is the likelihood function, and Q _i (z ^(j)) denotes the probability of x _i belonging to class z ^(j). After parameter optimization through CA, samples with a density from the MoG model less than the preset threshold are outliers.

Inspired by the MoG model with the EM algorithm, Woodward and Sain [51] used the EM algorithm to identify outliers from a mixture of normal distributions when there is missing data. They confirmed through simulations and examples that using the EM algorithm on the entire dataset resulted in higher detection probabilities than using only the complete data vectors, which is the subset of the entire dataset that includes only data vectors for which all of the variables were observed. The MoG model with the EM algorithm, can detect nuclear explosions from a large number of background signals (such as earthquakes and mining explosions) using seismic signals (or any other discriminant) [52]. Carrying out outlier detection to recognize heart disease, biological virus invasion, and electrical power grid faults has also been explored [53–55].

By the Jensen’s inequality of the logarithmic function in the expectation format, the lower bound of the likelihood function L(θ) of parameters in direct density estimation was discovered rapidly, precisely, and effectively. However, the principal drawback of this method is that the number of mixed components is data-dependent due to the requirement such as weights ∑ ^γ α _k = 1, so it is a tough choice, and the mixture of multiple Gaussian models requires more samples to overcome the curse of dimensionality [56].

Although direct density estimation method is adaptable to multiple different distributions estimation and efficient parametric optimization, it can not correctly reflect the pattern’s characteristics for most high-dimensional conditions, as multivariate functions are intrinsically difficult to estimate [57]. To solve this problem, indirect density estimation methods have been developed. The main reason for the name “indirect density estimation” is that it does not require density estimation. The goal of this method is to estimate the density ratio w(x), called importance, of the independent and identically distributed (i.i.d.) training samples { x i t r } i = 1 n t r and i.i.d. test samples { x j t e } j = 1 n t e : w x = p t e x / p t r x , (10) where p _te (x) and p _tr (x) are the probability density function [58] for the training data and test data, respectively. w(x) is non-negative because p _te (x) ≥ 0 and p _tr (x) > 0 for all x belonging to the data domain D ⊂ ( R d ) . w(x) for regular samples is close to one, while those for outliers tend to deviate substantially from one (i.e., close to 0) because the training data only contains regular samples, and p _te (x) would be close to 0 where outliers exist.

With the key constraint of avoiding estimating densities p _te (x) and p _tr (x), adhoc studies have estimated the w(x) to detect the outlier by convex techniques, in which kernel mean matching (KMM) [59], logistic regression (LogReg) [60], the Kullback-Leibler importance estimation procedure (KLIEP) [61,62], least squares importance fitting (LSIF) [63], and unconstrained least squares importance fitting (uLSIF) [64] are popular. For the above-listed methods, the convex expressions for the estimation and their appraisal are summarized in Table 1; we describe the latest two methods in detail.

Since proposed, the uLSIF method has received widespread usage in outlier detection. For instance, based on the experimental results of 12 datasets available from Rätsch’s benchmark repository [70], the SMART disk-failure dataset, and the in-house financial dataset, Hido et al. concluded that the uLSIF-based method is a reliable and computationally efficient alternative to existing outlier detection methods [71]. Umer et al. also illustrated its superiority in the detection of malicious poisoning attacks in 2019 [6]. In addition, change-point detection in time series data such as smart home time series data [72,73], outlying image detection in hand-written digit image and face image data [68], outlier detection in both synthetic and benchmark datasets [74], and computer game cheats detection in game-traffic data [75], all proved its excellence.

Since estimating density is complex (especially in high-dimensional space), a convex heuristic enables indirect density estimation methods against the curse of dimensionality without going through density estimation. The outliers tend to have smaller importance values (close to zero) and then they emerge by a suitable threshold. Optimization methods, such as Newton’s method, the conjugate gradient method, the Broyden–Fletcher–Goldfarb–Shanno algorithm for a convex nonlinear problem, the gradient descent method, KKT conditions method [44], and the inexact augmented lagrange multiplier (IALM) method for unconstrained and constrained quadratic programs, are not only efficient, but could find the global minima. The uLSIF-based method is highly scalable to large datasets, which is of critical importance in practical applications.

The indirect density estimation method, however, is of well-documented vulnerability to a poisoning attack, even with a modest number of attack points inserted into the training data [6]. When an intelligent adversary (the one with full or partial access to the training data) injects well-crafted malicious samples into the training data, an incorrect estimation of the w(x) can occur.

A data matrix S ∈ R n × m with n samples and m variables can be factorized by RPCA as: S = L + E , (15) where L ∈ R n × m is a low-rank component, E ∈ R n × m is a sparse matrix containing outliers and process faults, and E i j = 0 denotes that the jth variable in the ith sample is noise-free. The essence of the RPCA algorithm is to address the convex optimization programming demonstrated in Eq. 16: m i n ∥ L ∥ ∗ + λ ∥ E ∥ 1 s . t . S = L + E (16) In Eq. 16, ∥L ∥_* is the nuclear norm of the matrix L, obtained by the sum of the singular value of L. ∥E ∥₁ is the norm of matrix E, i.e., the sum of absolute values of all elements in E. Also, the parameter λ provides the trade-off between the norm factor ∥L ∥_* and ∥E ∥₁, which can be calculated according to the standard Eq. 17 λ = 1 m a x n , m , (17) and then adjusted slightly according to prior knowledge of the solution.

The optimization problem in Eq. 16 is convex and linearly constrained, and several efficient algorithms are available, including the alternating direction method of multipliers (ADMM) [81], IALM [82], and singular value thresholding (SVT) [83]. Key steps of the RPCA problem solved by IALM are demonstrated in Algorithm 1.

Algorithm 1

RPCA problem using IALM

After Isom and LaBarre [84] first applied RPCA in the monitoring of fuel cell power plants’ process fault detection, Zhang et al. [85] proposed an LRaSMD-based Mahalanobis distance (LSMAD) method for hyperspectral outlier detection. This algorithm dates to the SPCP model proposed by Zhou et al. [86], in which a noise item N (i.e., i.i.d. noise on each entry of the matrix) programming was min L , E ∥ L ∥ ∗ + λ ∥ E ∥ 1 s . t . ∥ X − L − E ∥ F ≤ δ , (18) where ∥⋅∥ _F denotes its Frobenius norm and ∥N ∥ _F ≤ δ for some δ > 0, thus L* and E* can be estimated more stably.

Xu et al. [87] suggested leveraging LRR [88] for anomaly detection in hyperspectral images (HSIs). The LRR model introduced a dictionary matrix D in the linear decomposition of the background matrix, and the convex optimization problem in Eq. 19 is solved for matrix factorization: min Z , E ∥ Z ∥ ∗ + λ ∥ E ∥ 2,1 s . t . X = D Z + E . (19)

In Eq. 19, ∥⋅∥_2,1 is defined to be the sum of ℓ ₂ norms of column vectors of a matrix, and ∥E ∥_2,1 represents the ℓ _2,1-norm to characterize the error term E. LRR could handle the data collected from multiple subspaces well.

In 2020, Su et al. [10] proposed an LRCRD method, and this model is primarily suitable for hyperspace. They employed another ℓ ₂ norm to collaborate the global background and anomaly feature as local representation process attribute on the foundation of LRR; thus, a functional outlier detection model with strong representation ability was built: min ∥ Z ∥ ∗ + β ∑ i = 1 N ∥ Z i ∥ 2 2 + λ ∥ E ∥ 2,1 s . t . X = D Z + E , (20) where ∥Z ∥_* is still the nuclear norm of Z, convexly approximating the rank of Z, N is the number of pixels, β > 0 and λ > 0 are both regularization coefficients.

When it comes to strongly corrupted data, such that the columns of all entries are corrupted, the RPCA via outlier pursuit (RPCA-OP) method, an efficient convex optimization-based algorithm, should be employed for outlier detection [89]. Experiments have confirmed that the RPCA-OP method can even endure column-sparse or row-sparse errors. It recovers the correct column space of the uncorrupted matrix rather than the exact matrix itself like RPCA. Its convex optimization program is shown in Eq. 21: min ∥ R ∥ ∗ + λ ∥ C ∥ 2,1 s . t . S = R + C , (21) where C is still a sparse matrix with some columns’ elements all be zero, ∥C ∥_2,1 promotes column-wise sparsity. To ensure success, we could tune parameter λ to 3 7 γ n with λ being the fraction of corrupted points. Outliers exist in the set of nonzero columns of C ̂ (i.e., I ̂ = { j : c ̂ i j ≠ 0 f o r s o m e i } ).

This method aims to discover a data description with a presupposed shape from the training dataset. A good description covers all target data but includes no superfluous space. Points outside the description in the test set will be detected as outliers.

Among the support vector domain methods, the support vector machine (SVM) [97] is a mainstream two-class classification method for fields such as text detection, human body recognition, and freight transportation [98]. The SVM separates two types of samples with a maximal margin by a hyperplane. For outlier detection, since there is often only the target sample in the training set due to the lack of negative examples, the original SVM is no longer applicable, and support vector data description (SVDD) was developed by Tax and Duin [43] for one-class classification. It looks for a spherical description as implicit mapping, as shown in Figure 5A. This description encloses most training samples x _i and minimizes the volume (i.e., minimizes R) of the hypersphere (R, a), where R is the radius and a is its center.

SVDD adopts the soft-margin criterion [99], and a slack variable ξ is introduced to penalize training samples outside the sphere (i.e., the red point in Figure 5A, with square distance to the center of the sphere is greater than R ²). It operates as Step 1 in the following SVDD algorithm to find the hypersphere with the penalty of ξ _i , where C is the regularization factor (i.e., the trade-off between the volume and the errors) for tighter description and higher accuracy. The detailed algorithm flow (SVDD by the Lagrange multiplier method) is presented as follows:

Step 1) min R ² + C∑ _i ξ _i s.t. ∥x _i − a ∥² ≤ R ² + ξ _i , ξ _i ≥ 0, ∀i;

Applying the Lagrange multiplier method [44], the dual problem can be obtained by the KKT conditions, and the problem that both minimum volume and maximum samples are expected to be fulfilled can be transformed into the above convex quadratic programming problem in Step 1 of the SVDD algorithm. Besides, the duality a = ∑ ⁿ α _i x _i could generate the sparse center of the sphere, which improves its test performance.

Figure 5A is a visual representation of SVDD, and the points on the surface with 0 < α _i < C are support vectors (SVs). The red circles are like the black ones (i.e., normal samples in the training set). However, the red circles are outside the hypersphere, so they are penalized.

To further enhance the flexibility of SVDD when negative examples are available, the following SVDD with negative samples (NSVDD) was also proposed by Tax and Duin [43]. NSVDD assumes that the target samples are in the hypersphere as much as possible (i.e., the black circle in Figure 5B), but the outliers are outside (i.e., the green circle in Figure 5B). Then, the normal points (i.e., the red circle) and the outliers (i.e., the blue dashed circle) should be penalized because they are not in the correct position. Eq. 23 describes how NSVDD works: m i n R 2 + C ∑ i ξ i s . t . y i R 2 − ∥ x i − a ∥ 2 ≥ 0 − ξ i , ξ i ≥ 0 , ∀ i , (23) in which y _i ∈ { − 1, 1} is the label of the training sample with “-1” denoting an outlier. NSVDD is identical to the normal SVDD when new variables α i ′ = y i α i are defined, and both are convex representations.

By employing two slack variables, NSVDD has shown higher classification accuracy with a varying radius of the hypersphere [100]. However, the outlier placed on the boundary of the description (i.e., the blue solid circle crossed by a curve in Figure 5B can not be distinguished from the SVs in the target class (i.e., black solid circle crossed by a curve) based on Step 7 in the SVDD algorithm. By applying kernel techniques, both SVDD and NSVDD can obtain a rigid hypersphere for nonlinear problems with greater flexibility and malleability.

SVDD is an unsupervised machine learning method for anomaly detection, while NSVDD is supervised. The related semi-supervised method [101] was developed in 2020 for rolling element bearings default detection by combining SVDD and cyclic spectral coherence (CSCoh) as domain indicators [102].

Although affected by noise and limited to hypersphere data, standard SVDD can be rated as a cornerstone in the field of anomaly detection. With its improvement, it has been explored for anomaly detection with high-dimensional and large-scale data [103], adversarial examples [104], contaminated data [105], and other anomaly detection situations. Furthermore, in 2020, Yuan et al. [106] demonstrated that this method can undertake robust process monitoring in over 20 real-life datasets, including vehicle evaluation, breast cancer, and process engineering.

By transforming the mini-volume and most-points problem into convex quadratic programming, convexity makes KKT conditions necessary and sufficient. The optimality of the convex program is adequate for solving the data description of the support vector domain method, which results in accuracy and efficiency for global outlier detection. This method ensures the accuracy of normal samples by minimizing the volume of the description and the error of outlier detection. Compared with other outlier detection methods, this method shows comparable or improved performance for sparse and complex datasets. However, for minuscule target error rates, the SVDD could break down, and this method is not preferred for high-dimensional samples.

The support vector domain method is a fundamental and special case of the convex hull method, in which the hypersphere is a convex hull (CH), and solutions are most provided by convex programming. The CH for a set of points S ∈ R n in a real vector space V is the minimal convex set containing S [107]. The CH classifier, belonging to the one-class classifier, builds the CH border according to the training set (comprising the points of the normal class), and samples outside the border in the test set are outliers. An example is illustrated in Figure 6.

The convex hull CH(S) can be calculated according to Eq. 24: C H S = ∑ i = 1 | S | β i x i | ∀ i : β i ≥ 0 ∩ ∑ i = 1 | S | β i = 1 , x i ∈ S . (24)

Since the existence of outliers in the training set may lead to an overfitting decision model, the CH can be corrected by a parameter λ ∈ [0, + ∞), according to Eq. 25 [108]: v λ : λ v + 1 − λ c | v ∈ C H S . (25)

In Eq. 25, v incorporates the vertices of the original convex hull in Eq. 24 regarding their center c = 1 D ∑ i x i , ∀ i = 1 , . . . , | D | ; thus, v _λ contains the modified vertices of CH(S). From this equation, it can be concluded that the CH would be expanded or contracted when λ is greater than 1 or lower than 1, respectively.

However, this approach shows two major drawbacks. First, the computation cost is high. And second, the training data’s boundary may not be well-modeled by a convex polytope. Calculating the CH of a high-dimension dataset requires a tremendous computational cost. If a dataset comprises N samples in R n , the cost of computing the CH is estimated as O(N ^(n/2)+1) [109]. This problem can be solved with the approximate polytope ensemble (APE) technique [110], which first constructs p random 2D projections of the original dataset and then model the CH for each 2D projection. Then, outliers are identified by those points which are outside of at least one of these projections. The main idea of this approach is demonstrated in Figure 7, where a dataset in R 3 is projected in two 2D planes, and the red dot out of the CH of projection #2 represents an outlier. Despite the good performance of the APE approach, an inaccurate classification would happen in non-convex sets. Hence, non-convex APE (NAPE), an extension of managing non-convex boundaries, is proposed. The underlying idea of this extension is to divide the non-convex boundary into a set of convex problems. Then, each convex problem can be solved using the APE algorithm.

Outlier detection performance was investigated using this method on over 200 datasets [111–113], even in multi-modal distributions of automated visual surveillance detection [114]. All exhibited a trade-off between the detection rate (true positive rate) and false alarm rate (false positive rate), and AUC greater than 0.9. In practice, CHs are usually adopted in industrial intelligent fault diagnosis, multiaxial high-cycle fatigue recognition, and other anomaly detection applications, some of which are described in He et al. [115]’s and Scalet [116]’s studies.

As a flexible geometric model, a CH is typically a substantial approximation of the target region. It can approximate a polytope without overfitting, even in a high-dimension situation. The low computational and memory storage requirements allow the APE method to be used under limited resources. By the vertex of the CH of the training set, outliers can be relatively easily recognized. However, the boundaries of the training data may not be well modeled by APE in more general non-convex scenarios. Furthermore, due to its ability to manage strong non-convex distributions, NAPE, a more general extension than the APE algorithm, outperforms the rest of the outlier detection methods including APE in many cases [109,110]. Nevertheless, further efforts are needed to reduce the computational costs of building NAPE.

Unlike the CH method, which is largely an offline algorithm, the online convex programming (OCP) method can be explored in online anomaly detection methods for the data stream. OCP, such as the online gradient descent (OGD) algorithm [117], as defined by Zinkevich [118], features a sequence of convex programmings with feasible sets that are identical, but the cost functions are diverse. According to what has been learned, the algorithm should always choose a point for the lowest cumulative cost before observing the cost function. Whenever the anomaly score (i.e., probability, density, or other custom metrics) efficiently and simply calculated by the OCP for the current state falls below the dynamic threshold, we declare an anomaly.

OCP can be broadly viewed as a game between two opponents: the Forecaster and the Environment [74,119]. The Forecaster constantly predicts changes in a dynamic Environment, where the influence of the Environment is depicted by a sequence of convex cost functions with arbitrary variations over a given feasible set, and the Forecaster attempts to pick the next feasible point in such a way to reduce the cumulative cost as much as possible.

An OCP problem with horizon T can be outlined by a convex feasible set U ⊆ R d and a family of convex functions F = f : U → R . The algorithm of OCP is described in the following section.

Algorithm 2

Online convex programming

The Forecaster will minimize the difference between the actual cost incurred after T rounds of the game and the smallest cumulative cost that could be achieved in hindsight using a single feasible point. Given a strategy μ ^T and a cost function tuple f ^T , the regret w.r.t. u ^T is defined as Eq. 26 R T μ T ; f T , u T ≜ ∑ t = 1 T f t u ̂ t − ∑ t = 1 T f t u t = ∑ t = 1 T f t μ t u ̂ t − 1 , f t − 1 − ∑ t = 1 T f t u t , (26) where u ^T = (u ₁, ..., u _T ) ∈ U ^T , a time-varying tuple, is a comparison strategy distinguishing from the Forecaster’s observation-driven strategy μ ^T and it does not depend on the previous points or cost functions but only on the time index t.

Then the goal would be to select a suitably restricted subset C T ⊂ U T and employ the Forecaster’s tactic μ ^T to ensure that the worst-case regret sup f T ∈ F T sup u T ∈ C T R T μ T ; f T , u T ≡ sup f T ∈ F T ∑ t = 1 T f t u ̂ t − inf u T ∈ C T ∑ t = 1 T f t u t (27) is sublinear in T. Whenever the anomaly score (i.e., probability, density, or other custom metrics) efficiently and simply calculated by the OCP for the current state falls below the dynamic threshold, we declare an anomaly.

Online anomaly detection based on OCP fulfills the needs of some fields, such as industrial production and network routing, where decisions should be made before the comprehension of true costs.

Inspired by recent developments in OCP, Raginsky et al. [120] designed and analyzed a so-called FHTAGN method consisting of assigning a belief (probability) and flagging potential anomalies according to the belief, exploring online anomaly detection methods with dynamic thresholding built on limited feedback. Nevertheless, classic statistical change point detection studies, such as this work [120], surveyed the transient outlier instead of the persistent change. Therefore, persistent change was considered for anomaly detection based on OCP. Further improvements have been made to achieve lower computational complexity [121] or higher anomaly detection accuracy [122].

Convex optimization provides a more versatile approach to tackling complex situations, especially sequential change point detection. Its efficiency and simplicity make it possible to perform computations in real-time. By the convex cost function of the Environment, schemes such as mirror descent for the OCP method are possible. It allows us to remarkably predict the extrinsic anomalous behavior for the next observation concerning the best model based on what we have seen in the past. However, this work has not been extended to any arbitrary anomaly detection method.

In machine learning, especially deep learning, a neural network (NN) is also an essential algorithm that CA contributes to anomaly detection, with its core gradient descent method being the most significant technique in CA [123]. For anomaly detection, a NN extracts the characteristics of abnormal behavior by adaptive learning and learns the normal behavior pattern from the training set. Then, samples with anomalously-related labels in the test set will be anomalies [124].

The loss function to be minimized in a NN is: L w = 1 ∣ X ∣ ∑ x ∈ X l x , w , (28) where w is the weight of the network, X is the training set with labels and l(x, w) denotes the loss calculated by the sample x ∈ X and its label.

The gradient descent method is a first-order optimization algorithm usually applied to find the minima of a function. An iterative search is performed to the point with the specified step size from the current point along the opposite direction of the gradient (or approximate gradient), which is the direction of steepest descent. As the most common gradient descent method in NN, minibatch stochastic gradient descent [45] is usually called simply stochastic gradient descent (SGD) in recent literature even though it operates on mini-batches. It performs the following parameter update: w t + 1 = w t − η 1 N ∑ x ∈ B ▽ l x , w t , (29) where B is the minibatch sampled from X and N = | B | is the minibatch size, η denotes the learning rate, t represents the iteration index, and ▽l(x, w _t ) represents the gradient of loss l(x, w). Therefore, the parameter update is a back-propagation process along the gradient, as demonstrated in Eq. 30: ▽ l x , w t = ∂ l x , w t ∂ w = 1 N ∑ n = 1 N ∂ l n x , w t ∂ w . (30)

Many neural networks have been applied to specific fields of anomaly detection and have been investigated with appealing results. For example, Zenati et al. [125] leveraged bidirectional generative adversarial networks (BiGAN) for image and network intrusion detection, Gao et al. [126] applied CNN for time series anomaly detection in 367 public benchmark datasets from Yahoo, and Xu et al. [127] proposed a cluster-based deep adaptation network (CDAN) model that is adaptable for the spinning power consumption anomaly detection problem in the real-environment yarn spinning workshop. These studies have achieved a desirable performance and high speed.

As an unconstrained optimization in convex optimization theory, the gradient descent method achieves a rapid decline in the loss function by the convex path, contributing considerably to the behavior learning of normal samples and anomalies. NN is a non-parametric method that typically employs gradient descent. With the best architecture and an efficient training procedure, anomaly detection by a NN exhibits higher AUC and F1 scores than other state-of-the-art methods, such as LRR [88] and isolation forests (IF) [128]. Nevertheless, a NN generally desires adequate training data for convergence. Another critical drawback of this method may be that it can not provide the analyst with clear interpretability of why the system believes an entity is potentially anomalous.