The main contribution of this research lies in proposing a lean resource management and reliable interaction framework for the middle platform based on distributed data governance in the context of the lean financing management environment for power grid companies. The paper addresses the pressing need for enhanced data management and control in the grid, particularly focusing on functions such as data cleaning and governance.

First, the existing research algorithms for anomalous data detection have encountered some limitations, such as a manual anomaly threshold setting and untimely threshold updating. In this regard, an anomalous data cleaning method is proposed, based on the dynamic adjustment of local outlier factor (LOF) anomaly thresholds, to achieve the optimal selection of the anomaly threshold for eliminating anomalous data and ensuring high standards of data quality and reliability in lean resource management.

Second, there are some shortcomings in various methods to complete missing data, such as the incomplete utilization of context information and data correlation. In this regard, a missing data-filling method based on an adaptive update domain genetic algorithm is proposed to ensure reliable data support for decision-making processes in the low-carbon-oriented new grid.

Finally, a data storage and traceability method was proposed, integrating blockchain with the InterPlanetary File System (IPFS) to ensure the authenticity and reliability of financial data during the interaction process, thereby enhancing the efficiency and efficacy of lean resource management and reliable interaction in the context of the new energy grid.

The remainder of the paper is structured as follows: Section 2 outlines the related work; Section 3 introduces the lean resource management and reliable interaction framework of the middle platform based on distributed data governance; Section 4 presents a distributed data governance approach for lean resource management of the middle platform in the low-carbon-oriented new grid; Section 5 introduces a data storage and traceability method for reliable interaction; Section 6 presents the simulation results; Section 7 presents the discussion and limitations; and Section 8 presents the conclusion.

The LOF algorithm is a classic unsupervised anomaly identification algorithm, mainly utilizing the density of the data to determine the data anomaly. However, the traditional LOF algorithm requires the LOF threshold to be set manually in advance, which is not applicable to massive financial data cleaning (Zheng et al., 2015; Salehi et al., 2016). This paper proposes a data anomaly cleaning method based on the dynamic adjustment of LOF anomaly thresholds, which adjusts and updates the anomaly threshold according to the number of samples of the LOF value. The proposed method realizes the optimal selection of anomaly thresholds, which is described as follows.

The kth reachable distance is calculated. The kth distance between the point farthest from d _i and d _i in all financial data points is defined; the distance S _k (d _i ) is the kth distance of d _i , and S _k (d _i , d _j ) denotes the distance between point d _i and point d _j . Thus, the kth reachable distance from point d _i to point d _i is denoted as S k ( d i , d j ) = max s ( d i , d j ) , S k ( d i ) .

The local reachable density for each financial data point is calculated. The kth distance domain of point d _i is denoted by V _k (d _i ), that is, all points within the kth distance of point d _i . The local reachability density ρ _k (d _i ) of point d _i is the inverse of the average reachability distance from all points within V _k (d _i ) to point d _i , reflecting the density between point d _i and points in the surrounding domain, which is given by the following expression: ρ k d i = V k d i ∑ d j ∈ V k d i S k d i , d j . (1)

The LOF is calculated for all financial data points in the sample. The local anomaly factor ξ _LOF(d _i ) for point d _i is given by the following equation: ξ LOF  d i = ∑ d j ∈ V k d i ρ k d j ρ k d i V k d i . (2) Here, ξ _LOF(d _i ) denotes the average value of the ratio of the local reachability density of points within the kth distance domain V _k (d _i ) of point d _i to the local reachability density of point d _i . The larger ξ _LOF (d _i ) is, the more likely that point d _i is an anomalous data point.

The LOF anomaly threshold is determined. After obtaining all LOF values, the anomaly threshold can be continuously adjusted based on the number of statistics of LOF values to realize the accurate identification of anomaly financial data, and the LOF anomaly threshold is defined as ξ LOF ′ , which is calculated as follows: ξ LOF  ′ = ξ ̄ LOF  + β ⋅ ∑ i = 1 n ξ LOF  d i − ξ ̄ LOF  2 n . (3) Here, ξ ̄ LOF  = 1 n ∑ i = 1 n ξ LOF d i is the mean of all LOF values. n is the sample size of LOF values. β is the anomaly skewness, which measures the extent to which the anomalous data differ from the normal data, and the larger the value of β, the larger the ξ LOF ′ is likely to be. When ξ _LOF(d _i ) is greater than ξ LOF ′ , d _i is an anomalous data point.

Since β is an important parameter affecting the LOF anomaly threshold, a too-large value of β will lead to a large anomaly identification error, while a too-small value of β will lead to slow identification efficiency. Therefore, the optimal β needs to be selected. This paper further adapts the parameter of β, which is given as follows: β = ∑ i = 1 n 1 − 1 e a i ⋅ ξ LOF d i − ξ ̄ LOF ξ LOF , max − ξ LOF ,  min  , (4) where a _i ∈ [0, 1] is an indicator variable for the mean value of the financial data. When a _i = 0, it indicates that ξ LOF ′ = ξ ̄ l o f ; otherwise, a _i = 1. ξ _LOF, max and ξ _LOF, min indicate the maximum and minimum values of the LOF financial data point, respectively. Using the above equation, the optimal β can be adaptively adjusted according to the number of samples of LOF values, and the optimal LOF anomaly threshold can be further obtained, which improves the efficiency and accuracy of financial data point anomaly identification.

The above steps are repeated until all anomalous financial data points are identified, and the anomalous financial data are cleaned to obtain a new financial dataset E = E 1 , E 2 , … , E m .

As the cleaning of anomalous financial data will result in missing financial data points, it is necessary to fill in the missing data to protect the integrity of financial data to support the lean resource management of financial financing. We assume that E = E 1 , E 2 , … , E m satisfies the m dimensional normal distribution, which is denoted as E = E _obs ∪ E _mis . E _obs is the set of financial data with observations, and E _mis is the set of missing financial data. In this paper, based on the adaptive update domain genetic algorithm, we estimate the log-likelihood function of the parameters μ and Ω of the financial dataset E as follows: Φ μ , Ω = − t 2 ln 2 π − t 2 ln | Ω | − 1 2 ∑ 1 = 1 t e 1 − μ T ⋅ Ω 1 − 1 e 1 − μ , (5) where μ = μ 1 , μ 2 , … , μ m is a vector of means for each financial data and Ω σ p q is the covariance matrix of variable E 1 , E 2 , … , E m . The initial values of μ and Ω are generally determined by the financial dataset E _obs , and e _l denotes the vector of variables corresponding to the financial data record l = {1, 2, …, t}, where t is the number of financial data records.

In this paper, Φ( μ , Ω) is used as the fitness function to calculate the fitness of each parameter individual in the population. The larger the Φ( μ , Ω) value is, the closer and more accurate the parameter. The following constraints must be met.  s.t.  μ 1 , min ≤ μ 1 ≤ μ 1 , max , … , μ m , m i n ≤ μ m ≤ μ m , m a x , (6) where μ _m, min and μ _m, max denote the minimum and maximum values of the mth anomalous financial data point, respectively, whose values are determined by E _obs .

In order to improve the speed of selecting the optimal parameters, the parameters determined by Φ( μ , Ω) are further crossed and mutated to realize the selection of the optimal parameters. Assuming that p _c is the crossover probability and there are t parameter individuals in the parameter population, tp _c parameter individuals are selected for crossover operation. Assuming that O = O 1 , O 2 , … , O t denotes the parent of the parameter population, two parameters are randomly chosen in O = O 1 , O 2 , … , O t to form the crossover pair O O r , O s . At the same time, v is randomly chosen in {1, 2, …, m}. Two offspring μ r v ′ , μ s v ′ are generated by performing a c-crossing operation on μ _rv , μ _sv in O O r , O s , which, in turn, yields a new parameter O r v ′ , O s v ′ . The crossover formula is expressed as follows: μ r v ′ = e μ r v + 1 − e μ s v μ s v ′ = 1 − e μ r v + e μ s v , (7) where e is the crossover random number and its value is within [0,1].

Assuming that p _x is the variation probability and there are t parameter individuals in the parameter population, tp _x parameter individuals are selected from the parameter population for crossover operation. O _h is denoted as an individual in the parameter population. μ h 1 , μ h n , ⋯ , μ h m is the set of means of O _h , and φ is randomly selected in {1, 2, …, m} for the mutation operation. Then, the mutated parameter is O h ′ , and the mean is μ h 1 ′ , μ h n ′ , … , μ h m ′ . The mutation formula is expressed as follows: μ h φ ′ = μ h φ + Δ g , max φ − μ h φ , r a n d o m ⋅ > 0 μ h φ − Δ g , μ h φ − min φ , r a n d o m ⋅ < 0 , (8) Δ g , x = x 1 − η 1 − g G β , (9) where random () is a random function that produces a uniform distribution. If a random number is greater than 0, the mean value after mutation will increase, that is, random(⋅) > 0; if a random number is less than 0, the mean value after mutation will decrease, that is, random(⋅) < 0; and if a random number is 0, the mean value after mutation will remain unchanged, that is, random(⋅) = 0. G is the maximum number of generations of variants, g is the current number of generations of variants, β is a parameter that determines the degree of non-consistency, and η is a random number in [0,1].

Considering that the genetic algorithm easily falls into the local optimum and has many iterations, this paper makes the algorithm jump out of the local optimum by the chaotic disturbance of excellent parameters to reduce the number of iterations. Let the fitness function of the current optimal parameter μ* be Φ* and the mean vector of the excellent parameter μ h * = μ h 1 * , μ h 2 * , … , μ h m * , then, the chaotic disturbance to μ h m * can be expressed by the following equation: μ h m * = μ h m − 1 * + 1 − y − 1 y n χ j − 1 . (10) Here, μ* is the value after chaotic perturbation in the traversal interval of the smaller feasible domain. 1 − ( y − 1 ) y n is the adjustment coefficient with respect to the number of iterations y. The value of χ _j−1 is randomly set as 1 or -1.

In order to further improve the optimization accuracy of the genetic algorithm, this paper introduces the search domain adaptive update mechanism. The update domain includes a total of two phases. Φ _o and Φ _o−1 are defined as the optimal adaptation values of the oth and o − 1th generations, respectively. α is a threshold, which takes the value of (0,1). If the difference between Φ _o and Φ _o−1 is less than a threshold α, the search domain update is in the first stage; otherwise, it is in the second stage. The stage discrimination formula of the search domain update is shown as follows: Φ * = Φ o − Φ o − 1 < α . (11)

When the search domain update is in the first stage, the lower bound of the search domain is increased, and the upper bound of the search domain is decreased, thus reducing the overall search domain. The upper and lower bounds of the search domain for the oth generation are calculated as follows: b o l o w = b o − 1 l o w + min b o − 1 u p − b o − 1 l o w / ε , b o u p = b o − 1 u p − min b o − 1 u p − b o − 1 l o w / ε , (12) where b o low and b o u p are the upper and lower bounds of the oth generation, respectively, and ɛ is the scale parameter.

When the difference between Φ _o and Φ _o−1 is larger than a threshold α, the replicated optimal individual enters the second stage, and then, the search domain is updated as follows: c o l o w = μ h φ ′ + min c o − 1 u p − c o − 1 l o w / ε , c o u p = μ h φ ′ − min c o − 1 u p − c o − 1 l o w / ε , (13) where c o l o w and c o u p are the adjusted lower and upper bounds, respectively. Φ^* is used as the criterion to shrink the boundaries one by one. When the distance of the optimal individual from the boundaries is less than the fault-tolerant variable, the boundaries are restored to the initially defined domain. The current optimal individual is preserved, and then, the genetic search is continued until reaching the maximum number of iterations.

In order to reduce the error of the estimated value of missing data, it is necessary to further estimate the missing anomalous financial data. Therefore, this paper uses the Markov chain Monte Carlo (MCMC) method to fill the missing data. This method iteratively estimates the missing data on the condition of incomplete datasets and parameters of incomplete data, and the filling process is as follows.

1) Each of the missing-type anomalous financial data are estimated according to the optimal parameters μ , Ω, and E _obs , and the value of E mis y + 1 is derived from the conditional distribution p(E _mis , E _obs , O _y ). p(E _mis , E _obs , O _y ) is the probability distribution associated with E _mis , E _obs , and O _y . μ and Ω are generally determined by the financial dataset E _obs .

2) The posterior mean vector and covariance matrix of the simulated data, that is, O _y+1, are obtained in p O ∣ E o b s , E m i s y + 1 , based on the filled complete financial dataset, which will be repeated in (1).

3) Filling the missing-type financial data by iterating (1) and (2) over each other produces a Markov chain Y 1 , O 1 , Y 2 , O 2 , … , Y y + 1 , O y + 1 , which exhibits a p E m i s , O ∣ E o b s distribution. When the distribution is stabilized, the filled missing data of E _mis will be obtained, yielding the complete financial dataset U = u 1 , … , u m .

After data governance, the dataset U = u 1 , … , u m is stored using the IPFS. The IPFS enables the data storage and retrieval based on the content of financial data and uses the idle storage resources in the network to establish a distributed data storage system. It divides the data to different network locations, supports fast retrieval and data sharing, and possesses a fault-tolerant nature.

Considering that the IPFS uses the hash value of data as the storage address, this feature is naturally consistent with the tamper-proof feature of blockchain storing data hash values. Therefore, this paper proposes a data storage scheme combining the IPFS and blockchain, which combines the IPFS to store data and blockchain to store data hash so as to realize distributed data storage and ensure data safety reliability, and traceability. When uploading data larger than 256 KB to the IPFS, the system automatically divides the data into 256 KB chunks and stores these chunks on different nodes in the network. Blockchain nodes store hash values of data elements, while nodes except leaf nodes store hash values of child nodes. Therefore, the hash value in the node is calculated as follows: A i , j = H a s h A i − 1,2 j − 1 , A i − 1,2 j , (14) where A ^i,j denotes the hash value of the jth target node in the ith level.

To represent the hash of the data, the IPFS uses a multi-hash format and Base58 encoding. The storage address A _d is represented as follows: A d = B a s e 58 A C o d e A L e n g h A H a s h , (15) where A _Code denotes the hash algorithm encoding; A _Lengh denotes the length of the hash value; and A _Hash denotes the hash value.

For each fragment, a unique hash value is generated. Subsequently, the IPFS concatenates the hash values of all fragments and computes the resulting hash value for the data, which is A H a s h = ∑ H a s h u m . (16)

In order to ensure the authenticity and traceability of financial data flow records (DFRs) of electric power companies, this paper proposes a data traceability method based on Merkle mountain proof, supporting the safety and reliability of financial data in the interaction process. Specifically, a new Merkle mountain block structure is introduced to construct a data storage structure, which includes two parts: block header and block body. Between them, the block header contains the data version number, time stamp, degree of confidentiality, business category, hash value of the previous block, Merkle tree root (MTR), and Merkle mountain range root (MMRR). The block body consists of the Merkle tree and Merkle mountain. As a special Merkle tree, the Merkle mountain has the advantage of dynamic data addition, and it is not necessary to rebuild the data structure. The data traceability method based on Merkle mountain proof includes two parts, Merkle mountain proof and data traceability of Merkle mountain proof based on data private blockchain (DPBC), which are introduced as follows.

Algorithm 1

The proposed Merkle mountain proof-based data traceability method.

1: Input:  h, MTR

This paper uses a sample set consisting of 10³–10⁴ distribution grid financial system data points collected for the purpose of identifying and cleaning anomalous financial data. The financial data points used are sourced from the transaction data and financial books of five departments in the distribution network financial system of a certain power supply company of the State Grid Corporation of China from January to March 2017 (Shouyu et al., 2019). To further validate the efficiency of the proposed algorithm in this paper, a comparative analysis is conducted with two existing algorithms for identifying and cleaning anomalous data: the quartile algorithm and the traditional LOF algorithm. The quartile algorithm demonstrates high cleaning efficiency but is prone to excessive removal, leading to identifying and cleaning some data points within the normal fluctuation range, resulting in a serrated pattern in the clustered regions of the data. The traditional LOF algorithm requires a manually preset threshold, heavily relying on expert experience. When applied to the cleaning of massive financial data characterized by high uncertainty, its efficiency and accuracy are notably compromised.

Figure 3 shows the anomalous data identifying and cleaning results under different numbers of data points. The proposed algorithm exhibits higher accuracy in identifying and cleaning anomalous data than the two comparing algorithms. Specifically, when the number of data points is set at 10⁴, the performance of the proposed algorithm improves by 76.4% compared to the quartile algorithm and 106.5% compared to the traditional LOF algorithm. This improvement stems from the adaptive adjustment of the anomaly threshold based on the sample size of LOF values in the proposed algorithm, enabling a dynamic optimal selection of the anomaly threshold and consequently enhancing the accuracy of anomalous data identifying and cleaning in massive financial datasets. The weaker ability of the quartile algorithm to identify biases in the data leads to the excessive removal of normal data, resulting in a decrease in accuracy in the identification and cleaning of anomalous data. The traditional LOF algorithm, when confronted with large datasets, has a fixed threshold, which limits its ability to identify and eliminate a significant portion of extreme anomalies, particularly in the context of multivariate high-dimensional data, thereby diminishing its accuracy in anomaly identification.

Figure 4 shows the number of false positives for anomalous data under different numbers of data points. The proposed algorithm exhibits a significantly lower count of misjudged anomalous data than the two comparison algorithms. Specifically, when the number of data points is set at 10⁴, the number of false positives for anomalous data in the proposed algorithm is 790, representing reductions of 83.5% and 85.7% compared to the quartile algorithm and traditional LOF algorithm, respectively. This noteworthy improvement stems from the real-time dynamic adjustment of the LOF threshold by the proposed algorithm, leading to a substantial decrease in the misjudgment count, particularly in the context of handling vast datasets. In contrast, the two comparison algorithms lack the capability to adapt to dynamic changes in financial data, resulting in an increase in the misjudgment count as data rapidly expand.

The performance of the algorithm is validated through simulation using a foundational financial dataset from the power grid financial system, aiming to demonstrate its data imputation capabilities in a multivariate dataset. The validation indicator is the data-filling accuracy, which refers to the similarity between the filled data and the original data. The specific attributes of the selected dataset are detailed in Table 1 (Wu et al., 2012).

To validate the performance of the proposed algorithm, the expectation maximization algorithm (EMA) and genetic algorithm (GA) are selected as comparison algorithms. The EMA assumes a distribution for a financial dataset with partially missing data and makes inferences based on the likelihood under this distribution, replacing missing data with expected values. The GA, on the other hand, derives the optimal combination of attribute weights, or the best chromosome, through selection, crossover, and mutation operations. Consequently, it estimates missing values in the dataset based on this optimal chromosome.

Figure 5 shows the variation in data-filling accuracy with the number of algorithm iterations. The proposed algorithm demonstrates superior data-filling accuracy and faster convergence than the two comparison algorithms. At the 120th iteration, the data-filling accuracy of the proposed algorithm surpasses those of the EMA and GA by 41.1% and 8.2%, respectively. This improvement is attributed to the adaptive updating mechanism of the search space introduced by the proposed algorithm, which dynamically identifies whether the improvement rate of the optimal individual meets the requirements, leading to adjustments in the updating space. Consequently, it conducts global optimization for the attributes of each sub-dataset. Although the EMA exhibits faster convergence than the proposed algorithm, its failure to consider the entire parameter space may result in estimating optimal parameters that are specific to local optima in individual sub-datasets, leading to a decrease in the overall data imputation accuracy. The GA lacks the ability to promptly use feedback information from the network, exhibiting a slower search speed, requiring more training epochs to achieve more accurate solutions.

To validate the performance of the proposed data traceability algorithm in this paper, simulation experiments are conducted, with the evaluation metrics being the amount of data downloaded and data traceability verification time. The amount of data downloaded refers to the size of data that nodes need to store locally when performing data traceability verification. The data traceability verification time is the time required to verify a specific transaction, encompassing the duration from submitting the proof of inclusion of a transaction to locating its corresponding hash value. The comparison algorithm chosen for this analysis is the simplified payment verification (SPV) algorithm. The impact of block height at various magnitudes on simulations is discussed, with the experimental setup including block heights of 0.01 × 10⁵, 0.02 × 10⁵, 0.02 × 10⁵, 0.1 × 10⁵, 0.2 × 10⁵, 0.3 × 10⁵, 0.6 × 10⁵, 1 × 10⁵, 1.5 × 10⁵, and 2 × 10⁵.

Figure 6 shows the amount of data downloaded at different blockchain heights. As the blockchain height increases, both the proposed algorithm and SPV algorithm experience an increase in the required data volume. However, at the same block height, the proposed algorithm necessitates a smaller data download than the comparison Algorithm. At a blockchain height of 2 × 10⁵, the amount of data downloaded for the proposed algorithm is 45.4 MB, representing a 16% reduction compared to the SPV algorithm. This discrepancy arises from the fact that during data verification, the SPV algorithm needs to download the block header information for the entire chain, whereas the proposed algorithm only requires the download of the latest block in the longest valid chain, thereby reducing the storage resource consumption for nodes.

Figure 7 shows the results of data traceability verification time at different blockchain heights. As the number of blocks in the blockchain network increases, the verification time for both the proposed algorithm and SPV algorithm gradually escalates. The proposed algorithm exhibits a shorter verification time than the SPV algorithm. This is attributed to the fact that the proposed algorithm only requires obtaining the Merkle mountain range, calculating the verification path to derive the MMRR and comparing it with the hash value in the latest block header at the current height. In contrast, the SPV verification process is more complex as it involves traversing downward from the latest block to trace back to the target block. At a blockchain height of 2 × 10⁵, SPV incurs a maximum time cost of approximately 36 ms, while the maximum time cost of the proposed algorithm is approximately 10 ms. Consequently, the proposed algorithm achieves a reduction of approximately 72% in verification time compared to the SPV algorithm, thereby enhancing the efficiency of the verification process in data traceability.