Relation between the measured power flows and state of the grid is (Schweppe and Wildes, 1970): z = h x + υ x = [ φ i , V i ] T z = [ V i , P i , Q i , P i j , Q i j ] T (1) where, z denotes the system measurement, h denotes the measurement equation, x denotes the state of the grid, υ denote the measurement noise, φ _i and V _i denote the voltage phase angle and magnitude of node i, P _i and Q _i denote the power of node i, P _ij and Q _ij denote the tributary power flow, from i to j, whose detailed equations are: P i = ∑ j ∈ N i V i V j G i j ⁡ cos φ i − φ j + B i j ⁡ sin φ i − φ j Q i = ∑ j ∈ N i V i V j G i j ⁡ sin φ i − φ j − B i j ⁡ cos φ i − φ j P i j = V i 2 g s i + g i j − V i V j c o s φ i − φ j − V i V j b i j ⁡ sin φ i − φ j Q i j = − V i 2 b s i + b i j − V i V j s i n φ i − φ j − V i V j b i j ⁡ cos φ i − φ j (2) where, G _ij , B _ij are the real and imaginary part of the i, j term in the node conduction matrix, g _ij , b _ij denote the conductance and susceptance between i, j, g _si + jb _si denotes the conductance of the parallel branch of i, g _sj + jb _sj denotes the conductance of the parallel branch of j.

Due to the pervasive noise, measurements can be unreliable and inconsistent with the actual state. Static state estimation filters the noise based on the current-measured data (Schweppe and Wildes, 1970). Due to this factor, when attacked by FDIAs, results of static state estimation will deviate significantly from the true state. That deviation is utilized in attack detection mechanism of this paper. WLS method is adopted in this paper, whose iterative form is: Δ z k = z − h x ̂ k Δ x ̂ k = H T R − 1 H − 1 H T R − 1 Δ z k x ̂ k + 1 = x ̂ k + Δ x ̂ k (3) where, k denotes the step of iteration, x ̂ denotes the estimated state, H x denotes the Jacobian matrix of the measurement equation, with the dimension of m × n. The iteration ends when Δ x ̂ is sufficiently small.

Dynamic state estimation bases on KF to estimate the state and eliminate noise. While static method focuses primarily on real-time states, dynamic method tries to predict the state of the next step and the estimation result is closer to true state under attacks. Due to the non-linearity of power system, EKF and UKF are applied in our works.

EKF is effective for non-linear models, and performs better in systems with weak non-linearities and perturbations (Li et al., 2015).

A second order expansion of h x ̂ around x ̃ is: h x ̂ = h x ̃ + H x ̃ Δ x + S (4) where, Δ x = x ̂ − x ̃ , x ̃ is the prior estimation of state, x ̂ is the posterior estimation of state, H denotes the Jacobian matrix of the measurement function, and S is the remainder term of the second and higher order. Omitting S , a linearized model of grid state is obtained. x k + 1 = F k x k + Q k z k + 1 = H x k + 1 + R k (5) where, F _k denotes the state-transition function, Q _k and R _k denote the system and measurement noise.

Equations of EKF are shown in Table 1, and the explanation is:

1. Prior estimation: Calculate x ̃ k + 1 and covariance matrix of prior estimation M _k+1 by the post estimation results of step k.

UKF applies KF to non-linear systems utilizing the Unscented Transformation (UT). UKF performs better under systems with strong non-linearity compared with EKF (Julier and Uhlmann, 2004). The non-linear form of grid state is: x k + 1 = f x k + ω k z k = h x k + υ k (6) where, f x k is n × 1 dimensional non-linear state-transition function, ω _k and υ _k are n × 1 and m × 1-dimensional Gaussian white noise with the zero mean.

Equations of UKF are shown in Table 2, and the explanation is:

1. Generate sigma-points: Generate 2n+1 sigma-points (Julier and Uhlmann, 2004).

Errors in the initial measurement data can be the source of distortion of estimated states, leading to wrong decisions of EMS. Therefore, detection of bad data in measurements is applied to detect possible errors. The most common method is constructing an empirical threshold and detect by the residual function (Merrill and Schweppe, 1971): τ < z − h x ̂ 2 (7) where, x ̂ denotes the state estimation results, z − h x ̂ 2 denotes the l ²-norm of residuals and τ denotes the empirical threshold generated form historical data.

Holding of Eq. 7 denotes that residuals of the estimated states exceed the threshold. Then a bad data alarm will be triggered, indicating the existence of bad data.

The DQN algorithm, used with replay buffer and target network, is a representative DRL algorithm. Applying DQN algorithm can overcome the complexity of storing Q-table in Q-learning. Other improvements of DQN over Q-learning are (Mnih et al., 2015):

1. Construct replay buffer: At each step, store the experiences in buffer D . When updating the neural network, a mini batch is extracted to update weights θ. Format of experience e _t is:

In this section, we construct FDIAs under the assumption of complete topology information and unlimited cost. Thus the attacker can extract whole measurement function h ( x ) and construct attack without considering the cost, resulting in the inefficacy of empirical bad data detection mechanism.

Equations for constructing attacks are (Liu et al., 2009): z a = z + a x ̂ a = H − 1 z a = x ̂ + c (10) where, z _a denotes the attacked measurement values that the system obtains, z denotes the real measurement values of the grid, a denotes the attack vector, x ̂ a denotes the estimated states under attacks, x ̂ denotes the estimated states without attacks, c denotes the change of state values.

As for a non-linear power system, FIDA can also satisfy the measurement equation z a = h x a by: a = z a − z = h x + c − h x = h x a − h x (11) z a − h x ̂ a 2 = z − h x ̂ + a + h x ̂ − h x ̂ a 2 = z − h x ̂ 2 (12)

If the static state estimation is operated as always, there is x ̂ a ≈ x a and h x ̂ a ≈ h x a . Comparing Eq. 12 with Eq. 7, ignoring the inherent Gaussian noise in Eq. 6, the residuals under valid FDIAs are the same as the residuals without an attack, namely z a − h x ̂ a 2 = z − h x ̂ 2 .

In other words, a FIDA constructed this way doesn’t change the residuals in Eq. 7. Therefore, a valid FIDA doesn’t trigger the bad data detection alarm mentioned in Section 3.4.

Considering the diversity of attacks and attackers’ intentions, types of FDIAs are also diverse. In this paper, we classified FDIAs by duration and variation of intensity.

According to the duration, attacks can be divided into transient attack and continuous attack (Jiang et al., 2020). The transient attack tends to have stronger perturbation in a short period, while the continuous attack can remain undetected for a longer period by applying weak perturbation.

According to the intensity, the attack can be divided into constant-intensity and variable-intensity attack. The constant-intensity attack vectors are similar in magnitude, while the variable-intensity attack vectors can be stochastic or asymptotic.

Detailed classification of FDIA is shown in Table 3. Three types of attacks are selected and studied in this paper, the strategies are shown in Algorithm 1 and the detailed equations are:

Algorithm 1

Strategies of three attacks.

1. Attack1. Continuous-constant-intensity attack:

Define the start and end of the attack as t _start and t _end . While t _start < t < t _end , construct and inject the attack by Eq. 13. x a = x + c ⋅ ω z a = h x a (13) where, c = [c _φ1, c _φ2, ⋯c _φn , c _V1, c _V2, ⋯c _Vn ] denotes the intended deviation of phase angle c _φi and magnitude c _Vi on the node voltage, ω is a standardized normal variable.

Attack-1 is a typical form of FDIA. When c _φi = c _Vi = 0, no attack will be injected on bus i, and c depends on the intention of attackers. Since we focus on detection, the programming problem of determining c is replaced by a Gaussian variable ω . Multiplying by ω , the attack vector varies in a reasonable range (0, c ). Thus, diversified attack intention can be included, and value of c can be fixed. For example, if we define c _V1 = 0.1p.u., all attacks with the intensity between 0 and 0.1p.u. on bus one are considered as long as there are enough episodes. Moreover, ω can also make FDIAs hard to be detected by empirical method.

2. Attack2. Transient-constant-intensity attack:

While t _start < t < t _end , at each step, with a probability of ɛ _attack to construct and inject the attack by Eq. 13. In other cases, no attack is conducted. Attack-2 aims to test the response speed of the detection method.

3. Attack3. Continuous-variable-intensity (incremental) attack:

While t _start < t < t _end , construct the attack by: x a = x a + c t end − t start z a = h x a (14)

Attack3 is valid during steady-state grid operation when state x undergoes little change. One obvious feature of Attack3 is that x _a is cumulative. Since the cumulation of deviation is slow, Attack3 is hard to be detected at an early stage.

According to Section 4.1, it is hard to detect well-constructed FDIAs by bad data detection. However, when attacked by FDIAs, results of different state estimation methods produce a significant difference: Result of static state estimation will deviate from the true state, since it only depends on the real-time measurements. Result of dynamic method is closer to the true state due to the prediction steps.

So in our works, we combine results of static method (WLS) with dynamic method (EKF and UKF) and detect attack based on their inconsistency: τ 1 = x K F − x WLS ∥ 2 > τ attack (15) where, x _KF and x _WLS denote the result of KF and WLS, τ _attack is the threshold for determining an attack. When Eq. 15 holds, the system is determined to be attacked.

However, grid states change abruptly sometimes due to other factors, which can also leads to the deviation of state estimation. Thus Eq. 15 can be false-positive, so we combine it with bad data detection: τ 2 = z − h x WLS 2 > τ baddata (16)

The mechanism is summarized as Figure 1, when Eq. 16 holds, the system is determined to have bad data, When Eq. 16 does’t hold but Eq. 15 holds, the system is determined to be attacked by FDIAs.

However, since the gird is vulnerable to disturbances, evaluating the performance of the method only by accuracy is incomplete. Considering the time sensitivity, an effective detection of FDIA in this paper is defined in Eq. 17. Utilizing Eq. 17, performance of detection is evaluated by detection rate. t start ⩽ t alarm ⩽ t start + 2 (17) where, t _start denotes the start of attack, and t _alarm denotes the time that the attack is detected.

If Eq. 17 holds, it shows that the attack is detected within a short period of time. Thus the detection is effective and the safety of grid can be protected.

Due to the changing load of grid and randomness of attacks, the threshold of Eq. 15 varies greatly in different scenarios. Therefore, it is impractical and costly to apply a certain empirical threshold τ _attack in a wide range of grids.

To address the shortcomings of empirical detection, FIDA detection is formulated as a MDP and trained utilizing DQN algorithm. Detection is achieved through neural network, equivalent to a dynamic threshold instead of the empirical threshold τ _attack in Eq. 15. The neural network is trained by interacting with the environment during the MDP of FDIA detection (An et al., 2019; Kurt et al., 2019). After an action of detection, agent receives a feedback (reward) from the environment for guiding the actions by updating the neural network (Sutton and Barto, 1998).

Extensive simulations performed to simulate the actual scenarios of FIDA attack-detection process. First, to ensure the practicability, simulations are based on IEEE 9, 14, 30 and 57-bus networks by MATPOWER (Zimmerman et al., 2011). Second, due to the diversity of attacks and attack intentions (An and Liu, 2019), three types of FDIAs are adopted, namely Attack-1, 2, and 3. Cases with single attack and multiple attacks are both considered during simulations. Then, the attacks aim at the magnitude of node voltages, with the intensity to cause voltage violation (Zhu and Liu, 2016; Zheng et al., 2020). Third, WLS is adopted in static estimation while EKF and UKF are adopted for dynamic state estimation in different cases. In addition, random seeds were used to reduce the random error. Details of settings are shown in Table 4.

Considering the change in voltage magnitude of static state estimation result, effects of three typical attacks is in Figure 3. The dashed lines indicate the start and end of attacks.

Figure 3A shows effect of Attack-1, namely continuous-constant-intensity attack. Attack-1 injects attack continuously and magnitude of the attack vector follows the same Gaussian distribution. Thus, deviation in voltage magnitude is obvious under Attack-1.

Figure 3B shows effect of Attack-2, namely transient-constant-intensity attack. Attack-2 injects the attack vector intermittently and magnitude of the vector follows the same Gaussian distribution. Deviation generated by Attack-2 is also considerable, but the attack duration is compressed. Thus, for the detector, higher response speed is required.

Figure 3C shows effect of Attack-3, namely continuous-variable-intensity (incremental) attack. Attack-3 injects the attack vector continuously and magnitude of the vector is cumulative. The deviation between two steps generated by Attack-3 is smaller than other attacks, which can avoid being detected. However, the deviation accumulates over a period of time and the amplitude at the end of attack is also considerable, so the consequence of Attack-3 can be severe.

Taking Attack-1 as an example, Figure 4 shows the differences between valid and invalid attacks. In Figure 4A, the residual modulus of the no-attack case is about 0, and almost overlaps with the valid-attack case. But in Figure 4B, the residual modulus fluctuates greatly at a high level under an invalid attack, exposing the attack to detector. In summary, the difference of effectiveness between valid and invalid attacks to bypass the bad data detection is obvious.

Changes in node voltage of IEEE 9-bus system when attacked by the FDIA are shown in Figure 5. Since the intended deviation of angle c _φi = 0°, the phase angle deviates slightly during the attack in Figure 5A. Moreover, the intended deviation of magnitude c _Vi = 0.1p.u., so the intensity of attack is within (0, c _j ). Correspondingly, the bus voltage magnitude undergoes a large deviation in Figure 5B, misleading the following grid operation.

Simulations in this section compare the optimized DQN-based method with the original DQN-based method and empirical threshold method. Moreover, cases with different dimension of state space are also compared Effectiveness of adopting sampled replay buffers and extending state space is proved.

First, to verify that effectiveness of extending state space, we changed the dimension of state space in different cases. Results are shown in Figure 6. The detection of original method is unstable during the training and reaches convergence after 3,000 episodes, the attack detection rate fluctuates at a low level (43%), and the detection rate is unstable with fluctuation.

With the extended state space, the detection rate increases by at least 21%–64%, and the fluctuation decreases significantly. Comparing the above cases, the detection rate reaches near 100% after convergence in cases that N ≥ 4, so in the rest of this paper N = 4.

Second, to prove efficacy of the optimized method, we simulated the detections with optimized DQN-based method, original DQN-based method and empirical method.

The empirical threshold method uses a fixed threshold consturcted from experiences. In this paper the algorithm is: By τ = x K F − x WLS 2 , calculate τ ₁ in no-attack cases and τ ₂ in attacked cases, τ _attack = τ ₂−τ ₁. So the detection rate is a fixed number and behaves as a horizontal line since the empirical threshold is hardly updated online in practice.

Results are shown in Figure 7. Each case is simulated in five parallel groups utilizing random seeds in Table 4. Detection rates are averaged and the shadows denote the standard deviations between different groups.

Comparing the performances in Figure 7, empirical method doesn’t perform well in detection. Detection rate of empirical method is 61% and 80% against Attack-1 and Attack-2, and is only 30% against Attack-3. What’s more, the method with original DQN performs well at certain episodes in Figures 7C, F, but the detection rate fluctuates substantially throughout the training in Figures 7A, D, E. The convergence of training with original DQN is difficult, too.

As for the optimized DQN-based method, cases with EKF converges around 8,000 episodes, and the converged detection rate is 98.42% against Attack-1, 99.70% against Attack-2, and 100% against Attack-3, with some fluctuation. Cases with UKF converges around 5,000 episodes, and the converged detection rate is 96.95% against Attack-1, 98.99% against Attack-2, and 100% against Attack-3 with little fluctuation. After convergence, fluctuation of detection rate has been restricted within 4%. In addition, the training process of EKF-based method is less stable than the UKF-based method.

In conclusion, detection rate is improved by at least 15.95% utilizing the optimized DQN-based method. Stability of the training is also improved fundamentally over the original DQN-based method, especially the UKF-based method.

In this section, we compared three types of FDIAs in power systems in different networks to prove that the proposed method is effective for multiple scenarios. In addition, each case was repeated at least three times and results are averaged. Results are shown in Figure 8 and Table 5.

In Figure 8, the training process converges after about 8,000 episodes against Attack-1 and 6,000 episodes against Attack-2. Meanwhile, the speed of convergence is faster in cases based on UKF, especially in the cases against Attack-3. At the final stages of training in above cases, the detection rates fluctuate by 2%. In addition, the convergence speed is slightly faster of a more complex network, since the accumulation of state deviation is faster.

First, in Table 5, the detection performances are similar in different networks, since the detection mechanism only depends on the state estimation performance and is not affected by the network complexity. Second, detection rates in different cases are consistently close to 100% after convergence. Third, UKF-based method performs better in detection than EKF-based method. In summary, the method performs well under different attacks in multiple scenarios.

To prove utility of the detection method, a hybrid attack model is constructed. In each episode, the type and start of attack is random and unknown. One of Attack-1, 2, 3 is randomly selected and conducted during the attack based on IEEE 14-bus network.

Rusult of training is shown in Figure 9. Detection rates are also averaged by results of five groups, and standard deviations are given by the shadows.

After training, the detection rate of EKF-based method reaches 99.01%, and the UKF-based method reaches 99.71%. In Figure 9, since the attack is hybrid, the detection rate fluctuates at the early stage of training. Trainings converge more slowly compared to the cases against single attack. EKF-based method converges at about 5,500 episodes and UKF-based method converges at about 8,500 episodes.