Considering the effect of temperature on the charging efficiency during the charging process of the battery (Powell and Meisel, 2016), a random variable is introduced into the charging/discharging process of the battery. The state of charge of the battery has Markov characteristics, in which C ( S t , x t ) is the revenue of the battery at stage t as follows: C t ( S t , x t ) = { d b , t x t − ( a d e p x t 2 + c d e p ) ,         x t > 0 − d s , t x t − ( a d e p x t 2 + c d e p ) ,       x t ≤ 0 ,   ∀ t ∈ T (1) F = max x t ∈ χ t E ∑ t = 0 t = T C t ( S t , x t ) , ∀ t ∈ T (2) where S t is the state of the battery; a d e p and c d e p are the depreciation factors; x t < 0 is the charging power, x t > 0 is the discharging power; x t = 0 is the float power; χ t is the feasible region of x t , and T is set of dispatch periods.

The objective function is subject to the constraints as follows: − P e ≤ x t ≤ P e (3) S O C min < S O C t < S O C max (4) S O C t = S O C t − 1 − g ( x t ) Δ T (5) where P e is the maximum power of battery; S O C max and S O C min are the maximum and minimum SOC of battery, respectively; Constraint (5) describes the SOC transfer procedure of battery, g ( x ) = ( 1 + η ) + ( 1 − η ) sgn ( x ) 2 x , in which η is the charging efficiency of battery.

MDP is a sequential optimization problem whose goal is to find a policy to maximise expected profits or minimise expected costs (Zhu et al., 2019). Let S t and x t be the state variables and decision variables of the battery at stage t , respectively, W t is the exogenous information that arrives during the stage interval from t to t + 1 ; V ( S t ) is the value function of state S t . The evolution of the battery can be described according to the state transition function: S t + 1 = S M ( S t , x t , W t + 1 ) (6)

The problem is to find the strategy from t = 0 to t = T with the objective function: max E { ∑ t = 0 t = T γ C ( S t , x t ) } (7)

According to the optimality principle proposed by Bellman in 1957, Bellman equation can be used to reformulate and recursively solve as: V ( S t ) = max { C ( S t , x t ) + γ E [ V ( S t + 1 ) | S t ] } (8) In Equation 8, the original multi-stage optimisation problem can be decomposed into a serial of single-stage sub-problems and solved in turn. When solving the sub-problem in each stage, the latest exogenous information can also be considered to deal with the uncertainty of the system. The discount factor is set to be 1.

MDP theory is based on the setting of sub-problems according to temporal decomposition. In the temporal decomposition framework, there are four classes of elements, namely state variables, decision variables, exogenous information, and transition function. They are defined as follows:

i. State variable: The state variable is used to reflect the relationships among sub-problems in the decision-making process. The state variables of battery can be expressed as

Because the MDP problem in this paper contains the calculation of expected value, it will often lead to difficulty in solving. Then the problem of curse of dimensionality is brought about. ADP uses the method of iteratively updating the approximate value function to find a strategy to make the value function approximately optimal. In the existing works, there are several methods to update the approximate value function. In this paper, the method proposed in (Zhu et al., 2019) is used to update the slopes of the value functions.

i. If S O C 1 > S O C 2 , the current strategy can be summarized as: first, the EVs connect to the charging pile to discharge to the grid at the peak period. Then the EVs will charge when the valley period comes.

i. If S O C 1 > S O C 2 and the next period is peak period. The EVs keep discharging during the peak period. The EVs will change to charge when the valley period comes.

For EVs who return during valley period, they will charge immediately.

In this paper, a distribution system with stakeholders such as MMG, batteries, and EV clusters is considered. Batteries and EV clusters are energy storage systems (ESS). According to the unbalanced power, an MG with power surplus is called “the seller MG” while one with power shortage is called “the buyer MG”. An ESS is called “the seller ES”’ when discharging while one is called “the buyer ESS” when charging. An ESS without charging or discharging is called “the balanced ESS”. The open electricity market is considered, and power is allowed to be transferred between the multi-stakeholder as well as between the stakeholder and the distribution system ({DS}). Then the upper-level non-cooperative game model G of the seller MGs, the seller ESSs and the {DS} is defined as follows: G = 〈 Γ ; S ; U 〉 (13) where G denotes a game; Γ represents the participants; S represents the strategy and U is the gain of the participants; N r M G , N s M G , and N b M G denote the number of seller MGs, buyer MGs, and MGs with balanced power, respectively. N r E S S , N s E S S , and N b E S S represent the number of the seller ESSs, the buyer ESSs, and the balanced ESSs, respectively. If N r M G + N r E S S = N r = 0 , all the buyers contain the buyer MGs and the buyer ESSs purchase power from the {DS} while all the sellers sell redundant power to the {DS} when N s M G + N s E S S = N s = 0 . Therefore, in these two cases, there is no need for sellers to play the game with the {DS}. In other words, a necessary and sufficient condition of the game is that: N r × N s ≠ 0 (14)

According to the power curves predicted by all stakeholders, the transactions in the system are studied. The transaction rules are formulated as follows:

a. The seller MGs and the seller ESSs fix the price for selling electricity sale price, respectively. Additionally, the {DS} fixes the service charge.

The non-cooperative model is an operation situation in which all stakeholders trade with the {DS}, but do not trade with each other.

In the non-cooperation mode, active power P i is transferred at a medium voltage U 0 between the stakeholder i and the {DS}. Stakeholder i is seller when P i > 0 , and is buyer when P i < 0 . Accompanied with the transfer of power, power loss incurs due to transmission lines and transformers expressed as P i 0 l o s s = R i 0 I i 0 2 + β f ( P i ) = R i 0 f 2 ( P i ) U 0 2 + β f ( P i ) (23) where R i 0 is the resistance of the line between the stakeholder i and the {DS}; β is the coefficient of power loss in transformers; f ( P i ) is the power exchanged between the stakeholder i and the {DS}, which is defined as: f ( P i ) = { P i , P i > 0 W i ∗ , P i < 0 0 , P i = 0 (24) where W i ∗ is the power offered by the {DS}. In order to ensure that the stakeholder i acquire the necessary power P i r e q u i r e d = ( − P i ) , W i ∗ is given by a solution to Eq. 23 W i = P i 0 l o s s + P i r e q u i r e d = R i 0 W i 2 U 0 2 + β W i − P i (25)

We define the power loss cost as the U i of the system model in the non-cooperative mode U i = d x P i 0 l o s s (26) where d x denotes the cost for unit power loss. d x = d b if the stakeholder i is a seller; d x = d s if the stakeholder i is a buyer.

Multi-stakeholder and the {DS} do not have direct power transactions. All stakeholders cooperate with each other to complete power transmission and form a coalition. With the coalition as a unit to conduct power transactions with the {DS}, the stakeholders within the coalition can exchange energy. At the same time, power exchange between stakeholders can avoid losses on the transformer.

In order to study further the possibility of forming a coalition and the assignment of the coalition’s cost, we define the cooperative multi-stakeholder coalition game as a pair ( N , v ) . N denotes the set of participants and v is a characteristic function corresponding with every coalition S , v ( S ) denotes the cost of the coalition S achieved by the cooperation of stakeholder in it, defined as follows: v ( S ) = c i ∑ i ∈ S r j ∈ S s P i j l o s s + d b ∑ i ∈ S r P i 0 l o s s + d s ∑ i ∈ S s P j 0 l o s s (27) s.t. S r ∪ S s = S (28) S r ` S s ≠ ∅ (29) d i s t i j ≤ D (30) where S r and S s are the set of sellers and buyers in the coalition S , respectively. d i s t i j is the distance between the seller i and the buyer j , and D is the distance threshold.

The constraints indicate that there are a seller and a buyer in a coalition at least, and two stakeholders are possible to cooperate only if they locate nearer than the distance threshold. The system should also satisfy the line capacity constraints, voltage constraints, and power flow equation constraints. The power loss P i j l o s s is formulated as P i j l o s s = ( P i j U 1 ) 2 ∗ R i j ∗ d i s t i j (31) where P i j is the power transferred from the seller i to the buyer j , and R i j is the resistance of transmission line.

The impedance loss of the transmission line and transformer loss P i 0 l o s s will exist when the stakeholder i trades with the {DS}. It is formulated as follows, where Z T is the impedance of the transformer, and n is the transformer ratio. P i 0 l o s s = P i 0 l i n e + P i 0 T = R i 0 I i 0 2 + Z T ( n I i 0 ) 2 (32)

Therefore, we define the objective function of the lower-level of the model as follows: min v ( S ) (33)

The objective function represents the smallest cost of the coalition S , the least power loss cost yielded in the power transfer namely. The cost of the stakeholder i in a coalition S is defined as given by ψ i ( S ) = α i ( v ( S ) − ∑ j ∈ S ψ ( j ) + ψ ( i ) ) (34) where ∑ α i = 1 and α i / α j = ψ ( i ) / ψ ( j ) in the coalition S .

The NE solutions are the key to the upper-level game. All the strategy combinations of participants will be enumerated so that no NE solutions will be omitted. Therefore, the strategies need to be discretised. The discrete steps are Δ c i M G , Δ c j E S S , and Δ s D , respectively. All the discrete steps are constant. The strategies are expressed as follows: { c i M G = k c i M G ⋅ Δ c i M G c j E S S = k c j E S S ⋅ Δ c j E S S s D = k s ⋅ Δ s D (35)

The upper and lower limits of the strategy set have been defined above. Therefore, the discrete strategy combinations are finite. Moreover, the number of participants is also finite in this game. According to Theorem 1, the NE’s existence is ensured.

Theorem 1: (Nash 1950): In the n-player normal-form game G = ( S 1 , S 2 , ... , S n ; U 1 , U 2 , ... U n ) , if n is finite and S i is finite for every i then there exists at least one NE, possibly involving mixed strategies.

NSGA-II is a representative multi-objective optimization algorithm and similar to the basic genetic algorithm. Its core idea is: to sort the population non-dominantly, and then obtain the virtual crowded distance of each individual in each level. On this basis, the operations of selection, crossover, mutation, and elite retention are completed. NSGA-II uses Pareto sorting to find the minimum electricity loss value of each stakeholder without increasing the power consumption of other stakeholders, such a coalition is called the optimal stakeholder coalition.

Assume that ( N r + N s ) stakeholders disjoint two coalitions C = { C 1 , … , C c } and K = { K 1 , … , K k } , and the elements in the coalition do not intersect each other. For the coalition C = { C 1 , … , C c } , the U j of the stakeholder j in the coalition C i is ψ j ( C i ) = ψ j ( C ) , where ψ j ( C ) is the cost of stakeholder j which is expressed in (Lin et al., 2017). The coalition C is better than the coalition K when the following formula is fulfilled in C < K ⇔ { ψ j ( C ) ≤ ψ j ( K ) ,       ∀ j ∈ C , K } , and at least one of the participants j meets ψ j ( C ) < ψ j ( K ) . In other words, from the coalition C to coalition K, at least one stakeholder can obtain more gain from the coalition C without harming the gain of other stakeholders. In the optimal coalition, no stakeholder can increase its own profits without harming other stakeholders.

Merge rule: Merge any collections of { S 1 , … , S i } into a coalition { U j = 1 i K j } , when { U j = 1 i K j } < { K 1 , … , K i } .

Split rule: Split any coalition of { U j = 1 i K j } into { K 1 , … , K i } , when { K 1 , … , K S } < { U j = 1 i K j } .