We consider a discrete-time linear time-invariant system which is represented by x k + 1 = A x k + w k , y k = C x k + v k , where x k ∈ R n is the state, y k ∈ R m is the output measured by the sensor, A and C are known time-invariant matrices of appropriate dimensions. The Gaussian random vectors w k ∈ R n and v k ∈ R m represent the process and measurement noise, respectively, which are zero-mean and their covariance matrices are given by W and V, respectively. Furthermore, the initial state x ₀ is also a Gaussian random vector with covariance P ₀ and it is uncorrelated with w _k and v _k .

Let I k s = { y 0 , y 1 , … , y k } denote the measurement history available at the sensor at time k. The sensor is capable of pre-processing the raw measurements before transmitting it to the remote estimator for improving the estimation. By utilizing the measurement history, the sensor can compute the minimum mean square error (MMSE) state estimate, i.e., x ̂ k s ≜ E { x k | I k s } , by running a local Kalman filter and the error covariance is given by P k s ≜ E { ( x k − x ̂ k s ) ( x k − x ̂ k s ) T | I k s } . To determine the error covariance, first we define functions h , g : S + n → S + n as h X ≜ A X A T + W , (1) g X ≜ X − X C T C X C T + V − 1 C X . (2) By assuming that the pairs (A, W ^1/2) and (A, C) are controllable and observable, respectively, the availability of the entire measurement history ensures that P k s is convergent. More specifically, the steady-state error covariance, denoted by P ̄ , is the unique positive semi-definite solution of g ◦ h ( P ̄ ) = P ̄ . By initializing the local filter from P ₀, P k s converges to P ̄ exponentially fast. Thus, we assume that the Kalman filter has already entered steady-state at the sensor side.

The information exchange between the sensor and remote estimator is supported by a wireless communication channel. This channel is modeled as a two-state Markov chain which is also known as the Gilbert-Elliott (GE) channel (Gilbert, 1960; Elliott, 1963). This model can capture the effects of shadow fading and burst error and is more general than the more commonly adopted model of i.i.d. packet dropouts, i.e., memoryless channel. Thus, the GE model is more accurate for industrial environments since the presence of large moving objects in such environments means intermittent obstruction of the radio links which leads to burst error and time correlated channel gains (Quevedo et al., 2013). The channel according to the GE model can be in two states, namely a good (G) or bad (B) state as illustrated in Figure 1. The transition probabilities from G to B and B to G are denoted by p and q, respectively. Note that 0 ≤ p, q ≤ 1 and we further assume that q ≤ 1 − p, i.e., positively correlated channels.

At each time k, the battery-powered sensor transmits a data packet containing x ̂ k s over the channel to be received at the estimator. We assume that the transmitter at the sensor supports two power levels denoted by δ and Δ, where δ < Δ. Let A ≜ { 0,1 } and a k ∈ A denote the action space and the action at time k. The selected transmission power at the sensor can then be defined as a k = 1 , x ̂ k s  is transmitted with energy  Δ , 0 , x ̂ k s  is transmitted with energy  δ . (3) When transmission is done with low power (δ), the data packet is dropped if the channel is B, otherwise it is successfully received at the estimator side. However, when the high power is utilized (Δ), data transmission is guaranteed to be successful regardless of the state of the channel. This scenario is in accordance with the reliable data flow offered by commercial sensors when using their highest energy level (Xiao et al., 2006). Since a sufficiently high transmission power results in a high Signal to Noise Ratio (SNR) regardless of the channel condition, the energy level Δ can be determined such that this assumption holds.

We assume that instantaneous packet acknowledgements-/negative-acknowledgements (ACK/NACKs) are available through an error-free feedback channel and let γ _k ∈ {0, 1} represent it, i.e., γ _k = 1 if transmission is successful and γ _k = 0 otherwise. Moreover, let θ denote the scheduling scheme adopted by the sensor that determines a _k in Eq. 3. The available information at the estimator at k can then be described by I k θ = γ 0 , x ̂ 0 s γ 0 , γ 1 , x ̂ 1 s γ 1 , … , γ k , x ̂ k s γ k .

The MMSE state estimate and error covariance at the remote estimator are given by x ̂ k θ ≜ E x k | I k θ = A t k x ̂ k − t k s , P k θ ≜ E x k − x ̂ k x k − x ̂ k T | I k θ = h t k P ̄ , where t _k ≜ k − max _t≤k {t : γ _t = 1} is the time elapsed since the last successful packet reception and the function h(⋅) is defined in Eq. 1.

Our aim is to find a power scheduling scheme to achieve the best estimation performance while the energy constraint imposed by the limited capacity of the sensor’s battery is satisfied. To this end, we choose the trace of the error covariance matrix at the remote estimator as the performance metric. By considering the performance over the infinite horizon, the average expected value of this metric for a given schedule θ is considered as the objective P a θ = lim sup T → ∞ 1 T E ∑ k = 0 T − 1 t r P k θ . (4) The average energy cost incurred by implementing θ is given by J a θ = lim sup T → ∞ 1 T E ∑ k = 0 T − 1 1 − a k δ + a k Δ . Assuming that the energy budget determined by the expected operational time of the sensor is given by L (δ ≤ L ≤ Δ), we can formulate our problem of interest as.

PROBLEM 1

min θ P a θ , subject to J a θ ≤ L . (5)

By assuming that the channel state is always observed regardless of the chosen transmission power, Problem 1 can be transformed into a constrained MDP. The state space of this MDP is given by S = { G , B } × { P ̄ , h ( P ̄ ) , h 2 ( P ̄ ) , … } . Recall that if the channel state is G, data transmission is certainly successful. Therefore, the states { G , h i ( P ̄ ) } with i ≥ 1 can be excluded from the state-space as they are non-existent. Consequently, we can rewrite the state-space as S = { h − 1 ( P ̄ ) , h 0 ( P ̄ ) , h ( P ̄ ) , … } , where h − 1 ( P ̄ ) corresponds to the case where the channel is G and the error covariance is given by h − 1 ( P ̄ ) ≜ P ̄ . Furthermore, for i ≥ 0 we use h i ( P ̄ ) to refer the case of channel state being B with i consecutive packet dropouts. The non-zero transition probabilities, P , for a given action a ∈ A are as follows: P h − 1 P ̄ | h − 1 P ̄ , a = 1 − p , P h 0 P ̄ | h − 1 P ̄ , a = a p , P h P ̄ | h − 1 P ̄ , a = 1 − a p , P h − 1 P ̄ | h i P ̄ , a = q , P P ̄ | h i P ̄ , a = a 1 − q , P h i + 1 P ̄ | h i P ̄ , a = 1 − a 1 − q , where i ≥ 0. Furthermore, we denote the immediate cost and the immediate cost related to the constraint by R ( h i ( P ̄ ) , a ) = ∑ h j ( P ̄ ) ∈ S h j ( P ̄ ) P { h j ( P ̄ ) | h i ( P ̄ ) , a } and d ( h i ( P ̄ ) , a ) = ( 1 ‐ a ) δ + a Δ , respectively. The constrained MDP (CMDP) over the infinite horizon can now be described by the tuple ( S , A , P { ⋅ | ⋅ , ⋅ } , R , d ) .

Solving Problem 1 is equivalent to finding the optimal policy for the CMDP with the objective being minimization of the average cost, i.e., lim sup T → ∞ 1 T E { ∑ k = 0 T − 1 R ( P k , a k ) } , subject to the constraint lim sup T → ∞ 1 T E { ∑ k = 0 T − 1 d ( P k , a k ) } ≤ L . Since R ( h i ( P ̄ ) , a ) and d ( h i ( P ̄ ) , a ) are bounded below, the optimal solution of Problem 1 is a stationary schedule Altman (1999). Let μ = (μ ₋₁, μ ₀, μ ₁, … ) denote the optimal stationary schedule, where μ _i ∈ [0, 1] is the probability of the sensor using Δ for transmission when was h i ( P ̄ ) at the last time step. By investigating the stationary distribution of Markov chain induced by adopting μ , it is proved in (Qi et al., 2017) that the optimal scheduling policy is given by μ = 0 − 1 , … , 0 i ∗ − 1 , μ i ∗ , 1 i ∗ + 1 (6) where i ^∗ is the state of the chain where the sensor switches to high power with probability μ i ∗ which is given by μ i ∗ = E − 1 + E 1 − q 2 1 − q i ∗ 1 + E 1 − q 2 q 1 − q i ∗ − 1 , if  i ∗ > 0 , E 1 − 1 − q 2 − 1 E q 1 − q , if  i ∗ = 0 , p E − 1 / q , if  i ∗ = − 1 , (7) where E = [(p + q)(L − δ)]/[pq (Δ − δ)].

Essentially, the policy in Eq. 6 implies that optimal performance is achieved when the sensor continuously transmits with low power δ until the Markov chain reaches state i ^∗. At this state, the sensor switches to high power Δ with probability μ i ∗ . This switching probability is determined by upperbounding the average energy consumption by L and is derived in closed form Eq. 7. Finally, if transmission at i ^∗ is unsuccessful and the chain transitions to i ^∗ + 1, the sensor utilizes high power with probability 1 which ensures that the error covariance reduces to P ̄ at the remote estimator. Consequently, the state of the chain returns to h − 1 ( P ̄ ) or h 0 ( P ̄ ) , depending on whether the state of the communication channel is G or B, respectively.

We now consider the more realistic case where the sensor has only partial observations of the channel state. More specifically, whenever it transmits with high power Δ, i.e., a _k = 1, the state of the GE channel cannot be inferred since high power transmission is always successful regardless of the channel state. We will show how the knowledge of channel parameters p and q can be used to exploit the optimal schedule in Eq. 6 for optimal scheduling despite the partial observations.

Let belief be defined as the probability of the channel being in G and at a given time k and denote it by b _k . If low power δ is utilized, i.e., a _k = 0, the channel state at k is observed and the belief at k + 1 is obtained by b k + 1 = 1 − p , if  s k = G , q , if  s k = B , where s _k ∈ {G, B} denotes the channel state at k. Although the exact channel state at k would be unknown to the sensor when a _k = 1, i.e., high power transmission, it will still be able to keep track of the belief as b k + 1 = b k 1 − p + 1 − b k q .

Problem 1 can be formulated as an equivalent POMDP that can be converted to a corresponding MDP with the belief as a state, since b _k is a sufficient statistic for decision making given the past action history and observation history (Kaelbling et al., 1998). Nevertheless, without resorting to value iteration methods for solving this problem, we show that under certain conditions, the schedule in Eq. 6 can be applied for optimal performance.

In essence, when i ^∗ ≥ 1 in Eq. 6, the policy μ can be applied for solving the POMDP despite the lack of channel state observations when high power is used. If the sensor transmits with high power at k, the state of the chain returns to h − 1 ( P ̄ ) or h 0 ( P ̄ ) , depending on whether the channel is in state G or B, respectively. Since i ^∗ ≥ 1 and after successful transmission we have i < 1, policy Eq. 6 ensures that a successful transmission is always followed by low power transmission. In other words, the sensor will transmit with low power at k + 1 (a _k+1 = 0) regardless of the exact state of the chain. Consequently, the sensor can determine the state of chain at k + 1 from the ACK/NACK signal of its low power transmission at k + 1. Hence, the missing observations when i ∈ { − 1, 0} have no effect on the performance of this policy for the POMDP scenario and it leads to the same result as the MDP counterpart.

As long as i ^∗ ≥ 1, the schedule in Eq. 6 is optimal for the resulting POMDP problem considered here, and it can be successfully applied as discussed. Next, we derive the condition which guarantees that i ^∗ ≥ 1 by using the following the result of the following lemma (Qi et al., 2017, Lemma 4.1):

LEMMA 1

Define a schedule μ ̄ ( i ̄ ) such that μ ̄ i ̄ = 0 − 1 , 0 0 , … , 0 i ̄ , 1 i ̄ + 1 . (8) Under this schedule, J a ( μ ̄ ( i ̄ ) ) is a monotonically decreasing function of i ̄ and for i ̄ ≥ 1 J a μ ̄ i ̄ = δ + p q Δ − δ p + q 1 − q 1 − i ̄ − 1 − q 2 . (9)

By comparing μ ̄ ( i ̄ ) with the schedule in Eq. 6, since μ i ∗ ≤ 1 by definition, it readily follows that J a ( μ ̄ ( i ̄ ) ) ≥ J a ( μ ( i ∗ ) ) for all i ̄ = i ∗ . Therefore, from Eq. 9 we obtain that i ^∗ ≥ 1 and thus θ ^∗(i ^∗) if the energy budget satisfies δ ≤ L ≤ δ + p q Δ − δ p + q 1 − 1 − q 2 . (10)

Remark 1

Although the schedule Eq. 6 is optimal and applicable for the POMDP problem only when the the energy budget satisfies (10), we can apply the schedule Eq. 8 with i ̄ = 1 when the constraint Eq. 10 is violated. Adopting this policy results in suboptimal performance due to its lower energy consumption than the given budget L, which guarantees that the energy constraint is satisfied.

Remark 2

The Kalman filter is always stable, i.e., the expected value of the error covariance at the estimator is bounded, as long as L > δ. Since the error covariance at the remote estimator shrinks to P ̄ when high power is used, L > δ ensures that i ^∗ < ∞ in Eq. 6 and thus the error covariance is bounded by h i ∗ ( ( P ̄ ) ) < ∞ . In case L = δ, then the sensor will always transmit with low power and the expected error covariance can grow to infinity. In such scenarios, it is proved that the filter is mean-square stable if and only if q > 1 − 1 / σ max 2 ( A ) (Liu et al., 2021).

In Bayesian inference, the prior of an uncertain quantity is the probability distribution that would express one’s beliefs about the quantity in question before new data about it becomes available. The uncertain quantities in this work are the transition probabilities of the GE model, i.e., p and q, which will be referred to as channel parameters hereon. If the posterior distribution of these quantities is in the same probability distribution family as their prior probability distribution, the prior and posterior are then called conjugate distributions, and the prior is called a conjugate prior for the likelihood function. The unknown transition probabilities are within the interval [0, 1] and they can be viewed as random variables consisting of the number of successes in Bernoulli trials with unknown probability of success p and q. Since Beta distribution is the conjugate prior for Bernoulli distributions, we assume the prior distribution of the channel parameters follow the Beta distribution, which is parameterized by Φ ≜ [ ϕ 1 , ϕ 2 , ϕ 3 , ϕ 4 ] ∈ Z + 4 that we refer to as posterior count henceforth. Since the probability distribution of p and q are independent we can write P p , q ; Φ = P p ; ϕ 1 , ϕ 2 P q ; ϕ 3 , ϕ 4 , where P p ; ϕ 1 , ϕ 2 = p ϕ 1 − 1 1 − p ϕ 2 − 1 B ϕ 1 , ϕ 2 , (11) P q ; ϕ 3 , ϕ 4 = q ϕ 3 − 1 1 − q ϕ 4 − 1 B ϕ 3 , ϕ 4 , (12) where B(⋅) denotes the Beta function.

Using these prior distributions highly facilitates the posterior update. More specifically, after new observations are made, the posterior update can easily be done by updating the posterior counts (ϕ ₁, ϕ ₂) for p and (ϕ ₃, ϕ ₄) for q. For instance, consider that the channel state is G and Φ = [3, 1, 2, 3]. The next three observation of the channel are G, B and B in consecutive order. More specifically, the channel stays G, then transitions to B, and finally stays B which can happen with probabilities 1 − p, p and 1 − q, respectively. Consequently, the updated posterior count is easily obtained by Φ = [3 + 1, 1 + 1, 2, 3 + 1]. Without loss of generality, we will assume that the initial posterior count is Φ = [1 1 1 1] meaning that the channel parameters p and q are between zero and one with equal probabilities and P { p , q } = 1 .

Let z _k ∈ {G, B, V} denote the observation at time k, where z _k = V corresponds to not observing the channel state. Therefore, if a _k = 0 (low power transmission), z _k ∈ {G, B}, and if a _k = 1 (high power transmission), the channel state remains unobserved, i.e., z _k = V. We denote the observation history as z ^k = {z ₁, …, z _k } which is sufficient for inferring the action history. In addition, the channel state history is crucial for the following framework which is denoted by s ^k = {s ₁, …, s _k }.

When the observation at time k is given by z _k = V, the channel state could be either G or B, i.e., s _k ∈ {G, B}. Hence, multiple channel state histories can lead to the same observation history. We denote the set of all these possible channel state histories by S (z ^k−1) which is defined as S z k − 1 ≜ s k − 1 | s t = z t , ∀ t ∈ τ | z τ ≠ V . In addition, multiple state histories could result in the same posterior count. For instance, consider s ^k−1 ∈ S (z ^k−1) and s ^k = {s ^k−1, s _k } which consists of c ₁, c ₂, c ₃, and c ₄ number of transitions from G to B, G to G, B to G, and B to B, respectively. Therefore, P s k | p , q P p , q = P s k | p , q = p c 1 1 − p c 2 q c 3 1 − q c 4 . Since the posterior count is independent of the order of occurrence of the state transitions, multiple state histories can lead to the same posterior count. We define appearance count, denoted by Ψ(Φ, S(z ^k−1), s _k ), as the total number of state histories up to k which lead to the same posterior count Φ from the initial condition Φ = [1 1 1 1].

We consider the joint probability distribution of the state and channel parameters given the observation history, i.e., P ( s k , p , q | z t − 1 ) , which can be fully described by the posterior count and appearance count as P s k , p , q | z k − 1 P z k − 1 = ∑ s k − 1 P z k − 1 , s k | p , q P p , q = ∑ s k − 1 ∈ S z k − 1 P s k | p , q P p , q = ∑ Φ Ψ Φ , S z k − 1 , s k p ϕ 1 − 1 1 − p ϕ 2 − 1 q ϕ 3 − 1 1 − q ϕ 4 − 1 , (13) where P ( z k − 1 ) is the normalization term. When a new observation is made, the posterior is updated recursively as (Zou et al., 2016) P s k + 1 , p , q | z k = ∑ s k P s k , p , q | z k − 1 P s k + 1 , z k | s k , p , q / P z k | z k − 1 . (14) When a _k = 0, the sensor transmits with low energy δ and the state of the channel is observed, i.e., s _k = G or s _k = B. Consequently, the number of posteriors remains constant according to (14). However, when a _k = 1 and high power is utilized, we obtain z _k = V which means that s _k ∈ {G, B}. Hence, the summation in Eq. 14 is taken over both possible channel states which increases the number of possible posteriors. Therefore, with each high power transmission the number of possible posterior counts increases which inevitably grows to infinity. Figure 2 demonstrates how the posterior count and appearance count are updated based on the observations and the effect of a _k = 1 on the growth of possible posterior counts.

The presented Bayesian framework enables us to incorporate the uncertainty in the channel parameters in the decision making process. More specifically, prior to transmission at each time k the probability distribution of the channel parameters is updated with respect to the available state and observation history up to k − 1. The estimated parameters are then used to evaluate Eq. 7 to obtain the optimal schedule Eq. 6. Then the sensor adopts this schedule for adjusting the transmission power which yields a new observation at k. Repeating this procedure over time increases the accuracy of the learned channel parameters and thus the readjusted schedule. There are, however, several challenges that need to be addressed for implementing this idea in practice.

First, as aforementioned, with each high power transmission the number of possible posterior counts grows and thus this number inevitably goes to infinity over time. A common approach to avoid this problem is to ignore the posterior update with high power transmission which leads to a constant number of posterior counts. This approach is not applicable in our problem since it assumes that the underlying two-state Markov chain of the channel remains frozen at those times, as it is in rested bandit problems (Maghsudi and Hossain, 2016). We instead adopt the idea of approximate belief monitoring (Ross et al., 2011; Zou et al., 2016). This method allows us to take into consideration the possible state changes during high power transmissions, while it ensures computational tractability. Essentially, the number of posterior counts kept in the update stage are limited to a constant number K. Moreover, the kept posterior counts are drawn randomly with respect to their appearance count.

The next challenge arises from the dependency of the uncertain channel parameters, optimal schedule and the resulting average energy consumption. The exploration/exploitation dilemma in the presented framework is addressed by applying the schedule in Eq. 6, which is derived based on the average energy constraint. Without explicitly including this constraint in the algorithm, a policy is derived which minimizes the trace of the expected error covariance based on the current posterior distribution of the state and parameters of the channel obtained via the aforementioned Bayesian framework. This tends to leverage the exploration/exploitation toward a better estimation performance at the cost of higher energy consumption and less accurate p and q. To overcome this, we explicitly enforce the energy constrain by utilizing a feedback mechanism in the algorithm as described in Algorithm 1.

We define ξ ^G ≜{[ϕ ₁, ϕ ₂, ϕ ₃, ϕ ₄], Ψ, Π} as the set of information on the posteriors of the channel being in G and let Π denote the joint probability distribution Eq. 13, which is set to 0.5 initially. Each posterior count in ξ ^G can transition into G or B with probability of 1 − p and p, respectively, where p is the mean of the corresponding Beta distribution (p = ϕ ₁/(ϕ ₁ + ϕ ₂)). Similarly, we define ξ ^B for the posterior counts associated with the channel state B and the corresponding transition probabilities are determined based on their respective q = ϕ ₃/(ϕ ₃ + ϕ ₄). These operations are shown by the two procedures on Line 19 and Line 23. After the update, these sets are merged which is shown by the operator ∪ , such that Ψ and Π of the items with identical posterior counts are summed and then only K posteriors are kept as shown in Line 17 and Line 17, respectively.

To ensure that exploration/exploitation is performed in a manner that the average energy consumption of the returned policy is L, the following corrections are made at each time step k:

• First, the average energy consumed in previous steps is computed in Line 4.

This method guarantees that the energy constraint is satisfied regardless of the accuracy of the learned parameters and leads to near-optimal performance as shown in Section 5.

Algorithm 1

Joint learning and scheduling design.