In this section, we present our proposed LG-H-PPO (Latent Graph-based Hierarchical PPO) framework in detail. The core objective of this framework is to significantly reduce the complexity of long-term planning for high-level policies in offline hierarchical reinforcement learning (Offline HRL), particularly within PPO-based frameworks, by introducing a latent variable graph structure. The overall architecture of LG-H-PPO is illustrated in Figure 1, comprising three organically integrated stages: latent variable encoder pre-training, latent variable graph construction, and graph-based hierarchical PPO training.

LG-H-PPO follows the fundamental paradigm of HRL by decomposing complex navigation tasks into two levels: high-level (subgoal selection) and low-level (subgoal arrival). The key innovation of LG-H-PPO lies in constructing a discrete latent variable graph G = ( V , E ) . The task of the high-level policy π h is no longer to “create” a latent subgoal z in a high-dimensional continuous space but is simplified to “select” an adjacent node as the next subgoal on this pre-constructed graph G . The low-level policy π l executes atomic actions, facilitating transitions between latent variable states represented by graph nodes. The entire framework is designed around training using a fixed offline dataset D to enhance sample efficiency and accommodate scenarios where extensive online interactions are impractical. We apply K-Means clustering to the latent representations z of all states in the offline dataset D to identify K representative cluster centroids, which form the nodes V of our graph. Edges E are then established between nodes that are temporally adjacent in the dataset. This entire process is visualized in Figure 2.

The first stage involves pretraining a latent variable encoder. The objective is to learn a high-quality, low-dimensional state representation z from offline data, which should capture key state information and reachability relationships between states. We aim to obtain an encoder q ϕ ( z | s ) . This model primarily consists of three key components: Encoder (Posterior Network) q ϕ ( z | s t G , s t + c G ) : Inputs the current state s t G (mapped to the target space, e.g., coordinates) and the future state s t + c G after c steps, outputting the posterior distribution parameters (mean μ post and variance σ post 2 ) of the latent variable z . Decoder Network p θ ( s t + c G | s t G , z ) : Inputs the current state s t G and the latent variable z ∼ q ϕ sampled from the posterior distribution, attempting to reconstruct the state s t + c G after c steps. Prior Network ρ ω ( z | s t G ) : Inputs only the current state s t G and outputs the prior distribution parameters of the latent variable z (mean μ prior and variance σ prior 2 ). All three networks are implemented using Multi-Layer Perceptrons (MLPs) with ReLU or GELU activation functions (Hafner et al., 2020). The training objective maximizes the Evidence Lower Bound (ELBO), combining reconstruction loss and KL divergence regularization terms: L V AE = E z ∼ q ϕ log p θ s t + c G | s t G , z − β D K L q ϕ z | s t G , s t + c G ‖ ρ ω z | s t G

Next comes the core innovation: we constructed a latent variable graph, discretizing the high-dimensional, continuous latent variable space Z learned in Phase 1 into a structured directed graph G = ( V , E ) . Nodes V represent representative states (or regions) within the latent space, while edges E denote the latent reachability between these states (based on observations in the offline dataset). We use the trained encoder q ϕ to encode all states { s i } i = 1 N in the offline dataset D , yielding the corresponding latent variable set Z = { z i } i = 1 N ⊂ R d z , where d z is the latent variable dimension. Select an appropriate number of nodes K (a key hyperparameter, e.g., K = 100, 200, 500). Apply the K-Means algorithm to cluster the latent variable set Z , aiming to minimize the sum of squared errors within clusters: argmin   C ∑ k = 1 K ∑ z i ∈ C k ‖ z i − c k ‖ 2

The K cluster centroids V = { c 1 , … , c K } are defined as the K nodes of the latent variable graph G . Each node v k represents an abstract region within the latent space.

To reflect dynamic reachability between states, we utilize trajectory information from the dataset D to construct graph edges. We traverse each trajectory segment τ = ( s t , a t , s t + 1 , … , s t + k ) in D (where k can be a fixed number of steps, such as c H T D (Baek et al., 2025), or the entire trajectory). Let z t = q ϕ ( s t ) and z t + k = q ϕ ( s t + k ) denote the initial and final latent variables of the segment, respectively. Use the K-Means model to find their nearest node indices: i = arg min l ‖ z t − c l ‖ 2 and j = arg min l ‖ z t + k − c l ‖ 2 . If i ≠ j , this indicates the presence of observations in the dataset moving from node v i ’s region to node v j ’s region. We then add a directed edge e ( v i , v j ) ∈ E to the graph G . E = e v i , v j | ∃ τ = s t , … , s t + k ∈ D , i = arg min l ‖ q ϕ s t − c l ‖ 2 , j = arg min l ‖ q ϕ s t + k − c l ‖ 2 , i ≠ j

Next, train the hierarchical PPO policy π = ( π h , π l ) on the constructed latent variable graph G to enable it to effectively utilize the graph structure for long-term planning and navigation. The task of the lower-level policy ( π l ) is to learn π l ( a t | s t , c j ) , enabling the agent to drive itself from the current state s t to the latent variable region represented by the target node c j . We adopt an offline Soft Actor-Critic (SAC) (Haarnoja et al., 2018) training paradigm based on Advantage-Weighted Action Cloning (AWAC/AWR) (Peng et al., 2019). At the high-level policy level ( π h - PPO), our task is to learn π h ( c j | v i , z final ) , which selects the optimal next neighbor node c j as a sub-goal on the graph G based on the current graph node v i and the final objective (encoded as z final ). The high-level state consists of the current encoded and mapped graph node v i (or its latent variable c i ) and the latent variable of the final goal z final . The discrete action space A h ( v i ) = { v j | e ( v i , v j ) ∈ E } represents the set of all neighboring nodes reached via outgoing edges from a node v i in graph G . During PPO training, the network architecture π h comprises Actor and Critic networks, both implemented using MLP. Compute the K logits output by the actor. Simultaneously, a mask is constructed to set the logits of all non-neighboring nodes v k ∉ A h ( v i ) to − ∞ . Subsequently, a Softmax function is applied to the logits of the remaining neighboring nodes to obtain a probability distribution, from which the next node is sampled: c j ∼ π h ( ⋅ | v i , z final ) . The high-level PPO relies on round data ( v t , c t + 1 , R h , t , v t + 1 ) , which we collect through simulated interactions. Using the collected high-level rollout data, we compute the Generalized Advantage Estimate (GAE) (Schulman et al., 2015): A ̂ t = ∑ l = 0 N − 1 γ λ l δ t + l , where  δ t = R h , t + γ V ϕ v t + 1 , z final − V ϕ v t , z final Specifically, to stabilize the high-level policy updates and prevent catastrophic policy collapse, we employ the standard clipped surrogate objective function of PPO. The optimization objective L CLIP ( θ ) for the high-level policy network π θ is defined as: L CLIP θ = E ̂ t min r t θ A ̂ t , clip r t θ , 1 − ϵ , 1 + ϵ A ̂ t where r t ( θ ) = π θ ( c j | v i , z final ) π θ old ( c j | v i , z final ) denotes the probability ratio between the new and old policies, and A ̂ t is the Generalized Advantage Estimate (GAE) computed via Equation 3. ϵ is the clipping hyperparameter (set to 0.2 in our experiments). This objective ensures that the updated policy does not deviate excessively from the behavior policy that generated the data, guaranteeing monotonic improvement during training. Subsequently, we update the Actor and Critic networks using multiple iterations (epochs) and mini-batches of data, minimizing the joint loss function of PPO. This ultimately yields the trained low-level policy π l and high-level PPO policy π h .

We evaluate on the Antmaze navigation benchmark, focusing on antmaze-medium-diverse-v2 and antmaze-large-diverse-v2. These environments simulate navigation tasks for quadrupedal robots in medium and large mazes, characterized by high state space dimensions (29-dimensional), continuous action space (8-dimensional), limited field of view, sparse rewards (+1 only upon reaching the goal), and long task durations (up to 1,000 steps). They serve as an ideal platform for testing long-term planning and offline learning capabilities. Training is based on the antmaze-medium-diverse-v2 and antmaze-large-diverse-v2 offline datasets. The diverse dataset contains a large number of suboptimal trajectories generated by medium-level policy exploration, offering broad coverage but few successful trajectories. This places high demands on the algorithm’s trajectory stitching capabilities and its ability to learn optimal policies from suboptimal data (Baek et al., 2025; Shin and Kim, 2023). Each dataset contains approximately one million transition samples.

Baselines: To comprehensively evaluate LG-H-PPO’s performance, we selected the following representative baselines for comparison: 1.

Guider (Shin and Kim, 2023): A state-of-the-art offline HRL algorithm based on VAE latent variables and continuous high-level actions. It serves as a crucial foundation and benchmark for our approach.

We use normalized scores as the primary performance metric. This score is linearly scaled based on the environment’s raw rewards, where 0 corresponds to the performance of a random policy and 100 corresponds to the performance of an expert policy. We run each algorithm and environment with five different random seeds, reporting the final policy’s average normalized score, standard deviation, maximum, and minimum over 100 evaluation rounds. Additionally, we plot the average normalized score curve (learning curve) during training to compare the convergence speed and training stability of different algorithms. We implement the LG-H-PPO framework using PyTorch. Clustering Implementation Details: For the latent graph construction, we utilize the KMeans module from the Scikit-learn library. To ensure high-quality cluster center initialization and faster convergence, we explicitly employ the ‘k-means++’ initialization strategy rather than random initialization. This is critical for generating representative graph nodes in the complex high-dimensional latent space. Discussion on Node Count K: The number of graph nodes K is a key hyperparameter balancing ‘planning resolution’ and ‘computational complexity’. A smaller K may result in nodes representing overly large regions, losing local details, while an excessively large K significantly increases the action space for the high-level policy, complicating learning. In our experiments, K = 200 was empirically selected as the optimal value for the Antmaze tasks. Detailed hyperparameter settings are shown in Table 1. All experiments are conducted on a server equipped with an NVIDIA RTX 4090 GPU.

We summarize the final performance of LG-H-PPO and various baseline algorithms on the D4RL Antmaze task in Table 2. To present the results more comprehensively, we report the average normalized score, standard deviation, and maximum and minimum scores across 100 evaluation rounds for five random seeds.

Table 2 clearly demonstrates the superiority of LG-H-PPO. In the antmaze-medium environment, the average scores of all hierarchical methods significantly outperform the non-hierarchical CQL + HER, highlighting the inherent advantage of hierarchical structures in handling long-temporal sequence problems. LG-H-PPO achieves a high score of 90.5 on this task, matching the performance of top methods HIQL and GAS while significantly outperforming continuous latent space planning-based approaches like Guider and H-PPO (Cont.). This performance advantage is further amplified in the more challenging antmaze-large environment. Confronted with longer paths and sparser rewards, the non-hierarchical method CQL + HER experiences a steep decline to 11.3. Guider and H-PPO (Cont.) also drop to 80.8 and 68.1 respectively, indicating significantly increased difficulty in long-term planning within continuous latent spaces. In contrast, LG-H-PPO, leveraging its planning capability on discrete latent variable graphs, maintained a high performance of 85.6. This substantially outperformed both Guider and H-PPO (Cont.), coming very close to HIQL’s performance. This strongly demonstrates that latent variable graph structures are key to overcoming the bottlenecks of long-term offline HRL planning. By discretizing the action space of high-level PPO, we enable policy gradient methods to more effectively learn long-range dependencies and select optimal subgoal sequences. Concurrently, LG-H-PPO exhibits a relatively small standard deviation, and the gap between its maximum and minimum values indicates stable performance across different random seeds.

To provide a more intuitive understanding of LG-H-PPO’s decision-making process, we visualize a planned trajectory in the Antmaze-Large environment in Figure 3. The visualization highlights the two-level hierarchical structure.As shown in Figure 3a, the high-level PPO policy ( π h ) , operating on its discrete action space (the graph nodes), selects an efficient sequence of latent subgoals (yellow stars) from the start state (S) to the final goal. This demonstrates its long-term planning capability. Figure 3b shows the low-level policy ( π l ) in action, executing primitive actions to successfully navigate and reach each of the discrete subgoals provided by the high-level.This qualitative result visually confirms that our framework effectively decomposes the complex, long-horizon task into a series of simpler, short-horizon navigation problems, validating the efficacy of our latent graph-based approach.