Based on the need to fairly compare algorithms (Henderson et al., 2018), benchmarks have proliferated in the Reinforcement Learning (RL) community. These cover a wide range of tasks, from the full collection of Atari 2,600 games (Bellemare et al., 2013) to 3D simulations in Mujoco (Laskin et al., 2021). However, in recent years, the focus of research has changed from producing more complex environments to producing a range of environments. Although undeniable progress has been made with respect to the capabilities of trained agents, much remains to be done for their capacity to generalize (Mnih et al., 2015). In practice, agents “will not learn a general policy, but instead a policy that will only work for a particular version of a particular task with particular initial parameters” (Risi and Togelius, 2020).

Thus, a recurring theme in modern RL research is the training of agents in various situations to avoid overfitting. Although some algorithms have built-in solutions to smooth out the learning process, e.g., TD3 (Fujimoto et al., 2018) (where small perturbations are applied to the actions), providing such a diversity of experience primarily originates from the environments themselves. To this end, numerous benchmarks now consist of a collection with varying degrees of homogeneity. Some of them have a similar structure, as in the Sonic benchmark (Nichol et al., 2018) where levels are small areas taken from three games in the franchise. In other cases, environments share very little: In the Arcade Learning Environment (ALE), the single common factors are the dimensions of the observation space (Bellemare et al., 2013). Intermediate test suites with distinct but complementary sets of “skill-building” tasks have also been designed, for example, with the Mujoco simulator (Wawrzyński, 2009; Yu et al., 2019; Laskin et al., 2021) or Meta-World (Yu et al., 2019). However, all of these examples share a common feature: the set of environments is predefined, generally the result of a costly human-tailored design procedure, e.g., Beattie et al. (2016).

To solve this generalization problem, agents must face sufficiently diverse situations so that the underlying principles are learned instead of a specific trajectory. Naturally, this requires generating environments that exhibit such diversity while still offering the same core challenges. For instance, in the context of maze navigation, reaching e.g., the exit might be the goal while the actual topology of said maze is only relevant insofar as giving agents a wide sample of states to learn from. One common way to address this later point is to use procedural generation (Beattie et al., 2016; Kempka et al., 2016; Harries et al., 2019; Juliani et al., 2019; Tomilin et al., 2022) or complementary techniques such as evolutionary algorithms (Alaguna and Gomez, 2018; Wang et al., 2019). For example, ProcGen (Cobbe et al., 2020) encompasses 16 different types of environment and serves as a generalizable alternative to ALE. Adapting more recent video game environments, either directly (Synnaeve et al., 2016) or in a light format (Tian et al., 2017), can help further push adaptability by providing finer-grained perceptions and actions. While such an approach can be used to create large training sets, the main difficulty becomes the design of a sufficiently tunable generator, i.e., one in which desirable features are easy to introduce.

Considering the challenges of generating a panel of demanding training environments, the contribution of this article is two-fold:

After highlighting, in Section 2, the niche this generator occupies in the current benchmark literature, we describe its main components in Section 3. Three alternative methodologies for training generalized maze-navigating agents are then detailed in Section 4 alongside two algorithms (A2C and PPO). The resulting performance in handling unknown environments is then thoroughly tested in Section 5, allowing us to draw conclusions about the relative benefits of the generator, the training processes, and the underlying algorithms.

To place this generator in perspective, we conducted an extensive comparison with a select number of commonly used benchmarks. As our library is primarily targeted at Python environments, we restricted the set of considered environments to those that could be reliably installed and used on an experimenter's machine. Timing was done on 1000 time steps averaged over 10 replicates on an i7-1185G7 (3GHz) using the Python 3.10 version of all libraries, except for the Unsupervised Reinforcement Learning Benchmark (URLB) (Laskin et al., 2021) and RetroGym (Nichol et al., 2018) which required Python 3.8. In the latter case, we used the ROMs linked in the library's documentation. The scripts, intermediate data and figures are available as part of AMaze's repository.

As detailed in the following section, AMaze can provide environments for fully discrete, fully continuous, and hybrid agents. Table 1 illustrates how the former case allows for fast simulation at the cost of low observable complexity. Based on the time taken to simulate 1,000 timesteps, only the simplest of the gymnasium suite (Sutton and Barto, 2018) is comparable to AMaze which, in addition, provides numerous unique and experimenter-controlled environments. In the hybrid case, where agents perceive images but still only take discrete steps, the library is on par with Classic Control tasks (Barto et al., 1983) such as Mountain Car or Cart Pole. ProcGen (Cobbe et al., 2020) addresses similar concerns as AMaze and is quite comparable in terms of speed, but has a stronger focus on randomness, with difficulty levels being the main way of controlling the resulting environments. DeepMind Lab2D (Beattie et al., 2020), while noticably slower, is also extensively customizable, albeit through lua scripting, and allows for heterogeneous multi-agent experiments.

With respect to the fully continuous case, the most computationally expensive of the three regimes, AMaze performs at a level similar to that of Box2D or Mujoco (Todorov et al., 2012), which, in traditional implementations such as gymnasium (Towers et al., 2024), lack customization capabilities. Purely vision-based benchmarks such as ALE (Bellemare et al., 2013), RetroGym (Nichol et al., 2018), Meta-world (Yu et al., 2019), Unsupervised Reinforcement Learning Benchmark (Laskin et al., 2021), LevDoom (Tomilin et al., 2022), or Maze Explorer (Harries et al., 2019), while offering a more challenging task than AMaze, also exhibit drastically higher costs with variable levels of experimenter control over the environments.

To summarize, AMaze has comparable computational costs with the most simple of environments (e.g., Classic Control or Toy Text) while providing much finer grained control over the challenges proposed to the agent. Conversely, even in its most expensive variation (fully continuous) it outperforms complex settings such as Mujoco-based environments, ALE or Retro-Gym. Combined with its extensive and intuitive parameterization, this makes it an ideal platform for trying out new algorithms, policies or hypotheses before deployment on more demanding contexts.

It follows that AMaze fills a very specific niche in the benchmarking landscape by providing a computationally inexpensive framework to design challenging environments. Control over the various characteristics of said environments is left in the hands of the experimenter through a number of high- and low-level parameters that will be described in the following section.

Learning to navigate mazes represents a flexible, diverse, yet challenging tested for training agents. Here, we propose a generator (AMaze) for this task with the following primary characteristics:

These features make it possible to generate a wide range of mazes for a variety of purposes, from a fast prototyping RL platform to a testbed for embodied computer vision in indirectly encoded NeuroEvolution. In the remainder of the section, we detail the major components of this generator namely the environment's parameters, the agents' capabilities, the reward function and, finally, unifying metrics for comparing widely different mazes.

To teach agents generalizable navigation skills, we define a training maze, presented in Figure 4a. Although, for simplicity, we only depict one variation of this maze, in practice, the agent is trained on all four rotations (Figure 4c). Thus, the agent will not overfit to a particular upper-diagonal type of behavior, but instead will have to develop a context-dependent strategy. Furthermore, intermediate evaluations of the agent's performance are performed in a similar maze (in terms of complexity) with a different seed, as shown in Figure 4b.

To showcase this benchmark's integration with current Reinforcement Learning frameworks, we used Stable Baselines 3 (Raffin et al., 2021) and more specifically their off-the-shelf Advantage Actor-Critic (A2C) and Proximal Policy Optimization (PPO) algorithms with all hyperparameters kept to their default values (Mnih et al., 2016; Schulman et al., 2017). The total training budget is of 3,000,000 timesteps, divided over the four rotational variations of the training maze with possible early stopping if the optimal trajectory is observed on all evaluation mazes.

Following the training protocols defined in the previous section, we evaluated the final agents on two complementary tasks to determine whether they had acquired generalized behavior in the target maze class. The first is straightforward: can the agent solve any maze of a given complexity or lower? To answer this, we generated 18 mazes, as shown in Figure 5, based on varying amounts of features (clues, lures, and traps) and Surprisingness. The agents are then evaluated with respect to two goals: their success (do they reach the goal) and their reward (cumulative normalized reward R̄, as in Equation 1). Although this allows for comparison between agents based on performance under “normal conditions,” this method suffers from cumulative failure: an error at a given time point may preclude any further success. In fact, agents who take a wrong turn somewhere have little information on how to get back on track. Thus, sub-optimal strategies may end up indistinguishable from trivially bad ones.

To counteract this trend, we also perform a complementary evaluation in a more abstract context. Because the input is discrete and thus enumerable, we can generate the complete set of possible input arrays. As we know which is the correct decision, we can assess which inputs are correctly processed by the agents among the four classes: empty corridor, corridor with lure, intersection with clue, and intersection with trap. Although less “natural,” this method ensures complete coverage of all the possible situations that an agent may encounter on an infinite number of mazes. Conversely, it also implies that we may be testing an agent on input configurations that it has never seen during training.

The interested reader might also refer to Supplementary material 2 for details on the dynamics of each group's error with the different sign types. The full data for every training dynamics is available in Godin-Dubois (2024).

In this work, we presented a benchmark generator that is geared toward the easy generation of feature-specific mazes and the intuitive understanding of the resulting agents' strategies. The visual cues (either pre-processed or raw) these agents must learn to use to successfully navigate mazes are designed in a CPU-friendly manner so as to drastically limit computational time. By grounding an embodied visual task in what is essentially a succession of lookup-table queries, we allow complex cognitive processes to take place while avoiding the cost of a full robotics simulator. As the agents have only access to local information, this generator is applicable across a broad range of research domains, e.g., from sequential decision making to embodied AI. To help future researchers in manipulating and comparing mazes with widely different characteristics, we introduced two partially orthogonal metrics that accurately capture two key features of such mazes: their Surprisingness and Deceptiveness.

Furthermore, to demonstrate the potential of this generator, we compared the training capabilities of the Advantage Actor-Critic (A2C) and Proximal Policy Optimization (PPO) algorithms in three different training regimens with varying levels of environmental diversity. Direct training was a brute-force approach with only a target maze, while the Interpolation case relied on a scaffolding approach presenting increasing challenges. Finally, an interactive methodology (EDHuCAT) was introduced to leverage human expertise as often as possible.

We evaluated the performance of both the maze navigation capabilities of trained agents and their ability to correctly process the entire observation space. Across all these metrics, it was shown that PPO significantly outperforms A2C in dynamic environments, demonstrating the relevance of the former in producing generalized agents. Furthermore, we found that EDHuCAT together with PPO was clearly one step above the alternatives when aiming for a general maze-navigating agent. At the same time, if one strives for more than a singular champion but, instead, for reproducible performance, then results point to both the Interpolation and interactive training setups as valid contenders when used in conjunction with PPO.

While demonstrating the potential of AMaze as a benchmark generator for AI agents, this work also raised a number of questions. First, we aim to confirm whether the observed higher performance of PPO is explained by its use of a trust region, which reduces learning speed and, in turn, overfitting. Furthermore, as we limited the study to two RL algorithms and a single neural architecture, many questions remain open with respect to the best choice of hyperparameters or even the applicability of other techniques, such as Evolutionary Algorithms. Second, we only briefly mentioned the impact of the human in the interactive case, and while preliminary data (Godin-Dubois, 2024) show tentative relationships between the human strategy, the training algorithm, and performance, dedicated studies are required to provide definitive answers. The strategy could be studied, as well as additional factors: Do youngsters train better than their elders? Does having a background in AI help? Or can laymen outperform experts?