Remote supervision of autonomous agents where a human operator is working remotely with an autonomous system to achieve a shared goal is a challenging yet important modality for future exploration or operational environments, including spaceflight, undersea robotics, manufacturing, and transportation. These environments are characterized by highly trained operators, where there are often consequences to safety and performance due to improper action, particularly when there is uncertainty in the state of the system. During supervisory control, human operators spend the majority of their time monitoring autonomous agents, and only a small portion of their time providing intermittent inputs to the system. Monitoring is critical for understanding system states, anticipating future issues, and quickly detecting and responding to failures (Sheridan, 2012), and is the portion of supervisory control of interest in this work. Monitoring requires operators to process complex data from many sources, often without the environmental context afforded to traditional modes of operation (Cummings et al., 2019). Remote operations are especially challenging because operators are separated spatially and/or temporally from the environment they seek to control. Thus, the operator’s perceptual processing abilities are decoupled from the physical environment (Chen et al., 2007), which creates a host of new challenges for human information display, including compromised situation awareness (SA). (Chen et al., 2007; Endsley, 2000). SA can be defined as having three levels where level 1 is “the perception of critical elements in the environment”, level 2 is “the comprehension of their meaning” and level 3 is “the projection of their status into the future” (Endsley, 1988). Low SA can make operations a difficult and cognitively challenging task, reducing mission effectiveness and task completion.

Spaceflight operations is one such monitoring application that is becoming increasingly important and has been chosen as the basis of this paper. Beyond the challenges described above, rendezvous missions and in-orbit servicing of satellites is an area of increasing interest and research focus for spacecraft owners and operators. On-orbit servicing requires satellites working in close proximity, with approach orbits controlled by complex, non-intuitive flight dynamics, and the relative motion sensing is impacted by time-varying lighting and space environment conditions. The close proximity of the satellites during a servicing mission increases the risk of collision. This risk is exacerbated by the uncertainty introduced by imperfect knowledge of the state of the satellites on-orbit (i.e., their exact positions, velocities, and orientations.) The close proximity reduces decision times compared to standard operations, particularly for the timing of collision avoidance maneuvers which may be required to be performed in seconds (Sellmaier and Frei, 2022).

One identified key necessity for next-generation supervisory control rooms is the development of a system capable of managing large volumes of data and presenting it in a way that enhances SA (Sim et al., 2008). Currently, spaceflight displays involve densely packed telemetry data. 2D graphical representations of system status, such as an orbital component vs. time, may be shown on occasion (Sim et al., 2008). These displays present information on 2D monitors, where human operators interact with the system using keyboards and mice. This can increase workload for operators, especially when the work involves processing and interacting with 3D data on a 2D screen (Bualat et al., 2013; Dan and Reiner, 2017; Woods and David, 2011). Thus, for spaceflight and similar operational environments, there is a need for improved understanding and SA, which VR or 3D visualizations may be able to provide. However, there is a gap in the literature in understanding how these display changes can affect spaceflight operations.

Virtual reality (VR), has been proposed as an alternative to traditional 2D display interfaces in many situations that would benefit from increased immersion in the environment and 3D interactions. Such displays increase telepresence, which is the feeling of being present in an environment other than where one is physically (Sheridan, 1992). VR may enhance performance through the improvement of perception, increased field of view, and the ability to change viewpoints without the loss of telepresence (Chen et al., 2007; Tittle et al., 2002; Elor et al., 2021). The ability to change viewpoints (i.e., teleportation) offers improvements in performance over a fixed viewpoint but comes with the risk of increasing cognitive load (Naceri et al., 2021). Furthermore, the ability to increase the field of view through natural head motions in VR can help improve collision avoidance and understanding of future vehicle states over the reduced field of view 2D displays. In prior work, immersive displays has been studied in the context of data visualizations and immersive analytics, and have been shown to improve estimations of depth, size, distance, cluster identification, and trajectories (Kraus et al., 2020; Whitlock et al., 2020; Kraus et al., 2022; Etemadpour et al., 2013; Hurter et al., 2019), further promoting the hypothesis that VR may be useful for operational monitoring.

Most of the prior research into using VR and remote operations has focused on direct teleoperation, where the human operator manipulates or controls a robot (Chen et al., 2007; Dima et al., 2019; Wilde et al., 2014; Naceri et al., 2021; Whitney et al., 2020a). This includes situations such as operating undersea robots (Elor et al., 2021), remotely driving a car (Hosseini and Lienkamp, 2016), or remotely manipulating robotic arms (Whitney et al., 2020a). Whitney et al. (Whitney et al., 2020a) found that using VR to complete a teleoperation task with a robotic arm led to faster completion time, lower workload, and improved usability compared to traditional monitor and keyboard interfaces. Similarly, Elor et al. concluded that stereoscopic VR displays led to faster completion times, increased usability, and increased perceived presence and performance over desktop displays in an underwater capture task. However, no differences were found in other metrics of workload (Elor et al., 2021).

Consistently, studies found that VR displays can improve depth perception and collision avoidance and lead to faster task completion, increased sense of presence, increased usability, and reduced perceived effort compared to the 2D displays (Chen et al., 2007; Tittle et al., 2002; Elor et al., 2021; Whitney et al., 2020a). However, there are mixed results on the effects of VR on task performance. For example, some studies find VR does not change workload (Elor et al., 2021) while others reported a decrease (Whitney et al., 2020b; a). With regards to SA, some studies have found improvements (Hosseini and Lienkamp, 2016), while others have reported no differences (Read and Saleem, 2017) between VR and screen displays. Some of these differences may be attributed to display design choices as there is a lack of validated design principles for VR operations and ineffective display design choices may result in reduced performance. In addition, the task type or complexity may also influence the outcome, as VR may not be appropriate for all direct control tasks.

A gap in prior work is the use of VR for remote supervisory and monitoring operations, particularly with a lack of research on satellite operations. The direct control paradigms that have been studied in depth may not be appropriate for all future operations, such as spaceflight, where time delay and bandwidth limitations inhibit direct control. Instead, autonomous systems are capable of performing many of the tasks needed while operators act in a supervisory control modality (Sheridan, 2021). Although there has been limited work assessing VR for monitoring, some of the benefits seen with direct control, such as improved collision avoidance, may still apply. However, other benefits may not be as applicable due to the differing control authority and cognitive demands on the operator. Manual control tasks tend to have higher workload (Hooey et al., 2018), which VR has been shown to reduce (Whitney et al., 2020b; a). Unlike direct control, monitoring tasks already have a low workload (Hooey et al., 2018) so VR might not reduce it further (Lager and Topp, 2019). However, it is still important to study VR for monitoring tasks, as successful monitoring relies heavily on an operator’s SA which VR may improve.

VR has been proposed for use in operations for many monitoring control rooms, such as for spaceflight (Sittner et al., 2023), maritime (Lager and Topp, 2019; Tsigkounis et al., 2021), and air traffic control (Cordeil et al., 2016; Gorbunov and Nechaev, 2022). While there has been research focusing on these monitoring applications using augmented reality (van den Oever et al., 2024; Torres et al., 2024; Rohacs et al., 2016; Reisman and Brown, 2006; Bagassi et al., 2024), or for training (Moesl et al., 2023), there have been limited experimental studies into VR for operations, particularly for spacecraft operations. Lager et al. compared the use of traditional 2D GUI, 3D screen-space GUI, and 3D VR GUI to remotely monitor autonomous surface vehicles and found that users were better able to detect collisions and had improved SA in both the 3D and VR displays compared to the 2D display. The 3D GUI had significantly reduced cognitive load (as measured through a proxy variable of a secondary task) compared to both the VR GUI and 2D GUI. However, participants subjectively felt that if they had many hours of training, the VR display would be best for the task, indicating that VR has the potential for monitoring applications (Lager and Topp, 2019). Other research has studied VR for maritime control room monitoring and has found that VR could replace complex monitoring dashboards (Tsigkounis et al., 2021). Although an experiment was conducted, no comparisons were made between VR and current maritime traditional displays, representing a gap in VR monitoring research and understanding how VR can facilitate improvements. A different study comparing VR to physical displays for the monitoring of autonomous cars found that VR increased task load and simulator sickness, and decreased usability (Kalamkar et al., 2023). However, the authors acknowledge that these differences are likely attributed to the hardware used (i.e., headset weight and resolution) and the fact that their VR display was not designed or optimized for use in VR, indicating that it is important to design with VR in mind. Finally, for air traffic control applications, VR has been demonstrated to reduce the number of errors and aid in identifying dangerous situations compared to the typical 2D view controllers see (Gorbunov and Nechaev, 2022), but no comparison was made to a 3D view, making it unclear if the benefits come from increased immersion or 3D visualizations.

While VR has promise as an operational display, there are also limitations present. Most of the current VR displays lack the resolution to display large blocks of text in a readable and understandable way, indicating that it is important to ensure text is readable before use. Interaction and selection techniques, such as controllers or raycast, can also be adjusted given design needs. However, using an inappropriate choice can result in selection errors, longer competition time, and decreased usability (Wentzel et al., 2024; Luong et al., 2023), resulting in a need to ensure the controls are appropriate. In addition, information overload and misinterpretation of the data are possible in a VR-based environment representation (Van de Merwe et al., 2019), especially with large chunks of data. As such, it is critical to verify that users are able to correctly interpret the data. Finally, operator buy in and susceptibility to cybersickness are influenced by display design choices, including viewpoint selection, field of view, amount of control over the environment, and headset properties (van Emmerik et al., 2011; Davis et al., 2014). Some of these limitations are due to hardware properties and are actively improving as technology progresses, but must still be considered to ensure current operational use and operator buy in. These limitations also necessitate subjective user evaluations of new displays to ensure the usability is not compromised.

The objective and contribution of this research is to compare the effects of 3D visualization and VR on a remote operator’s understanding of uncertainties using a specific application: monitoring of a satellite during operations. We focused only on monitoring tasks as it is how the operator spends the majority of their time during remote supervision and has critical implications towards being able to take appropriate actions. Throughout this work, the term visualization refers to the presentation of data in a simulated system representation, as opposed to plots of raw telemetry values. Immersion is defined as “the extent to which the computer displays are capable of delivering an inclusive, extensive, surrounding and vivid illusion of reality to the senses of a human participant” (Slater et al., 1996). In this work immersion is provided through head-mounted display VR. This work considers three displays with varying degrees of visualization and immersion, and their effects on SA, workload, usability, and subjective understanding of uncertainties. We hypothesize that 3D visualizations will improve SA, lower workload, and improve usability and subjective utility over displays without 3D visualizations. We further hypothesize that immersive displays, such as VR, will provide additional benefits over 3D visualizations.

In this research, three displays of increasing levels of visualization and immersion are designed and implemented to simulate the remote monitoring of spacecraft operations. The simulated remote monitoring task is a rendezvous mission scenario in which a servicer approaches a target vehicle, performs corrective burns, and changes its orientation to inspect the target. The three display designs are compared through a human subject evaluation.

A summary of the each metrics descriptive statistics can be seen in Table 2. Significant differences are seen between the display modalities for the 3 SA measures shown in Figure 3. There is a significant difference in the level 1 SA as measured by the subject average percent callouts made. The ANOVA comparing the linear mixed effect models found significance between conditions (F (2, 29.06) = 11.62, p < 0.005, η 2 = 0.44). Follow up pairwise comparisons with Tukey adjusted p-values found that the differences are between the VR and baseline display (t (29.1) = 4.76, p = 0.001, d = 2.04), and VR and screen visualization (t (29.1) = 2.95, p = 0.017, d = 1.26), but no differences between the baseline and screen visualization (t (29.1) = 2.95, p = 0.17, d = 0.78). Further analysis into level 1 SA compares the differences in display modalities across the different types of callouts, as seen in Figure 4. There are differences in modalities across the percent of satellite state callouts made. This includes the fuel and battery values, and is unique in the VR display as these values follow the satellite’s position and change location with time (F (2, 29.06) = 11.24, p < 0.005, η 2 = 0.44). Post-hoc pairwise comparisons with Tukey adjusted p-values found that the differences are between the VR and baseline display (t (29.1) = 4.66, p < 0.005, d = 2.00), and VR and screen visualization (t (29.1) = 2.97, p = 0.016, d = 1.27). However, there is not a significant difference in the percent of the callouts made correctly regarding the time until the next event (F (2,29.17) = 2.50, p = 0.10, η 2 = 0.15). These callouts are stationary in all three displays, as this information is contained in the HUD component of the VR display.

For both level 2 and 3 SA there is a significant difference between display modalities as measured by the subject averaged percent correct of the SPAM queries asked. For level 2 the ANOVA found a difference (F (2, 29.02) = 5.57, p = 0.0089, η 2 = 0.28). Post-hoc pairwise comparisons with Tukey adjusted p-values found that the differences are between the baseline display and screen visualization (t (29) = −3.13, p = 0.011, d = −1.34) and baseline display and VR display (t (29) = −2.56, p = 0.041, d = −1.10). Level 3 SA found similar results, with the ANOVA finding differences in display modalities (F (2, 30) = 4.90, p = 0.014, η 2 = 0.25), and post hoc comparisons finding the differences between the baseline display and screen visualization (t (29.3) = −2.81, p = 0.023, d = −1.21) and baseline display and VR display (t (29.3) = −2.52, p = 0.044, d = −1.08). The means and standard deviation for the different display modalities across the different SA levels can be seen in Figure 3.

No significant differences are found between the other measures collected. The comparison of workload between displays found no significant difference (F (2, 30) = 0.51, p = 0.61, η 2 = 0.03). For the system usability scale, no significance is found with the ANOVA (F (2, 30) = 0.97, p = 0.39, η 2 = 0.06). Finally, none of the utility questions yielded significant differences. However, the comparison of participants’ subjective understanding of the servicer uncertainty was nearly significant (H (2) = 5.35, p = 0.069, η 2 = 0.11). Trends in the data yield toward highest perceived utility for VR followed by screen visualization, followed by baseline. The underlying data for these measures is in Figure 5.

In sum, these results are in support of the hypothesis that 3D visualizations improve SA. They support the idea that 3D visualizations can improve level 2 and 3 SA, but not workload, usability, and utility. These results are contrary to the hypothesis that immersiveness will provide additional benefits over 3D visualizations.

This study is one of the first to investigate the use of VR for remote monitoring of spacecraft rendezvous operations. The objective measure of SA shows significant differences between display modalities, with 3D visualizations improving Level 2 and Level 3 SA, but with VR harming Level 1 SA. Contrary to our hypothesis, the subjective measures of workload, and usability, did not show statistical differences. Similarly, subjective assessment of utility did not reach statistical significance but trended toward higher evaluations for displays with visualizations and immersion. In sum, these results provide insight into the understudied area of the utility of 3D visualizations and VR for operators in a remote supervisory, rather than direct command, of autonomous systems.

3D Visualizations and VR impact levels 1, 2, and 3 SA differently. Improving SA is critical for improving performance and enabling appropriate decisions, and poor SA has been a contributor to many accidents or errors (Endsley, 1995). All three levels of SA are important, and typically build off each other, such that level 2 SA requires level 1 SA, and level 3 SA requires level 2 SA. However, for remote monitoring and supervision of unintuitive orbital systems, operators will need to have an appropriate level 3 SA to understand collision risk and project the consequences of avoidance maneuvers (Endsley, 1995), particularly under uncertainty. For satellite operations in particular this is especially critical as collisions can adversely effect the viability of space operations across all orbital regimes (Oltrogge and Vallado, 2019).

While the SA level 1 results indicate that VR led to significantly worse performance over the baseline and screen visualization displays, the difference was only derived from information that was not in a fixed location on the VR display. When comparing the SA callouts, further analysis found no differences in the display modalities for items that were always present in the same location. For the VR display, this includes information in the heads up display. However, there were significant differences in the analysis of items that are in a static location in the baseline and screen visualization display but are tied to specific objects and change location over time in the VR display. The dynamic motion of the satellite states in VR made it so these objects are less salient, and more effort is required to find them. Participants are thus unable to have a consistent scan pattern. Scan patterns are often described using the SEEV (Salience, effort, expectancy, and value) model. Effort and salience are important aspects of this model which has been shown to be predictive of level 1 SA (Wickens et al., 2005). Although it may be desired to have all the data in a glanceable HUD which can improve monitoring performance over other information displays (Lu et al., 2020), this also can block visualized information, increase clutter, and disrupt immersion (Iacovides et al., 2015; Rosyid et al., 2021; Caroux et al., 2023). While tying some information to the satellites may reduce level 1 SA, it still is an important design consideration to avoid some of the pitfalls of HUD, such as putting too much information into the HUD obscuring the visualization. These results imply that information most critical to the success of the mission or information that needs to be consistently monitored should be located in a stationary component of a VR display.

The SA level 2 and 3 results found that 3D visualizations lead to an improved performance over the baseline display without 3D visualizations, but found no differences between the screen visualization and the VR displays. These initial results indicate that in this monitoring task VR does not impact performance. This agrees with the results of a prior remote monitoring VR study (Lager and Topp, 2019), which used a proxy for SA. These results are also in agreement with a monitoring study that compared only 3D visualizations to 2D visualizations and found that the 3D visualizations increased SA (Rottermanner et al., 2020). Other studies found that VR improves SA, although they consider a direct control paradigm of interacting with robotic systems (Whitney et al., 2020a). The amount of control authority an operator has may be a contributing factor to these differences in results. There is a need for future work to consider other control paradigms that fall between direct control and monitoring, such as supervisory control.

The subjective measures of workload show no differences between display modalities. There are inconsistencies in previous literature as to whether VR increases (Lager and Topp, 2019) or decreases (Whitney et al., 2020a) workload over non-immersive displays. This research finds no differences. This may be due to the task and experimental paradigm itself: remote supervision, especially monitoring, is typically lower workload compared to direct control (Hooey et al., 2018; Huey and Wickens, 1993; Grier, 2015). Thus, it is not unexpected that most users experienced similar levels of workload, as they had no control authority. Additionally, participants using the baseline display could not customize their display, while users in screen visualization and VR display could. The effort towards customizing the display or finding appropriate camera viewpoints could inflate the workload of visualization-based displays relative to the baseline. Most of the previous tasks that have found workload differences have been for direct control, where the operators are interacting with a system either through VR or a computer display and there is typically a higher workload overall (Elor et al., 2021; Whitney et al., 2020a; Hooey et al., 2018; Huey and Wickens, 1993). Due to inconsistencies in the literature and the varied degrees of operator engagement, future work should investigate other degrees of control authority, like supervisory control. While an ideal display would decrease workload over alternative displays, these results may be considered positive in that they did not exacerbate workload, indicating overall good display design.

For usability and utility, no significant differences are found between display modalities. In prior work operators often subjectively rate VR displays to have a higher usability and prefer to work with them (Elor et al., 2021; Lager and Topp, 2019), or prefer 3D visualizations over 2D visualizations (Weiss et al., 2021). In this research, statistical significance is not achieved, but the utility results trend toward significance. As such, these results are consistent with that of the literature where users trended to subjectively prefer the utility of the VR display. A critical difference between our results and those in the literature is that many of these studies used a within subjects design where participants had a chance to experience multiple display modalities and thus their responses reflect these comparisons. By not doing a within subjects design, this study is unable to capture some of these subjective preferences. Like workload, no differences in usability may be positive, as having a significantly worse display may be more indicative of poor display design or issues due to limitations with VR technology.

A challenge of this research was implementing a VR display that leverages appropriate display design principles. Effort was made, through human factors evaluation, to ensure that results were not influenced by a participant’s inability to read the display or interpret and control the visualization effectively. Established principles for aerospace displays (such as MIL-SPEC and FAA regulations) and human factors (i.e., minimizing information access cost, contrast, and avoiding absolute judgment limitss) guided the layout of the VR display, but few VR-specific design principles are established, highlighting a need for further research into VR display design guidelines. The design guidelines applied here (such as the use of HUD and tying text to specific elements) and their impact on this study’s results can be extended by future designers of VR displays.

There are some limitations to this study. Our subject pool may not be representative of operators who would be trained in or use these types of systems and had varying backgrounds and familiarity with orbital mechanics and VR. While subject background knowledge is accounted for in our statistical analysis to reduce these effects, having a more representative subject pool would be ideal. Additionally, using participants with no prior familiarity with traditional displays makes it unclear how current, highly-trained operators would react to visualizations or a new system they are not as familiar with. Previous research for air traffic control found that while 3D visualizations improved SA among all participants, but those with extensive operational experience provided lower subjective ratings to 3D (Rottermanner et al., 2020). Future work should assess to see if the same is true for satellite operations, and if so how to best mitigate the issue of switching displays. In addition, the between-subject design used may impact the subjective measures as participants did not have a chance to experience all three displays and thus did not have the ability to compare between the features and limitations of each display.

While we monitored for symptoms of nausea during the experiment, we did not measure other cybersickness symptoms like eye strain or fatigue. These other symptoms may influence the outcomes of the metrics assessed in the study or may discourage the use of VR during future operations. Additionally, this research considers the monitoring aspect of remote supervisory control of a simplified, faster than real time satellite operation, in which participants had no control authority. This work focuses on the monitoring of a remote system, which is how operators in such systems will spend the majority of their time in supervisory control paradigms, and therefore foregoes the inclusion of the ability to intermittently provide input to the autonomous system. It is critical to understand how display modalities can impact monitoring performance. If a display fails to facilitate effective monitoring, it will be difficult to use in supervisory control. Future work could include the addition of opportunities for intermittent control or increased complexity in the monitored system. Finally, this research considers VR and immersion as applied through a head-mounted display. There are many other ways of providing an immersive environment, such as a CAVE system, and using a single display type represents a limitation of our work. Future research can study the impact of different degrees of immersion using different immersive systems on monitoring.

Future work involves expanding this experiment to consider remote supervision paradigms to understand how VR impacts these situations. This also more closely represents what operators might encounter during normal operations when remotely supervising autonomous or semi-autonomous agents. Understanding the impacts of display on monitoring is an important first step, as it is how the majority of an operator’s time is spent, providing the operator with limited control authority to make interventions while in supervisory mode may allow display differences to be seen in metrics of workload, usability, and utility. This will also fill in the gap of understanding the effect VR has on various degrees of control authority, as remote supervision is understudied. Additionally, while these results are specific towards satellite monitoring displays, these results may be applicable towards other systems that rely on remote monitoring of multiple systems under uncertainty such as air traffic control, submarine/maritime control rooms, and monitoring self driving cars. Future work should investigate the extent to which these conclusions transfer to other types of monitoring systems. Future work can also study changing the complexity or the pacing of the scenario used to understand if that changes the outcome. Furthermore, as VR display design principles continue to solidify, future work will continue to incorporate emerging best practices from the literature and human factors; these design decisions may impact results.

This study compares the effects of 3D visualizations and VR for remote monitoring of spacecraft operations on SA, workload, usability, and utility. Three displays, with varying degrees of 3D visualizations and immersion, were designed and evaluated through human subject testing. The results of this work indicate 3D visualizations may improve display interfaces for monitoring satellites; however, there is little evidence that VR such as that provided by VR, yields additional improvements. 3D visualizations improve level 2 and level 3 SA as measured through SPAM queries. VR reduces level 1 SA as measured through callouts; this reduction was only noticed when considering information that was not always present on the VR display. There are no differences between displays in workload, usability, and utility. While VR has been demonstrated to be a promising modality for direct control tasks, the benefits do not translate to remote monitoring of autonomous agents. Future work should assess the impact of VR and immersion in remote supervision of autonomous systems, where operators can make intermittent commands. These conclusions can also inform interface designs for future monitoring of spaceflight systems.