We present a workflow for developing HMD-VR exergames with immersive restorative environments for gait rehabilitation (see Figure 1). We selected real-world environments that meet quality criteria for both restoration and gait and balance training. For these selected real-world environments, we acquired 3D point cloud data and audio recordings, which we integrated into a game engine to create a coherent audiovisual VR environment. By manipulating the point cloud, we displayed or concealed objects selectively, which allowed for difficulty level customization and adaptation to everyday settings for therapy. Implementing redirect walking (RDW) enabled us to track user motion, effectively translating real-world movements into the virtual environment.

We applied our methodology to create three realistic immersive distinct HMD-VR exergame environments, each selected for their unique restorative qualities and alignment with our project’s objective of fostering restorative gait and balance training (Hartig et al., 2003; Hartig et al., 2014; Bowler et al., 2010; Ward Thompson, 2011; Bergeron et al., 2014; Hartig, 2021). There is strong evidence that the experience of the environment, especially the natural environment, can be beneficial to people’s health and wellbeing (Ward Thompson, 2011; Hartig et al., 2014; Hartig, 2021). Current evidence suggests that natural environments, i.e., those that are predominantly dominated by vegetation (e.g., gardens, parks, nature reserves or forests), are more valuable for restoring attention and reducing stress than non-vegetated urban environments (e.g., city streets or residential areas) (Hartig et al., 2003; Bowler et al., 2010). Other studies indicate that landscape preference is influenced more by structural parameters (e.g., complexity, openness, water sources) than by habitat type (Han, 2007; Ward Thompson, 2018). This can be connected to Ulrich’s qualities and components of a landscape, which promotes stress reduction as described in his theory of stress reduction (Ulrich et al., 1991). For example, landscapes should have moderate depth and complexity, as well as a focal point and natural content (vegetation or water) (Hartig et al., 2003).

Overall it has been shown, that there is a tendency to prefer forested landscapes over grassland (Han, 2007). However, it is noted that not only natural environments with lots of vegetation can have a positive effect on our health. Urban environments and familiar places evoke positive associations and memories, which in turn can impact wellbeing (Bergeron et al., 2014; Hartig et al., 2014; Hartig, 2021). This might not have a direct effect on recreation, but as meaningful content to remember the past, it can positively affect our wellbeing (Appel et al., 2020b). Moreover, it has been shown that emotional associations and attachments we form with a place, which are based on personal experiences such as the restorative effects of a favorite location, influence the value we attribute to a landscape and, consequently, our overall wellbeing (Eisenhauer et al., 2000). Expanding the restorative perspective in this way allows for a more comprehensive understanding of how different environments contribute to health and wellbeing, moving beyond traditional theories to encompass a wider range of phenomena, problems, and solutions (Hartig, 2021). To illustrate, one might consider the incremental impact of repeated restorative experiences on an individual’s connection to a specific place. Research demonstrates that people frequently form attachments to locations they regularly visit for restoration. These attachments can become a significant aspect of their identity. Over time, the emotional bonds with such a place might enhance its restorative qualities, creating a mutually reinforcing relationship between restorative experiences, place attachment, and personal identity (Korpela et al., 2001; Hartig, 2004).

Although landscapes are already utilized in healthcare, there remains limited research on patients’ preferences for different landscapes and their potential to enhance physical and mental recovery (Song et al., 2022). Therefore, we chose environments that encompass (1) an urban setting, replicating an attractive shopping area with outdoor cafés in an old Swiss town that is popular with national citizens and international tourists; (2) a rural nature reserve, offering psychological benefits like mental repose, revitalization, and the freedom of expansive surroundings (King et al., 2017); and (3) an open forest, a familiar and enriching gathering spot, as is often found in forests of the Swiss Plateau (see Figures 1A, 2). Each environment catered to the needs and preferences of our target audience, particularly the elderly, such as safety, aesthetics, naturalness, and opportunities for leisure and social engagement, offering unique opportunities for restorative experiences (Wen et al., 2018). Crucial for selecting environments for the prototype were practical considerations such as terrain complexity and spatial constraints. To ensure a seamless transition from physical to virtual space, environments with minimal slopes or steps were selected. Technical constraints, such as the need for an area with relatively low visual ranges of a few kilometres in all directions for smooth 3D point cloud-based visualization, were also considered to optimize the user experience and minimize potential discomfort. This limits the choice of suitable environments to smaller, contained areas.

The selection ensured comparability across environments based on restoration functions, following Kaplan’s ART criteria for the four components (see Table 1) (Kaplan, 1995). The component “Being Away” elucidates the sensation of emancipation and repose afforded by these environments. To illustrate, the rural setting provided tranquil soundscapes and interactions with distinctive flora, fostering a sense of tranquility. The concept of “Fascination” pertains to the manner in which the various elements within these settings—such as the vibrant café scenes in the urban environment, the prominent single tree surrounded by stones in the forest, or the platform with a small pond in the rural reserve—engage and captivate attention effortlessly. The term “Extent” is used to describe the perceived spaciousness of an environment. Even smaller areas, such as a cozy café terrace or the tree line on the horizon in the rural reserve, can evoke a sense of expansive, immersive space. The concept of “Compatibility” emphasizes how these environments align with human inclinations, making them more comfortable and enjoyable. This can be observed in the familiar layout of the urban area or the natural flow of the forest landscape (Kaplan, 1995).

Terrestrial Laser Scanning (TLS) offers a reliable method to scan environments for 3D visualization of real-world settings (see Figure 1B, left). Scanner type choice is flexible, prioritizing high-density point cloud generation and accurate point colorization with scene photos. Our study used the TLS RIEGL VZ-1000 with a maximum measurement range of 1400 m and a calibrated Nikon D700 camera for 3D point cloud colorization (RIEGL Laser Measurement Systems, 2023). For each location, multiple scan positions were chosen in a strategic manner to ensure comprehensive coverage and capture the entire landscape in high detail (number of scan positions per environment: urban = 25, rural = 31, forest = 28). This approach was necessary to obtain overlapping regions and to gather all essential spatial data with the required precision and at the required level of detail (Tang and Alaswad, 2012; Zhang et al., 2016).

The choice of location determines whether a short-range scan with a range of 450 m or a long-range measurement with a range of typically 1,200 m is conducted. A short-range scan lasts approx. 5 min, whereas a long-range scan takes approx. 20 min. To extend the spatial coverage for allowing long-range views, we integrated Airborne Laser Scanning (ALS) point clouds (see Figure 1B, middle). We used for this purpose 1 km² tiles of ALS data with an accuracy of 10 cm obtained from the Federal Office of Topograhy Swisstopo (Bundesamt für Landestopografie swisstopo, 2024).

The point cloud data underwent rigorous post-processing and refinement using a suite of specialized software tools, following methodologies outlined by Schalbetter et al. (2023) and Wissen Hayek et al. (2023) (see Figure 1C). The objective was to enhance data quality, eliminate noise, and prepare the point clouds for subsequent VR rendering. To ensure comprehensive coverage and minimize obstructions, at least two series of photos were taken for each scan location. This approach provided the images for colorizing the points and ensured that sections of the scanned environment were captured without cars or people obscuring the view.

The initial postprocessing step involved the use of Adobe Photoshop 25.6.0 (Adobe Systems Software Ireland Limited, 2024) to remove erroneous color information from the recorded images. The Layer Mask tool facilitated the seamless integration of portions of one image into the other when a moving object disrupted the initial shot. Furthermore, the Clone Stamp tool was employed to rectify color inconsistencies by duplicating colors from neighboring regions, particularly in areas that were covered several times (see Figure 3).

Subsequently, for further processing the point cloud scans we employed the RiSCAN Pro v2.14.1 (RIEGL Laser Measurement Systems, 2024) for Coarse Registration and Automatic Registration, aligning the overlapping scans to create a comprehensive representation of the entire scanned environment. With Automatic Registration the point clouds are shifted and rotated until the highest match between all scans is achieved. This automated workflow proved effective in the urban environment, as the building structure offers clear matching points. However, sufficiently flat surfaces and sharp edges are not given in landscapes with a high vegetation content. Therefore, Coarse Registration was used in the natural environments, where we searched individually in different scans for similar points and connected the clouds over them, with at least 4 connecting points per scan. This manual effort is relatively time-consuming, depending on the number of scan positions. Both registration methods resulted in an exact composition of all scans.

After registration of the individual scans, the Image Registration functionality enabled us to utilize the edited images for colorizing the point clouds for photorealistic representation of the environment. Then, with the Selection tool, we removed so-called noise points caused by reflective surfaces, such as windows, and other similar errors (see Figure 4). We additionally utilized CloudCompare v2.12.3 (CloudCompare, 2024) for tasks such as Scalar Field Filter by Value, Clone, Merge, and Segment. These functions were instrumental in eliminating superfluous points, such as those caused by moving vegetation. Further, gaps in the point clouds, e.g., due to objects blocking the laser beam, were filled by cloning and merging relevant sections of the point cloud. The Merge tool was further employed to combine the TLS and ALS point clouds. The rotation and transformation of the ALS data were adjusted based on the registration of the TLS data to align with the orientation of the TLS data. Subsequently, Lis Pro 3D v8.2.0 (Petrini-Monteferri, 2023) was utilized to Eliminate Isolated Points, thereby further reducing noise in the point cloud data.

The sequence in which these software tools were applied was meticulously selected based on their specific functionalities and interdependencies. For example, for eliminating noise points in CloudCompare with the Scalar Field Filter, the point cloud needed to be colorized with the images beforehand to receive the noise points that are colored white.

A significant challenge in dynamic HMD-VR is smoothly displaying voluminous point clouds with millions of points (see Table 2 for point cloud sizes used in our project). When displaying such complex scenes, the frame rate can often drop when the computation time for each frame exceeds the refresh rate of the device. In turn, this is negatively impacting the user’s experience and comfort.

For real-time rendering, we stored point clouds using the potree octree structure (Schütz, 2024) and rendered them in a game engine. We employed Unity v2022.2.13f1 (Unity Technologies, 2024c) and imported the point clouds into a 3D Unity project using Fraiss’ Point Cloud Shader (Fraiss, 2017). This setup allowed for dynamic loading of point clouds with their full resolution from local storage into Unity, keeping the project size manageable (see Table 2). Potree adapts the modifiable nested octree structure, saving the original point cloud in node subsamples (Scheiblauer, 2014; Schütz, 2016). The rendering process is dynamic, tracking the camera’s orientation and streaming only the necessary point cloud data within the field of view (Fraiss, 2017) (see Figure 1D). The chosen rendering technique successfully maintains a stable frame rate of 90 Hz, which is optimal for the HTC Vive Pro Eye.

In general, the auralization of environments in VR can be done using recordings taken at specific locations, or acoustic environments can be designed from scratch by compiling different sounds from specific sources such as cars, church bells, birds, crickets etc. (Lindquist, 2014; Pieren, 2018; Ferreira, 2021). In our methodology, we prioritized fidelity, choosing to use auditory data recorded in-situ from the actual environment (Lindquist, 2014; Pieren, 2018), which was then edited for refinement.

Geolocated recordings were acquired by first dividing the environment into acoustically separate locations (forest, pasture, pond, church, café, etc.) (see Figure 1B, right). Second, we captured the sounds of these locations in the field utilizing an ambisonic sound-field microphone (Soundfield MKII Portable (SoundField, 2024)). We chose the ambisonic recording technique, which results in a four-channel signal that can be transcoded into various output formats for use in any playback setup, as it is well suited for interactive reproduction of spatial sound (Hong et al., 2017). Third, for postprocessing the ambisonic audio recordings, we used the digital audio workstation ProTools (Avid Technology, 2024), synthesizing the coherent sound sequences by cutting out unwanted sound (e.g., airplanes, humans passing, etc.) for each of the recording positions in their distinct location. In a fourth step, we integrated these sound sequences into Unity by positioning them in the VR environment at the respective position that corresponds to the recording position in the real world. Finally, we used Steam Audio (Valve Corporation, 2024) to decode the 4-channel audio of the ambisonic sequences to a binaural signal that is spatialized according to the head movement (Valve Corporation, 2017; Broderick et al., 2018). The head-tracked binaural audio reproduction technique allows the playback of spatial sounds with a pair of loudspeakers such as headphones on an HMD (Hong et al., 2017). This technique has been proven to enable soundscape representations with high ecological validity (Hong et al., 2019).

Some adjustments to the audio source settings in Steam Audio were necessary to ensure optimal integration within the VR environment. Specifically, the sound files were configured to play in a continuous loop and to activate automatically upon the application’s start, ensuring seamless and uninterrupted audio playback throughout the session. To enable the audio to adapt dynamically to user movements, the spatialize option was selected, and the audio was blended in 3D spatial mode. The audio spread was set to 360°, allowing the sound to be perceived from all directions. Additionally, the minimum and maximum audible distances were carefully adjusted to correspond to the characteristics of the sound recordings. These settings were customized for each project to ensure the most accurate and immersive auditory experience. By moving through the scene, the sound sequences fade in and out providing an immersive auditory environment.

Point clouds provide a flexible means for manipulating virtual environments by adding or deleting points (Urech et al., 2020; 2022). This method enables the removal of elements that could detract from the restorative quality of the environment, such as parked cars in an urban setting. These removed point clouds of objects can be replaced, e.g., by segmented and cloned portions of the remaining point cloud using CloudCompare (see Figure 1E).

To enhance the training complexity in the HMD-VR exergame and support exercises such as dodging and leg lifts, we incorporated point clouds that could be toggled visible or invisible. This required scanning of common objects that fit naturally within the landscape, such as tree trunks for the forest and skateboards for the urban environment. Tree trunks, for example, were selected and exported from the processed point cloud of the forest landscape. To increase the quality and point density, we cloned and slightly transformed the selected point cloud subset multiple times.

When suitable objects were not found in the original point clouds, alternative methods were employed to acquire them. One approach involved using Polycam–LiDAR and 3D Scanner for iPhone and Android (Polycam, 2024), a cost-effective tool for scanning smaller 3D objects (Inal et al., 2023) (see Figure 5; left). The resulting scan was exported as *.laz file, imported into CloudCompare, where relevant parts of the scan were cut out, and then exported. Another approach was to download 3D objects (*.obj) from online sources and import them into CloudCompare. Using the “Sample Points” tool, a point cloud was automatically generated from these objects, with point density adjustments made based on the object’s size.

The resulting point clouds, whether derived from the original point cloud, Polycam scans or transformed 3D objects, were exported from CloudCompare as *.ply files (binary) and then imported into Unity and converted into game objects using the Pcx plugin (Takahashi, 2019). We chose this plugin over the potree octree structure because it simplifies the process of placing and using point clouds as therapeutic objects in Unity.

In our work, we also utilized point cloud modification to spatially reposition entire sections of the environment. For instance, a slightly elevated platform could be adjusted to the same level as the surrounding ground, ensuring consistency between the virtual and physical gradients, e.g., a flat ground without steps in a training room. For this propose the virtual object (here: the elevated platform) had to be selected and then translated in the vertical direction to meet the ground level (see Figure 5; right).

Utilizing point clouds for visualization in HMD-VR applications enables users to explore and interact with high-fidelity virtual environments in a natural and engaging manner, despite potential constraints in physical space (Nilsson et al., 2018). SteamVR™ Tracking (Valve Software, 2024a) provides an easy solution for free VR walking by tracking popular HMDs in 3D, easily integrated into Unity with the SteamVR Plugin (Unity Technologies, 2024b). We used the HTC VIVE Pro Eye headset (HTC Corporation, 2011b) with a wireless adapter (HTC Corporation, 2011c) and two VIVE SteamVR Base Stations 2.0 (HTC Corporation, 2011a), which track the HMD in the room (orientation, speed) using trigonometry (Valve Software, 2024b) (see Figure 1F left icon). Eliminating the need for tethered connections using the wireless adapter allows for greater freedom of movement, improving the overall user experience. Wireless VR headsets can minimize disruptions, and optimize user engagement and safety (Gonçalves et al., 2020).

The point cloud data imported into Unity are no game objects and, hence, no collider can be added to the points representing the ground surface. The collider is required to allow walking on the virtual ground. Therefore, we generated a digital surface model (DSM) to allow unimpeded traversal across the scanned point cloud terrain. This was achieved utilizing Esri’s ArcGIS v2.9.1 (ESRI, 2024) LAS Dataset To Raster, which is capable of directly interpolating the DSM from the point cloud files in the common LAS format. Depending on the environment and the point cloud itself, the interpolation type should be adjusted (e.g., binning, cell assignment: minimum/average, fill method: linear). The selection of the most appropriate interpolation type was an exploratory process, whereby several options were evaluated to identify the one that best matched the point cloud surface. In this regard, the specific landscape conditions to be accurately represented such as steps or slopes were of critical importance, which is why the interpolation type varied across different landscapes.

The resulting raster files (*.png) with an resolution of 25 cm were then imported into Unity through the Terrain Tools plugin (Makkonen, 2024). To guarantee that the raster is imported in the appropriate dimensions, it was essential to ascertain the maximum height discrepancy and the dimensions of the raster in ArcGIS. These values have to be entered as input in Unity for defining the extent and the extrusion of the raster cells. Subsequent minor adjustments were manually applied within the terrain tools as needed (e.g., set height, smooth height, raise or lower terrain).

The terrain model enabled the implementation of a game object equipped with a rigidbody component used to control the motion of an object by Unity’s physics engine. The rigidbody allows the virtual user’s game object (capsule) to react to movements in real-time, with the reactions being subject to gravity, preventing the virtual user from passing through or floating above the terrain. Working in tandem with the rigidbody, a capsule collider detects collisions between the virtual user and the terrain, facilitating realistic movement of the virtual user within the environment. A streaming camera, which is a child of the capsule, adapts its height based on the position data received from the VR headset in the real environment. This setup ensured that head movements, such as bending down, were accurately mimicked in the virtual world, thereby enhancing the realism of the VR experience (Wang et al., 2010) (see Figure 1F middle icon).

Additionally, a Redirect Walking (RDW) library, namely, OpenRDW (Li Y.-J. et al., 2021), was integrated into the Unity-based point cloud visualization framework. OpenRDW facilitates users to walk along virtual pathways, diverging slightly from their actual physical trajectories in the real environment (Razzaque, 2005; Li Y.-J. et al., 2021) (see Figure 1F right icon) while still providing high levels of immersion (Usoh et al., 1999; Lutfallah et al., 2024). Manipulation of the relationship between real and virtual motion is achieved by applying gains, including translation, rotation and curvature gains (Azmandian et al., 2016) which users do not notice since they primarily rely on their visual sense for orientation (Lutfallah et al., 2024). These gains create a subtle mismatch between the user’s perceived and actual movements, allowing for seamless redirection in the virtual environment. Each gain adjusts different aspects of motion, such as the user’s rotation, walking speed, or trajectory, while keeping changes within thresholds to ensure that the manipulations remain imperceptible (Steinicke et al., 2010; Williams and Peck, 2019; Lutfallah et al., 2024). This method is used to alter the user’s movements in the virtual environment, allowing redirection away from physical boundaries (Azmandian et al., 2016; Lutfallah et al., 2024). This feature lets users explore larger virtual environments beyond the 4 × 4 m² confines of the physical tracked space in our setup.

For the prototype of the HMD-VR exergame, we incorporated Unity’s default skybox with an unspectacular cloudless sky to establish a consistent natural ambiance across all environments. This choice represents a usual day conducive to outdoor recreation (Manyoky, 2015), providing a cost-effective way to achieve a convincing visual depiction of the sky (Raguman et al., 2019). Furthermore, we added a script to enable the navigation by jumping to predefined locations within the environment using get-key-down-commands (Unity Technologies, 2024a). The same commands were also used to manipulate the visibility of various training objects on the ground, such as skateboards or tree trunks (see Section 2.5). The final Unity scenes were compiled into a game application compatible with various platforms, including PC, Mac, and Linux, as well as VR headsets.

We configured our system with the following computational specifications: NVIDIA GeForce RTX 3090, Samsung SSD 980 PRO 2TB, 128 GB Memory, and 12th Gen Intel^® Core™ i9-12900K CPU. Additionally, we utilized the Vive Pro Eye HMD, featuring a Dual OLED 2.5″ diagonal screen with a resolution of 1440 × 1600 pixels per eye and a refresh rate of 90 Hz, and including integrated Hi-Res stereo headphones (HTC Corporation, 2011b).

The VR exergame’s tasks were developed in close collaboration with physiotherapists to align with rehabilitation goals, particularly for gait security training. These tasks simulate real-world navigation and balance challenges that are common in Swiss hospital rehabilitation programs and are selected based on activities of daily living commonly performed by the elderly, as outlined in the Activities-Specific Balance Confidence (ABC) Scale (Powell and Myers, 1995). Examples include: “Walk to the restaurant (and read its name)”, “Walk to the large stone and take a seat” (with a real chair for sitting), “Bend down to have a closer look at something on the ground”, and “Walk beside the field (and count the purple flowers).” The exercises, including those in brackets, are dual-task exercises designed to increase difficulty by requiring users to divide their attention between walking and performing a secondary cognitive activity, thereby increasing both physical and cognitive engagement (Tasseel-Ponche et al., 2023). Additional tasks can be activated through modified point cloud elements, including: “Step over all the skateboards and scooters, starting with the left/right foot,” “Weave through obstacles in a slalom,” and “Side-step over tree trunks.” These exercises encourage participants to walk frequently, reinforcing gait security by improving coordination (Chiu and Chou, 2013), spatial awareness (Osoba et al., 2019), and step control (Okubo et al., 2017). Each task was selected to enhance users’ mobility and balance.

Three distinct HMD-VR applications were successfully developed, each designed to replicate high-fidelity real-world environments using 3D point clouds from Terrestrial Laser Scanning (TLS) and Airborne Laser Scanning (ALS) data. These applications leveraged an advanced rendering method to ensure both high performance and visual fidelity. The applications utilized an advanced rendering method that ensured high performance and visual fidelity. The dynamic loading of point clouds facilitated smooth operation, even with large data sets, maintaining a stable frame rate of 90 Hz, which aligns with the graphics capabilities of the headset. No issues related to cyber sickness were reported by users, with none complaining of dizziness or other discomforts. Additionally, when asked about difficulties during the semi-structured interviews, several users reported no difficulties, noting that the movements within the virtual environment closely mirrored their real-world counterparts, contributing to a more natural and comfortable experience.

The auralization was effectively integrated, employing geolocated ambisonic sound recordings to create an immersive auditory environment. This environment was designed to dynamically adapt to users’ movements, aiming to enhance the sense of presence and engagement. Participant feedback gathered though the session observations indicated that this adaptive auditory environment contributed to a heightened sense of immersion. Several users specifically reported a noticeably calming effect of the natural sound, and appreciated the relaxing character of these auditory stimuli such as bird twittering or the rippling of water. Participants described specifically the rural environment’s sound atmosphere as calming. In contrast, the urban soundscape encouraged exploration and observation, with examples like searching for a church when hearing bells ring or being drawn to a bustling café after catching snippets of conversation. Overall, the participants in our study expressed appreciation for the soundscape with some variations in perception of its congruency depending on the environment. For instance, in the forest setting, participants often noted the sound of water sprinkling as a dominant feature, even when its source was not always visible. Additionally, some participants found the virtual environment visually quiet but perceived the ambient sound as noisy. In particular, participants in the urban setting criticized that the sounds of people engaged in conversation were particularly prominent, despite only a few individuals being visible within the VR scene.

The SteamVR™ Tracking system, combined with Unity’s rigidbody functionalities, enabled natural movement and terrain traversal within the VR environment (see Figure 6, or supplementary video). Redirect Walking (RDW), implemented via the OpenRDW library, allowed users to explore larger virtual spaces beyond the constraints of their physical area. Nevertheless, the activation of the redirected mechanism necessitates a certain degree of balance, as the resulting gains are still perceptible. In our clinical experience with patients exhibiting instability in their gait and preexisting balance issues, we found the RDW function unsuitable for this population. Due to their existing balance challenges, using RDW could have potentially introduced instability. Instead, we relied on get-key-down-inputs for changing location, which allowed participants to explore sufficient portions of the virtual landscape without the need for physical displacement. This method provided a safer and more controlled experience, eliminating the need for the RDW feature while maintaining an immersive and engaging environment. Additionally, the combination of simple, user-friendly tracking methods requires no technical expertise from either the patient or the physiotherapist, making it an ideal solution for gait rehabilitation in older adults.

The user experience regarding restoration and its subscales, measured using the PRS, was assessed in 49 sessions across urban (20), forest (20) and rural (9) landscapes, where participants engaged in 25 min of gait and balance training using the HMD-VR exergame. The analysis revealed that all environments fostered restoration (see Figure 7).

The ‘Being Away’ component, a measure of immersion, approached high value across all three virtual environments, indicating deep engagement [28.30 ± 2.91, (min 0 – max 30)]. The highest values were reached in the forest environment (28.80 ± 1.51), followed closely by the urban (27.78 ± 4.28) and rural (28.22 ± 1.79) environments. Participants frequently reported that they forgot they were in a hospital during the session, describing the experience as a welcome break from their daily routine and responsibilities.

The ‘Fascination’ component also recorded elevated scores in all environments, indicating that the environment effectively captured the participants’ attention [41.02 ± 5.61, (min 0 – max 48)]. The urban environment scored the highest with 42.39 ± 7.23, followed by the forest (40.35 ± 4.68) and rural (39.78 ± 3.31) environments. Distinctive landmarks such as fountains, ponds, and individual trees, helped participants orient themselves, as reported in the interviews and observed behaviors such as actively searching for these elements before performing the given task. Key features, like the church in the urban environment, the central tree in the stone circle in the forest, and in the rural environment the farm buildings (see Figure 2) served as reference points when participants navigated the virtual space or when the location in the virtual environment changed on get-key-down-inputs.

The “Coherence” and “Compatibility” components, which measure the comprehensibility of the environment and its support of participants’ needs, also demonstrated strong results [Coherence: 21.07 ± 2.58, (min 0 – max 24); Compatibility: 47.45 ± 6.09, (min 0 – max 54)]. In the Coherence scale, the urban and the forest environment almost scored the same (21.47 ± 3.41 vs. 21.25 ± 1.94). Also, in the Compatibility scale, the urban (48.89 ± 6.79) and forest (47.35 ± 4.78) environments both showed higher values than the rural environment (44.78 ± 6.96). None of the differences between the environment groups in the different components were significantly different from each other.

The incorporation of dynamic point cloud manipulation features, such as the selective visibility of elements like skateboards and tree trunks, facilitated tailored simulations for various gait rehabilitation exercises like stepping over or around obstacles and navigating figure-eight patterns. For instance, when a patient felt secure in walking within the VR environment, stepping over obstacles could be introduced as a more challenging task. The ability to provide diverse exercise scenarios not only motivated participants but also facilitated repeated practice in varied contexts (see Figure 8). Users appreciated the varied training setups in the same VR environment made possible by these modifications, finding them engaging and beneficial. Patients mentioned, for example, that they would feel insecure stepping over obstacles in the real world and appreciated the opportunity to practice this skill in the VR environment. The integration of get-key-down-inputs for changing location allowed therapists to quickly change the user’s position within the virtual environment. Adding obstacles with this function enabled rapid adjustments to different training settings, providing a flexible and responsive training experience tailored to individual needs. For instance, some participants said that they did not feel able to step over obstacles in the first sessions so that they were switched off again, whereas in later sessions they had no problems with this task and the obstacles could be used as training elements.

The aim was to establish a workflow for developing realistic environments that accurately mimic real-world settings using 3D geospatial data and motion tracking. Our prototypes allow users to navigate immersive environments and engage in tailored therapeutic activities. Dynamic point cloud manipulation in our prototypes presents users with challenges mirroring real-world obstacles in a safe environment (Dockx et al., 2016; Tasseel-Ponche et al., 2023). Our approach bridges the gap between traditional therapy and immersive virtual environments by accurately replicating real environments, facilitating easier skill transfer to everyday tasks (Montalbán and Arrogante, 2020). For instance, the elderly might navigate through a city, searching for restaurants and sitting down on a chair, or explore a forest, identifying the best path while avoiding tree roots. Future enhancements could be directed towards improving the adaptability and personalization of the exercises, such as enabling additional dual-task exercises or incorporating moving objects for dodging. However, the utilization of static 3D point clouds restricts the incorporation of moving objects, which could potentially enhance the realism of the environment and further facilitate the simulation of real-life tasks. Currently, approaches are being developed to coherently integrate animated point clouds for augmented reality, promising great opportunities for immersive experiences in virtual reality (Piazzolla et al., 2023).

Nevertheless, the generation of such realistic virtual reality environments is a time-consuming process. For instance, the capture of 20–30 point cloud scans for each environment requires approximately 10 h, exclusive of the planning phase. The subsequent postprocessing steps in RiSCAN Pro, CloudCompare, and Lis Pro 3D demand an additional 70 h per environment. These times were achieved by experienced users, although familiarity with the process does not significantly reduce the time and effort required due to the manual nature of these tasks. Additionally, the postprocessing time of images and the cleaning of point clouds are heavily influenced by moving objects; larger crowds and traffic can considerably extend the postprocessing duration. However, a well-planned process, where scanner locations are strategically chosen to minimize the number of scans (Jia and Lichti, 2022) while concurrently ensuring the inclusion of overlapping regions and the acquisition of all essential spatial data with the requisite levels of precision and granularity, can markedly diminish the workload (Tang and Alaswad, 2012; Zhang et al., 2016). In general, it takes about 40 h per landscape to integrate the processed data into Unity, depending on the amount of terrain modification required and the programmer’s familiarity with Unity’s features and dependencies.

Our system enhances realism and engagement by synchronizing sound with user interactions, which is known to contribute to the therapeutic effect by stress reduction (Lindquist et al., 2020). However, challenges remain in representing the sound of certain environments perceived as congruent in all aspects. Particularly, discrepancies between visual calm and auditory noise was identified as a potential disruptor to the immersive experience. Nevertheless, the user feedback underscores the potential of audiovisual HMD-VR exergames in providing immersive experiences during gait and balance training. This is highlighting the importance of further research and development in the field of the acoustic environment representation in VR to provide guidance for producing more realistic and congruent soundscapes.

Our exploration of RDW was motivated by the need to navigate longer distances in VR, despite limitations in real-world space (Schalbetter et al., 2022). Therefore, we adopted the RDW approach, which presents challenges associated with RDW-induced cybersickness (Rothacher, 2019). While RDW facilitates natural movement in confined spaces, it can also cause sensory conflicts and balance issues, especially in vulnerable populations, such as elderly participants in the exergame (Schmitz et al., 2018). Further advancements in RDW or better adjustments of its parameters are necessary to enable navigation through environments with reduced cybersickness risk, currently achievable only through SteamVR™ Tracking and additional manual keyboard inputs. However, this method does not allow for the expansion of environmental scope, calling for further research in motion translation. In the meantime, using get-key-down-inputs for changing locations is a viable solution, as it proved very suitable in the training sessions conducted.

Preliminary results from this ongoing study suggest that all three virtual environments (urban, forest, rural) promote restoration, as indicated by the PRS scale components. The ‘Being Away’ component, reflecting immersion, was high across all environments, with the forest environment showing the strongest engagement. This suggests that natural settings, which allow users to escape daily routines, provide greater immersion (Kaplan, 1995; Hartig et al., 1997b). Our results, consistent with previous studies (Han, 2007), show higher immersion in forested landscapes compared to rural settings, supporting the idea that forests are often favored for their restorative qualities.

However, in line with earlier research highlighting the positive effects of urban environments (Bergeron et al., 2014; Hartig et al., 2014; Hartig, 2021), the “Fascination” component—measuring the environment’s ability to capture attention—was highest in the urban environment. This is likely due to the variety of visual stimulus, such as historic buildings and store windows, that naturally draw attention. The “Coherence” component, which measures how well the environment’s elements are related and fit together, was similarly high in all three environments. These settings provided a strong sense of connectedness through their well-organized structures, whether it was the city’s historic buildings, the forest’s natural features, or the overview over the open rural landscape.

Finally, the “Compatibility” component, assessing how well the environment supports the participant’s intended activities, was also high in both the urban and forest environments. These settings aligned well with participants’ goals, whether it was strolling through the city or exploring nature, suggesting that both urban and natural environments can facilitate restorative activities in ways that enhance users’ experiences (Hartig et al., 1997b).