To test and implement a workflow for obtaining a virtual environment of a coastscape, we selected the case study site presented by Karstens et al. (2022). The study site, Stein beach, which is situated in northern Germany in the outer part of the Kiel Fjord (Baltic Sea), accommodates a diverse range of vegetation ( Figure 1 ). The seaward wetland edge is largely dominated by common reed (Phragmites australis) with a few patches of salt marsh bulrushes (Bolboschoenus maritimus), followed by dune vegetation in the sandy areas (e.g. Ammophila arenaria, Ammophila x baltica, Leymus arenarius). The swash margin is dominated by annual vegetation such as Cakile maritima, Atriplex littoralis and Atriplex prostrata. Accommodation space is limited as the vegetated area at the study site is bordered by a dike in the hinterland and a marina to the east. As a result of the ongoing shore-parallel sediment transport, most of the bays between Kiel and Fehmarn island are currently being cut off by spit formation. Bottsand, which extends to the west and has been advancing since 1880 (Niedermeyer et al., 2011), is the youngest spit; however, regular dredging at the marina impacts natural sediment transport and coastal dynamics at the study site.

Transitions from wet habitats that are suitable for plants like Phragmites australis, Bolboschoenus maritimus or Schoenoplectus tabernaemontani to dry habitats with dune vegetation occur across small timescales, such that climate change impacts, e.g. sea level rise or increased wave action during winter (Ahola et al., 2021), will influence vegetation composition and pattern distribution. Bolboschoenus maritimus has a higher resistance to salinity than Phragmites australis, and thus might outcompete reed at the wetland edge in the near future. This would have a significant impact on the coastscape because Phragmites australis, which grows up to >2m in height, is much larger and impacts the visual perception (“shielding”) significantly more than Bolboschoenus maritimus. In this study, we created two virtual representations of the study area: (i) as it is today, with Phragmites australis as the dominant species; and (ii) with Bolboschoenus maritimus replacing Phragmites australis at the wetland edge as it might in higher salinity conditions in the future.

Our workflow can be easily adapted to suit any geomorphometric data input. Nevertheless, a digital elevation model (DEM), together with segmentation and classification maps for the different terrain types, is required for further processing in EPIC Unreal Engine 5. The images must be converted to the same resolution, cover the same area and be in PNG format. For the best visualization of geomorphology, the size in cm² that is covered by a pixel and the total height covered by the DEM should be known ( Figure 2 ). Resolution of these images depends on flight height as well as the sensors used for the process, and can hence vary significantly from mm-scale to dm-scale. While airborne UAS can produce orthophotos in the mm- to cm-scale, the resolution of data from shipboard multibeam-echosounders may range from some to several tens of meters. The level of detail of the DEM, and the segmentation and classification maps defines the lateral appearance of the displayed landscape material and assets. To prevent visual steps related to pixel boundaries, we recommend the use of images with the highest possible resolution. The application of a Gaussian filter to interpolate pixel boundaries also helps to smooth lateral landscape limits. A Python-based tool that enables the modification of the DEM, segmentation and classification maps can be accessed from our GitHub repository (see “Data availability” chapter).

DEMs were generated for our study area during UAS surveys with a RGB camera in 2021/2022 (see Karstens et al., 2022). RGB imagery on the sub-decimeter scale was conducted with a DJI ZenmuseX5S RGB camera mounted on a DJI Inspire II UAS at a flight height of 70 m, resulting in a lateral resolution of 2 cm. Orthophotos and digital elevation models were generated based on structure-from-motion photogrammetry with the open-source software WebODM (Version 1.9.11, OpenDroneMap, 2022, see Mokrane et al., 2019; Vacca, 2020), that uses the structure-from-motion software library OpenSfM (OpenSfM, 2022) in combination with the Multi-View-Stereo (MVS, 2022) technique (Vacca, 2020). The geo-referenced point cloud data were used for the processing of DEMs with an inverse distance weighting interpolation method (Vacca, 2020; Karstens et al., 2022). Segmentation and classification maps were created from the orthophotos using the open-source Orfeo Toolbox (OTB Version 6.0, Grizonnet et al., 2017, see Karstens et al., 2022). The data were segmented with a spatial range of 50 and a radial range of 7, and supervised machine learning was carried out using a support vector machine (see Karstens et al., 2022). Classes for segmentation included, inter alia, “Phragmites australis”, “Other vegetation”, “Sand”, and “Water” ( Figure 2 ).

In this section, we describe our developed workflow for obtaining asset-based virtual landscapes from airborne RGB imagery ( Figure 3 ). For this study, we use Unreal Engine Blueprints (see “Data availability” chapter for the GitHub link). This workflow can easily be adapted to other sensor data, e.g. spaceborne and multispectral photography, or hydroacoustic remote sensing techniques, such as multi-beam echo-sounder (e.g. Backscatter) data. As mentioned above, the prerequisites are a DEM and segmentation/classification maps with the same lateral boundaries. The segmentation and classification maps can be derived from any geomorphometric parameter in the terrestrial and/or the marine realm. The acquired data need to be validated in the field to define the best fitting landscape materials and assets that should be used in the virtual instance of the scene. Within the blueprint, these landscape materials and assets can be replaced by any custom or purchased 3D model.

Game engines, such as EPIC Unreal Engine 5 (UE5), allow landscape materials to be generated and then draped on digital elevation models or any object within the game engine project. The setup of landscape material within the game engine (in our case UE5) is crucial for obtaining an asset-based virtual instance of original aerial imagery or any other geospatial dataset. The UE5 Blueprint presented in this study is built using a modular setup, so that customization is simple. The classification of the landscape materials is the basis for this workflow, and must be carried out before importing the DEM via the UE5 Landscape Editor.

The developed workflow highlights the possibilities for improving the appearance and perception of virtual geomorphologic landscapes, and can be considered as a blueprint for the generation of an asset-based virtual environment. In Figure 6 , a textured mesh is juxtaposed with the generated asset-based virtual environment. The major advantage of an asset-based virtual environment like this is that it provides a more “realistic” scenic view that is more easily recognized and perceived compared to the coastal landscape of the survey area. This will continue to hold true if the virtual environment is further developed to include additional scenarios (e.g. storm surges) or manipulations of the working area. The automatic spawning of vegetation is key to building this virtual instance, as precise imaging of vegetation is a challenge in both photogrammetry and structure-from-motion algorithms (Cunliffe et al., 2016).

The level of detail in the virtual environment, which is based on digital elevation models and classification maps, can be increased by manually placing additional assets (e.g. habitat matching fauna, which is not derivable from terrestrial and/or marine remote sensing). Furthermore, meteorically-induced water level fluctuations, which are high and not infrequent along the Baltic coast, can easily be integrated into the scene by adding water surfaces where necessary. The virtual coastscape allows stakeholders to experience places along the coast that are rarely accessible for them (e.g. large reed stands, which are protected by law and too dense to walk through). Furthermore, (potential) environmental changes can be discovered individually, enabling a more focused debate about possible implications between experts and policy makers, as well as with the general public. Virtual environments thus have great potential to aid dialogue-driven research, such as in transdisciplinary approaches, and in the design and establishment of living-labs. At our case study site, changes in water level and salinity could lead to a switch from Phragmites australis dominated wetland edges to Bolboschoenus maritimus ( Figure 7 ). The resulting decrease in vegetation height would shape the coastal visage and its perception by stakeholders. The view from land towards the sea would improve, but refuge and “hiding” options, which are used not only by fauna but also by beach visitors, would decrease. Several man-made footpaths are also present at the study site at Stein ( Figure 1 ), and modifications to the virtual environment help to demonstrate how changes in these anthropogenic structures would impact the vegetation patterns.

Remote sensing results, such as photogrammetry data from UAS RGB camera surveys or LIDAR data, can be quickly translated into immersive experiences for a variety of landscapes (e.g. Reinoso-Gordo et al., 2020; Rienow et al., 2020; Huang et al., 2021). A virtual environment is thus a powerful tool for visualizing and discussing localized scenarios with different stakeholders, which is particularly important in the dynamic and ever-changing coastal zone. Our asset-based approach allows us to create immersive virtual environments that can easily be modified into different coastscapes. Asset choices are large and often available for free or at low cost (e.g. www.unrealengine.com/marketplace; https://quixel.com/megascans/); yet, whenever a particular asset cannot be externally sourced (e.g. rare plant species, or special textures of rock and sediments), the work flow to create it is more labor-intensive compared to automated photogrammetry, and requires 3D modeling and texturing skills.

UAS surveys with RGB cameras have become popular in the coastal zone, where sediment dynamics (e.g. Albuquerque et al., 2018; Jayson-Quashigah et al., 2019; Robin et al., 2020), dune monitoring (e.g. Scarelli et al., 2017; van Puijenbroek et al., 2017; Laporte-Fauret et al., 2020), and litter detection (e.g. Bao et al., 2018; Bak et al., 2019; Andriolo et al., 2022) have become major themes in recent years. Integrating such UAS datasets with an RGB sensor into our UE blueprint virtual environment would not only enable a better understanding of the environmental dynamics or pollution issues, but also their communication with a non-scientific audience. Scientific communication continues to evolve (Hurd, 2000), and immersive environments are becoming more important; not only in scientific, but also in educational domains (Rubio-Tamayo et al., 2017). The use of immersive virtual environments will help to foster inclusion in geoscience education, making study areas more accessible for students who may not be able to go there in person (Harknett et al., 2022).

Climate change will impact vegetation patterns and species distribution in coastal areas. Our workflow allows users to create a simple virtual representation of a coastscape, where changes in species composition (e.g. Phragmites australis vs Bolboschoenus maritimus) can be performed easily for chosen segments. UAS surveys with RGB cameras have previously been applied for mapping invasive species in vegetated coastal areas (e.g. Samiappan et al., 2017; Zhu et al., 2019; Marzialetti et al., 2021), as well as for monitoring the spatial and temporal variability of vegetation patterns (e.g. Tóth, 2018; Doughty et al., 2021). These datasets can easily be processed to fit our workflow and allow an immersive display of environmental challenges that affect humankind. At our study site, where increased salinity might lead to Bolboschoenus maritimus outcompeting Phragmites australis, the visual perception of the wetland edge would completely change. Tall-growing reed currently forms a visual shield, which would disappear should B. maritimus, which is smaller and less dense, dominate. In order to fully understand and discuss potential environmental changes and their implications, stakeholders need to be an integral part of the research process from the beginning. An asset-based approach, such as that presented in this study, allows researchers to co-create and co-design virtual reality environments together with stakeholders by replacing assets or choosing different asset designs.

Using original research data, such as DEMs and segmentation/classification maps, to generate asset-based virtual environments has several advantages as demonstrated by our case study. In particular, resolution may be much higher than from publicly available data, and segmentation and classification may be more detailed, as the processing of the input data and geomorphometric analysis is carried out by the same person. Nevertheless, the use of original datasets results in comparatively small virtual environments, whose spatial extents are limited to the original dataset and are not comparable to “open world” scenes, in which a user can navigate through large landscapes. Future work will involve blending these specific scenes with open world scenes.

The workflow and blueprint presented in this study enable the semi-automatic generation of data-based virtual representations of coastscapes. We see great advantages to this simplified yet more immersive representation, as users outside academia may be better approached by a scene populated with familiar objects rather than with textured meshes and point clouds. Even more critically, assets and asset-based vegetation can also respond to in-game physics, displaying wind motion or being manipulated by a character within the virtual environment (e.g. Imbert et al., 2013). We consider our approach to be a baseline for future development of virtual environments using user-collected datasets for transdisciplinary research and decision making, dissemination and outreach. We also see a great field of educational purposes, as on the one hand, the visualization of geomorphometric analyses can be brought into the classroom in the field of Eath sciences, and on the other hand disciplines like social- and human sciences can use such virtual environments to assess other dimension of the visualized scenarios. For this purpose, we implemented in the function of “sea-level rise” into the Stand-Alone versions.

Moreover, the methodological approach and blueprint presented here enables any geomorphometric data to be used as the input for a scene. The created virtual environment does not necessarily need to image a vegetated beach scene, but could be a deep-sea habitat, geomorphological structure, or any in situ derived land to sea (L2S) environment.

In the context of digital twins and a metaverse, we emphasize that our blueprint does not currently meet the standards of a digital twin, as the described workflow has limitations including grain size, sediment distribution, and the loss or over-printing of small-scale features during the segmentation and classification process of the original orthophotos. In addition, the physical properties essential to a digital twin (Qi et al., 2021) are not included at this point, and would need to be implemented in a secondary step. The employment of this blueprint in a metaverse nonetheless presents many advantages, as the virtual representation of the study area displays an appropriate scene for further implementation of long-onset scenarios, including sea-level rise, and short-onset extreme events, such as storms, flooding, and droughts. Moreover, we see great potential for the implementation of L2S scenes, where submarine habitats can be added to the virtual environment. This could act as a basis for a fully accessible virtual environment of a coastscape. To make the scene as accessible as possible, we developed two different instances of the scene: a stand-alone version for the use on a desktop PC and a virtual reality version ready to be used on HMD or dome theatres.