The method of optical hand tracking is a type of VR input device that scans full hand motion, including natural mapping, in the full range of motion and motor control. Optical hand-tracking devices capture the movement of hands with cameras, either applied to HMD or as local sensors connected to a PC. As no device is connected to the user’s hand, no haptic feedback is included (Fahmi et al., 2020). Lower-cost optical hand-tracking devices are available as VR gloves, but these devices come with some limitations that must be considered. Optical tracking can only be conducted when the camera can see the hands. Tracking outside the camera field of view is not included. Additionally, precision, tolerance (e.g., towards light inference), limited data capture, and operating system compatibility are issues with optical hand-tracking devices that impede the technology (Gamboa et al., 2019). These limitations make the devices problematic in fast changing environments, as, for instance, when needed for the replication of manufacturing tasks (see implications for practice). This gave a further note of the importance of choosing a VR glove over an optical system in an operational use case (see Section 2.2.1). The use case resulted from the initial research gap of lacking qualitative studies on natural mapping with VR gloves.

To deliver an extensive picture between naturally mapped devices, differences between input devices must be outlined. The opposite of an abstracted movement with a controller is a naturally mapped movement (McEwan et al., 2014). Steuer (1992) defines natural mapping as the “ability of a system to map its controls to changes in the mediated environment in a natural and predictable manner.” The perception of naturalness is based on the ability to complete the mental models people use for real-world activities (Tomborini and Skalski, 2006). Mental models are internal representations of the external world that allow individuals to anticipate and predict the outcomes of their actions. McEwan et al. (2014) give an insightful note on natural mapping: “Naturally mapped control interfaces (NMCIs) for video games involve interactions with less abstraction between the task to be virtually achieved and the action required to achieve it” (McEwan et al., 2014). Hence, a VR glove can be considered closer to a realistic haptic interaction due to higher natural mapping capabilities than other VR input devices. Caeiro-Rodriguez et al. (2021) noted three ways in which commercial VR gloves map human actions. I) Hand and finger pose estimation motion tracking (degree of freedom), II) tactile feedback, and III) kinesthetic feedback. Caeiro-Rodriguez et al. (2021) depicted that a wide variety of active movements and natural mapping influences the sense of agency. Agency refers to the experience of control in one’s own actions (motor control), which contributes to perceived naturalness (McEwan et al., 2014).

Skalski et al. (2011) differentiated in a typology between four different mapping types: Directional natural mapping refers to devices that produce a correspondence between a direction used with a control and the actual movement of a body part. Devices that inherit directional natural mapping are, for instance, video game controllers. Kinesic natural mapping involves body movements that correspond to real-life actions. No tangible stimulation is included in this type of mapping. Optical hand-tracking devices fall into this category. Incomplete natural mapping is included in devices that partly map the “feel” of an object. However, the tangibility is not fully able to reproduce real-life sensations. Realistic natural mapping is the highest level of natural mapping, according to Skalski et al. (2011). It provides a realistic, tangible element to achieve the highest level of natural mapping. Considering a hand movement, including grabbing and touching, as natural tangible elements, a VR glove can be considered a realistic, naturally mapped device for hand operations. The motion capabilities of a human hand are displayed in Figure 1.

At this point, the gap between kinesic and realistic natural mapping for hand movements is not constantly given. According to Biocca (1997), close mapping of body movements and tangible stimulation is needed to address mental models for real-life behavior. Backed on this definition, Skalski et al. (2011) investigated a car steering wheel as a control device. This brings direct features of a clear mental model in relationship to driving into the simulation. This allows classifying the steering wheel as a realistic natural controller. The definition of a VR glove as a realistic, naturally mapped device is, therefore, also related to the mental model that it fulfills.

Perceived naturalness with controllers can lead to perceived presence (Skalski et al., 2011). Presence is a feeling of being virtually immersed in the media. It can be classified as the response of the human brain to the system (Berkman and Akan, 2019). In the investigation of Skalski et al. (2011) on natural mapping, perceived presence is directly influenced by perceived naturalness with natural mapped controllers. According to Tamborini and Bowman (2010), increased accessibility of mental models is expected to enhance spatial presence with naturally mapped devices (Tamborini and Bowman, 2010). Spatial presence is a type of presence that concerns the ability of a device to create a mediated experience to be immersed in a virtual environment or game space. (Tomborini and Skalski, 2006). Consequently, spatial presence is primarily of concern for this investigation of VR gloves.

If the input device creates a strong perceived presence, the user may also experience a situation in which they accept the virtual hand model as their own. This is a state of embodiment, as the virtual hand embodies the hand of the user (Gonzalez-Franco and Peck, 2018). Input devices in VR are mainly displayed as hand models (Caeiro-Rodríguez et al., 2021). For instance, the HTC Vive controller represents a model of a hand that is shown at the location of the VR input device. When the user pushes a button, they can observe how the virtual hand is grabbing an object in VR. Embodiment is also highlighted as a strong level of self-presence in the scientific literature and is strongly interrelated to cognitive absorption in VR (Schultze, 2010; Gonzalez-Franco and Peck, 2018). As the VR glove is a device that aims to mimic hand movements in VR, a state of embodiment should be reached where a user feels fully embodied with the hand model. We assume that the users have perceived embodiment with the hand model due to mapped controls.

Haptic feedback can affect presence with VR input devices and lead to tangible natural mapping (Skalski et al., 2011). The objective of haptic feedback in VR is to increase the user’s sense of reality (Gibbs et al., 2022). Haptic feedback in VR allows users to perceive the physical properties of virtual objects (e.g., weight and motion patterns) (Wang et al., 2022). Monteiro et al. (2020) highlighted a distinct dependency between new input device types and perceived presence in new interactions (Monteiro et al., 2020). On the other hand, a lack of haptic sensations can disorientate the users’ experience of presence (Wang et al., 2022). In current research on VR gloves, distinctions are made between certain aspects of a system. Gibbs et al. (2022) provided an extensive overview of the effect on presence between visuals and haptics of a VR system. In this research, we deliver a primary investigation on motor control and navigation aspects related to natural mapping (McEwan et al., 2014; Moon et al., 2023). Haptic feedback is included with VR controllers but is not of primary concern for the study of natural mapping movement, as we orient on the natural mapping definition from Steuer (1992), which defines natural mapping primarily as a feature of the system to map its controls to changes in the mediated environment (see Section 2.2.1). Haptic feedback is a complex field of study and implies a set of different technologies (Caeiro-Rodríguez et al., 2021) that must be examined in specialized investigations. Furthermore, the haptic feedback of current VR gloves does not realistically map real-life haptic processes. The human hand creates haptics in a complex interplay of multimodal stimulations (Kim et al., 2017). To this point, no commercial solutions can realistically map haptic feedback as in reality (Kim et al., 2017). Haptic tangible feedback is multimodal and involves a variety of senses. The interplay between haptics, changing features of the grabbed or touched object, sounds, etc., is highly complex (Gibbs et al., 2022) and would exceed the scope of this study with a focus on control mechanisms on naturally mapped VR controllers and VR input devices after Steuer (1992).

To this point, the connection of naturally mapped VR gloves on presence is outlined as a direct effect of the system. However, presence does not describe the perception of the task that is operated with a VR glove. To fully understand naturally mapped interactivity, a perspective on the operated task is necessary. Deep involvement in a task is described as cognitive absorption (CA) (Ryan et al., 2015). This is related to the trait absorption on its own, the engagement of the user in VR, and the state of flow that is experienced with the system (Agarwal and Karahanna, 2000). Cognitive absorption represents the extent to which a user is absorbed when using a system (Agarwal and Karahanna, 2000). Flow can be understood as a self-motivated state of immersion that results from an optimal balance between the challenge and skill with a system (Csikszentmihalyi, 1993). Therefore, this research gives an extensive overview of the presence response to the immersive capabilities of the VR glove. CA is additionally a precondition for learning, as it is described in the field of learning as a state in which an individual is completely involved in a task (Csikszentmihalyi, 1990). Based on this literature, we assume that natural mapping with VR gloves can lead to a high level of cognitive absorption into a task.

The scope of interest for this study is outlined toward this point. The goal of this research is to learn if a VR glove, due to natural mapping, automatically has stronger effects on presence and how this affects cognitive absorption and embodiment in the VR environment. The research question is accordingly:

The study aims

1) To analyze participant perceptions while using a VR glove compared to two other input devices on lower mapping levels.

The entire VR scenario was programmed with the Unreal Engine 4 Game Engine on a PC with 32 GB RAM, an i7 12th Generation-12700K processor, and a GeForce RTX 3080 Ti Graphics card for use with an HTC Vive Pro Eye VR headset. The HMD included an FOV of 110° and a frame rate of 90 Hz. The headset was tracked in the experiment through two HTC Vive 2.0 base stations that were attached in the corners of the laboratory. The HTC Vive was additionally equipped with a wireless adapter toolkit so that no direct cable connection to the computer was needed. For the experiment, three input devices on different natural mapping levels were chosen and are displayed in Figure 2.

The HTC Vive controller, the first device, is an input device with the lowest level of natural mapping. The HTC Vive controller is displayed in the VR as a virtual hand model that represents the hand of the user. The controller inherits directional natural mapping through applied buttons for grabbing and abstracts the grabbing motion through the directional pushing of a button with the index finger, which triggers a grabbing animation of the hand model. The positioning is accurately tracked in the VR through the two external sensors in the room that capture the position in the real world and set the controller model at the same position in VR. This creates partly realistic movement natural mapping by representing the basic hand movements. The HTC Vive controller delivers a full range of motion in the arm movement (three DoF) but no more complex finger movements. Haptic feedback is given in the HTC Vive controller through tactile vibration (Fahmi et al., 2020).

The second device, the index controller, partly abstracts grabbing physically with buttons for the index finger and thumb but also captures the movements of the middle, ring finger, and pinkie finger through sensors. The grabbing can be triggered by bending the fingers through flexion in contact with a virtual object. In contrast to the HTC Vive controller, the index controller mimics the finger movements through motion tracking but cannot completely map the finger movements of the user. For that reason, the index controller is considered here as an incomplete mapping input device. The index controller reaches six DoF in the hand movements and three DoF in the arm movements. Haptic feedback was included in the experiment with the VR controllers. The standard controller functions for Unreal Engine 4 were used for the HTC Vive and the index controller. The last of the chosen devices is the Noitom Hi5 VR glove, a fabric VR glove that allows almost full hand movement and real-time position tracking in VR. Each finger movement of the hand except for the side movements (abduction) was directly captured 1:1 from the real world into the VR while grabbing. In total, the Notom Hi5 glove supports 19 DoF in the hand; the real human hand has 23 DoF, as displayed in Table 1. However, as side movements are not necessarily needed for everyday operations (Skalski et al., 2011), the Hi5 glove comes close to the finger movement and mapping capabilities of the human hand. No tactile vibration feedback was provided for the grabbing due to limitations of the basic version of the Noitom Hi5-glove in Unreal Engine 4.

VR gloves are still highly complex and costly devices. However, in training fields, including surgery training, education, and industrial security training, VR gloves bring new possibilities to simulate critical situations (Caeiro-Rodriguez et al., 2021). Considering the current application of VR gloves and naturally mapped devices in the field, a construction scenario in an industrial context with a perspective on the transferability of practice was chosen (see the implication section for the practice). The Noitom Hi5 glove was selected for this investigation due to its strong motion-capturing capabilities and its light fiber design. A lightweight device was selected so as to not affect the movement and motion range capabilities of the system and allow optimal movement-mapping conditions. Secondary for the experiment design was good affordability, low latency, and a solid integration into Unreal Engine 4. The selection of the VR glove was made on the basis of a systematic analysis of VR gloves by Caeiro-Rodriguez et al. (2021). A scenario was selected for the experimental case where the participants practiced assembling an industrial medium-sized crane in VR. The VR scenario for investigating immersion effects was designed with a focus on the implication for practice (see Section 4.2). At the time of this study, VR gloves are in an experimental state and not an affordable option for the private consumer market (Caeiro-Rodríguez et al., 2021). Current customers for VR gloves are mainly companies that use VR gloves to display their processes in VR (Kim et al., 2017).

The VR level was designed as a plain white room to reduce distractions and allow full concentration on the operation with the input devices. The participant must assemble a crane from five crane parts. When the construction process with one device was finished, the construction procedure started again with the subsequent device in order. The order of the input devices switched randomly to avoid bias on the devices as the user already knew more about the construction process with each assembly round. The qualitative research process was divided into an experimental part and an interview part. The investigation took 1 hour.

Each participant received a short introduction to the procedure and the context of the investigation. To that, a short definition of immersion was given to explain the field of interest. After the introduction with the narrative, the participant was given a sheet that showed the finished constructed crane in the real world and the crane model with text labels to explain each construction part. The participant needed to remember the arrangement of the parts and construct the crane in VR from the crane parts displayed in the picture. The preparation of the experiment took an average of 15 min. After the interview, the participants were thanked for their attendance and asked for possible improvements. An interview took, on average, for 25–30 min. The five crane parts were placed in the VR level of the white room. A ghost model of the crane was presented next to the parts in the VR environment to represent where the crane should be assembled in the scenario (see Figure 3). The ghost model consisted of five blue transparent boxes, one box for one crane part. The user was instructed to take the parts and find their corresponding box in the ghost model. Here, the user should remember the position from the construction manual that was presented beforehand. When the user found the right location box and overlapped the part, the box changed color to green and showed a ghost model of the crane part. The VR controllers provided haptic vibration feedback during the grabbing activities. When the user released the crane part, it snapped to the equivalent position in the ghost model. The standard input mappings for the HTC Vive controller and the index controller were integrated into Unreal Engine 4. For the Noitom Hi5 glove, the grabbing mechanism was developed manually, as no input mapping pattern for the movements was provided by the manufacturer. The grabbing trigger with the VR glove was programmed with the Noitom Hi5 software development kit (SDK), which inherited special functions for data retrieval. The grabbing was equivalent to the movement of the fingertips to the hand base. An average between the distances of the fingers was counted. When the average distance between the fingers and the hand base was less than a predetermined threshold, the crane parts snapped to the hand model when the user operated a grabbing pose. The crane parts were released when the fingers opened, and the distance exceeded the threshold again.

The experiment took 20 min on average. After the experiment, the attendants were again asked for their permission on the audio recording. Then, they were interviewed according to the interview guidelines. In the interview, the participants replied to a general entry question to share their thoughts on immersion so they could express their impressions without being influenced by the interviewer. Then, more detailed categorical questions were asked. The interviewer could dynamically ask open follow-up questions on a category if needed to get more details on an insight. After sampling, a final question asked the users to give their overall impression of the devices and explain what they would need to be immersed with an input device.

Regarding natural mapping, the perceived freedom in the scenario was indicated. “I think the glove simulates grabbing in real life the best because you move your hand freely” (FG4). Most people perceived a gap between holding a device in their hand for interaction and moving their own hand in VR that limited the mapped activity: “I would say it does not affect my task with holding the controller in the hand, but it did not feel like really free. You had always to grab something. And that was a bit, I would say, a little bit limiting my task” (FG1). Regarding the index controller, the controls were criticized for not matching actual hand movements, which was seen as artificial.

In terms of realism, 13 of 20 participants claimed the glove was, in general, a realistic device. The normal controller, in comparison, was seen in three cases as realistic but also criticized in four cases as unrealistic. As an indication of the perception of the realism of the HTC Vive controller, the capability of young people to abstract experiences, and the decreasing importance of realism in VR and video game experiences, one participant noted:

When using the glove, 60% of the participants perceived to be more engaged in the scenario. In comparison, four participants claimed the HTC Vive controller was engaging. Two people rated the index controller as an engaging device in the scenario. The capability of the glove to be seamlessly in the scenario was valued by most participants: “With the glove, I was really focused on the crane, on the scenario. With the other devices, I had to focus more on the devices instead of the scenario” (FG2). Most participants noticed a thinking effort with the other devices that kept them from diving into the experience:

The analysis indicated that the VR glove created strong associations to natural hand movements compared to other state-of-the-art VR input devices. The prediction that the natural mapping motion capture of the VR glove would be perceived as more natural than VR controllers was highly supported (Skalski et al., 2011). However, flexibility does not directly imply a natural perception. The index controller allowed a similar range of movements as the VR glove in the experiment but could not reach a natural perception in most cases. According to the participants, the mapping must be in line with the expectations of the users to create a perception of naturalness and realism (Skalski et al., 2011). In contrast, a device with higher movement capabilities but weak mapping (the index controller) was perceived as less natural than a normal controller. In addition, participants highlighted a correlation between a stronger thinking effort of “translating” movements. This presents that a simpler device, such as the HTC Vive controller, can be perceived as natural and more realistic than a device with plenty of movement capabilities if the thinking effort with the complex device is too high. The VR glove, in contrast, shows that activation of existing movement patterns from mental models can work to create a naturally perceived interaction (McGloin & Krcmar, 2011).

The results with the devices indicate that the VR glove allowed the participants to be involved and engaged in a task, which could, according to Agarwal and Karahanna (2000), be related to increased CA. Jahn et al. (2018) highlight that CA can influence attitudes toward an intention to use a system that can positively influence learning outcomes. Kampling (2018) extensively outlines the importance of CA in VR learning environments due to CA, presence, and embodiment (Kampling, 2018). Nevertheless, further research would be needed to isolate a relationship between CA and learning for VR gloves.

The results indicate that the participants experienced different levels of spatial presence with the input devices. The perceived embodiment is a reliable indicator that perceived spatial presence due to a naturally mapped device can change the perception of the hand as a body part in VR (Gonzalez-Franco and Peck, 2018). The fact that the VR glove in this early development stage, without haptic feedback, can induce a strong perception of embodiment indicates the potential that can be unlocked in this field with further technological advancements.