The immersive VR environment features on the floor a visual representation of the European countries as 3D extruded polygons (see Figure 2F). Based on the generated data for 39 locations (see Section 3.1), 39 individual 3D Radar Charts are placed in the VR environment, each at the center of its associated country. Each 3D Radar Chart features five color coded, semitransparent data variable axes, one for each plant type, representing the data in that location (see Figure 2A). Thus, a total of 29,250 data values are displayed in the immersive VR environment. ⁸ Based on a room-scale two-by-two meter area, the user, wearing an HMD with a 3D gestural input device attached, can walk around to investigate and interact with those charts in close proximity. The 3D gestural input (see Figures 2C, 3) allows for the implementation of various (hand) interaction techniques in the VR environment, in our case a mixture of hand-based grasping, indirect widget, and bimanual (gestural command) metaphors (LaViola et al., 2017, Chapters 7, 8, and 9).

To move closer to charts in the virtual environment that are placed “beyond” the physical real world limitations, a target-based travel mechanism has been implemented using a mixture of gaze-based input and gestural command: By simply looking around, the user can center their gaze on one of the several 3D Radar Charts in the VR environment, which will prompt a visual outline for user feedback, at which point the user can then make a hand posture to point towards the chart (index finger extended, all others not extended) in a “I want to go there”-like motion (see Figure 3A), initiating a translation of the user’s virtual position to the center of the chart. Additionally to this spatial exploration, making general observations to get an overview and potentially identifying interesting visual patterns, the user can also engage into more active contextual interaction with an individual 3D Radar Chart, displaying details-on-demand in regard to the temporal data variables (Shneiderman, 1996). While the user is engaged into such details-on-demand investigation with an individual 3D Radar Chart, the gaze-and-point target-based travel mechanics are inactive. In order to be able to move again to “far away” charts, the user is required to first disengage and deactivate the details-on-demand state of a 3D Radar Chart.

Each chart features above it a minimalistic sphere as an Activation Toggle that the user hand simply touch to iterate through three states: Activate/Rotate, Reconfigure/Filter, and Deactivate (see Figure 2B). Once activated, the chart will display its Time Slice, a 2D mesh representing the classical radar chart pattern, integrated to connect the values of the data variable axes in order to represent the currently selected time event (see Figure 2C). Using 3D gestural input, the user can grab the Time Slice to move it up and down in order to make selections forward and backward in time, automatically adjusting the Time Slice’s mesh to represent the values for the updated time event (see Figure 3B). The integrated Time Slice aims to facilitate data interpretation and visual pattern detection when investigating the time-series data in more detail (Reski et al., 2020a). An Information Window is anchored and displayed next to the Time Slice (see Figure 2D), providing additional numerical information about the selected time event, by presenting a traditional radar chart visualization with annotated numerical values, names, and color coding for the selected time event as well as an outlined radar that represents the averages for each data variable across the displayed time-series. By using a two-hand pinch technique (index finger and thumb touching in each hand respectively) close to the chart, the user can select a time range from the time-series to focus on (see Figure 3C). The pinching allows for a “live sculpting” of the desired time range, which is visually highlighted by removing the color from the data variable segments that are not included in the selection. By keeping the colorless semi-transparent segments outside the time range selection visible, the user is still able to perceive a preview of the time-series, maintaining information accordingly – another change compared to our initial version (Reski et al., 2020a). It is noteworthy that the outlined radar chart as part of the Information Window, representing the displayed averages across all data variables, updates to the applied time range selection. The visualization also features a Rotation Handle, allowing for convenient rotation of the 3D Radar Chart in place (see Figure 2B). The Reconfigure/Filter Handle features color coded spheres that are placed above each of the data variable axes and connected to the time axis origin. By grabbing the individual spheres, the user can manipulate the angular position of the linked data variable axis and thus reconfigure (sort) the radial arrangement (see Figure 3D). Additionally, the user can also grab and pull each individual sphere away far enough from the time axis origin until its visual connection “snaps”, effectively removing the linked data variable axis, i.e., filter out undesired data (see Figure 3E). Finally, by making a two-hand index finger cross posture, the user is able to reset the state of the entire 3D Radar Chart, displaying all available data variable axes as well as automatically selecting the entire available time-series range (see Figure 3F).

The right part of the interface features a Map View, displaying outlined the individual countries across Europe (see Figure 4A). An interactive node is placed at the center of each country, allowing the user to left-click and select the corresponding location accordingly. It is noteworthy that each node is placed at the exact same position as the individual 3D Radar Charts in the immersive VR environment (see Section 3.2). Once a location has been selected, indicated through a colored node outline in the Map View, it updates the Climate View on the left part of the interface, composed of two line graphs, each representing one of the climate data variables (sunlight and humidity) for that country (see Figures 4B,C). Each line graph’s horizontal axis encodes time, while the vertical one encodes the data value. By hovering over a line graph, a vertical dashed Preview Line provides some additional visual feedback in regard to the hovered time event (see Figure 4E). The user can select the hovered time event via left-click. With a single time event selected, the user can also select a time range through a combination of holding the COMMAND-key and left-click, effectively spanning a continuous interval along the time-series data from the first time event selection to the new one, indicated through a visual overlay (see Figure 4D). Time event and range selections can be updated by simply making new time selections in the interface, replacing the prior ones accordingly. Furthermore, time event and range selections are synchronized across the two line graphs, i.e., making a selection in one line graph will automatically display the same selection in the respective other one.

The following collaborative information cues from the immersive VR environment are displayed in the non-immersive desktop terminal. The Map View features an added node that represents in real-time the position and field-of-view, i.e., orientation, of the VR user, allowing the desktop user to have an understanding of the VR user’s location in space (see Figure 5A, right), The location node, representing the 3D Radar Chart that the VR user is potentially actively interacting with (details-on-demand) is outlined accordingly in the Map View as well, indicating the VR user’s current engagement with it. If both collaborators are interacting with the same location, the Climate View features a vertical dashed line representing the VR user’s current time selection, i.e., the position of the Time Slice (see Figure 5B, right). Similarly, if the VR user applies a time range selection, an overlay in both of the line graphs is visualized in the Climate View accordingly (see Figure 5C, right). All interface elements representing information cues of the VR user are color coded differently to easily discern them from the desktop user.

The other way around, the following collaborative information cues from the non-immersive desktop terminal are displayed in the immersive VR environment. Location selections made in the Map View will temporally highlight the corresponding country, extruded in 3D on the floor in the VR environment, in a different color, assuming the VR user is not already actively interacting with the 3D Radar Chart at that location (see Figure 5A, left). Time event and time range selections made in any of the two line graphs of the Climate View will be represented as virtual annotations in the corresponding 3D Radar Chart (see Figures 5B,C, left). More specifically, the selected data values across all data variables in 3D will be highlighted, aiming to catch the VR user’s attention as well as, in case of a time range selection, indirectly providing information about the amount of selected time events (“resolution”). Furthermore, a virtual symbol in the form a magnifying glass, aligned in space with the respective time selection, provides an additional cue to catch the VR user’s attention in the immersive VR environment, figuratively indicating that the desktop user is currently “investigating in this time context”.

The study was set up in a controlled environment at our research group lab. The lab features a square two-by-two meter area, designated for the VR user to move freely without obstacles. There is enough space in the lab for the researcher’s workplace, from which the study moderation was conducted. Additionally, the lab features a workbench that is divided using a physical partition from the researcher’s workplace, allowing the VR user to complete their informed consent and questionnaires with pen and paper. The user operating the web application was seated alone in a dedicated separate office that featured a workbench with a computer, a 27-inch monitor, keyboard and mouse, as well as enough space to write down notes and complete the informed consent and questionnaires with pen and paper. Detailed information about hardware and software are described in Section 3.5. Additional remarks in general as well as in regard to the ongoing COVID-19 pandemic during the time of the study are stated in the article’s Ethics Statement.

The pair of participants used different interfaces (hybrid); the choice of interface additionally affected which part of the dataset they had access to, and therefore determined their role (asymmetric). Using the non-immersive interface provided access only to the climate data within the desktop application, while using the immersive interface provided access only to the plant data within the virtual environment. This made the users “climate” and “plant” experts respectively; these roles would flip with the switching of the interfaces for the second study task (within-subject design).

The collaborative nature of the tasks required them to work together to determine the correlations of each of the climate parameters to each of the plant parameters; the non-immersive user was additionally tasked to write down the answers, and any worthwhile observations, for both participants (using printed sheets that were provided; these are included in this article’s Supplementary Material).

Due to the artificial nature of the datasets, each of the pair’s answers could be checked against the model used to generate them, making the nature of this study a confirmative analysis (Aigner et al., 2011, Chapter 1.1) compared to the explorative analysis of our previous study (Reski et al., 2020b).

The choice of context (climate and flora at different locations across Europe, and how the climate conditions affect the plant growth) was chosen as concepts familiar to all participants. At the same time, any previous knowledge of geography and agriculture had to be dismissed, therefore the study was presented with a “science fictional” description to the participants that had to suspend their disbelief and pretend that they were exploring a parallel universe in the far future instead of working with real observations that followed known phenomena. The detailed task description, as it was presented to participants in our user interaction study, is included in the article’s Supplementary Material.

Each study session followed the same procedure of three stages: 1) introduction, 2) fruits task scenario, and 3) veggies task scenario. The overall duration was aimed at approximately 2 hours, including all three stages. The initial choice of which participant used which interface was random. For the two task stages, the participants were encouraged to explore the data and complete their task at their own pace. However, for practical purposes, the pair was given a duration of approximately 30 min to aim for and to have a frame of reference in regard to their task completion progress. Whether the pair required more or less time, was up to them. Consequently, each participant was anticipated to spent approximately 35 min (5 min warm-up; 30 min task) immersed in VR.

In the introduction, the participants were first welcomed and then asked to fill out an informed user consent in regard to their participation. Afterwards, demographic information about the participants’ background and prior virtual reality experiences were inquired. The moderator provided an overview about the two applications and their collaborative features, as well as about the data context and the task for their upcoming joint data explorations, i.e., the first and second task stages.

For the first task, each participant of the pair assumed their role and respective application. Using a special warm-up dataset, different from each of the two task scenario datasets, the participants were provided with the opportunity to warm-up and become familiar with their interfaces and the collaborative features. Once the pair felt comfortable, the moderator loaded the task scenario dataset and issued the start for the pair’s task completion by initiating the audio recording. During the tasks, the pair could only talk to each other, while the moderator refrained from making any comments to the pair, only writing down noteworthy observations. Once the pair considered themselves to be done with their task by speaking aloud “We are done with the data exploration” (or equivalent), the moderator stopped the audio recording. The participants were then asked to complete three questionnaires (in order): System Usability Scale (SUS), User Engagement Scale - Short Form (UES-SF), and our Spatio-Temporal Collaboration Questionnaire (see Section 4.2.3).

After a short break in which the moderator made several preparations, the participants switched their assumed roles and applications, and the second task stage started by following the same procedure as in the first one (warm-up, task, questionnaires). Finally, the pair was thanked for their participation and sent off. If they inquired about their task performance, they were informed after the study completion.

We collected system logs of all the participants’ interactions with their respective interfaces during the task sessions. Each log entry consists of a timestamp (in seconds), a user identifier, and multiple fields that describe the various contextual interactions in detail, e.g., a movement to a specific location or the selection of a specific time event or time range. Since all interactions in the non-immersive interface are communicated to the immersive interface via real-time communication interface, we decided to log the interactions for both users conveniently unified in one place. The outcome is a CSV file that can be easily processed according to our interests. For instance, an analysis can be conducted to identify when the collaborators were actively investigating the data in the same spatial location, or in regard to when each collaborator moved from location to location, to name just two examples. Within the scope of the presented collaborative system, we implemented the system logging for both interfaces as a separate lightweight module that is integrated as part of the VR application (see Section 3.5).

Based on the designed data analysis task as described in Section 4.1.2, we are able to assess the pair’s ability to collaboratively identify the potential correlations between the plant and the climate dimensions using their respective interfaces. For each task, this results in a total of ten correlation answers, i.e., five plant data dimensions × two climate data dimensions, each indicating either a negative, positive, or no correlation. While the option to answer no correlation was provided, a correlation was always defined by the models used in the fruits and veggies scenarios. Additionally, each of the ten correlation answers included an associated confidence (low, medium, high, or do not know), describing the pair’s reported confidence for their respective answers.

To make assessments about the general usability of each interface, the immersive VR and the non-immersive desktop one, we asked the participants to complete the System Usability Scale (SUS) questionnaire (Brooke, 2013, 1996). The SUS is composed of ten 5-point Likert scale items, resulting in an interpretable score between 0 (negative) to 100 (positive) (Brooke, 2013). Additionally, we choose to adopt the adjective ratings as proposed by Bangor et al. (2009), further facilitating the interpretation of the numerical score and explanation of the results.

Furthermore, to gain insights about the collaborators’ general engagement with their respective interface, we asked them to complete the User Engagement Scale - Short Form (UES-SF) questionnaire (O’Brien et al., 2018). As opposed to the much more extensive 30 item Long Form version, the UES-SF consists instead of only twelve 5-point Likert scale items, i.e., three items for each of its four factors: Focused Attention, Perceived Usability, Aesthetic Appeal, and Reward (O’Brien et al., 2018). Scores on a scale from 1 (negative) to 5 (positive) can be calculated for each of the four factors individually as well as an overall engagement score (O’Brien et al., 2018).

Assessing system usability and user engagement within the scope of our task allows to gain further insights into the implemented data exploration interfaces and their collaborative features. While both interfaces are using fundamentally different display and interaction technologies, we believe it is important to potentially identify factors that might impact the pair’s collaboration. The SUS and the UES-SF questionnaires are comparatively inexpensive data collection methods, and are both widely recognized and applied in the research community (Brooke, 2013; O’Brien et al., 2018).

With our research objective in mind, we are motivated to investigate aspects of the pair’s collaboration. In particular, additionally to observations through a researcher, we are interested in the pair’s own perception of their collaboration after the task completion. For that purpose, means of self-reporting through the participants are required, commonly implemented through Likert scale statements (quantitative) or open-ended interview-like questions (qualitative). To the best of our knowledge, there is no dedicated standardized CSCW questionnaire for the purpose of investigating collaboration in virtual environments. We also examined potential alternatives, for instance the Social Presence Module as part of the Game Experience Questionnaire (Poels et al., 2007; IJsselsteijn et al., 2013), but deemed those not specific enough within the scope and purpose of our investigation, where a pair is exploring and interacting with spatio-temporal data – a comparatively common use case (Fonnet and Prié, 2021).

Consequently, we set out to design a questionnaire to satisfy our needs. Based on relevant literature, we started by identifying important aspects and dimensions of collaboration. Dix (1994) presents a general framework for CSCW by dissecting its components in Cooperative Work and various aspects of Computer Support, i.e., Communication, Computerized Artefacts of Work, and Non-Computerized Artefacts. Throughout the framework, the importance of communicative aspects as part of cooperative work is emphasized, in particular as Computer Mediated Communication, arguing for its appropriate integration respectively (Dix, 1994). Within the context of CSCW, four key features that collaborative virtual environments should strive to support are defined by Snowdon et al. (2001) as follows: Sharing Context, Awareness of Others, Negotiation and Communication, and Transitions between Shared and Individual Activities. A conceptual framework and taxonomy by Gutwin and Greenberg (2002) is dedicated to awareness within the context of group work. Awareness, seen as a state of being attentive and informed about the events in a situation and environment, can be maintained rather easy and natural in face-to-face workspaces as opposed to groupware ones that do not feature face-to-face communication (Gutwin and Greenberg, 2002). Gutwin and Greenberg (2002) differentiate between situation awareness, workspace awareness, and awareness maintenance, and move on to propose a Workspace Awareness Framework to describe aspects related to environment, knowledge, exploration, and action. Pinelle et al. (2003) propose a task model to support Collaboration Usability Analysis. They categorize the mechanics of collaboration into different aspects of communication and coordination, and go on to describe their task model that consists of scenario, task (individual and collaborative), and action components. Andriessen (2001) proposes a heuristic classification of the major activities involved in cooperative scenarios according to interpersonal exchange processes (communication), task-oriented processes (cooperation, coordination, information sharing and learning), and group-oriented processes (social interaction). Within the more specific context of Collaborative Visual Analytics, Heer and Agrawala (2008) discuss important design considerations to facilitate collaborative data exploration, among others relevant to Common Ground and Awareness, Reference and Deixis, and Incentives and Engagement.

Based on the insights and impressions gained from the various classifications according to the described literature, all discussing collaboration in regard to similar themes from slightly different perspectives, we subjectively decided to follow and adopt the descriptions by Snowdon et al. (2001), emphasizing various key aspects that collaborative virtual environments should aim to support. We believe that the investigation of Sharing Context, Awareness of Others, Negotiation and Communication, and Transitions between Shared and Individual Activities should allow for the retrieval of insights in regard to different important collaborative aspects, thus providing a “bigger picture” of the collaboration during the completion of an analytical task (Snowdon et al., 2001).

Our self-constructed questionnaire, named the Spatio-Temporal Collaboration Questionnaire, was designed to assess aspects in a setting of synchronous collaboration as follows. It features 17 5-point Likert scale statements that are thematically relevant to the four dimensions as adopted from Snowdon et al. (2001), and described in the following way:

• Transitions between Shared and Individual Activities (TSIA): The interplay between individual and group efforts, including the ability to switch between these, within the scope of collaborative work.

Table 1 presents an overview of all item statements and their Likert scales across these four dimensions. The design of the individual item statements is held purposefully generic, anticipating re-usability, remix, and further adoption for evaluations in similar contexts in the future. Under consideration of our overall scenario and task as described in Section 3 and Section 4.1.2, only the items AO.2, AO.3, AO.5, and AO.6 are rather use case specific in regard to the collaborator’s ability to send and retrieve spatio-temporal references using their respective interfaces. Therefore, these four items inquire ratings about the collaborator’s location in space and time reference during group and individual efforts. In practice, the questionnaire is to be filled out by each participating collaborator individually and directly after the respective task completion. The evaluation of the answers should allow for a quantitative analysis of the system’s collaborative features, and provide insights in regard to the collaboration as perceived by the collaborators themselves. Furthermore, the results should be interpreted within the context of the tested system and against its anticipated design, for instance to assess if an anticipated role distribution (equal, unequal) between the collaborators was fulfilled as intended, to name just one example. A two-page print version of the questionnaire, as it was presented to participants in our user interaction study, can be found in the article’s Supplementary Material.

As the participants were located in two physically separated rooms during the task sessions, there needed to be a way for them to verbally communicate (besides the nonverbal communication features of our system, i.e., the spatio-temporal referencing). The Zoom Cloud Meetings ¹⁴ teleconferencing software was installed on both the machine that the non-immersed participant was using, as well as on the machine that run the immersive application. This enabled the participants to talk to each other via an audio call, which was recorded. Zoom conveniently allows the recording of separate audio streams for each call participant; therefore, at the end of each task session, three audio files (one of the combined audio, and one from each user) were obtained. Using the Audacity ¹⁵ audio editor software and its Sound Finder tool, it is possible to obtain timestamps that describe when sound was detected ¹⁶ in each participant’s audio file, and therefore roughly when they were (individually) speaking. Summing up the time intervals provides an estimation of each participant’s “speaking” amount, and it was also possible to calculate when and how much participants were “overlapping” (talking at the same time). These timestamps were further synchronized with the system log timestamps, by knowing when the audio recording of each session started (and ended). This can allow the correspondence of verbal communication activity and system events (including nonverbal communication cues).

The answers to the Spatio-Temporal Collaboration Questionnaire (see Section 4.2.3) from the perspectives of the immersed and non-immersed users are provided in Figure 7. The discussion of the results is based primarily on the median values and, where significant, on the interquartile range.

The participants reported that there were a few individual efforts (TSIA.1), and a lot of group efforts (TSIA.2); the non-immersed users also had the impression that they took a more leading role than the immersed users (TSIA.3).

The participants reported constant verbal communication (NC.1), and often nonverbal communication (NC.2). The participants reported that were almost constantly in dialog (NC.3) and that they sometimes negotiated (NC.4). Similarly to the responses as to who took more often the leading role (TSIA.3), the non-immersed users considered that they initiated these negotiations more often than the immersed users; however this was done equally on median (NC.5). Noteworthy for the context of the NC items, all paired medians were the same. Wilcoxon signed rank tests were conducted to determine whether the median values were the same for the immersive and non-immersive interface users, for all NC items; in all cases there was no significant difference (p = 1 for NC1, NC2 and NC3; p = 0.42 for NC4 and p = 0.018 for NC5).

The participants strongly agreed that the system allowed them to focus on the same subject as their partner (SC.1), and also to establish dialog (SC.2). While the participants disagreed that the collaborative features of the system distracted them from their individual efforts, the interquartile range for the immersed users was wider (SC.3).

The participants were always aware of their partner’s activities during group efforts and a lot during individual efforts (AO.1 and AO.4). The awareness of others during individual efforts was a little lower than during group efforts, as expected. The participants were always aware of their partners’ location during both group and individual efforts, except for the immersed users during individual efforts (AO.2 and AO.5). Finally, the participants were always aware of their partner’s time reference during group efforts (AO3), and a lot during individual efforts with quite wide interquartile range for both users (AO.6).

Table 3 presents the session duration for each task scenario in minutes (M = 28,  SD = 9), the number of unique places the participants visited (both, together, and independently), how long they were at the same place at the same time (M = 87%,  SD = 7%), how long the speaking time was for the immersive (M = 29%,  SD = 14%) and the non-immersive interface users (M = 39%,  SD = 13%), as well as how long their speaking overlapped each other (M = 2%,  SD = 0.2%).

One noticeable outlier regarding session duration was the second session by the fourth pair. However the short duration had no impact on the task performance (see Table 2) and was not due to the participants being in a hurry to complete their session. Overall, there was no significant difference of session duration means comparing the fruits (M = 30,  SD = 7) and veggies (M = 26,  SD = 11) scenarios; paired t-test, t (4) = 0.65, p = 0.55. A paired t-test was conducted to compare the normalized amounts that the participants were at the same place for the fruits (M = 88%,  SD = 7%) and veggies (M = 87%,  SD = 8%) scenarios; t (4) = 0.18, p = 0.86, indicating that the means were not significantly different. A paired t-test was conducted to compare the immersed and non-immersed users speaking time means; t (9) = −1.80, p = 0.11, indicating that the means were not significantly different. However, every participant that changed role from using the non-immersive to the immersive interface spoke less, and every participant that changed role from using the immersive to the non-immersive interface spoke more or about the same.

Additionally, we visualized the participants’ verbal communication activity based on the recorded audio (see Section 4.2.4), as well as plotted their spatial data exploration over time as 3D pathway visualizations ¹⁹ based on the collected system logs (see Section 4.2.1). Both visualizations for every task session can be found in the article’s Supplementary Material.

All reported usability scores (SUS) for both interfaces are above the good margin, indicating that the users were generally able to operate the interfaces for their intended purpose. Given the overall purposefully minimalistic but representative designed interactive InfoVis, the comparatively high usability scores (median above excellent) for the non-immersive desktop terminal are not that surprising, as it relied on rather established visualization approaches, i.e., line graphs and bird’s-eye view map (Lundblad et al., 2010; Ward et al., 2015). The comparatively novel approach as implemented in the immersive VR environment, and considering that all but one participant reported only minor experiences with VR in general before, we are particularly encouraged by the received positive usability feedback. All participants were able to quickly pick up and learn the various aspects of the immersive interface during their 5 minutes warm-up time, i.e., understand the concept of the 3D Radar Charts and the collaborative information cues, become comfortable with wearing the HMD, and utilize the 3D gestural input to interact in the VR environment. This is in line with the general anticipation of utilizing immersive technologies for their natural interaction techniques (Büschel et al., 2018; Skarbez et al., 2019). After all, enabling users to simply pick up the technology and start using it for data analysis purposes in an intuitive manner without extensive training allows them to focus at the subject matter at hand. The self-reported usability scores coincide also with the observations, confirming the VR user’s ability to operate the immersive interface in a natural and fluent manner. In fact, the majority of the participants made reflections at the very end of the study, i.e., after the completion of their second task, positively highlighting the “smoothness” of the VR experience and that they overall could have easily spent even more time with their joint data analysis activity. Considering these comments in regard to the measured session duration (M = 28,  SD = 9) and within the presented data analysis scenario and task, we believe this to be a step towards the direction of moving beyond comparatively brief “just a few minutes” VR experiences. This is also important keeping in mind the complexity inherent from CSCW, i.e., interpretation of data, information exchange, as well as discussion and negotiation take time (Andriessen, 2001; Heer and Agrawala, 2008). Following this line of thought and with respect to such multi-user interplay, one can potentially anticipate comparatively longer exposure times in VR within CIA scenarios compared to single user experiences. Among others, we believe the above good usability scores for both interfaces are important within the scope of our investigation for two reasons in particular. First, it validates that the interfaces could be operated as intended (see Section 3.2 and Section 3.3) and without any major usability flaws. Consequently for our study, we can assume that it is rather likely that a “difficult to operate interface” would have had a potentially negative impact on the pairs’ overall collaboration. And second, it also indirectly validates the usability of the designed and integrated synchronous collaborative features as part of each respective interface. The visual information cues (see Section 3.4) were easy to recognize and provided important contextual references about their partner’s activity, as also emphasized by Cruz et al. (2015). Additionally, these spatio-temporal references were integrated in a rather seamless manner, allowing for the automatic transmission of information as the users naturally interacted with their interfaces. As opposed to introducing additional dedicated actions in regard to what information to share and when, such as used by Welsford-Ackroyd et al. (2020), Reski et al. (2020b), and Peter et al. (2018), and based on our observations, this seemed to have allowed the collaborators to naturally interact with each other, seamlessly picking up and referring to their partner’s context without noticeable action delays, i.e., without the need to wait for a specific collaborative signal.

The median user engagement scores (UES-SF) for both interfaces are at or above 4, indicating overall high user engagement with the provided collaborative system during the data analysis task. This corroborates the observer’s impressions of the collaborators being motivated and eager to use their interfaces to explore the data and to find the correct answers, noticeably enjoying themselves and their collaboration during their confirmative analysis task. These results align well with the often stated argument that immersive technologies have the potential to provide “engaging” experiences that encourage data interaction and interpretation (Hackathorn and Margolis, 2016; Dwyer et al., 2018; Ens et al., 2021).

Furthermore, the results allow for a discussion in regard to the individual user engagement factors. The immersive interface users reported a higher Focused Attention compared to the users who operated the non-immersive desktop terminal. We can argue for a couple of potential reasons for this. Primarily, the characteristics of the applied display and interaction technologies have to be taken into account. Perceiving a virtual environment through an HMD and allowing for natural hand interaction, i.e., a comparatively high level of immersion, may have required the VR user to be generally somewhat more attentive, as there are many visual stimuli to process – both in regard to the data visualization in the immersive VR environment itself as well as due to the integrated visual information cues triggered through the non-immersed collaborator. Additionally, it also needs to be considered that while the non-immersed user explored two data variables (climate data) per location, the immersive interface was presented with five data variables (plant data) per location (see Section 3.1 and Section 4.1.2). Nevertheless, the general high Focused Attention across both interfaces can be attributed to the overall close collaboration between the users, e.g., the pairs investigating the same respective locations for the majority of the task duration (see were at same place column in Table 3), collaboratively making observations. In regard to the slightly lower Focused Attention score reported by the non-immersive interface users, another aspect comes to mind: The note-taking and completion of the pen-and-paper answer sheet (see Section 4.1.2 and Supplementary Material). During the task, they were in charge of keeping track and filling out the provided plant-climate answer matrix, which arguably may have effected their attention to the interface as they were required to temporary switch their focus to the paper answer sheet.

The reported Perceived Usability scores for the two interfaces were in line with the reported SUS scores, as discussed in the prior section.

Aspects in regard to the Aesthetic Appeal were rated similarly positive across both interfaces as well. At this stage of the collaborative system, we focused in both the immersive and the non-immersive interface on the essential parts that allow the users to explore and analyse data, trying to avoid unnecessary information or distracting elements in general. We are satisfied with the received Aesthetic Appeal scores, overall indicating that participants enjoyed the chosen graphical elements and visual design for each of the interfaces accordingly.

The positive Reward scores, reported with medians above 4.5 for both interfaces, are particularly interesting and encouraging to us. On the one hand, the collaborators were observed being particularly motivated to solve the given task correctly, often verifying their observations of the time-series data across multiple different locations to ensure that their answer was correct. A reoccurring expression across the different task sessions was along the lines of “I am sure we got it [ right ], but let’s just check one more [ location ]”. The investigative nature of the confirmative data analysis task, provided the pairs with a clear purpose for this activity, which is also important in regard to moving beyond initial novelty reactions (Snowdon et al., 2001). At the same time, it was completely up to them to organize their task solving approach, resulting in interesting data exploration strategies. The freedom of task approach, combined with the fact that they had to work together, using different types of display and interaction technologies, but still being able to have a notion of what their partner was up to, supported through the synchronous collaborative features integrated in the interfaces, are likely to have positively contributed to these Reward scores. The participants appeared genuinely excited that “it [ the collaborative system ] really worked” and “we [ the collaborators ] are able to see each other”, thus successfully enabling them to be mutually aware of each other (Benford and Fahlén, 1993; Benford et al., 1994; Heer and Agrawala, 2008). The positive Reward scores for both interfaces within the presented collaborative context arguably also indicate a comparatively equal user contribution: Of course each interface served their own purpose, but they were engaging for both collaborators alike, motivating them to partake in the data analysis activity – as anticipated (see Section 2.3). We believe it is also important to consider and discuss two more factors within the context of the positive Reward scores: The overall high rated usability, and the absence of an explicit time limitation. The users ability to use the interfaces as intended in a scenario where they were not pressured for time, informally confirmed by some of the participants’ expressions that they could have spent even more time with the developed system, likely also contributed beneficently to the Reward scores.

To summarize, both the immersive and non-immersive interface were assessed positively in regard to usability and user engagement. We believe it is important to highlight again that the reported scores are not meant to be compared in a “X is better than Y” manner, nor can they (due to the asymmetric role setup), as the interfaces serve different purposes. Instead, with the primary objective to investigate collaborative aspects when combining immersive and non-immersive interfaces into the same data analysis workflow, as motivated in Section 2.3, we believe it is crucial to have an understanding and assessment of the applied tools that are likely to impact the collaboration. After all, the empirical evaluation of collaborative immersive systems is complex (Billinghurst et al., 2018; Skarbez et al., 2019; Ens et al., 2021). Having received a similar assessment by the participants, we can assume that the two interfaces are appropriately balanced in regard to their purpose and operability within the scope of the presented context (see Section 3) as well as with respect to anticipated hybrid data analysis workflows (Isenberg, 2014; Cavallo et al., 2019; Wang et al., 2019). We argue that this serves as a good foundation for the assessment of various collaborative aspects, particularly when using heterogeneous device types and an asymmetric user role setup.

The analysis of the system logs as presented in Table 3 reveals that the collaborators spent the majority of their time investigating the time-series data at the respective same spatial location (min 75.0%, max 97.7%). This also becomes apparent when examining the audio analysis and pathway visualizations (see Supplementary Material). Consequently, we can infer that the collaborators found themselves in a state of rather close collaboration for the majority of the session duration, i.e., directly interacting with each other in the same data context, making efforts as a group to solve the task. They communicated about their observations and findings both verbally, i.e., by talking to each other to explain, discuss, and negotiate, and nonverbally, i.e., by making spatio-temporal references to point and highlight data for their peer (see Section 3.4). Collaboration relies inherently on complex personal and social processes (Heer and Agrawala, 2008; Billinghurst et al., 2018), therefore every human user has slightly different ways and approaches of interacting with one another. We believe this is well reflected in the results of the audio analysis in regard to the “speaking” time during the task sessions (see Table 3). For instance, some pairs (e.g., pair p2, fruits scenario, 24.4 min: 8.1 and 24.8%) communicated verbally less compared to others with much higher speaking time rates (e.g., pair p3, fruits scenario, 22.4 min: 20.8 and 69.1%). However, examining their verbal activity over time (see Supplementary Material), we can identify a steady activity in the majority of cases that also align with the positive user engagement results. Nevertheless, there were also some instances when the collaborators made some more individual efforts. These usually occurred when a pair set out to find a new spatial location to explore, either in regard to yet completely unexplored plant-climate correlations, or in order to verify and confirm previously made deductions. Most of the time, both users began to explore the data using their interfaces independently in order to find a place that contained “interesting” data, or in the words of the participants, time-series data visualizations that are “curvy or bumpy” and feature “peaks, spikes, slopes, or valleys”. An interesting case is pair p4 within the veggies scenario, whose verbal activity was much lower during these phases compared to the ones when they explore the same spatial location. However, during similar spatial exploration phases by other pairs, their overall verbal activity did not seem to change that much compared to the remainder of their joint data exploration, indicating that they generally kept talking to each other independent of whether they were making individual or group efforts.

In general, all pairs were able to collaboratively complete the tasks of identifying the various data correlations (ten in total per task scenario) in a satisfactory manner (see Section 5.2). Given the hybrid asymmetric setup (heterogeneous device types and different user roles), we did not anticipate any knowledge carry over from the fruit to the veggies scenario in regard to the respective interface’s operation. A carry over of data insights was also not possible as both task scenarios featured different datasets. There is the possibility for the pair’s task solving approach in the veggies task to be somewhat influenced and informed by their strategy in the prior fruit task. However, we have confirmed that there was no significant impact on their task performance (see Section 5.2). Given these circumstances as well as the prior described influences of the personal and social processes on collaboration (Heer and Agrawala, 2008; Billinghurst et al., 2018), we believe it is unlikely that there was a noticeable knowledge carry over between the two task scenarios.

With all the above in mind, throughout the following sections we can have a closer look at the collaborators’ self-reported assessments as collected using the Spatio-Temporal Collaboration Questionnaire (see Figure 7).

The pairs’ reporting in regard to the occurrence of individual and group efforts is in line with the observations and system log analysis. They considered having made a lot of (shared) group efforts, while only a few individual efforts during the task solving activity. Furthermore, the pairs had the impression that the non-immersed user was in a somewhat more leading or directing role compared to the immersed one. While the immersive interface users reported a median of more other, some me, the median for the non-immersive interface users lies between both equally and more me, some other. Based on our observations, it is likely that the task answer sheet responsibility has given the non-immersed user a little bit the edge towards being the task director. Even though the non-immersed user assumed such a “leading” role at times, we do not consider this to be the same as the dedicated guiding roles such as discussed by Welsford-Ackroyd et al. (2020) or Peter et al. (2018), but instead an overall rather balanced interplay between the collaborators for their own purposes that is conceptually similar to the scenarios as described by Lee et al. (2020), Sugiura et al. (2018), and Gugenheimer et al. (2017).

In regard to the pairs’ verbal communication frequency, both interface users reported that they talked pretty much constantly, which agrees with visual impressions from the audio activity analysis (see Supplementary Material). The pairs stated that they often utilized the nonverbal communication features, i.e., the provided collaborative synchronous features (see Section 3.4). Furthermore, the pairs considered dialog taking place for the majority of their verbal communication. Negotiation was reported taking place only sometimes if at all, rather similarly initiated from both interface users. All the above is interesting for a couple of reasons. First, the medians from both interface users across these five items are equal, overall indicating that the collaborators had a rather similar impression about their negotiation and communication independent of the interface type. Second, considering the higher share of dialog compared to the lower amount of negotiations, it seems that the collaborators were rather successful in their verbal and nonverbal communication, being overall able to follow their joint data descriptions and interpretations, without much need for additional negotiation. And third, the reported amounts of verbal and nonverbal communication, most of the time categorized as dialog, further indicate a close collaboration between the two interface users. It was also interesting to observe pairs establishing their individual reference terminologies (as presented in Section 5.4.3), including common and reoccurring expressions as well as more unique ones.

The results indicate that the implemented collaborative features allowed the users to focus on the same subject as their peer and to establish a dialog accordingly, which is a foundational aspect for successful CSCW (Snowdon et al., 2001; Heer and Agrawala, 2008; Cruz et al., 2015). Overall, the collaborators disagreed that these features distracted them from their individual efforts, however with a slightly bigger range of provided impressions. Generally, all the results in this category are favourable within the context of the presented setup and task. The ability to focus on the same subject matter and to establish dialog are crucial for any kind of collaboration (Dix, 1994). With both interface users confirming that they were able to do so, the overall design of the synchronous collaborative features across both the immersive and non-immersive interface can be considered validated within the presented context, of course assuming that a verbal communication channel is available (see Figure 1). These results are also relevant within the context of physically distributed collaboration environments (Skarbez et al., 2019), as it enables analysts to work together remotely, independent of their distance to each other. Furthermore, while the collaborators did not assess the implemented collaboration features as distractions during their individual efforts, it is important to consider the amount of reported individual effort that took place, i.e., only a few. On the one hand, the collaborators assessment is a promising trend in regard to the provided features design, allowing them to focus ad-hoc on their peer’s context without interfering with their own individual efforts. On the other, some further investigations using tasks that involve more individual efforts throughout the collaborative task are necessary to confirm or reject this trend. Finally, it was also interesting to observe that some pairs came up independently from each other with similar ways of utilizing the implemented collaborative features, such as the “live annotation” behaviour (see Section 5.4.3).

Both interface users reported high awareness about their respective partner’s activity in general, their location in space, and their time reference. These assessments allow again some reflections and discussion on the implemented system features that aimed to facilitate their awareness of one another. First, the assessments for joint awareness was reported slightly higher during group efforts, which is desired as this is arguably the situation when it is more important to know about the collaborator’s activity. Nevertheless, the awareness was still rated fairly high even during the few individual efforts, and seemingly in a non-distracting manner as discussed before. Second, the awareness of the immersed user was slightly higher perceived through the non-immersed user, with everyone agreeing that they were always aware of the activities of the user in the immersive VR environment. Reflecting on the characteristics of the implemented visual information cues across both interfaces (see Section 3.4), one aspect becomes apparent in regard to seemingly different update frequencies. For instance, the location and time reference updates from the immersive interface appear much more “continuous” in the non-immersive interface, e.g., the position and orientation of the immersed user are constantly updating, and even new time event and time range selections appear much more in motion and fluent due to the nature of the involved technologies. This arguably provides smooth visual transitions from one state to another, naturally updating the non-immersive interface accordingly. The other way around, collaborative information cues from the non-immersed user update more “discrete” and “event”-like in the immersive VR environment, i.e., new selections appear when they are done, providing comparatively fewer visual transition cues. We believe that this may be a good starting point for further investigation into this matter. Overall, based on the implemented collaborative information cues across both interfaces, it appears that each user was able to follow and have an understanding of their partner’s current investigation, closely coupled with the results of the Sharing Context category. Considering the general importance of mutual awareness for the design of collaborative systems (Benford et al., 1994; Snowdon et al., 2001; Cruz et al., 2015), not least as important foundation to establish communication that results in the subsequent interpretation and discussion of the data (Andriessen, 2001), the received awareness assessments can be interpreted positively. We can argue that the designed visual approaches for supporting spatio-temporal references worked well and as intended in both the immersive and non-immersive interface, allowing the pairs to point and highlight data to their peer accordingly. Our results also align well with the insights reported by Nguyen and Duval (2014), stating that rather simple awareness cues can often be sufficient to provide the collaborator with an understanding of the shared workspace.

Throughout all task sessions, the collaborators were able to work closely as a group for the majority of their joint session duration in order solve the given confirmative data analysis task in a satisfying manner (see Table 2). Considering that they had to provide ten answers (including ten accompanying confidence indications), they were overall quite busy during an average half hour for data exploration, observing, interpreting, and discussing their findings. They communicated a lot by having complementing dialog that was further facilitated through the various spatio-temporal referencing features of their interfaces. The setup allowed them to closely explore and interpret the data in space and time, making important observations and deductions along the way. One pair in particular highlighted the “detective work”-like nature of the task and their joint collaboration, reflecting on the great interplay between the two interfaces, rating the experience in a very positive manner. The participants’ overall excitement as well as the natural way of interacting with each other was a reoccurring theme through the different task sessions, likely positively contributing to their collaboration assessments. This can be further underlined through a selection of noteworthy participant comments after their task completion:

• “Oh, this was really fun and worked really well.”

Comments such as particularly the last one, are quite interesting given that there were no “avatar”-like representation of the users in either of the interfaces, such as for instance as utilized by Nguyen et al. (2019) or Benford et al. (1994), but just the provided visual references. It appears that the participants made the mental association between the visual references and their peers, themselves. Similar observations within the context of remote collaboration around interactive tabletop systems, also including rather abstract and minimal visual representations for the collaborator’s input, were made by Kim et al. (2010), reporting that users in their study “(…) felt as though the remote participants were in the room itself.” It would be interesting in the future to investigate effects on collaboration and empathy when there is no virtual user avatar but instead other similar more abstract means of user representation, identifying requirements and use cases where one approach is potentially preferable over the other. For instance, a CIA system presented by Nguyen et al. (2019) used a virtual avatar representation for the peer in the VR environment. How could an alternative approach without such an avatar look like, and what would the difference be in the (perceived) collaboration?

The expressed appreciation for the collaborative system and rewarding experience through the participants is much in line with the visions for such hybrid analysis environments, combining different types of technologies, as described by Wang et al. (2019) and Isenberg (2014). All in all, using different types of display and interaction technologies and facilitated through various collaborative features, we can summarize that within the scope of our experiment, all pairs were able to successfully collaborate with each other in a rather balanced shared workspace manner, as opposed to more common remote expert scenarios in similar technological setups (Ens et al., 2019).