Eight participants (six men and two women; college students aged 19–21 years) enrolled in the study. The participants were segregated into four groups, with each pair referred to as collaborators.

Recruitment was open for 1 week from 17 February 2023. Participants were given an explanation of the consent document in the laboratory, and the informed consent procedure was carried out. The participants were handed a paper that outlined the experiment and data-handling procedures, which were explained by the experimenter. All eight participants agreed to participate in the study. The experimental data were obtained using anonymized ID numbers. This ensured that the data were not linked to the participants’ names.

The VR systems were established in two separate rooms. Each system comprised a VIVE Pro Eye (HMD), two controllers (VIVE Controller 2018), two base stations (SteamVR Base Station 2.0), and a computer. The VR environment was created using Unity (2021.3.1f1) in a server-client network using “Netcode for Game Objects.” In this environment, paired participants entered the same virtual space and interacted via physical actions. No audio communication was available, and the VR environment featured two avatars, buttons, a display, and a mirror (Figure 1). All experimental configurations and spatial arrangements shown in Figure 1 represent the layout within the virtual reality environment, not the physical laboratory setup. The avatars were able to move based on six-coordinate data (three positions and three rotations) obtained from the HMD and two controllers. These avatars were boxy and lacked personality traits, and their movements were executed using the “Final IK (Inverse Kinematics)” asset. Red- and green-labeled reaction buttons were placed in front of the avatar in the VR space. The RTs were acquired via collision detection when the avatar touched a button.

The task comprised four phases: Phases 1-4 (time count, fixation cross-presentation, target presentation, and blanks, respectively). Phases 1 and 3 were the countdown and motion phases, respectively (See Figure 2).

The participants were instructed to touch a button corresponding to the target color, regardless of its location. Each session comprised 16 or 32 consecutive trials, with the target color and position randomized between the trials.

The participants were allowed to select either the client or host experimental booths. The terms “host” and “client” were designated because paired data were transmitted as streamed data from the client to the host PC. Following the instructions of the experimenter stationed at each booth, the participants were instructed to wear the HMDs and operate the controllers with both hands. Before each session, the participants were briefed on the colors of the stimuli for which they were responsible. Before commencing the joint task, the participants were instructed to view their collaborators directly and then confirm their avatars in the mirror set in the VR space.

The host stood on the right, whereas the client stood on the left. Figure 1 illustrates the experimental setup for the joint Simon task and conformity/competition tasks within the virtual reality environment. Participants interacted with color targets displayed on screen, responding by pressing corresponding buttons in the virtual environment according to task instructions. The host operated the buttons in the VR space using the left hand, whereas the client used the right hand. Thus, the right hand was not used on the host side, and the left hand was not utilized on the client side. The task involved pressing a button labeled with the corresponding color name when the assigned color appeared. The participants were instructed to halt if they felt uncomfortable, lift their HMDs at the end of each session, and take breaks as required.

The participants entered the space individually and completed eight practice trials for the Go/No-Go task. During the practice session, the correct answer was indicated when the correct button was touched. An incorrect answer was revealed when another button was touched or when a certain amount of time had elapsed without a touch being detected. If a participant failed in all eight trials, then the practice session was repeated. After the practice session was completed, the following sessions were conducted.

Session 2 involved the procedure shown in the upper section of Figure 1. The target colors in Session 3 were swapped to minimize the learning effects. Sessions 4 and 5 involved the procedure shown in the lower section of Figure 1. The detailed session structure for the preliminary experiment is presented in Table 1.

Sensor data from Sessions 4 (conformity) and 5 (competition) were used for machine learning and testing, respectively. In the conformity task session, the participants were instructed that “whichever target appears, touch the correct button at the same time as your companion.” During the competition task session, the participants were instructed that “whichever target appears, touch the correct button before your companion.”

In the preliminary experiment, we investigated the features necessary for distinguishing between interpersonal behaviors in VR environments. To identify subtle differences in subconscious movements, we used the sensor data during Phase 1, i.e., the time at which the participants were staring at the countdown, as shown in Figure 1. The phases and procedures illustrated in Figure 1 were all conducted within the virtual reality space.

The sensing data included gaze direction, eye position, pupil size (left and right), head position and rotation, and the position and rotation of the left and right controllers. For each of these, XYZ three-axis data were recorded where applicable. The transmission latency from the client to the host was approximately 0.01 s. Signals were sampled at a variable rate (80 Hz average), and after missing values were removed, the client and host data were linked at intervals of approximately 0.02 s and then used for machine learning. The features used for machine learning were: distance, which was obtained as the root sum of squares of the XYZ (Euler angle) of the position and gyro sensor at each sampling point; the velocity from the time difference; and the acceleration obtained from the time difference in velocity, which was used as the analysis data. After deleting samples with missing values, we used the Python sklearn Random Forest Classifier (n_estimators = 250, random_state = 42) as the random forest model. A set of decision trees was constructed for a subset of randomly sampled training data, and predictions based on a subset of these features were aggregated to obtain the final prediction. After testing various feature types, we found that distance features yielded relatively high classification accuracy, whereas models using velocity and acceleration performed poorly. Therefore, we decided to use only distance features, except for triaxial gaze data, which were retained as they are considered essential for synchronization. To evaluate classification accuracy, cross-validation was conducted by iteratively designating data from three out of eight participants as the test set, while data from the remaining five were used for training. This process was repeated to ensure that each participant appeared in the test set at least once. The final model was trained on the full dataset after cross-validation.

The participants of this experiment were recruited through a university website, and assigned to each experimental day. Recruitment was open for 2 months from 14 May 2023, and the informed consent procedure was the same as the preliminary experiment, but was obtained in advance via a web-based questionnaire in order to avoid coercion in obtaining consent due to face-to-face situations in the laboratory. Owing to the absence of one participant, the final number of participants was 18, which comprised seven men, nine women, and two other genders. The average age of the participants was 19.83 years (standard deviation [SD] = 1.04). All participants were assessed for handedness using a self-report questionnaire. Of the 18 participants, 16 were right-handed, 1 was left-handed, and 1 reported being ambidextrous or having no hand preference. After the experiment, the participants were instructed to complete a questionnaire survey and interview, for which they received an honorarium of approximately $10 (1,500 yen) after completion. The device and experimental procedures were identical to those used in the preliminary experiments. Two types of avatars, i.e., a box and a human, were designed for other research purposes (Figure 3).

All participants completed the sessions in the same fixed order shown in Table 2.

Correct response rate: Correct responses were counted when participants touched the correct color target in trials where they were required to respond and refrained from touching in trials where they were not. Subsequently, the correct response rate was divided by the number of trials.

JSE: The mean RT delay (RTs for incompatible targets minus RTs for compatible targets) during the joint Simon task (Sessions 2, 3, 6, and 7) minus that of the Go/No-Go task (Session 1) was calculated. The RTs for correct responses with more than two standard deviations from the mean RT were excluded as outliers. Additionally, pairwise data from participants whose RT could not be measured because of equipment failure were excluded.

Bot cognition: After the experiment, a structured assessment combining questionnaires and follow-up face-to-face interviews was conducted to systematically evaluate participants’ awareness of the bot condition. The assessment protocol was designed to minimize leading questions and retrospective bias. A participant who perceived the human collaborator to be a bot was assumed to be unaware of the discrimination between humans and bots. In the data analysis, binary values of 1 and 0 were used to indicate the awareness and unawareness of bots, respectively. As a proxy for perceived social presence, we also included a two-item measure rated on 7-point Likert scales. The items assessed how realistic and how human-like the avatar appeared. The sum of these two items was used as an index of social presence (Mean = 8.78, standard deviation [SD] = 2.29).

Sensor data: By performing the procedures of the preliminary experiment, the distance was obtained as the root sum of squares of the XYZ (Euler angle) of the position and gyro sensor at each sampling point; the velocity was the time difference between the two; and the acceleration was the time difference between the two. Because the accuracy of the classification model using velocity and acceleration data was low during the machine learning process, we adopted a model that used only distance features. The resulting features were the HMD position, HMD rotation, and 12 variables of position information for the left- and right-controller positions and rotations (Figure 2). The eye-gaze and pupil size features used in the preliminary experiments were not used in the main experiment because no sensing data corresponded to the bot. Under the bot condition, only trace data from Phase 3 were used; thus, data from Phase 3 were used to formulate the classification model. Samples with missing values on the host or client side were deleted. After removing missing values, the number of observations obtained from Sessions 3, 4, and 5 was 16,759 for training and testing, out of a total of 124,193 observations (approximately 86.9% of the original data were retained). Sensor data from Sessions 2, 6, and 7 were prepared as files on the host and client sides to compare the human and bot conditions.

Pair Activity Probability: Details of machine learning, angular transformations, and statistical models are described in Section 4.3.

Synchrony Index: As in the preliminary experiment, the most important feature for classifying paired activities was the rotation of the unused hand (host side right-hand rotation). Therefore, the cross-correlation (CCC) of the sensor data for the host side right-hand rotation and client-side left-hand rotation was calculated for each trial and then used as the interpair synchrony index using MATLAB’s XCORR function. After normalizing the sensor data for each trial, the maximum value obtained at lag0 was used as the CCC index.

We used the MATLAB Classification Layer application for the machine learning model to compare the decision trees, random forests, support vector machines (SVMs), and neural nets. The results showed that even a single decision tree provided a correct answer rate exceeding 90%, which is comparable to the performance of other methods. Thus, we adopted a decision tree model to identify the most important features. The Gini diversity index was used as the splitting criterion.

Each of the sensor datasets in the three training sessions was subjected to machine learning, with target variables for classification (10% was used for data verification and cross-validation).

The Receiver Operating Characteristic (ROC) value calculated from the true-positive and false-positive rates exceeded 0.98, which was sufficient for classification accuracy. The ROC curves are presented in the Supplementary Appendix. The final number of branching nodes was 191.

The most important features for classifying joint activities were the right-hand rotation unused by the host and the left-hand rotation unused by the client. The features of the model were similar to those of the preliminary experiments, which classified conformity and competition in the countdown phase, thus suggesting that joint action in the subconscious movement can be classified using the categorization model based on the motion phase.

Using MATLAB’s trained predict function, we applied the classification model to Sessions 2, 6, and 7 as test sessions using the 12 feature variables. The classification probability results for each observation were the activity indices of cooperation, conformity, and competition.

These probability values were angularly transformed using arcsin ( probability ) × 180 / π . We adjusted for a probability of 0 by setting arcsin ( 0.0833 ) × 180 / π and a probability of 1 by setting arcsin ( 1 − 0.0833 ) × 180 / π . These corrections were performed based on the usual adjustment ( 1 / 4 N ) for angular transformations. Thus, the minimum and maximum possible values were 16.54 and 78.69, respectively. These values were averaged for each trial and used as cooperation, conformity, and competition indices for the pair activity.

The term 1 / 4 N used in the angular transformation refers to the usual adjustment for proportions, where N is the number of response options. After applying this correction, the angular transformation of the minimum and maximum possible proportions (0 and 1) results in values of 16.54 and 78.69, respectively. These are dimensionless values resulting from the arcsine transformation of relative proportions; therefore, no physical units are associated with them.

Because the human condition comprised data that switched from the client to the host for comparison with the bot condition, we analyzed the condition effects via multilevel analysis. For the linear mixed model, paired groups were specified as random-effect factors after centralization was performed, in which the mean value of each paired group was subtracted from each indicator.

As the Akaike Information Criterion (AIC)s of each indicator’s random intercept and random slope models were similar or lower for the random slope model, we report the results for the random slope model here. Considering the few people in the random variable and a p-value that is likely to be high, we report the results of the robust model obtained via the log-likelihood ratio test. Owing to the low overall variance, we report the fixed-factor effects of the mixed model, as well as the results of the test using marginal mean estimation (in contrast to the human condition set to 1 and the bot condition set to 0).

Details of machine learning, angular transformations, and statistical models are described in Section 4.3.

The ROC value calculated from the true-positive and false-negative rates exceeded 0.98, which was sufficient for the classification accuracy. The confusion matrix and ROC curves are presented in the Supplementary Appendix. The final number of branching nodes was 191. The classification criteria for the top six branches of the decision tree are shown in Figure 4.

The confusion matrix is presented in Figure 5.

Avatar behavior varied systematically between conditions. In the human condition, avatars reflected participants’ real-time movements via the VR tracking system, providing natural responsiveness to each participant’s actions. In the bot condition, avatars replayed pre-recorded human movements from earlier experimental sessions, with no adaptive responses to participant behavior. All bot avatar motions were pre-recorded in all bot sessions (Sessions 6 and 7), and there was no dynamic adjustment to participant behavior.

Details of machine learning, angular transformations, and statistical models are described in Section 4.3.

These three activity indices (cooperation, conformity, and competition) exhibit mutually constrained relationships. The correlation between concordance and competition was uncorrelated in both conditions, whereas the correlation between cooperation and competition indicators was r = − .5432 ( p = 0.0198) in the human condition and − .5078 ( p = 0.0314) in the bot condition. The correlation between the cooperation and conformity indices was r = − .7821 ( p < 0.001) in the human condition and r = − .6061 ( p = 0.0077) in the bot condition, both of which were high. Therefore, to examine the effect of the conditions on the three activities, we examined each dependent variable individually. Figure 6 shows the angular-transformed mean values for each condition before centralization by the group mean.

Figure 6 shows the angular-transformed mean values for cooperation, conformity, and competition activity indices under human and bot conditions. Cooperation activity was significantly higher in the human condition, whereas conformity activity was significantly higher in the bot condition.

An analysis of the human or bot-condition effect with the cooperation indicator as the dependent variable revealed that the estimated value of 7.37 ( S E = 3.25) was significant ( t (9) = 2.27, p = 0.0495). Similarly, a comparison of the mean estimate of the neighborhood with the human and bot conditions of 1 and 0, respectively, was significant ( z = 2.22; Holm’s p = 0.0262). As shown in Figure 6, the ratio of cooperation activity in the human condition was higher than that in the bot condition. An examination of the human- or bot-condition effect with the conformity activity as the dependent variable showed that the estimated value of − 6.56 ( S E = 2.24) was significant ( t (9) = 2.93, p = 0.0167) and that the difference in the marginal mean estimate was substantial ( z = 2.93; Holm’s p = 0.0034), i.e., statistically significant. As shown in Figure 6, the conformity activity in the bot condition was higher than that in the human condition. An examination of the human-/bot-condition effect with competition activity as the dependent variable showed that the estimated value for the condition effect was insignificant, i.e., − 1.43 ( S E = 2.28). No difference in competitive activity was observed; however, cooperation activity was significantly greater in the human condition, whereas conformity activity was significantly greater in the bot condition.

An analysis of the human- and bot-condition effects with the JSE as the dependent variable revealed that the estimated value of 0.0033 ( S E = 0.009) was insignificant. Meanwhile, an examination of the human- and bot-condition effects with the percentage of correct responses to the joint Simon task as the dependent variable found that the estimated value of − 0.0111 ( S E = 0.0043) was significant ( t (8.55) = 2.61, p = 0.0294). Although a trend toward a higher percentage of correct responses was observed in the bot condition, a test of the difference between the marginal estimates showed z = 1.03 ( p = 0.0535), which was not significant. The correct response rates are presented in Figure 7. The variance in the percentage of correct responses was higher in the human condition, whereas that in the bot condition was minimal. This is presumably because the bots consistently provided correct answers to the questions. In the bot condition, bots replayed pre-recorded human movements taken from correct trials, resulting in consistently correct responses for every trial.

Figure 7 presents the percent correct responses for the joint Simon task in each condition using a raincloud plot, which shows the distributions and box plots for each condition.

An analysis of the effect of the human or bot conditions on the synchrony index as the dependent variable showed the estimated value was 0.274 ( S E = 0.0035), which was significant ( t (7.73) = 3.65, p < 0.001). Additionally, the difference in the marginal estimates was significant ( z = 6.68, p < 0.001). The results for each group are illustrated in Figure 8, where lower means and higher variances for synchrony were indicated under the bot condition. The lack of synchrony with the bot may have caused this difference, depending on whether the pair was aware or unaware of the bot. However, the difference in the mean bot awareness (0,1) was insignificant. Next, we performed mediation analysis based on the bot condition to determine whether bot awareness was a mediating variable.

Mediation analysis was conducted separately to investigate the effects of activity on the joint Simon task performance under the human and bot conditions. The variables considered were the three activities and synchrony indices as predictor variables, bot cognition as a mediating variable, and the JSE and correct response rate as the outcome variables. A 32-trial average was considered for the activity and synchrony indices to align with the correct response rate and sample size.

Owing to the high correlation between the three activity indicators that served as predictor variables, we performed principal component analysis as a standard procedure to avoid multiple linearities. Two principal components were extracted when eigenvalues greater than 1 were specified. The factor-loading matrix without rotation is presented in the Supplementary Appendix. Because the first principal component separated cooperation from other activities, we named the factor score of the first principal component the collaboration factor, i.e., collabH (as shown in Figure 9) and as collabB (as shown in Figure 10) for the bot condition. As the second principal component distinguished between competition and conformity, the score for the second principal component factor was named the competition factor, as indicated by competeH and competeB in Figures 9, 10, respectively. The first principal component was converted from negative to positive values, with higher values indicating greater cooperation. The outcome variables, JSE, and correct response rate were standardized and entered, and the regression coefficients were reported as standardized coefficients.

The path coefficients from the independent variables (two activity factors and synchrony) to the dependent variable (correct response rate) under the human condition are shown in Figure 9. The significant paths revealed that the collaboration factor increased the correct response rate, whereas the competition factor decreased the correct response rate. The statistics for each path are presented in Table 2 of the Supplementary Appendix. No effect on the JSE was observed, and bot cognition was not shown to be a mediating variable. The total R 2 values for the paths to the JSE, correct response rate, and bot cognition were 0.06, 0.28, and 0.11, respectively.

The path coefficients for the same variables under the bot condition are shown in Figure 10. The total R 2 values for the paths to the JSE, correct response rate, and bot cognition were 0.37, 0.10, and 0.35, respectively. As a key pathway, collaboration factors had a substantial total effect on enhancing bot cognition and reducing the JSE ( z = − 3.17, p = 0.0015).

To further examine the robustness of the mediation effect, we replaced the binary “Bot-Notice” variable with the continuous Social Presence index described in the methods section. However, when using Social Presence as a mediator, we did not observe any significant indirect effects on either the JSE or correct response rate in either condition.

The path analysis results for both conditions are presented in Figures 9, 10 (see Figures 9, 10). These figures illustrate the standardized path coefficients and demonstrate the differential effects of collaboration and competition factors on task performance under human and bot conditions.

The total R ² values for the paths to the JSE, correct response rate, and bot cognition were 0.06, 0.28, and 0.11, respectively, for the human condition. For the bot condition, the total R ² values were 0.37, 0.10, and 0.35, respectively. As a significant path, the collaboration factor substantially increased bot cognition and weakened the JSE as a total effect ( z = − 3.17, p = 0.0015).

We confirmed that an avatar’s appearance did not affect the JSE or bot cognition. Specifically, avatar differences under the avatar condition (Sessions 6 and 7) were compared based on the classification probability as repeated factors. The results of an ANOVA indicated that the main effect of the avatar condition and the interaction effect between the avatar condition and classification were insignificant. Moreover, no significant interaction effects involving the bot avatar appearance were indicated.

In the preliminary experiment, a two-category model comprising competition and conformity was established initially. However, because the participants showed higher synchrony in competitive activities, the model was refined into a three-category classification comprising cooperation, conformity, and competition in the main experiment. The results of the main experiment revealed the feasibility of classifying joint activities based on subtle movements during Phase 3, i.e., when the participants were in motion, and in Phase 1, i.e., when the participants remained still. These results corroborate the predictions of the preliminary experiment, which identified the potential for preparing joint activities during the countdown phase. Furthermore, the results above suggest that the classification model and synchrony index used in this study were valid. A notable finding was the consistent selection of similar features for the synchrony index, which emerged as the most crucial feature for classification in both the preliminary and main experiments. This feature, which is a rotation of the unused hand, would not be readily observed by oneself or others, which suggests that behavioral synchronization phenomena appear as unconscious responses.

In the main experiment, we hypothesized that synchrony and cooperative activity under the bot condition would decrease compared with the human condition. A linear mixed model was used to analyze the effects of human and bot conditions on joint activities and synchrony indices. The results revealed a higher ratio of cooperative activity in the human condition and a high ratio of conformity in the bot condition. This finding is consistent with previous research indicating that humans tend to conform more to non-human agents when the agent’s behavior is predictable or lacks social cues. The synchrony index was significantly lower in the bot condition, indicating reduced interpersonal synchrony with bot avatars. This suggests that while bots can elicit conformity, they may not facilitate the same level of behavioral coordination as human partners.

The mediation analysis further revealed that cooperation factors had a substantial total effect on enhancing bot cognition and reducing the JSE. However, bot cognition was not shown to be a mediating variable for the correct response rate or JSE. These findings highlight the complexity of human interaction with a non-responsive avatar and suggest that while participants may recognize non-responsive avatars as joint-action task partners, this recognition does not necessarily translate into improved task performance or increased synchrony.

These findings contribute to discussions on the mechanisms underlying the Joint Simon Effect (JSE). Rather than supporting one theoretical account over the other, our results indicate that bodily synchrony engages both processes simultaneously: participants appeared to use the partner as a spatial reference point while also sharing aspects of task representation. This suggests that interpersonal synchrony can serve as a behavioral signature of these intertwined mechanisms, highlighting how spatial coding and co-representation may co-occur rather than operate in isolation.