This experiment was approved by the University of California, Davis Institutional Review Board. All participants were briefed on the experimental procedure and provided written informed consent prior to participating in the experiments.

Ten students participated in this study: 6 males and 4 females, aged 21 to 31 years old, (M = 27, SD = 3). All participants were right handed and reported to have received 5 to 8 h of sleep the previous night, mean (M = 7.1, SD = 3.6). All completed the full experiment of 20 trials as described in Section 2.1.3. Each participant was briefed on the function of the instrumentation and testing procedure. Prior to the experiment, participants filled out a demographics survey which included questions regarding consumption of caffeine, prior experience with robotic or autonomous systems, and video gaming experience. All participants were compensated at a rate of $20 per hour.

This experiment investigated how changing levels of human trust in autonomous systems are reflected in brain activity, specifically, scalp voltages measured with an EEG headset. We designed our screen based experiment using a ROS simulation shown in Figure 1A. Our custom interaction panel allowed the human participant to remotely oversee the placement of stowage onto an equipment rack by a UR5e robotic arm, ostensibly located on the International Space Station (ISS). Each participant was introduced to the fictional scenario whereby a critical maintenance task was to be performed by crew members onboard the ISS. The complete maintenance task was segregated to create a sense of interdependancey between individual elements of a broader team: (1) a procedure planning portion of the ground crew; (2) the human-robot team, comprised of the participant and the simulated robot; and (3) the onboard crew. The participant worked in collaboration with the robot to ensure the proper placement of stowage required for the onboard crew to perform the repair/maintenance task. Participants were instructed that the proper stowage placement was critical to the effective performance of the maintenance task.

Within the simulation, specific articles of stowage were represented as colored cubes. The cube colors were designed to be maximally distinguishable by participants with colorblindness. Participants were informed that the procedure planning team had developed a maintenance plan which required a sequence of cubes to be placed onto a rack. The sequence was communicated to the participant through the Placement Request panel. Using a mouse, the participant would sort the cubes from a Tool Bin by selecting the proper colored cube and placing it into the Robot Command queue. Once the queue was full, the participant clicked a button to start the robot placement sequence. The participant was informed that a planning algorithm decided how the cubes would be placed onto the rack. As a means to elicit changes in the participant's trust, the capability of proper stowage placement was modulated between two levels of placement accuracy: 90% and 30%. These levels were randomized and unknown to the participant. The two levels of accuracy ensured that each participant was exposed to the approximately the same number of correct and incorrect placements over the course of the experiment. As the robot performed the placement task, the participant was instructed to self-report their trust in the robot via a horizontal slider located directly under the On Board Camera panel. High trust was reported by positioning the slider to the right, whereas low trust was reported by positioning to the left. Trust values vary from 0 (no trust) to 1 (complete trust). Each participant was allowed to determine when, and by how much they decided to report their level of trust. Once the robot was finished placing the 4 cubes, the window was reset and a new sequence of cubes was displayed in the Placement Request panel.

It was necessary that our simulated robot was perceived by participants as an individual agent and a teammate, occupying a distinct role (Wynne and Lyons, 2018) that it performed uniquely, with capabilities that could be evaluated subjectively. In this way, the human and robot arm would be working interdependently toward the common goal. To enforce this, our simulation ensured that each placement trajectory experienced by participants throughout the experiment was unique. Each four block placement was planned independently. While placement accuracy was used to determine how many cubes would be placed in their proper positions, the order of cube placement and the trajectories of the robotic arm were calculated prior to the presentation of the cubes in the Tool Bin. Each of these trajectories were determined using a probabalistic roadmap motion planning algorithm, PRM* (Kavraki et al., 1996) via waypoints assigned above the start and finish position of each cube. No two trajectories were ever the same. As a result, participants had to rely on the robot based on both its performance and behaviors.

Participants self-report changes in trust based upon their perception of risk, task importance, and capability of the robot. Changes in self-reported trust are important in both direction and magnitude compared to previously reported levels. Previous works have similarly obtained dynamic reports of trust via periodic prompts to cue participants for a response of an increase, decrease, or no change in trust compared to the previous response (Desai, 2012; Desai et al., 2013). We assume self-report occurs when the participant has accumulated enough information about the state of the system and made the decision to report a change trust. If there is no self-report, we cannot impute a correspondence between a level of trust and EEG signal versus any other external or internal stimuli of brain activity. Self-reported trust is only a single dimensional measure of a complex concept. We do not claim that self-report trust levels are comparable between participants. The models constructed from our data are personalized, and the analysis that follows is within-subject. Despite limited research exploring the correlation between the trust slider and other established subjective trust reporting methods, it has been acknowledged (Kohn et al., 2021) that the likert or sliding scale is functionally similar to the trust item used in Lee and Moray's trust and self-confidence measure for measuring trust in automation (Lee and Moray, 1994).

A detail of the experimental setup is shown in Figure 1A. Electrophysiological data were collected from each participant using a EEG recording suite manufactured by g.tec. The system was comprised of the g.HIamp amplifier and 62 channels of active electrodes mounted into a single flexible cap. Gaze position and pupil diameter were measured using a Tobii Nano Pro screen based gaze tracker. The gaze tracker captured pixel position of gaze, pupil diameter and blinks at a sample rate of 60Hz and is optimized for screen based experiments. In addition, mouse position and button clicks were recorded. Both the gaze and mouse data were used to observe the participants' interaction with the screen-based task. Additionally, gaze was used during baseline measurements that preceded each test block, and was crucial in identifying myoelectrical artifacts in the EEG signal due to blinking. All data were synchronized and recorded using Lab Streaming Layer at their native sample rates.

Prior to the experiment, all participants received guidance on executing the human-robot collaborative task using the screen based interface in Figure 1B. In addition, a training session was provided that introduced the background of the study including the scene, where the study took place, and the goal of the task. Furthermore, detailed instructions on how to evaluate trust according to the given task was highlighted to the participants. The instructions included two major constructs: (1) We want you to report your trust as your attitude that the autonomous system will help you achieve your goals given the uncertainty and vulnerability associated with this task (and this task only) (Lee and See, 2004) and (2) Your trust may include aspects related to the performance of the autonomous system and also may include your feelings toward the autonomous system. Thus, the participants were assumed to report their trust based on the same standard.

After receiving instructions, each participant was outfitted with the 62 channel EEG headset. The active electrodes were filled with conductive gel and electrode impedance was verified to be below 5kΩ using the g.tec data acquisition software. Next, the eye tracking device was calibrated for the particular participant. A complete experiment consisted of 5 GROUPs¹ of 4 trials for a total of 20 placement trials as shown in Figure 1C. Prior to the first GROUP a baseline measure of EEG activity was recorded for approximately 4 min: 2 min with eyes open while fixating on a crosshair, and 2 min with eyes closed. The following four GROUPs starting with a similar baseline measure, but with eyes open/closed periods of 45 seconds each. After each GROUP, a 5 min rest period was provided. The total experiment lasted approximately one hour. A short video of a single trial is available (UC Davis CHPS Lab, 2022).

Our EEG headset recorded 62 channels of scalp voltages at a sample rate of 512 Hz. Two electrode clips were attached to the right and left earlobes with an additional ground electrode required for active impedance control. All EEG signals were referenced to the right earlobe. The EEG data were filtered with a zero lag 4th-order bandpass filter (2–100Hz) with an additional notch filter (58–62Hz) for AC line noise. To reduce the impact of field spread, the EEG voltage data was transformed to current source density (μV/cm²) based on a spherical spline surface Laplacian method (Perrin et al., 1989). We performed a time-frequency decomposition using Morlet wavelets (Sanei and Chambers, 2007) over a set Ω of 34 frequencies between 9.5 and 83 Hz. The time-frequency transform yielded a temporal sequence of complex values given by:

where ω_i∈Ω. We extracted frequency dependent power P = A² (μV²/cm⁴) and phase ϕ (rad) directly from Equation 1 for each of the 62 channels in the EEG headset.

As we describe in Section 2.2.1.2, functional connectivity values are based on EEG inter-channel phase difference. For the sake of longer term fidelity of EEG signal phase, we chose not to use independent component analysis (ICA) to identify and remove myoelectrical artifacts (Makeig and Onton, 2011) due to blink events, which occurred infrequently. ICA can have indeterminate impacts on the phase content of a signal over the long time course (Thatcher et al., 2020). Additionally, the separate gaze data allowed us to specifically identify the occurrence of blinks and analyze their impact on the EEG signal. As a result, we ultimately chose to exclude frequencies below 9.5 Hz due to the excessive artifacts generated from blinking.

In this study, we assumed that the state of trust was based upon the participants' perception of risk, task importance, and capability of the robot to properly perform the placement task. We additionally assume that both trust and EEG features were continuous. Therefore, our predictive models assume a consistent correlation between the m independent variables, EEG features x_i, and the dependent variable, trust T, of the form shown in Equation 5. Regressors were selected from the complete feature sets for each feature type: x_i∈X_SP for the SP model, and xi∈XNMk for the seven region-networks. We constructed a series of multivariate linear models from each feature set as follows:

where T^n is the trust estimate using regressors from one of the eight feature types, m is the total number of regressors in the model, a₀ is a bias term, α_i are the coefficients of each regressor xik, and ϵ is a random noise term. We additionally assumed that interactions between features were negligible. Personalized linear models were generated for each participant using regressors selected from the SP feature set X_SP and seven NM feature sets XNMk for a total of eight regressor sets for each participant.

A major assumption in this study was that elicited changes in trust occurred over large timescales such they may be captured by examining average EEG measures at discrete points in time. Our participants self reported their trust from 0 (no trust) to 1 (complete trust), T_m∈[0, 1]. We use the temporal characteristics of T_m to define a discrete set of time points t_r, from which to build our models. Each participant's self reported trust over the entire experiment was collected into a single ensemble.

Our method of building and testing the performance of the linear models was based on N-fold cross validation. Sample points for each participant over an entire experiment were collecting into a single ensemble. All sampled points were randomly assigned to one of ten equally sized bins: S = {s₁, s₂, s₃, …, s₁₀}. A 10% holdout, S_test, of data was set aside testing while the remaining 90% S_train was used for model building. If we define the set B = {b₁, b₂, b₃, …, b_NB} where bi={Straini,Stesti}, there are NB=mm10 unique, non-overlapping sets in B. We built and evaluated linear models using 1,000 randomly selected train/test sets b_i∈B using the MATLAB function fitlm.m. For each set b_i, we constructed n = 8 models from the ranked regressor sets: one model for signal-power, and seven models for the region-networks. Prior to model construction, all trust and EEG data were temporally aligned to ensure common time stamps between signals of different sampling rates. During the model building phase, normality of the residuals was verified using a Shapiro-Wilk test.

The regressor types, whether SP or NM, are derived from EEG signals that measure neural activity arising from the same stimulus. Consequently, we assumed that model performance for all regressor types represents an ensemble of possible predictive performance using EEG for each individual participant.

Trust prediction accuracy was evaluated using the root mean squared error between the self reported and model predicted values of trust using the test set Stesti that was unseen during the model building phase. The root mean squared trust prediction error using models built from the n^th regressor set will be denoted as ETn and is given by

where T^in is the trust estimate, T_i is the true self reported trust, and N_test is the number of points in the test set. The ETn is the standard deviation of the trust prediction error. Given the range of T_m = [0, 1], a value of ETn = 0.25 for example, would indicate that estimated trust could differ from the actual trust by 25%. Additionally, we compared adjusted R² to determine how well regressor types capture the variability in self reported trust.

Despite the presence of normally distributed residuals during the training phase, there was a high incidence of skewed distributions of ETn in the prediction results. This trend was also present for the adjusted R². Consequently, we compared the medians of ETn and adjusted R² between SP and NM feature types using a non-parametric Wilcoxon rank-sum test for equal medians with a significance level of α = 0.05.

Our study investigates the difference in trust prediction accuracy between linear models using SP and NM regressors. We evaluated model predictive power by comparing the RMS trust prediction errors as defined by Equation 6. Data in Table 3 is given as the difference in median ETn between the SP model and seven NM models for the region-networks: ΔETk=ETSP-ETNMk. Positive values of ΔETk indicate a smaller trust prediction errors for NM based models and consequently improved trust prediction accuracy. The resulting p values are given for comparisons that failed the statistical test. Figure 6 illustrates our main findings. The percent change in trust prediction error, defined as ETSP-ETNMkETSP, is given in Figure 6A. Figure 6B shows how many participants' models yielded improved trust prediction accuracy when using NM features. Counts are given for each of the seven NM region-networks.

Participants with lowest improvement were Subject 3 at 1%–5%, Subject 5 at 3%–4%, Subject 7 at 3%–7%, and finally Subject 8 at 2%–8%. The greatest improvement was Subject 6 at 55%–67% followed by Subject 9 at 19%–35%. Of the seven region-networks, only F_N showed a decrease in RMS trust prediction error for all ten participants. This was followed by CPO_N and TPO_N, which showed decreases in nine participants; FN_N and AFC_N in eight participants; and FPO_N and FTPO_N in seven participants.

In addition to trust prediction error, we calculated the coefficient of determination (adjusted R²) (Kuhn and Johnson, 2013) to compare the proportion of variance in self reported trust that was captured by models using SP and NM regressors. These results are also shown in Figure 6. Figure 6C gives the difference in median adjusted R² between the SP model and seven NM models for the region-networks: ΔRk2=RkNM2-RSP2. A positive value of ΔRk2 indicates that NM based regressors better capture variability in the self reported trust. All median differences were statistically significant with p < 0.001. Figure 6D shows how many participant's models produced higher adjusted R² values when using NM features. Counts are given for each of the seven NM regions.

Participants that showed positive increases in adjusted R² for all seven of the region-networks were Subjects 2, 6, 7, and 8. Subjects 4 and 10 had a single region that yielded a reduction in adjusted R² while Subjects 1, 5, and 9 had two regions that produced a reduction. Subject 3 had the poorest performance with five of the seven region-networks showing a reduction in adjusted R². The participants with the greatest improvements were Subjects 2, 4, and 6, with ΔR² of 0.2 to 0.45. The participants with the smallest improvements were Subjects 5, 7, and 8 with 0.05 to 0.10. Subject 3 had only two regions that showed an increase in adjusted R² at 0.05.

Of the seven region-networks, FC_N and TPO_N showed improvements in nine participants. The AFC_N region-network showed improvements in seven participants, while the remaining region-networks showed improvements in eight participants.

Patterns in trust reporting behavior and model performance are summarized in Table 4. The first row orders participants based upon their average ΔRMS trust prediction errors across all region-networks from lowest to highest. Similarly, the second row orders participants by their average ΔR² across all region-networks from lowest to highest. Finally the third row orders the participants by their ΔTmΔt IQR from highest to lowest. Subjects 3, 5, 7, and 8 are in the bottom half of ranked performance measures. In addition, Subjects 3, 5, 7, and 8 are in the top half of ranked ΔTmΔt. Participants who reported trust more frequently and with larger magnitudes had the lowest performing models using EEG-derived features of both types.

From Figure 8A, the power region with the highest number of regressors is the CP_P at 28%, followed by the F_P at 22%, OP_P at 18%, and then the FC_P and T_P regions at 15% and 10%, respectively. As shown in Figure 9A, when power features are separated by frequency band, the CP_P and F_P regions remain the top contributors, accounting for 20%–26% of regressors across the Alpha through Lower Gamma bands. In the Upper Gamma band, however, the most important regressors shift to the OP_P region, followed by the CP_P.

In Figure 8A, the network-metric types with the highest count are A_N at 28% followed by T_N at 24% and E_N at 20%. Modularity (M_N) and Density (D_N) were selected at 16% and 12% respectively. When additionally separating the metrics by frequency among the highest performing region-networks, A_N and T_N are within the top three selected metrics for all regions and bands with the exception of CPO_N in the Alpha band as seen in Figure 8B. Similarly, Efficiency is within the top three for all but the FC_N in Lower Gamma, and the CPO_N in Beta and Upper Gamma.