A total of 12 individuals were enlisted in the study; however, data from one individual were excluded because of technical problems in registering keyboard responses. Therefore, this study analyzed 11 individuals who were all students (six undergraduates, three pursuing master’s degrees, and two enrolled in doctoral programs) at the Tokyo Institute of Technology. Two participants did not participate under the no-window condition. The participants had an average age of 23.9 years, with a standard deviation of 2.4 and an age range of 19–27 years. Among the participants, two were women and nine were men, including three Japanese, six Korean, and two Chinese. This study was announced to undergraduate students during their classes, and master’s and doctoral program students were informed through their advisors via email. Only participants who could read Chinese characters were recruited as we used characters with a consistent radical character (please refer to the section on stimuli). Each hour of participation in the experiment earned the participants a 1,000-yen Amazon gift card. All participants had normal or corrected-to-normal vision and provided informed consent before participation. The experiments were conducted in accordance with the guidelines of the Declaration of Helsinki. An institutional consent form concerning consent to publish was obtained from all study participants. The Tokyo Institute of Technology Ethics Review Committee for Epidemiological Research approved this study. Data were collected between April 26, 2022, and December 15, 2022. Subsequent to this period, statistical analyses and evaluations were performed to address the research questions.

Figure 3 illustrates the experimental setup. The participants completed the experiment in a dark room where they were presented with visual stimuli through an LCD (24 inches, XL2411t, BenQ Corp., Taipei, Taiwan, frame rate:144 Hz; viewing distance:60 cm). To minimize head movement, the participants were instructed to position their chin and forehead on a chin rest. The delay from eye movement to the monitor’s feedback was determined by the eye tracker’s sample delay (mean < 1.4 ms, SD < 0.4 ms), processing time (updated every 1 ms), and input lag on the display. Participants responded using a numeric keypad on a keyboard (SK-8825; Lenovo Group Ltd., Beijing, China). The EyeLink 1000 Plus (SR Research Ltd., Ottawa, Ontario, Canada) recorded the participant’s left-eye gaze with a spatial resolution of <0.02° and a sampling rate of 2,000 Hz. Calibration and validation were conducted using a nine-point procedure with a validated accuracy of <1°. After each session, participants took a break and underwent recalibration and validation. The software program for the experiment was modified from the GCWINDOW template provided by SR Research, Ltd.

In the visual search array, the characters were black (luminance: 4.71 ± 1.32 c d / m 2 ) against a gray background (luminance: 30.41 ± 3.15 c d / m 2 ). Each character was 3° in size, with horizontal and vertical distances of 11° between adjacent characters (Figure 4A). Blurred image (Figure 4B) was produced by applying a Gaussian filter to one image using Adobe Photoshop 2020. A low-pass filter with a standard deviation of 10 pixels (equivalent to 0.2°) is implemented. The signal was reduced by 3 dB at a spatial frequency of 0.57 cycles/°, and the attenuation became stronger at higher spatial frequencies. By applying a Gaussian filter, the luminance of the blurred Chinese characters increased (15.27 ± 3.76 c d / m 2 ), while the gray background’s luminance remained consistent.

A dynamic 10° square gaze-contingent window was centered on participants’ fixation locations and adjusted according to their eye movements (Figure 4C). The edge of the window was 5° vertically or horizontally from the fixation location. The image (Figure 4A) appeared only within the central moving window, whereas a blurred image (Figure 4B) was displayed in the surrounding external area (Figure 4C). The image was changed from the inner to the outer side of the square window using a step function. An example rendered image is shown in Figure 4D. Despite designing the display to allow reading of only one character through direct fixation, participants could adjust their gaze strategically to view two characters simultaneously. Nonetheless, our analysis suggests that such a strategy was not utilized by the participants (refer to Supplementary material S1 for details).

Nine Chinese characters were randomly arranged in an array according to the following three rules: First, each character had a Chinese character “fish” as a radical character, ensuring all characters belonged to the same meaning category (fish) to avoid meaning-or category-based attentional bias. Second, the characters had the same stroke count to ensure consistent spatial frequency, contrast, and complexity. Third, the central character matched one of the other eight characters with a 50% probability (target or non-target). Such uncertainty was implemented to prevent participants from guessing the target’s location before completing the visual search, which could influence their behavior. The minimum correct response rate for the tasks was 98%, demonstrating that participants actively participated in each trial without skipping (the worst error rate per individual was 2.1%; see Supplementary material S2 for details). A total of 239 Chinese characters (see Supplementary material S3) met the first two criteria. A total of 540 image files were created using preprogrammed random image-generating Unity software. We coded the software to specify the existence and location of the target using a random number generator function. As we did not sample the stimuli, the target existence and location probabilities were uneven. Of the 540 images, 272 were no-target arrays and 268 were target arrays, with varying target counts at each of the following eight locations (clockwise from the middle top): 34, 29, 32, 32, 35, 40, 37, and 29.

As shown in Figure 5, each trial commenced with a drift check, followed by a 650 ms display of blurred images, before initiating the visual search. This predetermined duration, which was consistent across all trials, prevented participants from expecting any deliberate temporal discrepancies. During the 650 ms waiting period, the participants were instructed to fixate on the center of the monitor. The visual search concluded when the participants identified and reported the target’s location and pressed the enter key. Subsequently, they completed the authorship questionnaire to assess whether they perceived the window movements as aligned with their gaze (self), temporally delayed (delay), or controlled by another agent (other). They then rated the degree to which they believed their eye movements influenced the window’s movement on a six-point scale, where six represented “I fully manipulated the window” and one indicated “I could not possibly manipulate the window”—this was the authorship rating questionnaire.

All statistical analyses were performed on the average values for each condition, representative of each participant.

To investigate the delay’s impact on the sense of agency, our study’s first question, we utilized two generalized linear mixed models (GLMM; SPSS 24.0) to analyze the 𝜇 value’s effect on categorical authorship reports. Specifically, for trials involving delays—excluding playback and no-delay trials—we compared “self” to “delay” responses and “other” to “delay” responses, using a multinomial probability distribution and a generalized logit link function. The participant served as a random effect, with variance components as the covariance structure, to capture unexplained variance, with 𝜇 value treated as a fixed effect. Due to data loss, four responses were excluded, resulting in 2,306 responses analyzed. For mean authorship rating scores, we conducted a one-way ANOVA with repeated measures on delayed trials. Mauchly’s test assessed sphericity, with degrees of freedom adjusted using Greenhouse–Geisser estimates.

To address our second question on the impact on participants’ behavior, we first analyzed response times. Response time was assessed as a measure of task complexity. The duration from stimulus onset to the registration of the participants’ final input was recorded. Previous inputs were disregarded as repeated registrations frequently occurred when the participants needed to reposition their hands on a keypad. We conducted a target × delay repeated-measures ANOVA on the response time under the delayed conditions. The same procedure was applied to adjust degrees of freedom. Post-hoc analyses were conducted using Bonferroni correction for pairwise comparisons. For eye movements, we examined fixation counts per trial, fixation counts per second, relative distribution for fixation duration, and saccade amplitude, all in relation to the 𝜇 value. we set the minimum thresholds for eye movements to 0.1°, 30.0°/s, and 8000.0°/s² to identify saccades. Any fixations that occurred outside the stimuli, began immediately before or after a blink, or lasted less than 120 ms were excluded from our analyses. We determined the relative distribution of fixation durations in 25 ms intervals within a 0–800 ms range and the distribution of saccade amplitudes in 0.5° intervals within a 0–15° range. Given the variability in response times across trials and participants, we opted for relative distributions. For the fixation counts per trial, we conducted a one-way (delay) repeated-measures ANOVA under the delayed conditions. Mauchly’s test assessed sphericity, with degrees of freedom adjusted using Greenhouse–Geisser estimates.

Our third aim was to identify the statistical attributes (e.g., minimum, mean, or maximum values of the distribution) that had the greatest influence on the sense of agency and eye movements. We compared our present results (i.e., the categorical authorship report and fixation count per second) with those of our previous experiments, in which a constant delay ranging from 0 to 4,000 ms was implemented (Kim and Yoshida, 2023). In the previous study, the same gaze-contingent window, as well as the no-window and playback conditions, was applied as in the current study. For comparison, we utilized data from delayed conditions ranging from 0 to 400 ms, which were retrieved from our previous publication on the open science framework (more details on the methods and results from the previous experiment can be found in Supplementary material S4). We only considered the mean (𝜇) value in the jittery distribution implemented in the current study, which represents the minimum value of that jittery distribution. If the comparison did not reveal a significant difference between the two experiments, we concluded that the minimum value was the most influential attribute.

With fixation count per second, we conducted two-way mixed ANOVA for the five delayed trials (0, 100, 200, 300, and 400) with delay as a within-subjects factor, and experiment type (constant and jittery) as a between-subjects factor. With categorical authorship reports, using the least-squares method, we fit a sigmoid function to the proportion of each of the two categories in the authorship report (self and delay). The other responses were omitted, as they were barely reported across both experiments. For each experiment, we fit two regression curves for each participant, yielding 38 curves. The sigmoid function Y = 1 / 1 + e a x − b was defined, with a and b as its two adjustable parameters. The b indicates the delay at which the response proportion reaches the 50% threshold, whereas a indicates the slope of the threshold. A positive value of a represents an increasing slope, whereas the opposite is true for a negative value. Removing a fit with a R 2 less than 0.70 resulted in the exclusion of one delay response curve from the current jittery delay experiment, leaving a total of 37 curves for comparison. We chose to calculate the 50% threshold, which is considered a reliable statistical measure of the curve (Sato, 2008; Shimada et al., 2010; Farrer et al., 2013) as it represents the alteration in the subjective sense of agency. Moreover, the slopes reflected the degree of uncertainty in the participants’ responses, with steeper slopes indicating higher certainty. Consequently, we fitted a sigmoid function as previously described by Farrer et al. (2013). With fitted slope values ( a ) and threshold delay values ( b ), one-way multivariate ANOVA was conducted for the delayed trials with the experiment type as a between-subjects factor.

The categorical authorship report includes three levels of responses (self, delay, and other). For each condition, we computed the mean and standard deviation of the proportion of each response type, as shown in Figure 6. The GLMM results revealed a significant main effect of delay in the comparison between “self” and “delay” responses (b = −0.018, SE = 0.002, t = −8.913, p = 0.000), indicating a less “self” than the “delay” responses with increasing 𝜇. All parameters and coefficients for the GLMM are available in Supplementary Table S4. When comparing the “other” response to the “delay” response, the main effect of the delay was not significant (b = 0.000, SE = 0.002, t = 0.122, p = 0.903) while only the intercept showed significance (b = −3.387, SE = 0.507, t = −6.683, p = 0.000), indicating the infrequent choice of “other” response in delay condition. All parameters and coefficients for the GLMM can be found in Supplementary Table S5.

The means and standard deviations for each fitted slope value ( a ) and threshold delay value ( b ), across each categorical authorship response in both previous (Kim and Yoshida, 2023) and current experiments, are detailed in Table 1 (for a visualized graph, see Figure 7). Note that the “self_jitter” and “delay_jitter” in Figure 7 correspond to the average proportion of the “self” and “delay” responses in Figure 6. The multivariate ANOVA, assessing the impact of experiment type (constant vs. jittery), did not yield a significant main effect (F [4, 13] = 0.459, p = 0.764, partial η 2 = 0.124). All parameters for the MANOVA are available in Supplementary Table S6.

The authorship rating with a scale of one to six gages the subjective authorship, where one implied “I could not possibly manipulate the window” and six represented “I fully manipulated the window.” The averages and standard deviations of the ratings were calculated for each participant (Figure 8). Mauchly’s test indicated that the assumption of sphericity had been violated ( χ 2 [20] = 99.822, p = 0.000). Therefore degrees of freedom were corrected using Greenhouse–Geisser estimates of sphericity (ε = 0.217). The main effect of the delay was significant (F [1.303, 13.025] = 65.974, p = 0.000, partial η 2 = 0.868), suggesting that delay affects the subjective authorship of the gaze-contingent window. All parameters for the ANOVA are available in Supplementary Table S7.

Both the mean and standard deviation of the response times were calculated based on each participant’s average response time under each condition and in the presence or absence of the target (Figure 9). Mauchly’s test indicated that the assumption of sphericity had been violated for the delay ( χ 2 [20] = 44.342, p = 0.002) and the interaction ( χ 2 [20] = 50.915, p = 0.000). Therefore degrees of freedom were corrected using Greenhouse–Geisser estimates of sphericity (ε = 0.289 for delay, ε = 0.399 for the interaction). The main effect was significant for the delay (F [1.734, 17.341] = 33.814, p = 0.000, partial η 2 = 0.772) and target (F [1, 10] = 23.471, p = 0.001, partial η 2 = 0.701), whereas the interaction was not significant. These results indicate that the response time is affected by temporal delay. All parameters for the ANOVA are available in Supplementary Table S8. A post hoc pairwise comparison with Bonferroni correction found that the response time significantly increased by a mean difference of 1,511 ms (p = 0.001) when the target-absent condition (M = 6,674 ms) was compared to the target-present condition (M = 5,163 ms), suggesting that our gaze-contingent display forces a visual search in a serial manner.

The fixation counts per trial were presented per participant (Figure 10) as the counts largely varied across participants. Mauchly’s test indicated a sphericity assumption violation for the delay ( χ 2 [20] = 53.481, p = 0.000), hence the Greenhouse–Geisser estimates were used to adjust degrees of freedom (ε = 0.292). Delay did not yield a significant main effect (F [1.751, 17.512] = 0.774, p = 0.460, partial η 2 = 0.072). All parameters for the ANOVA are available in Supplementary Table S9.

The mean fixation count per second and its standard deviations were calculated for both the current experiment (jitter delay, N = 11) and the previous study (constant delay, N = 8, Kim and Yoshida, 2023) (Figure 11). The experiment type did not yield a significant main effect (F [1, 17] = 0.961, p = 0.341, partial η 2 = 0.053). Delay showed a significant main effect in both experiments (F [1.473, 25.034] = 90.652, p = 0.000, partial η 2 = 0.842) where Mauchly’s test indicated that the sphericity violation ( χ 2 [9] = 52.956, p = 0.000) hence the degrees of freedom were corrected with Greenhouse–Geisser estimates (ε = 0.368). All parameters for the ANOVA are available in Supplementary Table S10.

The mean relative distribution of fixation duration was calculated for the 11 participants (Figure 12). In the no-window condition, the distribution mode appeared at 150 ms. Considering that the no-delay and playback conditions had similar modes at 200 ms, this seems to be due to the gaze-contingent window. As the delay was introduced, the mode shifted to 225 ms. Interestingly, from when the 𝜇 was 50 ms, we observed dual modes, one remaining around 125–225 ms, and another mode located at the position of the sum of 𝜇 and 200 ms (e.g., when the 𝜇 was 300 ms, the latter mode located at 500 ms).

The mean relative distribution of saccade amplitudes was again calculated among the 11 participants (Figure 13). Across all conditions, two distinct modes were observed: one at 1° and the other at 7°. Given that the shortest distance between the two characters was 8°, this likely explains the peak at 7°. Relative frequency below 1° seemed to rise with the gaze-contingent window when we compared no delay or playback to no window condition and further increased with the 𝜇.

Categorical authorship reports showed that the self responses decreased as the delay increased. The dissociation between “delay” response frequency and response time in playback conditions and the predominance of other responses in the conditions suggests that the “delay” responses are more closely associated with the subjective sense of agency than with task difficulty. Likewise, the consistent authorship rating between one and two for playback conditions suggests that these ratings likely indicate participants’ perception of the authoring agent rather than task difficulty. Predominantly reported delay response (i.e., partial sense of agency) led to a more pronounced decrease in the proportion of the self-response than the decrease in the authorship rating where the rating of one implied total absence of the sense of agency. The decrease in authorship ratings with temporal delay aligned well with prior research that utilized hand movements as the primary action (Sato and Yasuda, 2005; Ebert and Wegner, 2010; Wen et al., 2015a,b; Shibuya et al., 2018). Specifically, our categorical authorship reports mirror those of Farrer et al. (2008) in two key aspects: (1) self responses decreased and delay responses increased alongside the temporal discrepancy, and (2) other responses were barely reported even at longer temporal discrepancies.

However, Farrer et al.’s (2013) study found participants misattributing authorship and selecting other responses with longer temporal discrepancies. These misattributions can be accounted for by the degrees of freedom in the action–feedback experimental paradigm. Both the current study and Farrer et al.’s (2008) study allowed participants to incorporate movement direction into their authorship judgments (gaze direction and joystick manipulation respectively). Farrer et al.’s (2013) study offered only a single degree of freedom (a single-button press), making it challenging for participants to claim authorship at extended temporal discrepancies. Such variations in sensory data across studies might further explain the disparities in the attenuation degree of subjective reports. For example, Farrer et al.’s (2013) study documented that an increase in the partial control response proportion began at longer temporal discrepancies than in our study. Wen (2019) suggested that copious sensory data would constrict the time window for a sense of agency. These disparities do not notably hinder our assertion that our study generally replicates prior research, even with the use of eye movement as the action modality.

The fixation counts per trial showed no influence from the 𝜇 value, while the fixation counts per second exhibited a consistent decrease. This suggests that the number of fixations required to complete the visual search remained stable across different delay conditions, despite the delays disrupting their visual search time efficiency. Such a suggestion is further supported by the observed increase in response times. The observed decrease in fixation counts per second can be detailed by examining the fixation duration distributions. These distributions reveal two distinct patterns: (1) a shifting mode (hereafter referred to as “higher mode”) and (2) a consistent mode between 125 and 225 ms (hereafter, “lower mode”). The location of the higher mode closely correlates with the delay, moving in tandem with the 𝜇 value. Subtracting the 𝜇 value from the location of each higher mode across all delay conditions shows that all mode locations converge at 200 ms, except when 𝜇 is 0 ms, where it converges at 225 ms. This correlation likely reflects participant behavior waiting for the gaze-contingent window to update at their fixation point before reading the character, termed “wait-and-read” for the higher mode and identified as typical reading behaviors for the lower mode.

The distributions of the saccade amplitudes showed an effect of delay as the increase in the relative frequency of the 0–1° saccades. While we maintain some reservations about our camera-based eye-tracking capability to accurately record microsaccades, we can extrapolate the root causes of our findings from previous research on microsaccades. Previous research has proposed microsaccades as a proxy for covert attention allocation (Hafed and Clark, 2002; Engbert and Kliegl, 2003). We postulate that the separation between eye movement and attention allocation is a fitting explanation for our delayed gaze-contingent display. When the window tracks eye movements with a delay, it causes a distance between the participant’s gaze point on the monitor and the window. Such distance may necessitate participants to direct their attention to the window as part of their temporal mismatch perceptions while their gaze remains on the desired character to perform a visual search. As the delay increases, causing the distance between the window and their eye movement to expand, their frequency of covertly attending to the window increases.

Even though the action-effect delay could not significantly hinder the participants’ visual search, it did utilize their attentional resources. Although the low error rates and negligible influence of the μ value on fixation count per trial indicate a trivial impact of the delay on search behaviors, the variation in saccade amplitudes in response to 𝜇 values suggests an adaptive attention distribution and needs further exploration. Decreased visual search efficiency suggests higher attentional demands on participants’ working memory. The adoption of two fixation strategies indicates that participants chose to “wait and read” to reduce these demands. Those eye movement results hint at a possible proportional relationship between the amount of attention consumed and the length of the delay. However, this relationship cannot be confirmed since the delay did not affect task performance. Although attention is required for action-effect comparison (Wen et al., 2016) and is a limited resource essential for experiencing agency (Hon, 2017), our study, showing sustained search performance and a steady decline in subjective authorship, indicates that visual search and the sense of agency adequately shared the available attentional load. These findings underscore the importance of further manipulations concerning task difficulty.

Comparison with constant delay conditions (Kim and Yoshida, 2023) revealed no significant differences in either the subjective report parameters or the fixation counts per second. The relative distributions of fixation durations and saccade amplitudes also did not show any noticeable impacts. These results suggest that the minimum value of the distribution predominantly influences the sense of agency and eye movements. In generalizing our findings, qualitative differences in fitted values should be noted. Plots of fitted values per participant (Supplementary Figures S8, S9) indicate that jittery delay may result in less uncertainty and require shorter delays to impair the sense of agency. Further research is needed to validate our findings. Furthermore, these conclusions are constrained by our choice of distribution, specifically σ set at 50 ms.

Direct comparisons with previous studies are scarce due to methodological differences, such as continuous action–effect interactions and the application of jittery distributions across delay ranges. Normoyle et al. (2014) provide a partially relevant comparison by manipulating the θ value of a gamma distribution to achieve variances of 0, 50, 100, and 150 ms, with a constant mean of 200 ms. In their study, participants perceived delays only at the highest variance of 150 ms. Our study employed a truncated normal distribution with a variance of 628.29 ms across all delay conditions, identifying a subjective report threshold of 50% at a delay of 94 ms, the distribution’s minimum value, with a mean delay of 130.14 ms. While direct comparisons with gamma distributions are challenging, our findings suggest a broader variance in threshold delay (mean = 130 ms, variance = 628 ms, min = 94 ms, max = 194 ms) compared to previous studies (mean = 200 ms, variance = 150 ms). In contrast with our truncated distribution, previous studies likely encountered occasional longer delays (e.g., over 400 ms) due to low-probability extreme delays, which could explain the perceived temporal delay at a smaller variance. This discrepancy raises questions about whether perceptions of temporal mismatch are more influenced by delays near a distribution’s minimum rather than its maximum value, prompting the need for future research with a more extensive range of delays.

The above experiments suggest that the threshold for perceiving delays may depend on both the length of the delay and its frequency of occurrence. The sporadic extended delay is critical when considering data packet transmission over the internet; an extended delay in one packet often predicts similar delays in subsequent packets (Bolot, 1993). Furthermore, packet loss tends to occur in clusters, affecting consecutive packets (Jiang and Schulzrinne, 2000). In the realm of continuous action-effect HCI, even brief experiences of such sporadic, consecutive extended delays can significantly impair the sense of agency and potentially alter user behavior. Therefore, future research should not only encompass experiments with a wider range of delays but also investigate the effects of varying occurrence rates of the delays.

Low error rates and significant response time differences in target presence conditions confirm participants navigated the visual search in a serial manner without any skipping, experiencing the designed inconvenience of jittery delays. The influences of delays on subjective reports and behaviors validate our design approach and underscore the efficacy of jittery delays. Also, we observed no decrease in the overall saccade amplitude, consistently recording 6–8° saccades across all conditions. The 3 × 3 virtual grid utilized in our experiment was considered to prevent reductions in saccade amplitude caused by the gaze-contingent window, thereby distinguishing our methodology from those previous researches where saccade amplitudes are reported to decrease with peripheral blurring gaze-contingent window (Reingold et al., 2003; Foulsham et al., 2011; Laubrock et al., 2013; Cajar et al., 2016a,b). Hence, the eye-gaze interface that we implemented effectively minimized the effect of the window, allowing us to focus solely on investigating the effects of delay.

This study has certain limitations, such as a participant pool constrained to a specific age bracket and the use of a specific distribution. An explicit questionnaire may not fully capture the sense of agency (e.g., the feeling of agency proposed by Synofzik et al., 2008). In addition, subjective tools, such as rating scales and response criteria, have inherent limitations and are often prone to systematic bias. Utilizing intentional binding with the time interval estimates paradigm (Humphreys and Buehner, 2009), or the Libet clock paradigm (Haggard et al., 2002) is another option. However, in these cases, the chronostasis effect, in which perceived time is dilated immediately after a saccade, is problematic when designing an experiment (Yarrow et al., 2001). Furthermore, investigating jittery delay in a continuous action-effects system would be impossible with intentional binding paradigms. Future work could potentially benefit from objective measures using signal detection frameworks. Also, the altered eye movements in a temporally delayed gaze-contingent display may not be triggered by other types of stimuli (e.g., natural scenes).

While this study provides valuable insights into the impact of jittery delays on gaze-contingent HCI, it does not directly explore the origins of the sense of agency, particularly whether it stems from internal comparisons between intentional oculomotor signals and visual feedback or from comparisons within various contexts (e.g., within visual feedback or between goals and results). This question remains a theoretical limitation, extending beyond the scope of our objectives yet important for understanding the full spectrum of factors influencing the sense of agency in eye-gaze HCI. Establishing a baseline condition, such as using transcranial magnetic stimulation to induce unintentional actions (Haggard et al., 2002; Haggard and Clark, 2003), would be critical for addressing this aspect comprehensively, posing a challenge for future research. This limitation underscores the need for further investigation to delineate the mechanisms of delay perception and their implications for the design and evaluation of gaze-contingent interfaces.