The approach to ascertain the user’s distance from the screen and subsequently project the gaze vector onto the computer screen involves the computation of the matrix ^S T _G . The transformation matrix ^S T _G allows the transformation of the gaze vector ^G g ∈ R 3 from the gaze coordinate system G to the screen coordinate system S (Eqs 2, 3). g S = T G S ⋅ G g (2) = R G S t G S 0 T 1 λ G g ̂ (3) = R G S λ G g ̂ + t G S (4) The gaze unit vector, denoted as g ̂ G ∈ R 3 , and the scalar λ ∈ R , project the gaze vector onto the screen. To find λ such that the gaze vector g ̂ G intersects with the screen, the scalar product of the screen’s z-axis and the vector located on the screen’s plane must be zero (as shown in Eq. 5). z T G λ G g ̂ − t S G = 0 (5) z G = R S G ⋅ 0,0,1 T (6) For a clearer understanding of this equation, please refer to Figure 2, where the dashed green arrowheads represent the vectors involved. The point p corresponds to the point on the screen where the person is looking. The vector t S G originates from the transformation matrix T G S , which we will determine in the regression problem discussed in Section 3.3.3. The scalar product with the z-coordinate of the screen coordinate system becomes zero when the vector λ G g ̂ − t S G lies on the plane of the screen, allowing us to determine the scalar λ. Eq. 6 rotates the z-axis ([0,0,1]) in screen coordinates to the gaze coordinate system, where R S G follows from the definition of Figure 2. With that we can compute the scaling factor λ from Eq. 5. λ = z T G t S G z T G g ̂ G (7)

As mentioned earlier the “gaze-estimation-adas-0002” model takes as input the two eye images and the head rotation angle and delivers a gaze unit vector, whereby the coordinate system of the gaze vector is rotated so that its x/y-plane is parallel to the x/y-plane of the camera. If the webcam is mounted on top of the screen and the x/y-plane of the webcam coincides with the x/y-plane of the screen we can determine the rotation matrix R G S . R G S = − 1 0 0 0 − 1 0 0 0 1 (8) Therefore, the only unknown is the translation vector t G S ∈ R 3 in Eq. 4. This translation vector is part of the homogeneous transformation matrix, which is determined through a regression analysis (Section 3.3.3).

To gather data for the regression problem, we initiated a calibration process in which a user is presented with points on the screen and instructed to direct their gaze towards these points, which is the standard method. Our approach simplifies the calibration process, involving the presentation of just four calibration points, denoted as ^S g ̃ i ∈ C , on the screen. These calibration points, serve as the user’s focal points. Each point is observed for approximately 2 s, yielding multiple data points for the regression problem from a single point of focus. Transitioning to a new point of focus may introduce a brief lag in gaze redirection, leading to instances where the user does not precisely follow the presented point on the screen. Hence, in the data stream, any erroneous data points, arising when the user gazes at positions different from the presented point, are eliminated prior to the application of the regression model. It is important to note that users were directed to maintain a steady head position during the calibration process, despite not employing a headrest.

With a set C of known calibration points ^S g ̃ i ∈ C displayed on the screen and the gaze vector g ̂ G provided by any gaze vector estimation model, we can determine the translation vector t G S by formulating a regression problem in the following way (Eqs 9–12). min x ∑ i C ‖ S g ̃ i − g i S ‖ 2 2 (9) with  g i S = S R G λ i G g ̂ i + t G S (10) λ i = z T G ⋅ − R G T S t G S z T G g ̂ i G (11) x = t G x S , t G y S , t G z S T (12) The coordinates t G x , y , z S represent the x, y, and z coordinates of the vector t G S . g i S is derived in Eq. 4, while λ ⁱ is obtained from Eq. 7. Given the knowledge of the calibration points’ poses and thus the distances between these points on the screen, we have both a physical reference and a direction provided by the set of gaze vectors ^G g ⁱ . Leveraging this information, we can determine the 3D location of the gaze coordinate system G through the proposed regression problem. Subsequently, we can ascertain the homogeneous transformation matrix T G S and project the unit gaze vector g ̂ G onto the computer screen, as detailed in Section 3.3.

The accuracy of the gaze vector model, in this case, the OpenVino gaze vector, is constrained by inherent limitations. Notably, the accuracy of the gaze vector provided by OpenVino exhibits a Mean Absolute Error (MAE) of 6.95° OpenVINO (accessed 2023). This inaccuracy became apparent in the experiment, where four points forming a rectangular pattern were presented on the screen. Examination of the raw gaze data from the OpenVino model shows that the recorded coordinates do not align with the expected rectangular pattern (Figure 3) that were presented as calibration points on the screen. This occurs because variations in lighting conditions, such as light sources positioned at the side, can result in reflections in the eye, potentially distorting the output.

However, during the calibration process we are able to reduce this inaccuracy, which we will explain next. A homogeneous transformation matrix, incorporating a gaze vector as the translation vector from the calibration, yields an accurate projection for a specific calibration point, albeit limited to that point alone. Consequently, we derive additional transformation matrices from calibration points, each accurate only for its respective calibration point location. Instead the transformation matrix determined in the previous Section 3.3.3 provides a comprehensive estimation across all points. The subsequent equations provide these additional estimations (Eqs 13, 14). T p S = I t p S 0 T 1 (13) T G p = R G S − R G T S λ G g ̂ 0 T 1 (14) T G S = T p S T G p (15) p ∈ P is the set of calibration points p and g ̂ G is the unit gaze vector from a person looking at this point during the calibration process. t p S are the calibration points displayed on the screen during the calibration process. The scaling factor λ is computed over Eq. 7 by using t S G from the transformation matrix determined over the regression process. The matrix T G S computed in Eq. 15 transforms the unit gaze vector for this specific point to the screen with zero error.

In total, this procedure provides n + 1 transformations T G S , with n for the number of calibration points and an additional one determined through the regression problem, providing the distance between the user and the computer screen.

To determine the final gaze point on the screen, we first compute an initial gaze point on the screen through the transformation matrix determined through the regression analysis in Section 3.3.3, which will provide an initial guess. We improve this guess by computing the euclidean to the known calibration points on the screen and choosing as additional transformation matrix, the one closest to this calibration point. As an approximation we define the median between these two gaze estimation points as the final point on the screen.

Additionally, we propose a framework to compensate for head movements, specifically where a user moves to the left or right in front of the screen. To consider these movements we make use of structure from motion.

Structure from Motion (SfM) is a powerful technique in computer vision that aims to extract three-dimensional (3D) information from a collection of two-dimensional (2D) frames. One of the fundamental problems in SfM is to simultaneously estimate multiple crucial parameters from a set of point correspondences between two images (Hartley and Zisserman 2004). These parameters include the 3D coordinates of points in space ( ^W p), the relative motion of cameras ( ^W R, ^W t), and the intrinsic properties of the cameras (K ₁, K ₂). The intrinsic properties of the camera are determined over the OpenCV camera calibration library (OpenCV, 2023). For feature matching between consecutive frames we use the OpenVino “facial-landmarks-35-adas-0002” library (Intel Corporation, 2023). With these features we determine the essential matrix E and recover the rotation matrix and translation vector between two frames. By triangulating the points we are able to get 3D points in the world/camera coordinate system. However, we do not know the scaling factor, because there is no reference distance in the video stream. Therefore, we formulate a complete regression problem to determine the homogeneous transformation matrix ^W T _G (t). Moving the head to the left or to the right shifts the gaze coordinate system G. With SfM we are able to determine a unit translation vector of the transformation matrix ^W T _G (t) from the world coordinate system W to the gaze coordinate system G. As mentioned earlier, given the unknown scaling factor, we will derive the gaze vector g S ( t ) on the screen using the following equations (Eqs 16–18). g S t = S T W ⋅ W T G t ⋅ G g t (16) = R W S t W S 0 T 1 R G W μ W t ̂ G t 0 T 1 λ G g ̂ t (17) = R W S R G W λ G g ̂ t + R W S μ W t ̂ G t + t W S (18) With μ ∈ R and λ defined in Eq. 7, we compute t S G over the following equation. To enhance readability, we will hereafter omit explicit time dependencies in the variables: T G S = S T W T G W (19) R G S t G S 0 T 1 = R W S t W S 0 T 1 R G W t G W 0 T 1 (20) = R W S R G W R W S t G W + t W S 0 T 1 (21)

With Eqs 19–21 we can compute t S G in the following way (Eqs 22, 23). t G S = R W S t G W + t W S (22) t S G = − R G T S R W S μ W t ̂ G + t W S (23)

^W t ̂ G is determined over SfM (Section 3.3.5) and with the following regression model (Eqs 24–27) we determine μ and t W S . min x ∑ i C ‖ S g ̃ i − g i S ‖ 2 2 (24) with g i S = R W S R G i W λ i G g ̂ i + R W S μ W t ̂ G i + t W S (25) λ i = z T G ⋅ − R G T S R W S μ W t ̂ G i + t W S z T G g ̂ i G (26) x = μ , t W x S , t W y S , t W z S T (27) t W x , y , z S are the x, y, z coordinates of the vector t W S . By solving this regression problem and knowing the parameters in x we can compute the transformation matrix T G S in Eq. 19. To collect data, in order to, solve the regression problem we perform the calibration procedure described in Section 3.3.2. The methods presented in this work have been implemented in Python and are publicly available on GitHub (Falch, 2023).

For this comparison we use the same trial as described in the previous Section 4. Given that both the GazeRecorder software and our method rely on the webcam, we had to conduct an additional trial for GazeRecorder, as the webcam cannot be simultaneously used by both applications. In the experiment, the user was positioned in front of a computer screen at a distance of 800 mm. The screen had a resolution of 2560 × 1440 pixels and physical dimensions of 597 mm × 336 mm.

To ensure accuracy, we synchronized the recording of both webcam-based gaze tracking solutions with data from the Tobii Eye Tracker 5. In Figure 4, the trajectory of the presented target is depicted in blue, serving as the ground truth. The trajectory recorded by the Tobii Eye Tracker 5 is represented in orange, while our method in combination with the OpenVino model is shown in green and our method in combination with ETH-XGaze dataset is shown in black. Both the OpenVino model and the model trained on the ETH-XGaze dataset yield distinct outputs, both of which serve as inputs in our method for projecting the unit gaze vector onto the screen. This highlights the critical importance of an accurate unit gaze vector for our method. Despite their differences, the overall error along this exemplary trajectory between the two models is comparable (Table 1). It is worth noting that the number of samples collected by the Tobii Eye Tracker 5 surpasses that of the webcam based solutions.

In Figure 5, we present a comparison between GazeRecorder, Tobii Eye Tracker 5, and the ground truth. The color scheme in this figure corresponds to that in Figure 4 with the green trajectory for GazeRecorder. Notably, GazeRecorder exhibits a lower sampling rate compared to the Tobii Eye Tracker 5. This lower sampling rate can be attributed to the webcam’s lower frame rate, which affects the rate at which data is captured.

Furthermore, we calculated the Root Mean Squared Error (RMSE) among the three gaze tracking solutions for this trial. The RMSE was computed for the trajectories shown in both Figures 4, 5. The computed RMSE values are presented in Table 1. The RMSE in millimeters represents the discrepancy between the ground truth on the screen and the gaze point provided by each gaze tracking system, at an 800 mm screen-user distance. Additionally, the RMSE in degrees measures the angular error between the ground truth gaze vector and the corresponding vector delivered by each of the three gaze tracking systems. It is evident that comparable results were achieved between GazeRecorder, which stands out as one of the most accurate webcam-based gaze tracking solutions, and our proposed method. However, it is essential to note that these solutions still cannot rival the precision of the highly sophisticated Tobii Eye Tracker 5 solution.

For this specific trial, we did not attain the accuracy reported by GazeRecorder and Heck et al. (2023). This deviation could be attributed to the experiments not being conducted in a controlled laboratory environment, with not ideal lightning conditions, potential influences from the user’s attention and gaze tracking behavior when following a trajectory. However, it is essential to highlight that, when computing the RMSE in degrees for this trial, our method yields similar results to GazeRecorder. This serves as an additional indication that our approach to compute a transformation matrix from the gaze coordinate system to the screen is effective, enabling the precise projection of a unit gaze vector onto the screen.

In the context of head movements, we differentiate between two types: head movements where the user simply turns their head, corresponding to yaw and pitch movements and head movements where the user moves their head from left to right, or up and down.

For the next experiment, a point is presented at the center of the screen. The user is instructed to focus their gaze on this point while performing head movements: turning their head once to the left, once to the right, and once up and down. This essentially involves yaw and pitch movements. The movements were in the range ±30° for yaw movements and ±25° for pitch movements. The results are presented in Table 2.

In the following experiment, the user is once more tasked fixing their gaze on a displayed point. However, this time, the user is instructed to refrain from moving their head, but rather, to shift their body to the left and then to the right, all while maintaining their focus on the point. The movement was in a range of ±100 mm form the initial position, which was in the middle of the screen at a distance of 800 mm. The outcome of this experiment is detailed in Table 3.

The accuracy and precision of our projection depends solely on the gaze vector estimation model used. For instance, the Mean Absolute Error (MAE) of OpenVino is 6.95°, with a standard deviation of 3.58°. Accuracy and precision metrics for models trained on the ETH-XGaze dataset are available on the ETH-XGaze competition website CodaLab (accessed 2023). These initial experiments, though conducted on only one person, already indicate that 2D webcam-based solutions, including GazeRecorder, fail to effectively accommodate precise head movements. These experiments merely serve to highlight the persistent limitations associated with head movements when using 2D webcam-based gaze tracking. This limitation, also recognized by Heck et al. (2023), becomes evident after just one trial. Specifically, lateral movements, such as those to the left or right, are not adequately addressed in any of the trained models. For example, GazeRecorder restricts horizontal head movements by positioning the user centrally and issuing alerts for excessive shifts. In the subsequent section, we delve into the challenges associated with lateral movements, illustrate the issue using the OpenVino model, and propose a solution to address these challenges using SfM.

The proposed method compensates for horizontal head movements in which a user shifts left or right by utilizing SfM to track the user’s movement. To achieve this, we apply the model introduced in Section 3.3.5. In our first experiment using this model, we replicate the conditions detailed in Section 4.2.1, where the user is instructed to follow a target trajectory while keeping their head stationary. The results of this experiment align with those presented in Table 1, showcasing similar outcomes and reinforcing the validity of the approach.

In the subsequent experiment, participants are instructed turning their head to the left and right (yaw-movement). Head movements can introduce subtle changes in the translation vector of T G W since the eyes are not precisely aligned with the head’s axis of rotation. However, it is essential to note that the OpenVino CNN model already compensates for these head movements. Therefore, both yaw and pitch rotations need to be accounted for in the translation vector of T G W .

Furthermore, when a user moves to the left or right without actually performing a yaw head movement, the 2D camera perspective makes it appear as if a yaw movement occurred. This is because the user is filmed from the side, causing the linear movement to mimic a yaw movement from the CNN model’s perspective. These particular movements are not considered in the CNN model’s calculations.

With the proposed method, which determines the actual location of a person in front of the computer screen, we are able to illustrate the mentioned two scenarios in Figure 6. One movement features a pure yaw movement and another involves lateral head movements to the left and right without head rotation. The visual representation demonstrates that a pure yaw movement leads to a translation in the x-direction of the T G W translation vector, as indicated in blue. However, when a user moves laterally, a significant change occurs in the x-direction of the translation vector, along with a shift in the yaw angle. This change in the yaw angle happens even when the user does not physically turn their head, a phenomenon attributed to the side-view camera perspective, as previously explained.

To accurately distinguish between these movements, a CNN model should undergo specialized training. This training process should include an additional input: the user’s position in front of the screen, determined through SfM techniques. With existing eye gaze datasets like (Zhang et al., 2020 or Tonsen et al., 2016) we propose to account for lateral head movements and improve webcam based gaze tracking.