The experimental protocol was approved by the Ethical Committee of the University of Paris (N° CER 2014-34/2018-115) and all participant gave written informed consent in line with the Declaration of Helsinki.

The setup is very similar to what used in our previous studies (Tagliabue and McIntyre, 2011, 2012), consisting of the following components: an active-marker motion-analysis system (CODAmotion; Charnwood Dynamics) used for real-time recording of the three-dimensional position of 19 infrared LEDs (sub-millimeter accuracy, 200-Hz sampling frequency). Eight markers were distributed ∼10 cm apart on the surface of stereo virtual reality goggles (nVisor sx60, NVIS) worn by the subjects (field of view: 60°, frame rate: 60 Hz, resolution: 1,280 × 1,024 pixels, adjustable inter-pupillary distance); eight on the surface of a tool (350 g, isotropic inertial moment around the roll axis) that was attached to the subjects’ dominant hand; and three attached to a fixed reference frame placed in the laboratory. Custom C++ code was developed by the research team to optimally combine the information about the three-dimensional position of the infrared markers and the angular information from an inertial sensor (IS-300 Plus system from InterSense) placed on the VR headset to estimate in real-time the position and the orientation of the subject’s viewpoint and thus to update accordingly the stereoscopic images shown in the virtual reality goggles. For tracking the hand movement only infrared markers were used.

The three-dimensional virtual environment shown to the subjects through the head mounted display consisted of a cylindrical tunnel (Figure 1). Longitudinal marks parallel to the tunnel axis were added on the walls to help the subjects to perceive their own spatial orientation in the virtual word. The fact that the marks went from white in the “ceiling” to black on the “floor” facilitated the identification of the visual vertical.

The task consisted of three phases: (1) memorization of the target orientation, 2) lateral neck flexion, and (3) alignment of the tool to the remembered target orientation. As in our previous studies (Tagliabue and McIntyre, 2011, 2012, 2013, 2014; Tagliabue et al., 2013; Arnoux et al., 2017), we took advantage of the head rotation to introduce a sensory conflict with the subjects not noticing it (see below). The target could be laterally tilted with respect to the virtual vertical of −45°, −30°, −15°, 0°, +15°, +30° or +45°. The subjects had 2.5 s to memorize its orientation. After the target disappeared, the subject was guided to laterally tilt the head 15° to the right or to the left by a sound with a left-right balance and a volume corresponding to the direction and the distance from the desired inclination. If the subject was unable to extinguish the sound within 5 sec, the trial was interrupted and repeated later on, otherwise a go signal was given to indicate that he/she had to reproduce the target orientation with the tool. The subject clicked on the trigger of a trackball held in the hand to validate the response.

In order to quantify the sensory weighting in each experimental condition a sensory conflict was artificially introduced (Tagliabue and McIntyre, 2011): tracking the virtual reality goggles was normally used to hold the visual scene stable with respect to the real world during the lateral head tilt, but in half of the trials, a gradual, imperceptible conflict was generated such that, when the subjects laterally flex the neck, they received visual information corresponding to a larger head tilt. The amplitude of the angle between the visual vertical and subject body axis varied proportionally (by a factor of 0.6) with the actual head tilt, so that for a 15° lateral head roll a 9° conflict was generated. When, at the end of the experiment, the subjects were interviewed about the conflict perception, none of them reported to have noticed the tilt of the visual scene.

Each subject was tested in two postural conditions: Seated and Supine (Figures 1A,B). In order to compensate for possible learning effects, half of the subjects were tested first seated and then supine, and the other half in the opposite order. When the subjects performed the task in the supine position, they lay in a medical bed with their head supported by an articulated mechanical structure allowing for lateral neck flexions (Figure 1B). When the subject performed the task in a seated position the same head support was fixed to the back of the chair to restrain the head movements in a way similar to the supine condition (Figure 1A). Since the main axis of the virtual tunnel always corresponded to the anterior-posterior subject direction, it was horizontal and vertical in the Seated and Supine Condition, respectively.

As detailed below, the first three experiments presented in this study differed only by the sensory information available to acquire the target and to control the tool during the response (Figure 1C). The task used in the fourth, additional experiment was the same as for the main cross-modal experiment with the exception that the subject always kept the head aligned to the body.

The subjects’ performance was analyzed using Matlab (MathWorks, RRID: SCR_001622) in terms of the lateral inclination (roll) of the tool when they validated the response. In order to describe the variability of the subject responses, we computed the root mean square of the difference, RMSd, between the two responses, r, to each combination of target, t, and head, h, inclination in the trials without conflict.

To describe the characteristics of the average behavior of the subjects, the linear regression lines of their responses after tilting the head to the right and to the left were computed imposing their parallelism (see Figure 2A). Each of the two regression lines have the form r = mt+q_i, where r and t are the response and target orientation, respectively. The parameter “m,” common for the two lines, represents their slope. The intersection with the response axis “q_i” is different for the trials with rotation of the head to the right (i = hr) and to the left (i = hl). The parameters of the lines were used to quantify the following variables:

To quantify the specific effect of the sensory conflict in each condition we linearly interpolated the responses of the conflict-trials with right and left neck flexion constraining the lines to be parallel to regression lines of the no-conflict-trials (see Figure 2B). This procedure provides the responses-axis intercepts for the conflict trials. Subtracting to these parameters the corresponding values in the no-conflict-trials we obtain the average deviations of the response due to the tilt of the visual scene: Δq_r,hi. In order to convert the response deviation into the percentage weight given to visual information, we computed, for each conflict trial, the virtual displacement of the target expected if only visual information was used to code its orientation, which corresponds to t - head_angle × 0.6. We linearly interpolated these theoretical responses for right and left neck flexion separately, constraining the lines to be parallel to the one joining the targets (m = 1) and we obtained the response-axis intercepts (see Figure 2B). Subtracting from these parameters the intercept of the line joining the target in the no-conflict trials (q = 0), we obtain the average target deviation expected in case of fully visual encoding of their orientation: Δq_t,hi. The percentage weight given to the visual information, ω_V, can be then computed as it follows:

For each experiment, we assessed the effect of the subject posture on the subject performances by performing mixed model ANOVAs on the AMe, Dist, Acc, RMSd, and ω_V dependent variables, with the Posture (Seated, Supine) and Order (Seated-First and Supine-First) as within- and between- subjects independent variable, respectively. No between-experiment comparisons were performed, because they do not correspond to the goal of this study. Since we performed three distinct experiments, we applied a Bonferroni correction (n = 3) to the resulting p-values to reduce the probability of type I errors (false positive). Therefore, in the following, p < 0.05/3 (≃0.0167), p < 0.01/3 (≃0.0033), and p < 0.001/3 (≃0.00033) will be indicated with “*,” “^**,” “^***,” respectively. For the straight-neck experiment, we specifically wanted to test the “Gravity Hp,” that is whether the Supine position increased the subjects’ variable and constant errors. We therefore performed one-tail Student’s t-tests on RMSd and Acc. Since the subjects did not rotate their head, no conflict could be generated and no quantification of the sensory weighting was possible. All statistical analyses were performed using the Statistica 8 software (Statsoft, SCR_014213).

In order to quantify the predictions associated with the Gravity and Neck Hypotheses and compare them with the experimental results, we apply our Concurrent Model of optimal sensory integration (Tagliabue and McIntyre, 2011, 2014) to describe the information flow associated with the Seated and Supine postures for each of the three experiments. An illustration of the general model structure is reported in Figure 3A.

This model is based on the assumption that the target and hand position are compared in the visual and proprioceptive space concurrently (ΔV and ΔP) and then these two parallel comparisons are combined based on the Maximum Likelihood Principle (Ernst and Banks, 2002). From this optimality principle it follows that the relative weight, W_ΔV and W_ΔP, given to each comparison depends on their variance σ△⁢V2 and σ△⁢P2 as it follows:

which corresponds to the minimal achievable variance of motor vector estimation Δ

In Equations 3 and 4 the covariance between ΔV and ΔP, cov(ΔV,ΔP), is used to take into account the situations in which the two concurrent comparisons are not fully independent (Tagliabue and McIntyre, 2013). The application of MLP to multi-sensory integration therefore assumes that the brain can estimate the variability of the signals to be combined (σ△⁢V2 and σ△⁢P2) and to which extent they are independent (cov(△V,△P)). Although it is not clear whether, and how, the brain would actually estimate these specific parameters, perceptive and behavioral studies have shown that human sensory weighting is clearly modulated by signals’ variability as predicted by the MLP (Ernst and Banks, 2002) and that performances cannot be improved by combining two fully dependent signals (Tagliabue and McIntyre, 2013-2014), as expected if their covariance is taken into account.

For the cross-modal task without head rotation (Figure 3B), the model predicts a reconstruction of the proprioceptive target representation from the visual information and of a visual hand representation from the proprioceptive feedback (green arrows). These cross-modal transformations, which introduce additional errors, are associated to specific variance terms σV→P2 and σP→V2, and, as show in section 1 of Supplementary Material, Equations 3 and 4 become:

As illustrated in Figures 3C,D, the model predicts no cross-modal reconstructions for the unimodal tasks (Tagliabue and McIntyre, 2013): in these tasks, the direct comparison between the available information about the target and the hand fully covaries with any comparison reconstructed from the available cues. From equation 4 it follows that the reconstruction of concurrent comparisons cannot improve the precision of Δ and using equations 3 it results that the predicted sensory weights and the motor vector variance are:

for the visual and proprioceptive task, respectively.

For all tasks, once the motor vector is estimated, the motor system generates the muscle activations necessary to displace the hand in the defined direction and distance. This step introduces some additional noise, that we will call motor noise, σm2, so that the variance of the movement execution is σM⁢E2=σ△2+σm2. There might be additional factors, as the concentration and fatigue levels of the subject, that can contribute to the movement execution variability. For sake of simplicity, the present version of the model does not include them separately and they are all combined together in the σm2 term.

To simulate the effect on the information processing of head inclination with respect to gravity, or of the neck flexion, in these three tasks, the variance, σN2, is added to the σV→P2, σP→V2 terms. This extra noise is added to the cross-modal sensory transformations performed with the neck flexed, with neck muscle acting against gravity or with the head misaligned with respect to gravity, depending on the hypothesis to be tested.

In order to test which hypothesis, between the “Neck1,” “Neck2,” and “Gravity,” better predicts the experimental results, we compare the observed effect of posture on the subjects’ responses’ variability, D⁢M⁢S⁢d=R⁢M⁢S⁢dS⁢u⁢p⁢i⁢n⁢e2-R⁢M⁢S⁢dS⁢e⁢a⁢t⁢e⁢d2, and on the response deviation due to visual scene rotation, Dω_V=ω_V,Supine−ω_V,Seated, with corresponding parameters of the model: the difference between the Supine and Seated posture predicted by the model for the movement execution variability, D⁢σM⁢E2=σM⁢E,S⁢u⁢p⁢i⁢n⁢e2-σM⁢E,S⁢e⁢a⁢t⁢e⁢d2, and for the weight associated with visual representation of the task, DW_△V=W_△V,Supine−W_△V,Seated.

As shown in Supplementary Material (sections 3 and 4), the theoretical predictions depend only on two main parameters: the variance associated to the cross-modal sensory transformation, σP↔V2, and to the noise added to these transformations when performed with the head misaligned with respect to gravity and/or the body,σN2. In order to reduce even further the degrees of freedom of the model, and thus the possibility of overfitting the experimental data, the value of σP↔V2 is set to 23.19°²; a value that is computed from the results of Tagliabue and McIntyre (2011) in section 4.2 of Supplementary Material. To statistically test whether the predictions of the various hypotheses differed from the experimental data, a multivariate Hotelling’s T² test is performed with six dependent variables (Dω_V and DMSd for each of the three experiments) and the six corresponding model predictions (DW_△V and D⁢σM⁢E2) as reference values.

To go beyond average responses, we then assessed whether inter-individual variability can provide more insight on the sensory processing underlying the responses observed in the three experiments.

For the Seated condition of the unimodal visual and proprioceptive experiments, the concurrent model predicts, respectively, a negative and positive correlation between the visual weighting and the variability of the motor vector estimation. In fact, a visual weight smaller than 100% in the V-V task, or the larger than 0% in the P-P task, would both correspond to suboptimal solutions and thus to an increase of the variability of the motor vector estimation (see Supplementary Material, section 2). The correlation between visual weighting and the variability of the motor vector estimation measured in inter-individuals is reported in Table 2.

Although not statistically significant, the tendency to a negative correlation in the unimodal visual task reported in Table 2, is consistent with the model prediction, while the absence of correlation in the P-P experiment is not. This could be due to a significant contribution of the motor noise to RMSd in this task, because both memorization and response require active hand movements. Motor noise affects the response variability but not the sensory weight, thus it might hide an existing correlation between the variability of motor vector estimation and the sensory weighting. The potential influence of motor noise is supported by the fact that the expected correlation seems to exist for the V-V task, where the motor component should be irrelevant.

For the V-P task no clear correlation between ω_V and RMSd is to be expected, because, as shown in Equation 5, the sensory weight theoretically depends only on the noise attributed to the cross-modal sensory transformations, whilst the response variability depends also on the subject’s visual and proprioceptive acuity. Moreover, motor noise could play a role, as in the P-P task.

In order to understand whether between-subject differences while seated would affect an individual’s performance when supine, we evaluated the correlation between the individual performance in the Seated and Supine conditions. As shown in Figure 6, we evaluated the performance in terms of response variability, RMSd, and visual weight, ω_V.

The top part of Figure 6 shows that the ranking of the subject in terms of response precision in the Seated condition tends to be preserved when Supine, but only in the tasks with relevant proprioceptive and motor components (V-P and P-P). Consistent with the results of Table 2, this finding suggests that the individual motor noise contributes to the observed response variability and tends to be preserved between postures. The bottom part of Figure 6 show that in the tasks with a relevant visual component (V-P, V-V), the subjects that are most visuo-(in)dependent when seated, remain the most visuo-(in)dependent when supine. These correlations suggest that, although different levels of visual-dependency can be observed among the subjects, their visual-dependency ranking was not altered by posture. It follows that the effect of the postural change in the cross-modal task was quite consistent among all of participants.

Figure 7A graphically represents the model predictions associated with the hypotheses that the lateral neck flexion per se (Neck1 Hp), the increase of the noise in the neck muscles-spindles (Neck2 Hp) or the head misalignment with respect to gravity (Gravity Hp), interferes with the ability to perform cross-modal transformation (detailed model equations are presented in Supplementary Material, section 3). Their quantitative comparison with the experimental results is shown in Figure 7B in terms of differences between the Seated and Supine condition. Focusing these predictions on the effect of the postural change has two main advantages: first, it compensates for a possible role of individual motor precision or sensory acuity that, as we have shown above, might increase between-subject variability. Second, it simplifies the model by allowing a significant reduction of the number of parameters estimated.

Figure 7B show that the “Neck1 Hp,” which predicts no changes between Seated and Supine postures for all three, Cross-Modal, Unimodal Visual and Unimodal Proprioceptive tasks, is significantly different from the experimental observations [Hotelling’s test: T² = 93.0, F_(6,12) = 10.9, p = 0.0003]. The “Neck2 Hp” prediction also significantly differs from the experimental observations [Hotelling’s test: T² = 34.93 F_(6,12) = 4.11, p = 0.017]. Indeed, although this hypothesis appears to better match the increase of the visual weight when supine, it cannot account for the increase in response variability; since in the Supine posture the neck muscles never act against gravity the model must predict a decrease of the response variability with respect to task performed with the Seated posture, which require a neck muscles’ activation during the response phase to support the tilted head.

“Gravity Hp” appears to well capture the fact that the Supine posture increases both the response variability and the visual weight in the cross-modal task only [Hotelling’s test: T² = 9.65, F_(6,12) = 1.13, p = 0.40]. The matching between the Gravity Hp prediction and the experimental data is obtained with σN2=812, which means that the variance associated with the cross-modal transformation would increase by about 3.5 times when the head is not aligned with gravity.

To confirm the role of the head-gravity alignment on the visuo-proprioceptive transformations (experimental results and the model prediction of Figure 7) the precision and the accuracy of the subjects’ responses was compared between the Seated and Supine conditions of a cross-modal task performed without lateral neck movements. Figure 8 shows that, as for the main Cross-Modal Experiment, when supine the subjects are significantly less precise [one-tailed t-test: t₍₁₁₎ = 3.42, p = 0.04] and less accurate [one-tailed t-test: t₍₁₁₎ = 2.79, p = 0.009] than when seated.

The analyses of the between-subjects differences suggest that the effect of the head-gravity misalignments on cross-modal transformations is quite robust, since it does not appear to depend on individual characteristics such as visual dependency or precision, which can vary significantly between subjects. The observed inter-subject variability in the Seated condition also suggests that not all subjects perform optimally, in the “Maximum Likelihood” sense (Ernst and Banks, 2002), that is, some subjects sub-optimally combine the visual and proprioceptive representations of the task. As expected, however, those subjects who deviate from the theoretical optimal sensory weighting tends to show larger level of variability.

Lastly, the inter-subject analyses also suggest that the noise of the motor component of the task, which can be different between participants, might represent a relevant part of the performance variability. These observations confirm the rationale of basing our conclusions on within-subject comparisons.

The present section aims at discussing whether the behavioral findings reported here are compatible with the current knowledge about the anatomy and physiology of the central nervous system. First, the brain areas involved in visuo-proprioceptive transformations will be presented. Second, it will be discussed how the signals related to head orientation with respect to gravity might interact with these brain areas and hence with the cross-modal processing.

The idea that the brain performs cross-modal transformations is supported by several electrophysiological and brain imaging studies. For instance, the encoding of visual stimuli in somatosensory space is consistent with the observation that brain regions such as the somatosensory areas (S) and Broadman’s Area 5 (BA5), which are known to encode the hand grasping configuration and the position of tactile stimulation in the peripersonal space (Koch and Fuster, 1989; Deshpande et al., 2008; Lacey et al., 2009), are activated also by visual stimuli such as images of glossy and rough surfaces, which a have a strong “tactile content” (Sun et al., 2016), and by images of familiar manipulable objects (Vingerhoets, 2008). Similarly, the encoding of haptic/proprioceptive information in visual space is fully compatible with the finding that the visual area in the Lateral Occipital Complex, called LOtv, is activated not only by 3D objects images (Moore and Engel, 2001), but also when sensing familiar objects with the hand (Deshpande et al., 2008; Lacey et al., 2009).

A brain area which appears to be a good candidate for performing cross-modal transformations is the Intra-Parietal Sulcus (IPS) which has been shown to have neural activation compatible with the computation of visuo-tactile transformations in monkey (Avillac et al., 2005) and which is known to be involved in the visuo-motor transformations performed during grasp movements (McGuire and Sabes, 2011; Janssen and Scherberger, 2015). Monkey experiments have shown that, in this brain area, the information can be reencoded from the retinal space to the somatosensory space, and vice-versa, thanks to recurrent basis function neural networks (Pouget et al., 2002) which would use the sensory signals relative to the eye-body kinematic chain to “connect” the two sensory spaces. In humans, the Anterior part of IPS is strongly activated when comparing visual to haptic objects, and vice-versa (Grefkes et al., 2002) or when reaching a visual target without visual feedback of the hand (Beurze et al., 2010). Virtual lesions of this area through TMS interfere with visuo-tactile transformations, but not with uni-modal, visual and tactile, tasks (Buelte et al., 2008). The planning of cross-modal tasks, such as reach-and-grasp visual objects with an unseen hand, also appears affected by TMS of the anterior IPS (Verhagen et al., 2012).

Focusing on the main finding of the present study, one can ask through which neural pathway the head-gravity misalignment can affect the visuo-proprioceptive transformations occurring in the IPS. At the peripheral level, the information about the head orientation with respect to gravity is mainly provided by a complex integration of the signals from different areas of the otolithic organ (Chartrand et al., 2016) arising from both the left and right organs (Uchino and Kushiro, 2011). Semi-circular canal and neck proprioception, which are combined to otolithic information already at the level of the vestibular nuclei (Gdowski and McCrea, 2000; Dickman and Angelaki, 2002), can also contribute to the head orientation estimation. However, since in the Straight Neck Experiment the posture effect was also observed when no head rotations, nor neck flexions, occurred, we can conclude that the otolithic signals are sufficient to affect visuo-proprioceptive transformations. At the central level, it is known that the vestibular-otolithic information can reach the parietal cortex through the posterior vestibular thalamocortical pathway (Hitier et al., 2014; Cullen, 2019). Specific otolithic afferences have been indeed observed in the IPS: otolithic stimulations activate neurons of Ventral IPS in monkeys (Schlack et al., 2002; Chen et al., 2011), with half of the neurons in this area which receive vestibular inputs (Bremmer, 2005), and human fMRI studies also show IPS activations resulting from saccular stimulations (Miyamoto et al., 2007; Schlindwein et al., 2008). Electrical stimulations of the anterior-IPS have also been reported to elicit linear vestibular sensations in a patient (Blanke et al., 2000). Since head-gravity misalignment modulates the otolithic inputs and the otolithic system projects to the IPS, it is plausible that gravitational information would be integrated in the recurrent basis-function neural network of this brain areas (Pouget et al., 2002; Avillac et al., 2005) to “connect” the visual and the proprioceptive space. As a consequence, it is reasonable that an alteration of the otolithic gravitational input due to the head tilt can alter cross-modal transformations.

There are other neural structures involved in motor control, such as the cerebellum, that receive otolithic inputs (Büttner-Ennever, 1999), and could therefore contribute to the effect of the head-gravity misalignment observed here. However, the predictive functions of the cerebellum (Blakemore and Sirigu, 2003), which is fundamental for the control of rapid movements, probably plays only a marginal role in the slow, quasi-static, movements tested here.

Once we have established that the head-gravity misalignment affects visuo-proprioceptive transformations and which neural circuits could be responsible for this phenomenon, the following question remains open: “How does tilting the head interfere with the cross-modal sensory processing?” At least two possible explanations exist: first, the unusualness of performing eye-hand coordination tasks with the head tilted; second, a possible signal-dependent increase of the otolithic noise with the head tilt.

Some studies have been able to correctly predict the effect of tilting the head on subjective vertical experiments by assuming that the noise of the otolithic signals linearly increases with the signal amplitude (Vrijer et al., 2008), hence the second hypothesis appears reasonable. To our knowledge, however, there are no electrophysiological studies clearly supporting the signal-dependent modulation of the otolithic noise (Fagerson and Barmack, 1995; Yu et al., 2012), therefore, the fact that unusual tilt of the head could interfere with cross-modal sensory transformations should not be “a priori” discarded. The “usualness effect” appears consistent with IPS recurrent neural networks functioning (Pouget et al., 2002) in which the synaptic weights necessary to perform visuo-proprioceptive transformations are learnt through experience. Since the upright position is largely the most common head orientation in our everyday life, it is possible that these neural networks become “optimized” for such head position and significantly less effective when otolithic afferences signal a head tilt for which we have a limited experience. A way to test this hypothesis could be to perform experiments on subjects that are in a tilted position, or in weightlessness, for a long period of time and see whether they can learn to perform cross-modal transformations as effectively as in the upright position, despite the altered or lacking otolithic signals.