Mixed Reality is a class of simulators that combines both virtual and real objects to create a hybrid of the virtual and real worlds Ohta (1999). MR visualization acts as a bridge between virtual content and the real world, forming the core of the entire MR system. In recent years, advancements in graphic rendering algorithms and GPU hardware have made the blending of virtual objects with the real-world environment on MR devices increasingly natural and realistic. Enhancing light and shadow models necessitates the incorporation of temporal and dynamic effects, such as sudden changes in lighting and dynamic scenes, as well as the classification of algorithms tailored to different light paths Marques et al. (2018). Gierlinger et al. (2010) propsed a real-time rendering engine specifically tailored to the needs of MR visualization, which utilize image-based techniques for lighting and material acquisition allows for consistent integration of virtual objects into real-world environments. recommended using the fuzzy logic model to add soft shadows to a virtual object during embedding in a real scene. Nasr Eddine and Junjun (2019) used a holographic approach using georeferenced raster-based data integrated into a virtual world to enhance geographic visualization and data observation, with experiments conducted on the HoloLens to improve geographic edutainment. Zhou and Zhou (2023) proposed a mixed reality (MR) video fusion framework that dynamically projects video images onto 3D models as textures, utilizing remote rendering and browser-based implementation to overcome client limitations and reduce computational and bandwidth demands.

For inverse rendering, accurately Lighting estimation, particularly in indoor scenes, is a complex and essential task. Most current illumination estimation methods operate on single images, with a primary emphasis on integrating virtual objects into real images rather than making substantial alterations to the scene’s illumination Karsch et al. (2011); Garon et al. (2019); Zhan et al. (2021). While traditional methods like a single environment map Gardner et al. (2017); LeGendre et al. (2019) and spherical lobes Garon et al. (2019) have been used, they often neglect spatial variations and high-frequency details in lighting. Recent innovations Song and Funkhouser (2019); Srinivasan et al. (2020) have attempted to improve 3D lighting representation, but still grapple with challenges like spatial instability and the lack of HDR information. Li et al. (2020) propose per-pixel spatially-varying spherical Gaussians (SVSG) lighting representation to capture high-frequency effects and demonstrate that SGs are superior to spherical harmonics (SH) for depicting lighting details in indoor scenes. Neural-PIL Boss et al. (2021b) proposes a pre-integrated lighting network based on image-based lighting (IBL), showing better performances on conveying global illumination than SGs and SH. Hence, we utilize a neural HDR-radiance field to represent the IBL at any spatial point, thereby ensuring a more accurate and detailed depiction of indoor lighting scenarios with a focus on physically accurate HDR lighting prediction.

Material estimation in inverse rendering can be categorized into two levels: object level and scene level. Object-level estimation Zhang et al. (2021), Zhang et al. (2022); Munkberg et al. (2022); Liang et al. (2022); Boss et al. (2021a) focuses on individual objects, often in controlled or simplified environments. Object-level approaches are generally less complex, as they deal with fewer variables and more straightforward lighting conditions. In contrast, scene-level material estimation Choi et al. (2023); Li et al. (2023) is significantly more challenging due to the complexity and variability of entire scenes. This includes diverse lighting, multiple objects with different materials, and shadows.

The complexity of scene-level material estimation is further compounded by the choice between single-view and multi-view approaches. Single-view material estimation Li et al. (2020); Gardner et al. (2017), despite its simplicity and lower data requirements, often faces the ill-posed issue, where insufficient information leads to ambiguous or inaccurate estimations. This is particularly evident in complex scenes where a single viewpoint cannot capture the entirety of the scene’s lighting and material properties. In contrast, multi-view material estimation Choi et al. (2023); Zhang et al. (2022), Zhang et al. (2021); Munkberg et al. (2022) leverages images from multiple viewpoints, providing a more comprehensive understanding of the scene. It can significantly reduce the ambiguity associated with single-view estimations, allowing for more accurate and reliable material property extraction. Our work utilizes multi-view images for material estimation, specifically addressing the challenges at the scene level.

Since vision is crucial for perception, achieving the best fusion requires considering the overall impact of visual integration. Consequently, visual perception in MR systems has consistently been a primary research topic. Fleming (2014) proposed material perception plays a crucial role in the visual system by allowing us to effortlessly recognize and distinguish materials despite varying appearances. Zhdanov et al. (2019) explored virtual prototyping methods to assess and mitigate visual discomfort in AR, VR, and MR systems, addressing issues like vergence-accommodation conflicts and illumination differences. However, it may still face limitations in fully capturing the complexities of real-world visual perception. Petikam et al. (2018) study the relationship between real-world depth fidelity and visual quality in MR rendering, providing perceptual thresholds for various composition artifacts through user experiments. However, it is limited by its focus solely on depth information, without analyzing other rendering factors, which may constrain its applicability in real-world scenarios. Potemin et al. (2018) proposing a virtual prototyping approach to analyze visual perception problems in augmented and mixed reality devices, comparing physically correct images with expected ones. However, it is limited by its reliance on physical results without further analyzing the impact on human visual perception. Du et al. (2024) proposed how the integration of subtle visual cues, such as shadows, lighting, textures, blur, and distortions, in MR interfaces can enhance user experience by making digital elements appear as natural components of the physical environment. However, it is limited by its focus on how virtual objects blend into real-world settings, without further exploring how real objects can be seamlessly integrated into virtual environments to improve overall MR fusion.

When building MR systems, we typically embed 3D models of real-scanned elements into virtual 3D models. However, we have observed that since real-scanned objects often carry the lighting and shadow effects of the original scene, this residual light and shadow create disconnection in the new scene. Therefore, we consider how to eliminate the impact of the original lighting. Given that the material is not affected by lighting, we can obtain the material properties of the real elements to remove the interference of light and shadow. Nevertheless, compared to virtual objects (designed by artists), it is usually difficult to directly obtain the display material parameter values of real objects. Leveraging the recently developed Ref-GS technique Zhang et al. (2025), an AI-based inverse rendering approach built on differentiable neural rendering method, we achieve efficient PBR material and geometry reconstruction of avatars directly from multi-view images. This established method enables rapid decomposition of visual appearance into intrinsic material properties and geometric attributes, providing a practical solution for high-quality avatar creation in mixed reality applications.

Given a set of posed RGB images of an real-world human avatar, our goal is to accurately decompose the geometry, materials (Basecolor/Roughness) under unknown illumination, which can be used in re-render process with new environment light. Based on Ref-GS technique Zhang et al. (2025), we firstly use the 3D-GS Kerbl et al. (2023) to represent the avatar’s 3 days Neural Radiance Field, which include global illumination and geometry property. And then, we use the additional Neural Network MLP to represent an material field. Meanwhile, to eliminate residual shadow and efficiently improve the quality of material, we introduce a diffusion-based material estimation prior as material regularization, the pipeline as shown in Figure 2.

After obtaining precise geometry, we utilized differentiable rendering techniques to optimize materials, include basecolor A ̂ and roughness R ̂ . The rendering equation can be written as Equation 1: L o x ̂ , ω o = ∫ Ω + L i x ̂ , ω i A ̂ π + f s x ̂ , R ̂ , ω i , ω o ω i ⋅ n ̂ x d ω i (1) where n ̂ x is normal at surface point x ̂ , ω i is light incident direction, ω o is view direction.

Visual realism in VR is often achieved by estimating physically based parameters such as lighting, materials, and geometry. Many VR design workflows assume that physically accurate rendering will naturally lead to realistic visual experiences. However, observations from realistic VR applications show that this assumption does not always hold. Even when physical parameters are correctly estimated, users may still perceive the scene as unrealistic.

This mismatch suggests that physical realism and human perceptual realism are not always aligned in interactive VR environments. Based on this observation, this study is guided by the following hypothesis:

H1

Physically derived visual realism parameters do not always align with users’ perceptual judgments of realism in realistic VR scenes.

Testing this hypothesis requires access to realism parameters as perceived by users during interaction. However, existing VR systems mainly support physically based rendering and rely on parameters predefined by designers. They do not provide mechanisms for users to directly adjust realism-related parameters or to record perception-aligned values.

To address this gap, this study introduces a user-centered VR system that allows users to freely adjust visual realism parameters within realistic VR scenes. Through interactive adjustment, the system captures parameter values that reflect users’ perceptual preferences rather than physical correctness alone. These perception-aligned parameters enable direct comparison between physical and perceptual realism. This system includes global and local variables, and allows users to adjust different variable parameters to obtain the optimal parameter combination to achieve optimal immersion in MR scenes, as shown in Figure 3. User studies are conducted to compare physically derived parameters with user-adjusted parameters across three realistic VR scenarios: a Classroom, a Computer Room, and a Studio. These scenes represent common and socially relevant uses of realistic VR. The experimental results confirm the proposed hypothesis and provide the foundation for building an optimal visual realism parameter model in later studies.

Notably, inspired by Wei and Luximon (2024) and Disney’s PBR model, we select an group significant variables for visual fusion adjustment.

Global Variables: To ensure the avatar and background blend seamlessly, we need to blend the lighting and shadows of the avatar with those of the background scene Hughes et al. (2004). Therefore, we designed the lighting and shadow variables of the environment as global control variables for blending. However, we assume there might be a gap between the physical rendering results and human perception in some variables. For example, the direction of the shadows on the rendered avatar may not necessarily match the optimal blending perception from a human perspective.

Light Types: This is a discrete variable. And here, we simplify the types of lighting by using general illumination in MR, include point light, direction light and spot light.

Local Variables: Since the avatar is essentially a complex composite, made up of multiple parts and different materials, and because user have varying levels of visual perception for different parts, we assume that the visual blending of different parts affects the overall blending effect in an MR scene. We divided the avatar into five parts: hair, skin, top, pants, and shoes. For each part, we designed two local variables: roughness and base color saturation. As pants and shoes are black and therefore the base color saturation is not adjusted for pants and shoes.

Roughness: This is a continuous variable. Roughness describes the degree of smoothness of the material for the adjusted part. In our MR environment, we consider that the roughness calculated through inverse rendering may not fully achieve optimal blending. Therefore, the default parameter value for roughness is set to the result from inverse rendering. We then adjust this parameter based on the physical results, ultimately aiming to find the optimal roughness range for each part that aligns with the majority of users’ perception of seamless blending.

Furthermore, we use Participatory Design Method to obtain the optimal MR fusion system for our research scene. By allowing multiple users to adjust the parameters of all variables to achieve their perceived optimal fusion effect, we can continuously refine our optimal fusion parameter range through this process. The optimal equation is shown in Equation 2: D in = ∑ x ∈ L , R 1 R − L , D out = ∑ x ∉ L , R 1 1 − R − L , (2) where L and R is denote the left and right boundaries of the test interval, respectively. The term ∑ x ∈ [ L , R ] 1 represents the total number of data points within the interval [ L , R ] . For each data point x inside this interval, 1 is added. The denominator ( R − L ) is the width of the interval [ L , R ] . Therefore, D in is defined as the number of data points inside the interval [ L , R ] divided by the width of the interval, which is the density of data points inside the interval, and D out is external density.

We define the objective function f ( L , R ) which quantifies the difference in data point densities inside and outside the interval [ L , R ] , and then we optimization goal is to find the optimal as shown in Equation 3, and the objective function is Equation 4. f L , R = D in L , R − D out L , R (3) δ * = arg max δ f P − δ , P + δ (4) where P is the center of the densest region in the histogram, representing the peak of the distribution. δ is an offset used to explore intervals of different widths centered at P . By optimizing this objective f ( L , R ) , we can find the optimal offset that maximizes the density difference between inside and outside the interval δ * , thereby determining the corresponding optimal interval [ L * , R * ] .

The experimental results provide clear empirical evidence that this consistency does not always hold. Across all three scenarios, participants consistently favored perception-aligned parameter settings over physically computed values, supporting Hypothesis H1. These findings challenge the common assumption that physically accurate rendering alone is sufficient to achieve convincing visual realism in VR environments.

One possible explanation for this mismatch lies in the nature of human visual perception. While physically based rendering aims to simulate light transport and material interactions according to physical laws, human perception of realism is influenced by additional factors such as visual comfort, contextual expectation, and tolerance to physical inaccuracies. Users may therefore prefer parameter configurations that deviate from strict physical correctness but better match their subjective experience of what appears “natural” or “believable” in a given VR context.

The observed differences in preference strength across scenes further suggest that the perceptual–physical mismatch is context-dependent. In the Computer room and Classroom scenarios, strong preferences for the User-tuned condition indicate that users are particularly sensitive to realism-related parameters in structured indoor environments, where lighting consistency and material appearance strongly affect plausibility. In contrast, although the Studio scene also showed a significant preference for perception-aligned settings, the effect was comparatively more moderate. This may be attributed to greater perceptual tolerance in broadcast-style environments, where stylization and controlled lighting are more common and therefore less likely to violate user expectations.

Importantly, these findings highlight a fundamental limitation of current VR design workflows that rely exclusively on physically derived parameters. Without mechanisms to capture and incorporate user perceptual preferences, designers risk producing visually accurate yet perceptually unconvincing scenes. The results of this chapter demonstrate the necessity of treating perceptual realism as a first-class component in VR scene design rather than as a byproduct of physical simulation.

Our study results reveal that an MR system combining both physical and perceptual aspects can optimize users’ visual fusion in MR environments. We conducted evaluations across different scenarios and found that this approach is effective. During user testing, we had users observe the visual fusion between the inserted avatar and the environment. We found that, regardless of whether the MR environment was a classroom, computer room, or studio, users experienced a noticeable improvement in visual fusion before and after adjusting the MR system. This subjective feedback aligns closely with findings from Wei and Luximon (2024), indicating that there is still a gap between users’ perception of visual fusion in MR systems and the values derived from physics-based calculations. In comparison, we conducted a more in-depth exploration within the MR system, allowing users to make intuitive judgments, thereby providing more credible experimental results. Given the significant impact of ambient lighting on overall effectiveness, we selected test environments based on common lighting types. This allowed users to intuitively perceive the interaction effects between the inserted avatar and the MR environment, and based on their potential cognition of the scene and a certain type of lighting, we made our evaluations. During the testing process, we first had users observe the MR system designed with rendering factor values obtained through physics-based inverse rendering model. Users were then asked to judge the visual fusion in this state. The results showed that most users were dissatisfied with the current visual fusion, particularly noting that the global lighting variables and the clothing materials did not completely match the settings of the current scene, resulting in a noticeable sense of separation. We found that people tend to prefer point light sources as the optimal choice in classroom settings, while in studio environments, they lean towards spotlights. This reflects the differing lighting needs of various scenarios and highlights the sensitivity of human visual perception to these differences. It is understandable that classrooms typically require even lighting to ensure all students can clearly see the blackboard and other visual aids. In contrast, studios use spotlights to highlight specific objects or areas.

Another objective of our study is to identify the optimal parameters combination range for visual fusion in MR test scenarios. We conducted a optimal range analysis of all adjustable variables for each scenario according the Section 4.2. Delving deeper into underlying factors, we explored the issue from two aspects. First, by comparing the optimal adjustment ranges for each scenario with the default values obtained from the physical-based inverse rendering model, we found that some variables were distributed near the default values, while others deviated. This indicates that solely relying on physical-based models to design MR systems is still insufficient to fully meet users’ optimal perception of visual fusion. Specifically, most roughness-related variables, including Cloth_roughness, Hair_roughness, Pants_roughness, and Skin_roughness, have optimal ranges that are smaller compared to the default values in all scenes. This indicates that, from the users’ perceptual standpoint, the avatar’s overall roughness distribution in these test environments should appear rougher rather than more specular.

Secondly, we observed that the optimal ranges for the same variables may vary between different scenarios, while also sharing some commonalities. Although the default lighting types differ between computer bar and teaching room, most of the optimal ranges are similar since both scenarios are classroom types. The primary exceptions are the distributions of Pants_roughness and Skin_roughness, which show significant differences with almost no overlapping areas. This indicates that in the computer room and classroom scenarios, the roughness of pants and skin requires special attention. In the computer room environment, users believe that lower values of Pants_roughness and Skin_roughness enhance the visual fusion. Meanwhile, compared to the other two scenarios, Studio scenario has fewer variables with overlapping optimal ranges due to its greater environmental differences. Only a few variables—Light_intensity, Shadow_direction, Shadow_intensity, Cloth_basecolor, and Skin_basecolor overlapping ranges, meaning they are less influenced by the environment and need only slight tuning for scene fusion.

After participants completed the system adjustments, we also asked them to rank the importance of various variables. From the ranking data, we found that 77.78% of participants considered lighting to be the most important factor for global variables such as illumination and shadow. Regarding the impact of different body parts on the fusion effect of the avatar with the background environment, 77.78% of participants ranked skin as the most critical, while 50% considered shoes to have the least impact. Additionally, 44.44% ranked cloth as the second most important, with hair and pants having relatively similar weights in the middle rankings. Our findings are consistent with prior research work Gonçalves et al. (2023), both demonstrating that global lighting has the greatest impact on visual perception, followed by shadows. Meanwhile, users in MR environments perceive that the upper body of the avatar, including the skin and top garments, has a greater influence on their judgment of the fusion effect. This finding is also consistent with previous work Van der Veer et al. (2018).