Augmented Reality (AR) is widely recognized as the next-generation computing platform replacing smart phones and computers. In AR, information is presented to viewers with virtual objects such as graphics and captions fused with real environments without compromising the viewer’s natural vision (Olbrich et al., 2013; Choi J. et al., 2020; Yu et al., 2020; Chiam et al., 2021; Ong et al., 2021). Different from smart glasses which simply superimpose two-dimensional (2D) contents in a head-mounted display (HMD), AR allows the viewers to have more natural interactions with the virtual objects.

The central component of AR is a near-eye display (NED) which is worn by the viewers and is used to combine real and virtual imageries together so that both can be seen at the same time (Koulieris et al., 2019). Although AR NEDs offer a replacement for smartphones and computer monitors and provide visual experience to viewers, all designs for AR NEDs involve tradeoffs between a number of different metrics, including resolution, eyebox (Barten, 2004), form factor, correct focus cues (Zschau et al., 2010), field of view (FOV) (Wheelwright et al., 2018), eye relief, brightness, and full color. Therefore, the greatest challenge in AR NEDs is not in optimizing any individual metric, but instead simultaneously providing a wide FOV, variable focus to mitigate the vergence-accommodation-conflict (VAC), high resolution, a wide eyebox, ease of manufacturing, a slim form factor, etc (Hoffman et al., 2008). However, to counter the mentioned challenge with AR NEDs requires significant technological advancements. The requirement that an AR NED be see-through constrains the form factor and optical materials involved. The requirements on other metrics, including resolutions, FOV, eyebox, and eye relief push the boundaries of diffraction for visible light wavelengths.

In this paper, we present a review on the advancements and challenges towards AR NEDs. Although there are two main groups of AR NEDs (Rolland et al., 1994), namely video see-through and optical see-through, in this paper we will focus introducing the optical see-through AR NEDs because of their potential to provide an extremely high sense of immersion. We will begin our review by giving an overview of the leading types of AR NEDs. Then we will describe each of the leading types of AR NEDs in details with the principles and advancements to counter key challenges including eyebox, FOV and VAC. We conclude by outlining emerging technologies and unsolved challenges for future research.

The basic construction for AR NEDs normally includes: 1) a display unit or image source (e.g., a laser projector, a LCD display panel); 2) magnifying optics or relay optics; and 3) the medium to transmit and project the virtual imageries into the eyes of the viewers while allowing the lights from the real environment to pass through (e.g., half mirrors, holographic films) (Cakmakci and Rolland, 2006; Kress, 2020) (Figure 1). Kress and Sterner introduced the critical optical design challenges for AR NEDs, including providing sufficient resolution, large eyebox and wide FOV (Kress and Starner, 2013). Another key impediment and a key cause of discomfort with AR NEDs is the VAC issue (Yano et al., 2002; Koulieris et al., 2017), which is caused by a mismatch between the binocular disparity of a stereoscopic image and the single eye’s optical focus cues provided by the AR NED.

There have been various attempts from both industry and academia aiming to deliver compact AR NEDs with full color, high-resolution, large FOV and minimized VAC. Starting from beam splitter (BS) based AR NED, various AR NEDs technologies have been developed, such as waveguide based AR NEDs, holographic optical element (HOE) based AR NEDs, freeform optics based AR NEDs. Each of these technologies features their advantages in some of the metrics while having limitations for other metrics. There are also other emerging technologies, including pinlight based, transmissive mirror device (TMD) based, and meta-surface based AR NEDs. We will review each of them in the following sections.

Waveguide based AR NEDs use a waveguide as the medium to transmit and project the virtual imagery into the viewer’s eyes. As its name indicates, a waveguide can guide different types of waves (e.g., electromagnetic wave) to pass through the fibers and pipes and has been widely applied in various domains (Snyder and Love, 1983). In optics, a waveguide is used as a transmitter that transmits the light wave between two different materials by guiding the light waves as shown in Figure 2A. As the waveguide is able to transmit light waves with total-internal-reflection (TIR) mode and therefore without any loss of input signal, it has been deployed for AR NEDs. However, limited FOV is a common challenge with waveguide based AR NEDs due to the incident light angle requirements for TIR to take place, which is dependent on the refraction index of the waveguide (Shen et al., 2017).

Waveguide based AR NEDs normally need two coupling components: an in-coupler and an out-coupler. As its name indicates, an in-coupler is responsible to couple the light from the image source into the waveguide while the out-coupler is responsible to direct the light from the waveguide into the user’s eye. Based on the coupling components used, waveguide based AR NEDs can be categorized into two main types: reflective and diffractive waveguides based AR NEDs.

Holographic optical elements (HOEs) are optical devices based on holography technique and have optically see-through property due to their high angular selectivity. Therefore, HOEs have been employed in AR NEDs in recent years (Lee B. et al., 2020). The principle of the HOE depends on the hologram recording (Kim et al., 2017). When the reference beam illuminates the recorded hologram, the virtual 3D image close to the original object is reconstructed. Figure 4A shows the recording process of a reflection HOE film and Figure 4B shows the reconstructed signal beam by projecting the reference beam to the recorded HOE film. Depending on the geometry of the recording process, HOEs can be classified into two types: transmission type and reflection type (Xiong et al., 2021a). In a transmission HOE, both the signal beam and reference beam are on the same side of the recording material. In a reflection HOE, the signal beam and the reference beam are on the different sides of the recording material. Figure 4D illustrates a typical setup for HOE based AR NEDs in which the collimated lights are projected onto the recorded HOE and the reconstructed lights will be generated and projected into the user’s eye.

As shown in Figure 4A, the reference and signal beams are both collimated. After hologram recording, the collimated signal beam perpendicular to the HOE plane can be reconstructed with the illumination of the oblique reference beam. This kind of HOEs is always utilized as the in- and out-couplers of waveguide based AR NEDs because HOE is able to off-axis direct the light (Mukawa et al., 2008; Piao et al., 2013; Shi et al., 2012). As shown in Figure 5, the light waves from the source display are reflected on the in-coupler HOE with an incident angle and then travel through the waveguide, and finally the out-coupler HOE changes the directions of the light and projects the light toward the eye of the viewer. The combinational use of HOE and waveguide enables the optical see-through view in a compact form factor. However, as one HOE film reflects only one wavelength of light, in order to achieve full color display, three HOEs are needed to reflect red, green, and blue colors respectively. This not only adds cost for manufacturing but since the three HOEs need to be “sandwiched” together, each wavelength of the light is slightly diffracted by the other color hologram adding color “cross-talk” in the image (Mukawa et al., 2008). To solve the issue, Shin et al. proposed a novel recording method towards improving the diffraction efficiency and uniformity of full-color HOE (Shin et al., 2021). In their method, an analysis is first conducted on the inhibitory properties of the initial response and the optical characteristics of the late response of the recording medium for each wavelength. Then the analysis result is utilized to improve the diffraction efficiency and color uniformity of full-color HOE. Compared with the above-mentioned diffractive waveguide based AR NEDs with DOE as the in- and out-couplers, the rainbow effect or color crosstalk problem can be eliminated with the HOE as the in- and out-couplers due to its narrower spectral bandwidth.

HOE based AR NEDs have two natural strengths in comparison with other technologies. First, it can be manufactured in compact form factor as normally HOE can be printed in thin films (Jeong et al., 2019; Jang et al., 2020). Secondly, it can achieve large FOV as the HOE film can be recorded with a signal beam with large-angle which can be generated by placing an objective lens in front of the HOE film (Figure 4C). Additionally, HOE can also be combined with a spatial light modulator (SLM) to achieve natural depth perception of virtual imagery (Yaras et al., 2010) thus eliminating the VAC issue and these kinds of AR NEDs are normally referred as holographic AR NEDs (Maimone et al., 2017). Figure 6 shows a typical optical design for holographic AR NED which was proposed by the authors (Xia et al., 2020). Our design is made up of three components: a laser light source, a recorded HOE film as the optical combiner and a SLM. The laser light source is deployed to emit the light source; the SLM is utilized to generate the digital hologram pattern calculated for the 3D image to be displayed; and the recorded HOE film is utilized to reconstruct the virtual image via off-axis projection.

Holographic AR NEDs normally choose to use computer-generated holograms (CGHs) to directly and dynamically reconstruct the realistic-looking projections (Peng et al., 2020). CGH is the field of using computers to algorithmically generate holographic interference patterns and a SLM to display the hologram pattern. Figure 7 shows the typical system setup for a CGH. To calculate the interference pattern for CGH, there are mainly two methods, including Fourier holography (Makey et al., 2012) and Fresnel holography (Benton and Bove, 2008). However, the conventional calculation for these two CGH methods with heuristic solutions is usually time consuming with no guaranteed image quality (Maimone et al., 2017). To improve the image quality reconstructed with holographic displays, Padmanaban et al. introduced a novel overlap-add stereogram (OLAS) algorithm to invert the light field into a hologram via the short-term Fourier transform (Padmanaban et al., 2019). Their method takes more computing power thus increasing the computing time in comparison with other methods. To speed up the hologram calculation, a few algorithms have been proposed (Chen and Chu, 2015; Gilles et al., 2016; Wei et al., 2016; Askari et al., 2017). Deep learning based methods leveraging on the power of neural network are recently introduced and achieve both unprecedented image fidelity and real-time framerates Horisaki et al. (2018); Lee J. et al. (2020).

One major drawback with holographic AR NEDs is the trade-off between FOV and eyebox (Brooker, 2003) as the product of these two factors is limited by the total number of pixels of the SLM. However, adopting a high-resolution SLM with higher pixel density will lead to significantly increased manufacturing cost and large form factor. To increase the eyebox, Park and Kim proposed a novel HOE based NED which uses a HOE as multiplexed concave mirrors to replicate the eyebox, thus enabling the observation of the images in a wider range (Park and Kim, 2018). In their method, CGH is created with different range of angular spectrums to control the depth of field for the displayed 3D object individually. Jang et al. demonstrated their holographic AR NED with expanded eyebox by shifting the optical system’s exit pupil to cover the expanded eyebox area with pupil-tracking. In their method, they proposed a pupil-shifting holographic optical element (PSHOE) to reduce the form factor (Jang et al., 2017, 2018). Choi et al. introduced their novel technique for eyebox expanded holographic AR NEDs by replicating and stitching the base eyebox via the combined use of a HOE and high order diffractions of the SLM (Choi et al., 2020b). In 2019, an improved integration of holographic AR NED and Maxwellian-view display was presented by Lee et al. in which the holographic AR NED processes relatively few layers of the virtual 3D scene, while the remaining objects are processed with a Maxwellian-view display through a Gaussian smoothing filter (Lee et al., 2019). In 2020, the authors proposed a novel design to expand the eyebox for holographic AR NEDs by utilizing a lens-array HOE which replicates the same spatial frequencies comprising the high-resolution holographic image at each viewing position (Xia et al., 2020).

In 1968, Ivan Sutherland developed the first see-through AR NED with a flat beam splitter (BS) to superimpose the computer-generated images on the direct view of the real world (E., 1968). Based on the initial prototype of AR NED which used only a flat BS (Figure 8A), new designs chose to deploy a curved BS together with the flat BS (Figure 8B, C). As the curved BS looks like a typical birdbath, these AR NEDs are normally categorized as birdbath AR NEDs. Typical commercialized birdbath AR NEDs include ODG AR glasses, Google glasses, etc. Compared with the traditional flat BS, the birdbath AR NEDs always have wide FOV due to the magnification of curved BS. Whereas this magnification always causes the image distortion which can be compensated with the predistortion of the image source.

Beam splitter based AR NEDs usually are constraint by the conflict between form factor and FOV (Rotier, 1989; Droessler and Rotier, 1990). To overcome these constraints, freeform optics based AR NEDs were developed as freeform surfaces can introduce more variables to optimize the optical eyepiece to get a high performance and relative compact outlook. For instance, Wang et al. developed an off-axis single-element curved beam combiners for AR NED (Wang et al., 2016). Meta two is another representative of freeform optics based AR NED ² . In these designs, a freeform half-mirror is used as both magnifying optics and an optical combiner (Figure 9A). To achieve compact form factor, instead of using one single freeform reflector, recent designs sophisticatedly choose a combination of refraction surfaces, total-internal-reflection (TIR) surfaces and reflection surfaces to minimize the form factor while allowing a large FOV (Morishima et al., 1995; Hoshi et al., 1996; Yamazaki et al., 1999). For example, Cheng et al. developed a freeform prism based AR NED which achieved a compact form factor in 2009 (Figure 9B). In their design, the wedge-shaped freeform prism consists of three freeform surfaces, and rays from the image source are firstly refracted by one surface close to the image source. After two consecutive reflections by surfaces, the rays are transmitted through one surface and reach the exit pupil of the system. An auxiliary element, which consists of two freeform surfaces, is attached to the prism to obtain a see-through view, and freeform surface in the auxiliary element is designed in order to maintain a non-distortion real-world scene. Market available AR NED products (e.g., NED ARTM ³ , have also been successfully commercialized based on freeform prism.

Freeform optics based AR NEDs, together with BS based AR NEDs, rely on binocular parallax to generate depth perception for the viewers and the optical power of the combiners are usually fixed. Therefore, these designs always suffer from VAC issue (Zabels, 2019; Zhan et al., 2020; Rolland et al., 2021). To solve VAC, some vari-focal based methods have been proposed. For example, Stevens et al. used Alvarez lenses to mitigate VAC issue in their proposed NED (Stevens et al., 2018). Dunn et al. proposed to use a varifocal deformable membrane mirror for each eye and eye tracking technique to achieve a wide FOV and VAC mitigated AR NED (Dunn et al., 2017). Similarly, McQuaide et al. proposed to use a deformable membrane mirror to generate realistic 3D depth cues by variable focus thus their display allows the viewer to see 3D objects using the natural accommodative response of the eye (McQuaide et al., 2003). Hua and Javidi proposed a method to combine freeform surface techniques with integral imaging (Hua and Javidi, 2014). In 2014, Hu and Hua deployed the combination of freeform-prism based design, high-speed deformable membrane mirror device and high-frame-rate digital micromirror device (DMD) to demonstrate a multifocal bench-top prototype with an extended depth range reaching from 0 to 3 diopters (Hu and Hua, 2014). Lee et al. proposed three-dimensional (3D) HMD providing multi-focal and wearable functions by using polarization-dependent optical path switching in Savart plate (Lee et al., 2016). In 2018, Wilson and Hua developed a mechanical method by shifting two lateral freeform Alvarez lenses to create a compact, high-resolution and tunable optical see-through NED with adjustable optical power from 0 to 3 diopters (Wilson and Hua, 2018). However, their method is limited by the speed of the actuators and has limitations on FOV and eyebox.

To mitigate the VAC issue, Maxwellian view AR NEDs have been developed (Yuuki et al., 2012). Different from other methods which render virtual imagery with true focal cue for each eye, Maxwellian view AR NEDs, or retina scanning displays, use pinhole imaging to project and focus the virtual imagery onto the retina (Choi et al., 2020c). This method can help to alleviate the VAC issue. Recently, Song et al. developed a novel method to construct an optical see-through retinal-projection near-eye display using the Maxwellian view and a holographic method (Song et al., 2021). In their method, a single phase-only spatial light modulator (SLM) was employed to generate holographic virtual images which can be directly projected onto the retina. The virtual image can be projected at different depths and thus the presented method can resolve VAC issues. However, the eyebox size in Maxwellian view NEDs is always limited, and the alignment for the pupil is rather restrictive for the viewers.

Digital holography can also be integrated with the BS or freeform optics based AR NEDs to solve the VAC issue. This solution always employs the phase SLM as the image source and uses coherent light as the illumination source to create the 3D virtual image for the user’s eye (Gao et al., 2016; Gao and Liu, 2017; Chang et al., 2019). However, this kind of holographic NEDs always have a small FOV due to the etendue limitation of holographic displays.

Some other researchers have also explored other means to mitigate VAC issues, such as using Pancharatnam-Berry (PB) phase lenspolarization-dependent lens (Tan et al., 2018; Moon et al., 2019; Yoo et al., 2019; Xiong et al., 2021a), or using electrically tunable lens (Lee et al., 2019; Piskunov et al., 2020). With different polarization state, polarization-dependent lens can create different focal lengths and thus generate multiple focal planes for near-eye display. As electrically tunable lens is able to dynamically adjust the focus for the light, it can been combined with active shutters to achieve time-multiplexed focus adjusting for the virtual images and real images respectively (Liu et al., 2008; Xia et al., 2019). In this way, tunable lens based design can help to solve the VAC problem.

Besides the abovementioned AR NED technologies, there have been other emerging technologies developed and published in recent years, including:

AR is widely recognized as the next-generation computing platform with numerous potential applications in various sectors. As an indispensable component for AR, NEDs have been the subject of many investigations by academia and industry and are therefore experiencing rapid progress. In this paper, we first present an overview about various technologies and their advances for AR NEDs. Then we focus our review on the principles, challenges and advancements for three leading designs for AR NEDs, including half-mirror/prism based AR NEDs, HOE based AR NEDs and waveguide based AR NEDs. We also reviewed other emerging technologies, including pinlight based AR NEDs, TMD-based AR NEDs, PB phase lens based AR NEDs and meta-surface based AR NEDs.

Each of the reviewed methods has their own advantages and disadvantages and involves tradeoffs between different metrics. As shared by Chang et al., in 2020, future-ready AR NEDs will provide both a comfort experience (eyeglasses-style form factor, light weight, large FOV) and immersion experience (natural 3D perception, high refresh rate, high-quality image) to users (Chang et al., 2020). Although HOE based AR NED technologies are heavily investigated and have shown the promise to achieve future-ready AR NEDs, there are still significant challenges remaining, including low image quality and high computation demand. Novel methods from other technologies or from the synergy between different technologies could also be expected. Another point worthy of close monitoring is the fast developments of AR display engines, including LCoS, DLP, LED, OLED, and LBS (Zhan et al., 2020). The continuously improved display resolution and decreased size of display engines might bring breakthrough inventions toward the ultimate goal of AR NEDs.

The mutual occlusion is also an important issue for AR NEDs. Most of the current AR NEDs superimpose the virtual image onto the real environment, but the displayed virtual image is transparent and can not block the rear scene. Several recent research works are focused on the approaches obtaining hard-edge mutual occlusion in optical see-through AR NEDs. Most of the solutions use two SLMs to merge the real and virtual imagery. This method always utilize one SLM to add occlusion mask to the real scene and use the other SLM to display the virtual image and then superimpose the both for the eye (Wilson and Hua, 2017; Hamasaki and Itoh, 2019; Chae et al., 2021; Zhang et al., 2021). Some other researches utilize the time-multiplexed method with a single SLM to achieve the mutual occlusion for AR NEDs (Ju et al., 2020; Krajancich et al., 2020). The existing approaches can achieve mutual occlusion to some extent, but still suffers from limited FOV and bulky form factor.

We hope our discussions will help to inspire research on future directions to counter the challenges with AR NEDs and we look forward to these advances.