vSLAM (visual Simultaneous Localization and Mapping) is a self-positioning method that uses a camera to capture the inside of a building. vSLAM is a technology that simultaneously localizes a target and builds a map, but if the application is indoor navigation, it needs to be matched with a known map database.

iTracker is a system that can estimate the 6DoF of a smartphone in real time (Sun et al., 2019). It is based on the monocular vSLAM technique (Engel et al., 2014) and utilizes short-term inertial measurement unit (IMU) tracking and IMU camera local information for rapid recovery even if tracking is lost. Through experiments, the proposed real-time step-length adaption algorithm proved that the iTracker error and delay increased slightly over time and was stable at 7° and 15 ms.

Compared with VLP using a camera, vSLAM is advantageous because it does not require a transmitter such as a modulated LED. By contrast, there are some disadvantages: privacy issues must be considered in some existing systems, creating a database is difficult, and vulnerability to changes in object placement must be addressed.

A PD is a device that can receive LED signals with lower power consumption than a camera. Simulation-based methods using PDs, such as time of arrival (ToA) (Wang et al., 2013) and time difference of arrival (TDoA) (Jung et al., 2011), have been proposed. However, accurately measuring the arrival time of a signal arriving at the speed of light using an off-the-shelf PD is difficult; hence, AoA (Yang et al., 2014) and received signal strength (RSS) (Steendam et al., 2017) have been proposed for use in real devices.

Epsilon was the first real working VLP system designed in the academic community (Li et al., 2014). It detects the binary shift keying of the LED using the prototype device with the PD and derives the position by triangulation. Accuracies of 0.4, 0.7, and 0.8 m at the 90th percentile were achieved in three different office spaces. NALoc uses the same type of device as the ambient light sensor that is implemented in a smartphone. The results gave 90th percentile errors of less than 0.35 m for the 2-D position, but the device was evaluated separately from the smartphone and not in a built-in setting (Yang et al., 2018).

These methods have not been evaluated using sensors embedded in smartphones. In our study, we describe an application that can run using only a smartphone.

We focus on VLP using smartphone cameras. Camera-based methods allow geometrical separation of the light sources, allowing for more accurate positioning (Yoshino et al., 2008; Rahman et al., 2011; Zhu and Zhang, 2017).

Luxapose can compute the position and posture of the smartphone by capturing ceiling lights directly with a camera (Kuo et al., 2014). The error is less than 10 cm and 3°. It uses 7712 × 5360 pixels in a WindowsPhone 8 smartphone camera as the receiver. The calculation requires a cloud server for high-quality image processing.

PIXEL is a polarization-based localization method (Yang et al., 2015) that requires only 120 × 160 pixels. It can measure indoor positioning in several seconds with an accuracy of 0.4 m. However, a polarizing filter must be attached to the camera, so there is a risk of impairing the original image.

The approach by Rajagopal et al. uses light reflected by the floor (Rajagopal et al., 2014). It is similar to our idea, but the authors focus on using the rolling shutter distortion to receive an identifier (ID) from the reflected light. Carriers up to 8 kHz can be received with a channel separation of 200 Hz. Tag information is transmitted by assigning ON and OFF bits to different frequencies. The data rate is 10 bps, and up to 29 light sources can be uniquely separated. Positioning accuracy was not discussed in the paper because the research aimed to obtain semantic positioning from differences in packet reception rates (PRR). Furthermore, because MATLAB was used for calculation, processing was not in real time.

STARLIT (Yang et al., 2019) also uses a rolling shutter camera and obtains short and long exposure images of reflected floor lights to filter noises. STARLIT can achieve 3-D localization of a smartphone but cannot conduct its 6DoF estimation.

The approach by Nakazawa et al. uses a dual-facing camera and calculates its position by the relationship between the ceiling light and the reflected light on the floor. Large coverage and high accuracy are achieved using only two LEDs. This method also requires a large image, so the average processing time is 1.2 s (Nakazawa et al., 2017).

In iLAMP, the radiation pattern of each LED in the building is recorded by the camera at first, and the position of the camera is estimated by matching with a database that has been created (Zhu and Zhang, 2017). Because a single LED can be used to estimate the 6DoF of a smartphone, there are fewer restrictions on the formation of the LEDs and the height of a ceiling. However, a computing server is required for large image processing.

RainbowLight is a method for calculating 3-D positioning based on the properties of birefringence and polarization (Li et al., 2018). In the authors' system, a birefringent material such as cellophane tape and a polarizing film are attached to the LED or window as landmarks, which are photographed by a camera equipped with the polarizing film to enable positioning without modulating the light. In a 3-D positioning experiment where the distance between the light source and the camera was between 2 and 3 m, the average errors were 3.19 cm on the x-axis, 2.74 cm on the y-axis, and 23.65 cm on the z-axis.

These systems still have the problems described in Section 1: constraints of LED formation and the height of the ceiling because of the FoV of the camera, calculation overheads for real-time processing within the smartphone, and the lack of care regarding overcoming rolling shutter distortion.

Our goal is to estimate the 6DoF of a smartphone in indoor environments. The definitions of 6DoF and image plane in RefRec+ are shown in Figure 1. The mobile 3-D position is shown by (x, y, z) in the world X, Y, and Z coordinates, and the pose is shown by (θ_x, θ_y, θ_z), which means pitch, roll, and yaw angle along the X_C, Y_C, and Z_C axis of the smartphone coordinate system. The head of the smartphone points in the Y_c direction, and the camera faces the floor at its initial status (θ_x, θ_y, θ_z) = (0, 0, 0). 3-D space is transformed by the camera into a U-V image plane, as shown in Figure 1. An overview of our system is shown in Figure 2A. Our method is based on observations that the RSS from the LED decreases with the distance from the LED to the floor.

More than two LEDs are mounted on the ceiling, which is parallel to the X-Y plane. The kth LED's known 3-D coordinate is P_Lk = (x_Lk, y_Lk, z_Lk). Each LED broadcasts a sinusoidal wave with its own unique frequency. The receiver is given the assignment table of these frequencies in advance. Modulated light from the LEDs is reflected by the floor, which is parallel to the X-Y plane where Z = 0. The camera of the mobile device captures any part of the floor area and then obtains a light intensity value at the center of ith POI P_Fi = (x_Fi, y_Fi, 0)(i = 0, 1, ⋯ ) composed of multiple pixels by averaging their intensity values. The total number of LEDs should be more than two. The ith POI corresponds to the pixel (u, v) on the U-V plane shown in Figure 1.

The positioning algorithm is briefly described as follows (Figure 2B). dik represents the distance between the intersection point on the floor with a perpendicular line passing through the kth light source installed at the ceiling at P_Lk and P_Fi on the floor captured by the camera. First, the distance dik is estimated. Second, the coordinate of P_Fi corresponding to (u, v) on the U−V plane is estimated using dik (k = 0, 1, 2, ...) and the known position of the LEDs.

The 6DoF of the camera is estimated by optical AoA using more than two P_Fi. The following sections provide details of the process.

To find the distance dik, the camera detects the intensity of the received signal from the photographed image and substitutes it into the diffusion model of the LED's reflected light. To reduce the influence of other light sources, the LED blinks a sinusoidal wave orthogonal in frequency to those of other light sources. We define b_k(t), which includes a signal from the kth LED at time t, as follows:

where f_s is the fundamental signal frequency and A_k is a natural number so that the signal reaches a frequency that is sufficiently high to avoid causing flickering; α is the direct current component, which makes b_k(t) always a positive value; and m_k is a natural number that uniquely identifies the frequency for each LED.

Let us assume that the camera frame rate is f_c = f_s, the shutter cycle is T_c = 1/f_c, the exposure time ratio is η, and the exposure time is ηT_c. By taking N images with the camera and separating the received light in the frequency domain, it is possible to extract the signal intensities of the unique frequencies. Therefore, the number of detectable LEDs is N/2−1, and m_k<N/2 should be satisfied as per the sampling theorem. By capturing b_k(t) from the kth LED with a camera, the resulting brightness I_ni of the P_Fi on the nth image is an integral:

where δ is the delay of the shutter timing with respect to the signal, and X(dik) is the attenuation function that is determined by the distance and transfer efficiency from the kth LED as a transmitter to the receiver. Hence, our purpose is to calculate dik from the inverse function of X(dik) using I_ni.

Now, B_qi is obtained by the Fourier transform of the video stream I_i = (I_0i, I_1i, ..., I_N−1i):

By assuming β_mk obtained by Fourier transform of b_k(t), B_mki is shown as follows (Shimada et al., 2017; Sugimoto et al., 2017):

where f_k = A_kf_s+m_k and sincx = sinx/x hold. Thus, the unique frequency m_k can be extracted. Note that the amplitude spectrum |β_mk| is affected by η and the sinc function. The attenuation of magnitude affects the accuracy of positioning, so η must be set properly. Note that X(dik) is the product of attenuation of the LED's signal and transfer efficiency. We assume the attenuation of the LED's signal is inversely proportional to the square of the distance and attenuates by the cosine of the radiation angle θ. However, it is difficult to model the attenuations theoretically because reflecting properties are very complicated in the real environment (Zhang and Yang, 2020). Our previous work (Shimada et al., 2018, 2020) comparing different mathematical models showed attenuation on the floor can be approximated by a hyperbolic secant distribution as shown in Figure 3. Therefore, the following equation

is used, where σ is the radiation characteristic of the LED, and C_k is the transmission efficiency determined by the receiver sensitivity. Hence, the amplitude spectrum |B_mki| is shown as follows:

Thus, dik can be calculated as

where Ck′=ηCksinc(πfkη)|βmk|.

From Equation (7), the distance dik cannot be estimated unless Ck′ and σ are obtained in advance by calibration. However, if all materials from each LED used for positioning are the same, calibration-free ranging can be achieved by determining Ck′ and σ that minimize the following summation:

The second step is to obtain P_Fi-captured positions on the floor using three or more dik estimated by the previous step. Note that P_Fi has a one-to-one correspondence with (u, v) on the U−V plane. Now, we assume that three perpendiculars from the light sources pass through the floor at the points (x_L0, y_L0), (x_L1, y_L1), and (x_L2, y_L2) (see Figure 2). Suppose that the estimated distances from these intersection points to the point (x_Fi, y_Fi) are dik as explained in Section 3.1. (x_Fi, y_Fi) is given by solving the following equation as an optimization problem:

The last step is to estimate the 3-D position and pose of the camera. These are calculated using the optical AoA method with the camera matrix:

where s is the scale coefficient, M denotes the intrinsic properties, and [R t] represents the extrinsic properties (Bradski and Kaehler, 2008). p is the 2-D image coordinates, and P defines the 3-D world coordinates as follows:

where the 3-D world coordinates (x_Fi, y_Fi) are captured by a camera as 2-D image coordinates (u_Ii, v_Ii) on the image. The camera position and rotation matrix [R t] are obtained by minimization of ||M[R| t]P−sp||₂.

BXRE-50C4001-B-74-type LEDs from Bridgelux were used as the LED transmitter. RefRec+ assumes that the light source is not an area or a line source, e.g., a flat panel or a bar light. Extending the experiments to include these sources remains to be addressed in future work. Four LEDs above the room were set as shown in Figure 4. The height of the ceiling is 2.6 m, coverage is a square surface of 4.0 m × 4.0 m, and LEDs 0, 1, 2, and 3 were set at (1.0, 0.5, 2.6), (1.0, 3.0, 2.6), (3.2, 0.5, 2.6), and (3.2, 3.0, 2.6) m, respectively. This setting was the same as the original built-in formation of fluorescent lights in this room. No other objects were placed in the room to evaluate the performance properly in an ideal environment. The signal parameters were set at f_s = 50, α = 1, A_k = 2 (at any k), and m_k = (1, 6, 13, 20). Hence, the LEDs were modulated at 101, 106, 113, and 120 Hz. These frequencies are higher than the 100 Hz modulation frequency for fluorescent lights in east Japan to ensure people do not experience any flickering. The transmitter represented any signal by pulse density modulation (PDM). A 5-V pulse signal from a function generator (NF Corporation WF-1948) was amplified to 34 V using a metal-oxide-semiconductor field-effect transistor (MOS-FET) and a power supply. The frequency of the pulse signal was about 8 MHz. This was a prototype, and the transmitter can be made cheaper and smaller using a circuit similar to a dimmable off-the-shelf LED in our future experiments.

An iPhone 7 was used as a receiver. Smartphones in recent years have generally higher-performance chipsets and cameras than the iPhone 7. Therefore, we believe that the experiments in this paper can be reproduced on more current smartphones. The frame rate of f_c = N = 50 was set to avoid effects from other fluorescent lights. In east Japan, fluorescent lights are modulated by AC 100 Hz, so 50 fps is the orthogonal frequency, which will treat other fluorescent lights just as DC sources (the same as sunshine). The F value is fixed on f/1.8 in the case of the iPhone 7. The focus was fixed on the floor. The distortion of the image was calibrated using Zhang's method (Zhang, 2000). A tripod was used to fix the smartphone.

Several experiments for clarifying the advantages and limitations of RefRec+ and how the performance of RefRec+ was determined are described in the following sections. The accuracy of the proposed system is most affected by its ranging performance. To reveal the ranging error, each distance was estimated by the following equation:

Note that (x_F0, y_F0) is the floor 2-D coordinates P_F0, calculated using the corresponding pixel captured by the camera. Unless otherwise noted, all the experiments regarding the ranging performance evaluations were conducted as follows: only LED 0 (k = 0) was used and each measurement point was set at 0.5 m intervals from just below the LED (d₀ = 0) to d₀ = 3.5 m, and 100 measurements were taken at each point. Based on each amplitude spectrum obtained, the distance d₀ was estimated by the parameter estimation using equation (7). In experiments except those in Section 5.2.3, the floor was illuminated only by the four LEDs. Existing fluorescent lights were turned off and direct sunlight was blocked using window shades.

The experiments described in Section 5.2.2 showed that floor material might affect the performance of RefRec+ seriously. Thus, a floor made from a non-patterned, non-gloss mat was chosen in experiments other than those in Section 5.2.2.

For estimating the 6DoF of the smartphone, nine P_Fi (0 ≤ i < 9) were chosen from one image stream with the resolution of 1,920 × 1,080 pixels. The principal point was (u_c, v_c) = (959.07, 524.86). As Figure 12 shows, the 2-D image coordinates of the points chosen are (u_Ij, v_Ik) = (300j+200−u_c, 300k+200−v_c) (0 ≤ j < 3, 0 ≤ k < 3).

By using P_Fis in each of 50 consecutive images captured in 1 s (the camera frame rate is set to 50 fps), RefRec+ conducts the Fast Fourier Transform to obtain intensities of reflected lights coming from individual LEDs. Examples of time and frequency domain representations of signals captured at P_F0, P_F4, and P_F8 are shown in Figure 13. The amplitude peaks are observed at 1, 6, 13 and 20 Hz in Figure 13B, which correspond to the emitted signals from LED 0 (101 Hz), LED1 (106 Hz), LED2 (113 Hz) and LED 3 (120 Hz), respectively. The 6DOF of the smartphone is calculated as described in Section 3. RefRec+ completes the 6DOF calculation before a new image is captured. When the new image is available, RefRec+ restarts the calculation in the same manner. Figure 14 shows the time variances of intensities of individual LEDs calculated using different P_Fis in 50 consecutive images captured between the 1st and 150th camera frames.

A camera was set in the room with parameters (x, y, z) = (1.0, 0.5, 1.15) and (θx,θy,θz)=(0,0,45∙). Each P_Fi position was estimated 100 times in real time as shown in Figure 15.

The accuracy of the estimated θ_z regarding how multiple P_Fis were chosen was evaluated. The total number of P_Fis was nine, so the total number of combinations of choosing more than one P_Fi from these was 2⁹−1−9 = 502. The estimated errors using different numbers of P_Fi to calculate θ_z are shown in Figure 16, as a box-and-whisker diagram. Based on this figure, the maximum, middle, and minimum errors (highest, middle, and lowest horizontal bars of each whisker); average error (‘x' on each box); and 25th and 75th percentile error (top and bottom of each box) are represented. When the number of P_Fi used was two, the estimated errors became larger than those of the other numbers, depending on which two were chosen from the nine P_Fis. The best 90th percentile absolute error was 6.15°, with the worst 90th percentile absolute error at 27.09°. When the baseline length [e.g., the distance between (u_I0, v_I0) and (u_I2, v_I2)] was long, the accuracy was improved. By contrast, the results for the cases with short baseline lengths [e.g., distance between (u_I1, v_I2) and (u_I2, v_I2)] were inaccurate. As the number of P_Fis increased, the difference between combinations (standard deviation) decreased. When the number of P_Fis used was eight, the performance differences between each combination composed of different chosen points became even closer: the best 90th percentile absolute error was 3.25°, and the worst 90th percentile absolute error was 5.77°.

Experiments for 6DoF estimation were conducted using all the P_Fi-captured positions. Results of the 3-D position estimation are shown in Figure 17. From the CDFs in the figure, the absolute errors at the 90th percentile for the 3-D coordinates were 0.2073, 0.1713, and 0.002464 m. The absolute 3-D position error at the 90th percentile was 0.27 m. The accuracy limitations of the smartphone's angle estimation by RefRec+ were investigated for each pose of the smartphone. The pitch, roll, and yaw of the smartphone were changed by 10° each, and its pose was estimated 100 times. Because of the movable range of a fixing apparatus of a smartphone, the measurable angles of roll, pitch, and yaw were limited to −30° to 30°, 0° to 30°, and 0° to 90°, respectively. Each 90th percentile absolute angle error at each posture of the smartphone is shown in Figure 18. Means of the 90th percentile absolute angle error were 5.69°, 5.78°, and 3.96° for the roll, pitch, and yaw, respectively.

We conducted preliminary experiments for 2-D position estimation to evaluate the performance of RefRec+ in mobile environments. A user (male, age 27, height 1.8 m) holding a smartphone was asked to walk along the six paths from right to left as shown in Figure 19A. Two hundred measurements per path were conducted. Figures 19B,C show heatmaps regarding the absolute errors and standard deviations. Bright colors mean that errors and standard deviations are large. From the figure, the positioning results in the right-side area indicated larger absolute errors and standard deviations than the left-side area. It was confirmed that the user often blocked the direct lights from LED 2 and LED3 in the former area and thus their reflected lights from the floor where his shadow was casted were not captured by the smartphone camera. Except the shadowed areas, RefRec+ could achieve absolute errors of less than 0.4 m. In order to mitigate the problem and improve the performance, we further investigate the influence of the shadow and test a couple of methods that can identify and avoid choosing POIs from shadowed areas (Yamashita et al., 2021).

From experimental results shown in Section 5.2, the distance estimation performance of RefRec+ was affected by the camera parameters (ISO and SS), measurement environments (floor materials, ambient lights), and number of pixels, which were somehow related to SNRs. Lower SNRs seemed to deteriorate the estimation performance. Although decreasing the number of pixels makes the estimation performance worse, it is expected to reduce the computational complexity for image processing and improve the power consumption (Afzalan and Jazizadeh, 2019), which is critical for the run time of the smartphone battery.

The proposed (Equation 7) did not consider multipath environments and thus could not estimate the distance accurately in such situations. Calibrations of the equation by considering influences of multipath lights via reflective objects (Abou-Shehada et al., 2021) can improve the performance of RefRec+. A multipath propagation model by integrating diffuse and specular surface assumptions and its model parameter estimation method are proposed and evaluated using different reflection surface materials (De-La-Llana-Calvo et al., 2017). More intensive investigations regarding the mitigation of multipath influences are needed in our future work.

RefRec+ uses RSS values of LED lights each of which has its unique frequency among them. Therefore, theoretically, the distance estimation performance is not affected by ambient lights if their frequencies are different from those of the LED lights. However, it was not confirmed to hold in the experiments. One possible explanation regarding this issue is the nonlinearity characteristics of the camera, which compresses intensity values to avoid whiteout when the power of the signal increases. Applying gamma correction (Reinhard et al., 2010) to alleviate the problem is our another future work.

A comparison of the performance of VLP is presented in Table 1. Note that each experimental environment differs in terms of LED installation spacing and measurement area size. Luxapose (Kuo et al., 2014) and iLAMP (Zhu and Zhang, 2017) were evaluated in mobile environments and the others were evaluated in static environments. Accuracy(°) and accuracy(m) show the 90th percentile absolute errors of 6DoF and 2-D/3-D positioning estimation using each method. Our method has the worst estimation accuracy for 6DoF. However, we claim that this result is sufficient for our application case because it satisfied our initial target described in Section 1. Density means how many LEDs are mounted on the 1 m² ceiling. The performance index (PI) proposed by Afzalan and Jazizadeh (2019) is derived by dividing the accuracy(m) by the density. Therefore, a smaller PI value means that a larger area can be covered with higher positioning accuracy using fewer LEDs. The table shows that only RefRec+ and (Zhu and Zhang, 2017)'s method achieve a PI of less than 0.1. As mentioned in Section 2, compared with Zhu and Zhang (2017), RefRec+ has advantages in that it does not require any dataset or any cloud computing. The resolution shows that RefRec+ uses fewer pixels for positioning. As a result, our method can complete the 6DOF calculation using 50 consecutive images in the fastest time and achieve the update rate of 50 Hz.

For calibration-free positioning, the performance of all LEDs used for positioning should be the same. However, individual LEDs often show different luminescence characteristics even while belonging to the same model. In the experiment, only one smartphone was tested as a receiver, and thus we did not observe any difference in results between different smartphones. The investigation of the difference between individual devices and models of transmitters and receivers remains as future work.

A 4.0 m × 4.0 m room was chosen in our experiment to assume the office environment. Assuming a real-world application, we need to experiment in a larger-scale environment. However, allocatable frequencies need to be considered. When set to N = 50, the total number of frequencies that can be assigned is 25, which may be inadequate for real-world applications such as a navigation system in a shopping mall. Hence, a new allocation method needs to be discussed.

Using the concept of the coloring problem, there exists a method of reassigning the same frequencies so that they are not adjacent (Yang et al., 2018). We plan to utilize phase-shift keying (PSK) to identify more LEDs. β_mk in Equation (4) includes phase information of a sinusoidal wave from kth LED. If 16-PSK is used for modulation, the number of LEDs that can be assigned increases to 25 × 16 = 400.

We do not intensively address the movement of the person holding the smartphone, although we describe preliminary experiments in Section 5.4. In particular, if the material of the floor changes during capturing, it is expected that positioning will become difficult. However, our method still has an advantage with regard to tracking because it can increase the possibilities for correctly predicting and identifying floor materials by estimating the 3-D position and pose of the smartphone. Further performance improvement, including the integration of the proposed method with pedestrian dead reckoning (PDR), remains future work.