ALLPCB
Overview
Augmented reality (AR) and virtual reality (VR) technologies have attracted strong interest in industry and academia. They augment the real-world environment or replace it with simulated environments. End-to-end latency in AR/VR devices can significantly affect user experience. In particular, motion-to-photons latency, defined as the time from a user action to the corresponding visual feedback appearing on the display, is one of the main challenges for AR/VR applications. Motion-to-photons latency above about 20 ms can cause nausea or dizziness.
Why Predict Head Orientation
Reducing motion-to-photons latency is critical for improving users' virtual experience. A common approach is to predict the user's future head orientation so the device can render upcoming frames in advance. Accurate head-orientation prediction algorithms are therefore essential for AR/VR systems. Beyond AR/VR, orientation prediction is also important for drones, robots, and navigation systems.
Sensor Sources and Trade-offs
Orientation prediction uses measurements from various sources, such as inertial sensors or monocular/stereo cameras, to estimate the current and future orientation. Prior work has combined inertial sensor and camera measurements for orientation estimation: inertial sensors provide orientation estimates by integrating angular rates, while camera outputs compensate for integration drift. However, camera-based solutions add complexity and are less suitable for resource-constrained devices. They are also affected by lighting changes.
Advances in MEMS have enabled smaller and lower-cost inertial measurement units (IMUs). With high sampling rates, low latency, and small size, IMUs are widely used in modern AR/VR devices to provide positioning and orientation data.
IMU Measurement Principles
Commercial IMUs typically include a three-axis gyroscope, a three-axis accelerometer, and sometimes a magnetometer. The gyroscope measures angular velocity, which can be integrated over time to obtain orientation angles. However, integration accumulates sensor errors, causing estimated orientation to drift from true orientation over time. Accelerometers can measure gravity to provide a reference vector for correcting orientation estimates. Magnetometers measure the Earth's magnetic field and can also be used for correction, but they are sensitive to magnetic interference from nearby metal or electrical equipment, limiting their applicability.
Filtering Approaches
Several orientation estimation and prediction algorithms for AR/VR devices have been developed using IMU data with Kalman filtering (KF) and particle filtering (PF). Kalman-based algorithms perform recursive two-step prediction and correction to derive optimal estimates. In the prediction step, the filter predicts the future orientation and its uncertainty. Once measurements are observed, a weighted average corrects the prediction, assigning greater weight to estimates with lower uncertainty.
Particle filtering is a recursive Bayesian state estimation approach that represents the state probability density with random samples. Because particle filters can handle arbitrary state-space models, they are applicable to a wide range of problems. However, despite higher complexity, particle filters have not shown significant improvements over Kalman-based methods for this task.
As a result, Kalman filtering is more commonly used for orientation estimation and prediction. Standard Kalman filtering is optimal only under linear assumptions, so variants such as the extended Kalman filter (EKF) and unscented Kalman filter (UKF) are widely used for nonlinear systems. The EKF linearizes the state transition function around the current estimate to perform prediction. When the state transition and observation models are highly nonlinear, the EKF may perform poorly because of uncertainty propagation. The UKF replaces linearization with a deterministic sampling scheme called the unscented transform, which selects sample points around the mean. However, UKF-based algorithms can be computationally expensive and are not always suitable for devices with limited compute resources, such as many AR/VR headsets.
Proposed Improvement
Researchers from Dongguk University have proposed a new orientation prediction algorithm that addresses limitations of traditional predictors to improve accuracy. The proposed method compensates for errors introduced by linear prediction by estimating linear displacement from the current orientation and applying a subsequent correction.
The current prediction is adjusted based on the accuracy of the previous prediction. To do this, the algorithm computes the orientation difference between the prior linear prediction and the current orientation. A weight matrix is then defined to determine the optimal adjustment to the current prediction based on that orientation difference.
Similar to the Kalman derivation, where differences with lower uncertainty receive larger weights, the optimal weight matrix here is obtained as a minimum mean square error (MMSE) estimator using the prediction error between the current and previous predicted orientations.
Summary and Results
Overall, the researchers proposed an IMU-based orientation prediction algorithm for AR/VR devices that uses a Kalman-like error compensation approach and an IMU that includes a three-axis gyroscope and three-axis accelerometer. The main contribution is improving the current prediction using the accuracy of prior predictions and determining the optimal corrective adjustment based on orientation differences. Experimental results show the algorithm outperforms conventional orientation prediction methods on several public datasets.
