ALLPCB
Overview
In virtual reality and augmented reality systems, predictive tracking estimates an object or a body part's next orientation and/or position. For example, when your head turns in a particular direction, the system can simultaneously predict where your hand is likely to move.
Why predictive tracking is useful
A common application of predictive tracking is reducing motion-to-photon latency. Motion-to-photon latency is the time between a user performing an action in a VR or AR environment and that action being fully reflected on the display. Motion itself introduces delay, and additional time is required for the display update (see causes of latency below). If the system can predict the next orientation and position and prioritize updating those values on the display, perceived latency can be greatly reduced.
While predictive tracking is often discussed in VR contexts, it is equally important for AR, especially when the user moves suddenly in the real world and AR overlays must update on the display. For example, if an AR headset overlays graphics on top of a physical object, those overlays need to remain locked to the object even when the user rotates their head, since they represent part of the real scene. An object may be identified by the camera, but the camera still needs time to capture frames so a processor can determine the object's position within the frame, and the graphics processor then needs to render the overlay at the new position. Using predictive tracking can reduce the amount of motion processing required for the overlay compared with waiting for the real-world measurement chain to complete.
How predictive tracking works
If you see a car moving at a constant speed, predicting where the car will be one second later is relatively straightforward and usually accurate. You know the car's current position and its current or estimated speed, so you can infer its future position.
However, predicting the car's position over a period is not always 100% accurate, because the car could change direction or accelerate during that time. In general, the farther ahead you try to predict, the less accurate the prediction becomes. Predicting the car's position one second ahead will usually be more accurate than predicting one minute ahead.
Also, the more information you have about the object, the better the prediction. For example, measuring the car's acceleration in addition to its speed allows for a more accurate forecast.
Similarly, when tracking a head, knowing the head's angular velocity and potential rotation range lets you refine the tracking model for more accurate predictions. Eye tracking data can also be used to predict head motion, which can further improve predictive performance.
