ALLPCB
Overview
Sensor fusion combines multiple physical sensors to produce an accurate representation of reality even when individual sensors are unreliable. This article explains how and why sensor fusion is used.
Sensor fusion answers practical questions such as "Where am I?" or "What is actually happening?" For example, it is implemented in firmware for commercial quadcopters to prevent collisions with obstacles.
Sensors are not perfect and each has conditions that cause erroneous output.
Why multiple sensors help
An inertial measurement unit (IMU) can appear excellent on paper but behave poorly under certain conditions. For example, an IMU on a quadcopter without vibration damping can experience harmonics that match mechanical frequencies and produce false readings.
In such cases, an IMU can be affected by vibration within its specified range, leading to harmonic interference with nearby mechanical parts. An IMU designed for a smartphone may not tolerate the environment next to multiple motors spinning at high RPM. The sensor can produce readings that incorrectly indicate rotation, causing the controller to compensate and potentially lose control.
The typical response to imperfect sensors is to add more sensors, which reduces blind spots but increases mathematical complexity. Modern sensor fusion algorithms implement probabilistic belief propagation; the Kalman filter is a common example.
Kalman filters
A Kalman filter maintains a set of confidence factors for each sensor. On each iteration, sensor data is used to statistically improve the state estimate while also updating estimates of sensor quality.
Robotic systems include constraints that encode real-world knowledge, such as the expectation that physical objects move smoothly and continuously in space. That allows the algorithm to prefer consistent, plausible state transitions over abrupt, unlikely changes.
If a sensor that has been reliable suddenly reports implausible values (for example, a GPS when entering a tunnel), the filter reduces that sensor's confidence within a few iterations until its outputs are again consistent with other sensors.
Kalman filters are superior to simple averaging or voting because they can handle temporary sensor faults as long as at least one sensor remains reliable. They are an application of more general concepts such as Markov chains and Bayesian inference, which iteratively refine estimates based on evidence.
Kalman filters have been used for decades in satellite orbit maintenance and are increasingly applied in robotics because modern microcontrollers can run these algorithms in real time.
PID controllers
Simpler systems often use PID controllers. These can be thought of as a raw form of a Kalman filter where iterative statistical updates are replaced by three fixed tuning parameters. Whether tuned automatically or manually, the tuning process externalizes the belief-updating procedure and is performed by a human operator. The underlying principle of using feedback to correct estimates remains the same.
Custom and hybrid filters
Real systems are often hybrids. A full Kalman filter can include control inputs, for example: "I turned the steering left, the compass reports left, but the GPS suggests straight. Which to trust?"
Even simple control loops, like a thermostat, can be viewed in Kalman terms: the controller tweaks an actuator and observes the response to evaluate sensor and actuator quality. Where direct control influence is limited, cross-checking and non-teleportation constraints still help to reduce ambiguity.
Choosing and combining sensors
The remainder of this article focuses on physical position, but the same concepts apply to other quantities. Using multiple identical backup sensors can replicate the same weaknesses; heterogeneous sensor combinations are generally stronger.
Consider the example goal "I do not want my quadcopter to crash." No single commodity sensor provides 100% reliability. Robust operation is achieved by combining multiple sensors so that failure modes are isolated and contradictory evidence is resolved until the true state remains.
Below are typical sensors on a quadcopter, with their advantages, limitations, and roles in sensor fusion.
Positioning systems (GPS)
GPS is an obvious choice but has limitations. Sample error can be on the order of meters, and bias can drift with satellite geometry. High-precision centimeter-level positioning requires fixed reference and multi-day measurements, which is impractical for most mobile applications.
Even a 100 Hz GPS update is slow compared to a flight controller main loop and cannot provide smooth timing for high-speed aerial maneuvers. GPS also does not indicate heading directly; heading is inferred from movement direction.
Vertical resolution (altitude) is typically worse than horizontal resolution and can be on the order of a tenth of horizontal accuracy, so allow for substantial vertical uncertainty. GPS measures altitude relative to mean sea level, not distance to local ground, so additional measures are required for reliable altitude over varied terrain.
A single GPS receiver is insufficient for safe, low-altitude, high-speed flight without differential corrections, high-performance receivers, and good terrain models.
Sonar, LiDAR, Radar, Optical flow
If satellite positioning is insufficient for avoiding terrain, direct range sensing can be used. Common ranging techniques include sonar, LiDAR, and radar.
All ranging sensors have limitations: some surfaces do not reflect certain signals. Curtains and carpets absorb ultrasound, dark paint can absorb LiDAR, and water absorbs microwaves. A sensor cannot detect what does not reflect its signal.
Sensors of the same type placed near each other can interfere with one another. Addressing crosstalk requires waiting for echoes to die out, encoding signals, or randomizing ping schedules to avoid synchronization with others. Each approach increases complexity.
Optical flow uses a camera to observe whether features in the field of view expand, shrink, or translate, indicating motion toward, away from, or parallel to surfaces. Cameras do not interfere like sonar and can provide rich information, but they require surface texture and lighting; they perform poorly on featureless surfaces similarly to how optical mice fail on glass. Optical flow is increasingly feasible as flow computation can be implemented on FPGAs or fast embedded processors.
Geometry and tilt
Geometry matters. A sensor mounted on an aircraft sees ground distance affected by tilt: when tilted, the ground appears farther than it actually is. With sufficient tilt, a sensor can miss imminent collision even when only centimeters away. Correcting this requires knowing the platform's attitude.
Gyroscopes
Gyroscopes and accelerometers are commonly combined in an IMU because they naturally cover each other's weaknesses. Gyroscopes provide clean continuous measurements of rotational rate.
MEMS gyroscopes are based on microfabricated resonators. They are relatively immune to linear accelerations and small vibrations, within specifications, because those are orthogonal to rotational sensing.
Gyroscopes are used in inner control loops to maintain attitude. Some pilots disable accelerometers entirely and rely on gyro feedback, which highlights the gyro's importance.
The primary weakness of gyroscopes is drift: small biases accumulate over time and require periodic correction using external references.
Accelerometers
Accelerometers detect the direction of "down" in the presence of gravity and are therefore complementary to gyroscopes, but their measurements mix gravity, linear acceleration, centrifugal forces from rotation, vibration, and sensor noise. As a result, accelerometer data is noisy and ambiguous, even if sampled at high rates with good MEMS accuracy.
Converting acceleration to position requires two integrations, which accumulates errors rapidly. This is why GPS or other absolute references are used to periodically correct position drift, analogous to using accelerometer data to correct gyro drift in attitude estimation.
Magnetometers
A magnetic compass should not be overlooked. Like accelerometers, magnetometers are often used to control long-term gyroscope drift. Knowing magnetic north and gravity allows the system to determine true orientation on Earth.
Magnetometer readings are noisy, especially near motors or ferrous structures, and are affected by environmental factors. They are commonly paired with GPS: one provides coarse position, the other provides coarse heading, and together they mitigate mutual weaknesses. For ground robots constrained to wheels, this pairing is often sufficient.
