ALLPCB
Introduction
Technological development is rapidly changing daily life. Where cash or card used to be the common payment question, people now ask whether to use mobile payment apps. Carrying cash gave way to carrying cards, and now a mobile phone is often sufficient.
In the future, people may not even need to carry a mobile phone when they go out; they may only need to "carry their face" as we move toward widespread face-based authentication. Personal information and assets could be bound to a person's facial identity, enabling authentication by face alone. This article provides a detailed overview of face recognition technology.
What Is Face Recognition
Face recognition is a biometric identification technology that uses facial feature information to verify identity. Cameras capture images or video streams that contain faces, and the system automatically detects and tracks faces in the imagery, then applies a series of techniques to the detected faces. It is also referred to as portrait recognition or facial recognition.
Face recognition is a prominent research area in computer vision and belongs to biometric recognition technologies, which distinguish individual organisms, typically humans, based on biological characteristics.
Biometric recognition studies characteristics such as face, fingerprint, palm print, iris, retina, voice (speech), body shape, and behavioral patterns (for example, typing dynamics or signature). Corresponding recognition systems include face recognition, fingerprint recognition, palmprint recognition, iris recognition, retina recognition, voice recognition (which can be used for identity verification or speech content recognition; only the former is a biometric identification use), body-shape recognition, keystroke dynamics, and signature verification.
Three Key Detection Techniques
1. Feature-based face detection
Detects faces using color, contours, texture, structure, or histogram features.
2. Template-matching face detection
Extracts facial templates from a database and applies template-matching strategies so that a captured face image matches templates in the library. Correlation levels and matched template sizes determine face size and location.
3. Statistical face detection
Builds large positive and negative sample libraries of face and non-face images and uses statistical methods to train the system to detect and classify face versus non-face patterns.
Four Feature Types
1. Geometric features
Use distances and ratios between facial points as features. This approach is fast, requires relatively little memory, and is less sensitive to illumination.
2. Model-based features
Extract face image features based on the probabilistic states of different facial attributes.
3. Statistical features
Treat the face image as a random vector and apply statistical methods to distinguish facial patterns. Typical methods include eigenfaces, independent component analysis, and singular value decomposition.
4. Neural network features
Use large numbers of neural units to associative-store and memorize face image features, and identify faces based on probabilistic states of neural units.
Ten Major Challenges
1. Illumination
Lighting variation is the most critical factor affecting face recognition performance and determines how practical the technology can be. The 3D structure of faces creates shadows that can amplify or obscure facial features. Poor lighting, especially at night, can dramatically reduce recognition rates and hinder practical deployment. Experiments show that differences in appearance of the same person under different lighting conditions can exceed differences between different individuals under the same lighting. Illumination is a long-standing problem in machine vision and is particularly challenging for face recognition. Potential solutions include 3D face imaging and thermal imaging, but these approaches are still immature and yield mixed results.
2. Pose
Face recognition primarily relies on frontal facial appearance. Pose-induced facial changes are a major difficulty, involving rotations of the head around three axes in 3D space. Rotations perpendicular to the image plane can cause partial loss of facial information. Most research focuses on frontal or near-frontal images, so recognition rates drop sharply for large yaw or pitch angles.
3. Expression
Large facial expressions such as crying, laughing, or anger also affect recognition accuracy. Current techniques handle many expressions reasonably well; methods like 3D modeling and pose-expression normalization can correct for open mouths and exaggerated expressions.
4. Occlusion
When faces are captured non-cooperatively, occlusion is a serious problem. In surveillance environments, subjects often wear glasses, hats, or other accessories that can make face images incomplete, affecting feature extraction and recognition, and sometimes causing face detection to fail.
5. Aging
Facial appearance can change significantly as a person ages, reducing recognition rates. Different age groups may yield different algorithmic performance. A direct example is ID photo verification: in China, ID validity is generally 20 years, during which a person's appearance can change substantially, creating recognition challenges.
6. Face similarity
Different individuals can have highly similar facial structures, even similar shapes of facial organs. While this similarity helps with face localization, it makes distinguishing individuals more difficult.
7. Dynamic recognition
In non-cooperative scenarios, motion blur or incorrect camera focus caused by subject movement can severely impact recognition success. This difficulty is evident in security applications such as subway, highway checkpoints, stations, retail loss prevention, and border control.
8. Anti-spoofing
Common spoofing approaches include creating 3D models or grafting facial expressions. As anti-spoofing methods, 3D face recognition, and advanced camera-based visual computing evolve, spoofing success rates decrease.
9. Image quality
Face images come from diverse sources and capture devices with varying quality. Low-resolution, high-noise, or otherwise poor-quality images from mobile cameras or remote surveillance present recognition challenges. High-resolution images also introduce different effects that need study. Most research uses images of similar size and clarity, but real-world applications require handling far more variability.
10. Limited samples
Statistical learning methods are currently dominant in face recognition, but they require large training sets. Face images are distributed on an irregular manifold in high-dimensional space, and available samples represent only a small portion of that space. Solving statistical learning under small-sample conditions remains an open problem. In addition, most public training databases are populated with faces from non-Asian populations; image databases for Chinese and Asian faces are much rarer, which complicates training models for those populations.
Application Dimensions
1. Dynamic scenarios: two modes
First, 1:1 verification. This compares two faces to verify identity. It is commonly used in finance and identity verification scenarios where precision and security are essential, such as bank account opening and real-name verification for payment platforms.
Second, 1:N identification. This searches a database or a gallery to determine whether a person exists in that gallery. It is an identification process used dynamically and non-cooperatively, for example in security when locating fugitives. Commercial scenarios also often involve non-cooperative subjects, such as VIP customers, employees, or members who are not asked to consciously interact with cameras.
2. Business scenario dimensions
First, market scale sufficient to support long-term company development. Second, data feedback loops. Third, whether the scenario involves frequent usage. Fourth, whether the solution is replicable and can evolve from a 1+0 model to 1+N to improve efficiency.
3. Visualization system dimensions
First, personnel access control. Second, integration of sensor networks. Third, integration of commercial real estate and new retail systems.
Application domains include finance, judiciary, security, border control, aerospace, utilities, education, and healthcare.
Four commercial opportunities: turnstiles, transportation, banking, and mobile devices.
Specific Application Scenarios
1. Financial Services
1. Self-service face recognition terminals
Used for manual review, self-service account opening, service changes, and password resets for personal banking services.
2. Mobile banking and sales
Remote identity verification, including user identity confirmation and portable face-enabled devices used by financial institutions for on-site services.
3. Teller systems
Networked face verification for account opening and other counter services in banks, insurance, and securities institutions.
2. Airport Applications in China
Three key milestones: initial trials, boarding, and full automation.
Notable events:
1. In 2009, Beijing Capital Airport conducted the first attempts to apply face recognition, representing the first step for airports in China to explore the technology. Due to the technology level at the time, magnetic cards were used for cross-validation to ensure unique identities. Recognition accuracy and speed then were not comparable to systems using deep learning.
2. In 2014, Nanjing Lukou Airport first trialed face recognition for boarding. While technical and commercial limitations prevented fully autonomous processing, the project provided experience for future deployments.
3. In December 2016, Yinchuan Airport implemented comprehensive automation, marking a new stage in airport intelligence. Beyond security screening and self-service boarding, face recognition and related computer vision techniques were applied to dynamic control, crowd guidance, smart flight displays, VIP reception, trajectory retrieval, and cleaning reminders, helping pave the way for broader adoption in 2017.
China Southern Airlines was the first airline in China to use face recognition technology for boarding. Flight CZ3384 became the first to board with the new technology; passengers could pass through the gate quickly by face scan without holding a boarding pass.
3. Pedestrian Enforcement in China
1. Face recognition can reduce enforcement costs.
2. Authorities must adhere to the rule of law to prevent enforcement abuses.
3. It helps resolve conflicts over right-of-way and avoid ad hoc enforcement.
Example: Jinan police reported that face recognition systems were used to capture pedestrians and non-motorized vehicle riders who ran red lights, producing clear images even at night. Short videos and enlarged headshots of violators were displayed on intersection screens. The system was connected to resident identity databases, and identified violators' names and ID numbers were shown on electronic displays.
After deployment in Jinan, the system captured over 6,200 pedestrian and non-motor vehicle red-light violations in one month. The visible deterrent reduced violations significantly at some intersections, with daily violator counts dropping from over one hundred to around a dozen.
In Chongqing Jiangbei, the pilot face recognition system reportedly increased pedestrian compliance from 60% to over 97%.
Risks: Public exposure of personal information raises privacy concerns. Experts recommend that authorities announce information collection areas in advance, informing the public that behavior will be recorded and publicly displayed. Collected data should undergo appropriate technical processing to obscure private information not suitable for public exposure.
Root causes: Experts note that poorly designed traffic facilities are often the main reason pedestrians run red lights. Some urban road networks emphasize main roads while secondary roads lack sufficient density, concentrating pedestrians and non-motorized traffic on main arteries. Poor signal timing can require excessive patience or speed to comply. Comprehensive planning to resolve conflicts between pedestrians and vehicles is necessary to address the underlying causes of jaywalking.
4. Education
Face-based authentication is used for exam candidate verification, campus and dormitory access control, and related scenarios.
Example: In 2016, exam authorities in multiple provinces adopted combined face and fingerprint biometric systems to verify candidate identity and prevent impersonation and cheating during major examinations.
As pilots and cross-sector implementations expand and operational models mature, industry reports projected potential large-scale adoption of face recognition.
5. Public Security
Key capabilities include face capture and tracking, face recognition computation, and face modeling and retrieval. Law enforcement uses both backend dynamic face recognition systems and front-end handheld face recognition devices and identity comparison terminals.
6. Healthcare
1. Community health check applications
When community users record biometric readings with digital devices such as electronic blood pressure monitors, scales, and glucose meters, combining those readings with live face data creates unique, verifiable identity-linked records. Each authenticated record is logged, enabling efficient feedback to clinicians and patients and facilitating personalized treatment plans.
2. Secondary and higher-care institutions
Face recognition at kiosks, counters, and clinics can link patient records via facial identifiers, enabling patients to retrieve medical histories and records by face scan.
7. Smart City Applications
1. Pension disbursement management
Face recognition can verify recipients and reduce pension fraud.
2. Tax office authentication systems
Face recognition can compare captured facial images with identity photos held by public security agencies to complete real-name authentication in real time, easing counter workload and improving tax service efficiency while reducing tax-related risks.
3. Suspect tracking systems
Using face recognition to monitor public venues like long-distance bus stations and train stations can compare faces in surveillance video against suspect databases; when a match occurs, alerts are generated, reducing staff workload and improving apprehension efficiency.
4. Community management
Non-cooperative face recognition can assist property management with visitor control and notifications such as utility or garage information, improving resident experience.
5. Building access control
Face recognition access systems can provide fast and accurate identity verification to open doors and enhance security for buildings and residences.
6. Candidate verification systems
Exam systems combining computing, communications, networking, face recognition, and databases can extract and verify candidate identity information to create more efficient and fair exam environments.
7. Driver trainee verification and safety management
Systems can verify attendance, authenticate trainees, log vehicle entry and exit, and control driving times.
8. Intelligent dining systems
Face recognition at dining points can record students' meal choices and compare them with health check data to recommend dietary adjustments and record excessive food waste for menu optimization.
9. Commercial intelligence
Face recognition can infer customer attributes such as gender, age, and mood to deliver targeted content and support customer flow and precision sales. Observing and learning from different audience responses can improve the accuracy of content recommendations over time.
Safety and Security Concerns
1. Face recognition consists of two main components:
Face matching, which determines whether a presented face matches the claimed identity, and liveness detection, which verifies that the presented face is live and not a spoof.
2. Common attack methods:
Matching can be easily spoofed if an attacker possesses a photograph of the target. If a person frequently posts selfies, those images can be used for spoofing, and taking surreptitious photos can also be trivial.
Liveness detection, while critical, can be bypassed using simple mechanisms originally developed for beauty or novelty camera features that overlay accessories on faces. By mapping such overlays onto still photos and replaying them, attackers can sometimes bypass liveness checks.
These weaknesses can render face recognition ineffective if not properly addressed. As face recognition expands into healthcare, social security, rail access, airport security, and other sensitive domains, vendors and deployers must remain vigilant about security risks that can accompany rapid adoption.
Experts recommend combining face recognition with other biometric modalities such as voiceprint, fingerprint, and iris, or with other authentication signals, rather than relying on face recognition alone in high-security scenarios. Multi-modal authentication significantly improves overall security.
Ultimately, whether face recognition is secure is a technical question, but the critical issue is the broader recognition and management of security risks in artificial intelligence and related technologies.
