ALLPCB
Overview
Virtual reality (VR) refers to a set of technologies that simulate the real and virtual worlds and immerse users in a three-dimensional space, producing stereoscopic visual, auditory, haptic, and sometimes olfactory sensations that closely resemble the real world within a defined scope. VR is commonly described by three characteristics (3R): real-time rendering, real space, and real interaction.
The ongoing digitization has shifted many real-world activities into virtual spaces. Beyond communication, shopping, learning, manufacturing, gaming, and social interactions, people now spend increasing amounts of time in virtual environments.
Definitions: VR, AR, and MR
1. VR
Virtual reality is a computer-simulated system that creates and enables experience of a virtual world. It generates an artificial environment through multi-source information fusion, interactive three-dimensional dynamic scenes, and simulated physical behaviors so users feel immersed. VR emphasizes real-time rendering, real spatial representation, and authentic interaction.
Over decades, VR has significantly affected gaming, film, professional training, tourism, manufacturing, and military uses. With the spread of head-mounted displays and advances in remote communication, integrating virtual reality into real environments has become attractive, giving rise to augmented reality (AR). Advances that better fuse real and virtual environments have expanded mixed reality (MR).
2. AR
Augmented reality (AR) overlays virtual content onto a live view of the real world by computing camera position and orientation in real time and rendering images, video, or 3D models aligned to that view. AR augments the user's perception of the physical environment using computer-generated or sensor-derived inputs such as audio, video, graphics, or GPS data. Because AR depends on the user's real environment, it is often mobile-focused.
3. MR
Mixed reality (MR) merges real and virtual worlds to produce new environments where physical and digital objects coexist and interact in real time. MR encompasses both augmented reality and augmented virtuality and supports deeper integration between virtual and physical content.
Figure 1. Continuum from reality to virtuality
The left end of the continuum represents the real environment and the right end the virtual environment. Slightly modified real-world states are referred to as amplified reality, where computational methods enrich the physical properties of objects. Another intermediate concept is mediated reality. Augmented virtuality describes views from a virtual world that include real elements. In this article, VR, AR, and MR are collectively referred to as virtual reality (VR) for brevity.
History
The term "Virtual Reality" was popularized by Jaron Lanier in the late 1980s, though the roots of VR extend much earlier. The field progressed significantly in the 21st century as computing power, smartphones, high-resolution displays, and 3D rendering capabilities increased. The emergence of consumer headsets, such as Google Cardboard, Samsung Gear, Oculus Rift, and HTC Vive, accelerated development. Cloud computing enabled server-backed VR applications, and AR and MR gained mainstream attention. Major technology companies began building AR/MR platforms and ecosystems.
Microsoft demonstrated the HoloLens mixed reality device in 2015, showcasing holographic integration with the real world and opening interfaces for third-party developers to build applications on the platform.
Key Technical Areas
VR involves many technical domains. The following focuses on core areas, starting with modeling methods since modeling underpins virtual environments.
Virtual Reality Modeling
3D object modeling defines the objects that interact in real and virtual spaces. Modeling methods include:
1. Scene presentation modeling
- Depth image based modeling:
- Depth image acquisition and registration
- 3D surface reconstruction
- Model repair
- Image-based modeling:
- Geometry reconstruction from a single image
- Stereo vision and structured light methods for geometry reconstruction
- Landscape reconstruction using prior knowledge
- Contour-based geometric reconstruction
- Material and illumination modeling:
- Volume illumination modeling
- Active alteration models
- Varying density emission models
- Material classification and hybrid models
- Field modeling:
- Vector fields
- Scalar fields
2. Behavior modeling
Modeling behaviors of active objects (people, vehicles) is critical. Common methods include:
- Classification of autonomous object types
- Finite state machine based models
- Expert systems for modeling
- Agent-based modeling
- Aggregation and disaggregation models
3. Real-virtual fusion modeling
AR and MR require models that combine real and virtual scene elements. Challenges include:
- Acquisition and representation of real environment information
- 3D registration and storage between real and virtual spaces
- Handling inconsistencies between real and virtual elements
- Methods for compositing virtual and real components
4. Physics-based modeling
- Rigid body modeling:
- Rigid body dynamics simulation
- Collision detection
- Connectivity and constraint modeling
- Deformable object modeling:
- Discrete particle methods
- Continuum models
- Collision detection for deformable objects
- Virtual human motion modeling:
- Motion data acquisition
- Motion data processing
- Motion control
Computer Graphics and Animation
Visual perception is fundamental to VR, so computer graphics and animation are core technologies. Key topics include coordinate systems, vectors, transform matrices for visualization and animation, 3D geometric modeling, and rendering using lighting models, lights, color, and texture. Since virtual worlds are dynamic, animation techniques are required to synthesize desired motion effects for objects over time.
System Architecture
VR system architecture is a technical challenge. Architectures have evolved from primarily local deployments toward cloud-based approaches. A driving simulator VR system is a representative example.
Figure 2. Example driving simulator
For developing MR applications with Microsoft HoloLens, typical hardware and software requirements include a PC running 64-bit operating systems (Windows 10, macOS, or Linux), sufficient disk space (e.g., 30 GB free), at least 6 GB of RAM, and adequate CPU performance. Recommended CPUs are Intel Core i5 or i7 or equivalent, and a discrete GPU or graphics card.
For using the HoloLens emulator, minimum system requirements commonly cited include 64-bit Windows 10 Pro/Enterprise/Education, a quad-core CPU, 8 GB of RAM, host OS support for hardware virtualization, Second Level Address Translation (SLAT), and hardware Data Execution Prevention (DEP). A DirectX 11.0 or later GPU with WDDM 1.2 or later drivers is also required. Dedicated hardware such as HoloLens or an emulator is typically needed. 3D modeling can be performed using software such as Unity.
Perception Technologies
1. Vision
Display technologies have evolved from CRTs to LCDs, with LCD dominating current mainstream displays. VR displays are delivered via head-mounted displays (HMDs), projection systems, or desktop screens, with HMDs being the most popular. Typical HMDs use one or two small displays with lenses and can employ LCD, LCOS, or OLED panels. Many HMDs integrate audio output via headphones or speakers. Advanced devices such as HoloLens integrate multiple processors (CPU, GPU, and HPU), inertial measurement units (accelerometers, gyroscopes, magnetometers), multiple visible-light cameras, an infrared camera, a depth projector, microphones, and other sensors.
HMDs paired with tracking systems detect orientation and position changes so the VR application can render correct scenes. In medical applications, X-ray imagery may be composited with conventional visuals. While conventional HMDs deliver much of the immersion, limitations remain in display resolution, field of view, and other areas.
2. Auditory
Sound carries substantial information and is inherently spatial. Reproducing realistic spatial audio in virtual environments is a major challenge. Roles of audio in VR include:
- Supplementary information: audio provides cues for object direction, distance, and environment size through reverberation and echoes.
- Feedback: audio can confirm user actions or selections.
- Interaction channel: voice recognition and synthesis can enhance VR interaction.
Technical tasks include recording and reproducing spatial audio, synthesizing spatial sound fields that represent source position and environment characteristics, and real-time audio rendering synchronized with animation. Requirements for VR audio systems include precise 3D localization of virtual sources, acoustic space simulation to reflect room characteristics, and performance trade-offs between physical realism and real-time generation. Mapping virtual source positions to a limited number of physical speakers, constrained by installation, is an additional challenge.
3. Haptics
Most VR applications provide visual and auditory feedback but fewer provide force feedback. Many scenarios require haptic and kinesthetic perception to let users "touch" virtual objects. Haptics includes cutaneous skin contact and proprioceptive cues from muscles. Typical haptic devices and techniques include:
- Data gloves: Gloves are a primary input for hand-based interaction. They embed sensors to capture finger bends and motion trackers for position and orientation. Some gloves include actuators such as inflatable bladders to simulate contact forces, letting users feel pressure when grasping virtual objects. Commercial examples include CyberGlove and other data gloves with rich sensing and wireless interfaces.
- Haptic rendering and force generation: Haptic rendering computes forces in response to user interactions with virtual objects. Approaches include:
- 3-DOF haptic rendering for a single contact point with three translational degrees of freedom.
- 6-DOF haptic rendering for position and orientation, providing both forces and torques.
System Engineering and Software Framework
How to build a VR system
Developing and operating a VR system requires deep knowledge across sensing and tracking, stereoscopic displays, multimodal interaction, computer graphics, geometry modeling, dynamics and physical simulation, and performance tuning. Compared with conventional software systems, VR systems have distinguishing features: strict real-time requirements; modeling of object appearance, physical properties, and behavior; and diverse interaction paradigms tied to different devices and tasks. VR development presents multi-objective decision problems with potentially conflicting goals.
VR systems are typically developed iteratively. The first step is requirement analysis: describe the intended experience, workflows, scenario structure, timing, and interaction conditions, and estimate input/output devices and compute needs. Based on requirements, model the main virtual objects using CAD tools, then implement object behavior using graphics and VR libraries. Scenes are organized from these objects and rendered at high frame rates (for example 20 Hz or higher) to ensure smooth animation. Specific VR sensors and displays serve as interfaces. Through successive refinement, the system converges to the required functionality.
A representative iterative development outline:
- 1. Script design and requirement analysis
- 2. Object, global behavior, and system structure modeling
- System overview design and requirement revision
- System design
- System simulation, verification, performance tuning, and task allocation
- 3. Performance and task decomposition, interaction model, information, function, and behavior refinement
- 4. Presentation and special effects implementation
VR system engineering
VR/AR/MR development is a specialized systems engineering discipline. Key areas include interaction design, software and hardware co-design, lifecycle management, and application development.
Interaction design
Interaction frameworks for MR treat mixed objects as hybrid entities with both virtual and real attributes, guiding iterative user-centered design and embedded MR environments. Engineering-focused approaches consider coordinated hardware and software design and model-driven development, evolving conceptual models into executable software through model transformations.
Lifecycle
Like other software systems, VR systems require lifecycle management for design, development, maintenance, and evolution. Managing the lifecycle of VR systems is a nontrivial engineering challenge.
Applications
VR technologies are applied beyond entertainment to healthcare, education, industry, and defense.
Healthcare
- Virtual surgery: Preoperative rehearsal and skill improvement in virtual environments can assist surgeons in preparing for operations.
- Virtual rehabilitation and therapy: VR systems for rehabilitation integrate haptics and advanced sensors to recognize motion patterns, generate therapeutic tasks, and facilitate diagnosis through data collection and model-based analysis.
- Psychotherapy: VR can be used to expose patients to controlled social environments, for example helping autistic children acclimate to crowds in a staged virtual setting.
- Virtual anatomy: Medical students can use VR to learn anatomical structures and procedures without the constraints of physical specimen availability.
Industrial and manufacturing
Computational tools such as CAD, CAM, and assembly planning are widely used in product lifecycles. VR enhances CAD/CAM capabilities and aids tasks like assembly, maintenance, and training. For example, Boeing developed a head-up perspective system in the 1990s to overlay virtual wiring harness guides on workers' view during aircraft assembly, improving installation efficiency and reducing errors. The term "augmented reality" was coined in this context to describe overlaying virtual guidance based on user position and head orientation.
Figure 3. VR in product design
Military
VR has long been used for military applications, including situation awareness by fusing reconnaissance data from satellites, radar, and sensors into a virtual battlefield representation, and for training in pilot, missile, and ship systems to reduce cost and risk. Other applications include individual soldier systems and flight simulators.
Trends and Challenges
VR is advancing rapidly. Investment and market forecasts have indicated strong growth globally and in the Chinese market across technologies, hardware, games, platforms, education, architecture, and industry applications.
Key technical challenges and opportunities include:
- Processor demands: The 3R requirements place high demands on CPUs, GPUs, and emerging processing units like HPUs, and on their integration.
- Perception and actuation hardware/software: Sensors, trackers, and actuators require continued development.
- Modeling technologies: Continued progress is needed in VR modeling techniques.
- Development methods: VR requires hardware-software co-design methods distinct from conventional software development.
- Big data processing: VR systems collect large volumes of sensor data that must be fused and processed in real time.
- Artificial intelligence: AR and MR rely on AI and large-scale data processing; algorithms for AR/MR remain in early stages.
- Context-aware computing: VR systems benefit from adaptive, responsive, and context-sensitive computation that remains an open research area.
