ALLPCB
Editor's note
In August 2023 the Beihang University Hangzhou Innovation Research Institute hosted the "5th Domestic Embedded Operating System Technology and Industry Development Forum and Embedded Systems Association Thematic Meeting" (session 29). The meeting theme was "Research and Applications of Hybrid-Critical Systems for Heterogeneous Multicore Intelligent Chips". The journals invited the research teams led by Professor Jianwei Niu of Beihang University and Professor Jinhhu Jiang of Fudan University to submit two papers.
One paper, "Operating System Intrinsic Security for Multikernel Systems", proposes a multikernel-based intrinsic security approach to address functional safety and cybersecurity issues arising from increasing device intelligence and connectivity. The other paper, "An Operating System for Industrial Robots Based on Hybrid-Critical Systems", proposes a hybrid-critical system architecture for robot operating systems to improve real-time performance, safety, and intelligence. The presentation materials are available from the Embedded Systems Association website.
0 Introduction
As devices proliferate and become more intelligent and networked, new challenges and attack surfaces emerge for both functional safety and cybersecurity. The operating system is the software foundation that abstracts hardware resources and mediates between user applications and the hardware, and it therefore faces significant security challenges. The theory and practice of operating system security are critical.
Traditional cybersecurity has been predominantly static and reactive, driven by security events, while functional safety has relied on redundancy. Traditional network security has evolved through three phases: system hardening to prevent intrusion, monitoring and response to limit damage, and survivability techniques to maintain system function under compromise. Each phase addresses different eras of networked system threats but has inherent limitations. Hardening relies on clearly defined boundaries and controls such as authentication and isolation, but cannot fully mitigate vulnerabilities or backdoors. Monitoring and response rely heavily on detection capabilities and generally cannot catch novel attack techniques. Survivability techniques aim to minimize the impact of intrusions, but large-scale deployments are limited and models depend on extensive prior knowledge, which does not cover unknown attack patterns.
With the increasing intelligence and networking of digital infrastructure, cybersecurity and functional safety are converging into the intrinsic security problem. Intrinsic security, proposed in 2013, is based on the internal security capabilities of network elements and leverages system architecture, algorithms, mechanisms, and scenarios to derive security properties. Intrinsic security has two core characteristics: built-in construction and adaptive growth. Built-in construction means security must be integrated during system design and construction. Adaptive growth requires systems to interact with their environment so security capabilities can improve dynamically as conditions change. Intrinsic security techniques have been applied to honeypots, firewalls, switches, routers, processors, web and DNS servers, cloud platforms, and network operating systems. Their built-in and adaptive characteristics pose new challenges for computing systems.
This article analyzes operating systems from the intrinsic security perspective and discusses the challenges and opportunities for operating system intrinsic security.
1 Current status of research
Intrinsic security techniques for operating systems fall into two categories. One targets functional safety using redundancy and fault-tolerance, for example triple-redundant systems used on spacecraft and redundant autonomous driving systems developed by industrial vendors. The other targets cybersecurity using mobile-target defense (MTD) and morphing architectures. MTD has seen substantial industrial adoption; research on MTD theory, strategy, and evaluation is active in academia and industry.
China's mainstream intrinsic security approach is the morphing defense framework. Morphing defense assumes vulnerabilities and backdoors are inevitable and unknown; it uses dynamic heterogeneous redundancy to address both functional safety and cybersecurity, making conditional avoidance so attackers cannot form effective attack paths. Morphing defense has been applied to network resilience engineering, routing and switching devices, industrial control systems, cloud and vehicular networks, wireless system security, AI systems, and intelligent chips, with some large-scale industrial deployments. Thus, morphing-architecture-based intrinsic security is a promising way to fundamentally enhance operating system protection.
2 Challenges and opportunities
The operating system is the core system software. It must abstract hardware resources and provide services to applications, offering standardized interfaces and a stable execution environment. It also manages hardware resources and application lifecycles, acting as the system coordinator and manager. Security is one of its key responsibilities.
Traditional operating systems rely on passive defenses based on known attacks and vulnerabilities and cannot handle unknown attacks. Dynamic heterogeneous redundancy based on morphing security is the main proactive defense approach, but implementing intrinsic security at the operating system layer faces several key challenges.
First, many existing operating systems (for example Linux, macOS, and other single-kernel systems) have architectural fragility, monotony, and static behavior that prevent the heterogeneous redundancy required for intrinsic security. Fragility means small errors can disable critical components and lead to system-level collapse. Redundancy can mitigate such functional safety problems, but current OSes lack on-machine redundancy; most components lie on the critical path, so compromise of any core module can cause system-wide failure. Monotony and lack of diversity make differential problems prominent and prevent removing inherent design defects. High static determinism provides attackers with a consistent and transparent system view. Many attacks rely on runtime probing of key data; dynamic temporal variability reduces attack success probability. Current functional safety and network defenses also struggle to handle post-escape attack scenarios. A new OS architecture is needed to provide heterogeneous redundancy.
Second, how to build a dynamic heterogeneous redundancy mechanism at the OS layer. Such a mechanism requires redundancy, heterogeneity, and dynamism. Redundancy means the OS must provide multiple application-transparent execution environments: from a single application image, the OS must construct multiple execution environments and run identical instances in each. Heterogeneity means providing functionally equivalent but heterogeneous execution environments so the application produces equivalent outputs across environments. Dynamism requires the OS to support iterative changes in the set of execution environments.
Third, how to design an efficient consensus mechanism at the OS layer. The same application may produce inconsistent outputs across heterogeneous environments; the system must determine the cause of divergence. Differences may stem from external attacks, inherent heterogeneity, or non-determinism in redundant environments, and the OS must adjudicate. The OS must monitor and collect runtime information from each kernel and derive consensus via voting based on these observations and decision policies.
3 Operating system intrinsic security design
As the core foundational software between hardware and applications, the OS's intrinsic security architecture directly determines coordination during dynamic heterogeneous redundancy. Multikernel operating systems that run multiple kernels concurrently are naturally suited to intrinsic security. A multikernel OS can host homogeneous or heterogeneous kernels that coordinate via efficient inter-kernel communication. Treating kernels as scene elements allows dynamic construction of heterogeneous redundant kernels to provide differentiated execution environments, making the multikernel OS an effective platform for intrinsic security.
Fig. 1 Multikernel operating system architecture
3.1 Intrinsic security architecture
The intrinsic security architecture for a multikernel operating system is shown in Fig. 2. The core challenge is maintaining efficient coordination during runtime while providing ease of use. This requires well-defined service interface, kernel interaction, and resource management layers to achieve easy-to-use services upward, efficient inter-kernel coordination internally, and efficient resource utilization downward.
The service interface layer studies how to provide simple, application-facing interfaces after performing heterogeneous redundancy transformations at the kernel and file-system level, enabling application-system interaction and cooperation. The kernel interaction layer studies the basic information and interfaces needed for efficient communication between heterogeneous kernels, including communication mechanisms and consensus interfaces. The resource management layer studies hardware partitioning and management: resources should be relatively independent and isolated across kernels, including compute, memory, and devices. Virtualization and related techniques partition resources and manage access control definitions.
Fig. 2 Internal security architecture of the operating system
3.2 Heterogeneous redundancy mechanism
The kernel is the core privileged control module of the software stack, so kernel-level heterogeneous redundancy is central to intrinsic security. It protects critical components and provides stronger support for upper-layer applications. Kernel heterogeneity should be achieved via multi-dimensional control and data transformations; this requires studying transformable dimensions, appropriate transformation methods, potential heterogeneity, overhead, and usability. Design must also consider application memory layout and execution path requirements. The system architecture for kernel heterogeneous redundancy is shown in Fig. 3.
Fig. 3 Heterogeneous redundant architecture of kernel
Redundancy design includes multi-path execution and inter-kernel isolation. Applications are dispatched to different kernels for multi-path execution; all kernels in the execution set perform the task in a semantically consistent environment. External inputs, environment variables, and system configuration are replicated and dispatched to multiple kernels. Each kernel independently executes and monitors its user programs, producing outputs that should match expectations. Kernels must also be isolated from each other. In traditional systems, applications run at low privilege and the kernel holds high privilege and all system authority. Unknown network attacks may leverage applications to attack the kernel. To provide intrinsic security redundancy, a kernel crash should not collapse the entire system; the OS must provide kernel-level fault tolerance. Since kernels may contain unknown vulnerabilities or backdoors and could be compromised, the system must limit kernel privileges to prevent a malicious kernel from probing or attacking others, so the multikernel OS must support kernel-level protection.
Heterogeneity can be designed across both the control plane and data plane. The kernel control plane includes system calls, execution paths, and instruction heterogeneity; kernel heterogeneity therefore centers on these aspects. System calls are the interaction interface that kernel provides to applications and directly determine available services; runtime randomization of system call interfaces and functional transformations of interface semantics can realize system-call-level heterogeneity. Execution-path heterogeneity can be achieved by runtime randomization of application execution paths, such as global scheduling order, lock acquisition/release sequences, and random number generation. However, system-function randomization faces challenges in implementation and scalability: randomizing every kernel's system calls complicates adding or removing kernels, and similar system calls across kernels create coupling that attackers can exploit. Execution-path heterogeneity also creates consensus and implementation challenges. Identifying application execution paths across kernels requires efficient consensus mechanisms, but kernel complexity and execution-path divergence make consensus difficult. Thus, relying only on system-call and execution-path heterogeneity is insufficient for intrinsic security. We propose instruction-level randomization to complement these approaches.
Instruction-level heterogeneity is primarily achieved by instruction randomization: encrypting machine code instructions at compile time and decrypting them before execution. This blocks many classes of code-reuse or binary-level attacks. Instruction randomization provides natural heterogeneity and meets intrinsic security requirements at a lower, finer granularity than system-call randomization while remaining transparent to applications and hardware.
For the kernel data plane, heterogeneity centers on memory layout and executable formats. Memory layout heterogeneity can be achieved through multi-target, multi-granularity address-space randomization: load base addresses of binaries and shared libraries, user and kernel stack offsets, ordering of code and data segments, encrypted representations on the stack, and so on, creating heterogeneous execution characteristics at runtime. Executable-format heterogeneity can be supported by transforming executable file formats and adapting loaders to handle the transformed formats. In summary, multikernel heterogeneous redundancy can be realized via instruction heterogeneity, memory heterogeneity, and loader/executable-format heterogeneity.
3.3 Efficient communication and consensus mechanisms
Inter-kernel communication and consensus are the basis for efficient interaction and coordination in an intrinsic security system. Inter-kernel communication efficiency directly affects coordination performance. In single-kernel systems, inter-process communication and remote procedure calls are rich and well-studied. For cross-kernel processes or modules on a single host, data transfer and message notification require new designs. Consensus built on efficient inter-kernel communication is key to high-performance secure agreement: kernels execute large workloads and produce abundant behavior and result information, so fast consensus is essential. The inter-kernel communication and consensus architecture is shown in Fig. 5.
Fig. 4 Inter-kernel communication and consensus architecture
Efficient inter-kernel communication explores secure and efficient mechanisms on a single host. Traditional inter-process or remote procedure calls often require high-privilege involvement to enforce security isolation and are implemented and managed by the kernel. In a multikernel architecture, kernels act as the communication principal and run in privileged mode, so new high-privilege communication methods are needed. Existing RPC or network-based mechanisms are not ideal for single-host communication; on-chip and chip-to-chip memory-based high-performance methods should be explored. Inter-kernel communication must balance sharing and isolation to leverage single-host efficiency while protecting kernel privacy.
Consensus mechanisms need to be efficient and accurate and require infrastructure for request distribution, proxy aggregation, and coordinated voting. Heterogeneous kernels exhibit divergent execution logic and may produce differentiated execution results, such as differing system-call types and counts or inter-process communication volumes. The system must implement heterogeneous monitoring components to collect kernel-specific runtime data and then vote across all kernel responses to reach consensus. Intrinsic security requires consensus algorithms tolerant of Byzantine-like faults; predefined adjudication algorithms and policies must determine whether observed discrepancies are random errors or deliberate attacks.
4 Related work
Industrial examples of intrinsic-secure OS architectures include SpaceX's triple-redundant system and industrial redundant autonomous driving platforms; academic work has focused on mobile-target defense and morphing defense. MTD increases attacker cost by providing random, diverse, and dynamic system views, but it does not change the logical nature of vulnerabilities or backdoors; attackers may still exploit short circuits or coordinated paths to bypass defenses. OS-level morphing defenses use dynamic heterogeneous redundancy to address both functional safety and cybersecurity. A recent 2023 proposal explored implementing dynamic heterogeneous redundancy on multikernel operating systems, providing a blueprint for OS-level intrinsic security.
5 Conclusion
This article presents an operating-system-level intrinsic security approach to address growing intrinsic security challenges. It identifies limitations of single-kernel OS architectures for intrinsic-security heterogeneity, the immaturity of dynamic heterogeneous redundancy at the OS level, and the difficulty of designing consensus mechanisms. A multikernel-based intrinsic security architecture is proposed, and design directions for heterogeneous redundancy, efficient inter-kernel communication, and consensus mechanisms are outlined.
As morphing techniques integrate more deeply with AI, IoT, cloud, data, and SDN technologies, an integrated "morphing+" ecosystem will emerge. AI models such as CNNs, RNNs, GNNs, and GRUs can optimize multimodal adjudication, situational awareness, and data processing, enabling AI to analyze OS security posture deficiencies. OS intrinsic security can be applied to IoT scenarios such as advanced driver assistance and vehicular network defenses, addressing risks from pervasive connectivity, data aggregation, heterogeneous platforms, complex devices, and reliability concerns. However, intrinsic security still faces challenges: improving empirical security effectiveness, reducing deployment cost, closing attack escape spaces under non-cooperative multimodal decision making, balancing heterogeneity gains with execution synchronization, and addressing limited morphing entropy in current intrinsic security components. Many research directions remain to be explored.
