ALLPCB
Overview
Over the past decade, deep learning has dominated multiple areas of artificial intelligence, including natural language processing, computer vision, and biomedical signal processing. While model accuracy has improved significantly, deploying these models on resource-constrained devices such as smartphones and microcontrollers remains challenging. This review provides a practical design guide for such devices, covering the design of lightweight models, compression techniques, and hardware acceleration strategies. The goal is to identify methods and concepts that overcome hardware limits without unduly sacrificing accuracy. The review also highlights two prominent future directions for lightweight deep learning: TinyML and methods for deploying large language models (LLMs) on resource-limited hardware.
Trends in Model Scale and the Need for Efficiency
The importance of neural networks has risen sharply, extending into many everyday applications and supporting increasingly complex tasks. Since AlexNet in 2012, there has been a persistent trend toward deeper, more complex networks to improve accuracy. For example, Model Soups achieved strong ImageNet accuracy but required over 1.843 billion parameters. Similarly, GPT-4 demonstrated strong NLP performance while scaling to roughly 1.76 trillion parameters. Deep learning compute requirements grew dramatically—about 300,000 times from 2012 to 2018—creating the practical challenges addressed in this review.
Scope of the Review
Recent research has focused on lightweight modeling, model compression, and acceleration techniques for resource-constrained deployment. Workshops and challenges such as Mobile AI at CVPR (2021–2023) and AIM at ICCV/ECCV have emphasized image and video processing on devices like ARM Mali GPUs and Raspberry Pi 4. Prior surveys often concentrate on a single aspect, such as quantization. This review covers the full workflow from architecture design through compression to hardware acceleration, aiming to clarify interactions and trade-offs across these areas.
Lightweight Architectures
The review analyzes classic lightweight architectures and groups them into families for clarity. Many architectures advance performance by introducing efficient convolutional blocks. For example, depthwise separable convolutions prioritize high accuracy while reducing computation. Importantly, parameter count and FLOPs do not always correlate with inference time: early lightweight models like SqueezeNet and MobileNet aimed to reduce parameters and FLOPs, but this often increased memory access cost (MAC), slowing inference. The review highlights these trade-offs to inform practical deployment choices.
Compression Techniques
Beyond architecture design, the review examines efficient algorithms applied to compress an existing model. Quantization reduces storage by replacing 32-bit floats with 8-bit, 16-bit, or lower-precision representations, even down to binary values. Pruning removes unnecessary parameters; simple methods remove individual weights, while structured approaches remove entire channels or filters. Knowledge distillation transfers knowledge from a large, pretrained "teacher" model to a smaller "student" model, though later methods can modify a single network iteratively to avoid requiring an explicit teacher. In practice, multiple compression techniques are often combined—for example, pruning together with quantization.
Hardware Acceleration
The review surveys popular hardware accelerators for deep learning applications, including GPUs, field-programmable gate arrays (FPGAs), and tensor processing units (TPUs). It describes different dataflow styles and explores methods for optimizing data locality, which underpin efficient deep learning workloads. The review also discusses popular deep learning libraries tailored to accelerate inference and training, and it examines co-design approaches that jointly consider hardware architecture and compression methods to achieve optimized results.
Challenges and Emerging Directions
Despite technical advances, deploying lightweight models in resource-constrained environments remains challenging. The review identifies open problems and emerging techniques for accelerating and applying deep learning in TinyML and LLM contexts.
TinyML
TinyML enables deep learning on ultra-low-power IoT devices with power budgets below 1 mW. Designing TinyML models is difficult due to extremely constrained hardware. Low-end IoT devices commonly rely on microcontrollers (MCUs) for cost reasons, but MCU-targeted libraries such as CMSIS-NN and TinyEngine tend to be platform-dependent and lack the cross-platform support offered by GPU-oriented frameworks like PyTorch and TensorFlow. As a result, TinyML work is frequently tailored to specific applications rather than promoting broad research reuse, which can slow overall progress.
MCU-based libraries
Because TinyML operates under harsh resource constraints, MCU libraries are often designed for specific use cases. CMSIS-NN is a pioneering MCU library for ARM Cortex-M devices, providing efficient kernels split into NN functions (convolutions, pooling, activations) and support functions (data transforms, activation tables). CMIX-NN offers an open mixed- and low-precision toolchain for quantizing weights and activations to 8-, 4-, and 2-bit representations. MCUNet introduced a co-design framework for commercial MCUs that combines TinyNAS for efficient architecture search and TinyEngine for code-generation-based compilation and in-place depthwise convolutions to tackle memory limits. MCUNetV2 added a patch-based inference mechanism that runs on small spatial regions of feature maps to reduce peak memory. MicroNet uses differentiable NAS to search for efficient models with low operation counts and supports TensorFlow Lite Micro (TFLM). MicroNet reported state-of-the-art results on TinyMLPerf tasks such as visual wake words, Google speech commands, and anomaly detection.
Bottlenecks to rapid TinyML progress
TinyML growth is limited by resource constraints, hardware and software heterogeneity, and a lack of suitable datasets. Tiny devices may have minimal RAM and less than 1 MB of flash, complicating model design and deployment. Heterogeneous hardware and limited framework compatibility mean many TinyML solutions are device-specific, hindering wide distribution. Existing datasets may not match the sensing characteristics of edge devices, creating a need for standardized datasets suitable for training TinyML models. Addressing these research challenges is necessary before large-scale IoT and edge deployment becomes feasible.
Lightweight Large Language Models
Building compact LLMs for resource-limited environments is an active research direction. LLMs have shown strong performance across tasks and can be useful in practice, but they often have billions or trillions of parameters and require GPU-class hardware and tens of gigabytes of memory for inference. Quantizing and compressing LLMs is challenging because embeddings and weight distributions vary. Transforming large, resource-intensive LLMs into compact models suitable for mobile devices is a key area for future work.
Progress in model compression without retraining
Recent work applies common DL pruning and quantization techniques to large LLMs. SparseGPT demonstrated that large pretrained generative Transformer models can be pruned to at least 50% sparsity in a single step without retraining and with minimal accuracy loss. Wanda (Pruning by Weights and Activations) introduces sparsity in pretrained LLMs by pruning weights with minimal magnitude, without requiring retraining or weight updates. Pruned LLMs can be used directly, improving practical utility. Wanda outperforms established magnitude-based baselines and competes effectively with methods requiring extensive weight updates.
Model design and parameter-efficient tuning
Lightweight LLMs can also be created from the outset by reducing parameter counts. A promising approach is prompt tuning, which optimizes LLM behavior while keeping the base model largely intact. Visual prompt tuning (VPT) is a notable example in vision tasks: it introduces a small set of learnable prompt parameters (often less than 1% of trainable parameters) in the input space while keeping the backbone fixed. CALIP proposes a parameter-free attention mechanism to facilitate efficient interaction between vision and text features, producing text-aware image features and visually guided text features. Developing adaptive fine-tuning strategies that dynamically adjust model architecture and parameters to task requirements is a promising avenue to avoid unnecessary parameter growth while optimizing task-specific performance.
Diffusion Models and Visual Transformers (ViTs)
Diffusion models. Denoising diffusion models and score-based models have advanced generative quality, but moving inference to edge devices is difficult. The inference process reverses a stochastic diffusion process from Gaussian noise to real data and is computationally expensive. Compressing diffusion models risks degrading image quality because simplifications or approximations can impair accurate reconstruction. Achieving compact diffusion models that still generate high-quality images remains an open challenge for resource-limited scenarios.
Deploying ViTs. Lightweight vision transformers are emerging, but deploying ViTs on constrained hardware remains challenging. Reported latency and energy for ViT inference on mobile devices can be up to 40 times higher than for comparable CNNs. Self-attention in ViTs computes pairwise relations between image patches, and compute grows quadratically with the number of patches. The feed-forward network (FFN) layers also often dominate computation time. Structural reductions such as removing redundant attention heads and FFN layers can reduce latency; for example, DeiT-Tiny reduced latency by 23.2% with only a 0.75% accuracy trade-off.
Hardware-software co-design for ViTs
Several works propose co-design solutions for embedded systems such as FPGAs. DiVIT and VAQF present hardware-software co-designs for ViTs. DiVIT exploits patch locality with differential attention and an incremental patch encoding that supports an array of differential-attention processing engines with differential dataflow communication. Exponential operations are implemented via lookup tables to minimize computation and hardware cost. VAQF applies binarization to ViTs for FPGA mapping and quantization-aware training; it can generate quantization precisions and accelerator descriptions targeting specific frame-rate constraints for direct software and hardware implementations.
Two future directions for ViT deployment
1) Algorithmic optimizations. Beyond designing efficient ViT models, bottlenecks such as matrix multiplication should be targeted for acceleration or reduction. Improvements in integer quantization and operator fusion can also help.
2) Hardware accessibility. Unlike CNNs, most mobile devices and AI accelerators provide hardware primitives optimized for CNNs rather than ViTs. For example, ViTs may not run directly on some mobile GPUs or on certain VPUs because key operators are unsupported or require different tensor dimensionality. LayerNorm is common in transformers but not in many CNN accelerators. Investigating hardware support for ViT operators on resource-constrained devices is necessary for broader adoption.
Contributions and Conclusions
- The review organizes lightweight architectures into families (for example, grouping MobileNetV1–V3 and MobileNeXt into a MobileNet family) and provides a historical perspective on lightweight model development.
- To cover the full lifecycle of lightweight deep learning, the review integrates architecture design, compression methods, and hardware acceleration, clarifying the interactions among these areas.
- As part of surveying frontiers in lightweight DL, the review examines current challenges and future directions, including TinyML for extreme resource-constrained devices and efforts to leverage LLMs at the edge.
Recent emphasis in computer vision on energy efficiency, lower carbon footprint, and cost effectiveness highlights the importance of lightweight models for edge AI. This review provides a comprehensive look at lightweight deep learning, covering important models such as MobileNet variants and efficient transformer variants, as well as popular strategies for optimization, including pruning, quantization, knowledge distillation, and neural architecture search. Practical guidance is offered for tailoring lightweight models, with analysis of strengths and weaknesses. The review also examines hardware acceleration in detail, discussing architectures, dataflow styles, and data locality optimization techniques to aid understanding of accelerated training and inference. The interplay between hardware and software (co-design) is emphasized as critical for future progress. Finally, the review identifies open research areas in TinyML and LLM deployment that require creative solutions to advance lightweight deep learning in constrained environments.
