Efficient NPU–GPU scheduling for real-time deep learning inference on mobile devices | Journal of Real-Time Image Processing
As the need for on-device artificial intelligence (AI) has increased in recent years, mobile devices tend to be equipped with multiple heterogeneous processors, including CPU, GPU, and Neural Processing Unit (NPU). While NPUs can offer low-cost and real-time AI processing capabilities for Deep Neural Network (DNN) inference, its limited resources often lead to a trade-off between performance and accuracy, potentially resulting in a non-trivial accuracy drop. To address this problem, we propose a new NPU–GPU Scheduling (NGS) framework for DNN-based video analytics. The main challenge lies in determining when and how to execute inference on the NPU/GPU to satisfy the performance objectives. To make more precise scheduling decisions, we first propose a new image complexity assessment model to replace the existing normalized edge density metric. Then, we formulate the scheduling problem with the objective of maximizing inference accuracy under the given latency constraint, and introduce an adaptive solution based on dynamic programming to determine which frames should be processed on the GPU and when to exit from inference for each of them. Extensive experiments conducted on a real mobile device show that our NGS framework substantially outperforms other solutions, and achieves a close-to-oracle performance.
