Introduction
Despite the dominance of powerful cloud-based systems, the reliance on modern neural network architectures presents notable challenges, including high costs, energy consumption, and scalability issues. Artificial Intelligence, particularly Deep Neural Networks, has advanced to the point where large-scale cloud computing has become essential for further breakthroughs. Yet, we believe the future of AI may shift to smaller, more efficient models on edge devices. This direction reflects the increasing availability of tiny, powerful microcomputers and the need for localized AI processing. Achieving this shift depends on advancements in compact model architectures.
Microcontrollers and EdgeAI
Microcontrollers are everywhere, from industrial systems to IoT devices. They collect vast amounts of data, yet their computational power has historically been limited. For many AI tasks, the data captured by these devices cannot be processed locally, requiring a reliable connection to send data to the cloud for further analysis. The emergence of platforms like NVIDIA Jetson boards, Raspberry Pi, and Google TPUs proves the need for a bridge between traditional microcontrollers and cloud-based systems, enabling AI deployment directly onsite. These “microcomputers” – regarding their high computing resources compared to microcontrollers – which we refer to as edge devices, allow faster, safer, and localized decision-making. Initial models for computer vision tasks like classification or object detection were not very accurate when scaled down for edge devices. However, newer low-parameter architectures are closing the gap between cloud-based solutions like Visual Transformers and mobile-optimized designs like MobileViTv3 [18].
Case Study: Estimating Vineyard Yields
Located in the heart of vineyards in Reims, our lab collaborates with Vranken Pommery to improve grape yield estimation using AI. In the industrial application of yield estimation, we try to use Computer Vision by directly counting grapevines with edge devices directly mounted on tractors that pass through the vineyards. For this application, an object detector – a deep learning algorithm that directly “sees” and surrounds where grapes are located – identifies and counts grapes, even in challenging environments where dense foliage obscures visibility. We considered increasing frame rates to enhance the accuracy as it theoretically provides better yield estimates by giving the model “more chance” to see occluded grapes caused by leaves during several frames, thanks to the camera angle. However, this solution requires significant bandwidth, which is not practical across expansive vineyards with limited connectivity. Instead, we deployed AI models locally using Jetson TX2 devices. The solution records grape counts and coordinates in simple text files for later communication. This approach minimized memory and bandwidth demands, proving edge AI’s potential in real-world applications.
Combining Academic and Industrial Research
As part of the EdgeAI project, our research lab conducts academic studies [14, 5], industrial research [4, 17], and projects that bridge the two. In the context of grape cluster detection, we developed a new academic contribution to the MSCOCO [10] dataset: LeYOLO (Lightweight and Efficient YOLO). This model reduces parameter count and theoretical computation costs (FLOP) by incorporating key elements from efficient neural network architectures like MobileNet.
LeYOLO [5] performs on par with YOLOv9 [19], achieving 42% fewer FLOPs. It is also 9% more accurate than YOLOv8 Nano [8] on MSCOCO, with 25% fewer parameters. For grape cluster detection on the WGISD [16] dataset, LeYOLO performs 1.2% worse than YOLOv8 but still uses 25% fewer parameters and achieves a 72% reduction in theoretical FLOPs.
Doing More with Fewer Parameters
Reducing parameters in neural networks offers an academic and practical challenge. Two perspectives help explain this:
- Large-scale datasets: Models like MobileNetV1 [7], introduced in 2017, proved that low-parameter designs could perform competitively on datasets like ImageNet [9]. MobileViTv3 further demonstrated this by achieving 80% accuracy with 5.1M parameters, compared to the 77% accuracy of ViT-H [1], which uses 80M parameters.
- Small networks and toy datasets: Studies [2, 3] reveal that neural networks can create efficient feature representations using far fewer activations than initially thought. This efficiency is tied to superposition, where a single neuron represents multiple features, compressing information without significant loss.
Anthropic AI’s work on superposition highlights this phenomenon. Their experiments demonstrated that even highly compressed neural networks could effectively generalize and memorize data.
The Era of Efficient AI: From MobileNets to MobileViTs
Compact AI architectures [7, 15, 6, 12, 13, 18], from MobileNetV1 to MobileViTv3, show how innovation can optimize performance without bloated designs. Since the introduction of MobileNetV1 in 2017, researchers have demonstrated that compact models can achieve remarkable – if not better – accuracy on large-scale datasets like ImageNet. MobileNetV1 leverages depthwise separable convolutions to drastically reduce the parameter count while maintaining very competitive performance at that time. In recent years, this trend has accelerated, especially between 2020 and 2023, with the emergence of very influential architecture, like MobileViT [12], which combines transformer architecture with MobileNet principals, balancing computational efficiency and accuracy. MobileViT, a vision transformer model designed for mobile devices, achieves 80% Top-1 Accuracy on ImageNet with just 5.1 million parameters. In contrast, traditional vision transformers like ViT require over 80 million parameters to barely reach 77% accuracy on the same dataset. These principles have influenced even largescale models like ConvNeXt [11], which integrates parameter-reduction techniques while maintaining high performance. The benefits of compact AI extend beyond edge devices, offering solutions across AI research and deployment.
Conclusion: Embrace the Journey of Tiny AI Revolution
The future of AI lies in efficiency – achieving more with less. This principle is essential for edge applications, advancing compact architectures, and building sustainable AI systems. Achieving more efficient AI architecture is not just a necessity for constrained environments, it is a path toward sustainable and scalable AI innovations. By prioritizing parameter efficiency, we can create accessible, high-performance AI solutions. The journey from MobileNet to MobileViT and beyond demonstrates that resource constraints no longer demand compromises in performance. Efficient AI is at the corner of the future of intelligence, enabling researchers to bring the power of AI to every corner of the world through microcontrollers, computers, and beyond.
Blog signed by: URCA team
References
[1] Alexey Dosovitskiy. “An image is worth 16×16 words: Transformers for image recognition at scale”. In: arXiv preprint arXiv:2010.11929 (2020).
[2] Nelson Elhage et al. “Toy models of superposition”. In: arXiv preprint arXiv:2209.10652 (2022).
[3] Tom Henighan et al. “Superposition, memorization, and double descent”. In: Transformer Circuits Thread 6 (2023), p. 24.
[4] Lilian Hollard and Lucas Mohimont. “Applying Knowledge Distillation on Pre-Trained Model for Early Grapevine Detection”. In: Workshop Proceedings of the 19th International Conference on Intelligent Environments (IE2023). IOS Press. 2023, pp. 149–156.
[5] Lilian Hollard et al. “LeYOLO, New Scalable and Efficient CNN Architecture for Object Detection”. In: arXiv preprint arXiv:2406.14239 (2024).
[6] Andrew Howard et al. “Searching for mobilenetv3”. In: Proceedings of the IEEE/CVF international conference on computer vision. 2019, pp. 1314– 1324.
[7] Andrew G Howard et al. “Mobilenets: Efficient convolutional neural networks for mobile vision applications”. In: arXiv preprint arXiv:1704.04861 (2017).
[8] Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics YOLO. Version 8.0.0. Jan. 2023. url: https://github.com/ultralytics/ultralytics.
[9] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. “ImageNet Classification with Deep Convolutional Neural Networks”. In: Advances in Neural Information Processing Systems. Vol. 25. Curran Associates, Inc.,
2012. url: https://proceedings.neurips.cc/paper_files/paper/ 2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html
[10] Tsung-Yi Lin et al. “Microsoft coco: Common objects in context”. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer. 2014, pp. 740– 755.
[11] Zhuang Liu et al. “A convnet for the 2020s”. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022, pp. 11976– 11986.
[12] Sachin Mehta and Mohammad Rastegari. “MobileViT: Light-weight, Generalpurpose, and Mobile-friendly Vision Transformer”. In: International Conference on Learning Representations. 2022. url: https://openreview. net/forum?id=vh-0sUt8HlG.
[13] Sachin Mehta and Mohammad Rastegari. Separable Self-attention for Mobile Vision Transformers. June 2022. doi: 10.48550/arXiv.2206.02680.
[14] Lucas Mohimont, Lilian Hollard, and Luiz Angelo Steffenel. “Improving Edge-AI Image Classification Through the Use of Better Building Blocks.” In: CLOSER. 2024, pp. 303–310.
[15] Mark Sandler et al. “Mobilenetv2: Inverted residuals and linear bottlenecks”. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2018, pp. 4510–4520.
[16] Thiago Santos et al. Embrapa Wine Grape Instance Segmentation Dataset – Embrapa WGISD. Version 1.0.0. Zenodo, Aug. 2019. doi: 10.5281/ zenodo.3361736. url: https://doi.org/10.5281/zenodo.3361736.
[17] Luiz Angelo Steffene, Axel Langlet, and Lilian Hollard. “AI-Driven Strategies to Implement a Grapevine Downy Mildew Warning System”. In: Industrial Artificial Intelligence Technologies and Applications (2023), p. 177.
[18] Shakti N Wadekar and Abhishek Chaurasia. “Mobilevitv3: Mobile-friendly vision transformer with simple and effective fusion of local, global and input features”. In: arXiv preprint arXiv:2209.15159 (Oct. 2022). doi:
10.48550/arXiv.2209.15159.
[19] Chien-Yao Wang, I-Hau Yeh, and Hong-Yuan Mark Liao. “YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information”. In: arXiv preprint arXiv:2402.13616 (2024). doi: 10.48550/arXiv. 2402.13616.