The modern computing paradigm, characterized by a strict separation of computation and memory, has been a cornerstone of technological advancement [1]. However, this design is also incredibly wasteful, particularly in terms of energy efficiency. For instance, a DRAM read operation requires approximately 640 pJ to retrieve 32 bits, whereas the energy cost of a 32-bit addition is just 0.1 pJ [2]. This discrepancy highlights the inefficiency of constantly shuttling data between memory and processing units. In stark contrast, biological systems, like the human brain, integrate memory and computation seamlessly, enabling remarkable efficiency. This has inspired researchers to explore in-memory computing (IMC) as a means to revolutionize artificial intelligence (AI) by drastically reducing its energy footprint.
Fig.1 – Energy consumption related to different operation in a classical Von Neuman architecture. Extracted from [2].
In-Memory Computing for Artificial Neural Networks
The fundamental operations of neural networks—multiplication, accumulation, and non-linear activation—can be effectively implemented using in-memory approach. This has led to extensive research into resistive memories, or memristors, as analog synapses [3-4]. By leveraging Kirchhoff’s laws, these memories can perform computations directly within the memory array, eliminating the need for data movement and achieving remarkable energy efficiency [3-4].
Fig. 2 – Simple implementation using Kirchhoff’s law, a voltage is applied to the top electrode of a resistive device and the current that flow through it is proportional to the voltage V multiplied by the conductance G
Emerging Memories: The Core of a New Paradigm
Resistive memories, often referred to as memristors, are fast, non-volatile, and capable of being embedded at the core of CMOS logic. These devices store information as electrical resistance, switching between high-resistance (HRS) and low-resistance (LRS) states. Several flavors of device physics exists, including oxygen vacancy resistive memory, phase-change memory or magnetoresistive memory.
Fig. 3 – Schematic structure of various resistive memories: Phase Change Memories, Oxygen Vacancy Resistive Memories and Magnetic Random Access Memories based on Magnetic Tunnel Junctions.
Major microelectronic industry players such as Samsung, TSMC, Intel, and STMicroelectronics are already producing these memories on a test scale. Their ability to merge logic and memory opens up unprecedented opportunities for efficient AI hardware.
Overcoming Memristive Device Imperfections
Despite their promise, resistive memories are not without challenges [5]. Among other, variability in their electrical properties poses significant challenges.
Fig. 4- Three different programming conditions give rise to three different distributions.
(a) is a very strong programming condition with very low bit error rate but the cyclability is very low.
(b) is an intermediate programing condition with low bit error rate, low energy consumption and relatively high cyclability and (c) is a weak programming condition with high bit error rate but large cyclability. Extracted from [6].
In order to address memristive imperfections, CEA has been working on two different neural network flavors:
- Binarized Neural Networks (BiNNs): By pushing quantization to its extreme, BNNs use binary values for weights and activations, significantly simplifying computations.
- Bayesian Neural Networks (BayNNs): These models embrace the statistical nature of memristors, using them for Markov Chain Monte Carlo (MCMC) sampling to estimate probability distributions.
- Binarized Neural Networks: Efficiency Through Simplicity
BinNNs simplify neural network operations to the bare minimum: multiplication becomes XNOR, accumulation becomes bit counting, and non-linear activation becomes a simple sign function. Remarkably, these simplified operations are well-suited for in-memory or near-memory computing architectures [6-8].
However, programming conditions for resistive memories can strongly influence their characteristics. Strong programming conditions yield low bit-error rates (BER) but consume high energy and reduce cyclability. Conversely, weaker programming conditions improve energy efficiency and cyclability but increase BER. In this context, error correction codes (ECC) have been developed to mitigate those bit errors, but are not well-suited for IMC due to their area overhead. Instead, alternative techniques, such as differential approaches (e.g., 2T2R structures), are more compatible with IMC architectures. Notably, this flexibility enables BNNs to operate effectively even under conditions with high error rates.
Altogether, BinNNs have been demonstrated to be inherently resilient to errors, making them ideal candidates for leveraging emerging non-volatile memory devices with higher error rates.
Fig. 5 – Accuracy over the Bit Error Rate on weights. Extracted from [7].
Moreover, BNNs also offer advantages in fully digital architectures. Integer circuits of just a few bits can perform summations, eliminating the need for analog-to-digital conversions. This digital simplicity ensures compatibility with various neural network architectures while maintaining low energy consumption and a compact area footprint.
- Bayesian Neural Networks: Embracing Uncertainty
Bayesian neural networks take a different approach by interpreting network parameters and activations as probability distributions [9]. This approach introduces a measure of uncertainty into the model, which is particularly valuable in scenarios with limited data or noisy inputs. Key benefits of uncertainty estimation include:
- Improved robustness against overfitting and noisy data.
- Propagation of uncertainty to outputs, enabling more reliable decision-making
Fig. 6 – (a) Principle of a Bayesian Neural Network and Uncertainty measure. (b) Hardware implementation using multiple devices samples to program a probability distribution. Extracted from [9].
Bayesian neural networks are particularly well-suited to applications where data is scarce, making them a critical tool in domains that demand both precision and adaptability.
Exploiting Memristive Variability for Bayesian Models
The statistical nature of resistive memories is not merely a challenge but an opportunity. By harnessing device variability, these memories can be used for MCMC sampling, an essential component of Bayesian learning. This compatibility enables efficient, resistive memory-based learning systems that align with the probabilistic framework of Bayesian models.
Conclusion
Binarized neural networks and Bayesian neural networks represent two complementary paths for leveraging resistive memories in in-memory computing. While BNNs excel in energy-efficient inference by tolerating bit errors, Bayesian models exploit device variability for probabilistic learning. Together, these approaches illustrate the transformative potential of integrating emerging memories into AI hardware. By embracing imperfections and variability, researchers are unlocking new possibilities for energy-efficient, high-performance neural networks that push the boundaries of what is achievable in modern AI.
Blog signed by: CEA team
References
[1] A. Pedram et al., “Dark memory and accelerator-rich system optimization in the dark silicon era”, IEEE Design Test 34, 39-50 (2017)
[2] V. Sze, “Efficient Processing of Deep Neural Networks: from Algorithms to Hardware Architectures”, presented at NEURIPS, Vancouver, Canada, 2019.
[3] C. -X. Xue et al., “24.1 A 1Mb Multibit ReRAM Computing-In-Memory Macro with 14.6ns Parallel MAC Computing Time for CNN Based AI Edge Processors,” 2019 IEEE International Solid-State Circuits Conference – (ISSCC), San Francisco, CA, USA, 2019, pp. 388-390, doi: 10.1109/ISSCC.2019.8662395.
[4] D. Strukov et al., “Building brain-inspired computing”, Nat. Commun. 10, 4838 (2019)
[5] D. Ielmini et al., “Emerging Neuromorphic Devices”, Nanotech. 21;31(9):092001 (2020)
[6] M. Bocquet et al., “In-Memory and Error-Immune Differential RRAM Implementation of Binarized Deep Neural Networks”, IEDM2018 Tech. Dig., 20.6.1.
[7] T. Hirtzlin et al., “Digital Biologically Plausible Implementation of Binarized Neural Networks With Differential Halfnium Oxide Resistive Memory Arrays”, Front. Neurosci. 13:1383.
[8] M. Ezzadeen et al., “Low-Overhead Implementation of Binarized Neural Networks Employing Robust 2T2R Resistive RAM Bridges,” ESSCIRC 2021 – IEEE 47th European Solid State Circuits Conference (ESSCIRC), Grenoble, France, 2021, pp. 83-86, doi: 10.1109/ESSCIRC53450.2021.9567742.
[9] T. Dalgaty et al., “In situ learning using intrinsic memristor variability via Markov chain Monte Carlo sampling”. Nat Electron 4, 151–161 (2021). https://doi.org/10.1038/s41928-020-00523-3