As AI technology advances, addressing the memory wall is essential to unlock its full potential.
by Sangeetha Chandran Sarala, Principal Emulation Engineer, ASIC at Micron Technology
The rapid progress of Generative AI is transforming how we approach complex engineering problems and accelerate scientific discovery, driving greater efficiency and sustainability. This revolution depends on massive data analysis and processing, which in turn demands high-performance, scalable, and power-efficient hardware. The semiconductor industry is advancing in unison, developing ecosystems to handle substantial AI workloads. A critical challenge in this journey is overcoming the “memory wall” — a significant bottleneck in AI hardware performance.
Generative AI for data centers is highly compute-intensive and requires systems built with High-Performance Computing (HPC) components, including processors for calculations, memory and storage for data handling, and a networking fabric for data communication. Components are selected based on AI use cases to strike a balance between performance and cost.
At the heart of these systems are processors designed for resource efficiency and multitasking, which are essential for handling inference and training datasets. GPUs and TPUs excel in high-performance tasks due to their inherent parallelism and scalability. However, power efficiency remains a key concern. High-speed interconnects are equally vital for enhancing system efficiency by enabling fast and seamless data movement within and across servers. New networking infrastructure is evolving to streamline communication between processors, memory, and peripheral devices, ensuring scalable, high-speed operation for accelerated computing.
Memory solutions
To match the performance of top-tier processors, memory solutions must deliver high bandwidth and low latency while consuming minimal energy. The key challenge to enhancing overall system efficiency is addressing the “memory wall” issue. As CPU speeds far outpace memory speeds, processors are left waiting for data, creating a bottleneck. Today, this disparity is nearly 50%, significantly hindering system performance.
One way to mitigate this is by placing memory closer to the processor. Modern SoCs integrate on-chip SRAM near the processors to enable faster access—an approach known as near-memory computing. While effective, this strategy is expensive and limited by the small memory capacity that can be integrated on-chip.
Near-memory computing also involves placing compute logic closer to memory, utilizing advanced technologies such as High-Bandwidth Memory (HBM). Common in data centers and HPC, HBM stacks 3D DRAM dies alongside the processor using a silicon interposer. This setup delivers high bandwidth and capacity with excellent power efficiency. The latest iteration, HBM3, enhances memory density and performance per watt, albeit at the expense of increased design complexity and cost.
For applications that require high bandwidth but face cost constraints, Graphics Double Data Rate (GDDR) memory offers an alternative, particularly for AI inference. Initially developed for graphics processing, GDDR6/7 now serve near-memory computing with relatively high bandwidth and low latency. While more affordable than HBM, GDDR consumes more power at high data rates. The table accompanying this column highlights key differences between HBM and GDDR technologies.
Looking forward
To scale AI hardware effectively, chiplets are becoming the new standard in SoC design. This modular approach breaks a system into smaller, specialized components, which are then integrated into a single package. It allows designers to scale functionality with minimal redesign and greater flexibility.
High-speed communication between memory and logic chiplets is enabled by emerging interconnect standards, such as Universal Chiplet Interconnect Express (UCIe). UCIe standardizes chiplet connectivity, simplifying the integration of components from different vendors into a cohesive package.
Another promising development is Compute Express Link (CXL), a cache-coherent interconnect that maintains memory consistency between host and device. CXL enables memory pooling, allowing dynamic allocation across GPUs, SoCs, and other components. This optimizes memory use and boosts overall system performance—an essential capability for memory-intensive AI workloads. As adoption grows, CXL is expected to have a significant impact on data center architectures.
A particularly innovative area is in-memory computing, where logic operations are performed directly within memory arrays. This reduces data movement and boosts performance by minimizing latency. Non-volatile memory, such as flash, is being researched for this purpose, as it supports both data storage and logic functions. Although the concept is not new, renewed interest in this approach reflects its potential for addressing the memory wall, provided that energy and architectural challenges can be resolved.
Finally, photonic interconnects are being explored to reduce latency and power consumption further. By using light instead of electricity for data transfer, photonics could redefine communication between processors and memory in future data centers.
Conclusion
As AI technology advances, addressing the memory wall is essential for unlocking its full potential. By leveraging high-performance memory technologies such as HBM and GDDR, exploring modular chiplet designs, adopting interconnect standards like UCIe and CXL, and developing forward-looking solutions like in-memory computing and photonics, the semiconductor industry is laying the groundwork for robust, efficient, and scalable AI systems. These innovations will help close the performance gap and drive the next generation of AI to new heights.