Scaling The Storage Pyramid With Micron’s Datacenter Memory And Flash

SPONSORED CONTENT  Consider, for a moment, the current state of AI accelerators and datacenter GPUs. Now, try to imagine this landscape without Micron Technology’s entry into the High Bandwidth Memory (HBM) market. Micron’s robust leadership and groundbreaking innovation in meeting the memory bandwidth demands of AI workloads have ushered in a new era of AI capabilities. Their vision and commitment have propelled the industry forward, setting new standards and cementing their essential role in the tech ecosystem.

So, if you’re investing in AI accelerators for high-performance computing and datacenters – then don’t forget to send Micron a thank you note because their advanced memory solutions, like LPDDR5X and HBM3E, play a critical role in enabling these cutting-edge systems.

Micron, which started out 46 years ago as a semiconductor design company and, thanks in part to funding from potato magnate JR Simplot, who wanted to get into computer chips as well as potato chips and McDonald’s French fries, the company shifted to also making computer memory. And now, nearly five decades later, Micron spans the gamut from high-capacity drives used to build data lakes, to fast drives for local storage, and various kinds of dynamic random access memory (DRAM) – DDR, LPDDR, GDDR, and HBM – that link directly to compute engines of all kinds and literally give them the memory space to calculate.

To get a feel for what is going on with memory up and down the hierarchy as it relates to HPC and AI systems, we sat down with Praveen Vaidyanathan, vice president and general manager of Data Center Business at Micron, to talk about all things memory and to get some sense as to the growing demand for denser HBM memory chips and the opportunities enabled by their wider adoption..

Timothy Prickett Morgan: Let’s start out by talking about the many kinds of DRAM that Micron makes and how they are best suited for different devices and applications in the datacenter.

Praveen Vaidyanathan: Sure! It’s helpful to think about the trends that are driving requirements for memory, and why we focus on certain things. I put considerations into four categories or a vector matrix – performance, capacity, power, volume.

First is the trend for performance requirements across various applications – whether you are talking about general purpose compute applications, AI training and inference applications, or a networking card. There are certain performance requirements that are driving trends within each of these unique applications.

The second category is a capacity requirement. How much memory do you need to support your datasets?

The third category is very important – and probably going to be even more important with each year – and that’s power. How much power you budget and consume will define the next generation of memory that we are going to build.

TPM: Isn’t price the fourth factor?

Praveen Vaidyanathan: Price is a different conversation. Yes, we have to look at acquisition costs and do TCO calculations. But the more important fourth category which drives a memory technology choice is the volumetric space that memory occupies in X and Y dimensions and Z height. How much room do you have to pack that performance, to pack that power, and to pack that capacity? These four variables answer the question of when to use DDR, GDDR, LPDDR and HBM.

For instance, if your primary focus is on memory bandwidth, HBM is a great choice for an AI accelerator due to its high data transfer rates. However, if you also need to consider high memory capacity, DDR5 for a CPU attaches memory could be a good solution. We see LPDDR being a really good option in today’s datacenters to drive lower power consumption. So that’s the overall thought process: if you look at these four vectors for memory, and you add acquisition cost and TCO, the choices will evolve very naturally, and you will see why certain memory fits into one use case and not in others.

In the old days, it used to be simple: a customer would need a 16 Gb piece of memory, and they wanted a lot of it. The challenge we have now is how do we define a memory roadmap to address these four-categories or vector matrix – performance, capacity, power, volume – that customers can choose from for the applications they use and need. To us, that’s what is so exciting– you have to really understand the needs of your customers and their environment.

TPM: With AI workloads evolving rapidly and demanding greater memory performance, how do you see HBM playing a critical role in the future of computing, and how is your company driving innovation in this space?

Praveen Vaidyanathan: Great question. As AI and data-intensive applications continue to push the limits of computing, it’s clear that traditional memory architectures can no longer keep up with the sheer volume of data being processed. HBM is essential for meeting these demands, offering the high-speed data access and efficiency required for next-generation workloads.

We’re seeing a fundamental shift in computing where many AI-driven workloads are not just compute-bound but also memory-bound. Generative AI, for example, requires massive memory bandwidth, and now, with the rise of agentic AI – systems that can autonomously reason, adapt, and make decisions in real-time – the demand for high-performance memory is accelerating even further. These workloads require seamless data access to operate efficiently, making HBM an indispensable part of AI infrastructure.

Our company is at the forefront of HBM innovation, continuously advancing our technology to provide higher capacities, increased bandwidth, and optimized power efficiency. We are committed to enabling AI, cloud computing, and high-performance computing applications with cutting-edge memory solutions that unlock new possibilities. As the industry moves toward more memory-centric architectures, HBM is becoming the foundation for the future of computing.

TPM: Is HBM a better business than normal DRAM? I assume HBM throws off more gross margins than does DDR and GDDR.

Praveen Vaidyanathan: We have publicly said that HBM is accretive to the overall margin of our business.

TPM: Do you partner directly with HPC centers as a memory supplier? Or do you pitch to HPC centers through Hewlett Packard Enterprise, Atos, Nvidia, Supermicro, Lenovo, Dell, or whoever is building the machines in an HPC cluster? Because it would be interesting to see an HPC center come directly to you to get insight into how to solve their memory capacity and bandwidth problems.

Praveen Vaidyanathan: We do both. For instance, last year during SC24, we met with CERN to discuss a significant compute problem where they wanted to combine all this memory. And they are working with us directly to install some of our technology and see how it helps them solve some of these challenges.

The HPC community is very big, so it’s hard for us to interact with all of them. We do indirect reach through our OEMs, but in some areas, we also do a direct reach.

TPM: Do the HPC centers get early access to funky memory stuff? Is that a possibility? That happened for a long time in the past with compute engines, and it is still happening to a certain degree.

Praveen Vaidyanathan: Not as much funky stuff, as you put it, because these organizations have such big compute environments that they require something that is stable. If a technology is too early and not stable, it is actually a lot more disruptive to what they are working on. But we do drive ideas for innovation and deliver products based on the needs of HPC applications.

TPM: Can you, and do you, do a portfolio pitch to HPC centers? You have various kinds of flash and memory. Is there a benefit to having Micron memory top to bottom and front to back?

Praveen Vaidyanathan: Absolutely, and that is part of what we bring into the supercomputer space. It all starts with the memory and storage hierarchy

We start from near memory, which is HBM, and go all the way out to data lakes. The memory and storage hierarchy is bi-directional: it goes up to higher and higher in performance as you go towards near memory, and it moves to higher and higher capacity as you move down to data lakes. That’s the spectrum of the portfolio of products that we bring to the HPC community.

TPM: What about HBM4? What is going on with that product?

Praveen Vaidyanathan: I’ll recap the history and then speak to HBM4. We had an HBM2E product we revealed four years ago, and at that time, we were also thinking about where to go next. From there we made the decision to focus on delivering a best-in-class HBM3E product.

We were first to market with HBM3E, and we surprised people because we came out saying this runs at 1.2 TB/sec when the expectation for bandwidth was more in the 1 TB/sec range. We had 20 percent better performance coming out of the chute. Moreover, for a given performance, we had 30 percent lower power than any other HBM3E product out there – at a minimum. In addition, our 24 GB stack, with each layer being 24 gigabits, offered higher capacity than the industry standard of 16 GB per stack. In June 2023, we announced that product and by February 2024 we were in production with Nvidia with the H200.

Last fall, we were ready with our twelve-high, 36 GB HBM3E stack, which has the same performance specs and the same power benefits. In fact, as we crunched the numbers, we realized that our 36 GB HBM3E still uses 20 percent less power than the eight-high 24 GB stacks from other vendors.

TPM: With HBM4, you’re not going to skip right to HBM4E, are you?

Praveen Vaidyanathan: No, no. We have a strong position, and with HBM4 we want to maintain that position. We are excited about Micron’s HBM leadership roadmap for the rest of this decade. Leveraging our strong foundation and continued investments in 1-beta process technology, we expect HBM4 to maintain time-to-market and power efficiency leadership while increasing bandwidth by over 60 percednt compared to HBM3E. We anticipate HBM4 to ramp up in high volume for the industry in calendar year 2026.

TPM: Can you maintain that performance and power gap with the competition with HBM4 and HBM4E?

Praveen Vaidyanathan: The competition will be fighting pretty hard, but we will continue to drive power differentiation, and we think over time, it’s going to remain a differentiator for us not just in general purpose compute and accelerators, but for cloud service providers building custom silicon. They can take greater advantage of this gap by optimizing for specific workloads, leading to OPEX savings or increased compute power and speed.

Last fall, we announced that we have HBM4 mechanical samples ready, which is an important milestone. We deliver the power and capacity performance improvements not just by silicon design, but with packaging innovations. This is where mechanical test vehicles are helpful – it’s basically an HBM cube that is electrically the same as a real HBM stack, but not yet the full design. Mechanically, it looks identical to the real HBM4 stack. Customers can take those mechanical test vehicles and build their systems and run the packaging and manufacturing testing now, so that they are prepared for the real parts that come later – all of which results in reduced time to market.

Development work is well underway with multiple customers on HBM4E, which will follow HBM4. HBM4E will introduce a paradigm shift in the memory business by incorporating an option to customize the logic base die for certain customers using an advanced logic foundry manufacturing process from TSMC. We expect this customization capability to drive improved financial performance for Micron.

Sponsored by Micron