Key insights
- The article highlights potentially unsustainable operating margins in the semiconductor sector, particularly for AI-related chips, with companies like Micron, SK Hynix, Nvidia, and TSMC seeing significant expansions. Risks include efficiency breakthroughs reducing compute demand, increased capacity from memory chip makers, and China's potential to catch up in GPU and memory chip production. A slowdown in capex by major AI model makers could also disproportionately impact margins. The sustainability of current margin levels is questioned, especially concerning the future demand for memory chips driven by AI advancements.
Micron operating profit margins have expanded to about 50%, and they are forecast to go to something like 70%. The average over recent history has been in the 20-30% range. SK Hynix is already at a 60% operating margin and is also forecast to go towards 70%. Nvidia, which used to have a high 20% operating profit margin, has been coasting at a high 60% operating margin for some time now. TSMC has always had pretty margins, in the high 30% range, but is now in the mid 50% operating margin and forecast to go towards 60% operating margin. These margins all seem quite high by historical standards. One big risk is a big efficiency breakthroughs in the models. This might improve capabilities without expanding compute as much, getting more bang for the buck. Another is that there does seem to be more capacity coming, at least from the big memory chip companies. TSMC is expanding capacity at a slower pace. A third is that China catches up in a meaningful way. There are dozens of GPU startups in China now and many memory chip makers as well. They don’t have to get as good as an SK Hynix at memory and certainly not as a good as Nvidia at GPUs to have an impact, as long as you can string enough of them together and and direct less intensive work towards it, it can have an impact on margins. Finally, any slow down in capex expansion from one of the big model makers might lead to a disproportionately large move in margins.
From my read there has been this steady march higher of compute demand and more capabilities unlocked at every step. In particular, it seems like it’s continued expansion of the context window that has enabled a lot of the big gains in AI functionality recently. The further expansions in the context window will mean more and more memory required. But at some point do we get diminishing returns of expanding the context window? And at that point, what happens to the incremental demand for memory chips?