Google splits AI training and inference into separate TPU chips in latest Nvidia challenge

Google is separating AI training and inference into distinct processors in the eighth generation of its tensor processing unit, or TPU, as it tries to strengthen its position against Nvidia in AI hardware. Both chips are expected to become available later this year.

"With the rise of AI agents, we determined the community would benefit from chips individually specialized to the needs of training and serving," Amin Vahdat, a Google senior vice president and chief technologist for AI and infrastructure, said in a blog post.

The move comes as competition in custom AI semiconductors intensifies. In March, Nvidia highlighted forthcoming silicon designed to help models respond quickly to users' questions, using technology obtained in its $20 billion deal with chip startup Groq. Google remains a major Nvidia customer, but it also offers TPUs as an alternative for companies using its cloud services.

Major technology companies are increasingly developing custom chips for artificial intelligence to improve efficiency and support specialized use cases. Apple has long included neural engine AI components in its in-house iPhone chips. Microsoft announced a second-generation AI chip in January. Last week, Meta said it is working with Broadcom to develop multiple versions of AI processors.

Google was an early entrant in the field. The company began using internally designed processors for AI models in 2015 and started renting them to cloud customers in 2018. Amazon Web Services announced the Inferentia chip for handling AI requests in 2018 and introduced the Trainium processor for training AI models in 2020.

DA Davidson analysts estimated in September that the TPU business, together with the Google DeepMind AI group, would be worth about $900 billion.

Even so, large technology companies have not displaced Nvidia. Google is not directly comparing the performance of its new chips with Nvidia's products. The company said the new training chip delivers 2.8 times the performance of the seventh-generation Ironwood TPU, announced in November, at the same price. It also said performance is 80% better for the new inference processor.

Nvidia said its upcoming Groq 3 LPU hardware will rely on large amounts of static random-access memory, or SRAM, a component also used by Cerebras, the AI chipmaker that filed to go public earlier this month. Google's new inference chip, called TPU 8i, also uses SRAM. Each chip contains 384 megabytes of SRAM, three times the amount in Ironwood.

The architecture is designed "to deliver the massive throughput and low latency needed to concurrently run millions of agents cost-effectively," Sundar Pichai, CEO of Google parent Alphabet, wrote in a blog post.

Google said adoption of its AI chips is increasing. Citadel Securities has built quantitative research software using Google's TPUs, and all 17 U.S. Energy Department national laboratories use AI co-scientist software built on the chips. Anthropic has committed to using multiple gigawatts worth of Google TPUs.