Six months ago, AI chip startup Groq was flying under the radar, making scant headlines when CEO Jonathan Ross sent Elon Musk a sarcastic cease-and-desist letter complaining about xAI’s similarly named chatbot, Grok. But that all changed on Monday, when Silicon Valley-based Groq, one of the startups buying parts of Nvidia’s AI chip business, announced it had raised $640 million in a BlackRock-led funding round that valued the company at $2.8 billion. Industry leaders like Meta CEO Mark Zuckerberg have been promoting Groq, and Meta’s chief AI scientist Yann LeCun is on board as a technical adviser.
But it was that buzz moment in February 2024 that set the stage for today’s massive funding round. “That was a 100% game changer,” Ross told Fortune magazine in a recent interview.
Unlike Nvidia GPUs, which are used for both training and model output (a process known as “inference”) of today’s most sophisticated AI models, Groq’s AI chips are strictly focused on improving inference speed, meaning they provide extremely fast text output of large language models (LLMs) at a much lower cost than Nvidia GPUs.
But Groq struggled with how to demonstrate the power of its chip to potential users. In the end, the answer for Groq was to create a unique experience like ChatGPT.
Groq put its own conversational chatbot on its website in February, claiming to have broken the speed record for LLM output among open-source models, including Meta’s Llama. A developer then posted a short video on X showing how Groq, installed on Paris startup Mistral’s LLM, could answer a question of a few hundred words in less than a second. Ross says the company’s fortunes quickly changed. Suddenly, thousands of developers were clamoring to build AI tools using Groq’s powerful AI chips. Just six months later, 300,000 developers now have access to Groq’s solutions and hardware through its AI cloud service.
AI chip on the cloud
But for now, most developers use Groq’s service for free, so it’s unclear how the 200-employee company plans to make money. “Right now, we only offer paid access to a little over 30 customers,” Ross said, because capacity is limited. The company expects to have “significant revenue” this year, he said, but “one of the benefits of being a private company is we don’t have to talk about revenue.”
With little insight into revenue, it may seem easy to question Groq’s long-term prospects, but Ross has outperformed expectations for years. After dropping out of high school because he was “bored,” he studied computer programming and managed to get into New York University after studying at Hunter College. There, he did a PhD program for two years as an undergraduate but dropped out again. “I didn’t want to limit my income opportunities by graduating,” he joked. That led to a job at Google, where he helped invent Google’s AI chip, the TPU, before leaving in 2016 to start Groq.
Ross says Groq isn’t a venture-funded startup; it wants to build a sustainable business. “Sam Altman said he doesn’t care how much money we lose,” he says. “We’re going to recoup our investment with the actual money we raise, so we’re going to recoup every dollar with the hardware we deploy.” He explains that Groq has been able to raise more than $500 million because “there’s more demand than we can serve.” The investment will allow the company to build out more hardware and charge customers who want higher rate limits.
Groq isn’t the only AI chip startup looking to challenge Nvidia: Cerebras, for example, recently filed privately for an IPO. Also joining the fray are SambaNova, Etched and Fractile. And of course, incumbent GPU chipmakers like AMD are also ramping up their AI efforts. But analyst Daniel Newman recently told Fortune that “Nvidia doesn’t have a natural adversary right now.”
That said, even if Groq were to only get a small piece of Nvidia’s pie, it would still bring in plenty of business. “I don’t know if Nvidia would notice how much of the pie we were eating, but we’d be happy with it,” Ross said. “It would be a great multiple in terms of our valuation going forward.”
