The Blog Post That Set the AI World on Fire
An incendiary post from a prominent researcher suggests that AI is actually more predictable than we thought.
For the past 24 hours, Silicon Valley has been obsessed with a 26,000-word blog post by the AI researcher Nick Cammarata. The document has been called the “the most important essay,” “the most important result,” and “the single best piece of news” of the year, or in the history of AI research—or, perhaps, ever. Even OpenAI CEO Sam Altman lauded it. Cammarata makes a series of bold, almost astonishing claims about the nature of AI that, if true, would upend some of his industry’s basic assumptions about the very thing it is building. And after months of AI-hacking sprees and mounting public backlash against the industry, many people see Cammarata as offering a glimmer of hope when it comes to not just understanding but also controlling the technology being unleashed on the world.
Boiled down, Cammarata’s main contention is that generative-AI models are not actually black boxes, contrary to the common perception that these systems are so big and labyrinthine as to be nearly impenetrable. In fact, Cammarata argues, there is a relatively clear path toward understanding them. If that’s the case, the material significance could be tremendous: It would provide a way to map AI models’ inner workings and, in turn, perhaps prevent them from wreaking all sorts of harm—autonomously hacking companies and sovereign nations, encouraging users’ delusions, appearing to deceive AI researchers themselves. There is an entire field of research, called “mechanistic interpretability,” devoted to the study of just this.
The argument is just as important as the person making it. Cammarata, a former OpenAI employee, is a respected figure in his field. Years ago, he played a key role in establishing the opposite claim, which is now common wisdom: that neural networks, the software architecture that generative AI is built on, are impossible, or at least very difficult, to understand. In this new essay, Cammarata, who now works at the AI-research company Goodfire, essentially says that he and his entire field have been wrong ever since. “The claims here are viscerally shocking,” one observer wrote. Or, as another wroteon X, they are “big big big big.”
Neural networks, inspired by the human brain, consist of layered connections among individual nodes, or “neurons,” that activate and send information to one another when responding to a query. Earlier research, including Cammarata’s, established that individual machine neurons can activate in response to many different, unrelated topics—for instance, one neuron can correspond to images of cat faces, cat legs, and the front of cars. Nobody was sure why this happened, but one explanation was that, because AI models contain far more information than they have neurons, they have to repurpose these nodes. Regardless, the basic units of an AI model having seemingly arbitrary, multitudinous functions would make it very difficult to understand what is going on under the hood.