Summary
Highlights
Andrej Karpathy argues that the current excitement around AI agents is overblown, characterizing the current period as the 'decade of agents' rather than the 'year of agents.' He explains that while current agents like Claude and Codex are impressive, they lack intelligence, multimodality, continual learning, and cognitive abilities necessary for truly effective operation, suggesting it will take a decade to overcome these issues.
Karpathy reflects on nearly two decades in AI, noting seismic shifts in the field. He discusses the early adoption of deep learning, the transition to task-specific neural networks, and the 'misstep' of focusing on reinforcement learning in games (like Atari and OpenAI Universe). He emphasizes that foundational technologies, such as large language models providing strong representations, are necessary before agents can become truly competent.
Karpathy is cautious about drawing direct analogies between AI and animal intelligence. He highlights that animals are products of evolution with built-in hardware and maturation, while AI models are 'ghosts' or 'spirits' trained by imitating human data from the internet. He suggests that pre-training is a 'crappy evolution' for AI, providing a starting point for learning, but is distinct from biological evolution.
Karpathy distinguishes between knowledge gained during pre-training (a 'hazy recollection' due to massive compression) and information accessible in the context window (more like 'working memory'). He notes that while in-context learning shows signs of intelligence, potentially involving internal gradient descent mechanisms, the ultimate goal is to strip models of excessive 'knowledge' and retain a 'cognitive core' of intelligence and problem-solving algorithms.
Karpathy suggests that current LLMs resemble a piece of cortical tissue, capable of pattern recognition, but lack many other 'brain parts' crucial for human-like intelligence, such as emotions, instincts, and perhaps even a dedicated 'hippocampus.' He believes these cognitive deficits prevent LLMs from being effective 'employees' and contribute to the longer timeline for agent development.
Karpathy discusses the absence of continual learning in LLMs, contrasting it with human learning and memory distillation during sleep. He highlights the problem of 'model collapse,' where LLMs generate data with reduced diversity and richness. He explains that while current applications don't always demand diversity, this limitation hinders progress in synthetic data generation and sustained learning.
Karpathy notes a trend where state-of-the-art models are becoming smaller. He believes that models can achieve a very productive 'cognitive core' with as few as a billion parameters, especially if trained on higher-quality, more curated datasets. He emphasizes that the internet, as a primary training source, is 'garbage,' and improving dataset quality is crucial for more efficient and smaller, yet more capable, models.
Karpathy views AI progress as a continuum of automation, analogous to advances in computing since the 1970s. He prefers the definition of AGI as a system capable of performing any economically valuable task at human-level or better. He observes that AI's impact is gradually diffusing across the economy, noting that coding is an ideal first task for LLMs due to its text-based nature and extensive pre-built infrastructure.
Self-driving demonstrates the 'demo-to-product gap,' where early impressive demonstrations are far from production-ready. Karpathy explains that achieving reliability requires a 'march of nines,' with each improvement demanding significant effort. He applies this to AI, especially in critical domains like software engineering, noting that the cost of failure can be high in both, justifying longer deployment timelines.
Karpathy envisions superintelligence as a natural extrapolation of automation, leading to increased autonomy but also a 'gradual loss of control and understanding' for humans. He believes that the 'intelligence explosion' is already happening, manifested as continuous economic growth and technological advancement over centuries. He predicts AI will integrate smoothly into this existing exponential growth curve rather than causing a sudden, discrete jump.
Karpathy's new endeavor, Eureka, aims to build an 'elite institution' for technical knowledge, like a 'Starfleet Academy,' to prevent humanity from being disempowered by AI. He envisions AI-powered tutors that provide personalized, challenging, and engaging learning experiences, elevating human capabilities. He believes that learning should become as enjoyable and rewarding as physical fitness, driven by intrinsic human desire for self-improvement rather than purely economic motivation.
Karpathy shares insights into his teaching philosophy, heavily influenced by his physics background. He emphasizes breaking down complex topics into 'first-order terms' and building 'ramps to knowledge' that simplify concepts for learners. He illustrates this with examples like his 'micrograd' project (100 lines of code explaining backpropagation) and his approach to teaching transformers, starting with simple bigrams. He also highlights the 'curse of knowledge' and the importance of active engagement (explaining concepts to others) for deeper understanding.