Summary
Highlights
Andrej Karpathy clarifies his statement that it's the 'decade of agents,' not the 'year of agents.' He explains that while current agents like Claude and Codex are impressive, there's significant work remaining. He believes it will take about a decade to address issues like lack of general intelligence, multimodality, computer use, and continual learning in AI.
Karpathy reflects on seismic shifts in AI over the past 15 years. He describes early deep learning, the reorientation caused by AlexNet, the misstep of focusing on reinforcement learning for games (like Atari), and his own early work on agents using keyboard/mouse to operate web pages, which was prematurely attempted due to insufficient representational power.
Karpathy differentiates AI intelligence from animal intelligence. He argues that animals evolve with built-in hardware, while AI, particularly LLMs, are 'ghosts' or 'spirits' trained by imitating human data on the internet. He suggests that while we can make AI more 'animal-like' over time, their learning process is fundamentally different from biological maturation.
In-context learning is highlighted as a visible form of AI intelligence, emerging from pre-training and resembling a working memory. Karpathy suggests that AI models' strong memorization capabilities can be a hindrance, and future research should focus on extracting a 'cognitive core' (intelligence and problem-solving algorithms) from knowledge to avoid over-reliance on learned facts.
Karpathy discusses the parts of human intelligence not yet replicated in LLMs, drawing analogies to brain structures like the cortical tissue, prefrontal cortex, basal ganglia, and hippocampus. He emphasizes the lack of a 'distillation phase' in LLMs, akin to human sleep, where learning is consolidated into long-term memory, leading to issues like model collapse.
Karpathy anticipates that in 10 years, AI will still involve large neural networks trained with gradient descent, but with significant improvements in data quality, hardware, software, and algorithms. He notes that historical progress in AI has been a simultaneous improvement across all these fronts, rather than a single breakthrough.
Karpathy shares his experience building 'nanochat,' a simple ChatGPT clone, noting that coding models were of limited help. He categorizes interactions with coding models into full rejection, autocomplete (his preferred method), and 'vibe coding' (full automation). He explains that AI agents struggle with novel, intellectually intense code and can make a codebase 'sloppy' due to their cognitive deficits and tendency to rely on boilerplate examples.
Karpathy views AI as a continuum of computing and automation, not a distinct technology. He highlights how various tools, from code editors to search engines, have progressively automated tasks and abstracted human roles. He sees AI, including agents, as part of this 'autonomy slider,' where humans do less low-level work, moving up the abstraction layer.
Karpathy criticizes reinforcement learning (RL) as 'terrible' for intelligence tasks, describing it as 'sucking supervision through a straw' due to its high-variance estimation. He explains that current RL systems often upweight entire trajectories based on single outcomes, ignoring process correctness. He discusses how using LLMs as judges in process-based supervision is vulnerable to adversarial examples and 'model collapse' due to their collapsed data distribution and lack of diverse output.
Karpathy speculates on the optimal size for a 'cognitive core' of intelligence, suggesting it could be around a billion parameters in 20 years, enabling productive conversations without factual memorization. He believes that cleaning up current, 'terrible' internet training data and focusing on cognitive components could lead to smaller, more effective models.
Karpathy defines AGI as systems that can perform any economically valuable task at human or superhuman levels. He notes that the definition has evolved, often removing physical work. He proposes measuring progress by identifying jobs amenable to automation, like call center employees, and observing the 'autonomy slider' where AI handles increasing portions of tasks, with humans supervising.
Karpathy views superintelligence as a natural extrapolation of societal automation and recursive self-improvement that has been ongoing for centuries. He posits that AI won't cause a sudden, discrete jump in economic growth but will continue the existing hyper-exponential trend, leading to a 'firecracker event' observed in slow motion. He expresses concern over a gradual loss of human control and understanding in this increasingly automated world, potentially leading to chaotic competition among autonomous entities.
Karpathy explains that coding is a natural first domain for LLMs due to its text-based nature and existing infrastructure (IDEs, diffs). He contrasts this with domains like presentation slides, which lack such text-based structure and tools, making automation much harder. He acknowledges that even in language-in/language-out tasks, LLMs can struggle to deliver consistent economic value.
Drawing on his experience at Tesla, Karpathy highlights the significant 'demo-to-product gap' in fields with high costs of failure, like self-driving. He describes achieving full reliability as a 'march of nines,' where each additional 'nine' of accuracy requires a constant, substantial amount of work. He views software engineering with production-grade code as having similar safety requirements, leading to extended timelines for real-world deployment.
Karpathy is focusing on education, building 'Starfleet Academy' (Eureka) to empower humans in an AI future. He envisions an AI-powered tutor providing a highly personalized learning experience, adapting to individual needs like an excellent human tutor. He recognizes that current AI capabilities aren't sufficient for this vision, but projects a future where learning is trivialized and pursued for enjoyment, much like physical fitness today.
Karpathy's physics background informs his educational philosophy: abstracting core concepts and building 'ramps to knowledge.' He illustrates this with 'micrograd,' a 100-line Python code that demystifies backpropagation. He advocates for presenting the 'pain' (problem) before the 'solution,' enabling students to deeply understand the 'why' behind each concept, maximizing 'eurekas per second'.
Karpathy addresses the 'curse of knowledge,' where experts struggle to explain concepts to beginners. He suggests using tools like ChatGPT to simulate a beginner's questions and encourages peer explanation as a powerful learning tool. He highlights that often, concise conversational explanations are more insightful than formalized academic writing, which tends towards jargon and abstraction, masking the core ideas.