I Built Self-Evolving Claude Code Memory w/ Karpathy's LLM Knowledge Bases

The video introduces the concept of LLM knowledge bases, highlighting Andrej Karpathy's influential tweet on using LLMs to build personal knowledge bases for research. The speaker shares his own memory system built on this concept, emphasizing its simplicity and effectiveness. Karpathy's approach involves ingesting external information, organizing it, and making it queryable by AI agents, often employing Obsidian as a core tool. The architecture covers data ingestion, viewing, querying, formatting, and health checks.

Karpathy's system is explained using a compiler analogy. Raw source code (articles, papers) is ingested into Obsidian's 'raw' folder. An LLM acts as the 'compiler,' processing this raw data to create a 'wiki' (executable) with summaries and linked documents, leveraging Obsidian's graph view for connections. A 'test suite' (linting) ensures data integrity, identifying gaps, stale data, and broken links. Finally, 'runtime' involves agents querying the wiki using an index file, which guides the agent without needing complex RAG systems or vector databases.

The speaker introduces his custom implementation, which adapts Karpathy's architecture for internal codebase data. Instead of external information, this system gives Claude Code a memory that evolves with the codebase. It captures session logs using hooks, automatically extracting and structuring knowledge articles. This allows the coding agent to become smarter by remembering past decisions and project evolutions. The sponsor, InsForge, an open-source platform, is highlighted for enabling coding agents to ship full-stack applications with an integrated database, authentication, storage, and AI models.

To set up this memory system, users can simply send a specific prompt to their Claude Code agent, which will configure everything. While not mandatory, using Obsidian as a vault is recommended to visualize the memories and wiki. The system is entirely driven by Claude Code hooks defined in `settings.json`. A `session_start` hook loads the `agents.mmd` (global rules) and `index.mmd` (actively maintained file list) to provide context to the agent. This enables the agent to answer questions efficiently by referencing its knowledge base.

The system utilizes `pre_compact` and `session_end` hooks to capture and summarize conversations, storing them in daily log files. These summaries, containing decisions, lessons, and action items, are processed by a separate Claude agent (using the Claude agent SDK). A daily 'flush' process extracts concepts and connections from these logs and populates the wiki. This creates a compounding loop: asking questions leads to synthesized answers, which are then filed and added to the wiki, making the knowledge base grow and the agent's ability to search and provide answers improve over time.