启发式学习Heuristic Learning | OpenAI翁家翌 | 超越梯度的学习 | 持续学习的困境 | 灾难性遗忘 | 用代码Agent来代替神经网络权重 | 吸收反馈 | 压缩历史

Weng Jiayi's idea for HL originated from a need for low-cost, repeatable heuristic strategies to test game environments efficiently. Using Codex and GPT-5.4 to write purely rule-based strategies, initial experiments on Atari Breakout, MuJoCo Ant, and HalfCheetah robots yielded unexpected results, reaching or surpassing deep reinforcement learning benchmarks. Most notably, in Atari57 game tests, heuristic strategies with fixed interaction steps significantly outperformed PPO, all without training any neural networks. This demonstrated HL's ability to achieve high performance by continuously growing and iterating software systems rather than optimizing model parameters.

Weng Jiayi formally defines Heuristic Learning (HL) as a paradigm where the core carrier of learning is program code. Unlike deep reinforcement learning, which updates neural network parameters, HL updates the structure of software. Its feedback is diverse, including environmental rewards, test cases, logs, and human feedback. Update mechanisms involve coding agents directly modifying policy code, detectors, and memory. The Heuristic System (HS) is the core object maintained by HL, comprising a complete, iterative, and verifiable system of rules, feedback, history, and update paths.

The video outlines six key differences between HL and Deep Reinforcement Learning (DRL). HL's policy is human-readable code, while DRL's is opaque neural network parameters. HL's state is explicitly human-readable; DRL's is implicit observation data. HL actions are generated by code logic, DRL's by neural networks. HL's feedback is comprehensive, while DRL's is primarily fixed reward functions. HL updates by direct code modification, DRL by gradient descent. HL explicitly stores memories including experiment records and failure reasons, while DRL has limited or no explicit memory. These differences give HL advantages in interpretability, sample efficiency, regression testability, overfitting control, and mitigating catastrophic forgetting.

Traditional expert systems and rule-based methods failed due to prohibitively high maintenance costs, leading to brittle and unmaintainable 'code monsters.' Coding agents, however, industrialize the maintenance process, allowing for automated iterations and evolutionary changes. The agent perceives environmental feedback, test failures, and log anomalies, then modifies code, tests, or memory, reruns the program, and records the results. This automated, iterative process makes maintainable heuristic systems possible, a capability lacking in past expert systems.

While HL can still experience forgetting (e.g., new rules breaking old functions), it's treatable and reversible, unlike neural network catastrophic forgetting. Old capabilities are solidified into verifiable forms like regression test cases, rather than being compressed into weights. A healthy HS must absorb new feedback and compress historical data by integrating scattered patches into a simpler architecture. This transforms the problem of continual learning from 'how to update neural network parameters' to 'how to maintain a software system that continuously absorbs feedback.'

Coupling complexity refers to the policy complexity an agent must manage for HL to iterate stably. It's not about code lines but the interconnectedness of states, rules, tests, feedback, and historical constraints during updates. Code-side factors (modular design, stable interfaces, test coverage) and coding agent-side factors (model capability, context length, memory quality, tool quality) influence this complexity. Effective engineering practices like modularization and testing can externalize some complexity, while memory and tools enhance the agent's effective context length, preventing systems from becoming unmanageable 'code piles.'

HL has the potential to become a mainstream paradigm, particularly for online and continual learning, as it effectively addresses problems that can be continuously iterated. However, HL is limited by code expressiveness and cannot replace neural networks in tasks like complex perception or long-range generalization. The true future lies in their deep integration: HL rapidly processes online data, converting real-time experience into high-quality, testable data which then periodically updates neural networks. This dual-system approach, with HL and shallow neural networks handling System One tasks and large language models (System Two) guiding HL and overall updates, offers a robust path forward for AI.

The video opens by introducing the long-standing challenge of catastrophic forgetting in AI, where models learning new tasks forget old capabilities. It then presents OpenAI engineer Weng Jiayi's new paper, 'Learning Beyond Gradients,' which proposes Heuristic Learning (HL) as a solution. HL uses coding agents to update code rules and maintain software systems, achieving performance comparable to deep reinforcement learning without neural networks. This new approach addresses the limitations of traditional rule-based systems and offers a promising direction for continual learning.