Summary
Highlights
Lance introduces Pete from Manis and sets the stage for the discussion on context engineering. He explains that while prompt engineering emerged with ChatGPT, context engineering became prominent with the rise of AI agents, particularly around May of the current year. Agents accumulate a large amount of context through tool calls, which can lead to performance degradation if not managed properly. Context engineering is defined as the art and science of filling the context window with only the necessary information for the next step.
Lance outlines common themes in context engineering: context offloading, reducing context, retrieving context, and context isolation. Offloading involves moving information out of the main message history to an external storage (like a file system) to be retrieved only when needed. Reducing context can be done by summarizing or compressing information. Retrieving context involves different strategies like indexing/semantic search or simple file-based search tools. Context isolation is often achieved by splitting context across multiple sub-agents, each with its own context window.
Pete from Manis emphasizes that context engineering is crucial to avoid being trapped by model iteration speeds. He explains that trying to fine-tune models too early or focusing on specialized models can hinder product innovation, especially for startups. Instead, leaning on general models and context engineering for as long as possible is a more sustainable approach, as the AI landscape is constantly changing.
Pete distinguishes between compaction and summarization for context reduction. Compaction involves stripping out reconstructible information and is reversible, making it crucial for agents that chain predictions. Summarization, on the other hand, is irreversible and is used when compaction is no longer sufficient. Manis uses a threshold-based approach, prioritizing compaction first and only resorting to summarization when necessary, always ensuring that recent tool calls remain in full detail.
Pete discusses context isolation, borrowing wisdom from multi-process coordination. He highlights two patterns: 'by communicating' and 'by sharing memory'. 'By communicating' involves a main agent sending a prompt to a sub-agent with a limited context, suitable for clear, short tasks. 'By sharing memory' means sub-agents can access the entire previous context, necessary for complex tasks where full history is vital, such as detailed research. However, sharing context is more expensive in terms of tokens and KV cache usage.
Pete explains Manis' approach to context offloading through a layered action space to manage tool context effectively and avoid 'context confusion'. This includes: 1) Function calling (atomic functions for basic operations), 2) Sandbox utilities (pre-installed tools in a virtual machine environment), and 3) Packages and APIs (for complex computations and external services). This layered approach keeps the model's interface simple while expanding its capabilities significantly.
Pete concludes that context engineering is a delicate balance of conflicting objectives: offloading, reducing, retrieving, isolating, and caching. He stresses the importance of avoiding 'context over-engineering' and emphasizes that simplifying the architecture often leads to faster, more stable, and smarter systems. The goal is to make the model's job simpler, not harder.
Pete answers questions about how the LLM discovers and invokes shell tools, explaining that hints in the system prompt and the '-help' flag are used. He also discusses Manis' approach to memory, where each sandbox session is new, relying on 'knowledge' for explicit user-confirmed memory, avoiding index databases for speed, and exploring self-improving agents through collective feedback.
Pete explains that Manis frequently refactors its architecture (5 times since March) due to constant model changes and improvements. He highlights testing architecture by switching between weaker and stronger models to understand future-proofness. Regarding data formats, Manis prioritizes line-based formats over markdown for easier processing with Linux tools and to avoid unnecessary bullet points from the model.
Pete elaborates on summarization, suggesting structured outputs with schemas instead of free-form prompts to ensure stability and completeness. For compaction, he explains that for token-heavy tool outputs like search results, Manis stores them in the file system and provides a minimal reference (like a file name or query) back to the agent. Complex search scenarios use 'agent as tool' to manage detailed workflows and return a structured output.
Pete discusses agent-to-agent communication, explaining that Manis uses a shared sandbox for context transfer and defines output schemas with constraint decoding for sub-agents to ensure structured results. He also touches on model choice, noting that while they don't use open-source models due to cost and KV cache efficiency, they leverage various frontier models (Anthropic, Gemini, OpenAI) based on the specific subtask requirements.
Pete recommends a rule of thumb for tool selection: no more than 30 tools in the direct action space. Manis keeps its core functions atomic (10-20 tools) and offloads everything else to the sandbox. He clarifies that Manis uses a hybrid approach, allowing direct tool calls for simple actions and empowering the agent to write and execute scripts in the sandbox for vastly expanded capabilities.
Pete details Manis' planning mechanism, which now uses a separate planner agent for structured planning, moving away from simple to-do lists to save tokens. He discusses guardrailing in the sandbox, ensuring data doesn't leave without authorization and requiring manual user confirmation for sensitive operations. For evaluations, Manis prioritizes user ratings, internal automated tests, and human interns for qualitative assessments over academic benchmarks.
Pete advises against heavy reinforcement learning (RL) for most companies, especially with the rapidly changing action spaces of agents and foundational models. He explains that RL becomes effectively building a foundational model, and it's more beneficial to leverage the progress of large model providers. Manis instead focuses on parameter-free online learning and collective feedback for personalization.