Summary
Highlights
Todd Toller introduces the session, explaining that GitHub Copilot's transition to usage-based billing has raised questions about token consumption. He argues that focusing on agent quality is more effective than solely on cost, as higher quality agents naturally lead to fewer tokens spent. He uses an analogy of launching rockets to the moon, suggesting that it's better to launch fewer, more accurate rockets than many imprecise ones.
The current 'gambling' approach with agents, where users send off many agents hoping one succeeds, is unsustainable. Investing in agent quality increases the return on investment. Furthermore, large language models are non-deterministic, and errors can compound in multi-step workflows. Even at 99% accuracy per step, 50 steps can reduce overall accuracy to 61%. Quality improvements dramatically increase success rates and reduce wasted tokens from failed attempts.
A high-level overview of LLMs as text-in, text-out machines that predict the next most probable word or instruction. The core principle for response quality is 'context balance': provide as little context as possible, but as much as required. Too much irrelevant context biases the model, while too little can lead to hallucinations. Agents interact with LLMs, sending entire conversations, causing tokens and context to compound with each turn. Critical problems with context windows include the 'lost in the middle' bias, where models favor content at the beginning and end, and 'recency bias,' where models forget earlier instructions if the context window grows too large (beyond 50-60% capacity).
Optimization efforts should scale with usage. For infrequent users, minimal optimization is needed. For 'AI engineers' orchestrating many agents, every percentage of token saving or quality increase is significant. The first major lever is model choice: common patterns can use cheaper, older models, resulting in significant cost savings (e.g., 24x difference between Claude Opus 4.7 and GPT 54 mini). Auto mode in VS Code dynamically assigns the best model based on the task. The second lever is providing only relevant context, avoiding stuffing prompts with unnecessary information. Using the `/clear` command to start fresh for new tasks or when the context window becomes too long is crucial.
Prompts should be optimized for steering the agent correctly, not for fewer tokens. Be precise with instructions (e.g., 'fix issue #45' instead of 'fix the bug'). Work in phases (research, plan, implement) and create new context windows between phases to avoid carrying irrelevant information. For planning, use reasoning models, and for implementation, smaller models may suffice to prevent the reasoning model from 'going rogue'. Deterministic controls, such as writing comprehensive tests, are essential. They ensure agents build on a stable foundation and bring them back on track when errors occur, ultimately saving tokens by preventing compound bugs.
Agent configurations, especially `copilot-instructions.mmd` and `agent.md`, provide always-on guidance. These should be kept concise, human-written, and contain non-negotiables, agent 'miss' prevention, and output trimming instructions. Custom agents can force specific roles or workflows, invoked manually or automatically, and can restrict tool access to prevent agents from going down unintended paths. Skills are dynamically loaded markdown descriptions that further refine agent behavior based on the task, allowing for efficient context offloading. It's important not to overdo skills, avoid redundancy, and use them only for capabilities the agent wouldn't otherwise possess.
Model Context Protocol (MCPs) enable external API calls and dynamic tools for agents. While powerful, they can bloat tool descriptions (wasting tokens) and lead agents to call undesired tools. It's recommended to deactivate unnecessary MCPs or use them with custom agents only when needed (e.g., Playwright MCP for web front-end work). Sub-agents open secondary context windows for specific tasks (like research), preventing the main session from filling with irrelevant information. They improve main session quality but involve a trade-off in tokens spent within the sub-agent.
Power user tips include thinking in code to filter outputs, using CLIs over MCPs where models are already proficient, optimizing shell output with tools like RTK, using `chronicle` to analyze session logs for prompt optimization, and collapsing tool calls to reduce turns. Important future traits for successful agentic development are building analytical skills (agents execute but don't strategize), applying good architecture (domain-driven design, hexagonal architecture act as guardrails), and continuously iterating on prompts and agent configurations (context engineering is an ongoing process).
A recap of the five biggest tips to improve agent quality and token spend: choose the right model for the task, provide clear guidance in prompts, split tasks (research, plan, implement) to manage context size, provide deterministic guardrails (tests, linters, security scans), and maintain a concise, human-written `copilot-instructions` file. The overarching principle is to provide 'as little context as required and as much as necessary' to make every token count.