Context Management

← Infrastructure

Based on strategies from Effective Context Engineering for AI Agents by Anthropic’s Applied AI team.

Prax manages its LLM context window to prevent overflow, reduce token costs, and maintain response quality. The ContextManager sits between the orchestrator and the LLM, automatically budgeting, clearing, and compacting the context before every invocation.

Why this matters

LLM context windows are finite resources. As conversations grow:

• Context rot — accuracy degrades as token volume increases
• Attention dilution — important information gets lost in noise
• Cost escalation — more tokens = more API spend per turn
• Hard failures — exceeding the window causes errors

Architecture

┌──────────────────────────────────────────────────────────┐
│                     Orchestrator                         │
│                                                          │
│  System Prompt (~11K tokens)                             │
│       + Memory Context (STM + LTM, ~500 token budget)    │
│       + Workspace Context                                │
│       + Difficulty / Metacognitive Hints                  │
│       + Conversation History (unbounded without mgmt)    │
│       + Current User Message                             │
│                         │                                │
│                         ▼                                │
│           ContextManager.prepare_context()                │
│                         │                                │
│            ┌────────────┼────────────┐                   │
│            ▼            ▼            ▼                   │
│     1. Clear old   2. Count     3. Compact               │
│        tool results   tokens       or truncate           │
│        → stubs                     if over budget        │
│            │            │            │                   │
│            └────────────┼────────────┘                   │
│                         ▼                                │
│              Prepared messages (within budget)           │
│                         │                                │
│                         ▼                                │
│                   graph.invoke()                         │
└──────────────────────────────────────────────────────────┘

flowchart TD
    SP[System Prompt ~11K] --> CM[ContextManager]
    MEM[Memory STM+LTM ~500] --> CM
    HIST[Conversation History] --> CM
    TR[Tool Results] --> CM
    UM[User Message] --> CM

    CM --> S1[1. Clear Old Tool Results]
    S1 --> S2[2. Count Tokens]
    S2 --> S3{Over Budget?}
    S3 -- No --> OUT[Prepared Messages]
    S3 -- Yes --> S4[3. LLM Compaction]
    S4 --> S5{Still Over?}
    S5 -- No --> OUT
    S5 -- Yes --> S6[4. Hard Truncation]
    S6 --> OUT
    OUT --> LLM[LLM Invocation]

Strategies

1. Token budgeting — per-model limits, capped

The budget for a call is worked out in three steps:

1. The model is the one that will run. A component configured by tier only (the usual case for the orchestrator) uses its tier’s model.
2. Its window comes from the table below, else from what its provider reports (OpenRouter’s public catalogue lists context_length for every model), less 20% for the response; else the tier fallback.
3. The result is capped by CONTEXT_BUDGET_TOKENS (default 64,000; 0 = up to the window). A 1M-token window is not a reason to resend 800k tokens on every tool call.

Fixed 2026-10. Before this, an empty model name always meant the tier fallback, so production’s 1M-window OpenRouter models were budgeted at 50,000. Every turn in one long scheduled conversation (75 of 96 turns in a month of logs) started at about 128k tokens, was summarized, and was still truncated. Nothing was persisted, so the next turn did it all again.

The orchestrator logs token usage every turn:

Context budget: 8432/100000 tokens (system=6200, history=2232)

2. Tool result clearing — old results to stubs

The safest, lightest-touch form of compaction. Old tool results (beyond the last 6) are replaced with stubs:

Before: ToolMessage(content="<3000 chars of search results>")
After:  ToolMessage(content="[Result from background_search — cleared for context efficiency]")

The agent’s interpretation of the results (in its AIMessage response) is preserved — only the raw output is cleared. This is applied first, before any token counting or budget checks.

3. LLM compaction — summarize oldest half

When the conversation exceeds the budget after tool result clearing, the oldest half of turns are summarized by a cheap LLM.

• What the summarizer sees. All of that half: each message is clipped to 4,000 characters and tool results to 600, up to 240,000 characters in total, most recent first. It used to see only the first 8,000 characters, a few of the oldest messages.
• Reuse. A prefix summarized before is reused, and only what came after it is summarized, with the earlier summary as context. If exactly that part was summarized before, no model call is made. The cache is in memory, so a restart costs one full summary.
• The split never separates a tool result from its call.

The summary preserves:

• Key decisions and their reasoning
• Unresolved issues or bugs
• User preferences and corrections
• File paths, URLs, and identifiers

The most recent turns are kept verbatim. This creates a natural “working memory” effect — recent context is detailed, older context is summarized.

If the LLM summarization call fails, compaction falls back to hard truncation.

4. Hard truncation — oldest-first, always preserve system message

If compaction still leaves the context over budget, the oldest messages are dropped one at a time. The system message is always preserved. After dropping, the message list is adjusted to ensure it starts with a valid HumanMessage (not an orphaned AIMessage or ToolMessage).

4b. Inside the turn — before every model call (CONTEXT_IN_LOOP_ENABLED)

Steps 1–4 run once, when a turn starts. Inside the turn, every tool result is re-sent on every later call. Spokes start from just their task, so nothing managed their context at all. With CONTEXT_IN_LOOP_ENABLED, a loop middleware (ContextBudget) applies the same policy before each model call:

• past 60% of the budget, tool results beyond the last six become stubs;
• past 85%, the older part is summarized.

Only the outgoing request changes; the loop’s state and the record keep everything. There is no recall tool: “An Empirical Study of Harness Design for Coding Agents” (assessment) finds models almost never use one.

5. Spoke architecture — 67 to 35 tools

Context quality also depends on tool count — every tool in the system prompt costs tokens for its description, even when the agent never calls it. The spoke architecture reduces the orchestrator’s tool count:

Each spoke agent gets its own clean context window with only the tools it needs.

Per-model context window lookup

The context manager maintains a lookup table of known models and their context budgets (with ~20% headroom reserved for the response):

When a model is not in the table, get_context_limit tries substring matching (e.g., claude-sonnet-4-6-20260101 matches claude-sonnet-4-6), then the window its provider reports (model_catalog.context_window, from OpenRouter’s catalogue, cached with the model list), then the tier-based defaults. Whichever applies, CONTEXT_BUDGET_TOKENS caps it:

Model override system

Users can override the orchestrator’s model at runtime. The override takes precedence over the config default.

Runtime API

PUT /teamwork/model
{ "model": "claude-opus-4-6" }

• Setting "model": "auto" clears the override and reverts to the config default.
• Setting "model": null also clears the override.

Conversational

Users can say “switch to Claude Opus” or “use GPT-5.4-pro” in chat, and the orchestrator picks it up via the model picker tool.

TeamWork UI

The Memory panel’s Model section shows the current model and provides a dropdown to switch. The change takes effect on the next message.

Implementation

In orchestrator.py:

_model_override: str | None = None

def set_model_override(model: str | None) -> None:
    """Pass None or 'auto' to clear."""
    global _model_override
    if model and model.lower() == "auto":
        model = None
    _model_override = model

def get_model_override() -> str | None:
    return _model_override

The ConversationAgent.__init__ resolves the effective model as:

effective_model = _model_override or model or cfg.get("model")

Context inspector

The TeamWork Memory panel’s Context tab shows real-time context usage:

• History messages — number of messages in the conversation
• History tokens — token count of the conversation history
• System prompt tokens — token count of the system prompt (including memory, hints)
• Total tokens — combined total
• Per-tier limits — the budget ceilings for each tier
• Overflow flag — whether the context exceeded its budget

This is served by the GET /teamwork/context/stats endpoint.

Token budget breakdown example

A typical turn for a gpt-5.4-mini (100K budget):

For a gpt-5.4-nano (12K budget), the same conversation would trigger compaction around turn 10-15, and hard truncation if the compacted summary plus recent turns still exceeds 12K.

API

GET /teamwork/context/stats

Returns current context usage for the user:

{
  "history_messages": 42,
  "history_tokens": 8500,
  "system_prompt_tokens": 11200,
  "total_tokens": 19700,
  "limits": {
    "low": 12000,
    "medium": 50000,
    "high": 100000,
    "pro": 160000
  }
}

Configuration

Context limits are defined per model in context_manager.py, with provider-reported windows and tier-based fallbacks. The orchestrator’s tier is determined by the orchestrator component config in llm_routing.yaml. The model override (if set) takes precedence over both.

References

• Effective Context Engineering for AI Agents — Anthropic’s Applied AI team guide (the basis for this implementation)
• Building Effective Agents — Anthropic’s research on agent architectures
• context_manager.py — Implementation: token counting, tool clearing, compaction, truncation
• orchestrator.py — Model override system, context preparation call site
• model_tiers.py — Tier resolution and model selection