Research · negative result · 26 July 2026 · updated 8 October 2026

Does compacting tool output lower a coding agent's API bill?

A paired experiment on DeepSeek V4 Pro: 75 sessions, 4.81 MB of repeated tool output removed, and no measurable saving.

Coding agents resend old tool output on every turn, so replacing it with something smaller looks like an obvious saving. I built that as a Torana plugin, broke it once, rebuilt it, and measured it. On DeepSeek V4 Pro the saving was about 2% per session, smaller than ordinary run-to-run variation, because prompt caching had already made the repeated output nearly free.

This write-up covers the idea, what broke, the final design, the experiment setup and methodology, the results, and what's next.

Experiment setup ↓ · Results ↓ · Why it didn't pay ↓ · What's next ↓

How coding agents resend tool output

The model behind a coding agent can't read files or run commands itself. It asks for a tool call, the agent runs it locally, and the output is appended to the conversation. On every following turn, the agent sends the whole visible conversation back to the model, including every earlier tool result.

This applies to harnesses that resend visible history, typically over Chat Completions or Messages-style APIs. When a provider owns the history (for example OpenAI Responses with previous_response_id), Torana can't rewrite earlier items; native provider compaction is a separate mechanism.

The idea: a cheaper assistant for the hosted model

If the hosted model is the brain doing the real work, a cheaper model could summarise tool output before the brain reads it. A summary is only useful if it knows what matters, so Torana's intent plugin changes the tool definitions the model sees to request a short reason with every tool call, then strips that field before the harness sees the call. The compactor uses the captured intent to decide what to keep.

Attempt 1: summarise every large output

The first compactor summarised any tool output over 2,000 characters. It broke coding agents immediately (issue #166). Coding agents edit files with exact search and replace; when a file read came back as a natural-language summary, edits stopped matching. The agent re-read the file, failed again, and looped, burning tokens instead of saving them.

Attempt 2: exact by default, compact only older output

The second design leaves everything exact by default. Policies are ordered, first-match and case-insensitive; unmatched tools are never changed:

• exact: never altered. Mutation tools, diffs, failed commands, errors, stack traces and non-zero exits are forced exact even if a broader rule matches them.
• deterministic: keeps bounded head and tail evidence plus size, SHA-256, omitted-byte count and a rerun instruction. With first_pass, it applies from the first time the model sees the output, so the prompt never needs a later rewrite.
• keyword (keyword_compactor): keeps intent-matching lines, only after at least one exact exposure.
• model (compactor): an intent-guided summary, only after at least one exact exposure, and only when the economic gate predicts a net saving.
• source (file reads): accepted in configuration but behaves as exact (see what broke).

Rewriting an older message changes the prompt prefix from that point on. The model-gated compactor therefore runs two economic gates: a preflight that checks whether even a best-case reduction could repay the cache rewrite (before paying for a summary), and a final check on the actual candidates. It applies a batch only when the estimated net saving is positive.

Experiment setup and methodology

Environment

• Date: 21 July 2026.
• Harness: OMP 16.2.9 (coding agent).
• Target model: DeepSeek V4 Pro via https://api.deepseek.com/beta (OpenAI-compatible format).
• Summariser (model-gated arm): DeepSeek V4 Flash on the same endpoint.
• Torana: commit e09d225 from PR #179. That PR was the experiment branch; the plugins were later moved to torana-plugins, and current releases differ. This is the implementation used for the historical run; the unpublished task prompts and run artifacts below prevent exact reproduction from this article alone.

Prices used (July 2026, per million tokens)

Arms

• Control: Torana in the request path with no compaction plugin.
• Deterministic: schema_translator → intent → keyword_compactor, summariser disabled. This is the committed config.dogfood.json. The run itself used the https://api.deepseek.com/beta endpoint for the target model.

{
  "providers": {
    "deepseek": { "url": "https://api.deepseek.com", "format": "openai" },
    "deepseek-anthropic": { "url": "https://api.deepseek.com/anthropic", "format": "anthropic" }
  },
  "plugins": {
    "order": ["schema_translator", "intent", "keyword_compactor"],
    "config": {
      "keyword_compactor": {
        "tool_policies": [
          { "match": "read*", "mode": "exact" },
          { "match": "view*", "mode": "exact" },
          { "match": "web_search", "mode": "deterministic", "first_pass": true,
            "rerun": "Repeat the search to recover every result." },
          { "match": "grep*", "mode": "keyword" }
        ]
      }
    }
  },
  "offload": { "enabled": false, "provider": "deepseek", "model": "deepseek-v4-flash" },
  "limits": { "concurrency": 10, "rpm": 100 }
}

• Model-gated: schema_translator → intent → compactor, with explicit prices, expected_applications: 6 and V4 Flash as the summariser. Grep-like output was eligible for model summaries; the same exact and first-pass rules applied otherwise. Reconstructed from the run record in the shape of the committed example (the committed file covers only the deterministic arm).

{
  "providers": {
    "deepseek": {
      "url": "https://api.deepseek.com/beta",
      "format": "openai",
      "pricing": {
        "deepseek-v4-pro":   { "input_usd_per_mtok": 0.435, "output_usd_per_mtok": 0.87,
                               "cache_read_usd_per_mtok": 0.003625, "cache_write_usd_per_mtok": 0.435 },
        "deepseek-v4-flash": { "input_usd_per_mtok": 0.14,  "output_usd_per_mtok": 0.28,
                               "cache_read_usd_per_mtok": 0.0028,   "cache_write_usd_per_mtok": 0.14 }
      }
    }
  },
  "plugins": {
    "order": ["schema_translator", "intent", "compactor"],
    "config": {
      "compactor": {
        "expected_applications": 6,
        "tool_policies": [
          { "match": "read*", "mode": "exact" },
          { "match": "view*", "mode": "exact" },
          { "match": "web_search", "mode": "deterministic", "first_pass": true,
            "rerun": "Repeat the search to recover every result." },
          { "match": "grep*", "mode": "model" }
        ]
      }
    }
  },
  "offload": { "enabled": true, "provider": "deepseek", "model": "deepseek-v4-flash" }
}

Protocol

• Tasks: five repository tasks run with OMP: architecture, economics, regression review, safety (exercising mutations, patches, errors and non-zero exits), and a stress task.
• Design: five repetitions × three arms = 75 sessions. Within each paired block (task × repetition), the order of the three arms was randomised.

Results

Deterministic arm: 15 of 25 treatment runs transformed something; 399 applications (34 new transformations, 365 cache reuses) removed 4,809,164 bytes in total. Median paired saving $0.000524 (~2.1%), 95% interval −$0.003714 to +$0.003725. Mean saving −$0.0000559; cheaper in 13 of 25 pairs. Median paired input-token saving 29,117 (interval −57,989 to +60,161). Transformed-only runs had a median change of −$0.000262, a small increase.

Model-gated arm: only four runs transformed output, all through the deterministic rule (6 transformations, 79 reuses). Median paired saving $0.000478 (~1.9%), interval −$0.000998 to +$0.004973; mean $0.001569, cheaper in 15 of 25 pairs. The preflight made zero Flash calls: with six expected applications at DeepSeek's cache-hit price, no rewritten prefix could repay its cache misses. This measures the gate, not model-written summaries.

Loop stops (request-cap terminations) happened in the control arm and in treatment runs with no transformations, so they reflect agent variance rather than compaction.

Why it didn't pay: prompt-cache economics

About 92% of input tokens were cache hits in every arm, and a cache hit on V4 Pro cost 0.83% of a miss: about 120× cheaper. The repeated tool output being removed was already nearly free. A rewrite changes the prompt from that point on, so everything after it is billed as new input until the cache is rebuilt.

That is a property of DeepSeek's pricing. Providers that charge more for cached input, or have no prompt cache at all, could change the result. That is the next experiment (issue #193).

The break-even condition

Let N be future turns, R the rewritten cached span, W the provider's write/input rate, C its cache-read rate, D tokens removed per future turn, and S the summariser cost. Compaction needs:

N > (R × max(0, W − C) + S) / (D × C)

This simplified model assumes positive D × C and matching units: rates per token when S is in dollars. It doesn't model every provider's caching rules or changes in agent behaviour. When economic inputs are missing or the expected net is not positive, the gate does nothing, which is a correct outcome.

What broke along the way

1. File-read markers caused re-read loops. An earlier, excluded pilot replaced an old file read with a "reread" marker. The agent fetched different line ranges of the same file until it hit the request cap; deduplicating identical arguments didn't help because the offsets kept changing. Source reads are now always exact (#178 tracks a bounded alternative).
2. The gate missed a cost. An uncached summary candidate was mislabelled as a cache reuse, so the preflight left out the prefix-rewrite premium. Torana paid for Flash summaries that the final gate rejected. Uncached candidates are now treated as rewrites; the corrected run made zero uneconomic summariser calls.
3. Cache hits were invisible. DeepSeek reports cache usage as prompt_cache_hit_tokens, not the OpenAI-style prompt_tokens_details.cached_tokens. Torana now reads both in streaming, JSON and model-service responses, and /stats reports summariser token usage.

Limitations

• One provider, one harness, five tasks, five repetitions per task, and July 2026 prices.
• Agent trajectories varied between runs, and many treatment sessions were no-ops.
• Bytes are not tokenizer counts; token savings are provider-reported.
• The quality heuristic understated semantic quality and was only used for triage.
• The excluded pilot is retained only as negative safety evidence.
• Compaction can still matter where caching is expensive or absent, or when an agent runs out of context.

What's next

Providers with expensive cache reads. A fixed transcript replay to isolate pricing from agent behaviour, then live paired runs on providers whose cache pricing doesn't make history nearly free (#193).

Decide per tool call, not by size. Some output must stay exact: file reads, edits, errors. Some never needed to stay in the conversation: npm install logs, build output, script runs. Small local decision models such as Jev and Laya can classify a tool call into a fixed set of choices. I'm building a version where one decides, per tool call, whether its output stays exact or can be compacted. That is in progress, not a measured result.

Try it or build on it

compactor, keyword_compactor and intent are Torana plugins. Install them by name, review their permissions and bind a summariser following the compactor guide. Measure with paired runs on your own tool mix and prices, using provider-reported tokens and including summariser cost.

The same plugin SDK powers the local PII check. To build your own, start with the plugin-authoring guide. Results from other providers, especially on #193, are welcome.

How this experiment became Torana: the pivot story →