public/dotfiles
2
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

mcp-config: render gateway context limits into Hermes and pi, not just opencode

Browse Source 
OpenAI's /v1/models schema carries no context window, so a client that only
lists models cannot know how large a request it may send, and oversizing one
is a hard ContextWindowExceededError 400 — vLLM counts prompt + max_tokens
against its window. opencode already read the window from /v1/model/info.
Hermes and pi never consult that endpoint: they take it from static config,
which was hand-copied and drifted. Hermes named a model the gateway had
stopped serving while its replacement (the one model with a SMALLER window)
had no entry; pi's reserve was half its maxTokens. Both fail only once a
session grows, which is why it looked like the model or the GPU.

This renders one /v1/model/info response into all three clients. It is not a
plugin per harness and needs no upstream change — it extends the renderer that
already translates the MCP servers into Hermes' schema, with the same
manifest discipline.

SAME SOURCE, THREE DIFFERENT NUMBERS, deliberately — each client means
something different by "context":

  opencode  limit.context = window - 1024   reserves limit.output and targets
                                            context - output, so needs headroom
  Hermes    context_length = window        omits max_tokens entirely; vLLM
                                            bounds output by what is left
  pi        maxTokens <= reserveTokens-2048 compacts at window - reserveTokens
                                            (compaction.js shouldCompact) then
                                            requests maxTokens

pi's maxTokens is derived FROM its reserve rather than the reserve being raised
to fit: reserveTokens is one global setting shared by every provider pi knows,
so raising it for this gateway would shrink everyone's usable context.

Safety properties, each tested in a sandbox against copies of the real
configs before touching live ones (11 cases):

- Ownership is per FIELD. A value is retracted only if this set it and the
  gateway stopped declaring it. Hand-added models and fields are never
  touched; a model removed upstream loses only what was rendered.
- Vision is retracted, not just added. The chart states that supports_vision
  is what makes clients send image parts, so if a backend falls back to
  language-only, the `image` input this added is taken back — but a user's own
  `image` on an entry is left alone.
- Raw /v1/model/info is cached on each successful fetch, because refresh is
  throttled and a throttled shell never fetches. A bad fetch (403 body, HTML
  error page, empty list, rows without model_name) never replaces a good cache.
- Never invents a window: claude-* declares none and is skipped. A gateway
  that suddenly declares no windows is treated as broken, not empty.
- One output cap (MCP_SYNC_OUTPUT_RESERVE_CAP) is read by opencode and pi, so
  the clients cannot drift apart on it. Parses under dash as well as bash,
  since install.sh sources this under /bin/sh.

First live run found a bug nobody knew about: pi's Claude entries carried
maxTokens 64000 against a 34816 reserve, so claude-haiku-4-5 sent
(200000 - 34816) + 64000 = 229184 tokens at a 200000 window — still broken
after the earlier manual reserve fix, which only covered 32768. Now 32768, and
every served model satisfies the invariant. It also gave Hermes limits for the
Claude routes it had none for.

hermes/gateway.yaml: the static map is marked as a first-boot seed that the
next sync overwrites, so nobody hand-edits it believing it is authoritative.
The source of truth is model_info.max_input_tokens in ai-k8s-iac's
charts/inference/values.yaml. README documents all of the above.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FoDK4wwB1Twty5gwiriYLf
This commit is contained in:
noconnor
2026-09-12 18:50:02 -04:00
co-authored by Claude Opus 5
parent 9519cbe609
commit d42d0b29ff
3 changed files with 401 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mcp-config/README.md
+47
View File
@@ -56,6 +56,53 @@ needs PyYAML (it warns if that's missing). Nothing needs adding to Hermes'
`platform_toolsets` — MCP servers are auto-enabled by name; listing *some*
would turn it into an allowlist, and `no_mcp` disables them all.
### Context limits — discovered, not hand-copied
OpenAI's `/v1/models` carries **no context window**, so a client that only
lists models cannot know how big a request it may send, and oversizing one is a
hard `ContextWindowExceededError` 400 (vLLM counts `prompt + max_tokens` against
its window). The window lives in LiteLLM's `/v1/model/info`, and `sync.sh`
renders it into every client from that one response:
| client | reads | what `sync.sh` writes | why that number |
|---|---|---|---|
| opencode | `opencode.gateway.json` | `limit.context = window − 1024`, `limit.output = min(out, cap)` | reserves `output` and targets `context − output`, so needs headroom |
| Hermes | `~/.hermes/config.yaml` | `context_length = window` | omits `max_tokens`; vLLM bounds output by what is left, so no margin |
| pi | `~/.pi/agent/models.json` | `contextWindow = window`, `maxTokens = min(out, cap, reserveTokens − 2048)` | compacts at `window − reserveTokens` then asks for `maxTokens`, so `maxTokens` must fit inside the reserve |
Same source, three different numbers — each client means something different by
"context". Do not "simplify" them into one.
**The single source of truth is `model_info.max_input_tokens` in ai-k8s-iac's
`charts/inference/values.yaml`.** A wrong window is fixed there and propagates on
the next sync. Editing a rendered value in a client config is overwritten.
Requirements and behaviour:
- **The key needs `/v1/model/info`.** A virtual key scoped to `llm_api_routes`
gets a 403 and every client silently falls back to its own defaults. Grant it
in the LiteLLM UI (Virtual Keys → key → Allowed Routes). The match is on the
exact path: `/v1/model/info` does not also grant `/model/info`.
- **Cached** at `~/.config/mcp-sync/model-info.json` on each successful fetch, so
a throttled shell still renders. A bad fetch (403, error page, empty list)
never replaces a good cache.
- **Ownership is tracked, per field.** `hermes-models-managed.json` and
`pi-managed.json` record what was set, so a value is retracted only if `sync.sh`
set it and the gateway stopped declaring it. That includes vision: if a backend
falls back to language-only and `supports_vision` flips off, the `image` input
`sync.sh` added is taken back rather than left to send image parts the backend
rejects. Models and fields you add by hand are never touched.
- **Never invents a window.** A wildcard route like `claude-*` declares none, so
it is skipped. A gateway that suddenly declares no windows at all is treated as
broken, not empty — nothing is retracted.
- **`pi`'s `maxTokens` is derived from its reserve**, not the other way round:
`reserveTokens` is one global setting shared by every provider pi knows, so
raising it to suit this gateway would shrink everyone's context. Raise
`compaction.reserveTokens` in pi's settings to buy back output. The shared
output cap is `MCP_SYNC_OUTPUT_RESERVE_CAP` (default 32768), read by opencode
and pi alike.
- Opt out per client with `MCP_SYNC_HERMES_CONFIG=` or `MCP_SYNC_PI_MODELS=`.
Credentials are referenced as `{env:VAR}` and are **never** written into the
config. Each user exports them in `~/.zshrc.local`: