OpenAI's /v1/models schema carries no context window, so a client that only
lists models cannot know how large a request it may send, and oversizing one
is a hard ContextWindowExceededError 400 — vLLM counts prompt + max_tokens
against its window. opencode already read the window from /v1/model/info.
Hermes and pi never consult that endpoint: they take it from static config,
which was hand-copied and drifted. Hermes named a model the gateway had
stopped serving while its replacement (the one model with a SMALLER window)
had no entry; pi's reserve was half its maxTokens. Both fail only once a
session grows, which is why it looked like the model or the GPU.
This renders one /v1/model/info response into all three clients. It is not a
plugin per harness and needs no upstream change — it extends the renderer that
already translates the MCP servers into Hermes' schema, with the same
manifest discipline.
SAME SOURCE, THREE DIFFERENT NUMBERS, deliberately — each client means
something different by "context":
opencode limit.context = window - 1024 reserves limit.output and targets
context - output, so needs headroom
Hermes context_length = window omits max_tokens entirely; vLLM
bounds output by what is left
pi maxTokens <= reserveTokens-2048 compacts at window - reserveTokens
(compaction.js shouldCompact) then
requests maxTokens
pi's maxTokens is derived FROM its reserve rather than the reserve being raised
to fit: reserveTokens is one global setting shared by every provider pi knows,
so raising it for this gateway would shrink everyone's usable context.
Safety properties, each tested in a sandbox against copies of the real
configs before touching live ones (11 cases):
- Ownership is per FIELD. A value is retracted only if this set it and the
gateway stopped declaring it. Hand-added models and fields are never
touched; a model removed upstream loses only what was rendered.
- Vision is retracted, not just added. The chart states that supports_vision
is what makes clients send image parts, so if a backend falls back to
language-only, the `image` input this added is taken back — but a user's own
`image` on an entry is left alone.
- Raw /v1/model/info is cached on each successful fetch, because refresh is
throttled and a throttled shell never fetches. A bad fetch (403 body, HTML
error page, empty list, rows without model_name) never replaces a good cache.
- Never invents a window: claude-* declares none and is skipped. A gateway
that suddenly declares no windows is treated as broken, not empty.
- One output cap (MCP_SYNC_OUTPUT_RESERVE_CAP) is read by opencode and pi, so
the clients cannot drift apart on it. Parses under dash as well as bash,
since install.sh sources this under /bin/sh.
First live run found a bug nobody knew about: pi's Claude entries carried
maxTokens 64000 against a 34816 reserve, so claude-haiku-4-5 sent
(200000 - 34816) + 64000 = 229184 tokens at a 200000 window — still broken
after the earlier manual reserve fix, which only covered 32768. Now 32768, and
every served model satisfies the invariant. It also gave Hermes limits for the
Claude routes it had none for.
hermes/gateway.yaml: the static map is marked as a first-boot seed that the
next sync overwrites, so nobody hand-edits it believing it is authoritative.
The source of truth is model_info.max_input_tokens in ai-k8s-iac's
charts/inference/values.yaml. README documents all of the above.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FoDK4wwB1Twty5gwiriYLf
noconnorandClaude Opus 5
d42d0b29ff