OpenAI's /v1/models schema carries no context window, so a client that only
lists models cannot know how large a request it may send, and oversizing one
is a hard ContextWindowExceededError 400 — vLLM counts prompt + max_tokens
against its window. opencode already read the window from /v1/model/info.
Hermes and pi never consult that endpoint: they take it from static config,
which was hand-copied and drifted. Hermes named a model the gateway had
stopped serving while its replacement (the one model with a SMALLER window)
had no entry; pi's reserve was half its maxTokens. Both fail only once a
session grows, which is why it looked like the model or the GPU.
This renders one /v1/model/info response into all three clients. It is not a
plugin per harness and needs no upstream change — it extends the renderer that
already translates the MCP servers into Hermes' schema, with the same
manifest discipline.
SAME SOURCE, THREE DIFFERENT NUMBERS, deliberately — each client means
something different by "context":
opencode limit.context = window - 1024 reserves limit.output and targets
context - output, so needs headroom
Hermes context_length = window omits max_tokens entirely; vLLM
bounds output by what is left
pi maxTokens <= reserveTokens-2048 compacts at window - reserveTokens
(compaction.js shouldCompact) then
requests maxTokens
pi's maxTokens is derived FROM its reserve rather than the reserve being raised
to fit: reserveTokens is one global setting shared by every provider pi knows,
so raising it for this gateway would shrink everyone's usable context.
Safety properties, each tested in a sandbox against copies of the real
configs before touching live ones (11 cases):
- Ownership is per FIELD. A value is retracted only if this set it and the
gateway stopped declaring it. Hand-added models and fields are never
touched; a model removed upstream loses only what was rendered.
- Vision is retracted, not just added. The chart states that supports_vision
is what makes clients send image parts, so if a backend falls back to
language-only, the `image` input this added is taken back — but a user's own
`image` on an entry is left alone.
- Raw /v1/model/info is cached on each successful fetch, because refresh is
throttled and a throttled shell never fetches. A bad fetch (403 body, HTML
error page, empty list, rows without model_name) never replaces a good cache.
- Never invents a window: claude-* declares none and is skipped. A gateway
that suddenly declares no windows is treated as broken, not empty.
- One output cap (MCP_SYNC_OUTPUT_RESERVE_CAP) is read by opencode and pi, so
the clients cannot drift apart on it. Parses under dash as well as bash,
since install.sh sources this under /bin/sh.
First live run found a bug nobody knew about: pi's Claude entries carried
maxTokens 64000 against a 34816 reserve, so claude-haiku-4-5 sent
(200000 - 34816) + 64000 = 229184 tokens at a 200000 window — still broken
after the earlier manual reserve fix, which only covered 32768. Now 32768, and
every served model satisfies the invariant. It also gave Hermes limits for the
Claude routes it had none for.
hermes/gateway.yaml: the static map is marked as a first-boot seed that the
next sync overwrites, so nobody hand-edits it believing it is authoritative.
The source of truth is model_info.max_input_tokens in ai-k8s-iac's
charts/inference/values.yaml. README documents all of the above.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FoDK4wwB1Twty5gwiriYLf
Folds in the in-flight local-checkout work and fixes the context accounting
on top of it. Both halves are the same bug seen from two ends: clients were
sending requests larger than the backend's window and getting hard
ContextWindowExceededError 400s, intermittently, in a way that looked like
the model or the GPU misbehaving.
Already in flight here (unchanged in intent):
- sync.sh prefers the LOCAL CHECKOUT over re-downloading from Gitea, and
install.sh records the checkout path so it can be found. That recording
is load-bearing: without it sync.sh cannot find the repo, silently falls
back to Gitea, and every local edit to opencode.gateway.json is ignored.
Observed on this box — ~/.config/mcp-sync/source did not exist, the
default ~/.dotfiles is not where the checkout lives, and the Gitea base
still served models:{}.
- the hermes custom-provider wiring.
Fixed on top:
opencode had NO context limit on ANY gateway model, not a wrong one. Two
faults compounded. /model/info 403s for a day-to-day virtual key (only
management-scoped keys may read it), which sync.sh treats as best-effort and
falls back to names-only — as designed and as commented. But the merge then
did `prov["models"] = models`, replacing the map wholesale, so limits
declared in the base config were discarded too and there was no floor left.
Base specs are now a floor that live /model/info overrides field by field:
discovery still wins where present, so this adds no drift, and a 403 no
longer strips the limits. Verified: ornith 195584/32768, the qwens
261120/32768.
hermes' model map named `qwen3-vl-30b-a3b` at 262144 — a model the gateway
stopped serving when Ornith replaced it — while ornith-1.0-35b, the model
actually on cg-1 AND the only one with a window smaller than its neighbours
(196608), had no entry at all and inherited Hermes' default. The one model
needing a lower ceiling was the one not declaring one, which is exactly the
shape that fails only once a session grows.
NOTE for other boxes: hermes reads ~/.hermes/config.yaml, which install.sh
produces by MERGING gateway.yaml. Editing gateway.yaml alone changes nothing
until install.sh re-runs — this machine's live config was patched by hand.
Not addressed here, deliberately: the gateway-side context_window_fallback
(ai-k8s-iac). It is wired but inert while cg-2 is scaled down, so preventing
the oversized request client-side is what actually works today.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FoDK4wwB1Twty5gwiriYLf
Hermes had no MCP servers at all -- no mcp_servers: key -- so "use the gitea
MCP" had nothing to call. The model said so ("there's no Gitea-specific skill"),
which was true, and then looped reaching for the browser as the only way in.
The obvious fix -- paste a gitea block into ~/.hermes/config.yaml -- is the
wrong one: it makes a SECOND definition of a server this repo already defines
once, free to drift (one pinned to gitea-mcp@0.0.10, the other to whatever got
typed). README already calls this out: "all MCP servers are defined here", and
listed other clients as "not done yet". So render, don't duplicate: sync.sh
already translates LiteLLM's /model/info into opencode's schema; this is the
same move into Hermes' schema, from opencode.gateway.json's mcp block. Add a
server once, both clients get it within the same TTL.
Not a shared server PROCESS, which is what "use the same one" would imply: MCP
stdio servers are subprocesses spawned per client over stdin/stdout and dying
with it. Each client running its own `npx -y <pkg>` is the design -- same
package, same pinned version, same per-user credential, same identity. Sharing
one process would need a persistent daemon plus a stdio->HTTP bridge, for
nothing. And it stays client-side: the gateway hosts no MCP servers on purpose
(values.yaml:892; user-scoped APIs, no shared tokens), and gitea-mcp hard-exits
without a token so LiteLLM's headerless listing never registers it anyway.
Translation notes: Hermes' `command` must be a STRING (a list stringifies to
"['npx', '-y']" and fails at exec), so opencode's argv list splits into
command + args. `{env:VAR}` becomes `${VAR}` -- both by-name, so no secret is
written to either file. Hermes gives an MCP child only a filtered baseline env,
so every secret must be listed in env:, which the rendered block does. Nothing
is added to platform_toolsets: MCP servers auto-enable by name, and listing some
would turn it into an allowlist.
Rendered names are recorded in ~/.config/mcp-sync/hermes-managed.json so a
server dropped upstream is dropped from Hermes, while servers the user added by
hand -- never in the manifest -- are left alone. Only writes when the result
changes, skips silently when Hermes isn't installed, warns when PyYAML is
missing (an absent MCP block is invisible rather than obviously broken).
Verified via Hermes' own discovery: 72 tools register -- 30 gitea (incl.
issue__list) and 42 proxmox -- with ${GITEA_KEY} / ${PROXMOX_MCP_*} resolving
and no secret in config.yaml.
Also fixes a stale README claim that MCP tools come from the gateway's /mcp
endpoint; they never do, and values.yaml:892 says why.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sync runs only at shell start and is throttled to MCP_SYNC_TTL (1h), so a
model added on the gateway can take up to an hour AND a new terminal before
opencode sees it. There was no way to say "fetch it now", so the recurring
symptom was "I added a model and it isn't showing up" -- with a healthy
gateway and a correctly-scoped key.
`mcp-sync` re-sources the hook with the throttle disabled and prints the
resulting model list, so it also answers "what does the client actually have?"
in one step.
README gains a section splitting the two failure modes: a model the gateway
returns but opencode lacks is a stale client; a model the gateway does not
return is a LiteLLM model_list / key-scoping problem.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every API worth exposing (Gitea, Proxmox) is user-scoped, so hosting MCP
centrally means one shared token per server: everyone acts as the same
identity and attribution is lost. LiteLLM's per-request BYOK cannot cover
all of them either -- it builds its tool->server map from a headerless
background listing, and gitea-mcp hard-exits without a token, so it never
registers. A split brain (some MCPs server-side, some client-side) is worse
than either, so all of them move client-side.
Servers are declared local, spawned via npx with pinned versions, and read
each user's OWN credentials via {env:VAR}. No token is ever written to the
config; the file stays identical for every user.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Models and MCP servers are discovered from the LiteLLM gateway at sync time
(/v1/models, and the aggregate /mcp/ endpoint). Skills have no gateway side to
query: LiteLLM's /v1/skills is a passthrough to Anthropic's Skills API, which
lists skills on an Anthropic account rather than serving skill files, and
/public/skill_hub is a marketplace catalog. So the repo is the source of truth
and the tree is fetched whole, from the repo tarball.
Deploy to ~/.config/mcp-sync/skills rather than ~/.config/opencode/skills so
the atomic replace can never delete a skill the user wrote by hand; opencode
picks the tree up via skills.paths, which expands {env:HOME} — verified — so
the shared config stays identical for every user.
Same throttle and non-fatal discipline as the config sync: a fetch that yields
no skills/ leaves the last-good tree in place instead of emptying it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two changes to the synced opencode config:
- Model provider. opencode can't list models from a custom OpenAI-compatible
provider itself, so sync.sh queries the gateway's /v1/models with the
user's key and materializes the map client-side. New gateway models land
on every box at the next sync with no config edit. If discovery fails the
previous map is carried over, so a gateway hiccup never empties the list.
- Aggregate MCP endpoint. One entry at /mcp/ (trailing slash required — the
bare path 307-redirects and MCP clients don't follow it) exposes every
server the key can access, replacing the per-server gitea entry.
sync.sh also exports OPENCODE_CONFIG: opencode does not auto-load extra
files from its config dir, so without it the synced file was ignored.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
noconnorandClaude Opus 5