fe8b9a3ec0fae5a310c94f41fa2d388b9aebf58f
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  noconnorandClaude Opus 5
|
d42d0b29ff |
mcp-config: render gateway context limits into Hermes and pi, not just opencode
OpenAI's /v1/models schema carries no context window, so a client that only
lists models cannot know how large a request it may send, and oversizing one
is a hard ContextWindowExceededError 400 — vLLM counts prompt + max_tokens
against its window. opencode already read the window from /v1/model/info.
Hermes and pi never consult that endpoint: they take it from static config,
which was hand-copied and drifted. Hermes named a model the gateway had
stopped serving while its replacement (the one model with a SMALLER window)
had no entry; pi's reserve was half its maxTokens. Both fail only once a
session grows, which is why it looked like the model or the GPU.
This renders one /v1/model/info response into all three clients. It is not a
plugin per harness and needs no upstream change — it extends the renderer that
already translates the MCP servers into Hermes' schema, with the same
manifest discipline.
SAME SOURCE, THREE DIFFERENT NUMBERS, deliberately — each client means
something different by "context":
opencode limit.context = window - 1024 reserves limit.output and targets
context - output, so needs headroom
Hermes context_length = window omits max_tokens entirely; vLLM
bounds output by what is left
pi maxTokens <= reserveTokens-2048 compacts at window - reserveTokens
(compaction.js shouldCompact) then
requests maxTokens
pi's maxTokens is derived FROM its reserve rather than the reserve being raised
to fit: reserveTokens is one global setting shared by every provider pi knows,
so raising it for this gateway would shrink everyone's usable context.
Safety properties, each tested in a sandbox against copies of the real
configs before touching live ones (11 cases):
- Ownership is per FIELD. A value is retracted only if this set it and the
gateway stopped declaring it. Hand-added models and fields are never
touched; a model removed upstream loses only what was rendered.
- Vision is retracted, not just added. The chart states that supports_vision
is what makes clients send image parts, so if a backend falls back to
language-only, the `image` input this added is taken back — but a user's own
`image` on an entry is left alone.
- Raw /v1/model/info is cached on each successful fetch, because refresh is
throttled and a throttled shell never fetches. A bad fetch (403 body, HTML
error page, empty list, rows without model_name) never replaces a good cache.
- Never invents a window: claude-* declares none and is skipped. A gateway
that suddenly declares no windows is treated as broken, not empty.
- One output cap (MCP_SYNC_OUTPUT_RESERVE_CAP) is read by opencode and pi, so
the clients cannot drift apart on it. Parses under dash as well as bash,
since install.sh sources this under /bin/sh.
First live run found a bug nobody knew about: pi's Claude entries carried
maxTokens 64000 against a 34816 reserve, so claude-haiku-4-5 sent
(200000 - 34816) + 64000 = 229184 tokens at a 200000 window — still broken
after the earlier manual reserve fix, which only covered 32768. Now 32768, and
every served model satisfies the invariant. It also gave Hermes limits for the
Claude routes it had none for.
hermes/gateway.yaml: the static map is marked as a first-boot seed that the
next sync overwrites, so nobody hand-edits it believing it is authoritative.
The source of truth is model_info.max_input_tokens in ai-k8s-iac's
charts/inference/values.yaml. README documents all of the above.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FoDK4wwB1Twty5gwiriYLf
|
||
|
noconnorandClaude Opus 5
|
135112eee5 |
mcp-config/hermes: make the gateway's context limits actually reach the clients
Folds in the in-flight local-checkout work and fixes the context accounting
on top of it. Both halves are the same bug seen from two ends: clients were
sending requests larger than the backend's window and getting hard
ContextWindowExceededError 400s, intermittently, in a way that looked like
the model or the GPU misbehaving.
Already in flight here (unchanged in intent):
- sync.sh prefers the LOCAL CHECKOUT over re-downloading from Gitea, and
install.sh records the checkout path so it can be found. That recording
is load-bearing: without it sync.sh cannot find the repo, silently falls
back to Gitea, and every local edit to opencode.gateway.json is ignored.
Observed on this box — ~/.config/mcp-sync/source did not exist, the
default ~/.dotfiles is not where the checkout lives, and the Gitea base
still served models:{}.
- the hermes custom-provider wiring.
Fixed on top:
opencode had NO context limit on ANY gateway model, not a wrong one. Two
faults compounded. /model/info 403s for a day-to-day virtual key (only
management-scoped keys may read it), which sync.sh treats as best-effort and
falls back to names-only — as designed and as commented. But the merge then
did `prov["models"] = models`, replacing the map wholesale, so limits
declared in the base config were discarded too and there was no floor left.
Base specs are now a floor that live /model/info overrides field by field:
discovery still wins where present, so this adds no drift, and a 403 no
longer strips the limits. Verified: ornith 195584/32768, the qwens
261120/32768.
hermes' model map named `qwen3-vl-30b-a3b` at 262144 — a model the gateway
stopped serving when Ornith replaced it — while ornith-1.0-35b, the model
actually on cg-1 AND the only one with a window smaller than its neighbours
(196608), had no entry at all and inherited Hermes' default. The one model
needing a lower ceiling was the one not declaring one, which is exactly the
shape that fails only once a session grows.
NOTE for other boxes: hermes reads ~/.hermes/config.yaml, which install.sh
produces by MERGING gateway.yaml. Editing gateway.yaml alone changes nothing
until install.sh re-runs — this machine's live config was patched by hand.
Not addressed here, deliberately: the gateway-side context_window_fallback
(ai-k8s-iac). It is wired but inert while cg-2 is scaled down, so preventing
the oversized request client-side is what actually works today.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FoDK4wwB1Twty5gwiriYLf
|
||
|
noconnorandClaude Opus 4.8
|
e42807c054 |
hermes: wire into dotfiles; fix opencode's generated context limits
Two client-side fixes against the covert-gpt LiteLLM gateway, both traced from opencode throwing intermittent ContextWindowExceededError 400s. opencode (mcp-config/sync.sh): spec_for() mapped LiteLLM's /model/info onto opencode's schema 1:1, but the two mean different things by "output". LiteLLM's max_output_tokens is the whole window (vLLM caps TOTAL context, not output separately, which is correct for LiteLLM); opencode's limit.output is a RESERVE it subtracts from context to pick a compaction point. Copying one to the other told opencode to reserve the entire 262144 window, so it targeted an input of context-output and the server counted one token more than opencode did -- landing on context+1 and a hard 400, every time (measured: 196609 in + 65536 out = 262145 vs a 262144 window; the retries shrinking max_tokens by 65 never helped because the input grew to compensate). Capping the reserve alone would NOT have fixed it -- opencode would still sit exactly on the boundary, just at a different input size. So: cap the reserve (OUTPUT_RESERVE_CAP) and hold a margin back from context (CONTEXT_SAFETY_MARGIN). Generated at sync time, so it fixes every gateway model at once, including third-party ones. hermes (hermes/gateway.yaml, hermes/install.sh): Hermes reached the gateway by pretending it WAS OpenRouter -- the LiteLLM key in OPENROUTER_API_KEY, the gateway in OPENROUTER_BASE_URL. Requests routed fine, but fetch_openrouter_models hardcodes openrouter.ai/api/v1/models and ignores OPENROUTER_BASE_URL, so the picker listed OpenRouter's ~342-model catalog -- models this gateway cannot serve -- while the gateway's own models, including cg-1's, never appeared at all. Replaced with a named custom provider. install.sh now merges gateway.yaml into ~/.hermes/config.yaml, so a fresh box gets a Hermes that can actually talk to the gateway instead of one pointed at nothing. The merge is idempotent and narrow: only the covert-gpt entry is touched, model.default/provider are seeded only when unset, and a box still on the masquerade is migrated off it (fingerprint: provider openrouter + base_url pointing at this gateway; a genuine OpenRouter user never has that, and is left alone). No secrets: gateway.yaml names the key via key_env, and mcp-config/sync.sh is what exports it. Hermes needs no context safety margin -- it omits max_tokens entirely, so vLLM bounds output by whatever context is left. Also folds in the in-flight hermes bootstrap wiring. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |