Two client-side fixes against the covert-gpt LiteLLM gateway, both traced
from opencode throwing intermittent ContextWindowExceededError 400s.
opencode (mcp-config/sync.sh):
spec_for() mapped LiteLLM's /model/info onto opencode's schema 1:1, but the
two mean different things by "output". LiteLLM's max_output_tokens is the
whole window (vLLM caps TOTAL context, not output separately, which is
correct for LiteLLM); opencode's limit.output is a RESERVE it subtracts from
context to pick a compaction point. Copying one to the other told opencode to
reserve the entire 262144 window, so it targeted an input of context-output
and the server counted one token more than opencode did -- landing on
context+1 and a hard 400, every time (measured: 196609 in + 65536 out =
262145 vs a 262144 window; the retries shrinking max_tokens by 65 never
helped because the input grew to compensate).
Capping the reserve alone would NOT have fixed it -- opencode would still sit
exactly on the boundary, just at a different input size. So: cap the reserve
(OUTPUT_RESERVE_CAP) and hold a margin back from context
(CONTEXT_SAFETY_MARGIN). Generated at sync time, so it fixes every gateway
model at once, including third-party ones.
hermes (hermes/gateway.yaml, hermes/install.sh):
Hermes reached the gateway by pretending it WAS OpenRouter -- the LiteLLM key
in OPENROUTER_API_KEY, the gateway in OPENROUTER_BASE_URL. Requests routed
fine, but fetch_openrouter_models hardcodes openrouter.ai/api/v1/models and
ignores OPENROUTER_BASE_URL, so the picker listed OpenRouter's ~342-model
catalog -- models this gateway cannot serve -- while the gateway's own models,
including cg-1's, never appeared at all.
Replaced with a named custom provider. install.sh now merges gateway.yaml into
~/.hermes/config.yaml, so a fresh box gets a Hermes that can actually talk to
the gateway instead of one pointed at nothing. The merge is idempotent and
narrow: only the covert-gpt entry is touched, model.default/provider are
seeded only when unset, and a box still on the masquerade is migrated off it
(fingerprint: provider openrouter + base_url pointing at this gateway; a
genuine OpenRouter user never has that, and is left alone).
No secrets: gateway.yaml names the key via key_env, and mcp-config/sync.sh is
what exports it. Hermes needs no context safety margin -- it omits max_tokens
entirely, so vLLM bounds output by whatever context is left.
Also folds in the in-flight hermes bootstrap wiring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
/v1/models returns ONLY model ids, so every model was deployed as {} and
opencode ran all of them on defaults -- no context window (so it cannot manage
truncation), no cost, and no idea whether the model can call tools. opencode
requires this metadata in the models map: it has no /v1/models discovery of its
own, and models.dev only covers known providers, not a private LiteLLM gateway.
sync.sh now also queries /model/info and maps LiteLLM's metadata onto opencode's
per-model spec:
max_input_tokens -> limit.context
max_output_tokens -> limit.output
input/output_cost_per_token -> cost.input/output (per-token -> $ per 1M)
supports_function_calling -> tool_call
supports_reasoning -> reasoning
supports_vision -> attachment
Best-effort and backwards-compatible: /model/info is outside the llm_api_routes
scope most virtual keys have, so an un-updated key 403s and falls back to
names-only -- exactly today's behaviour, no regression.
Note the gateway only reports metadata for models in LiteLLM's registry. The
self-hosted vLLM entries (qwen3-*) carry no model_info in the gateway's
model_list, so they still deploy bare. Fixing that is a gateway-side change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sync runs only at shell start and is throttled to MCP_SYNC_TTL (1h), so a
model added on the gateway can take up to an hour AND a new terminal before
opencode sees it. There was no way to say "fetch it now", so the recurring
symptom was "I added a model and it isn't showing up" -- with a healthy
gateway and a correctly-scoped key.
`mcp-sync` re-sources the hook with the throttle disabled and prints the
resulting model list, so it also answers "what does the client actually have?"
in one step.
README gains a section splitting the two failure modes: a model the gateway
returns but opencode lacks is a stale client; a model the gateway does not
return is a LiteLLM model_list / key-scoping problem.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
~/.zshrc is MANAGED: zsh/install.sh overwrites it wholesale (cp) on every
bootstrap run. install.sh appended the mcp-sync hook to that same file, so
the next bootstrap silently deleted it -- sync.sh then never ran, so
OPENCODE_CONFIG was never exported and opencode came up with no models and
no skills. It looked like a broken gateway or a bad model; it was a wiped
shell hook.
~/.zshrc.local is the unmanaged file the managed zshrc sources last, so the
hook now survives a re-bootstrap. Verified: hook lands in ~/.zshrc.local
(0600), managed ~/.zshrc is left untouched, and OPENCODE_CONFIG still
exports after simulating the bootstrap overwrite.
Also warn instead of failing silently when python3 is missing: sync.sh then
deploys the base config verbatim with an EMPTY models map, which presents as
"the gateway has no models" rather than "a dependency is absent".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every API worth exposing (Gitea, Proxmox) is user-scoped, so hosting MCP
centrally means one shared token per server: everyone acts as the same
identity and attribution is lost. LiteLLM's per-request BYOK cannot cover
all of them either -- it builds its tool->server map from a headerless
background listing, and gitea-mcp hard-exits without a token, so it never
registers. A split brain (some MCPs server-side, some client-side) is worse
than either, so all of them move client-side.
Servers are declared local, spawned via npx with pinned versions, and read
each user's OWN credentials via {env:VAR}. No token is ever written to the
config; the file stays identical for every user.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Models and MCP servers are discovered from the LiteLLM gateway at sync time
(/v1/models, and the aggregate /mcp/ endpoint). Skills have no gateway side to
query: LiteLLM's /v1/skills is a passthrough to Anthropic's Skills API, which
lists skills on an Anthropic account rather than serving skill files, and
/public/skill_hub is a marketplace catalog. So the repo is the source of truth
and the tree is fetched whole, from the repo tarball.
Deploy to ~/.config/mcp-sync/skills rather than ~/.config/opencode/skills so
the atomic replace can never delete a skill the user wrote by hand; opencode
picks the tree up via skills.paths, which expands {env:HOME} — verified — so
the shared config stays identical for every user.
Same throttle and non-fatal discipline as the config sync: a fetch that yields
no skills/ leaves the last-good tree in place instead of emptying it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two changes to the synced opencode config:
- Model provider. opencode can't list models from a custom OpenAI-compatible
provider itself, so sync.sh queries the gateway's /v1/models with the
user's key and materializes the map client-side. New gateway models land
on every box at the next sync with no config edit. If discovery fails the
previous map is carried over, so a gateway hiccup never empties the list.
- Aggregate MCP endpoint. One entry at /mcp/ (trailing slash required — the
bare path 307-redirects and MCP clients don't follow it) exposes every
server the key can access, replacing the per-server gitea entry.
sync.sh also exports OPENCODE_CONFIG: opencode does not auto-load extra
files from its config dir, so without it the synced file was ignored.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
noconnorandClaude Opus 4.8