noconnorandClaude Opus 5
d42d0b29ff
OpenAI's /v1/models schema carries no context window, so a client that only
lists models cannot know how large a request it may send, and oversizing one
is a hard ContextWindowExceededError 400 — vLLM counts prompt + max_tokens
against its window. opencode already read the window from /v1/model/info.
Hermes and pi never consult that endpoint: they take it from static config,
which was hand-copied and drifted. Hermes named a model the gateway had
stopped serving while its replacement (the one model with a SMALLER window)
had no entry; pi's reserve was half its maxTokens. Both fail only once a
session grows, which is why it looked like the model or the GPU.
This renders one /v1/model/info response into all three clients. It is not a
plugin per harness and needs no upstream change — it extends the renderer that
already translates the MCP servers into Hermes' schema, with the same
manifest discipline.
SAME SOURCE, THREE DIFFERENT NUMBERS, deliberately — each client means
something different by "context":
opencode limit.context = window - 1024 reserves limit.output and targets
context - output, so needs headroom
Hermes context_length = window omits max_tokens entirely; vLLM
bounds output by what is left
pi maxTokens <= reserveTokens-2048 compacts at window - reserveTokens
(compaction.js shouldCompact) then
requests maxTokens
pi's maxTokens is derived FROM its reserve rather than the reserve being raised
to fit: reserveTokens is one global setting shared by every provider pi knows,
so raising it for this gateway would shrink everyone's usable context.
Safety properties, each tested in a sandbox against copies of the real
configs before touching live ones (11 cases):
- Ownership is per FIELD. A value is retracted only if this set it and the
gateway stopped declaring it. Hand-added models and fields are never
touched; a model removed upstream loses only what was rendered.
- Vision is retracted, not just added. The chart states that supports_vision
is what makes clients send image parts, so if a backend falls back to
language-only, the `image` input this added is taken back — but a user's own
`image` on an entry is left alone.
- Raw /v1/model/info is cached on each successful fetch, because refresh is
throttled and a throttled shell never fetches. A bad fetch (403 body, HTML
error page, empty list, rows without model_name) never replaces a good cache.
- Never invents a window: claude-* declares none and is skipped. A gateway
that suddenly declares no windows is treated as broken, not empty.
- One output cap (MCP_SYNC_OUTPUT_RESERVE_CAP) is read by opencode and pi, so
the clients cannot drift apart on it. Parses under dash as well as bash,
since install.sh sources this under /bin/sh.
First live run found a bug nobody knew about: pi's Claude entries carried
maxTokens 64000 against a 34816 reserve, so claude-haiku-4-5 sent
(200000 - 34816) + 64000 = 229184 tokens at a 200000 window — still broken
after the earlier manual reserve fix, which only covered 32768. Now 32768, and
every served model satisfies the invariant. It also gave Hermes limits for the
Claude routes it had none for.
hermes/gateway.yaml: the static map is marked as a first-boot seed that the
next sync overwrites, so nobody hand-edits it believing it is authoritative.
The source of truth is model_info.max_input_tokens in ai-k8s-iac's
charts/inference/values.yaml. README documents all of the above.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FoDK4wwB1Twty5gwiriYLf
78 lines
4.7 KiB
YAML
78 lines
4.7 KiB
YAML
# hermes/gateway.yaml — the covert-gpt LiteLLM gateway, as a Hermes custom
|
|
# provider. Merged into ~/.hermes/config.yaml by hermes/install.sh.
|
|
#
|
|
# NO SECRETS, EVER: the key is referenced by NAME via `key_env: LITELLM_KEY`,
|
|
# never by value. mcp-config/sync.sh exports LITELLM_KEY into every interactive
|
|
# shell from ~/.config/mcp-sync/key (0600), and Hermes — launched from such a
|
|
# shell — resolves it at runtime. So this file is safe in a PUBLIC repo, and
|
|
# `hermes/install.sh` has no key to prompt for. Run mcp-config/install.sh first;
|
|
# without it LITELLM_KEY is unset and Hermes falls back to "no-key-required",
|
|
# which the gateway rejects with a 401.
|
|
#
|
|
# WHY a custom provider instead of `model.provider: openrouter` + a base_url:
|
|
# that older shape pointed Hermes at the gateway by pretending it WAS OpenRouter
|
|
# (OPENROUTER_API_KEY held the LiteLLM key, OPENROUTER_BASE_URL held the gateway
|
|
# URL). Requests reached the gateway, but the MODEL PICKER did not: Hermes'
|
|
# fetch_openrouter_models hardcodes openrouter.ai/api/v1/models and ignores
|
|
# OPENROUTER_BASE_URL, so the picker listed OpenRouter's ~342-model catalog —
|
|
# models this gateway cannot serve — while the gateway's own models never
|
|
# appeared. A named custom provider probes THIS base_url instead.
|
|
#
|
|
# NOTE ON COMMENTS: Hermes rewrites ~/.hermes/config.yaml itself (save_config)
|
|
# and does not preserve comments, so none of this survives into the deployed
|
|
# file. That is exactly why the reasoning lives HERE, in the repo, rather than
|
|
# inline in the generated config.
|
|
custom_providers:
|
|
- name: covert-gpt
|
|
base_url: https://llm.covert-gpt.com/v1
|
|
key_env: LITELLM_KEY
|
|
api_mode: chat_completions
|
|
# `discover_models` is deliberately left at its default (true): with a key
|
|
# present Hermes probes the gateway's /v1/models and REPLACES the picker's
|
|
# list with the live catalog, so new gateway models show up with no edit
|
|
# here. The `models:` map below is NOT the picker list — it exists purely to
|
|
# carry per-model context_length overrides, which are read from config
|
|
# independently of discovery.
|
|
models:
|
|
# NO LONGER HAND-MAINTAINED. mcp-config/sync.sh renders this map from the
|
|
# gateway's /v1/model/info on every sync (_mcp_sync_hermes_models), so the
|
|
# single source of truth is `model_info.max_input_tokens` in ai-k8s-iac's
|
|
# charts/inference/values.yaml. Fix a wrong window THERE, not here: an
|
|
# edit to these entries is overwritten by the next sync.
|
|
#
|
|
# What follows is only a first-boot seed, so a brand-new box has sane
|
|
# limits before its first sync. Anything you add that the gateway does
|
|
# not serve is left alone — the renderer only touches fields it set.
|
|
#
|
|
# Unlike opencode, Hermes needs NO safety margin here. opencode reserves
|
|
# `limit.output` and targets an input of exactly context - output, landing
|
|
# one token over the window (a hard 400); Hermes omits max_tokens entirely
|
|
# unless model.max_tokens is set, so vLLM just bounds output by whatever
|
|
# context remains. Do NOT add model.max_tokens to work around anything: it
|
|
# is GLOBAL, would apply to every provider (including Anthropic), and only
|
|
# shrinks the effective input budget Hermes compacts against.
|
|
# CORRECTED 2026-09-11. This map had drifted badly and the drift was
|
|
# not visible from inside Hermes: it listed `qwen3-vl-30b-a3b` at
|
|
# 262144, a model the gateway stopped serving when Ornith replaced it,
|
|
# while Ornith — the model actually on cg-1, and a SMALLER window at
|
|
# 196608 — had no entry at all. A model with no entry gets whatever
|
|
# default Hermes carries, so the one model on this gateway that needs a
|
|
# lower ceiling than its neighbours was the one model not declaring one.
|
|
# That is the right shape to produce exactly the reported symptom: fine
|
|
# on short turns, hard 400 once a task grows.
|
|
ornith-1.0-35b: # cg-1 — reasoning + tool calling, multimodal
|
|
# 196608, NOT 262144. Lower than the others on purpose — see
|
|
# vllm.nodes[cg1] --max-model-len.
|
|
context_length: 196608
|
|
supports_vision: true
|
|
qwen3.8-27b: # cg-2 — general, NVFP4
|
|
context_length: 262144
|
|
qwen3-coder-next: # cg-2 — agentic coder
|
|
# Shares cg-2 with qwen3.8-27b; only one is enabled at a time, so at
|
|
# most one of these two is ever live. Both are listed because which
|
|
# one it is changes without touching this file.
|
|
context_length: 262144
|
|
# Models the gateway proxies to third parties (kimi, glm, …) are
|
|
# intentionally absent: discovery still lists them, and inventing a
|
|
# context_length we have not verified is worse than letting Hermes probe.
|