noconnorandClaude Opus 5
135112eee5
Folds in the in-flight local-checkout work and fixes the context accounting
on top of it. Both halves are the same bug seen from two ends: clients were
sending requests larger than the backend's window and getting hard
ContextWindowExceededError 400s, intermittently, in a way that looked like
the model or the GPU misbehaving.
Already in flight here (unchanged in intent):
- sync.sh prefers the LOCAL CHECKOUT over re-downloading from Gitea, and
install.sh records the checkout path so it can be found. That recording
is load-bearing: without it sync.sh cannot find the repo, silently falls
back to Gitea, and every local edit to opencode.gateway.json is ignored.
Observed on this box — ~/.config/mcp-sync/source did not exist, the
default ~/.dotfiles is not where the checkout lives, and the Gitea base
still served models:{}.
- the hermes custom-provider wiring.
Fixed on top:
opencode had NO context limit on ANY gateway model, not a wrong one. Two
faults compounded. /model/info 403s for a day-to-day virtual key (only
management-scoped keys may read it), which sync.sh treats as best-effort and
falls back to names-only — as designed and as commented. But the merge then
did `prov["models"] = models`, replacing the map wholesale, so limits
declared in the base config were discarded too and there was no floor left.
Base specs are now a floor that live /model/info overrides field by field:
discovery still wins where present, so this adds no drift, and a 403 no
longer strips the limits. Verified: ornith 195584/32768, the qwens
261120/32768.
hermes' model map named `qwen3-vl-30b-a3b` at 262144 — a model the gateway
stopped serving when Ornith replaced it — while ornith-1.0-35b, the model
actually on cg-1 AND the only one with a window smaller than its neighbours
(196608), had no entry at all and inherited Hermes' default. The one model
needing a lower ceiling was the one not declaring one, which is exactly the
shape that fails only once a session grows.
NOTE for other boxes: hermes reads ~/.hermes/config.yaml, which install.sh
produces by MERGING gateway.yaml. Editing gateway.yaml alone changes nothing
until install.sh re-runs — this machine's live config was patched by hand.
Not addressed here, deliberately: the gateway-side context_window_fallback
(ai-k8s-iac). It is wired but inert while cg-2 is scaled down, so preventing
the oversized request client-side is what actually works today.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FoDK4wwB1Twty5gwiriYLf
72 lines
4.3 KiB
YAML
72 lines
4.3 KiB
YAML
# hermes/gateway.yaml — the covert-gpt LiteLLM gateway, as a Hermes custom
|
|
# provider. Merged into ~/.hermes/config.yaml by hermes/install.sh.
|
|
#
|
|
# NO SECRETS, EVER: the key is referenced by NAME via `key_env: LITELLM_KEY`,
|
|
# never by value. mcp-config/sync.sh exports LITELLM_KEY into every interactive
|
|
# shell from ~/.config/mcp-sync/key (0600), and Hermes — launched from such a
|
|
# shell — resolves it at runtime. So this file is safe in a PUBLIC repo, and
|
|
# `hermes/install.sh` has no key to prompt for. Run mcp-config/install.sh first;
|
|
# without it LITELLM_KEY is unset and Hermes falls back to "no-key-required",
|
|
# which the gateway rejects with a 401.
|
|
#
|
|
# WHY a custom provider instead of `model.provider: openrouter` + a base_url:
|
|
# that older shape pointed Hermes at the gateway by pretending it WAS OpenRouter
|
|
# (OPENROUTER_API_KEY held the LiteLLM key, OPENROUTER_BASE_URL held the gateway
|
|
# URL). Requests reached the gateway, but the MODEL PICKER did not: Hermes'
|
|
# fetch_openrouter_models hardcodes openrouter.ai/api/v1/models and ignores
|
|
# OPENROUTER_BASE_URL, so the picker listed OpenRouter's ~342-model catalog —
|
|
# models this gateway cannot serve — while the gateway's own models never
|
|
# appeared. A named custom provider probes THIS base_url instead.
|
|
#
|
|
# NOTE ON COMMENTS: Hermes rewrites ~/.hermes/config.yaml itself (save_config)
|
|
# and does not preserve comments, so none of this survives into the deployed
|
|
# file. That is exactly why the reasoning lives HERE, in the repo, rather than
|
|
# inline in the generated config.
|
|
custom_providers:
|
|
- name: covert-gpt
|
|
base_url: https://llm.covert-gpt.com/v1
|
|
key_env: LITELLM_KEY
|
|
api_mode: chat_completions
|
|
# `discover_models` is deliberately left at its default (true): with a key
|
|
# present Hermes probes the gateway's /v1/models and REPLACES the picker's
|
|
# list with the live catalog, so new gateway models show up with no edit
|
|
# here. The `models:` map below is NOT the picker list — it exists purely to
|
|
# carry per-model context_length overrides, which are read from config
|
|
# independently of discovery.
|
|
models:
|
|
# These MUST track each node's --max-model-len in ai-k8s-iac
|
|
# (charts/inference/values.yaml > vllm.nodes[].extraArgs). They are a
|
|
# separate copy of that number; there is no single source across repos.
|
|
#
|
|
# Unlike opencode, Hermes needs NO safety margin here. opencode reserves
|
|
# `limit.output` and targets an input of exactly context - output, landing
|
|
# one token over the window (a hard 400); Hermes omits max_tokens entirely
|
|
# unless model.max_tokens is set, so vLLM just bounds output by whatever
|
|
# context remains. Do NOT add model.max_tokens to work around anything: it
|
|
# is GLOBAL, would apply to every provider (including Anthropic), and only
|
|
# shrinks the effective input budget Hermes compacts against.
|
|
# CORRECTED 2026-09-11. This map had drifted badly and the drift was
|
|
# not visible from inside Hermes: it listed `qwen3-vl-30b-a3b` at
|
|
# 262144, a model the gateway stopped serving when Ornith replaced it,
|
|
# while Ornith — the model actually on cg-1, and a SMALLER window at
|
|
# 196608 — had no entry at all. A model with no entry gets whatever
|
|
# default Hermes carries, so the one model on this gateway that needs a
|
|
# lower ceiling than its neighbours was the one model not declaring one.
|
|
# That is the right shape to produce exactly the reported symptom: fine
|
|
# on short turns, hard 400 once a task grows.
|
|
ornith-1.0-35b: # cg-1 — reasoning + tool calling, multimodal
|
|
# 196608, NOT 262144. Lower than the others on purpose — see
|
|
# vllm.nodes[cg1] --max-model-len.
|
|
context_length: 196608
|
|
supports_vision: true
|
|
qwen3.8-27b: # cg-2 — general, NVFP4
|
|
context_length: 262144
|
|
qwen3-coder-next: # cg-2 — agentic coder
|
|
# Shares cg-2 with qwen3.8-27b; only one is enabled at a time, so at
|
|
# most one of these two is ever live. Both are listed because which
|
|
# one it is changes without touching this file.
|
|
context_length: 262144
|
|
# Models the gateway proxies to third parties (kimi, glm, …) are
|
|
# intentionally absent: discovery still lists them, and inventing a
|
|
# context_length we have not verified is worse than letting Hermes probe.
|