hermes: wire into dotfiles; fix opencode's generated context limits

Two client-side fixes against the covert-gpt LiteLLM gateway, both traced
from opencode throwing intermittent ContextWindowExceededError 400s.

opencode (mcp-config/sync.sh):
  spec_for() mapped LiteLLM's /model/info onto opencode's schema 1:1, but the
  two mean different things by "output". LiteLLM's max_output_tokens is the
  whole window (vLLM caps TOTAL context, not output separately, which is
  correct for LiteLLM); opencode's limit.output is a RESERVE it subtracts from
  context to pick a compaction point. Copying one to the other told opencode to
  reserve the entire 262144 window, so it targeted an input of context-output
  and the server counted one token more than opencode did -- landing on
  context+1 and a hard 400, every time (measured: 196609 in + 65536 out =
  262145 vs a 262144 window; the retries shrinking max_tokens by 65 never
  helped because the input grew to compensate).

  Capping the reserve alone would NOT have fixed it -- opencode would still sit
  exactly on the boundary, just at a different input size. So: cap the reserve
  (OUTPUT_RESERVE_CAP) and hold a margin back from context
  (CONTEXT_SAFETY_MARGIN). Generated at sync time, so it fixes every gateway
  model at once, including third-party ones.

hermes (hermes/gateway.yaml, hermes/install.sh):
  Hermes reached the gateway by pretending it WAS OpenRouter -- the LiteLLM key
  in OPENROUTER_API_KEY, the gateway in OPENROUTER_BASE_URL. Requests routed
  fine, but fetch_openrouter_models hardcodes openrouter.ai/api/v1/models and
  ignores OPENROUTER_BASE_URL, so the picker listed OpenRouter's ~342-model
  catalog -- models this gateway cannot serve -- while the gateway's own models,
  including cg-1's, never appeared at all.

  Replaced with a named custom provider. install.sh now merges gateway.yaml into
  ~/.hermes/config.yaml, so a fresh box gets a Hermes that can actually talk to
  the gateway instead of one pointed at nothing. The merge is idempotent and
  narrow: only the covert-gpt entry is touched, model.default/provider are
  seeded only when unset, and a box still on the masquerade is migrated off it
  (fingerprint: provider openrouter + base_url pointing at this gateway; a
  genuine OpenRouter user never has that, and is left alone).

  No secrets: gateway.yaml names the key via key_env, and mcp-config/sync.sh is
  what exports it. Hermes needs no context safety margin -- it omits max_tokens
  entirely, so vLLM bounds output by whatever context is left.

Also folds in the in-flight hermes bootstrap wiring.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-07-16 10:38:12 -04:00
co-authored by Claude Opus 4.8
parent fdc3a71236
commit e42807c054
6 changed files with 293 additions and 2 deletions
+17 -2
View File
@@ -134,13 +134,28 @@ except Exception:
# actually reports are emitted; a model LiteLLM knows nothing about (a
# self-hosted vLLM entry with no model_info in the gateway's model_list) simply
# stays {}, which is what every model got before this existed.
# opencode's limit.output is a RESERVE it subtracts from limit.context to decide
# when to compact — NOT a server-side cap. LiteLLM reports max_output_tokens as
# the whole window for self-hosted vLLM (vLLM caps TOTAL context, not output
# separately — see charts/inference/values.yaml model_info), so copying it
# verbatim made opencode reserve the entire window and target an input of
# context - output. Cap the reserve at a generous agent-output budget instead.
OUTPUT_RESERVE_CAP = 32768
# opencode targets input == context - output EXACTLY, and the server counts one
# more token than opencode does, landing at context + 1 and getting a hard 400
# (ContextWindowExceededError). Observed on qwen3-coder-next: 196609 in + 65536
# out = 262145 vs a 262144 window, one token over, every time. Hold a margin
# back so opencode's target can never sit on the boundary.
CONTEXT_SAFETY_MARGIN = 1024
def spec_for(mi):
spec = {}
limit = {}
if mi.get("max_input_tokens"):
limit["context"] = mi["max_input_tokens"]
limit["context"] = max(1, mi["max_input_tokens"] - CONTEXT_SAFETY_MARGIN)
if mi.get("max_output_tokens"):
limit["output"] = mi["max_output_tokens"]
limit["output"] = min(mi["max_output_tokens"], OUTPUT_RESERVE_CAP)
if limit:
spec["limit"] = limit
# LiteLLM prices PER TOKEN; opencode (like models.dev) uses $ per 1M tokens.