From e42807c05460a1e19be19708ff2c3f7d9eb084e1 Mon Sep 17 00:00:00 2001 From: Nic O'Connor Date: Thu, 16 Jul 2026 10:32:37 -0400 Subject: [PATCH] hermes: wire into dotfiles; fix opencode's generated context limits Two client-side fixes against the covert-gpt LiteLLM gateway, both traced from opencode throwing intermittent ContextWindowExceededError 400s. opencode (mcp-config/sync.sh): spec_for() mapped LiteLLM's /model/info onto opencode's schema 1:1, but the two mean different things by "output". LiteLLM's max_output_tokens is the whole window (vLLM caps TOTAL context, not output separately, which is correct for LiteLLM); opencode's limit.output is a RESERVE it subtracts from context to pick a compaction point. Copying one to the other told opencode to reserve the entire 262144 window, so it targeted an input of context-output and the server counted one token more than opencode did -- landing on context+1 and a hard 400, every time (measured: 196609 in + 65536 out = 262145 vs a 262144 window; the retries shrinking max_tokens by 65 never helped because the input grew to compensate). Capping the reserve alone would NOT have fixed it -- opencode would still sit exactly on the boundary, just at a different input size. So: cap the reserve (OUTPUT_RESERVE_CAP) and hold a margin back from context (CONTEXT_SAFETY_MARGIN). Generated at sync time, so it fixes every gateway model at once, including third-party ones. hermes (hermes/gateway.yaml, hermes/install.sh): Hermes reached the gateway by pretending it WAS OpenRouter -- the LiteLLM key in OPENROUTER_API_KEY, the gateway in OPENROUTER_BASE_URL. Requests routed fine, but fetch_openrouter_models hardcodes openrouter.ai/api/v1/models and ignores OPENROUTER_BASE_URL, so the picker listed OpenRouter's ~342-model catalog -- models this gateway cannot serve -- while the gateway's own models, including cg-1's, never appeared at all. Replaced with a named custom provider. install.sh now merges gateway.yaml into ~/.hermes/config.yaml, so a fresh box gets a Hermes that can actually talk to the gateway instead of one pointed at nothing. The merge is idempotent and narrow: only the covert-gpt entry is touched, model.default/provider are seeded only when unset, and a box still on the masquerade is migrated off it (fingerprint: provider openrouter + base_url pointing at this gateway; a genuine OpenRouter user never has that, and is left alone). No secrets: gateway.yaml names the key via key_env, and mcp-config/sync.sh is what exports it. Hermes needs no context safety margin -- it omits max_tokens entirely, so vLLM bounds output by whatever context is left. Also folds in the in-flight hermes bootstrap wiring. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 14 +++ bootstrap.sh | 3 + hermes/gateway.yaml | 55 +++++++++++ hermes/install.sh | 199 ++++++++++++++++++++++++++++++++++++++++ mcp-config/sync.sh | 19 +++- zsh/zshrc.local.example | 5 + 6 files changed, 293 insertions(+), 2 deletions(-) create mode 100644 hermes/gateway.yaml create mode 100755 hermes/install.sh diff --git a/README.md b/README.md index 9e9b5c6..b8d3627 100644 --- a/README.md +++ b/README.md @@ -101,6 +101,20 @@ Variables modules look for: These are NOT used by `ai-k8s-iac` / `task up`: the LiteLLM gateway hosts no MCP servers and stores no MCP credentials. See `mcp-config/README.md`. +- `hermes/` — installs Hermes Agent (Nous Research's self-improving AI agent) + via npm. Hermes uses npx to spawn and downloads the Python hermes-agent from + PyPI. Everything under $HOME, so it survives Kasm redeploys. + + It then merges `hermes/gateway.yaml` into `~/.hermes/config.yaml`, wiring + Hermes to the covert-gpt LiteLLM gateway as a named custom provider. The merge + is idempotent and touches only that provider entry, so the rest of your + personal config is left alone; `model.default` / `model.provider` are seeded + only on a fresh box. Unlike `mcp-config/`, this deploys at BOOTSTRAP time, not + per-shell — re-run `hermes/install.sh` to pick up a gateway.yaml change. + + Depends on `mcp-config/install.sh` having run: gateway.yaml names the key + (`key_env: LITELLM_KEY`) but never contains it, and `mcp-config/sync.sh` is + what exports LITELLM_KEY into your shell. Without it Hermes 401s. Secrets never go in this repo or in `zsh/zshrc` — that file is shared config applied to every box. diff --git a/bootstrap.sh b/bootstrap.sh index 154a51e..1cd4bb2 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -66,6 +66,9 @@ run_module bin # Provides npx, which opencode needs to spawn the npm-based MCP servers in # mcp-config/. Without it every MCP server silently fails to start. run_module node +# Hermes Agent (Nous Research) is distributed as an npm package with a Python +# dependency. Runs after node/ ensures npx is available. +run_module hermes run_module ansible # Runs last: needs rootless Docker already up (started at boot) so it can # delegate cgroups to it. No-ops where that does not apply. diff --git a/hermes/gateway.yaml b/hermes/gateway.yaml new file mode 100644 index 0000000..ab6ebac --- /dev/null +++ b/hermes/gateway.yaml @@ -0,0 +1,55 @@ +# hermes/gateway.yaml — the covert-gpt LiteLLM gateway, as a Hermes custom +# provider. Merged into ~/.hermes/config.yaml by hermes/install.sh. +# +# NO SECRETS, EVER: the key is referenced by NAME via `key_env: LITELLM_KEY`, +# never by value. mcp-config/sync.sh exports LITELLM_KEY into every interactive +# shell from ~/.config/mcp-sync/key (0600), and Hermes — launched from such a +# shell — resolves it at runtime. So this file is safe in a PUBLIC repo, and +# `hermes/install.sh` has no key to prompt for. Run mcp-config/install.sh first; +# without it LITELLM_KEY is unset and Hermes falls back to "no-key-required", +# which the gateway rejects with a 401. +# +# WHY a custom provider instead of `model.provider: openrouter` + a base_url: +# that older shape pointed Hermes at the gateway by pretending it WAS OpenRouter +# (OPENROUTER_API_KEY held the LiteLLM key, OPENROUTER_BASE_URL held the gateway +# URL). Requests reached the gateway, but the MODEL PICKER did not: Hermes' +# fetch_openrouter_models hardcodes openrouter.ai/api/v1/models and ignores +# OPENROUTER_BASE_URL, so the picker listed OpenRouter's ~342-model catalog — +# models this gateway cannot serve — while the gateway's own models never +# appeared. A named custom provider probes THIS base_url instead. +# +# NOTE ON COMMENTS: Hermes rewrites ~/.hermes/config.yaml itself (save_config) +# and does not preserve comments, so none of this survives into the deployed +# file. That is exactly why the reasoning lives HERE, in the repo, rather than +# inline in the generated config. +custom_providers: + - name: covert-gpt + base_url: https://llm.covert-gpt.com/v1 + key_env: LITELLM_KEY + api_mode: chat_completions + # `discover_models` is deliberately left at its default (true): with a key + # present Hermes probes the gateway's /v1/models and REPLACES the picker's + # list with the live catalog, so new gateway models show up with no edit + # here. The `models:` map below is NOT the picker list — it exists purely to + # carry per-model context_length overrides, which are read from config + # independently of discovery. + models: + # These MUST track each node's --max-model-len in ai-k8s-iac + # (charts/inference/values.yaml > vllm.nodes[].extraArgs). They are a + # separate copy of that number; there is no single source across repos. + # + # Unlike opencode, Hermes needs NO safety margin here. opencode reserves + # `limit.output` and targets an input of exactly context - output, landing + # one token over the window (a hard 400); Hermes omits max_tokens entirely + # unless model.max_tokens is set, so vLLM just bounds output by whatever + # context remains. Do NOT add model.max_tokens to work around anything: it + # is GLOBAL, would apply to every provider (including Anthropic), and only + # shrinks the effective input budget Hermes compacts against. + qwen3-vl-30b-a3b: # cg-1 — multimodal, tool calling + context_length: 262144 + supports_vision: true + qwen3-coder-next: # cg-2 — agentic coder + context_length: 262144 + # Models the gateway proxies to third parties (kimi, glm, …) are + # intentionally absent: discovery still lists them, and inventing a + # context_length we have not verified is worse than letting Hermes probe. diff --git a/hermes/install.sh b/hermes/install.sh new file mode 100755 index 0000000..b4ec858 --- /dev/null +++ b/hermes/install.sh @@ -0,0 +1,199 @@ +#!/usr/bin/env sh +# hermes/install.sh — install Hermes Agent by Nous Research, and point it at +# the covert-gpt LiteLLM gateway. +# +# WHY: Hermes requires Python 3.11+ and creates a self-contained venv at +# ~/.hermes. The official installer handles the complexity of managing its +# own uv-based Python environment. This module simply runs the official +# installer with --skip-setup so it completes in non-interactive mode. +# Everything under $HOME, survives Kasm redeploys. +# +# It then merges hermes/gateway.yaml into ~/.hermes/config.yaml, so a fresh box +# gets a Hermes that can actually talk to the gateway. Without that step Hermes +# installs and then points at nothing — the asymmetry mcp-config/ does not have, +# since opencode rebuilds its config from Gitea on every shell via sync.sh. +# +# No secrets are handled here: gateway.yaml references the key by NAME +# (key_env: LITELLM_KEY) and mcp-config/sync.sh exports it. Run +# mcp-config/install.sh first. +set -eu + +SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)" +REPO_ROOT="$(dirname -- "$SCRIPT_DIR")" + +# shellcheck source=../lib/fetch.sh +. "$REPO_ROOT/lib/fetch.sh" + +say() { printf '%s\n' "$*" >&2; } + +# Hermes Agent version (synced with PyPI/npm). +HERMES_VERSION="v0.18.2" + +# --- 1. the binary -------------------------------------------------------- +# Check if hermes is already installed at the correct version using the +# .versions stamp pattern from bin/install.sh. NOTE: this only skips the +# DOWNLOAD — the config merge below still runs, so a config fix ships to an +# already-installed box on the next bootstrap. +if [ -x "$HOME/.local/bin/hermes" ] && [ -f "$HOME/.local/bin/.versions/hermes" ] && [ "$(cat "$HOME/.local/bin/.versions/hermes")" = "$HERMES_VERSION" ]; then + say "hermes-agent $HERMES_VERSION already installed." +else + say "Installing hermes-agent via official installer..." + # Run the official installer in non-interactive mode, skipping the setup + # wizard. The installer creates ~/.hermes/venvs/hermes with its own + # uv-managed Python. + curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash -s -- --non-interactive --skip-setup + + # Verify installation by checking the binary exists AND produces expected output. + if command -v hermes >/dev/null 2>&1; then + mkdir -p "$HOME/.local/bin/.versions" + printf '%s\n' "$HERMES_VERSION" >"$HOME/.local/bin/.versions/hermes" + say "Installed hermes-agent $HERMES_VERSION successfully." + else + say "ERROR: hermes CLI not found after install" + exit 1 + fi +fi + +# --- 2. point it at the gateway ------------------------------------------- +# Merge hermes/gateway.yaml into ~/.hermes/config.yaml. Idempotent, and +# deliberately NON-DESTRUCTIVE: config.yaml is a large personal file +# (personalities, display prefs, toolsets) that Hermes also rewrites itself, so +# this replaces ONLY the covert-gpt custom_providers entry and never rewrites +# the rest. model.default / model.provider are seeded only when unset, so a +# deliberate model choice is preserved across bootstraps. +# +# PyYAML is needed to merge. Prefer the system python3; fall back to Hermes' own +# venv, which always has it. If neither works the config step is skipped with a +# warning rather than failing bootstrap — the binary is still installed, and a +# missing provider is a far smaller problem than a broken box. +GATEWAY_YAML="$SCRIPT_DIR/gateway.yaml" +HERMES_CONFIG="$HOME/.hermes/config.yaml" + +PY="" +for _c in python3 "$HOME/.hermes/venvs/hermes/bin/python" "$HOME/.hermes/hermes-agent/venv/bin/python"; do + if command -v "$_c" >/dev/null 2>&1 && "$_c" -c 'import yaml' >/dev/null 2>&1; then PY="$_c"; break; fi +done + +if [ -z "$PY" ]; then + say "WARNING: no python3 with PyYAML found — skipping Hermes gateway config." + say " Install PyYAML and re-run $0 to finish wiring the gateway." +elif [ ! -f "$GATEWAY_YAML" ]; then + say "WARNING: $GATEWAY_YAML missing — skipping Hermes gateway config." +else + HERMES_GATEWAY_YAML="$GATEWAY_YAML" HERMES_CONFIG_PATH="$HERMES_CONFIG" "$PY" - <<'PYEOF' +import os, sys, tempfile +import yaml + +gw_path = os.environ["HERMES_GATEWAY_YAML"] +cfg_path = os.environ["HERMES_CONFIG_PATH"] + +with open(gw_path) as f: + gw = yaml.safe_load(f) or {} +entries = gw.get("custom_providers") or [] +if not entries: + print("hermes: gateway.yaml has no custom_providers — nothing to do", file=sys.stderr) + raise SystemExit(0) + +cfg = {} +if os.path.exists(cfg_path): + try: + with open(cfg_path) as f: + cfg = yaml.safe_load(f) or {} + except Exception as e: + # A corrupt config is the user's to fix; never overwrite it blind. + print(f"hermes: {cfg_path} is not valid YAML ({e}) — skipping", file=sys.stderr) + raise SystemExit(0) +if not isinstance(cfg, dict): + print(f"hermes: {cfg_path} is not a mapping — skipping", file=sys.stderr) + raise SystemExit(0) + +existing = cfg.get("custom_providers") +if not isinstance(existing, list): + existing = [] + +changed = [] +for entry in entries: + name = entry.get("name") + if not name: + continue + for i, cur in enumerate(existing): + if isinstance(cur, dict) and cur.get("name") == name: + if cur != entry: + existing[i] = entry + changed.append(f"updated custom_providers[{name}]") + break + else: + existing.append(entry) + changed.append(f"added custom_providers[{name}]") +cfg["custom_providers"] = existing + +# `default` is a BARE model id and `provider` is the custom: form; both +# are the user's to change, so they are seeded on a fresh box and otherwise left +# alone — EXCEPT to migrate the old masquerade, below. +primary = entries[0]["name"] +model = cfg.get("model") +if not isinstance(model, dict): + model = {} + +gw_url = (entries[0].get("base_url") or "").rstrip("/") +cur_provider = (model.get("provider") or "").strip() +cur_base = (model.get("base_url") or "").rstrip("/") + +# MIGRATE the old masquerade: `provider: openrouter` + `base_url: `. That shape routed requests here but left the picker listing +# openrouter.ai's catalog, so leaving it in place would silently preserve the +# exact bug this module exists to fix. `base_url` pointing at OUR gateway is the +# fingerprint — a genuine OpenRouter user never has that. Any OTHER provider is +# a deliberate choice and is never touched. +if gw_url and cur_base == gw_url and cur_provider in ("", "openrouter", "custom"): + if cur_provider != f"custom:{primary}": + model["provider"] = f"custom:{primary}" + changed.append( + f"migrated model.provider {cur_provider or ''} -> custom:{primary}" + ) +elif not cur_provider: + model["provider"] = f"custom:{primary}" + changed.append(f"set model.provider=custom:{primary}") + +if not model.get("default"): + first_model = next(iter((entries[0].get("models") or {})), None) + if first_model: + model["default"] = first_model + changed.append(f"set model.default={first_model}") + +# model.base_url is ignored once a named custom provider supplies it, and a +# stale value here is an active footgun — drop it if it points at this gateway. +if gw_url and cur_base == gw_url: + model.pop("base_url", None) + changed.append("removed dead model.base_url") +cfg["model"] = model + +if not changed: + print("hermes: gateway config already current", file=sys.stderr) + raise SystemExit(0) + +d = os.path.dirname(cfg_path) or "." +os.makedirs(d, exist_ok=True) +fd, tmp = tempfile.mkstemp(dir=d) +with os.fdopen(fd, "w") as f: + yaml.safe_dump(cfg, f, sort_keys=False, default_flow_style=False, allow_unicode=True) +os.chmod(tmp, 0o600) +os.replace(tmp, cfg_path) +for c in changed: + print(f"hermes: {c}", file=sys.stderr) +PYEOF + + # The provider is useless without the key. mcp-config/sync.sh exports it; warn + # rather than fail, since bootstrap ordering may put mcp-config after this. + if [ -z "${LITELLM_KEY:-}" ] && [ ! -f "$HOME/.config/mcp-sync/key" ]; then + say "NOTE: LITELLM_KEY is unset and ~/.config/mcp-sync/key is missing." + say " Run mcp-config/install.sh, or Hermes will 401 against the gateway." + fi + # The old OpenRouter masquerade actively breaks the model picker (it lists + # openrouter.ai's catalog, not the gateway's), so flag it if it lingers. + if [ -n "${OPENROUTER_API_KEY:-}" ] && [ "${OPENROUTER_API_KEY:-}" = "${LITELLM_KEY:-x}" ]; then + say "NOTE: OPENROUTER_API_KEY is set to the LiteLLM key in your environment." + say " That is the old masquerade; unset it in ~/.hermes/.env so the" + say " model picker shows the gateway's models instead of openrouter.ai's." + fi +fi diff --git a/mcp-config/sync.sh b/mcp-config/sync.sh index 81c6b45..6fddfe3 100644 --- a/mcp-config/sync.sh +++ b/mcp-config/sync.sh @@ -134,13 +134,28 @@ except Exception: # actually reports are emitted; a model LiteLLM knows nothing about (a # self-hosted vLLM entry with no model_info in the gateway's model_list) simply # stays {}, which is what every model got before this existed. +# opencode's limit.output is a RESERVE it subtracts from limit.context to decide +# when to compact — NOT a server-side cap. LiteLLM reports max_output_tokens as +# the whole window for self-hosted vLLM (vLLM caps TOTAL context, not output +# separately — see charts/inference/values.yaml model_info), so copying it +# verbatim made opencode reserve the entire window and target an input of +# context - output. Cap the reserve at a generous agent-output budget instead. +OUTPUT_RESERVE_CAP = 32768 +# opencode targets input == context - output EXACTLY, and the server counts one +# more token than opencode does, landing at context + 1 and getting a hard 400 +# (ContextWindowExceededError). Observed on qwen3-coder-next: 196609 in + 65536 +# out = 262145 vs a 262144 window, one token over, every time. Hold a margin +# back so opencode's target can never sit on the boundary. +CONTEXT_SAFETY_MARGIN = 1024 + + def spec_for(mi): spec = {} limit = {} if mi.get("max_input_tokens"): - limit["context"] = mi["max_input_tokens"] + limit["context"] = max(1, mi["max_input_tokens"] - CONTEXT_SAFETY_MARGIN) if mi.get("max_output_tokens"): - limit["output"] = mi["max_output_tokens"] + limit["output"] = min(mi["max_output_tokens"], OUTPUT_RESERVE_CAP) if limit: spec["limit"] = limit # LiteLLM prices PER TOKEN; opencode (like models.dev) uses $ per 1M tokens. diff --git a/zsh/zshrc.local.example b/zsh/zshrc.local.example index 552768c..7214e8e 100644 --- a/zsh/zshrc.local.example +++ b/zsh/zshrc.local.example @@ -16,6 +16,11 @@ # setting it here. Only set it here on a box without mcp-sync: # export LITELLM_KEY=sk-... # +# Hermes Agent uses the same key and endpoint. The base_url is configured +# in ~/.hermes/config.yaml to point to your covert-gpt gateway +# (https://llm.covert-gpt.com). Run hermes setup or update config.yaml +# to set the default model (see ~/.hermes/config.yaml for available models). + # To develop mcp-config/ or skills/ against a working checkout instead of # Gitea's main, point the sync at it and source it directly: # MCP_SYNC_RAW_URL="file://$HOME/source/dotfiles/mcp-config/opencode.gateway.json"