public/dotfiles
2
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

mcp-config: render gateway context limits into Hermes and pi, not just opencode

Browse Source 
OpenAI's /v1/models schema carries no context window, so a client that only
lists models cannot know how large a request it may send, and oversizing one
is a hard ContextWindowExceededError 400 — vLLM counts prompt + max_tokens
against its window. opencode already read the window from /v1/model/info.
Hermes and pi never consult that endpoint: they take it from static config,
which was hand-copied and drifted. Hermes named a model the gateway had
stopped serving while its replacement (the one model with a SMALLER window)
had no entry; pi's reserve was half its maxTokens. Both fail only once a
session grows, which is why it looked like the model or the GPU.

This renders one /v1/model/info response into all three clients. It is not a
plugin per harness and needs no upstream change — it extends the renderer that
already translates the MCP servers into Hermes' schema, with the same
manifest discipline.

SAME SOURCE, THREE DIFFERENT NUMBERS, deliberately — each client means
something different by "context":

  opencode  limit.context = window - 1024   reserves limit.output and targets
                                            context - output, so needs headroom
  Hermes    context_length = window        omits max_tokens entirely; vLLM
                                            bounds output by what is left
  pi        maxTokens <= reserveTokens-2048 compacts at window - reserveTokens
                                            (compaction.js shouldCompact) then
                                            requests maxTokens

pi's maxTokens is derived FROM its reserve rather than the reserve being raised
to fit: reserveTokens is one global setting shared by every provider pi knows,
so raising it for this gateway would shrink everyone's usable context.

Safety properties, each tested in a sandbox against copies of the real
configs before touching live ones (11 cases):

- Ownership is per FIELD. A value is retracted only if this set it and the
  gateway stopped declaring it. Hand-added models and fields are never
  touched; a model removed upstream loses only what was rendered.
- Vision is retracted, not just added. The chart states that supports_vision
  is what makes clients send image parts, so if a backend falls back to
  language-only, the `image` input this added is taken back — but a user's own
  `image` on an entry is left alone.
- Raw /v1/model/info is cached on each successful fetch, because refresh is
  throttled and a throttled shell never fetches. A bad fetch (403 body, HTML
  error page, empty list, rows without model_name) never replaces a good cache.
- Never invents a window: claude-* declares none and is skipped. A gateway
  that suddenly declares no windows is treated as broken, not empty.
- One output cap (MCP_SYNC_OUTPUT_RESERVE_CAP) is read by opencode and pi, so
  the clients cannot drift apart on it. Parses under dash as well as bash,
  since install.sh sources this under /bin/sh.

First live run found a bug nobody knew about: pi's Claude entries carried
maxTokens 64000 against a 34816 reserve, so claude-haiku-4-5 sent
(200000 - 34816) + 64000 = 229184 tokens at a 200000 window — still broken
after the earlier manual reserve fix, which only covered 32768. Now 32768, and
every served model satisfies the invariant. It also gave Hermes limits for the
Claude routes it had none for.

hermes/gateway.yaml: the static map is marked as a first-boot seed that the
next sync overwrites, so nobody hand-edits it believing it is authoritative.
The source of truth is model_info.max_input_tokens in ai-k8s-iac's
charts/inference/values.yaml. README documents all of the above.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FoDK4wwB1Twty5gwiriYLf
This commit is contained in:
noconnor
2026-09-12 18:50:02 -04:00
co-authored by Claude Opus 5
parent 9519cbe609
commit d42d0b29ff
3 changed files with 401 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mcp-config/sync.sh
+345 -1
View File
@@ -76,6 +76,21 @@ MCP_SYNC_HERMES_CONFIG="${MCP_SYNC_HERMES_CONFIG-$HOME/.hermes/config.yaml}"
# Names we rendered last time, so a server REMOVED upstream can be removed here
# without touching MCP servers the user added to Hermes by hand.
MCP_SYNC_HERMES_MANIFEST="${MCP_SYNC_HERMES_MANIFEST:-$MCP_SYNC_HOME/hermes-managed.json}"
# Model limits for the non-opencode clients. The gateway's /v1/model/info is
# cached here on every successful fetch, because the refresh above is throttled
# and a throttled shell never fetches — the renderers must still have something
# to render from.
MCP_SYNC_MODELINFO_CACHE="${MCP_SYNC_MODELINFO_CACHE:-$MCP_SYNC_HOME/model-info.json}"
# Per-model FIELDS we set in Hermes last time, so a field can be retracted
# without touching fields the user set by hand. Separate from the MCP manifest.
MCP_SYNC_HERMES_MODELS_MANIFEST="${MCP_SYNC_HERMES_MODELS_MANIFEST:-$MCP_SYNC_HOME/hermes-models-managed.json}"
# pi. Set MCP_SYNC_PI_MODELS to empty to opt out entirely.
MCP_SYNC_PI_MODELS="${MCP_SYNC_PI_MODELS-$HOME/.pi/agent/models.json}"
MCP_SYNC_PI_SETTINGS="${MCP_SYNC_PI_SETTINGS:-$HOME/.pi/agent/settings.json}"
MCP_SYNC_PI_MANIFEST="${MCP_SYNC_PI_MANIFEST:-$MCP_SYNC_HOME/pi-managed.json}"
# The most output any client is told it may reserve/request. ONE value, read by
# the opencode, Hermes and pi renderers alike, so they cannot drift apart on it.
MCP_SYNC_OUTPUT_RESERVE_CAP="${MCP_SYNC_OUTPUT_RESERVE_CAP:-32768}"
# Throttle: only re-sync if the deployed copy is older than this many seconds
# (default 1h). Keeps every new shell from hammering Gitea/the gateway.
MCP_SYNC_TTL="${MCP_SYNC_TTL:-3600}"
@@ -229,6 +244,33 @@ _mcp_sync_refresh() {
unset _mi_path
fi
# Cache it for the Hermes and pi renderers — but only a response that really
# is model info. curl -f already rejects a 403, yet a 200 carrying an error
# page or an empty list is possible, and caching that would strip every
# client's limits until the next good fetch. A bad fetch leaves the last good
# cache in place.
if [ -n "$_info" ] && command -v python3 >/dev/null 2>&1; then
MCP_SYNC_PY_INFO="$_info" MCP_SYNC_PY_CACHE="$MCP_SYNC_MODELINFO_CACHE" \
python3 - <<'PYEOF' 2>/dev/null || true
import json, os, tempfile
try:
data = json.loads(os.environ["MCP_SYNC_PY_INFO"])
rows = data.get("data")
except Exception:
raise SystemExit(0)
if not isinstance(rows, list) or not any(
isinstance(r, dict) and r.get("model_name") for r in rows):
raise SystemExit(0)
cache = os.environ["MCP_SYNC_PY_CACHE"]
d = os.path.dirname(cache) or "."
os.makedirs(d, exist_ok=True)
fd, tmp = tempfile.mkstemp(dir=d)
with os.fdopen(fd, "w") as f:
json.dump({"data": rows}, f)
os.replace(tmp, cache)
PYEOF
fi
# 3. Merge and deploy. python3 validates the base (don't clobber good config
# with an error page), injects the discovered models, and — when discovery
# failed — carries over the previous deployed models map so a gateway
@@ -237,6 +279,7 @@ _mcp_sync_refresh() {
if command -v python3 >/dev/null 2>&1; then
MCP_SYNC_BASE="$_base" MCP_SYNC_MODELS="$_models" MCP_SYNC_MODELINFO="$_info" \
MCP_SYNC_PY_DEST="$MCP_SYNC_DEST" MCP_SYNC_PY_PROVIDER="$MCP_SYNC_PROVIDER" \
MCP_SYNC_PY_OUTPUT_CAP="$MCP_SYNC_OUTPUT_RESERVE_CAP" \
python3 - <<'PYEOF' 2>/dev/null || true
import json, os, tempfile
@@ -263,7 +306,7 @@ except Exception:
# separately — see charts/inference/values.yaml model_info), so copying it
# verbatim made opencode reserve the entire window and target an input of
# context - output. Cap the reserve at a generous agent-output budget instead.
OUTPUT_RESERVE_CAP = 32768
OUTPUT_RESERVE_CAP = int(os.environ.get("MCP_SYNC_PY_OUTPUT_CAP") or 32768)
# opencode targets input == context - output EXACTLY, and the server counts one
# more token than opencode does, landing at context + 1 and getting a hard 400
# (ContextWindowExceededError). Observed on qwen3-coder-next: 196609 in + 65536
@@ -505,6 +548,302 @@ os.replace(tmp, manifest_path)
PYEOF
}
# --- render the gateway's MODEL LIMITS into Hermes' config (non-fatal) ------
#
# WHY THIS EXISTS: OpenAI's /v1/models schema carries no context window, so a
# client that only lists models cannot know how big a request it may send.
# opencode gets the window from /v1/model/info above. Hermes does not consult
# that endpoint at all — it takes context_length from a static `models:` map
# under its custom provider — so the number used to be hand-copied into
# hermes/gateway.yaml and kept "in sync" by nobody. It drifted: the map named a
# model the gateway had stopped serving, while the model that replaced it (and
# the only one with a SMALLER window than its neighbours) had no entry at all.
# Short turns were fine; long tasks hit ContextWindowExceededError 400s.
#
# NO MARGIN, deliberately, and this is the one number that differs from the
# opencode renderer. opencode reserves limit.output and targets an input of
# context - output, so it needs headroom. Hermes omits max_tokens entirely, so
# vLLM bounds the output by whatever context is left: the raw window is right.
#
# OWNERSHIP IS PER FIELD, not per model. The manifest records which fields this
# set on which model. A field is retracted only if WE set it and the gateway no
# longer declares it (a model removed upstream, or supports_vision flipped off
# because a backend fell back to language-only — see the drift warning on
# supports_vision in ai-k8s-iac's values.yaml). Anything the user typed into the
# map by hand was never in the manifest and is never touched.
#
# Only fills in the provider install.sh created; never creates one.
_mcp_sync_hermes_models() {
[ -n "${MCP_SYNC_HERMES_CONFIG:-}" ] || return 0
[ -f "$MCP_SYNC_HERMES_CONFIG" ] || return 0
[ -f "$MCP_SYNC_MODELINFO_CACHE" ] || return 0
command -v python3 >/dev/null 2>&1 || return 0
python3 -c 'import yaml' >/dev/null 2>&1 || return 0 # warned about by _mcp_sync_hermes
MCP_SYNC_PY_CACHE="$MCP_SYNC_MODELINFO_CACHE" \
MCP_SYNC_PY_HERMES="$MCP_SYNC_HERMES_CONFIG" \
MCP_SYNC_PY_MANIFEST="$MCP_SYNC_HERMES_MODELS_MANIFEST" \
MCP_SYNC_PY_PROVIDER="$MCP_SYNC_PROVIDER" \
python3 - <<'PYEOF' 2>/dev/null || true
import json, os, tempfile
import yaml
provider = os.environ["MCP_SYNC_PY_PROVIDER"]
dest = os.environ["MCP_SYNC_PY_HERMES"]
manifest_path = os.environ["MCP_SYNC_PY_MANIFEST"]
try:
with open(os.environ["MCP_SYNC_PY_CACHE"]) as f:
rows = json.load(f).get("data") or []
except Exception:
raise SystemExit(0)
rendered = {}
for r in rows:
if not isinstance(r, dict):
continue
name = r.get("model_name")
mi = r.get("model_info") or {}
window = mi.get("max_input_tokens")
# A wildcard route (claude-*) declares no window. Never invent one: a guessed
# context_length is worse than letting Hermes use its own default.
if not name or not isinstance(window, int) or window <= 0:
continue
fields = {"context_length": window}
if mi.get("supports_vision") is True:
fields["supports_vision"] = True
rendered[name] = fields
# A gateway that suddenly declares no windows at all is broken, not empty. Do
# not treat that as "retract everything".
if not rendered:
raise SystemExit(0)
try:
with open(dest) as f:
cfg = yaml.safe_load(f) or {}
except Exception:
raise SystemExit(0) # never overwrite a config we cannot parse
if not isinstance(cfg, dict) or not isinstance(cfg.get("custom_providers"), list):
raise SystemExit(0)
target = next((p for p in cfg["custom_providers"]
if isinstance(p, dict) and p.get("name") == provider), None)
if target is None:
raise SystemExit(0)
models = target.get("models")
if not isinstance(models, dict):
models = {}
before = json.dumps(models, sort_keys=True, default=str)
try:
with open(manifest_path) as f:
prev = json.load(f)
prev = prev if isinstance(prev, dict) else {}
except Exception:
prev = {}
for name, owned in prev.items():
entry = models.get(name)
if not isinstance(owned, list) or not isinstance(entry, dict):
continue
still = rendered.get(name, {})
for field in owned:
if field not in still:
entry.pop(field, None)
if not entry:
models.pop(name, None)
for name, fields in rendered.items():
entry = models.get(name)
entry = entry if isinstance(entry, dict) else {}
entry.update(fields)
models[name] = entry
if json.dumps(models, sort_keys=True, default=str) != before:
target["models"] = models
d = os.path.dirname(dest) or "."
fd, tmp = tempfile.mkstemp(dir=d)
with os.fdopen(fd, "w") as f:
yaml.safe_dump(cfg, f, sort_keys=False, default_flow_style=False,
allow_unicode=True)
os.chmod(tmp, 0o600)
os.replace(tmp, dest)
d = os.path.dirname(manifest_path) or "."
os.makedirs(d, exist_ok=True)
fd, tmp = tempfile.mkstemp(dir=d)
with os.fdopen(fd, "w") as f:
json.dump({n: sorted(fl) for n, fl in rendered.items()}, f, sort_keys=True)
os.replace(tmp, manifest_path)
PYEOF
}
# --- render the gateway's MODEL LIMITS into pi's models.json (non-fatal) ----
#
# Same reason as Hermes: pi reads a static models.json and never asks the
# gateway, so contextWindow/maxTokens were hand-copied and drifted.
#
# THE INVARIANT, because getting this wrong is a guaranteed 400 rather than a
# degraded answer. pi compacts when contextTokens > contextWindow - reserveTokens
# (compaction.js shouldCompact) and then sends max_tokens = maxTokens. vLLM
# counts prompt + max_tokens against its window. So the largest request pi can
# make is (window - reserveTokens) + maxTokens, and it fits only if
#
# maxTokens <= reserveTokens - margin
#
# This used to be violated outright: pi's default reserve is 16384 while
# maxTokens was set to 32768, so a full context sent 196608 - 16384 + 32768 =
# 212992 tokens at a 196608 window.
#
# maxTokens is DERIVED FROM the reserve, rather than the reserve being raised to
# fit maxTokens. reserveTokens is one global setting shared by every provider pi
# knows about; raising it to suit this gateway would silently shrink usable
# context for all of them. Capping maxTokens per model keeps the fix local, and
# means raising reserveTokens in settings.json is how you buy back output.
#
# OWNERSHIP: the manifest records models this ADDED and models it added "image"
# to. contextWindow and maxTokens are always managed on gateway models, because
# a stale value there is the bug. Models the user created are never removed, and
# a user's own "image" is never taken away — only one this added.
#
# Only fills in the provider the user configured; never creates one, because a
# provider needs a baseUrl and key this does not own.
_mcp_sync_pi() {
[ -n "${MCP_SYNC_PI_MODELS:-}" ] || return 0
[ -f "$MCP_SYNC_PI_MODELS" ] || return 0
[ -f "$MCP_SYNC_MODELINFO_CACHE" ] || return 0
command -v python3 >/dev/null 2>&1 || return 0
MCP_SYNC_PY_CACHE="$MCP_SYNC_MODELINFO_CACHE" \
MCP_SYNC_PY_PI="$MCP_SYNC_PI_MODELS" \
MCP_SYNC_PY_PI_SETTINGS="$MCP_SYNC_PI_SETTINGS" \
MCP_SYNC_PY_MANIFEST="$MCP_SYNC_PI_MANIFEST" \
MCP_SYNC_PY_PROVIDER="$MCP_SYNC_PROVIDER" \
MCP_SYNC_PY_OUTPUT_CAP="$MCP_SYNC_OUTPUT_RESERVE_CAP" \
python3 - <<'PYEOF' 2>/dev/null || true
import json, os, tempfile
provider = os.environ["MCP_SYNC_PY_PROVIDER"]
dest = os.environ["MCP_SYNC_PY_PI"]
manifest_path = os.environ["MCP_SYNC_PY_MANIFEST"]
cap = int(os.environ.get("MCP_SYNC_PY_OUTPUT_CAP") or 32768)
# Headroom between pi's compaction point and the server's hard limit. pi counts
# from the server's own usage.totalTokens, so this only has to absorb the new
# message appended since the last measured turn.
MARGIN = 2048
# pi's built-in default when compaction.reserveTokens is unset.
PI_DEFAULT_RESERVE = 16384
try:
with open(os.environ["MCP_SYNC_PY_CACHE"]) as f:
rows = json.load(f).get("data") or []
except Exception:
raise SystemExit(0)
reserve = PI_DEFAULT_RESERVE
try:
with open(os.environ["MCP_SYNC_PY_PI_SETTINGS"]) as f:
r = (json.load(f).get("compaction") or {}).get("reserveTokens")
if isinstance(r, int) and r > 0:
reserve = r
except Exception:
pass
ceiling = reserve - MARGIN
if ceiling <= 0:
raise SystemExit(0) # no output is provably safe under this reserve; do not guess
rendered = {}
for r in rows:
if not isinstance(r, dict):
continue
name = r.get("model_name")
mi = r.get("model_info") or {}
window = mi.get("max_input_tokens")
if not name or not isinstance(window, int) or window <= 0:
continue
out = mi.get("max_output_tokens")
# LiteLLM's max_output_tokens is the WHOLE window for a vLLM backend (it caps
# total context, not output), so it is an upper bound, never a value to copy.
out = out if isinstance(out, int) and out > 0 else cap
rendered[name] = {
"contextWindow": window,
"maxTokens": min(out, cap, ceiling),
"vision": mi.get("supports_vision") is True,
}
if not rendered:
raise SystemExit(0)
try:
with open(dest) as f:
cfg = json.load(f)
except Exception:
raise SystemExit(0)
prov = (cfg.get("providers") or {}).get(provider) if isinstance(cfg, dict) else None
if not isinstance(prov, dict) or not isinstance(prov.get("models"), list):
raise SystemExit(0)
before = json.dumps(cfg, sort_keys=True)
try:
with open(manifest_path) as f:
prev = json.load(f)
prev = prev if isinstance(prev, dict) else {}
except Exception:
prev = {}
added_prev = set(prev.get("added") or [])
image_prev = set(prev.get("image") or [])
# Drop models WE added that the gateway no longer serves. A user-created entry
# is never removed, even if the gateway stops listing it.
models = [m for m in prov["models"]
if not (isinstance(m, dict) and m.get("id") in added_prev
and m.get("id") not in rendered)]
by_id = {m.get("id"): m for m in models if isinstance(m, dict)}
added_now, image_now = set(), set()
for name, f in rendered.items():
m = by_id.get(name)
if m is None:
m = {"id": name, "name": name, "reasoning": False, "input": ["text"],
"cost": {"input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0}}
models.append(m)
by_id[name] = m
added_now.add(name)
elif name in added_prev:
added_now.add(name)
m["contextWindow"] = f["contextWindow"]
m["maxTokens"] = f["maxTokens"]
inp = m.get("input") if isinstance(m.get("input"), list) else ["text"]
if f["vision"]:
if "image" not in inp:
inp = inp + ["image"]
image_now.add(name)
elif name in image_prev:
image_now.add(name)
elif name in image_prev and "image" in inp:
# We added it and the gateway no longer claims vision: take it back, or
# pi keeps sending image parts the backend will reject.
inp = [x for x in inp if x != "image"]
m["input"] = inp
prov["models"] = models
if json.dumps(cfg, sort_keys=True) != before:
d = os.path.dirname(dest) or "."
fd, tmp = tempfile.mkstemp(dir=d)
with os.fdopen(fd, "w") as f:
json.dump(cfg, f, indent=2)
f.write("\n")
os.chmod(tmp, 0o600) # the provider block can carry an apiKey
os.replace(tmp, dest)
d = os.path.dirname(manifest_path) or "."
os.makedirs(d, exist_ok=True)
fd, tmp = tempfile.mkstemp(dir=d)
with os.fdopen(fd, "w") as f:
json.dump({"added": sorted(added_now), "image": sorted(image_now)}, f)
os.replace(tmp, manifest_path)
PYEOF
}
# --- deploy the shared skills tree (throttled, non-fatal) ------------------
_mcp_sync_skills() {
# Same TTL as the config, tracked on the deployed directory's own mtime.
@@ -565,6 +904,11 @@ _mcp_sync_export_key
_mcp_sync_refresh
# After refresh: renders the just-deployed MCP servers into Hermes' schema.
_mcp_sync_hermes
# Then the gateway's model limits into the two clients that do not discover
# them on their own. Both run from the cache, so a throttled shell still
# converges on the last known good values.
_mcp_sync_hermes_models
_mcp_sync_pi
_mcp_sync_skills
# opencode only merges config from known locations; point OPENCODE_CONFIG at