mcp-config: enrich discovered models with limits, cost and capabilities
/v1/models returns ONLY model ids, so every model was deployed as {} and
opencode ran all of them on defaults -- no context window (so it cannot manage
truncation), no cost, and no idea whether the model can call tools. opencode
requires this metadata in the models map: it has no /v1/models discovery of its
own, and models.dev only covers known providers, not a private LiteLLM gateway.
sync.sh now also queries /model/info and maps LiteLLM's metadata onto opencode's
per-model spec:
max_input_tokens -> limit.context
max_output_tokens -> limit.output
input/output_cost_per_token -> cost.input/output (per-token -> $ per 1M)
supports_function_calling -> tool_call
supports_reasoning -> reasoning
supports_vision -> attachment
Best-effort and backwards-compatible: /model/info is outside the llm_api_routes
scope most virtual keys have, so an un-updated key 403s and falls back to
names-only -- exactly today's behaviour, no regression.
Note the gateway only reports metadata for models in LiteLLM's registry. The
self-hosted vLLM entries (qwen3-*) carry no model_info in the gateway's
model_list, so they still deploy bare. Fixing that is a gateway-side change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
+53
-2
@@ -94,13 +94,25 @@ _mcp_sync_refresh() {
|
||||
"$MCP_SYNC_GATEWAY/v1/models" 2>/dev/null)" || _models=""
|
||||
fi
|
||||
|
||||
# 2b. Enrich with per-model metadata. /v1/models returns ONLY ids — no context
|
||||
# window, no cost, no capability flags — and opencode needs those or it
|
||||
# runs every model on defaults (it cannot manage truncation without a
|
||||
# context limit, and won't know a model can call tools). /model/info has
|
||||
# them. Best-effort: a key still scoped to llm_api_routes 403s here, and
|
||||
# we fall back to names-only, exactly as before.
|
||||
_info=""
|
||||
if [ -n "${LITELLM_KEY:-}" ]; then
|
||||
_info="$(curl -fsS -m 8 -H "Authorization: Bearer $LITELLM_KEY" \
|
||||
"$MCP_SYNC_GATEWAY/model/info" 2>/dev/null)" || _info=""
|
||||
fi
|
||||
|
||||
# 3. Merge and deploy. python3 validates the base (don't clobber good config
|
||||
# with an error page), injects the discovered models, and — when discovery
|
||||
# failed — carries over the previous deployed models map so a gateway
|
||||
# hiccup never empties the model list. Without python3, deploy the raw
|
||||
# base as-is (no discovery).
|
||||
if command -v python3 >/dev/null 2>&1; then
|
||||
MCP_SYNC_BASE="$_base" MCP_SYNC_MODELS="$_models" \
|
||||
MCP_SYNC_BASE="$_base" MCP_SYNC_MODELS="$_models" MCP_SYNC_MODELINFO="$_info" \
|
||||
MCP_SYNC_PY_DEST="$MCP_SYNC_DEST" MCP_SYNC_PY_PROVIDER="$MCP_SYNC_PROVIDER" \
|
||||
python3 - <<'PYEOF' 2>/dev/null || true
|
||||
import json, os, tempfile
|
||||
@@ -118,10 +130,49 @@ try:
|
||||
except Exception:
|
||||
raise SystemExit(0) # invalid base: keep last-good deployed config
|
||||
|
||||
# LiteLLM /model/info -> opencode's per-model spec. Only fields the gateway
|
||||
# actually reports are emitted; a model LiteLLM knows nothing about (a
|
||||
# self-hosted vLLM entry with no model_info in the gateway's model_list) simply
|
||||
# stays {}, which is what every model got before this existed.
|
||||
def spec_for(mi):
|
||||
spec = {}
|
||||
limit = {}
|
||||
if mi.get("max_input_tokens"):
|
||||
limit["context"] = mi["max_input_tokens"]
|
||||
if mi.get("max_output_tokens"):
|
||||
limit["output"] = mi["max_output_tokens"]
|
||||
if limit:
|
||||
spec["limit"] = limit
|
||||
# LiteLLM prices PER TOKEN; opencode (like models.dev) uses $ per 1M tokens.
|
||||
cost = {}
|
||||
for src, dst in (("input_cost_per_token", "input"),
|
||||
("output_cost_per_token", "output")):
|
||||
if mi.get(src) is not None:
|
||||
cost[dst] = mi[src] * 1_000_000
|
||||
if cost:
|
||||
spec["cost"] = cost
|
||||
for src, dst in (("supports_function_calling", "tool_call"),
|
||||
("supports_reasoning", "reasoning"),
|
||||
("supports_vision", "attachment")):
|
||||
if mi.get(src) is not None:
|
||||
spec[dst] = bool(mi[src])
|
||||
return spec
|
||||
|
||||
info = {}
|
||||
try:
|
||||
for entry in (json.loads(os.environ.get("MCP_SYNC_MODELINFO") or "{}")
|
||||
.get("data", [])):
|
||||
name = entry.get("model_name")
|
||||
if name:
|
||||
info[name] = spec_for(entry.get("model_info") or {})
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
models = {}
|
||||
try:
|
||||
data = json.loads(os.environ.get("MCP_SYNC_MODELS") or "{}")
|
||||
models = {m["id"]: {} for m in data.get("data", []) if m.get("id")}
|
||||
models = {m["id"]: dict(info.get(m["id"], {}))
|
||||
for m in data.get("data", []) if m.get("id")}
|
||||
except Exception:
|
||||
pass
|
||||
if not models:
|
||||
|
||||
Reference in New Issue
Block a user