A bad token paste (or a vault/item that did not match the login-name
default) used to surface only as mcp-config asking for a LiteLLM key,
hiding that every vault key was missing. Now:
- the token is verified with `op whoami` and re-prompted on a bad paste
- vault and item are validated, auto-picked when the token sees only one,
and otherwise prompted with the list it can see
- bootstrap skips mcp-config and hermes, with an error naming the cause,
when no LITELLM_KEY came from the vault
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Secrets were pulled in each box's hand-written ~/.zshrc.local with one
`op read` per field on EVERY new shell: every terminal waited on
1Password, every shell broke when it was unreachable, and the loop lived
outside the repo.
New secrets/ module (after bin/, which provides op and jq): one vault per
user and one item, both defaulting to the login name, whose field labels
are the variable names. It asks for the service-account token once
(hidden), fetches the item with a single `op item get`, and writes every
valid-named field to ~/.config/dotfiles/secrets.env (0600), replaced
atomically so a failed fetch keeps the previous copy. Values go through
jq @sh, so quotes, $() and newlines in a secret can't break or inject.
The token is kept in op.env (0600), which shells never source.
bootstrap loads secrets.env right after the module, so mcp-config takes
LITELLM_KEY from the env instead of prompting. A LITELLM_KEY rotated in
the vault also updates ~/.config/mcp-sync/key, since sync.sh re-exports
from that file after secrets.env loads and would otherwise win.
The managed zshrc sources secrets.env (no op call at startup) and adds
envr / envc / envd / envl / envs to read, upsert, delete, list and
refresh; they load the token only in a subshell.
Existing boxes still need the old op loop removed from ~/.zshrc.local.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Folds in the in-flight local-checkout work and fixes the context accounting
on top of it. Both halves are the same bug seen from two ends: clients were
sending requests larger than the backend's window and getting hard
ContextWindowExceededError 400s, intermittently, in a way that looked like
the model or the GPU misbehaving.
Already in flight here (unchanged in intent):
- sync.sh prefers the LOCAL CHECKOUT over re-downloading from Gitea, and
install.sh records the checkout path so it can be found. That recording
is load-bearing: without it sync.sh cannot find the repo, silently falls
back to Gitea, and every local edit to opencode.gateway.json is ignored.
Observed on this box — ~/.config/mcp-sync/source did not exist, the
default ~/.dotfiles is not where the checkout lives, and the Gitea base
still served models:{}.
- the hermes custom-provider wiring.
Fixed on top:
opencode had NO context limit on ANY gateway model, not a wrong one. Two
faults compounded. /model/info 403s for a day-to-day virtual key (only
management-scoped keys may read it), which sync.sh treats as best-effort and
falls back to names-only — as designed and as commented. But the merge then
did `prov["models"] = models`, replacing the map wholesale, so limits
declared in the base config were discarded too and there was no floor left.
Base specs are now a floor that live /model/info overrides field by field:
discovery still wins where present, so this adds no drift, and a 403 no
longer strips the limits. Verified: ornith 195584/32768, the qwens
261120/32768.
hermes' model map named `qwen3-vl-30b-a3b` at 262144 — a model the gateway
stopped serving when Ornith replaced it — while ornith-1.0-35b, the model
actually on cg-1 AND the only one with a window smaller than its neighbours
(196608), had no entry at all and inherited Hermes' default. The one model
needing a lower ceiling was the one not declaring one, which is exactly the
shape that fails only once a session grows.
NOTE for other boxes: hermes reads ~/.hermes/config.yaml, which install.sh
produces by MERGING gateway.yaml. Editing gateway.yaml alone changes nothing
until install.sh re-runs — this machine's live config was patched by hand.
Not addressed here, deliberately: the gateway-side context_window_fallback
(ai-k8s-iac). It is wired but inert while cg-2 is scaled down, so preventing
the oversized request client-side is what actually works today.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FoDK4wwB1Twty5gwiriYLf
Two client-side fixes against the covert-gpt LiteLLM gateway, both traced
from opencode throwing intermittent ContextWindowExceededError 400s.
opencode (mcp-config/sync.sh):
spec_for() mapped LiteLLM's /model/info onto opencode's schema 1:1, but the
two mean different things by "output". LiteLLM's max_output_tokens is the
whole window (vLLM caps TOTAL context, not output separately, which is
correct for LiteLLM); opencode's limit.output is a RESERVE it subtracts from
context to pick a compaction point. Copying one to the other told opencode to
reserve the entire 262144 window, so it targeted an input of context-output
and the server counted one token more than opencode did -- landing on
context+1 and a hard 400, every time (measured: 196609 in + 65536 out =
262145 vs a 262144 window; the retries shrinking max_tokens by 65 never
helped because the input grew to compensate).
Capping the reserve alone would NOT have fixed it -- opencode would still sit
exactly on the boundary, just at a different input size. So: cap the reserve
(OUTPUT_RESERVE_CAP) and hold a margin back from context
(CONTEXT_SAFETY_MARGIN). Generated at sync time, so it fixes every gateway
model at once, including third-party ones.
hermes (hermes/gateway.yaml, hermes/install.sh):
Hermes reached the gateway by pretending it WAS OpenRouter -- the LiteLLM key
in OPENROUTER_API_KEY, the gateway in OPENROUTER_BASE_URL. Requests routed
fine, but fetch_openrouter_models hardcodes openrouter.ai/api/v1/models and
ignores OPENROUTER_BASE_URL, so the picker listed OpenRouter's ~342-model
catalog -- models this gateway cannot serve -- while the gateway's own models,
including cg-1's, never appeared at all.
Replaced with a named custom provider. install.sh now merges gateway.yaml into
~/.hermes/config.yaml, so a fresh box gets a Hermes that can actually talk to
the gateway instead of one pointed at nothing. The merge is idempotent and
narrow: only the covert-gpt entry is touched, model.default/provider are
seeded only when unset, and a box still on the masquerade is migrated off it
(fingerprint: provider openrouter + base_url pointing at this gateway; a
genuine OpenRouter user never has that, and is left alone).
No secrets: gateway.yaml names the key via key_env, and mcp-config/sync.sh is
what exports it. Hermes needs no context safety margin -- it omits max_tokens
entirely, so vLLM bounds output by whatever context is left.
Also folds in the in-flight hermes bootstrap wiring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bootstrap failed outright on Oracle Linux: docker/install.sh piped
get.docker.com to sh, and that script allowlists distros by /etc/os-release
ID, so it aborted with "ERROR: Unsupported distribution 'ol'".
On a distro it does recognize, the convenience script only adds the repo and
installs the packages -- exactly what install_rhel() now does directly. Doing
it ourselves drops the allowlist, so every EL rebuild (Oracle, Rocky, Alma,
CentOS, RHEL) goes through one code path instead of needing to be enumerated.
Debian-family still uses get.docker.com. containerd.io needs --allowerasing to
replace the runc that Oracle Linux ships.
Also stop one failed module from killing the whole run: modules were bare
calls under `set -eu`, so docker/'s exit meant bin/, node/ and ansible/
silently never ran -- an unsupported distro string left the box with no CLIs,
no npx and no ansible. run_module warns and continues, then bootstrap exits
non-zero with a summary of what failed. Same guard 35befbe added for
individual tools in bin/install.sh, one level up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a minimal Debian/RHEL image neither python3 nor unzip is present.
lib/fetch.sh's install_zip_bin needs one of them, so the 1Password CLI
install failed -- and under `set -eu` that killed bin/install.sh outright.
install_1p sits immediately before install_opencode and install_zarf, so
neither installed; bin/install.sh's non-zero exit then aborted bootstrap.sh,
so node/ (npx) and ansible/ (python3) never ran either.
Net effect on a fresh box: no opencode, no npx for the MCP servers, and no
python3 for model discovery. It presented as "opencode has no models or
skills". Reproduced in a clean debian:13 container.
Two fixes:
- bin/install.sh: each tool now installs via try(), which warns and
continues instead of aborting, and reports a summary of what failed. One
optional CLI is not worth the rest of the bootstrap. The subshell
re-enables set -e so a failed download can't fall through to mark_version
and stamp a version that was never installed.
- bootstrap.sh: ensure python3 before bin/ runs. It is a hard dependency,
not an extra -- lib/fetch.sh needs it (or unzip) for .zip assets, and
mcp-config/sync.sh needs it to inject discovered models.
Verified end to end on fresh debian:13: bootstrap completes, opencode/zarf/
op/npx/node/python3 all present, and opencode resolves all 3 covert-gpt
models.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The MCP servers in mcp-config/ are npm packages that opencode spawns as
`npx -y <pkg>`. Without npx on PATH they all fail to start, and opencode
reports that only as a dead server -- easy to misread as a broken config.
Node is a tree, not a single binary, so it follows the ansible module:
unpack under ~/.local/opt and symlink the CLIs into ~/.local/bin.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
kind never worked reliably against rootless Docker on Kasm, even with the
cgroup delegation from docker-rootless-cgroup/. So install full Docker
where we can, and flip bin/'s kind gate from "rootless is available" to
"a rootful daemon answers" — kind is simply skipped on Kasm now.
docker/install.sh no-ops when a rootless daemon is already running so it
never fights Kasm's setup. It runs before bin/ in bootstrap so the kind
gate can see the daemon it just started. zsh/zshrc re-execs via newgrp
when the account gained the docker group but the live shell predates it.
docker-rootless-cgroup/ keeps its delegation (still useful for
per-container resource limits); only its kind-centric comments change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rootless Docker only gets cgroup v2 controllers delegated when systemd
sets Delegate=yes on the user session. Kasm images have no systemd, so
the daemon comes up with CgroupDriver=none and `kind create cluster`
fails. This module reproduces that delegation by hand: evacuates the
cgroup-namespace root, delegates controllers into a user-owned /rootless
slice, installs a PATH shim that gives the daemon a private cgroup leaf
so it can self-evacuate, and restarts it. Guarded to no-op on systemd,
rootful, and non-Docker hosts; wired into bootstrap.sh last.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the placeholder ripgrep/fd/bat/fzf set with devops tooling, keeping
jq: kubectl, helm, k9s, kubectx/kubens, stern, yq, OpenTofu, and terragrunt
— all self-contained, statically linked release binaries in ~/.local/bin,
so they persist across Kasm redeploys with no reinstall.
Switch idempotency from parsing each tool's --version (kubectl/helm/k9s have
no uniform flag) to a version stamp under ~/.local/bin/.versions, and add
fetch_arch_deb() for the amd64/arm64 naming most devops tools use.
Ansible is Python, not a single binary, so it gets its own module: a venv at
~/.local/opt/ansible with its CLIs symlinked into ~/.local/bin. The guard
verifies ansible actually runs (a redeploy can wipe the system python the
venv links to) and rebuilds if not.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New bin/install.sh downloads self-contained release binaries (ripgrep, fd,
bat, fzf, jq) into ~/.local/bin — no sudo, nothing outside $HOME, so they
persist across Kasm redeploys with no reinstall. Statically linked where
available, so they keep working even after the root fs is wiped. Idempotent:
pinned versions already present are skipped.
lib/fetch.sh adds arch detection + tarball/raw binary download helpers.
zsh/zshrc puts ~/.local/bin on PATH; bootstrap.sh runs the module.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Piping bootstrap.sh into sh left $0 as a non-path, so SCRIPT_DIR pointed
at the caller's cwd and the module scripts couldn't be found (exit 127).
When no checkout is present, clone/update the repo into ~/.dotfiles and
re-exec from there. Add a Quick start section to the README.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cross-distro (RHEL/Debian) VM setup: zsh + oh-my-zsh with a self-healing
bash fallback for environments that reset the root filesystem or hardcode
bash for new terminals (e.g. Kasm Workspaces), plus git identity and
HTTPS credential-store configuration.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
noconnorandClaude Opus 5