From c6e5f22acbae444d7c79d92e227db064b3851d63 Mon Sep 17 00:00:00 2001 From: Nic O'Connor Date: Thu, 17 Sep 2026 15:30:24 +0000 Subject: [PATCH] secrets: read the 1Password vault once into a 0600 env file Secrets were pulled in each box's hand-written ~/.zshrc.local with one `op read` per field on EVERY new shell: every terminal waited on 1Password, every shell broke when it was unreachable, and the loop lived outside the repo. New secrets/ module (after bin/, which provides op and jq): one vault per user and one item, both defaulting to the login name, whose field labels are the variable names. It asks for the service-account token once (hidden), fetches the item with a single `op item get`, and writes every valid-named field to ~/.config/dotfiles/secrets.env (0600), replaced atomically so a failed fetch keeps the previous copy. Values go through jq @sh, so quotes, $() and newlines in a secret can't break or inject. The token is kept in op.env (0600), which shells never source. bootstrap loads secrets.env right after the module, so mcp-config takes LITELLM_KEY from the env instead of prompting. A LITELLM_KEY rotated in the vault also updates ~/.config/mcp-sync/key, since sync.sh re-exports from that file after secrets.env loads and would otherwise win. The managed zshrc sources secrets.env (no op call at startup) and adds envr / envc / envd / envl / envs to read, upsert, delete, list and refresh; they load the token only in a subshell. Existing boxes still need the old op loop removed from ~/.zshrc.local. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 21 +++++- bootstrap.sh | 9 +++ secrets/install.sh | 155 ++++++++++++++++++++++++++++++++++++++++ zsh/zshrc | 67 +++++++++++++++++ zsh/zshrc.local.example | 16 +++-- 5 files changed, 259 insertions(+), 9 deletions(-) create mode 100755 secrets/install.sh diff --git a/README.md b/README.md index fd87011..6c76df0 100644 --- a/README.md +++ b/README.md @@ -53,6 +53,19 @@ Already have a clone? Just run it directly: redeploys with no reinstall. Idempotent: each tool records its version under `~/.local/bin/.versions` and is skipped when the pinned version is already present. `zsh/zshrc` puts `~/.local/bin` on `PATH`. +- `secrets/` — reads your 1Password vault once and writes every field to + `~/.config/dotfiles/secrets.env` (0600), which the managed `~/.zshrc` + sources, so opening a shell never calls `op`. One vault per user and one + item in it, both named after your login by default; each field's label is + the variable name (`GITEA_KEY`, `LITELLM_KEY`, `PROXMOX_TOKEN_SECRET`, …). + The first run prompts for the service-account token (and vault/item), and + keeps them in `~/.config/dotfiles/op.env` (0600, not loaded into shells). + Runs after `bin/` (needs `op` and `jq`); bootstrap loads the result so later + modules get the keys without prompting. Day to day: `envr NAME` reads a + field, `envc NAME [VALUE]` creates or updates one (prompts, hidden, without + VALUE), `envd NAME` deletes one; both of those refresh the file and this + shell. `envl [VAULT [ITEM]]` shows every field with its current value, and + `envs` refreshes after a change made elsewhere. No sudo. opencode is configured by `mcp-config/`: it points at the covert-gpt LiteLLM gateway (`https://llm.covert-gpt.com/v1`) for models. MCP tools do @@ -87,7 +100,8 @@ rc sources last (so it can override anything) and bootstrap never touches: cp zsh/zshrc.local.example ~/.zshrc.local && chmod 600 ~/.zshrc.local ``` -Variables modules look for: +Variables modules look for. With `secrets/` set up, put these in your vault +item instead of `~/.zshrc.local`: - `GITEA_KEY` — personal access token for `git.nic-oconnor.com`. Used for API/CLI work, and by `bootstrap.sh`/`mcp-config/` to read this repo when it @@ -98,8 +112,9 @@ Variables modules look for: don't set this by hand: run `mcp-config/install.sh` once and its shell hook exports it from `~/.config/mcp-sync/key`. Set it in `~/.zshrc.local` only on a box without mcp-sync. -- `OP_SERVICE_ACCOUNT_TOKEN` — 1Password service-account token for - non-interactive `op read` / `op run`. +- `OP_SERVICE_ACCOUNT_TOKEN` — 1Password service-account token. Not a shell + variable any more: `secrets/install.sh` asks for it once and stores it in + `~/.config/dotfiles/op.env`. Pass it in the env only for an unattended run. - `GITEA_KEY`, `PROXMOX_TOKEN_ID`, `PROXMOX_TOKEN_SECRET` — **your own** credentials for the client-side MCP servers in `mcp-config/`. opencode reads them from your environment (`{env:VAR}`) and passes them to the MCP server it diff --git a/bootstrap.sh b/bootstrap.sh index e00c394..098c244 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -82,6 +82,15 @@ run_module git # Runs before bin/ so the kind install gate can see the rootful daemon. run_module docker run_module bin +# Needs op and jq from bin/. Runs before everything that consumes a secret, and +# its output is loaded into THIS process so later modules see the vault's +# values: mcp-config/ takes LITELLM_KEY from the env instead of prompting for +# it, and GITEA_KEY is there for anything that reads the repo over the API. +run_module secrets +if [ -f "$HOME/.config/dotfiles/secrets.env" ]; then + # shellcheck source=/dev/null + . "$HOME/.config/dotfiles/secrets.env" +fi # Provides npx, which opencode needs to spawn the npm-based MCP servers in # mcp-config/. Without it every MCP server silently fails to start. run_module node diff --git a/secrets/install.sh b/secrets/install.sh new file mode 100755 index 0000000..5aea8ac --- /dev/null +++ b/secrets/install.sh @@ -0,0 +1,155 @@ +#!/usr/bin/env sh +# secrets/install.sh — write this user's 1Password vault item to a local, +# 0600 env file that the managed ~/.zshrc sources. +# +# WHY: secrets used to be pulled live in ~/.zshrc.local, one `op read` per +# field on EVERY new shell. That made each terminal wait on 1Password, broke +# every shell when the network or the service account was down, and meant each +# box carried its own hand-written copy of the loop. Now the vault is read once +# here (at bootstrap, or on demand with `envs`), and shells read a local file. +# +# Layout: one vault per user (default: the login name), holding one item +# (default: also the login name) whose field LABELS are the variable names — +# GITEA_KEY, LITELLM_KEY, PROXMOX_TOKEN_SECRET, ... Every such field is +# exported. Add a secret by adding a field (`envc NAME`), not by editing +# this repo. +# +# Files, all under ~/.config/dotfiles (0700), none in the repo: +# op.env OP_SERVICE_ACCOUNT_TOKEN, OP_VAULT, OP_ITEM (0600). Kept so a +# refresh needs no prompt. Deliberately NOT sourced by shells: +# the token stays out of every process's environment, and only +# the env* helpers in zsh/zshrc load it, in a subshell. +# secrets.env the exported fields (0600), replaced atomically, so a failed +# fetch leaves the previous copy in place. +# +# Prompts for the token on the controlling TTY the first time (bootstrap is +# often piped into sh, so stdin is not the terminal). Without a TTY, pass +# OP_SERVICE_ACCOUNT_TOKEN (and optionally OP_VAULT / OP_ITEM) in the env. +# Needs op and jq, which bin/ installs. No sudo. +set -eu + +say() { printf '%s\n' "$*" >&2; } + +CONF_DIR="${DOTFILES_CONFIG_DIR:-$HOME/.config/dotfiles}" +OP_ENV="$CONF_DIR/op.env" +SECRETS_ENV="$CONF_DIR/secrets.env" +MCP_SYNC_KEYFILE="$HOME/.config/mcp-sync/key" + +umask 077 +mkdir -p "$CONF_DIR" +chmod 0700 "$CONF_DIR" + +PATH="$HOME/.local/bin:$PATH" +for _tool in op jq; do + if ! "$_tool" --version >/dev/null 2>&1; then + say "secrets: $_tool is not installed or does not run; run bin/install.sh first." + exit 1 + fi +done + +# --- 1. resolve the service account, vault and item ------------------------- +# Precedence: the environment, then op.env, then a prompt (token) / defaults. +_env_token="${OP_SERVICE_ACCOUNT_TOKEN:-}" +_env_vault="${OP_VAULT:-}" +_env_item="${OP_ITEM:-}" +if [ -f "$OP_ENV" ]; then + # shellcheck source=/dev/null + . "$OP_ENV" +fi +OP_SERVICE_ACCOUNT_TOKEN="${_env_token:-${OP_SERVICE_ACCOUNT_TOKEN:-}}" +OP_VAULT="${_env_vault:-${OP_VAULT:-}}" +OP_ITEM="${_env_item:-${OP_ITEM:-}}" +_user="$(id -un)" + +# ask — read one line from the TTY; echoes the default on empty. +ask() { + printf '%s [%s]: ' "$1" "$2" >&2 + IFS= read -r _ans < /dev/tty || _ans="" + printf '%s' "${_ans:-$2}" +} + +if [ -z "$OP_SERVICE_ACCOUNT_TOKEN" ]; then + if [ -f "$SECRETS_ENV" ]; then + say "secrets: no service-account token; keeping existing $SECRETS_ENV." + exit 0 + fi + # Probe in a subshell: in POSIX sh a failed redirect on a special builtin + # like `:` exits the whole script, which would skip the message below. + if ! ( : < /dev/tty ) 2>/dev/null; then + say "secrets: no OP_SERVICE_ACCOUNT_TOKEN and no TTY to prompt." + say " re-run as: OP_SERVICE_ACCOUNT_TOKEN=ops_... $0" + exit 1 + fi + printf '1Password service-account token (ops_..., input hidden): ' >&2 + stty -echo < /dev/tty 2>/dev/null || true + IFS= read -r OP_SERVICE_ACCOUNT_TOKEN < /dev/tty || true + stty echo < /dev/tty 2>/dev/null || true + printf '\n' >&2 + OP_SERVICE_ACCOUNT_TOKEN="$(printf '%s' "$OP_SERVICE_ACCOUNT_TOKEN" | tr -d ' \t\r\n')" + [ -n "$OP_SERVICE_ACCOUNT_TOKEN" ] || { say "secrets: no token entered."; exit 1; } + [ -n "$OP_VAULT" ] || OP_VAULT="$(ask 'Vault' "$_user")" + [ -n "$OP_ITEM" ] || OP_ITEM="$(ask 'Item' "$_user")" +fi +OP_VAULT="${OP_VAULT:-$_user}" +OP_ITEM="${OP_ITEM:-$_user}" +export OP_SERVICE_ACCOUNT_TOKEN + +# --- 2. fetch the item once and render it ----------------------------------- +# One `op item get` instead of one `op read` per field. Labels that are not +# valid shell identifiers (and the item's built-in notes field) are skipped +# rather than eval'd; values go through jq's @sh, so quotes, $ and newlines in +# a secret cannot break or inject into the file. +if ! _json="$(op item get "$OP_ITEM" --vault "$OP_VAULT" --reveal --format json)"; then + say "secrets: could not read item '$OP_ITEM' in vault '$OP_VAULT'." + [ -f "$SECRETS_ENV" ] && say "secrets: keeping the previous $SECRETS_ENV." + exit 1 +fi + +_skipped="$(printf '%s' "$_json" | jq -r ' + [.fields[] | select(.purpose != "NOTES") + | select((.label // "") | test("^[A-Za-z_][A-Za-z0-9_]*$") | not) | .label] + | join(", ")')" +[ -z "$_skipped" ] || say "secrets: skipped fields whose labels are not variable names: $_skipped" + +_tmp="$(mktemp "$CONF_DIR/secrets.env.XXXXXX")" +trap 'rm -f "$_tmp"' EXIT +{ + printf '# Generated by dotfiles secrets/install.sh from op://%s/%s — do not edit.\n' "$OP_VAULT" "$OP_ITEM" + printf '# Refresh with: envs (change the vault with: envc NAME / envd NAME)\n' + printf '%s' "$_json" | jq -r ' + .fields[] + | select(.purpose != "NOTES") + | select((.label // "") | test("^[A-Za-z_][A-Za-z0-9_]*$")) + | select((.value // "") != "") + | "export \(.label)=\(.value | @sh)"' +} > "$_tmp" +chmod 0600 "$_tmp" +_count="$(grep -c '^export ' "$_tmp" || true)" +if [ "$_count" -eq 0 ]; then + say "secrets: item '$OP_ITEM' has no exportable fields; keeping the previous file." + exit 1 +fi +mv -f "$_tmp" "$SECRETS_ENV" +trap - EXIT +say "secrets: wrote $_count variables to $SECRETS_ENV (0600)." + +# Remember the account only after it has actually worked. +{ + printf 'OP_SERVICE_ACCOUNT_TOKEN=%s\n' "$(printf '%s' "$OP_SERVICE_ACCOUNT_TOKEN" | jq -Rr @sh)" + printf 'OP_VAULT=%s\n' "$(printf '%s' "$OP_VAULT" | jq -Rr @sh)" + printf 'OP_ITEM=%s\n' "$(printf '%s' "$OP_ITEM" | jq -Rr @sh)" +} > "$OP_ENV" +chmod 0600 "$OP_ENV" + +# --- 3. keep mcp-sync's copy of the gateway key in step ---------------------- +# mcp-config/sync.sh exports LITELLM_KEY from its own key file on every shell, +# AFTER secrets.env is loaded, so a key rotated in the vault would otherwise be +# silently overridden by the stale copy. Only update a file that already exists: +# creating it is mcp-config/install.sh's job. +_litellm="$(printf '%s' "$_json" | jq -r '.fields[] | select(.label == "LITELLM_KEY") | .value // empty' | head -n 1)" +if [ -n "$_litellm" ] && [ -f "$MCP_SYNC_KEYFILE" ] \ + && [ "$(tr -d ' \t\r\n' < "$MCP_SYNC_KEYFILE")" != "$_litellm" ]; then + printf '%s' "$_litellm" > "$MCP_SYNC_KEYFILE" + chmod 0600 "$MCP_SYNC_KEYFILE" + say "secrets: LITELLM_KEY changed; updated $MCP_SYNC_KEYFILE (re-run hermes/install.sh for Hermes)." +fi diff --git a/zsh/zshrc b/zsh/zshrc index 0893519..cc10594 100644 --- a/zsh/zshrc +++ b/zsh/zshrc @@ -74,6 +74,73 @@ grmb() { return $rc } +# Secrets from this user's 1Password vault, written by dotfiles/secrets/install.sh. +# Reading a local 0600 file keeps shell startup off the network: nothing here +# calls op. envc/envd refresh the file themselves; after editing the vault +# anywhere else (the 1Password app, another box), run `envs`. +_DOTFILES_CONF="$HOME/.config/dotfiles" +[ -f "$_DOTFILES_CONF/secrets.env" ] && source "$_DOTFILES_CONF/secrets.env" + +# envr NAME read one field straight from the vault +# envc NAME [VALUE] create or update a field (prompts, hidden, when VALUE is +# omitted, which keeps the secret out of shell history), +# then refresh and reload this shell +# envd NAME delete a field, refresh, and unset it in this shell +# envl [VAULT [ITEM]] show every field and its current value (default: your +# vault and item from op.env) +# envs re-fetch the vault into secrets.env and reload this shell +# +# The service-account token lives in op.env and is loaded only inside these +# subshells, so it never lands in the environment of everything you run. +_dotfiles_op() { + [ -f "$_DOTFILES_CONF/op.env" ] || { print -u2 "no $_DOTFILES_CONF/op.env; run: envs"; return 1; } + ( set -a; source "$_DOTFILES_CONF/op.env"; set +a; "$@" ) +} +_dotfiles_env_name() { + [[ "$1" =~ '^[A-Za-z_][A-Za-z0-9_]*$' ]] && return 0 + print -u2 "$2: '$1' is not a valid variable name"; return 2 +} +envr() { + _dotfiles_env_name "$1" envr || return + _dotfiles_op sh -c 'op read "op://$OP_VAULT/$OP_ITEM/$1"' envr "$1" +} +envc() { + emulate -L zsh + local name=$1 value + _dotfiles_env_name "$name" envc || return + if (( $# >= 2 )); then + value=$2 + else + read -rs "value?$name: " && print + fi + # [concealed] makes it a password field; an existing field of that name is updated. + _dotfiles_op sh -c 'op item edit "$OP_ITEM" --vault "$OP_VAULT" "$1[concealed]=$2" >/dev/null' \ + envc "$name" "$value" && envs +} +envd() { + _dotfiles_env_name "$1" envd || return + _dotfiles_op sh -c 'op item edit "$OP_ITEM" --vault "$OP_VAULT" "$1[delete]" >/dev/null' envd "$1" \ + && unset "$1" && envs +} +envl() { + _dotfiles_op sh -c ' + vault=${1:-$OP_VAULT} item=${2:-$OP_ITEM} + printf "op://%s/%s\n" "$vault" "$item" >&2 + op item get "$item" --vault "$vault" --reveal --format json' envl "$@" \ + | jq -r ' + [.fields[] | select(.purpose != "NOTES")] as $f + | ([$f[] | .label // "" | length] | max // 0) as $w + | $f[] + | ((.label // "") + " " * ($w - ((.label // "") | length))) as $name + | ((.value // "") | split("\n")) as $lines + | "\($name) \($lines[0] // "")", + ($lines[1:][] | (" " * ($w + 2)) + .)' +} +envs() { + "${DOTFILES_DIR:-$HOME/.dotfiles}/secrets/install.sh" \ + && source "$_DOTFILES_CONF/secrets.env" +} + # Per-machine customizations (secrets, keys, local overrides) live in # ~/.zshrc.local, which bootstrap never touches. See zsh/zshrc.local.example # for the variables modules look for. Loaded last so it can override anything. diff --git a/zsh/zshrc.local.example b/zsh/zshrc.local.example index 0e31d3b..59d143c 100644 --- a/zsh/zshrc.local.example +++ b/zsh/zshrc.local.example @@ -1,7 +1,9 @@ # ~/.zshrc.local — per-machine customizations. Copy this file to # ~/.zshrc.local and fill in what this box needs. It is sourced at the end # of the managed ~/.zshrc, is never overwritten by bootstrap, and never -# lives in the repo — this is the ONLY place to put secrets. +# lives in the repo. Secrets normally come from your 1Password vault instead +# (secrets/install.sh -> ~/.config/dotfiles/secrets.env); set one here only on +# a box without 1Password, or to override the vault's value on this machine. # # cp zsh/zshrc.local.example ~/.zshrc.local && chmod 600 ~/.zshrc.local @@ -30,9 +32,11 @@ # Only the bootstrap-managed checkout (~/.dotfiles) is ever `git pull`-ed by the # hook, and only when clean — a tree you work in is read, never touched. -# --- 1Password CLI ------------------------------------------------------- -# Service-account token for non-interactive `op read` / `op run`. -# export OP_SERVICE_ACCOUNT_TOKEN=ops_... +# --- 1Password ----------------------------------------------------------- +# Nothing to set here. Do NOT loop over `op read` in this file: that ran on +# every shell. secrets/install.sh stores the service-account token in +# ~/.config/dotfiles/op.env and writes the vault's fields to secrets.env, which +# the managed ~/.zshrc loads. Use envr / envc / envd (see zsh/zshrc). # --- Proxmox ------------------------------------------------------------- # One pair of credentials, several consumers: the proxmox MCP server @@ -40,8 +44,8 @@ # rke2-devops-airgap, whose Taskfile composes PROXMOX_VE_API_TOKEN from them. # Unprefixed on purpose — these are Proxmox credentials, not MCP ones. # -# The op-vault loop above already exports them from the `noconnor` item, so -# these lines are only needed without 1Password. Never committed. +# secrets.env already exports them from your vault item, so these lines are +# only needed without 1Password. Never committed. # export PROXMOX_TOKEN_ID="root@pam!terragrunt-proxmox" # export PROXMOX_TOKEN_SECRET="00000000-0000-0000-0000-000000000000" #