Compare commits

...
11 Commits
Author SHA1 Message Date
noconnorandClaude Opus 5 db3a3e6e82 claude-obsidian: don't count skipped empty sessions as still queued
The first full drain reported "105 written, 0 failed, 9 still queued" when
the 9 were empty sessions already recorded as skipped -- nothing was left.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KF322NzozeaFjpgWvh4F4g
2026-09-17 16:14:45 -04:00
noconnorandClaude Opus 5 376ec03428 claude-obsidian: unattended overnight runs -- lock, URL fallback, config fix
Needed to leave a full backlog drain running unattended:

- Run lock (flock): enabling the timer fired an immediate catch-up run
  (Persistent=true) while a manual run was about to start; without a
  lock the two would race on state.json and usage.jsonl.
- OBSIDIAN_API_URL accepts a comma-separated list, first reachable wins,
  so the permanent address and an interim port-forward share one config.
- Fix: inline "# comments" in config.env were parsed into values, so
  BUDGET_USD_PER_RUN=0.50  # ... crashed float(). The shipped
  config.env.example had exactly that shape on several lines -- anyone
  copying it would have hit the crash. A '#' inside a value (URL fragment)
  still survives. Tested against both the example and a real config.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KF322NzozeaFjpgWvh4F4g
2026-09-17 16:14:45 -04:00
noconnorandClaude Opus 5 2bfa7e9a2d claude-obsidian: meter and cap usage; cut per-session cost ~5.6x
The user asked for usage to be monitored closely, and measuring it turned
up that most of the spend was avoidable.

Metering: every model call is appended to usage.jsonl with BILLED tokens
(modelUsage -- the visible `usage` turn undercounted by half) and the
CLI's list-price cost. `usage` reports 5h/24h/7d. Each call's cost is
estimated first and refused if it would cross the per-run or rolling 24h
budget, so budgets cannot be overshot. Removed --max-budget-usd: it
aborts after spending and discards the result ($0.26 lost in one call).

Cost, measured on one session ($0.057 -> $0.010, same summary content):
- alwaysThinkingEnabled=false: extended thinking was ~85% of output.
- DISABLE_PROMPT_CACHING=1: one-shot prompts were being written to the
  1h cache at 2x input price.
- CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1: a hidden side request
  re-read the whole prompt, doubling billed input (9,638 vs 4,663).

Summaries are cached until their note is written, so a vault failure
never re-pays; --prefetch summarizes while the vault is unreachable.
Offline tests (fake claude binary): budget refusal pre-flight, cache
reuse with zero calls, map-reduce, env/argv levers, billed-token ledger.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KF322NzozeaFjpgWvh4F4g
2026-09-17 16:14:45 -04:00
noconnorandClaude Opus 5 11138d90a9 claude-obsidian: summarize Claude conversations into Obsidian notes
One note per Claude Code session (incremental, hourly user timer) and a
claude.ai data-export importer, written through the Obsidian Local REST API
plugin -- whose built-in /mcp/ server is also how Claude reads them back.

Transcripts are condensed (tool output dropped, one line per tool call,
~60x smaller), redacted, then summarized by headless `claude -p` on the
user's own plan. Three things learned the hard way are encoded here:

- --no-session-persistence, or every summary becomes a transcript that
  the next run summarizes.
- A LiteLLM gateway passing through the caller's Claude subscription has
  no key of its own, so a background job cannot use its claude-* models.
- Summaries share the plan's usage limits. A parallel ~100-session
  backfill exhausted a session limit in minutes. So one job, a per-run
  cap, and the run stops at the first usage-limit error (exit 75) instead
  of failing the whole queue; a stop flag, because Future.cancel() cannot
  stop work a thread already picked up. Covered by an offline test of
  stop / resume / idempotent rerun.

Nothing environment-specific is committed: hosts, keys and paths live in
~/.config/claude-obsidian/ (this repo is public).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KF322NzozeaFjpgWvh4F4g
2026-09-17 16:14:45 -04:00
noconnorandClaude Opus 5 7a7e987268 mcp-config: put the shell hook in ~/.zshrc.local on a fresh box
install.sh chose the rc file from $SHELL. On a fresh box bootstrap runs
from the user's original bash: zsh/ has already switched the login shell
to zsh, but $SHELL in that process still says bash. The hook went into
~/.bashrc, below zsh/'s auto-exec hook, so every new terminal exec'd into
zsh before reaching it, sync.sh never ran, OPENCODE_CONFIG was never
exported, and opencode came up unconfigured. Boxes already on zsh were
fine, which is why it only showed up on a new machine.

Now the dotfiles-managed ~/.zshrc, or zsh as the account's login shell
(getent), selects ~/.zshrc.local regardless of $SHELL. Standalone
`curl | sh` installs without these dotfiles behave as before.

Reproduced and verified in a clean ubuntu:24.04 container (zsh/ then
mcp-config/, dummy key): before, hook in .bashrc and OPENCODE_CONFIG
unset in a new zsh; after, hook in .zshrc.local and OPENCODE_CONFIG
points at a config with the covert-gpt provider and gitea/proxmox MCP
servers. A box already broken this way is repaired by re-running the
module, and re-runs leave a single hook.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 16:53:12 +00:00
noconnorandClaude Opus 5 0426ebc973 git: authenticate to our Gitea from $GITEA_KEY
git never reads GITEA_KEY on its own. With only `credential.helper
store`, auth depended on whatever was typed at a first prompt and cached
in ~/.git-credentials: it went stale when the token was rotated in the
vault, and a box that never pushed had no credential at all, so pushes
failed even with a key exported.

Add a credential helper scoped to https://git.nic-oconnor.com that hands
git $GITEA_KEY as the password (username from GITEA_USER, else the login
name). An empty helper entry first clears the inherited `store` for that
host, so a rejected token is never cached and a stale one can't win.
Other hosts keep `store`. With GITEA_KEY unset it prints nothing and git
prompts as before. Idempotent: re-runs leave one copy.

Verified: `git credential fill` returns exactly $GITEA_KEY for the host,
other hosts are unaffected, ~/.git-credentials stays absent. A dry-run
push still fails, because the key currently in the vault is rejected by
Gitea (401 on the API too); that is the token, not this wiring.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 16:36:20 +00:00
noconnorandClaude Opus 5 1159197e4b secrets: rename the vault helpers from env* to op*
They act on 1Password, not on the environment, so the prefix now says so:
opr (read), opc (create/update), opd (delete), opl (list), ops (refresh).
No behavior change. Checked the new names don't collide with any command
or oh-my-zsh alias, and verified opr/opl/ops against the real vault
(values compared, not printed).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 16:36:20 +00:00
noconnorandClaude Opus 5 c6e5f22acb secrets: read the 1Password vault once into a 0600 env file
Secrets were pulled in each box's hand-written ~/.zshrc.local with one
`op read` per field on EVERY new shell: every terminal waited on
1Password, every shell broke when it was unreachable, and the loop lived
outside the repo.

New secrets/ module (after bin/, which provides op and jq): one vault per
user and one item, both defaulting to the login name, whose field labels
are the variable names. It asks for the service-account token once
(hidden), fetches the item with a single `op item get`, and writes every
valid-named field to ~/.config/dotfiles/secrets.env (0600), replaced
atomically so a failed fetch keeps the previous copy. Values go through
jq @sh, so quotes, $() and newlines in a secret can't break or inject.
The token is kept in op.env (0600), which shells never source.

bootstrap loads secrets.env right after the module, so mcp-config takes
LITELLM_KEY from the env instead of prompting. A LITELLM_KEY rotated in
the vault also updates ~/.config/mcp-sync/key, since sync.sh re-exports
from that file after secrets.env loads and would otherwise win.

The managed zshrc sources secrets.env (no op call at startup) and adds
envr / envc / envd / envl / envs to read, upsert, delete, list and
refresh; they load the token only in a subshell.

Existing boxes still need the old op loop removed from ~/.zshrc.local.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 15:30:24 +00:00
noconnorandClaude Opus 5 1acbd28c6e bin: add the coder CLI
Pinned to v2.36.5 (latest release). Same shape as the other tarball
installs: the asset drops the tag's leading v and the binary sits at the
archive root. Verified on this box: installs, `coder version` reports
v2.36.5, and a re-run skips it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 15:30:24 +00:00
noconnorandClaude Opus 5 5fac9cb63b mcp-config: make install.sh executable, so bootstrap can run it
It was created 100644 for the curl | sh path, where the mode doesn't
matter. Since bootstrap.sh started calling it via run_module, which execs
the script directly, the module failed with "Permission denied" (126) and
the sync hook / LITELLM_KEY never got deployed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 14:09:36 +00:00
noconnorandClaude Opus 5 6ac2a02f1b mcp-config: coerce /model/info numbers, so a string price can't break opencode
The litellm-helm chart renders the proxy config with Go's toYaml, which writes
0.000005 as `5e-06`. LiteLLM loads it with PyYAML (YAML 1.1), which only reads
an exponent float that contains a dot, so /model/info returns input_cost_per_token
as the STRING "5e-06" for opus/sonnet/haiku while 2.5e-05 stays a float.
`"5e-06" * 1_000_000` is string repetition in Python: sync.sh wrote a 20 MB
opencode.gateway.json with a string cost, opencode rejected the whole config
("Expected number"), and every new shell redeployed it.

- num(): accept int/float/numeric strings, drop anything non-finite or non-numeric
- clean_spec(): validate every model spec before writing, whatever its source,
  so the discovery-failed carry-over path heals an already-corrupted file
  instead of preserving it
- refuse to deploy a config over 1 MB; keep the last-good file

Tested against the live gateway: costs render 5/25, 3/15, 1/5 $/1M, and
`opencode models covert-gpt` lists all 7 models; with no key, the 20 MB
corrupted file heals to 2.8 KB.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Duzwo1tmEqD2kYYyNKm7xs
2026-09-14 13:04:28 -04:00
16 changed files with 1358 additions and 22 deletions
+20 -4
View File
@@ -45,13 +45,27 @@ Already have a clone? Just run it directly:
kubectx/kubens, stern, yq, OpenTofu, terragrunt, task (go-task, the kubectx/kubens, stern, yq, OpenTofu, terragrunt, task (go-task, the
Taskfile runner), the 1Password CLI `op` for secrets storage, the Taskfile runner), the 1Password CLI `op` for secrets storage, the
opencode coding agent, zarf (DevSecOps for Offline and Airgapped opencode coding agent, zarf (DevSecOps for Offline and Airgapped
Kubernetes), and packer (HashiCorp's infrastructure build tool). Kubernetes), packer (HashiCorp's infrastructure build tool), and the
coder CLI for Coder workspaces.
kind is installed only when full (non-rootless) Docker is available — kind is installed only when full (non-rootless) Docker is available —
it doesn't work against rootless Docker, so it is skipped on Kasm. it doesn't work against rootless Docker, so it is skipped on Kasm.
No sudo, nothing outside `$HOME`, so tools persist across Kasm No sudo, nothing outside `$HOME`, so tools persist across Kasm
redeploys with no reinstall. Idempotent: each tool records its version redeploys with no reinstall. Idempotent: each tool records its version
under `~/.local/bin/.versions` and is skipped when the pinned version under `~/.local/bin/.versions` and is skipped when the pinned version
is already present. `zsh/zshrc` puts `~/.local/bin` on `PATH`. is already present. `zsh/zshrc` puts `~/.local/bin` on `PATH`.
- `secrets/` — reads your 1Password vault once and writes every field to
`~/.config/dotfiles/secrets.env` (0600), which the managed `~/.zshrc`
sources, so opening a shell never calls `op`. One vault per user and one
item in it, both named after your login by default; each field's label is
the variable name (`GITEA_KEY`, `LITELLM_KEY`, `PROXMOX_TOKEN_SECRET`, …).
The first run prompts for the service-account token (and vault/item), and
keeps them in `~/.config/dotfiles/op.env` (0600, not loaded into shells).
Runs after `bin/` (needs `op` and `jq`); bootstrap loads the result so later
modules get the keys without prompting. Day to day: `opr NAME` reads a
field, `opc NAME [VALUE]` creates or updates one (prompts, hidden, without
VALUE), `opd NAME` deletes one; both of those refresh the file and this
shell. `opl [VAULT [ITEM]]` shows every field with its current value, and
`ops` refreshes after a change made elsewhere. No sudo.
opencode is configured by `mcp-config/`: it points at the covert-gpt opencode is configured by `mcp-config/`: it points at the covert-gpt
LiteLLM gateway (`https://llm.covert-gpt.com/v1`) for models. MCP tools do LiteLLM gateway (`https://llm.covert-gpt.com/v1`) for models. MCP tools do
@@ -86,7 +100,8 @@ rc sources last (so it can override anything) and bootstrap never touches:
cp zsh/zshrc.local.example ~/.zshrc.local && chmod 600 ~/.zshrc.local cp zsh/zshrc.local.example ~/.zshrc.local && chmod 600 ~/.zshrc.local
``` ```
Variables modules look for: Variables modules look for. With `secrets/` set up, put these in your vault
item instead of `~/.zshrc.local`:
- `GITEA_KEY` — personal access token for `git.nic-oconnor.com`. Used for - `GITEA_KEY` — personal access token for `git.nic-oconnor.com`. Used for
API/CLI work, and by `bootstrap.sh`/`mcp-config/` to read this repo when it API/CLI work, and by `bootstrap.sh`/`mcp-config/` to read this repo when it
@@ -97,8 +112,9 @@ Variables modules look for:
don't set this by hand: run `mcp-config/install.sh` once and its shell don't set this by hand: run `mcp-config/install.sh` once and its shell
hook exports it from `~/.config/mcp-sync/key`. Set it in `~/.zshrc.local` hook exports it from `~/.config/mcp-sync/key`. Set it in `~/.zshrc.local`
only on a box without mcp-sync. only on a box without mcp-sync.
- `OP_SERVICE_ACCOUNT_TOKEN` — 1Password service-account token for - `OP_SERVICE_ACCOUNT_TOKEN` — 1Password service-account token. Not a shell
non-interactive `op read` / `op run`. variable any more: `secrets/install.sh` asks for it once and stores it in
`~/.config/dotfiles/op.env`. Pass it in the env only for an unattended run.
- `GITEA_KEY`, `PROXMOX_TOKEN_ID`, `PROXMOX_TOKEN_SECRET` — **your own** - `GITEA_KEY`, `PROXMOX_TOKEN_ID`, `PROXMOX_TOKEN_SECRET` — **your own**
credentials for the client-side MCP servers in `mcp-config/`. opencode reads credentials for the client-side MCP servers in `mcp-config/`. opencode reads
them from your environment (`{env:VAR}`) and passes them to the MCP server it them from your environment (`{env:VAR}`) and passes them to the MCP server it
+12
View File
@@ -204,6 +204,17 @@ install_terragrunt() {
mark_version terragrunt "$ver" mark_version terragrunt "$ver"
} }
# coder — CLI for Coder workspaces (coder login, coder ssh, coder config-ssh).
# Release tag is vX but asset drops the v; binary is at the archive root.
install_coder() {
ver=2.36.5
have_version coder "$ver" && { fsay "coder $ver already installed."; return 0; }
install_tarball_bin \
"https://github.com/coder/coder/releases/download/v$ver/coder_${ver}_linux_$DEB.tar.gz" \
coder coder
mark_version coder "$ver"
}
# try <install_fn> — run one tool's installer; on failure, warn and keep going. # try <install_fn> — run one tool's installer; on failure, warn and keep going.
# #
# WHY: these installs used to be bare calls, so under `set -eu` the FIRST # WHY: these installs used to be bare calls, so under `set -eu` the FIRST
@@ -242,6 +253,7 @@ try install_1p
try install_opencode try install_opencode
try install_zarf try install_zarf
try install_packer try install_packer
try install_coder
if [ -n "$FAILED" ]; then if [ -n "$FAILED" ]; then
fsay "" fsay ""
+9
View File
@@ -82,6 +82,15 @@ run_module git
# Runs before bin/ so the kind install gate can see the rootful daemon. # Runs before bin/ so the kind install gate can see the rootful daemon.
run_module docker run_module docker
run_module bin run_module bin
# Needs op and jq from bin/. Runs before everything that consumes a secret, and
# its output is loaded into THIS process so later modules see the vault's
# values: mcp-config/ takes LITELLM_KEY from the env instead of prompting for
# it, and GITEA_KEY is there for anything that reads the repo over the API.
run_module secrets
if [ -f "$HOME/.config/dotfiles/secrets.env" ]; then
# shellcheck source=/dev/null
. "$HOME/.config/dotfiles/secrets.env"
fi
# Provides npx, which opencode needs to spawn the npm-based MCP servers in # Provides npx, which opencode needs to spawn the npm-based MCP servers in
# mcp-config/. Without it every MCP server silently fails to start. # mcp-config/. Without it every MCP server silently fails to start.
run_module node run_module node
+113
View File
@@ -0,0 +1,113 @@
# claude-obsidian — Claude conversations → Obsidian notes
Summarizes every Claude conversation into one searchable Obsidian note: what happened, what
was decided and why, the facts and gotchas worth remembering, and what was left open.
- **Claude Code** sessions sync automatically and incrementally (hourly timer).
- **claude.ai** chats import from a data export (claude.ai has no API, so each export is manual).
Notes are written through the [Obsidian Local REST API](https://github.com/coddingtonbear/obsidian-local-rest-api)
plugin, so the vault can live anywhere the plugin is reachable. The same plugin exposes a
**built-in MCP server** (`/mcp/`), which is how Claude reads the notes back — no separate MCP
process.
```
~/.claude/projects/*/*.jsonl ─┐ ┌─ Claude/Code/2026/…md
claude.ai export .zip ────────┼─ condense → redact → summarize → PUT ────┼─ Claude/Web/2026/…md
└─ (headless claude -p, no transcript) └─ Claude/Index.md
```
## Install
1. In Obsidian, install **Local REST API**, enable its **HTTP server**, and make it reachable
from this machine. Put the API key in `~/.config/claude-obsidian/api-key` (`chmod 600`).
2. `./install.sh` — links the binary into `~/.local/bin`, enables the hourly user timer, and
creates `~/.config/claude-obsidian/config.env` from `config.env.example` if absent.
3. Edit `config.env` (at minimum `OBSIDIAN_API_URL`), then preview one note:
```bash
claude-obsidian-sync stats # sizes, no network
claude-obsidian-sync sync --dry-run --limit 1 # prints the note, writes nothing
```
## Use
```bash
claude-obsidian-sync sync # what the timer runs
claude-obsidian-sync sync --session <session-id> # one session, now, ignoring the settle window
claude-obsidian-sync import-claude-ai ~/Downloads/data-2026-09-13.zip
claude-obsidian-sync sync --prefetch --limit 3 # summarize into the local cache, vault untouched
claude-obsidian-sync usage # calls, billed tokens, spend: last 5h / 24h / 7d
```
Claude reads the vault through the plugin's MCP server:
```bash
claude mcp add --scope user --transport http obsidian http://<host>:27123/mcp/ \
--header "Authorization: Bearer $(cat ~/.config/claude-obsidian/api-key)"
```
## How it behaves — and why
- **Summaries run on your own Claude plan** via headless `claude -p` (`LLM_BACKEND=claude-cli`).
No API key to manage. An OpenAI-compatible backend (`LLM_BACKEND=openai`) exists for local
models. Note a LiteLLM gateway that fronts Claude by *passing through the caller's*
subscription has no key of its own, so a background job cannot use its `claude-*` models.
- **`--no-session-persistence` is load-bearing.** Without it every summary writes a new
transcript into `~/.claude/projects`, and the next run summarizes its own summarizing.
- **Usage is metered and capped.** Every model call is appended to
`~/.local/state/claude-obsidian/usage.jsonl` with *billed* tokens (the CLI's `modelUsage`, not
the visible turn) and `total_cost_usd` — API list-price equivalent, a faithful proxy for how much
was consumed; the plan's own meter is only visible in Claude Code's `/usage`. Before each call
its cost is **estimated** (+20% margin) and the call is refused if it would cross
`BUDGET_USD_PER_RUN` or the rolling `BUDGET_USD_PER_DAY` — so budgets are never overshot. Not
`claude -p --max-budget-usd`: that aborts *after* the tokens are spent and discards the result
(measured: $0.26 for nothing). Every run ends with a one-line usage report.
- **Three cost levers, measured on one session — $0.057 → $0.010, same summary content:**
`alwaysThinkingEnabled=false` (extended thinking was ~85% of output tokens),
`DISABLE_PROMPT_CACHING=1` (the CLI writes each prompt to the 1-hour cache at 2× input price, and
a one-shot call never reads it back), and `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1` (otherwise a
hidden side request re-reads the entire prompt, doubling billed input). A whole ~110-session
backlog is about $2 at list price with all three.
- **Summaries are cached until their note is written**, so a vault outage never pays for the same
summary twice, and `--prefetch` can do the summarizing while the vault is unreachable.
- **It shares your plan's usage limits.** A first backfill of ~100 sessions in parallel hit a
session limit within minutes — which also limits *interactive* use. So: one job at a time,
`MAX_PER_RUN` sessions per run (newest first), and the run **stops at the first usage-limit
error** (exit 75, which the systemd unit treats as success) instead of failing the rest of
the queue. The next hourly run resumes. A 100-session backlog drains in about ten hours.
- **Secrets are redacted before anything leaves the machine**, and again in the note: private
keys, bearer tokens, `password=`-style assignments, JWTs, provider tokens, ≥48-char hex, and
random mixed-case strings. It over-redacts on purpose; git SHAs, md5s and paths survive.
The summarizer is also told never to reproduce secrets.
- **One run at a time** (a `flock` on `run.lock`): an hourly timer firing during a long manual run
exits immediately instead of racing it on the state and ledger files.
- **Several vault URLs** can be listed; the first that answers wins, so one config covers both a
permanent address and a temporary `kubectl port-forward`.
- **Incremental and idempotent.** State (`~/.local/state/claude-obsidian/state.json`) records
each session's size+mtime. A resumed session is re-summarized into the *same* note path. A
session is left alone until idle for `SETTLE_MINUTES`, so a live conversation isn't
summarized mid-flight. Empty sessions are recorded and skipped.
- **Condensed, not truncated.** Tool output is dropped and each tool call becomes one line;
prose is kept. Typical compression is ~60× (271 MB of transcripts → 4.4 MB). Sessions over
300 KB condensed are summarized in parts, then combined.
- **Subagent transcripts** (`<session>/subagents/**`) are part of their parent session and are
not synced separately.
## Note format
```yaml
type: claude-conversation
source: claude-code # or claude-ai
session_id: 3f9c2a1e-…
title: Fix flaky CI cache after runner upgrade
project: my-service # basename of the working directory
branch: …
started / ended: …
prompts: 1 # user prompts, not tool-loop records
summarized_by: claude-haiku-4-5-20251001
tags: [claude, claude-code, ci, caching, …]
```
followed by a summary callout and **Outcomes / Decisions / Facts & gotchas / Open threads**,
and a `claude --resume <id>` line. `Claude/Index.md` lists every note, newest first.
+798
View File
@@ -0,0 +1,798 @@
#!/usr/bin/env python3
"""claude-obsidian-sync -- summarize Claude conversations into an Obsidian vault.
Reads Claude Code transcripts (~/.claude/projects/<project>/<session>.jsonl) and claude.ai
data exports, condenses each conversation, redacts secrets, summarizes it with an
OpenAI-compatible LLM endpoint (e.g. LiteLLM), and writes one note per conversation into
Obsidian through the Local REST API plugin. Standard library only.
claude-obsidian-sync stats # sizes and counts, no network
claude-obsidian-sync sync [--dry-run] # Claude Code sessions (incremental)
claude-obsidian-sync import-claude-ai EXPORT.zip|conversations.json
Configuration comes from the environment, falling back to ~/.config/claude-obsidian/config.env
(KEY=VALUE lines). Nothing environment-specific lives in this file. See README.md.
"""
from __future__ import annotations
import argparse
import concurrent.futures as cf
import datetime as dt
import fcntl
import hashlib
import json
import os
import pathlib
import re
import shutil
import subprocess
import sys
import threading
import time
import urllib.error
import urllib.parse
import urllib.request
import zipfile
HOME = pathlib.Path.home()
CONFIG_DIR = HOME / ".config" / "claude-obsidian"
# --------------------------------------------------------------------------- config
def load_config() -> dict:
cfg: dict[str, str] = {}
f = CONFIG_DIR / "config.env"
if f.exists():
for line in f.read_text().splitlines():
line = line.strip()
if line and not line.startswith("#") and "=" in line:
k, v = line.split("=", 1)
v = re.split(r"\s+#", v, maxsplit=1)[0] # inline comment; a '#' inside a value survives
cfg[k.strip()] = v.strip().strip('"').strip("'")
cfg.update({k: v for k, v in os.environ.items() if k in DEFAULTS or k.startswith("LLM_") or k.startswith("OBSIDIAN_")})
for k, v in DEFAULTS.items():
cfg.setdefault(k, v)
return cfg
DEFAULTS = {
"OBSIDIAN_API_URL": "http://127.0.0.1:27123",
"OBSIDIAN_API_KEY_FILE": str(CONFIG_DIR / "api-key"),
"LLM_BACKEND": "claude-cli", # claude-cli | openai
"CLAUDE_BIN": "claude",
"LLM_BASE_URL": "",
"LLM_API_KEY_FILE": "",
"LLM_API_KEY_ENV": "LITELLM_KEY",
"LLM_MODEL": "haiku",
"VAULT_FOLDER": "Claude",
"SETTLE_MINUTES": "30",
"CLAUDE_PROJECTS_DIR": str(HOME / ".claude" / "projects"),
"STATE_FILE": str(HOME / ".local" / "state" / "claude-obsidian" / "state.json"),
"JOBS": "1",
# Cap per run. Summaries draw on the same plan limits as interactive Claude use, so the
# timer drips a backlog instead of bursting it (a 107-session burst exhausted a session limit).
"MAX_PER_RUN": "10",
# Spend guards, in API list-price USD as reported by `claude -p` (total_cost_usd). Not the
# plan's own meter -- that is only visible in Claude Code's /usage -- but a faithful proxy
# for how much was consumed. Hitting one stops the run like a usage limit (exit 75).
"BUDGET_USD_PER_RUN": "0.50",
"BUDGET_USD_PER_DAY": "2.00",
# Pre-call estimate prices (USD per million tokens, list). A call is refused BEFORE it runs
# if the estimate would cross a budget. Deliberately NOT `claude -p --max-budget-usd`: that
# aborts after the tokens are spent and throws the result away (measured: $0.26 for nothing).
"LLM_PRICE_IN_PER_M": "1.0",
"LLM_PRICE_OUT_PER_M": "5.0",
}
def secret(cfg: dict, file_key: str, env_key: str | None = None) -> str:
path = cfg.get(file_key)
if path:
p = pathlib.Path(os.path.expanduser(path))
if p.exists():
return p.read_text().strip()
if env_key and os.environ.get(cfg.get(env_key, "") or ""):
return os.environ[cfg[env_key]].strip()
raise SystemExit(f"missing secret: set {file_key} (or {env_key})")
# --------------------------------------------------------------------------- redaction
# Applied to everything sent to the LLM AND to the note written. Over-redaction is the
# accepted failure mode: a summary missing a token is fine, a vault holding one is not.
_REDACTIONS = [
(re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----.*?-----END [A-Z ]*PRIVATE KEY-----", re.S), "[REDACTED PRIVATE KEY]"),
(re.compile(r"(?i)\b(bearer|token)\s+[A-Za-z0-9._~+/=-]{16,}"), r"\1 [REDACTED]"),
(re.compile(r"(?i)((?:password|passwd|pwd|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|client[_-]?secret|NB_API_KEY|PAT)[\"']?\s*[:=]\s*[\"']?)([^\s\"'`,;]{6,})"), r"\1[REDACTED]"),
(re.compile(r"\beyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}"), "[REDACTED JWT]"),
(re.compile(r"\b(?:sk|pk|rk)-[A-Za-z0-9_-]{20,}|\bgh[pousr]_[A-Za-z0-9]{30,}|\bglpat-[A-Za-z0-9_-]{20,}|\bxox[abpr]-[A-Za-z0-9-]{10,}"), "[REDACTED TOKEN]"),
# >=48 hex: API keys / sha256 tokens. Deliberately leaves 40-hex git SHAs and 32-hex md5s.
(re.compile(r"\b[a-fA-F0-9]{48,}\b"), "[REDACTED HEX]"),
# Random mixed-case+digit strings (generated passwords). No '/' so paths survive.
(re.compile(r"(?<![A-Za-z0-9_./-])(?=[A-Za-z0-9+]*[0-9])(?=[A-Za-z0-9+]*[a-z])(?=[A-Za-z0-9+]*[A-Z])[A-Za-z0-9+]{20,}={0,2}(?![A-Za-z0-9_./-])"), "[REDACTED]"),
]
def redact(text: str) -> str:
for pat, repl in _REDACTIONS:
text = pat.sub(repl, text)
return text
# --------------------------------------------------------------------------- parsing
_NOISE_BLOCKS = re.compile(
r"<(system-reminder|local-command-stdout|local-command-stderr|local-command-caveat|task-notification)>.*?</\1>",
re.S,
)
_CMD = re.compile(r"<command-name>(.*?)</command-name>.*?(?:<command-args>(.*?)</command-args>)?", re.S)
MAX_MSG = 6000
def _clip(text: str) -> str:
return text if len(text) <= MAX_MSG else text[:4000] + "\n[...clipped...]\n" + text[-1500:]
def _clean_user(text: str) -> str:
text = _NOISE_BLOCKS.sub("", text)
m = _CMD.search(text)
if m:
args = (m.group(2) or "").strip()
text = f"{m.group(1).strip()} {args}".strip()
text = re.sub(r"</?command-(message|args|name)>", "", text)
return text.strip()
def _tool_line(block: dict) -> str:
name = block.get("name", "tool")
inp = block.get("input") or {}
desc = inp.get("description") or inp.get("command") or inp.get("file_path") or inp.get("query") \
or inp.get("url") or inp.get("prompt") or inp.get("pattern") or ""
desc = " ".join(str(desc).split())[:160]
return f"[{name}: {desc}]" if desc else f"[{name}]"
def parse_claude_code(path: pathlib.Path) -> dict:
meta = {"session_id": path.stem, "cwd": None, "branch": None, "first": None, "last": None,
"title": None, "version": None, "prompts": 0}
turns: list[list[str]] = []
for line in path.open(encoding="utf-8", errors="replace"):
try:
r = json.loads(line)
except json.JSONDecodeError:
continue
t = r.get("type")
if t == "ai-title" and r.get("aiTitle"):
meta["title"] = r["aiTitle"]
if t == "custom-title" and r.get("customTitle"):
meta["title"] = r["customTitle"]
if t not in ("user", "assistant") or r.get("isSidechain") or r.get("isMeta"):
continue
ts = r.get("timestamp")
if ts:
meta["first"] = meta["first"] or ts
meta["last"] = ts
meta["cwd"] = meta["cwd"] or r.get("cwd")
meta["branch"] = r.get("gitBranch") or meta["branch"]
meta["version"] = r.get("version") or meta["version"]
content = (r.get("message") or {}).get("content")
if t == "user":
if isinstance(content, str):
text = content
elif isinstance(content, list):
text = "\n".join(b.get("text", "") for b in content if b.get("type") == "text")
else:
continue
text = _clean_user(text)
if text:
role = "SUMMARY-OF-EARLIER-CONTEXT" if r.get("isCompactSummary") else "USER"
turns.append([role, _clip(text)])
meta["prompts"] += role == "USER"
else:
parts = []
for b in content if isinstance(content, list) else []:
if b.get("type") == "text" and b.get("text", "").strip():
parts.append(_clip(b["text"].strip()))
elif b.get("type") == "tool_use":
parts.append(_tool_line(b))
if not parts:
continue
if turns and turns[-1][0] == "ASSISTANT": # a tool loop is many records
turns[-1][1] += "\n" + "\n".join(parts)
else:
turns.append(["ASSISTANT", "\n".join(parts)])
meta["transcript"] = "\n\n".join(f"{role}: {text}" for role, text in turns)
return meta
def parse_claude_ai_export(src: pathlib.Path) -> list[dict]:
"""claude.ai 'Export data' -> conversations.json. Tolerant of field-name drift."""
if src.suffix == ".zip":
with zipfile.ZipFile(src) as z:
name = next((n for n in z.namelist() if n.endswith("conversations.json")), None)
if not name:
raise SystemExit(f"no conversations.json inside {src}")
data = json.loads(z.read(name))
else:
data = json.loads(src.read_text())
out = []
for c in data if isinstance(data, list) else data.get("conversations", []):
turns, count = [], 0
for m in c.get("chat_messages") or c.get("messages") or []:
text = m.get("text") or ""
if not text and isinstance(m.get("content"), list):
text = "\n".join(b.get("text", "") for b in m["content"] if b.get("type") == "text")
for a in m.get("attachments") or []:
if a.get("extracted_content"):
text += f"\n[attachment {a.get('file_name', '')}]: {a['extracted_content'][:1500]}"
if not text.strip():
continue
role = "USER" if (m.get("sender") or m.get("role")) in ("human", "user") else "ASSISTANT"
turns.append(f"{role}: {_clip(text.strip())}")
count += role == "USER"
if not turns:
continue
out.append({
"session_id": c.get("uuid") or hashlib.sha1(json.dumps(c, sort_keys=True).encode()).hexdigest(),
"title": c.get("name") or None, "first": c.get("created_at"), "last": c.get("updated_at"),
"cwd": None, "branch": None, "version": None, "prompts": count,
"transcript": "\n\n".join(turns),
})
return out
# --------------------------------------------------------------------------- LLM
SYSTEM = """You write concise knowledge-base notes summarizing a conversation between a user and Claude.
The reader is the same user, months later, searching their notes. Capture what happened and
what was learned, not a play-by-play. Be specific: name the systems, files, commands, errors
and decisions. Never include passwords, API keys, tokens or other secrets, even if present;
refer to them generically ("the API key"). Text marked [REDACTED] must stay redacted.
Reply with ONLY a JSON object, no prose, with these keys:
"title": short specific title (<= 70 chars)
"summary": 3-6 sentences
"outcomes": list of what was actually built, fixed or answered
"decisions": list of choices made and why
"facts": list of reusable facts, gotchas, root causes worth remembering
"open_threads": list of unfinished work or follow-ups
"tags": 3-8 lowercase kebab-case topic tags
Use empty lists where nothing applies."""
CHUNK_SYSTEM = """You are condensing one part of a long conversation between a user and Claude so that a
later step can summarize the whole. Output dense bullet notes: goals, actions taken, results,
errors and their causes, decisions, open questions. Be specific. Never reproduce secrets."""
CHUNK_CHARS = 300_000
class Ledger:
"""Append-only record of every model call: tokens and list-price cost. Also enforces budgets."""
def __init__(self, cfg: dict):
self.path = pathlib.Path(os.path.expanduser(cfg["STATE_FILE"])).parent / "usage.jsonl"
self.run_budget = float(cfg["BUDGET_USD_PER_RUN"])
self.day_budget = float(cfg["BUDGET_USD_PER_DAY"])
self.lock = threading.Lock()
self.run_cost = 0.0
self.run_calls = 0
self.run_in = self.run_out = 0
def entries(self, since: dt.datetime | None = None) -> list[dict]:
if not self.path.exists():
return []
out = []
for line in self.path.read_text().splitlines():
try:
e = json.loads(line)
except json.JSONDecodeError:
continue
if since is None or dt.datetime.fromisoformat(e["ts"]) >= since:
out.append(e)
return out
def spent_since(self, hours: float) -> float:
since = dt.datetime.now(dt.timezone.utc) - dt.timedelta(hours=hours)
return sum(e.get("cost_usd") or 0 for e in self.entries(since))
def check(self, estimate: float = 0.0) -> None:
"""Called BEFORE each model call with that call's estimated cost -- so a budget is never overshot."""
with self.lock:
if self.run_cost + estimate > self.run_budget:
raise BudgetExhausted(f"run budget ${self.run_budget:.2f}: ${self.run_cost:.3f} spent, "
f"next call est. ${estimate:.3f}")
day = self.spent_since(24)
if day + estimate > self.day_budget:
raise BudgetExhausted(f"24h budget ${self.day_budget:.2f}: ${day:.3f} spent, next call est. ${estimate:.3f}")
def record(self, label: str, model: str, usage: dict, cost: float | None) -> None:
e = {"ts": dt.datetime.now(dt.timezone.utc).isoformat(timespec="seconds"), "item": label, "model": model,
"input": (usage.get("input_tokens") or 0) + (usage.get("cache_creation_input_tokens") or 0)
+ (usage.get("cache_read_input_tokens") or 0),
"output": usage.get("output_tokens") or 0, "cost_usd": cost}
with self.lock:
self.path.parent.mkdir(parents=True, exist_ok=True)
with self.path.open("a") as f:
f.write(json.dumps(e) + "\n")
self.run_calls += 1
self.run_cost += cost or 0
self.run_in += e["input"]
self.run_out += e["output"]
def report(self) -> str:
return (f"usage this run: {self.run_calls} model calls, {self.run_in:,} in / {self.run_out:,} out tokens, "
f"${self.run_cost:.3f} list-price equiv | last 24h ${self.spent_since(24):.3f} of ${self.day_budget:.2f}")
class BudgetExhausted(RuntimeError):
"""A configured spend budget is reached -- stop the run the same way as a usage limit."""
class QuotaExhausted(RuntimeError):
"""The plan/API usage limit is hit. Every further call would fail the same way -- stop the run."""
_QUOTA = re.compile(r"(?i)hit your (?:session |usage |weekly )?limit|usage limit|quota|credit balance|resets? (?:at )?\d")
class LLM:
"""Two backends.
claude-cli (default): headless Claude Code on the user's own plan -- no API key to manage.
--no-session-persistence is load-bearing: without it every summary writes a new transcript
into ~/.claude/projects and the next sync would summarize its own summarizing.
openai: any OpenAI-compatible /chat/completions endpoint (LiteLLM, local models).
"""
def __init__(self, cfg: dict, ledger: "Ledger | None" = None):
self.ledger = ledger or Ledger(cfg)
self.price_in = float(cfg["LLM_PRICE_IN_PER_M"]) / 1e6
self.price_out = float(cfg["LLM_PRICE_OUT_PER_M"]) / 1e6
self.label = threading.local()
self.backend = cfg["LLM_BACKEND"]
self.model = cfg["LLM_MODEL"]
self.resolved = cfg["LLM_MODEL"]
if self.backend == "claude-cli":
self.bin = shutil.which(os.path.expanduser(cfg["CLAUDE_BIN"]))
if not self.bin:
raise SystemExit(f"claude CLI not found ({cfg['CLAUDE_BIN']})")
self.workdir = pathlib.Path(os.path.expanduser(cfg["STATE_FILE"])).parent # no project CLAUDE.md here
self.workdir.mkdir(parents=True, exist_ok=True)
elif self.backend == "openai":
if not cfg["LLM_BASE_URL"]:
raise SystemExit("LLM_BASE_URL is not set (see config.env.example)")
self.url = cfg["LLM_BASE_URL"].rstrip("/") + "/chat/completions"
self.key = secret(cfg, "LLM_API_KEY_FILE", "LLM_API_KEY_ENV")
else:
raise SystemExit(f"unknown LLM_BACKEND {self.backend!r}")
def estimate(self, system: str, user: str, max_tokens: int) -> float:
# ~3 chars/token over-estimates input (measured 3.1 on transcripts); assume the full output;
# +20% margin for anything unmeasured. Errs high on purpose: refusing early beats overshooting.
return 1.2 * ((len(system) + len(user)) / 3.0 * self.price_in + max_tokens * self.price_out)
def chat(self, system: str, user: str, max_tokens: int = 2500) -> str:
self.ledger.check(self.estimate(system, user, max_tokens))
return self._chat_cli(system, user) if self.backend == "claude-cli" else self._chat_openai(system, user, max_tokens)
def _chat_cli(self, system: str, user: str) -> str:
# Cost levers, measured on one session: $0.057 -> $0.015 with identical summary content.
# alwaysThinkingEnabled=false -- extended thinking was ~85% of output tokens
# DISABLE_PROMPT_CACHING=1 -- Claude Code writes prompts to the 1h cache at 2x input
# price; a one-shot call never reads it back
# CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 -- otherwise a hidden side request re-reads the
# whole prompt (~20 tokens out, title-generation shaped),
# doubling billed input: 9,638 vs 4,663 visible tokens.
cmd = [self.bin, "-p", "--model", self.model, "--no-session-persistence", "--tools", "",
"--strict-mcp-config", "--output-format", "json",
"--settings", json.dumps({"alwaysThinkingEnabled": False}), "--system-prompt", system]
env = dict(os.environ, DISABLE_PROMPT_CACHING="1", CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC="1")
last = ""
for attempt in range(5):
try:
proc = subprocess.run(cmd, input=user, capture_output=True, text=True, timeout=900, cwd=self.workdir, env=env)
except subprocess.TimeoutExpired:
last = "timeout"
else:
try:
d = json.loads(proc.stdout)
except json.JSONDecodeError:
d = {"is_error": True, "result": (proc.stdout or proc.stderr)[-300:]}
if d.get("usage") or d.get("total_cost_usd"): # failed calls can still cost
# modelUsage is what was BILLED across every request the CLI made; `usage` is only
# the visible turn and undercounted by half before the nonessential-traffic fix.
mu = list((d.get("modelUsage") or {}).values())
billed = {"input_tokens": sum((v.get("inputTokens") or 0) + (v.get("cacheReadInputTokens") or 0)
+ (v.get("cacheCreationInputTokens") or 0) for v in mu),
"output_tokens": sum(v.get("outputTokens") or 0 for v in mu)} if mu else (d.get("usage") or {})
self.ledger.record(getattr(self.label, "v", "?"), self.model, billed, d.get("total_cost_usd"))
if not d.get("is_error") and d.get("result"):
used = list((d.get("modelUsage") or {}).keys())
if used:
self.resolved = used[0]
return d["result"]
last = str(d.get("result") or proc.stderr)[-300:]
if _QUOTA.search(last):
raise QuotaExhausted(last.strip())
if not re.search(r"(?i)rate.?limit|overloaded|429|529|timeout|temporar", last):
break # not transient -- do not hammer
time.sleep(2 ** attempt * 10)
raise RuntimeError(f"claude -p failed: {last}")
def _chat_openai(self, system: str, user: str, max_tokens: int) -> str:
body = json.dumps({"model": self.model, "temperature": 0.2, "max_tokens": max_tokens,
"messages": [{"role": "system", "content": system},
{"role": "user", "content": user}]}).encode()
for attempt in range(5):
req = urllib.request.Request(self.url, data=body, method="POST", headers={
"Authorization": f"Bearer {self.key}", "Content-Type": "application/json"})
try:
with urllib.request.urlopen(req, timeout=600) as r:
d = json.load(r)
u = d.get("usage") or {}
self.ledger.record(getattr(self.label, "v", "?"), self.model,
{"input_tokens": u.get("prompt_tokens"), "output_tokens": u.get("completion_tokens")}, None)
return d["choices"][0]["message"]["content"]
except urllib.error.HTTPError as e:
detail = e.read()[:300].decode(errors="replace")
if _QUOTA.search(detail):
raise QuotaExhausted(detail) from None
if e.code in (429, 500, 502, 503, 504, 529) and attempt < 4:
time.sleep(2 ** attempt * 5)
continue
raise RuntimeError(f"LLM HTTP {e.code}: {detail}") from None
except (urllib.error.URLError, TimeoutError) as e:
if attempt < 4:
time.sleep(2 ** attempt * 5)
continue
raise RuntimeError(f"LLM unreachable: {e}") from None
raise RuntimeError("LLM retries exhausted")
def summarize(self, meta: dict) -> dict:
text = redact(meta["transcript"])
if len(text) > CHUNK_CHARS: # map-reduce for long sessions
chunks, cur = [], []
size = 0
for part in text.split("\n\n"):
if size + len(part) > CHUNK_CHARS and cur:
chunks.append("\n\n".join(cur)); cur, size = [], 0
cur.append(part); size += len(part) + 2
chunks.append("\n\n".join(cur))
notes = [self.chat(CHUNK_SYSTEM, f"Part {i + 1} of {len(chunks)}:\n\n{c}", 3000)
for i, c in enumerate(chunks)]
text = "Condensed notes from each part of a long conversation, in order:\n\n" + \
"\n\n---\n\n".join(notes)
header = f"Working directory: {meta.get('cwd')}\nGit branch: {meta.get('branch')}\n" \
f"Existing title: {meta.get('title')}\n\n"
raw = self.chat(SYSTEM, header + text)
m = re.search(r"\{.*\}", raw, re.S)
if not m:
raise RuntimeError(f"LLM did not return JSON: {raw[:200]!r}")
out = json.loads(m.group(0))
for k in ("outcomes", "decisions", "facts", "open_threads", "tags"):
out[k] = [redact(str(x)) for x in (out.get(k) or []) if str(x).strip()]
out["summary"] = redact(str(out.get("summary", "")))
out["title"] = redact(str(out.get("title") or meta.get("title") or meta["session_id"]))
return out
# --------------------------------------------------------------------------- vault
class Vault:
def __init__(self, cfg: dict):
# OBSIDIAN_API_URL may list several URLs, comma-separated, tried in order at ping() -- e.g. the
# permanent address first and a port-forward second, so one config works either way.
self.candidates = [u.strip().rstrip("/") for u in cfg["OBSIDIAN_API_URL"].split(",") if u.strip()]
self.base = self.candidates[0]
self.key = secret(cfg, "OBSIDIAN_API_KEY_FILE")
def put(self, path: str, content: str) -> None:
req = urllib.request.Request(f"{self.base}/vault/{urllib.parse.quote(path)}",
data=content.encode(), method="PUT", headers={
"Authorization": f"Bearer {self.key}",
"Content-Type": "text/markdown"})
with urllib.request.urlopen(req, timeout=60) as r:
if r.status not in (200, 201, 204):
raise RuntimeError(f"vault PUT {path}: HTTP {r.status}")
def ping(self) -> None:
"""Checked BEFORE any summarizing, so an unreachable vault never spends usage limits."""
errors = []
for base in self.candidates:
req = urllib.request.Request(f"{base}/", headers={"Authorization": f"Bearer {self.key}"})
try:
with urllib.request.urlopen(req, timeout=10) as r:
info = json.load(r)
self.base = base
break
except (urllib.error.URLError, TimeoutError, OSError, ValueError) as e:
errors.append(f"{base}: {getattr(e, 'reason', e)}")
else:
raise SystemExit("Obsidian REST API unreachable -- " + "; ".join(errors))
print(f"vault: {self.base}", flush=True)
if not info.get("authenticated"):
raise SystemExit("Obsidian REST API reachable but the API key was rejected")
_BAD = re.compile(r'[\\/:*?"<>|#^\[\]]+')
def note_path(cfg: dict, kind: str, meta: dict, title: str) -> str:
when = (meta.get("first") or dt.datetime.now(dt.timezone.utc).isoformat())[:10]
slug = " ".join(_BAD.sub(" ", title).split())[:60].strip() or "untitled"
return f"{cfg['VAULT_FOLDER']}/{kind}/{when[:4]}/{when} {slug} ({meta['session_id'][:8]}).md"
def render(meta: dict, s: dict, source: str, model: str, src_file: str | None) -> str:
project = pathlib.Path(meta["cwd"]).name if meta.get("cwd") else None
tags = ["claude", source] + [t for t in s["tags"] if t not in ("claude", source)]
fm = {
"type": "claude-conversation", "source": source, "session_id": meta["session_id"],
"title": s["title"], "project": project, "cwd": meta.get("cwd"), "branch": meta.get("branch"),
"started": meta.get("first"), "ended": meta.get("last"), "prompts": meta.get("prompts"),
"summarized_by": model, "synced": dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
"tags": tags,
}
lines = ["---"] + [f"{k}: {json.dumps(v)}" for k, v in fm.items() if v not in (None, "")] + ["---", "",
f"# {s['title']}", "", "> [!summary]"] + [f"> {l}" for l in s["summary"].splitlines() or [""]]
for heading, key in (("Outcomes", "outcomes"), ("Decisions", "decisions"),
("Facts & gotchas", "facts"), ("Open threads", "open_threads")):
if s[key]:
lines += ["", f"## {heading}", ""] + [f"- {x}" for x in s[key]]
lines += ["", "---"]
if source == "claude-code":
lines.append(f"Resume: `claude --resume {meta['session_id']}`" + (f" · transcript `{src_file}`" if src_file else ""))
else:
lines.append(f"claude.ai conversation `{meta['session_id']}`")
return "\n".join(lines) + "\n"
# --------------------------------------------------------------------------- state
class State:
def __init__(self, path: str):
self.path = pathlib.Path(os.path.expanduser(path))
self.lock = threading.Lock()
self.data = json.loads(self.path.read_text()) if self.path.exists() else {}
def save(self) -> None:
self.path.parent.mkdir(parents=True, exist_ok=True)
tmp = self.path.with_suffix(".tmp")
tmp.write_text(json.dumps(self.data, indent=1, sort_keys=True))
tmp.replace(self.path)
def record(self, key: str, value: dict) -> None:
with self.lock:
self.data[key] = value
self.save()
def write_index(cfg: dict, vault: Vault, state: State) -> None:
rows = sorted((v for v in state.data.values() if v.get("note")), key=lambda v: v.get("started") or "", reverse=True)
out = ["---", 'type: "claude-index"', "---", "", "# Claude conversations", "",
f"{len(rows)} conversations. Regenerated by claude-obsidian-sync.", "",
"| Date | Title | Project | Source |", "|---|---|---|---|"]
for v in rows:
link = v["note"][:-3]
title = v.get("title", "").replace("|", "-")
out.append(f"| {(v.get('started') or '')[:10]} | [[{link}\\|{title}]] | {v.get('project') or ''} | {v.get('source')} |")
vault.put(f"{cfg['VAULT_FOLDER']}/Index.md", "\n".join(out) + "\n")
# --------------------------------------------------------------------------- commands
def top_level_sessions(cfg: dict) -> list[pathlib.Path]:
root = pathlib.Path(os.path.expanduser(cfg["CLAUDE_PROJECTS_DIR"]))
return sorted(root.glob("*/*.jsonl"), key=lambda p: p.stat().st_mtime) # nested = subagents
def process(cfg, llm, vault, state, key, meta, source, kind, fingerprint, src_file, dry_run) -> str:
# Cache the summary until the note is written, so a vault failure never pays for it twice.
cache = pathlib.Path(os.path.expanduser(cfg["STATE_FILE"])).parent / "cache" / \
f"{hashlib.sha1(key.encode()).hexdigest()[:16]}-{hashlib.sha1(fingerprint.encode()).hexdigest()[:8]}.json"
llm.label.v = key
if cache.exists():
s = json.loads(cache.read_text())
else:
s = llm.summarize(meta)
cache.parent.mkdir(parents=True, exist_ok=True)
cache.write_text(json.dumps(s))
prev = state.data.get(key, {})
path = prev.get("note") or note_path(cfg, kind, meta, s["title"]) # stable across re-syncs
body = render(meta, s, source, llm.resolved, src_file)
if dry_run == "prefetch":
return f"summary cached ({path})"
if dry_run:
print(f"\n===== {path} =====\n{body}")
return path
vault.put(path, body)
cache.unlink(missing_ok=True)
state.record(key, {"note": path, "title": s["title"], "fingerprint": fingerprint, "source": source,
"started": meta.get("first"),
"project": pathlib.Path(meta["cwd"]).name if meta.get("cwd") else None})
return path
SKIPPED = "(empty, skipped)"
EX_TEMPFAIL = 75 # "try again later" -- systemd logs it, the next timer run retries
def run_jobs(cfg, work, fn) -> tuple[int, int, str | None, int]:
ok = failed = skipped = 0
quota = None
stop = threading.Event() # Future.cancel() cannot stop a job a worker already picked up
def guarded(*w):
if stop.is_set():
raise cf.CancelledError()
try:
return fn(*w)
except (QuotaExhausted, BudgetExhausted):
stop.set()
raise
with cf.ThreadPoolExecutor(max_workers=int(cfg["JOBS"])) as ex:
futs = {ex.submit(guarded, *w): w for w in work}
for fut in cf.as_completed(futs):
label = futs[fut][0]
try:
result = fut.result()
if result == SKIPPED:
print(f"skip {label} (no conversation content)", flush=True); skipped += 1
else:
print(f"ok {label} -> {result}", flush=True); ok += 1
except (QuotaExhausted, BudgetExhausted) as e:
if quota is None:
quota = str(e)
for f in futs: # don't launch the rest
f.cancel()
except cf.CancelledError:
pass
except Exception as e: # keep going; next run retries
print(f"FAIL {label}: {e}", file=sys.stderr, flush=True); failed += 1
return ok, failed, quota, skipped
def finish(ok: int, failed: int, quota: str | None, dry_run, remaining: int, ledger: "Ledger | None" = None) -> None:
verb = {"prefetch": "summarized into cache (vault untouched)"}.get(dry_run) or \
("rendered (dry run, nothing written)" if dry_run else "written")
print(f"done: {ok} {verb}, {failed} failed, {remaining} still queued")
if ledger:
print(ledger.report())
if quota:
print(f"stopped: {quota}", file=sys.stderr)
sys.exit(EX_TEMPFAIL)
sys.exit(1 if failed else 0)
def cmd_usage(cfg, args) -> None:
led = Ledger(cfg)
rows = led.entries()
if not rows:
print("no model calls recorded yet"); return
for label, hours in (("last 5h", 5), ("last 24h", 24), ("last 7d", 168)):
since = dt.datetime.now(dt.timezone.utc) - dt.timedelta(hours=hours)
es = [e for e in rows if dt.datetime.fromisoformat(e["ts"]) >= since]
print(f"{label:>9}: {len(es):4d} calls {sum(e['input'] for e in es):>10,} in "
f"{sum(e['output'] for e in es):>8,} out ${sum(e.get('cost_usd') or 0 for e in es):.3f}")
print(f"budgets: ${led.run_budget:.2f}/run, ${led.day_budget:.2f}/24h (list-price equiv; plan meter: /usage in Claude Code)")
def cmd_stats(cfg, args) -> None:
total = 0
rows = []
for p in top_level_sessions(cfg):
m = parse_claude_code(p)
n = len(redact(m["transcript"]))
total += n
rows.append((n, p.stat().st_size, m["prompts"], m["title"] or p.stem))
rows.sort(reverse=True)
for n, raw, msgs, title in rows[: args.top]:
print(f"{raw / 1e6:7.2f} MB raw -> {n / 1e3:7.1f} KB condensed {msgs:4d} prompts {title[:60]}")
print(f"\n{len(rows)} sessions, {total / 1e6:.1f} MB condensed (~{total / 4 / 1e6:.2f}M tokens in), "
f"{sum(1 for r in rows if r[0] > CHUNK_CHARS)} need map-reduce")
def run_lock(cfg: dict):
"""One run at a time: a timer run overlapping a manual one would race on state and ledger files."""
path = pathlib.Path(os.path.expanduser(cfg["STATE_FILE"])).parent / "run.lock"
path.parent.mkdir(parents=True, exist_ok=True)
fh = path.open("w")
try:
fcntl.flock(fh, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
print("another claude-obsidian-sync run is in progress -- exiting")
sys.exit(0)
return fh # held until the process exits
def cmd_sync(cfg, args) -> None:
_lock = run_lock(cfg)
state = State(cfg["STATE_FILE"])
settle = dt.timedelta(minutes=int(cfg["SETTLE_MINUTES"]))
now = time.time()
work = []
for p in top_level_sessions(cfg):
if args.session and p.stem != args.session:
continue
st = p.stat()
fp = f"{st.st_size}:{int(st.st_mtime)}"
key = f"claude-code:{p.stem}"
if not args.force and state.data.get(key, {}).get("fingerprint") == fp:
continue
if not (args.force or args.session) and now - st.st_mtime < settle.total_seconds():
continue # still being written
work.append((p, fp, key))
total = len(work)
cap = args.limit or int(cfg["MAX_PER_RUN"])
work = work[-cap:] if not args.oldest else work[:cap] # newest first by default
print(f"{total} session(s) pending, summarizing {len(work)} this run", flush=True)
if not work:
return
prefetch = getattr(args, "prefetch", False)
llm, vault = LLM(cfg), (None if args.dry_run or prefetch else Vault(cfg))
if vault:
vault.ping()
mode = "prefetch" if prefetch else args.dry_run
def one(p, fp, key):
meta = parse_claude_code(p)
if not meta["transcript"].strip():
if not args.dry_run and not prefetch:
state.record(key, {"note": None, "fingerprint": fp, "source": "claude-code"})
return SKIPPED
src = str(p).replace(str(HOME), "~")
return process(cfg, llm, vault, state, key, meta, "claude-code", "Code", fp, src, mode)
ok, failed, quota, skipped = run_jobs(cfg, [(w[0].name, *w) for w in work], lambda _l, p, fp, key: one(p, fp, key))
if vault and ok:
write_index(cfg, vault, state)
finish(ok, failed, quota, mode, total - ok - skipped, llm.ledger)
def cmd_import(cfg, args) -> None:
_lock = run_lock(cfg)
state = State(cfg["STATE_FILE"])
convs = parse_claude_ai_export(pathlib.Path(args.export))
work = []
for m in convs:
key = f"claude-ai:{m['session_id']}"
fp = m.get("last") or str(len(m["transcript"]))
if args.force or state.data.get(key, {}).get("fingerprint") != fp:
work.append((m, fp, key))
total = len(work)
work = work[: args.limit or int(cfg["MAX_PER_RUN"])]
print(f"{len(convs)} conversations in export, {total} new or changed, summarizing {len(work)} this run", flush=True)
if not work:
return
llm, vault = LLM(cfg), (None if args.dry_run else Vault(cfg))
if vault:
vault.ping()
ok, failed, quota, skipped = run_jobs(cfg, [((m.get("title") or m["session_id"])[:50], m, fp, key) for m, fp, key in work],
lambda _l, m, fp, key: process(cfg, llm, vault, state, key, m, "claude-ai", "Web",
fp, None, args.dry_run))
if vault and ok:
write_index(cfg, vault, state)
finish(ok, failed, quota, args.dry_run, total - ok - skipped, llm.ledger)
def main() -> None:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = ap.add_subparsers(dest="cmd", required=True)
sub.add_parser("usage", help="model calls, tokens and spend from the local ledger")
s = sub.add_parser("stats", help="condensed sizes, no network")
s.add_argument("--top", type=int, default=15)
s = sub.add_parser("sync", help="summarize Claude Code sessions")
s.add_argument("--dry-run", action="store_true", help="print notes instead of writing them")
s.add_argument("--prefetch", action="store_true",
help="summarize into the local cache without touching the vault; a later sync writes them at no extra cost")
s.add_argument("--force", action="store_true", help="ignore state and the settle window")
s.add_argument("--session", help="only this session id")
s.add_argument("--limit", type=int, help="max sessions this run (default MAX_PER_RUN)")
s.add_argument("--oldest", action="store_true", help="take the oldest pending sessions instead of the newest")
s = sub.add_parser("import-claude-ai", help="import a claude.ai data export")
s.add_argument("export", help="the export .zip or its conversations.json")
s.add_argument("--dry-run", action="store_true")
s.add_argument("--force", action="store_true")
s.add_argument("--limit", type=int, help="max conversations this run (default MAX_PER_RUN)")
args = ap.parse_args()
cfg = load_config()
{"stats": cmd_stats, "usage": cmd_usage, "sync": cmd_sync, "import-claude-ai": cmd_import}[args.cmd](cfg, args)
if __name__ == "__main__":
main()
+28
View File
@@ -0,0 +1,28 @@
# ~/.config/claude-obsidian/config.env (chmod 600; never commit the real one)
# Every key can also be set as an environment variable, which wins.
# Obsidian Local REST API plugin -- the plain-HTTP server (Settings -> Local REST API ->
# Enable HTTP server). Point this at wherever that port is reachable from this machine.
# Several URLs, comma-separated, are tried in order -- e.g. a permanent address, then a port-forward.
OBSIDIAN_API_URL=http://obsidian.example.internal:27123,http://127.0.0.1:27123
OBSIDIAN_API_KEY_FILE=~/.config/claude-obsidian/api-key
# Summarizer. claude-cli = headless `claude -p` on your own Claude plan (no API key needed).
LLM_BACKEND=claude-cli
LLM_MODEL=haiku # sonnet writes noticeably sharper notes for ~1-3x the usage; set prices to match below
# LLM_PRICE_IN_PER_M=1.0 # used only for the pre-call budget estimate (haiku list; sonnet 3.0 / 15.0)
# LLM_PRICE_OUT_PER_M=5.0
# CLAUDE_BIN=claude
# Or any OpenAI-compatible endpoint (LiteLLM, a local model):
# LLM_BACKEND=openai
# LLM_BASE_URL=https://litellm.example/v1
# LLM_MODEL=some-model
# LLM_API_KEY_FILE=~/.config/claude-obsidian/llm-key # or LLM_API_KEY_ENV=LITELLM_KEY
VAULT_FOLDER=Claude
SETTLE_MINUTES=30 # leave sessions alone until they have been idle this long
JOBS=1
MAX_PER_RUN=10 # summaries share your plan's usage limits; the timer drips a backlog
BUDGET_USD_PER_RUN=0.50 # list-price USD, refused pre-flight; see `claude-obsidian-sync usage`
BUDGET_USD_PER_DAY=2.00
+14
View File
@@ -0,0 +1,14 @@
#!/bin/sh
# Install claude-obsidian-sync for the current user: binary into ~/.local/bin, systemd user
# timer enabled. Idempotent. Does not create config or secrets -- see README.md.
set -eu
HERE="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
mkdir -p "$HOME/.local/bin" "$HOME/.config/systemd/user" "$HOME/.config/claude-obsidian"
chmod 700 "$HOME/.config/claude-obsidian"
ln -sf "$HERE/claude-obsidian-sync" "$HOME/.local/bin/claude-obsidian-sync"
cp "$HERE/systemd/claude-obsidian-sync.service" "$HERE/systemd/claude-obsidian-sync.timer" "$HOME/.config/systemd/user/"
[ -f "$HOME/.config/claude-obsidian/config.env" ] || { cp "$HERE/config.env.example" "$HOME/.config/claude-obsidian/config.env"; chmod 600 "$HOME/.config/claude-obsidian/config.env"; echo "created config.env from the example -- edit it"; }
systemctl --user daemon-reload
systemctl --user enable --now claude-obsidian-sync.timer
echo "installed. check: systemctl --user list-timers claude-obsidian-sync.timer"
echo "timers only run while logged in unless lingering is on: loginctl enable-linger \"\$USER\""
@@ -0,0 +1,11 @@
[Unit]
Description=Summarize Claude conversations into Obsidian
After=network-online.target
[Service]
Type=oneshot
ExecStart=%h/.local/bin/claude-obsidian-sync sync
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
# 75 = usage limit reached. The run stopped cleanly and the next timer run resumes -- not a failure.
SuccessExitStatus=75
Nice=10
@@ -0,0 +1,10 @@
[Unit]
Description=Hourly Claude -> Obsidian conversation sync
[Timer]
OnCalendar=hourly
RandomizedDelaySec=10m
Persistent=true
[Install]
WantedBy=timers.target
+22 -1
View File
@@ -30,4 +30,25 @@ say "fetch.prune = true (stale remote-tracking refs cleaned on every fetch)"
git config --global credential.helper store git config --global credential.helper store
say "credential.helper = store (~/.git-credentials, created on first auth)" say "credential.helper = store (~/.git-credentials, created on first auth)"
say "Done. On the next HTTPS push/pull to a new host, git prompts once and caches the credential in ~/.git-credentials." # Our own Gitea authenticates from $GITEA_KEY (exported by secrets/ from the
# vault), not from ~/.git-credentials.
#
# WHY: git never reads GITEA_KEY by itself. With only `store`, the token was
# whatever got typed at the first prompt, copied into ~/.git-credentials, and
# it silently went stale when the vault's key was rotated — or, on a box that
# never pushed, there was no credential at all and the push just failed. Now
# `opc GITEA_KEY` is the only step when the token changes.
#
# The empty helper first clears the inherited `store` for this host, so a
# rejected token is never cached and a stale cached one can never win. Gitea
# accepts the token as the password; the username is informational (GITEA_USER
# overrides it). With GITEA_KEY unset the helper prints nothing and git falls
# back to prompting, as before.
GITEA_URL="https://git.nic-oconnor.com"
git config --global --unset-all "credential.$GITEA_URL.helper" 2>/dev/null || true
git config --global --add "credential.$GITEA_URL.helper" ''
git config --global --add "credential.$GITEA_URL.helper" \
'!f() { test "$1" = get && test -n "${GITEA_KEY:-}" && printf "username=%s\npassword=%s\n" "${GITEA_USER:-$(id -un)}" "$GITEA_KEY"; }; f'
say "credential helper for $GITEA_URL = \$GITEA_KEY (from secrets.env)"
say "Done. $GITEA_URL uses \$GITEA_KEY; other HTTPS hosts prompt once and cache in ~/.git-credentials."
Regular → Executable
+14 -2
View File
@@ -120,11 +120,23 @@ say "Key stored at $KEYFILE (0600)."
# bootstrap run — and then models and skills silently vanish, because sync.sh # bootstrap run — and then models and skills silently vanish, because sync.sh
# never runs and OPENCODE_CONFIG is never exported. ~/.zshrc.local is the # never runs and OPENCODE_CONFIG is never exported. ~/.zshrc.local is the
# unmanaged file the managed zshrc sources last, so the hook persists. # unmanaged file the managed zshrc sources last, so the hook persists.
case "${SHELL:-}" in #
# Do NOT pick the file from $SHELL alone. On a fresh box bootstrap runs this
# from the user's ORIGINAL shell: zsh/ has just switched the login shell to zsh,
# but $SHELL in this process still says bash. The hook then went into ~/.bashrc,
# below zsh/'s auto-exec hook, so every new terminal exec'd into zsh before
# reaching it, zsh never sourced sync.sh, and opencode came up unconfigured.
# The dotfiles-managed ~/.zshrc (or zsh as the account's login shell) wins.
_login_shell="$(getent passwd "$(id -un)" 2>/dev/null | cut -d: -f7)"
if grep -qs 'Managed by dotfiles/zsh/install.sh' "$HOME/.zshrc"; then
RC="$HOME/.zshrc.local"
else
case "${_login_shell:-${SHELL:-}}" in
*zsh) RC="$HOME/.zshrc.local" ;; *zsh) RC="$HOME/.zshrc.local" ;;
*bash) RC="$HOME/.bashrc" ;; *bash) RC="$HOME/.bashrc" ;;
*) [ -f "$HOME/.zshrc" ] && RC="$HOME/.zshrc.local" || RC="$HOME/.bashrc" ;; *) [ -f "$HOME/.zshrc" ] && RC="$HOME/.zshrc.local" || RC="$HOME/.bashrc" ;;
esac esac
fi
touch "$RC" touch "$RC"
# It sits alongside the user's secrets; keep it non-readable by others. # It sits alongside the user's secrets; keep it non-readable by others.
[ "$RC" = "$HOME/.zshrc.local" ] && chmod 0600 "$RC" 2>/dev/null || true [ "$RC" = "$HOME/.zshrc.local" ] && chmod 0600 "$RC" 2>/dev/null || true
+72 -6
View File
@@ -315,21 +315,50 @@ OUTPUT_RESERVE_CAP = int(os.environ.get("MCP_SYNC_PY_OUTPUT_CAP") or 32768)
CONTEXT_SAFETY_MARGIN = 1024 CONTEXT_SAFETY_MARGIN = 1024
# NEVER trust /model/info's number TYPES. The litellm-helm chart renders the
# proxy config with Go's toYaml, which writes 0.000005 as `5e-06`; LiteLLM
# parses it with PyYAML (YAML 1.1), which only reads an exponent float if it has
# a dot. So `5e-06` arrives as the STRING "5e-06" while `2.5e-05` is a float.
# `"5e-06" * 1_000_000` is not an error in Python — it is the string repeated a
# million times. That wrote a 20 MB opencode.gateway.json with a string where
# opencode requires a number, opencode refused to start, and every new shell
# redeployed it. Coerce every number through here; anything that is not a
# finite number is dropped, never passed through.
def num(v):
if isinstance(v, bool):
return None
if isinstance(v, (int, float)):
f = float(v)
elif isinstance(v, str) and len(v) <= 64:
try:
f = float(v.strip())
except ValueError:
return None
else:
return None
if f != f or f in (float("inf"), float("-inf")):
return None
return int(f) if isinstance(v, int) or f.is_integer() else f
def spec_for(mi): def spec_for(mi):
spec = {} spec = {}
limit = {} limit = {}
if mi.get("max_input_tokens"): max_in, max_out = num(mi.get("max_input_tokens")), num(mi.get("max_output_tokens"))
limit["context"] = max(1, mi["max_input_tokens"] - CONTEXT_SAFETY_MARGIN) if max_in and max_in > 0:
if mi.get("max_output_tokens"): limit["context"] = int(max(1, max_in - CONTEXT_SAFETY_MARGIN))
limit["output"] = min(mi["max_output_tokens"], OUTPUT_RESERVE_CAP) if max_out and max_out > 0:
limit["output"] = int(min(max_out, OUTPUT_RESERVE_CAP))
if limit: if limit:
spec["limit"] = limit spec["limit"] = limit
# LiteLLM prices PER TOKEN; opencode (like models.dev) uses $ per 1M tokens. # LiteLLM prices PER TOKEN; opencode (like models.dev) uses $ per 1M tokens.
cost = {} cost = {}
for src, dst in (("input_cost_per_token", "input"), for src, dst in (("input_cost_per_token", "input"),
("output_cost_per_token", "output")): ("output_cost_per_token", "output")):
if mi.get(src) is not None: per_token = num(mi.get(src))
cost[dst] = mi[src] * 1_000_000 if per_token is not None and per_token >= 0:
# round() so 5e-06 * 1e6 is 5.0, not 4.999999999999999.
cost[dst] = round(per_token * 1_000_000, 6)
if cost: if cost:
spec["cost"] = cost spec["cost"] = cost
for src, dst in (("supports_function_calling", "tool_call"), for src, dst in (("supports_function_calling", "tool_call"),
@@ -378,10 +407,47 @@ if not models:
except Exception: except Exception:
pass pass
# Validate EVERY spec before it is written, whatever its source. The carry-over
# path above re-reads the deployed file, so a bad value that was ever written
# (e.g. the 20 MB repeated-string cost) would otherwise be preserved forever;
# this makes the next sync heal it instead. opencode rejects the WHOLE config
# over one wrongly typed field, so dropping a bad field is always the better
# failure.
def clean_spec(spec):
if not isinstance(spec, dict):
return {}
out = {}
for group, keys, cast in (("limit", ("context", "output"), int),
("cost", ("input", "output", "cache_read", "cache_write"), float)):
vals = {}
for k in keys:
v = num((spec.get(group) or {}).get(k)) if isinstance(spec.get(group), dict) else None
if v is not None and v >= 0:
vals[k] = cast(v)
if vals:
out[group] = vals
for k, v in spec.items():
if k in ("limit", "cost"):
continue
if isinstance(v, (bool, int, float)) or (isinstance(v, str) and len(v) <= 256):
out[k] = v
elif isinstance(v, (dict, list)) and len(json.dumps(v)) <= 4096:
out[k] = v
return out
models = {k: clean_spec(v) for k, v in (models or {}).items()
if isinstance(k, str) and len(k) <= 256}
prov = cfg.get("provider", {}).get(provider) prov = cfg.get("provider", {}).get(provider)
if prov is not None and models: if prov is not None and models:
prov["models"] = models prov["models"] = models
# Last line of defence: the shared config is a few KB. Anything wildly larger is
# a bug upstream of here, and deploying it is what breaks opencode — keep the
# last-good file (or none) instead.
if len(json.dumps(cfg)) > 1_000_000:
raise SystemExit(0)
fd, tmp = tempfile.mkstemp(dir=os.path.dirname(dest)) fd, tmp = tempfile.mkstemp(dir=os.path.dirname(dest))
with os.fdopen(fd, "w") as f: with os.fdopen(fd, "w") as f:
json.dump(cfg, f, indent=2) json.dump(cfg, f, indent=2)
+155
View File
@@ -0,0 +1,155 @@
#!/usr/bin/env sh
# secrets/install.sh — write this user's 1Password vault item to a local,
# 0600 env file that the managed ~/.zshrc sources.
#
# WHY: secrets used to be pulled live in ~/.zshrc.local, one `op read` per
# field on EVERY new shell. That made each terminal wait on 1Password, broke
# every shell when the network or the service account was down, and meant each
# box carried its own hand-written copy of the loop. Now the vault is read once
# here (at bootstrap, or on demand with `ops`), and shells read a local file.
#
# Layout: one vault per user (default: the login name), holding one item
# (default: also the login name) whose field LABELS are the variable names —
# GITEA_KEY, LITELLM_KEY, PROXMOX_TOKEN_SECRET, ... Every such field is
# exported. Add a secret by adding a field (`opc NAME`), not by editing
# this repo.
#
# Files, all under ~/.config/dotfiles (0700), none in the repo:
# op.env OP_SERVICE_ACCOUNT_TOKEN, OP_VAULT, OP_ITEM (0600). Kept so a
# refresh needs no prompt. Deliberately NOT sourced by shells:
# the token stays out of every process's environment, and only
# the op* helpers in zsh/zshrc load it, in a subshell.
# secrets.env the exported fields (0600), replaced atomically, so a failed
# fetch leaves the previous copy in place.
#
# Prompts for the token on the controlling TTY the first time (bootstrap is
# often piped into sh, so stdin is not the terminal). Without a TTY, pass
# OP_SERVICE_ACCOUNT_TOKEN (and optionally OP_VAULT / OP_ITEM) in the env.
# Needs op and jq, which bin/ installs. No sudo.
set -eu
say() { printf '%s\n' "$*" >&2; }
CONF_DIR="${DOTFILES_CONFIG_DIR:-$HOME/.config/dotfiles}"
OP_ENV="$CONF_DIR/op.env"
SECRETS_ENV="$CONF_DIR/secrets.env"
MCP_SYNC_KEYFILE="$HOME/.config/mcp-sync/key"
umask 077
mkdir -p "$CONF_DIR"
chmod 0700 "$CONF_DIR"
PATH="$HOME/.local/bin:$PATH"
for _tool in op jq; do
if ! "$_tool" --version >/dev/null 2>&1; then
say "secrets: $_tool is not installed or does not run; run bin/install.sh first."
exit 1
fi
done
# --- 1. resolve the service account, vault and item -------------------------
# Precedence: the environment, then op.env, then a prompt (token) / defaults.
_env_token="${OP_SERVICE_ACCOUNT_TOKEN:-}"
_env_vault="${OP_VAULT:-}"
_env_item="${OP_ITEM:-}"
if [ -f "$OP_ENV" ]; then
# shellcheck source=/dev/null
. "$OP_ENV"
fi
OP_SERVICE_ACCOUNT_TOKEN="${_env_token:-${OP_SERVICE_ACCOUNT_TOKEN:-}}"
OP_VAULT="${_env_vault:-${OP_VAULT:-}}"
OP_ITEM="${_env_item:-${OP_ITEM:-}}"
_user="$(id -un)"
# ask <prompt> <default> — read one line from the TTY; echoes the default on empty.
ask() {
printf '%s [%s]: ' "$1" "$2" >&2
IFS= read -r _ans < /dev/tty || _ans=""
printf '%s' "${_ans:-$2}"
}
if [ -z "$OP_SERVICE_ACCOUNT_TOKEN" ]; then
if [ -f "$SECRETS_ENV" ]; then
say "secrets: no service-account token; keeping existing $SECRETS_ENV."
exit 0
fi
# Probe in a subshell: in POSIX sh a failed redirect on a special builtin
# like `:` exits the whole script, which would skip the message below.
if ! ( : < /dev/tty ) 2>/dev/null; then
say "secrets: no OP_SERVICE_ACCOUNT_TOKEN and no TTY to prompt."
say " re-run as: OP_SERVICE_ACCOUNT_TOKEN=ops_... $0"
exit 1
fi
printf '1Password service-account token (ops_..., input hidden): ' >&2
stty -echo < /dev/tty 2>/dev/null || true
IFS= read -r OP_SERVICE_ACCOUNT_TOKEN < /dev/tty || true
stty echo < /dev/tty 2>/dev/null || true
printf '\n' >&2
OP_SERVICE_ACCOUNT_TOKEN="$(printf '%s' "$OP_SERVICE_ACCOUNT_TOKEN" | tr -d ' \t\r\n')"
[ -n "$OP_SERVICE_ACCOUNT_TOKEN" ] || { say "secrets: no token entered."; exit 1; }
[ -n "$OP_VAULT" ] || OP_VAULT="$(ask 'Vault' "$_user")"
[ -n "$OP_ITEM" ] || OP_ITEM="$(ask 'Item' "$_user")"
fi
OP_VAULT="${OP_VAULT:-$_user}"
OP_ITEM="${OP_ITEM:-$_user}"
export OP_SERVICE_ACCOUNT_TOKEN
# --- 2. fetch the item once and render it -----------------------------------
# One `op item get` instead of one `op read` per field. Labels that are not
# valid shell identifiers (and the item's built-in notes field) are skipped
# rather than eval'd; values go through jq's @sh, so quotes, $ and newlines in
# a secret cannot break or inject into the file.
if ! _json="$(op item get "$OP_ITEM" --vault "$OP_VAULT" --reveal --format json)"; then
say "secrets: could not read item '$OP_ITEM' in vault '$OP_VAULT'."
[ -f "$SECRETS_ENV" ] && say "secrets: keeping the previous $SECRETS_ENV."
exit 1
fi
_skipped="$(printf '%s' "$_json" | jq -r '
[.fields[] | select(.purpose != "NOTES")
| select((.label // "") | test("^[A-Za-z_][A-Za-z0-9_]*$") | not) | .label]
| join(", ")')"
[ -z "$_skipped" ] || say "secrets: skipped fields whose labels are not variable names: $_skipped"
_tmp="$(mktemp "$CONF_DIR/secrets.env.XXXXXX")"
trap 'rm -f "$_tmp"' EXIT
{
printf '# Generated by dotfiles secrets/install.sh from op://%s/%s — do not edit.\n' "$OP_VAULT" "$OP_ITEM"
printf '# Refresh with: ops (change the vault with: opc NAME / opd NAME)\n'
printf '%s' "$_json" | jq -r '
.fields[]
| select(.purpose != "NOTES")
| select((.label // "") | test("^[A-Za-z_][A-Za-z0-9_]*$"))
| select((.value // "") != "")
| "export \(.label)=\(.value | @sh)"'
} > "$_tmp"
chmod 0600 "$_tmp"
_count="$(grep -c '^export ' "$_tmp" || true)"
if [ "$_count" -eq 0 ]; then
say "secrets: item '$OP_ITEM' has no exportable fields; keeping the previous file."
exit 1
fi
mv -f "$_tmp" "$SECRETS_ENV"
trap - EXIT
say "secrets: wrote $_count variables to $SECRETS_ENV (0600)."
# Remember the account only after it has actually worked.
{
printf 'OP_SERVICE_ACCOUNT_TOKEN=%s\n' "$(printf '%s' "$OP_SERVICE_ACCOUNT_TOKEN" | jq -Rr @sh)"
printf 'OP_VAULT=%s\n' "$(printf '%s' "$OP_VAULT" | jq -Rr @sh)"
printf 'OP_ITEM=%s\n' "$(printf '%s' "$OP_ITEM" | jq -Rr @sh)"
} > "$OP_ENV"
chmod 0600 "$OP_ENV"
# --- 3. keep mcp-sync's copy of the gateway key in step ----------------------
# mcp-config/sync.sh exports LITELLM_KEY from its own key file on every shell,
# AFTER secrets.env is loaded, so a key rotated in the vault would otherwise be
# silently overridden by the stale copy. Only update a file that already exists:
# creating it is mcp-config/install.sh's job.
_litellm="$(printf '%s' "$_json" | jq -r '.fields[] | select(.label == "LITELLM_KEY") | .value // empty' | head -n 1)"
if [ -n "$_litellm" ] && [ -f "$MCP_SYNC_KEYFILE" ] \
&& [ "$(tr -d ' \t\r\n' < "$MCP_SYNC_KEYFILE")" != "$_litellm" ]; then
printf '%s' "$_litellm" > "$MCP_SYNC_KEYFILE"
chmod 0600 "$MCP_SYNC_KEYFILE"
say "secrets: LITELLM_KEY changed; updated $MCP_SYNC_KEYFILE (re-run hermes/install.sh for Hermes)."
fi
+67
View File
@@ -74,6 +74,73 @@ grmb() {
return $rc return $rc
} }
# Secrets from this user's 1Password vault, written by dotfiles/secrets/install.sh.
# Reading a local 0600 file keeps shell startup off the network: nothing here
# calls op. opc/opd refresh the file themselves; after editing the vault
# anywhere else (the 1Password app, another box), run `ops`.
_DOTFILES_CONF="$HOME/.config/dotfiles"
[ -f "$_DOTFILES_CONF/secrets.env" ] && source "$_DOTFILES_CONF/secrets.env"
# opr NAME read one field straight from the vault
# opc NAME [VALUE] create or update a field (prompts, hidden, when VALUE is
# omitted, which keeps the secret out of shell history),
# then refresh and reload this shell
# opd NAME delete a field, refresh, and unset it in this shell
# opl [VAULT [ITEM]] show every field and its current value (default: your
# vault and item from op.env)
# ops re-fetch the vault into secrets.env and reload this shell
#
# The service-account token lives in op.env and is loaded only inside these
# subshells, so it never lands in the environment of everything you run.
_dotfiles_op() {
[ -f "$_DOTFILES_CONF/op.env" ] || { print -u2 "no $_DOTFILES_CONF/op.env; run: ops"; return 1; }
( set -a; source "$_DOTFILES_CONF/op.env"; set +a; "$@" )
}
_dotfiles_env_name() {
[[ "$1" =~ '^[A-Za-z_][A-Za-z0-9_]*$' ]] && return 0
print -u2 "$2: '$1' is not a valid variable name"; return 2
}
opr() {
_dotfiles_env_name "$1" opr || return
_dotfiles_op sh -c 'op read "op://$OP_VAULT/$OP_ITEM/$1"' opr "$1"
}
opc() {
emulate -L zsh
local name=$1 value
_dotfiles_env_name "$name" opc || return
if (( $# >= 2 )); then
value=$2
else
read -rs "value?$name: " && print
fi
# [concealed] makes it a password field; an existing field of that name is updated.
_dotfiles_op sh -c 'op item edit "$OP_ITEM" --vault "$OP_VAULT" "$1[concealed]=$2" >/dev/null' \
opc "$name" "$value" && ops
}
opd() {
_dotfiles_env_name "$1" opd || return
_dotfiles_op sh -c 'op item edit "$OP_ITEM" --vault "$OP_VAULT" "$1[delete]" >/dev/null' opd "$1" \
&& unset "$1" && ops
}
opl() {
_dotfiles_op sh -c '
vault=${1:-$OP_VAULT} item=${2:-$OP_ITEM}
printf "op://%s/%s\n" "$vault" "$item" >&2
op item get "$item" --vault "$vault" --reveal --format json' opl "$@" \
| jq -r '
[.fields[] | select(.purpose != "NOTES")] as $f
| ([$f[] | .label // "" | length] | max // 0) as $w
| $f[]
| ((.label // "") + " " * ($w - ((.label // "") | length))) as $name
| ((.value // "") | split("\n")) as $lines
| "\($name) \($lines[0] // "")",
($lines[1:][] | (" " * ($w + 2)) + .)'
}
ops() {
"${DOTFILES_DIR:-$HOME/.dotfiles}/secrets/install.sh" \
&& source "$_DOTFILES_CONF/secrets.env"
}
# Per-machine customizations (secrets, keys, local overrides) live in # Per-machine customizations (secrets, keys, local overrides) live in
# ~/.zshrc.local, which bootstrap never touches. See zsh/zshrc.local.example # ~/.zshrc.local, which bootstrap never touches. See zsh/zshrc.local.example
# for the variables modules look for. Loaded last so it can override anything. # for the variables modules look for. Loaded last so it can override anything.
+10 -6
View File
@@ -1,7 +1,9 @@
# ~/.zshrc.local — per-machine customizations. Copy this file to # ~/.zshrc.local — per-machine customizations. Copy this file to
# ~/.zshrc.local and fill in what this box needs. It is sourced at the end # ~/.zshrc.local and fill in what this box needs. It is sourced at the end
# of the managed ~/.zshrc, is never overwritten by bootstrap, and never # of the managed ~/.zshrc, is never overwritten by bootstrap, and never
# lives in the repo — this is the ONLY place to put secrets. # lives in the repo. Secrets normally come from your 1Password vault instead
# (secrets/install.sh -> ~/.config/dotfiles/secrets.env); set one here only on
# a box without 1Password, or to override the vault's value on this machine.
# #
# cp zsh/zshrc.local.example ~/.zshrc.local && chmod 600 ~/.zshrc.local # cp zsh/zshrc.local.example ~/.zshrc.local && chmod 600 ~/.zshrc.local
@@ -30,9 +32,11 @@
# Only the bootstrap-managed checkout (~/.dotfiles) is ever `git pull`-ed by the # Only the bootstrap-managed checkout (~/.dotfiles) is ever `git pull`-ed by the
# hook, and only when clean — a tree you work in is read, never touched. # hook, and only when clean — a tree you work in is read, never touched.
# --- 1Password CLI ------------------------------------------------------- # --- 1Password -----------------------------------------------------------
# Service-account token for non-interactive `op read` / `op run`. # Nothing to set here. Do NOT loop over `op read` in this file: that ran on
# export OP_SERVICE_ACCOUNT_TOKEN=ops_... # every shell. secrets/install.sh stores the service-account token in
# ~/.config/dotfiles/op.env and writes the vault's fields to secrets.env, which
# the managed ~/.zshrc loads. Use opr / opc / opd (see zsh/zshrc).
# --- Proxmox ------------------------------------------------------------- # --- Proxmox -------------------------------------------------------------
# One pair of credentials, several consumers: the proxmox MCP server # One pair of credentials, several consumers: the proxmox MCP server
@@ -40,8 +44,8 @@
# rke2-devops-airgap, whose Taskfile composes PROXMOX_VE_API_TOKEN from them. # rke2-devops-airgap, whose Taskfile composes PROXMOX_VE_API_TOKEN from them.
# Unprefixed on purpose — these are Proxmox credentials, not MCP ones. # Unprefixed on purpose — these are Proxmox credentials, not MCP ones.
# #
# The op-vault loop above already exports them from the `noconnor` item, so # secrets.env already exports them from your vault item, so these lines are
# these lines are only needed without 1Password. Never committed. # only needed without 1Password. Never committed.
# export PROXMOX_TOKEN_ID="root@pam!terragrunt-proxmox" # export PROXMOX_TOKEN_ID="root@pam!terragrunt-proxmox"
# export PROXMOX_TOKEN_SECRET="00000000-0000-0000-0000-000000000000" # export PROXMOX_TOKEN_SECRET="00000000-0000-0000-0000-000000000000"
# #