Compare commits
21
Commits
c6e5f22acb
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
41a90a10fb | ||
|
|
5f0dec5270 | ||
|
|
fe8b9a3ec0 | ||
|
|
72d85eb9ec | ||
|
|
7bbd911365 | ||
|
|
b6d36ba113 | ||
|
|
c0f1fc21b7 | ||
|
|
86b8e51646 | ||
|
|
ffedc84cfb | ||
|
|
cb7ab17769 | ||
|
|
5bc8a27f60 | ||
|
|
8ec3269f4a | ||
|
|
46052cf573 | ||
|
|
d3393edfdd | ||
|
|
db3a3e6e82 | ||
|
|
376ec03428 | ||
|
|
2bfa7e9a2d | ||
|
|
11138d90a9 | ||
|
|
7a7e987268 | ||
|
|
0426ebc973 | ||
|
|
1159197e4b |
@@ -44,9 +44,9 @@ Every module's `install.sh` satisfies all of these. Copy the shape from
|
|||||||
## Wiring it up
|
## Wiring it up
|
||||||
|
|
||||||
**`bootstrap.sh`** — add the call. Order is load-bearing and each existing
|
**`bootstrap.sh`** — add the call. Order is load-bearing and each existing
|
||||||
out-of-order entry carries a comment explaining why (`docker/` before `bin/`
|
out-of-order entry carries a comment explaining why (`secrets/` after `bin/`
|
||||||
so kind's gate sees the daemon; `docker-rootless-cgroup/` last because it
|
because it needs `op`; `docker-rootless-cgroup/` last because it needs the
|
||||||
needs the rootless daemon already up). If your module has an ordering
|
rootless daemon already up). If your module has an ordering
|
||||||
constraint, add it *with* a comment; if it doesn't, append it near related
|
constraint, add it *with* a comment; if it doesn't, append it near related
|
||||||
modules.
|
modules.
|
||||||
|
|
||||||
@@ -61,7 +61,7 @@ Never commit the value itself.
|
|||||||
## Non-systemd hosts
|
## Non-systemd hosts
|
||||||
|
|
||||||
Kasm has no systemd. Anything that would `systemctl enable` must degrade
|
Kasm has no systemd. Anything that would `systemctl enable` must degrade
|
||||||
gracefully — see `docker/install.sh`, which detects the situation and no-ops
|
gracefully — see `docker-rootless-cgroup/install.sh`, which detects the situation and no-ops
|
||||||
rather than failing the whole bootstrap run.
|
rather than failing the whole bootstrap run.
|
||||||
|
|
||||||
## Before you're done
|
## Before you're done
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ Everything in this repo is version-pinned on purpose. Nothing floats.
|
|||||||
| CLI tools | `bin/install.sh` | `ver=v1.2.3` inside each `install_<name>()` |
|
| CLI tools | `bin/install.sh` | `ver=v1.2.3` inside each `install_<name>()` |
|
||||||
| Node.js | `node/install.sh` | `NODE_VERSION="v24.18.0"` |
|
| Node.js | `node/install.sh` | `NODE_VERSION="v24.18.0"` |
|
||||||
| MCP servers | `mcp-config/opencode.gateway.json` | `["npx", "-y", "pkg@0.0.10"]` |
|
| MCP servers | `mcp-config/opencode.gateway.json` | `["npx", "-y", "pkg@0.0.10"]` |
|
||||||
| oh-my-zsh, Docker | `zsh/`, `docker/` | installed from upstream scripts — not pinned |
|
| oh-my-zsh | `zsh/` | installed from upstream script — not pinned |
|
||||||
|
|
||||||
Grep before assuming: `grep -rnE 'ver=|_VERSION=|@[0-9]+\.[0-9]+' --include='*.sh' --include='*.json'`
|
Grep before assuming: `grep -rnE 'ver=|_VERSION=|@[0-9]+\.[0-9]+' --include='*.sh' --include='*.json'`
|
||||||
|
|
||||||
|
|||||||
@@ -63,8 +63,8 @@ faithful rehearsal.
|
|||||||
|
|
||||||
Kasm has no systemd and no rootful Docker. Anything that would `systemctl` must
|
Kasm has no systemd and no rootful Docker. Anything that would `systemctl` must
|
||||||
no-op rather than abort — you can't fully rehearse this in a container, so read
|
no-op rather than abort — you can't fully rehearse this in a container, so read
|
||||||
the code path and confirm the guard exists (`docker/install.sh` and
|
the code path and confirm the guard exists (`docker-rootless-cgroup/install.sh`
|
||||||
`docker-rootless-cgroup/install.sh` are the references).
|
is the reference).
|
||||||
|
|
||||||
## Full run
|
## Full run
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,2 @@
|
|||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
@@ -32,16 +32,9 @@ Already have a clone? Just run it directly:
|
|||||||
- `git/` — sets git `user.name`/`user.email` and configures
|
- `git/` — sets git `user.name`/`user.email` and configures
|
||||||
`credential.helper store` (plaintext `~/.git-credentials`, 0600) for
|
`credential.helper store` (plaintext `~/.git-credentials`, 0600) for
|
||||||
HTTPS auth.
|
HTTPS auth.
|
||||||
- `docker/` — installs full (rootful) Docker Engine from Docker's upstream
|
|
||||||
package repo (get.docker.com on Debian-family; the docker-ce repo wired up
|
|
||||||
directly on RHEL-family, since the convenience script rejects EL rebuilds
|
|
||||||
like Oracle Linux by name), enables the service (systemd hosts), and adds
|
|
||||||
the invoking user to the
|
|
||||||
`docker` group (takes effect at next login). No-ops where a rootless
|
|
||||||
Docker daemon is already running (e.g. Kasm) so it never fights that
|
|
||||||
setup. Uses sudo.
|
|
||||||
- `bin/` — installs userland devops CLIs as self-contained, statically
|
- `bin/` — installs userland devops CLIs as self-contained, statically
|
||||||
linked release binaries into `~/.local/bin`: jq, kubectl, helm, k9s,
|
linked release binaries into `~/.local/bin`: jq, kubectl, kubelogin (the
|
||||||
|
`kubectl oidc-login` plugin, for OIDC auth to Omni clusters), helm, k9s,
|
||||||
kubectx/kubens, stern, yq, OpenTofu, terragrunt, task (go-task, the
|
kubectx/kubens, stern, yq, OpenTofu, terragrunt, task (go-task, the
|
||||||
Taskfile runner), the 1Password CLI `op` for secrets storage, the
|
Taskfile runner), the 1Password CLI `op` for secrets storage, the
|
||||||
opencode coding agent, zarf (DevSecOps for Offline and Airgapped
|
opencode coding agent, zarf (DevSecOps for Offline and Airgapped
|
||||||
@@ -49,6 +42,8 @@ Already have a clone? Just run it directly:
|
|||||||
coder CLI for Coder workspaces.
|
coder CLI for Coder workspaces.
|
||||||
kind is installed only when full (non-rootless) Docker is available —
|
kind is installed only when full (non-rootless) Docker is available —
|
||||||
it doesn't work against rootless Docker, so it is skipped on Kasm.
|
it doesn't work against rootless Docker, so it is skipped on Kasm.
|
||||||
|
Dotfiles does not install Docker itself; install it on the host (or bake
|
||||||
|
it into the image) where you want it.
|
||||||
No sudo, nothing outside `$HOME`, so tools persist across Kasm
|
No sudo, nothing outside `$HOME`, so tools persist across Kasm
|
||||||
redeploys with no reinstall. Idempotent: each tool records its version
|
redeploys with no reinstall. Idempotent: each tool records its version
|
||||||
under `~/.local/bin/.versions` and is skipped when the pinned version
|
under `~/.local/bin/.versions` and is skipped when the pinned version
|
||||||
@@ -58,14 +53,18 @@ Already have a clone? Just run it directly:
|
|||||||
sources, so opening a shell never calls `op`. One vault per user and one
|
sources, so opening a shell never calls `op`. One vault per user and one
|
||||||
item in it, both named after your login by default; each field's label is
|
item in it, both named after your login by default; each field's label is
|
||||||
the variable name (`GITEA_KEY`, `LITELLM_KEY`, `PROXMOX_TOKEN_SECRET`, …).
|
the variable name (`GITEA_KEY`, `LITELLM_KEY`, `PROXMOX_TOKEN_SECRET`, …).
|
||||||
The first run prompts for the service-account token (and vault/item), and
|
The first run prompts for the service-account token and checks it with
|
||||||
keeps them in `~/.config/dotfiles/op.env` (0600, not loaded into shells).
|
1Password straight away, re-prompting on a bad paste. It then picks the vault
|
||||||
|
and item: the only one the token can see, or a prompt that lists them. It keeps
|
||||||
|
all three in `~/.config/dotfiles/op.env` (0600, not loaded into shells). If
|
||||||
|
that fails, bootstrap skips mcp-config/ and hermes/ with an error rather
|
||||||
|
than asking for the LiteLLM key by hand.
|
||||||
Runs after `bin/` (needs `op` and `jq`); bootstrap loads the result so later
|
Runs after `bin/` (needs `op` and `jq`); bootstrap loads the result so later
|
||||||
modules get the keys without prompting. Day to day: `envr NAME` reads a
|
modules get the keys without prompting. Day to day: `opr NAME` reads a
|
||||||
field, `envc NAME [VALUE]` creates or updates one (prompts, hidden, without
|
field, `opc NAME [VALUE]` creates or updates one (prompts, hidden, without
|
||||||
VALUE), `envd NAME` deletes one; both of those refresh the file and this
|
VALUE), `opd NAME` deletes one; both of those refresh the file and this
|
||||||
shell. `envl [VAULT [ITEM]]` shows every field with its current value, and
|
shell. `opl [VAULT [ITEM]]` shows every field with its current value, and
|
||||||
`envs` refreshes after a change made elsewhere. No sudo.
|
`ops` refreshes after a change made elsewhere. No sudo.
|
||||||
|
|
||||||
opencode is configured by `mcp-config/`: it points at the covert-gpt
|
opencode is configured by `mcp-config/`: it points at the covert-gpt
|
||||||
LiteLLM gateway (`https://llm.covert-gpt.com/v1`) for models. MCP tools do
|
LiteLLM gateway (`https://llm.covert-gpt.com/v1`) for models. MCP tools do
|
||||||
@@ -79,6 +78,14 @@ Already have a clone? Just run it directly:
|
|||||||
`~/.local/bin`. Persists across redeploys; only rebuilt when the venv is
|
`~/.local/bin`. Persists across redeploys; only rebuilt when the venv is
|
||||||
missing or broken. Building it may install `python3`/`python3-venv` via
|
missing or broken. Building it may install `python3`/`python3-venv` via
|
||||||
the system package manager (the one place this module uses sudo).
|
the system package manager (the one place this module uses sudo).
|
||||||
|
- `vscode/` — installs Visual Studio Code from upstream's portable tarball into
|
||||||
|
`~/.local/opt/vscode`, symlinks `code` into `~/.local/bin`, and writes a
|
||||||
|
user-level `.desktop` entry so it shows up in the application menu. **No
|
||||||
|
sudo, nothing outside `$HOME`** — the same contract as `bin/`, so it survives
|
||||||
|
a Kasm redeploy. Pinned via `VSCODE_VERSION` in the module; bump it to upgrade.
|
||||||
|
Also sets `password-store: basic` in `~/.vscode/argv.json` so KDE's kwallet
|
||||||
|
does not prompt for an unlock on every launch (see below). **No-ops on
|
||||||
|
headless hosts** — it's a ~400MB GUI app — override with `DOTFILES_VSCODE`.
|
||||||
- `docker-rootless-cgroup/` — on systemd-less hosts (e.g. Kasm), hand-delegates
|
- `docker-rootless-cgroup/` — on systemd-less hosts (e.g. Kasm), hand-delegates
|
||||||
cgroup v2 controllers to the rootless Docker daemon so per-container cgroups
|
cgroup v2 controllers to the rootless Docker daemon so per-container cgroups
|
||||||
work (rootless Docker otherwise comes up with
|
work (rootless Docker otherwise comes up with
|
||||||
@@ -112,6 +119,18 @@ item instead of `~/.zshrc.local`:
|
|||||||
don't set this by hand: run `mcp-config/install.sh` once and its shell
|
don't set this by hand: run `mcp-config/install.sh` once and its shell
|
||||||
hook exports it from `~/.config/mcp-sync/key`. Set it in `~/.zshrc.local`
|
hook exports it from `~/.config/mcp-sync/key`. Set it in `~/.zshrc.local`
|
||||||
only on a box without mcp-sync.
|
only on a box without mcp-sync.
|
||||||
|
- `DOTFILES_VSCODE` — controls the `vscode/` module. Unset (default) installs
|
||||||
|
VS Code only when a graphical session is detected; `1` forces the install on
|
||||||
|
a headless host (e.g. for `code tunnel`); `0` skips it everywhere. A plain
|
||||||
|
shell/env var, not a vault secret.
|
||||||
|
|
||||||
|
On the keyring: VS Code's secret storage reaches for the OS keyring through
|
||||||
|
libsecret, which on KDE makes kwalletd prompt for an unlock before the editor
|
||||||
|
is usable — every session, since `kwallet-pam` is not installed. The module
|
||||||
|
sets `"password-store": "basic"` in `~/.vscode/argv.json` to stop that, which
|
||||||
|
means VS Code's stored tokens are obfuscated on disk rather than sealed
|
||||||
|
behind a wallet password. To use the wallet instead, install `kwallet-pam` so
|
||||||
|
it unlocks at login, then change that value to `"kwallet5"`.
|
||||||
- `OP_SERVICE_ACCOUNT_TOKEN` — 1Password service-account token. Not a shell
|
- `OP_SERVICE_ACCOUNT_TOKEN` — 1Password service-account token. Not a shell
|
||||||
variable any more: `secrets/install.sh` asks for it once and stores it in
|
variable any more: `secrets/install.sh` asks for it once and stores it in
|
||||||
`~/.config/dotfiles/op.env`. Pass it in the env only for an unattended run.
|
`~/.config/dotfiles/op.env`. Pass it in the env only for an unattended run.
|
||||||
|
|||||||
@@ -49,6 +49,18 @@ install_kubectl() {
|
|||||||
mark_version kubectl "$ver"
|
mark_version kubectl "$ver"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# kubelogin — kubectl OIDC plugin, so omnictl-generated kubeconfigs can
|
||||||
|
# authenticate (they exec `kubectl oidc-login get-token`). Installed as
|
||||||
|
# kubectl-oidc_login, the name kubectl's plugin lookup expects.
|
||||||
|
install_oidc_login() {
|
||||||
|
ver=v1.36.4
|
||||||
|
have_version kubectl-oidc_login "$ver" && { fsay "kubectl-oidc_login $ver already installed."; return 0; }
|
||||||
|
install_zip_bin \
|
||||||
|
"https://github.com/int128/kubelogin/releases/download/$ver/kubelogin_linux_$DEB.zip" \
|
||||||
|
kubelogin kubectl-oidc_login
|
||||||
|
mark_version kubectl-oidc_login "$ver"
|
||||||
|
}
|
||||||
|
|
||||||
# helm — Kubernetes package manager (from get.helm.sh).
|
# helm — Kubernetes package manager (from get.helm.sh).
|
||||||
install_helm() {
|
install_helm() {
|
||||||
ver=v4.2.2
|
ver=v4.2.2
|
||||||
@@ -235,6 +247,7 @@ try() {
|
|||||||
|
|
||||||
try install_jq
|
try install_jq
|
||||||
try install_kubectl
|
try install_kubectl
|
||||||
|
try install_oidc_login
|
||||||
try install_helm
|
try install_helm
|
||||||
try install_k9s
|
try install_k9s
|
||||||
if docker_full_available; then
|
if docker_full_available; then
|
||||||
|
|||||||
+27
-7
@@ -66,7 +66,7 @@ fi
|
|||||||
#
|
#
|
||||||
# WHY: these used to be bare calls, so under `set -eu` the FIRST failing module
|
# WHY: these used to be bare calls, so under `set -eu` the FIRST failing module
|
||||||
# aborted bootstrap and every later module silently never ran — one unsupported
|
# aborted bootstrap and every later module silently never ran — one unsupported
|
||||||
# distro in docker/ was enough to leave a box with no CLIs, no npx and no
|
# distro in one module was enough to leave a box with no CLIs, no npx and no
|
||||||
# ansible. The modules are independent, so a failure in one is not a reason to
|
# ansible. The modules are independent, so a failure in one is not a reason to
|
||||||
# skip the rest. (bin/install.sh does the same for individual tools; this is
|
# skip the rest. (bin/install.sh does the same for individual tools; this is
|
||||||
# that guard one level up.)
|
# that guard one level up.)
|
||||||
@@ -79,8 +79,6 @@ run_module() {
|
|||||||
|
|
||||||
run_module zsh
|
run_module zsh
|
||||||
run_module git
|
run_module git
|
||||||
# Runs before bin/ so the kind install gate can see the rootful daemon.
|
|
||||||
run_module docker
|
|
||||||
run_module bin
|
run_module bin
|
||||||
# Needs op and jq from bin/. Runs before everything that consumes a secret, and
|
# Needs op and jq from bin/. Runs before everything that consumes a secret, and
|
||||||
# its output is loaded into THIS process so later modules see the vault's
|
# its output is loaded into THIS process so later modules see the vault's
|
||||||
@@ -103,11 +101,33 @@ run_module node
|
|||||||
# be repaired by re-running bootstrap. Losing ~/.zshrc.local (or the hook line
|
# be repaired by re-running bootstrap. Losing ~/.zshrc.local (or the hook line
|
||||||
# in it) left opencode with no covert-gpt provider and no way to get it back
|
# in it) left opencode with no covert-gpt provider and no way to get it back
|
||||||
# short of curl-ing this module by hand.
|
# short of curl-ing this module by hand.
|
||||||
run_module mcp-config
|
#
|
||||||
# Hermes Agent (Nous Research) is distributed as an npm package with a Python
|
# Both it and hermes/ are SKIPPED when the vault did not deliver LITELLM_KEY
|
||||||
# dependency. Runs after node/ ensures npx is available.
|
# (and no box-local copy exists): mcp-config/ would otherwise fall back to a
|
||||||
run_module hermes
|
# "paste your LiteLLM key" prompt, which reads like a LiteLLM problem and hides
|
||||||
|
# the real one — a bad 1Password token, vault or item, which also means every
|
||||||
|
# other vault key is missing.
|
||||||
|
if [ -z "${LITELLM_KEY:-}" ] && [ ! -s "$HOME/.config/mcp-sync/key" ]; then
|
||||||
|
echo "" >&2
|
||||||
|
echo "bootstrap: ERROR: no LITELLM_KEY from 1Password — skipping mcp-config and hermes." >&2
|
||||||
|
case " $FAILED " in
|
||||||
|
*" secrets "*) echo "bootstrap: the secrets module failed (see its messages above)." >&2 ;;
|
||||||
|
*) echo "bootstrap: the vault item has no LITELLM_KEY field; add it with: opc LITELLM_KEY" >&2 ;;
|
||||||
|
esac
|
||||||
|
echo "bootstrap: fix it with $SCRIPT_DIR/secrets/install.sh, then re-run bootstrap." >&2
|
||||||
|
echo "" >&2
|
||||||
|
FAILED="$FAILED mcp-config hermes"
|
||||||
|
else
|
||||||
|
run_module mcp-config
|
||||||
|
# Hermes Agent (Nous Research) is distributed as an npm package with a Python
|
||||||
|
# dependency. Runs after node/ ensures npx is available.
|
||||||
|
run_module hermes
|
||||||
|
fi
|
||||||
run_module ansible
|
run_module ansible
|
||||||
|
# GUI editor, installed from upstream's tarball into ~/.local/opt (no sudo, same
|
||||||
|
# contract as bin/). No ordering constraints — it needs nothing from bin/,
|
||||||
|
# secrets/ or node/ — and no-ops on headless hosts.
|
||||||
|
run_module vscode
|
||||||
# Runs last: needs rootless Docker already up (started at boot) so it can
|
# Runs last: needs rootless Docker already up (started at boot) so it can
|
||||||
# delegate cgroups to it. No-ops where that does not apply.
|
# delegate cgroups to it. No-ops where that does not apply.
|
||||||
run_module docker-rootless-cgroup
|
run_module docker-rootless-cgroup
|
||||||
|
|||||||
@@ -0,0 +1,113 @@
|
|||||||
|
# claude-obsidian — Claude conversations → Obsidian notes
|
||||||
|
|
||||||
|
Summarizes every Claude conversation into one searchable Obsidian note: what happened, what
|
||||||
|
was decided and why, the facts and gotchas worth remembering, and what was left open.
|
||||||
|
|
||||||
|
- **Claude Code** sessions sync automatically and incrementally (hourly timer).
|
||||||
|
- **claude.ai** chats import from a data export (claude.ai has no API, so each export is manual).
|
||||||
|
|
||||||
|
Notes are written through the [Obsidian Local REST API](https://github.com/coddingtonbear/obsidian-local-rest-api)
|
||||||
|
plugin, so the vault can live anywhere the plugin is reachable. The same plugin exposes a
|
||||||
|
**built-in MCP server** (`/mcp/`), which is how Claude reads the notes back — no separate MCP
|
||||||
|
process.
|
||||||
|
|
||||||
|
```
|
||||||
|
~/.claude/projects/*/*.jsonl ─┐ ┌─ Claude/Code/2026/…md
|
||||||
|
claude.ai export .zip ────────┼─ condense → redact → summarize → PUT ────┼─ Claude/Web/2026/…md
|
||||||
|
└─ (headless claude -p, no transcript) └─ Claude/Index.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
1. In Obsidian, install **Local REST API**, enable its **HTTP server**, and make it reachable
|
||||||
|
from this machine. Put the API key in `~/.config/claude-obsidian/api-key` (`chmod 600`).
|
||||||
|
2. `./install.sh` — links the binary into `~/.local/bin`, enables the hourly user timer, and
|
||||||
|
creates `~/.config/claude-obsidian/config.env` from `config.env.example` if absent.
|
||||||
|
3. Edit `config.env` (at minimum `OBSIDIAN_API_URL`), then preview one note:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
claude-obsidian-sync stats # sizes, no network
|
||||||
|
claude-obsidian-sync sync --dry-run --limit 1 # prints the note, writes nothing
|
||||||
|
```
|
||||||
|
|
||||||
|
## Use
|
||||||
|
|
||||||
|
```bash
|
||||||
|
claude-obsidian-sync sync # what the timer runs
|
||||||
|
claude-obsidian-sync sync --session <session-id> # one session, now, ignoring the settle window
|
||||||
|
claude-obsidian-sync import-claude-ai ~/Downloads/data-2026-09-13.zip
|
||||||
|
claude-obsidian-sync sync --prefetch --limit 3 # summarize into the local cache, vault untouched
|
||||||
|
claude-obsidian-sync usage # calls, billed tokens, spend: last 5h / 24h / 7d
|
||||||
|
```
|
||||||
|
|
||||||
|
Claude reads the vault through the plugin's MCP server:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
claude mcp add --scope user --transport http obsidian http://<host>:27123/mcp/ \
|
||||||
|
--header "Authorization: Bearer $(cat ~/.config/claude-obsidian/api-key)"
|
||||||
|
```
|
||||||
|
|
||||||
|
## How it behaves — and why
|
||||||
|
|
||||||
|
- **Summaries run on your own Claude plan** via headless `claude -p` (`LLM_BACKEND=claude-cli`).
|
||||||
|
No API key to manage. An OpenAI-compatible backend (`LLM_BACKEND=openai`) exists for local
|
||||||
|
models. Note a LiteLLM gateway that fronts Claude by *passing through the caller's*
|
||||||
|
subscription has no key of its own, so a background job cannot use its `claude-*` models.
|
||||||
|
- **`--no-session-persistence` is load-bearing.** Without it every summary writes a new
|
||||||
|
transcript into `~/.claude/projects`, and the next run summarizes its own summarizing.
|
||||||
|
- **Usage is metered and capped.** Every model call is appended to
|
||||||
|
`~/.local/state/claude-obsidian/usage.jsonl` with *billed* tokens (the CLI's `modelUsage`, not
|
||||||
|
the visible turn) and `total_cost_usd` — API list-price equivalent, a faithful proxy for how much
|
||||||
|
was consumed; the plan's own meter is only visible in Claude Code's `/usage`. Before each call
|
||||||
|
its cost is **estimated** (+20% margin) and the call is refused if it would cross
|
||||||
|
`BUDGET_USD_PER_RUN` or the rolling `BUDGET_USD_PER_DAY` — so budgets are never overshot. Not
|
||||||
|
`claude -p --max-budget-usd`: that aborts *after* the tokens are spent and discards the result
|
||||||
|
(measured: $0.26 for nothing). Every run ends with a one-line usage report.
|
||||||
|
- **Three cost levers, measured on one session — $0.057 → $0.010, same summary content:**
|
||||||
|
`alwaysThinkingEnabled=false` (extended thinking was ~85% of output tokens),
|
||||||
|
`DISABLE_PROMPT_CACHING=1` (the CLI writes each prompt to the 1-hour cache at 2× input price, and
|
||||||
|
a one-shot call never reads it back), and `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1` (otherwise a
|
||||||
|
hidden side request re-reads the entire prompt, doubling billed input). A whole ~110-session
|
||||||
|
backlog is about $2 at list price with all three.
|
||||||
|
- **Summaries are cached until their note is written**, so a vault outage never pays for the same
|
||||||
|
summary twice, and `--prefetch` can do the summarizing while the vault is unreachable.
|
||||||
|
- **It shares your plan's usage limits.** A first backfill of ~100 sessions in parallel hit a
|
||||||
|
session limit within minutes — which also limits *interactive* use. So: one job at a time,
|
||||||
|
`MAX_PER_RUN` sessions per run (newest first), and the run **stops at the first usage-limit
|
||||||
|
error** (exit 75, which the systemd unit treats as success) instead of failing the rest of
|
||||||
|
the queue. The next hourly run resumes. A 100-session backlog drains in about ten hours.
|
||||||
|
- **Secrets are redacted before anything leaves the machine**, and again in the note: private
|
||||||
|
keys, bearer tokens, `password=`-style assignments, JWTs, provider tokens, ≥48-char hex, and
|
||||||
|
random mixed-case strings. It over-redacts on purpose; git SHAs, md5s and paths survive.
|
||||||
|
The summarizer is also told never to reproduce secrets.
|
||||||
|
- **One run at a time** (a `flock` on `run.lock`): an hourly timer firing during a long manual run
|
||||||
|
exits immediately instead of racing it on the state and ledger files.
|
||||||
|
- **Several vault URLs** can be listed; the first that answers wins, so one config covers both a
|
||||||
|
permanent address and a temporary `kubectl port-forward`.
|
||||||
|
- **Incremental and idempotent.** State (`~/.local/state/claude-obsidian/state.json`) records
|
||||||
|
each session's size+mtime. A resumed session is re-summarized into the *same* note path. A
|
||||||
|
session is left alone until idle for `SETTLE_MINUTES`, so a live conversation isn't
|
||||||
|
summarized mid-flight. Empty sessions are recorded and skipped.
|
||||||
|
- **Condensed, not truncated.** Tool output is dropped and each tool call becomes one line;
|
||||||
|
prose is kept. Typical compression is ~60× (271 MB of transcripts → 4.4 MB). Sessions over
|
||||||
|
300 KB condensed are summarized in parts, then combined.
|
||||||
|
- **Subagent transcripts** (`<session>/subagents/**`) are part of their parent session and are
|
||||||
|
not synced separately.
|
||||||
|
|
||||||
|
## Note format
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
type: claude-conversation
|
||||||
|
source: claude-code # or claude-ai
|
||||||
|
session_id: 3f9c2a1e-…
|
||||||
|
title: Fix flaky CI cache after runner upgrade
|
||||||
|
project: my-service # basename of the working directory
|
||||||
|
branch: …
|
||||||
|
started / ended: …
|
||||||
|
prompts: 1 # user prompts, not tool-loop records
|
||||||
|
summarized_by: claude-haiku-4-5-20251001
|
||||||
|
tags: [claude, claude-code, ci, caching, …]
|
||||||
|
```
|
||||||
|
|
||||||
|
followed by a summary callout and **Outcomes / Decisions / Facts & gotchas / Open threads**,
|
||||||
|
and a `claude --resume <id>` line. `Claude/Index.md` lists every note, newest first.
|
||||||
Executable
+798
@@ -0,0 +1,798 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""claude-obsidian-sync -- summarize Claude conversations into an Obsidian vault.
|
||||||
|
|
||||||
|
Reads Claude Code transcripts (~/.claude/projects/<project>/<session>.jsonl) and claude.ai
|
||||||
|
data exports, condenses each conversation, redacts secrets, summarizes it with an
|
||||||
|
OpenAI-compatible LLM endpoint (e.g. LiteLLM), and writes one note per conversation into
|
||||||
|
Obsidian through the Local REST API plugin. Standard library only.
|
||||||
|
|
||||||
|
claude-obsidian-sync stats # sizes and counts, no network
|
||||||
|
claude-obsidian-sync sync [--dry-run] # Claude Code sessions (incremental)
|
||||||
|
claude-obsidian-sync import-claude-ai EXPORT.zip|conversations.json
|
||||||
|
|
||||||
|
Configuration comes from the environment, falling back to ~/.config/claude-obsidian/config.env
|
||||||
|
(KEY=VALUE lines). Nothing environment-specific lives in this file. See README.md.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import concurrent.futures as cf
|
||||||
|
import datetime as dt
|
||||||
|
import fcntl
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import pathlib
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
import urllib.error
|
||||||
|
import urllib.parse
|
||||||
|
import urllib.request
|
||||||
|
import zipfile
|
||||||
|
|
||||||
|
HOME = pathlib.Path.home()
|
||||||
|
CONFIG_DIR = HOME / ".config" / "claude-obsidian"
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- config
|
||||||
|
def load_config() -> dict:
|
||||||
|
cfg: dict[str, str] = {}
|
||||||
|
f = CONFIG_DIR / "config.env"
|
||||||
|
if f.exists():
|
||||||
|
for line in f.read_text().splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if line and not line.startswith("#") and "=" in line:
|
||||||
|
k, v = line.split("=", 1)
|
||||||
|
v = re.split(r"\s+#", v, maxsplit=1)[0] # inline comment; a '#' inside a value survives
|
||||||
|
cfg[k.strip()] = v.strip().strip('"').strip("'")
|
||||||
|
cfg.update({k: v for k, v in os.environ.items() if k in DEFAULTS or k.startswith("LLM_") or k.startswith("OBSIDIAN_")})
|
||||||
|
for k, v in DEFAULTS.items():
|
||||||
|
cfg.setdefault(k, v)
|
||||||
|
return cfg
|
||||||
|
|
||||||
|
|
||||||
|
DEFAULTS = {
|
||||||
|
"OBSIDIAN_API_URL": "http://127.0.0.1:27123",
|
||||||
|
"OBSIDIAN_API_KEY_FILE": str(CONFIG_DIR / "api-key"),
|
||||||
|
"LLM_BACKEND": "claude-cli", # claude-cli | openai
|
||||||
|
"CLAUDE_BIN": "claude",
|
||||||
|
"LLM_BASE_URL": "",
|
||||||
|
"LLM_API_KEY_FILE": "",
|
||||||
|
"LLM_API_KEY_ENV": "LITELLM_KEY",
|
||||||
|
"LLM_MODEL": "haiku",
|
||||||
|
"VAULT_FOLDER": "Claude",
|
||||||
|
"SETTLE_MINUTES": "30",
|
||||||
|
"CLAUDE_PROJECTS_DIR": str(HOME / ".claude" / "projects"),
|
||||||
|
"STATE_FILE": str(HOME / ".local" / "state" / "claude-obsidian" / "state.json"),
|
||||||
|
"JOBS": "1",
|
||||||
|
# Cap per run. Summaries draw on the same plan limits as interactive Claude use, so the
|
||||||
|
# timer drips a backlog instead of bursting it (a 107-session burst exhausted a session limit).
|
||||||
|
"MAX_PER_RUN": "10",
|
||||||
|
# Spend guards, in API list-price USD as reported by `claude -p` (total_cost_usd). Not the
|
||||||
|
# plan's own meter -- that is only visible in Claude Code's /usage -- but a faithful proxy
|
||||||
|
# for how much was consumed. Hitting one stops the run like a usage limit (exit 75).
|
||||||
|
"BUDGET_USD_PER_RUN": "0.50",
|
||||||
|
"BUDGET_USD_PER_DAY": "2.00",
|
||||||
|
# Pre-call estimate prices (USD per million tokens, list). A call is refused BEFORE it runs
|
||||||
|
# if the estimate would cross a budget. Deliberately NOT `claude -p --max-budget-usd`: that
|
||||||
|
# aborts after the tokens are spent and throws the result away (measured: $0.26 for nothing).
|
||||||
|
"LLM_PRICE_IN_PER_M": "1.0",
|
||||||
|
"LLM_PRICE_OUT_PER_M": "5.0",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def secret(cfg: dict, file_key: str, env_key: str | None = None) -> str:
|
||||||
|
path = cfg.get(file_key)
|
||||||
|
if path:
|
||||||
|
p = pathlib.Path(os.path.expanduser(path))
|
||||||
|
if p.exists():
|
||||||
|
return p.read_text().strip()
|
||||||
|
if env_key and os.environ.get(cfg.get(env_key, "") or ""):
|
||||||
|
return os.environ[cfg[env_key]].strip()
|
||||||
|
raise SystemExit(f"missing secret: set {file_key} (or {env_key})")
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- redaction
|
||||||
|
# Applied to everything sent to the LLM AND to the note written. Over-redaction is the
|
||||||
|
# accepted failure mode: a summary missing a token is fine, a vault holding one is not.
|
||||||
|
_REDACTIONS = [
|
||||||
|
(re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----.*?-----END [A-Z ]*PRIVATE KEY-----", re.S), "[REDACTED PRIVATE KEY]"),
|
||||||
|
(re.compile(r"(?i)\b(bearer|token)\s+[A-Za-z0-9._~+/=-]{16,}"), r"\1 [REDACTED]"),
|
||||||
|
(re.compile(r"(?i)((?:password|passwd|pwd|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|client[_-]?secret|NB_API_KEY|PAT)[\"']?\s*[:=]\s*[\"']?)([^\s\"'`,;]{6,})"), r"\1[REDACTED]"),
|
||||||
|
(re.compile(r"\beyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}"), "[REDACTED JWT]"),
|
||||||
|
(re.compile(r"\b(?:sk|pk|rk)-[A-Za-z0-9_-]{20,}|\bgh[pousr]_[A-Za-z0-9]{30,}|\bglpat-[A-Za-z0-9_-]{20,}|\bxox[abpr]-[A-Za-z0-9-]{10,}"), "[REDACTED TOKEN]"),
|
||||||
|
# >=48 hex: API keys / sha256 tokens. Deliberately leaves 40-hex git SHAs and 32-hex md5s.
|
||||||
|
(re.compile(r"\b[a-fA-F0-9]{48,}\b"), "[REDACTED HEX]"),
|
||||||
|
# Random mixed-case+digit strings (generated passwords). No '/' so paths survive.
|
||||||
|
(re.compile(r"(?<![A-Za-z0-9_./-])(?=[A-Za-z0-9+]*[0-9])(?=[A-Za-z0-9+]*[a-z])(?=[A-Za-z0-9+]*[A-Z])[A-Za-z0-9+]{20,}={0,2}(?![A-Za-z0-9_./-])"), "[REDACTED]"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def redact(text: str) -> str:
|
||||||
|
for pat, repl in _REDACTIONS:
|
||||||
|
text = pat.sub(repl, text)
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- parsing
|
||||||
|
_NOISE_BLOCKS = re.compile(
|
||||||
|
r"<(system-reminder|local-command-stdout|local-command-stderr|local-command-caveat|task-notification)>.*?</\1>",
|
||||||
|
re.S,
|
||||||
|
)
|
||||||
|
_CMD = re.compile(r"<command-name>(.*?)</command-name>.*?(?:<command-args>(.*?)</command-args>)?", re.S)
|
||||||
|
MAX_MSG = 6000
|
||||||
|
|
||||||
|
|
||||||
|
def _clip(text: str) -> str:
|
||||||
|
return text if len(text) <= MAX_MSG else text[:4000] + "\n[...clipped...]\n" + text[-1500:]
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_user(text: str) -> str:
|
||||||
|
text = _NOISE_BLOCKS.sub("", text)
|
||||||
|
m = _CMD.search(text)
|
||||||
|
if m:
|
||||||
|
args = (m.group(2) or "").strip()
|
||||||
|
text = f"{m.group(1).strip()} {args}".strip()
|
||||||
|
text = re.sub(r"</?command-(message|args|name)>", "", text)
|
||||||
|
return text.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def _tool_line(block: dict) -> str:
|
||||||
|
name = block.get("name", "tool")
|
||||||
|
inp = block.get("input") or {}
|
||||||
|
desc = inp.get("description") or inp.get("command") or inp.get("file_path") or inp.get("query") \
|
||||||
|
or inp.get("url") or inp.get("prompt") or inp.get("pattern") or ""
|
||||||
|
desc = " ".join(str(desc).split())[:160]
|
||||||
|
return f"[{name}: {desc}]" if desc else f"[{name}]"
|
||||||
|
|
||||||
|
|
||||||
|
def parse_claude_code(path: pathlib.Path) -> dict:
|
||||||
|
meta = {"session_id": path.stem, "cwd": None, "branch": None, "first": None, "last": None,
|
||||||
|
"title": None, "version": None, "prompts": 0}
|
||||||
|
turns: list[list[str]] = []
|
||||||
|
for line in path.open(encoding="utf-8", errors="replace"):
|
||||||
|
try:
|
||||||
|
r = json.loads(line)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
continue
|
||||||
|
t = r.get("type")
|
||||||
|
if t == "ai-title" and r.get("aiTitle"):
|
||||||
|
meta["title"] = r["aiTitle"]
|
||||||
|
if t == "custom-title" and r.get("customTitle"):
|
||||||
|
meta["title"] = r["customTitle"]
|
||||||
|
if t not in ("user", "assistant") or r.get("isSidechain") or r.get("isMeta"):
|
||||||
|
continue
|
||||||
|
ts = r.get("timestamp")
|
||||||
|
if ts:
|
||||||
|
meta["first"] = meta["first"] or ts
|
||||||
|
meta["last"] = ts
|
||||||
|
meta["cwd"] = meta["cwd"] or r.get("cwd")
|
||||||
|
meta["branch"] = r.get("gitBranch") or meta["branch"]
|
||||||
|
meta["version"] = r.get("version") or meta["version"]
|
||||||
|
content = (r.get("message") or {}).get("content")
|
||||||
|
if t == "user":
|
||||||
|
if isinstance(content, str):
|
||||||
|
text = content
|
||||||
|
elif isinstance(content, list):
|
||||||
|
text = "\n".join(b.get("text", "") for b in content if b.get("type") == "text")
|
||||||
|
else:
|
||||||
|
continue
|
||||||
|
text = _clean_user(text)
|
||||||
|
if text:
|
||||||
|
role = "SUMMARY-OF-EARLIER-CONTEXT" if r.get("isCompactSummary") else "USER"
|
||||||
|
turns.append([role, _clip(text)])
|
||||||
|
meta["prompts"] += role == "USER"
|
||||||
|
else:
|
||||||
|
parts = []
|
||||||
|
for b in content if isinstance(content, list) else []:
|
||||||
|
if b.get("type") == "text" and b.get("text", "").strip():
|
||||||
|
parts.append(_clip(b["text"].strip()))
|
||||||
|
elif b.get("type") == "tool_use":
|
||||||
|
parts.append(_tool_line(b))
|
||||||
|
if not parts:
|
||||||
|
continue
|
||||||
|
if turns and turns[-1][0] == "ASSISTANT": # a tool loop is many records
|
||||||
|
turns[-1][1] += "\n" + "\n".join(parts)
|
||||||
|
else:
|
||||||
|
turns.append(["ASSISTANT", "\n".join(parts)])
|
||||||
|
meta["transcript"] = "\n\n".join(f"{role}: {text}" for role, text in turns)
|
||||||
|
return meta
|
||||||
|
|
||||||
|
|
||||||
|
def parse_claude_ai_export(src: pathlib.Path) -> list[dict]:
|
||||||
|
"""claude.ai 'Export data' -> conversations.json. Tolerant of field-name drift."""
|
||||||
|
if src.suffix == ".zip":
|
||||||
|
with zipfile.ZipFile(src) as z:
|
||||||
|
name = next((n for n in z.namelist() if n.endswith("conversations.json")), None)
|
||||||
|
if not name:
|
||||||
|
raise SystemExit(f"no conversations.json inside {src}")
|
||||||
|
data = json.loads(z.read(name))
|
||||||
|
else:
|
||||||
|
data = json.loads(src.read_text())
|
||||||
|
out = []
|
||||||
|
for c in data if isinstance(data, list) else data.get("conversations", []):
|
||||||
|
turns, count = [], 0
|
||||||
|
for m in c.get("chat_messages") or c.get("messages") or []:
|
||||||
|
text = m.get("text") or ""
|
||||||
|
if not text and isinstance(m.get("content"), list):
|
||||||
|
text = "\n".join(b.get("text", "") for b in m["content"] if b.get("type") == "text")
|
||||||
|
for a in m.get("attachments") or []:
|
||||||
|
if a.get("extracted_content"):
|
||||||
|
text += f"\n[attachment {a.get('file_name', '')}]: {a['extracted_content'][:1500]}"
|
||||||
|
if not text.strip():
|
||||||
|
continue
|
||||||
|
role = "USER" if (m.get("sender") or m.get("role")) in ("human", "user") else "ASSISTANT"
|
||||||
|
turns.append(f"{role}: {_clip(text.strip())}")
|
||||||
|
count += role == "USER"
|
||||||
|
if not turns:
|
||||||
|
continue
|
||||||
|
out.append({
|
||||||
|
"session_id": c.get("uuid") or hashlib.sha1(json.dumps(c, sort_keys=True).encode()).hexdigest(),
|
||||||
|
"title": c.get("name") or None, "first": c.get("created_at"), "last": c.get("updated_at"),
|
||||||
|
"cwd": None, "branch": None, "version": None, "prompts": count,
|
||||||
|
"transcript": "\n\n".join(turns),
|
||||||
|
})
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- LLM
|
||||||
|
SYSTEM = """You write concise knowledge-base notes summarizing a conversation between a user and Claude.
|
||||||
|
The reader is the same user, months later, searching their notes. Capture what happened and
|
||||||
|
what was learned, not a play-by-play. Be specific: name the systems, files, commands, errors
|
||||||
|
and decisions. Never include passwords, API keys, tokens or other secrets, even if present;
|
||||||
|
refer to them generically ("the API key"). Text marked [REDACTED] must stay redacted.
|
||||||
|
|
||||||
|
Reply with ONLY a JSON object, no prose, with these keys:
|
||||||
|
"title": short specific title (<= 70 chars)
|
||||||
|
"summary": 3-6 sentences
|
||||||
|
"outcomes": list of what was actually built, fixed or answered
|
||||||
|
"decisions": list of choices made and why
|
||||||
|
"facts": list of reusable facts, gotchas, root causes worth remembering
|
||||||
|
"open_threads": list of unfinished work or follow-ups
|
||||||
|
"tags": 3-8 lowercase kebab-case topic tags
|
||||||
|
Use empty lists where nothing applies."""
|
||||||
|
|
||||||
|
CHUNK_SYSTEM = """You are condensing one part of a long conversation between a user and Claude so that a
|
||||||
|
later step can summarize the whole. Output dense bullet notes: goals, actions taken, results,
|
||||||
|
errors and their causes, decisions, open questions. Be specific. Never reproduce secrets."""
|
||||||
|
|
||||||
|
CHUNK_CHARS = 300_000
|
||||||
|
|
||||||
|
|
||||||
|
class Ledger:
|
||||||
|
"""Append-only record of every model call: tokens and list-price cost. Also enforces budgets."""
|
||||||
|
|
||||||
|
def __init__(self, cfg: dict):
|
||||||
|
self.path = pathlib.Path(os.path.expanduser(cfg["STATE_FILE"])).parent / "usage.jsonl"
|
||||||
|
self.run_budget = float(cfg["BUDGET_USD_PER_RUN"])
|
||||||
|
self.day_budget = float(cfg["BUDGET_USD_PER_DAY"])
|
||||||
|
self.lock = threading.Lock()
|
||||||
|
self.run_cost = 0.0
|
||||||
|
self.run_calls = 0
|
||||||
|
self.run_in = self.run_out = 0
|
||||||
|
|
||||||
|
def entries(self, since: dt.datetime | None = None) -> list[dict]:
|
||||||
|
if not self.path.exists():
|
||||||
|
return []
|
||||||
|
out = []
|
||||||
|
for line in self.path.read_text().splitlines():
|
||||||
|
try:
|
||||||
|
e = json.loads(line)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
continue
|
||||||
|
if since is None or dt.datetime.fromisoformat(e["ts"]) >= since:
|
||||||
|
out.append(e)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def spent_since(self, hours: float) -> float:
|
||||||
|
since = dt.datetime.now(dt.timezone.utc) - dt.timedelta(hours=hours)
|
||||||
|
return sum(e.get("cost_usd") or 0 for e in self.entries(since))
|
||||||
|
|
||||||
|
def check(self, estimate: float = 0.0) -> None:
|
||||||
|
"""Called BEFORE each model call with that call's estimated cost -- so a budget is never overshot."""
|
||||||
|
with self.lock:
|
||||||
|
if self.run_cost + estimate > self.run_budget:
|
||||||
|
raise BudgetExhausted(f"run budget ${self.run_budget:.2f}: ${self.run_cost:.3f} spent, "
|
||||||
|
f"next call est. ${estimate:.3f}")
|
||||||
|
day = self.spent_since(24)
|
||||||
|
if day + estimate > self.day_budget:
|
||||||
|
raise BudgetExhausted(f"24h budget ${self.day_budget:.2f}: ${day:.3f} spent, next call est. ${estimate:.3f}")
|
||||||
|
|
||||||
|
def record(self, label: str, model: str, usage: dict, cost: float | None) -> None:
|
||||||
|
e = {"ts": dt.datetime.now(dt.timezone.utc).isoformat(timespec="seconds"), "item": label, "model": model,
|
||||||
|
"input": (usage.get("input_tokens") or 0) + (usage.get("cache_creation_input_tokens") or 0)
|
||||||
|
+ (usage.get("cache_read_input_tokens") or 0),
|
||||||
|
"output": usage.get("output_tokens") or 0, "cost_usd": cost}
|
||||||
|
with self.lock:
|
||||||
|
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
with self.path.open("a") as f:
|
||||||
|
f.write(json.dumps(e) + "\n")
|
||||||
|
self.run_calls += 1
|
||||||
|
self.run_cost += cost or 0
|
||||||
|
self.run_in += e["input"]
|
||||||
|
self.run_out += e["output"]
|
||||||
|
|
||||||
|
def report(self) -> str:
|
||||||
|
return (f"usage this run: {self.run_calls} model calls, {self.run_in:,} in / {self.run_out:,} out tokens, "
|
||||||
|
f"${self.run_cost:.3f} list-price equiv | last 24h ${self.spent_since(24):.3f} of ${self.day_budget:.2f}")
|
||||||
|
|
||||||
|
|
||||||
|
class BudgetExhausted(RuntimeError):
|
||||||
|
"""A configured spend budget is reached -- stop the run the same way as a usage limit."""
|
||||||
|
|
||||||
|
|
||||||
|
class QuotaExhausted(RuntimeError):
|
||||||
|
"""The plan/API usage limit is hit. Every further call would fail the same way -- stop the run."""
|
||||||
|
|
||||||
|
|
||||||
|
_QUOTA = re.compile(r"(?i)hit your (?:session |usage |weekly )?limit|usage limit|quota|credit balance|resets? (?:at )?\d")
|
||||||
|
|
||||||
|
|
||||||
|
class LLM:
|
||||||
|
"""Two backends.
|
||||||
|
|
||||||
|
claude-cli (default): headless Claude Code on the user's own plan -- no API key to manage.
|
||||||
|
--no-session-persistence is load-bearing: without it every summary writes a new transcript
|
||||||
|
into ~/.claude/projects and the next sync would summarize its own summarizing.
|
||||||
|
openai: any OpenAI-compatible /chat/completions endpoint (LiteLLM, local models).
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, cfg: dict, ledger: "Ledger | None" = None):
|
||||||
|
self.ledger = ledger or Ledger(cfg)
|
||||||
|
self.price_in = float(cfg["LLM_PRICE_IN_PER_M"]) / 1e6
|
||||||
|
self.price_out = float(cfg["LLM_PRICE_OUT_PER_M"]) / 1e6
|
||||||
|
self.label = threading.local()
|
||||||
|
self.backend = cfg["LLM_BACKEND"]
|
||||||
|
self.model = cfg["LLM_MODEL"]
|
||||||
|
self.resolved = cfg["LLM_MODEL"]
|
||||||
|
if self.backend == "claude-cli":
|
||||||
|
self.bin = shutil.which(os.path.expanduser(cfg["CLAUDE_BIN"]))
|
||||||
|
if not self.bin:
|
||||||
|
raise SystemExit(f"claude CLI not found ({cfg['CLAUDE_BIN']})")
|
||||||
|
self.workdir = pathlib.Path(os.path.expanduser(cfg["STATE_FILE"])).parent # no project CLAUDE.md here
|
||||||
|
self.workdir.mkdir(parents=True, exist_ok=True)
|
||||||
|
elif self.backend == "openai":
|
||||||
|
if not cfg["LLM_BASE_URL"]:
|
||||||
|
raise SystemExit("LLM_BASE_URL is not set (see config.env.example)")
|
||||||
|
self.url = cfg["LLM_BASE_URL"].rstrip("/") + "/chat/completions"
|
||||||
|
self.key = secret(cfg, "LLM_API_KEY_FILE", "LLM_API_KEY_ENV")
|
||||||
|
else:
|
||||||
|
raise SystemExit(f"unknown LLM_BACKEND {self.backend!r}")
|
||||||
|
|
||||||
|
def estimate(self, system: str, user: str, max_tokens: int) -> float:
|
||||||
|
# ~3 chars/token over-estimates input (measured 3.1 on transcripts); assume the full output;
|
||||||
|
# +20% margin for anything unmeasured. Errs high on purpose: refusing early beats overshooting.
|
||||||
|
return 1.2 * ((len(system) + len(user)) / 3.0 * self.price_in + max_tokens * self.price_out)
|
||||||
|
|
||||||
|
def chat(self, system: str, user: str, max_tokens: int = 2500) -> str:
|
||||||
|
self.ledger.check(self.estimate(system, user, max_tokens))
|
||||||
|
return self._chat_cli(system, user) if self.backend == "claude-cli" else self._chat_openai(system, user, max_tokens)
|
||||||
|
|
||||||
|
def _chat_cli(self, system: str, user: str) -> str:
|
||||||
|
# Cost levers, measured on one session: $0.057 -> $0.015 with identical summary content.
|
||||||
|
# alwaysThinkingEnabled=false -- extended thinking was ~85% of output tokens
|
||||||
|
# DISABLE_PROMPT_CACHING=1 -- Claude Code writes prompts to the 1h cache at 2x input
|
||||||
|
# price; a one-shot call never reads it back
|
||||||
|
# CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 -- otherwise a hidden side request re-reads the
|
||||||
|
# whole prompt (~20 tokens out, title-generation shaped),
|
||||||
|
# doubling billed input: 9,638 vs 4,663 visible tokens.
|
||||||
|
cmd = [self.bin, "-p", "--model", self.model, "--no-session-persistence", "--tools", "",
|
||||||
|
"--strict-mcp-config", "--output-format", "json",
|
||||||
|
"--settings", json.dumps({"alwaysThinkingEnabled": False}), "--system-prompt", system]
|
||||||
|
env = dict(os.environ, DISABLE_PROMPT_CACHING="1", CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC="1")
|
||||||
|
last = ""
|
||||||
|
for attempt in range(5):
|
||||||
|
try:
|
||||||
|
proc = subprocess.run(cmd, input=user, capture_output=True, text=True, timeout=900, cwd=self.workdir, env=env)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
last = "timeout"
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
d = json.loads(proc.stdout)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
d = {"is_error": True, "result": (proc.stdout or proc.stderr)[-300:]}
|
||||||
|
if d.get("usage") or d.get("total_cost_usd"): # failed calls can still cost
|
||||||
|
# modelUsage is what was BILLED across every request the CLI made; `usage` is only
|
||||||
|
# the visible turn and undercounted by half before the nonessential-traffic fix.
|
||||||
|
mu = list((d.get("modelUsage") or {}).values())
|
||||||
|
billed = {"input_tokens": sum((v.get("inputTokens") or 0) + (v.get("cacheReadInputTokens") or 0)
|
||||||
|
+ (v.get("cacheCreationInputTokens") or 0) for v in mu),
|
||||||
|
"output_tokens": sum(v.get("outputTokens") or 0 for v in mu)} if mu else (d.get("usage") or {})
|
||||||
|
self.ledger.record(getattr(self.label, "v", "?"), self.model, billed, d.get("total_cost_usd"))
|
||||||
|
if not d.get("is_error") and d.get("result"):
|
||||||
|
used = list((d.get("modelUsage") or {}).keys())
|
||||||
|
if used:
|
||||||
|
self.resolved = used[0]
|
||||||
|
return d["result"]
|
||||||
|
last = str(d.get("result") or proc.stderr)[-300:]
|
||||||
|
if _QUOTA.search(last):
|
||||||
|
raise QuotaExhausted(last.strip())
|
||||||
|
if not re.search(r"(?i)rate.?limit|overloaded|429|529|timeout|temporar", last):
|
||||||
|
break # not transient -- do not hammer
|
||||||
|
time.sleep(2 ** attempt * 10)
|
||||||
|
raise RuntimeError(f"claude -p failed: {last}")
|
||||||
|
|
||||||
|
def _chat_openai(self, system: str, user: str, max_tokens: int) -> str:
|
||||||
|
body = json.dumps({"model": self.model, "temperature": 0.2, "max_tokens": max_tokens,
|
||||||
|
"messages": [{"role": "system", "content": system},
|
||||||
|
{"role": "user", "content": user}]}).encode()
|
||||||
|
for attempt in range(5):
|
||||||
|
req = urllib.request.Request(self.url, data=body, method="POST", headers={
|
||||||
|
"Authorization": f"Bearer {self.key}", "Content-Type": "application/json"})
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=600) as r:
|
||||||
|
d = json.load(r)
|
||||||
|
u = d.get("usage") or {}
|
||||||
|
self.ledger.record(getattr(self.label, "v", "?"), self.model,
|
||||||
|
{"input_tokens": u.get("prompt_tokens"), "output_tokens": u.get("completion_tokens")}, None)
|
||||||
|
return d["choices"][0]["message"]["content"]
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
detail = e.read()[:300].decode(errors="replace")
|
||||||
|
if _QUOTA.search(detail):
|
||||||
|
raise QuotaExhausted(detail) from None
|
||||||
|
if e.code in (429, 500, 502, 503, 504, 529) and attempt < 4:
|
||||||
|
time.sleep(2 ** attempt * 5)
|
||||||
|
continue
|
||||||
|
raise RuntimeError(f"LLM HTTP {e.code}: {detail}") from None
|
||||||
|
except (urllib.error.URLError, TimeoutError) as e:
|
||||||
|
if attempt < 4:
|
||||||
|
time.sleep(2 ** attempt * 5)
|
||||||
|
continue
|
||||||
|
raise RuntimeError(f"LLM unreachable: {e}") from None
|
||||||
|
raise RuntimeError("LLM retries exhausted")
|
||||||
|
|
||||||
|
def summarize(self, meta: dict) -> dict:
|
||||||
|
text = redact(meta["transcript"])
|
||||||
|
if len(text) > CHUNK_CHARS: # map-reduce for long sessions
|
||||||
|
chunks, cur = [], []
|
||||||
|
size = 0
|
||||||
|
for part in text.split("\n\n"):
|
||||||
|
if size + len(part) > CHUNK_CHARS and cur:
|
||||||
|
chunks.append("\n\n".join(cur)); cur, size = [], 0
|
||||||
|
cur.append(part); size += len(part) + 2
|
||||||
|
chunks.append("\n\n".join(cur))
|
||||||
|
notes = [self.chat(CHUNK_SYSTEM, f"Part {i + 1} of {len(chunks)}:\n\n{c}", 3000)
|
||||||
|
for i, c in enumerate(chunks)]
|
||||||
|
text = "Condensed notes from each part of a long conversation, in order:\n\n" + \
|
||||||
|
"\n\n---\n\n".join(notes)
|
||||||
|
header = f"Working directory: {meta.get('cwd')}\nGit branch: {meta.get('branch')}\n" \
|
||||||
|
f"Existing title: {meta.get('title')}\n\n"
|
||||||
|
raw = self.chat(SYSTEM, header + text)
|
||||||
|
m = re.search(r"\{.*\}", raw, re.S)
|
||||||
|
if not m:
|
||||||
|
raise RuntimeError(f"LLM did not return JSON: {raw[:200]!r}")
|
||||||
|
out = json.loads(m.group(0))
|
||||||
|
for k in ("outcomes", "decisions", "facts", "open_threads", "tags"):
|
||||||
|
out[k] = [redact(str(x)) for x in (out.get(k) or []) if str(x).strip()]
|
||||||
|
out["summary"] = redact(str(out.get("summary", "")))
|
||||||
|
out["title"] = redact(str(out.get("title") or meta.get("title") or meta["session_id"]))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- vault
|
||||||
|
class Vault:
|
||||||
|
def __init__(self, cfg: dict):
|
||||||
|
# OBSIDIAN_API_URL may list several URLs, comma-separated, tried in order at ping() -- e.g. the
|
||||||
|
# permanent address first and a port-forward second, so one config works either way.
|
||||||
|
self.candidates = [u.strip().rstrip("/") for u in cfg["OBSIDIAN_API_URL"].split(",") if u.strip()]
|
||||||
|
self.base = self.candidates[0]
|
||||||
|
self.key = secret(cfg, "OBSIDIAN_API_KEY_FILE")
|
||||||
|
|
||||||
|
def put(self, path: str, content: str) -> None:
|
||||||
|
req = urllib.request.Request(f"{self.base}/vault/{urllib.parse.quote(path)}",
|
||||||
|
data=content.encode(), method="PUT", headers={
|
||||||
|
"Authorization": f"Bearer {self.key}",
|
||||||
|
"Content-Type": "text/markdown"})
|
||||||
|
with urllib.request.urlopen(req, timeout=60) as r:
|
||||||
|
if r.status not in (200, 201, 204):
|
||||||
|
raise RuntimeError(f"vault PUT {path}: HTTP {r.status}")
|
||||||
|
|
||||||
|
def ping(self) -> None:
|
||||||
|
"""Checked BEFORE any summarizing, so an unreachable vault never spends usage limits."""
|
||||||
|
errors = []
|
||||||
|
for base in self.candidates:
|
||||||
|
req = urllib.request.Request(f"{base}/", headers={"Authorization": f"Bearer {self.key}"})
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=10) as r:
|
||||||
|
info = json.load(r)
|
||||||
|
self.base = base
|
||||||
|
break
|
||||||
|
except (urllib.error.URLError, TimeoutError, OSError, ValueError) as e:
|
||||||
|
errors.append(f"{base}: {getattr(e, 'reason', e)}")
|
||||||
|
else:
|
||||||
|
raise SystemExit("Obsidian REST API unreachable -- " + "; ".join(errors))
|
||||||
|
print(f"vault: {self.base}", flush=True)
|
||||||
|
if not info.get("authenticated"):
|
||||||
|
raise SystemExit("Obsidian REST API reachable but the API key was rejected")
|
||||||
|
|
||||||
|
|
||||||
|
_BAD = re.compile(r'[\\/:*?"<>|#^\[\]]+')
|
||||||
|
|
||||||
|
|
||||||
|
def note_path(cfg: dict, kind: str, meta: dict, title: str) -> str:
|
||||||
|
when = (meta.get("first") or dt.datetime.now(dt.timezone.utc).isoformat())[:10]
|
||||||
|
slug = " ".join(_BAD.sub(" ", title).split())[:60].strip() or "untitled"
|
||||||
|
return f"{cfg['VAULT_FOLDER']}/{kind}/{when[:4]}/{when} {slug} ({meta['session_id'][:8]}).md"
|
||||||
|
|
||||||
|
|
||||||
|
def render(meta: dict, s: dict, source: str, model: str, src_file: str | None) -> str:
|
||||||
|
project = pathlib.Path(meta["cwd"]).name if meta.get("cwd") else None
|
||||||
|
tags = ["claude", source] + [t for t in s["tags"] if t not in ("claude", source)]
|
||||||
|
fm = {
|
||||||
|
"type": "claude-conversation", "source": source, "session_id": meta["session_id"],
|
||||||
|
"title": s["title"], "project": project, "cwd": meta.get("cwd"), "branch": meta.get("branch"),
|
||||||
|
"started": meta.get("first"), "ended": meta.get("last"), "prompts": meta.get("prompts"),
|
||||||
|
"summarized_by": model, "synced": dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
||||||
|
"tags": tags,
|
||||||
|
}
|
||||||
|
lines = ["---"] + [f"{k}: {json.dumps(v)}" for k, v in fm.items() if v not in (None, "")] + ["---", "",
|
||||||
|
f"# {s['title']}", "", "> [!summary]"] + [f"> {l}" for l in s["summary"].splitlines() or [""]]
|
||||||
|
for heading, key in (("Outcomes", "outcomes"), ("Decisions", "decisions"),
|
||||||
|
("Facts & gotchas", "facts"), ("Open threads", "open_threads")):
|
||||||
|
if s[key]:
|
||||||
|
lines += ["", f"## {heading}", ""] + [f"- {x}" for x in s[key]]
|
||||||
|
lines += ["", "---"]
|
||||||
|
if source == "claude-code":
|
||||||
|
lines.append(f"Resume: `claude --resume {meta['session_id']}`" + (f" · transcript `{src_file}`" if src_file else ""))
|
||||||
|
else:
|
||||||
|
lines.append(f"claude.ai conversation `{meta['session_id']}`")
|
||||||
|
return "\n".join(lines) + "\n"
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- state
|
||||||
|
class State:
|
||||||
|
def __init__(self, path: str):
|
||||||
|
self.path = pathlib.Path(os.path.expanduser(path))
|
||||||
|
self.lock = threading.Lock()
|
||||||
|
self.data = json.loads(self.path.read_text()) if self.path.exists() else {}
|
||||||
|
|
||||||
|
def save(self) -> None:
|
||||||
|
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
tmp = self.path.with_suffix(".tmp")
|
||||||
|
tmp.write_text(json.dumps(self.data, indent=1, sort_keys=True))
|
||||||
|
tmp.replace(self.path)
|
||||||
|
|
||||||
|
def record(self, key: str, value: dict) -> None:
|
||||||
|
with self.lock:
|
||||||
|
self.data[key] = value
|
||||||
|
self.save()
|
||||||
|
|
||||||
|
|
||||||
|
def write_index(cfg: dict, vault: Vault, state: State) -> None:
|
||||||
|
rows = sorted((v for v in state.data.values() if v.get("note")), key=lambda v: v.get("started") or "", reverse=True)
|
||||||
|
out = ["---", 'type: "claude-index"', "---", "", "# Claude conversations", "",
|
||||||
|
f"{len(rows)} conversations. Regenerated by claude-obsidian-sync.", "",
|
||||||
|
"| Date | Title | Project | Source |", "|---|---|---|---|"]
|
||||||
|
for v in rows:
|
||||||
|
link = v["note"][:-3]
|
||||||
|
title = v.get("title", "").replace("|", "-")
|
||||||
|
out.append(f"| {(v.get('started') or '')[:10]} | [[{link}\\|{title}]] | {v.get('project') or ''} | {v.get('source')} |")
|
||||||
|
vault.put(f"{cfg['VAULT_FOLDER']}/Index.md", "\n".join(out) + "\n")
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- commands
|
||||||
|
def top_level_sessions(cfg: dict) -> list[pathlib.Path]:
|
||||||
|
root = pathlib.Path(os.path.expanduser(cfg["CLAUDE_PROJECTS_DIR"]))
|
||||||
|
return sorted(root.glob("*/*.jsonl"), key=lambda p: p.stat().st_mtime) # nested = subagents
|
||||||
|
|
||||||
|
|
||||||
|
def process(cfg, llm, vault, state, key, meta, source, kind, fingerprint, src_file, dry_run) -> str:
|
||||||
|
# Cache the summary until the note is written, so a vault failure never pays for it twice.
|
||||||
|
cache = pathlib.Path(os.path.expanduser(cfg["STATE_FILE"])).parent / "cache" / \
|
||||||
|
f"{hashlib.sha1(key.encode()).hexdigest()[:16]}-{hashlib.sha1(fingerprint.encode()).hexdigest()[:8]}.json"
|
||||||
|
llm.label.v = key
|
||||||
|
if cache.exists():
|
||||||
|
s = json.loads(cache.read_text())
|
||||||
|
else:
|
||||||
|
s = llm.summarize(meta)
|
||||||
|
cache.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
cache.write_text(json.dumps(s))
|
||||||
|
prev = state.data.get(key, {})
|
||||||
|
path = prev.get("note") or note_path(cfg, kind, meta, s["title"]) # stable across re-syncs
|
||||||
|
body = render(meta, s, source, llm.resolved, src_file)
|
||||||
|
if dry_run == "prefetch":
|
||||||
|
return f"summary cached ({path})"
|
||||||
|
if dry_run:
|
||||||
|
print(f"\n===== {path} =====\n{body}")
|
||||||
|
return path
|
||||||
|
vault.put(path, body)
|
||||||
|
cache.unlink(missing_ok=True)
|
||||||
|
state.record(key, {"note": path, "title": s["title"], "fingerprint": fingerprint, "source": source,
|
||||||
|
"started": meta.get("first"),
|
||||||
|
"project": pathlib.Path(meta["cwd"]).name if meta.get("cwd") else None})
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
SKIPPED = "(empty, skipped)"
|
||||||
|
EX_TEMPFAIL = 75 # "try again later" -- systemd logs it, the next timer run retries
|
||||||
|
|
||||||
|
|
||||||
|
def run_jobs(cfg, work, fn) -> tuple[int, int, str | None, int]:
|
||||||
|
ok = failed = skipped = 0
|
||||||
|
quota = None
|
||||||
|
stop = threading.Event() # Future.cancel() cannot stop a job a worker already picked up
|
||||||
|
|
||||||
|
def guarded(*w):
|
||||||
|
if stop.is_set():
|
||||||
|
raise cf.CancelledError()
|
||||||
|
try:
|
||||||
|
return fn(*w)
|
||||||
|
except (QuotaExhausted, BudgetExhausted):
|
||||||
|
stop.set()
|
||||||
|
raise
|
||||||
|
|
||||||
|
with cf.ThreadPoolExecutor(max_workers=int(cfg["JOBS"])) as ex:
|
||||||
|
futs = {ex.submit(guarded, *w): w for w in work}
|
||||||
|
for fut in cf.as_completed(futs):
|
||||||
|
label = futs[fut][0]
|
||||||
|
try:
|
||||||
|
result = fut.result()
|
||||||
|
if result == SKIPPED:
|
||||||
|
print(f"skip {label} (no conversation content)", flush=True); skipped += 1
|
||||||
|
else:
|
||||||
|
print(f"ok {label} -> {result}", flush=True); ok += 1
|
||||||
|
except (QuotaExhausted, BudgetExhausted) as e:
|
||||||
|
if quota is None:
|
||||||
|
quota = str(e)
|
||||||
|
for f in futs: # don't launch the rest
|
||||||
|
f.cancel()
|
||||||
|
except cf.CancelledError:
|
||||||
|
pass
|
||||||
|
except Exception as e: # keep going; next run retries
|
||||||
|
print(f"FAIL {label}: {e}", file=sys.stderr, flush=True); failed += 1
|
||||||
|
return ok, failed, quota, skipped
|
||||||
|
|
||||||
|
|
||||||
|
def finish(ok: int, failed: int, quota: str | None, dry_run, remaining: int, ledger: "Ledger | None" = None) -> None:
|
||||||
|
verb = {"prefetch": "summarized into cache (vault untouched)"}.get(dry_run) or \
|
||||||
|
("rendered (dry run, nothing written)" if dry_run else "written")
|
||||||
|
print(f"done: {ok} {verb}, {failed} failed, {remaining} still queued")
|
||||||
|
if ledger:
|
||||||
|
print(ledger.report())
|
||||||
|
if quota:
|
||||||
|
print(f"stopped: {quota}", file=sys.stderr)
|
||||||
|
sys.exit(EX_TEMPFAIL)
|
||||||
|
sys.exit(1 if failed else 0)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_usage(cfg, args) -> None:
|
||||||
|
led = Ledger(cfg)
|
||||||
|
rows = led.entries()
|
||||||
|
if not rows:
|
||||||
|
print("no model calls recorded yet"); return
|
||||||
|
for label, hours in (("last 5h", 5), ("last 24h", 24), ("last 7d", 168)):
|
||||||
|
since = dt.datetime.now(dt.timezone.utc) - dt.timedelta(hours=hours)
|
||||||
|
es = [e for e in rows if dt.datetime.fromisoformat(e["ts"]) >= since]
|
||||||
|
print(f"{label:>9}: {len(es):4d} calls {sum(e['input'] for e in es):>10,} in "
|
||||||
|
f"{sum(e['output'] for e in es):>8,} out ${sum(e.get('cost_usd') or 0 for e in es):.3f}")
|
||||||
|
print(f"budgets: ${led.run_budget:.2f}/run, ${led.day_budget:.2f}/24h (list-price equiv; plan meter: /usage in Claude Code)")
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_stats(cfg, args) -> None:
|
||||||
|
total = 0
|
||||||
|
rows = []
|
||||||
|
for p in top_level_sessions(cfg):
|
||||||
|
m = parse_claude_code(p)
|
||||||
|
n = len(redact(m["transcript"]))
|
||||||
|
total += n
|
||||||
|
rows.append((n, p.stat().st_size, m["prompts"], m["title"] or p.stem))
|
||||||
|
rows.sort(reverse=True)
|
||||||
|
for n, raw, msgs, title in rows[: args.top]:
|
||||||
|
print(f"{raw / 1e6:7.2f} MB raw -> {n / 1e3:7.1f} KB condensed {msgs:4d} prompts {title[:60]}")
|
||||||
|
print(f"\n{len(rows)} sessions, {total / 1e6:.1f} MB condensed (~{total / 4 / 1e6:.2f}M tokens in), "
|
||||||
|
f"{sum(1 for r in rows if r[0] > CHUNK_CHARS)} need map-reduce")
|
||||||
|
|
||||||
|
|
||||||
|
def run_lock(cfg: dict):
|
||||||
|
"""One run at a time: a timer run overlapping a manual one would race on state and ledger files."""
|
||||||
|
path = pathlib.Path(os.path.expanduser(cfg["STATE_FILE"])).parent / "run.lock"
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
fh = path.open("w")
|
||||||
|
try:
|
||||||
|
fcntl.flock(fh, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||||
|
except BlockingIOError:
|
||||||
|
print("another claude-obsidian-sync run is in progress -- exiting")
|
||||||
|
sys.exit(0)
|
||||||
|
return fh # held until the process exits
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_sync(cfg, args) -> None:
|
||||||
|
_lock = run_lock(cfg)
|
||||||
|
state = State(cfg["STATE_FILE"])
|
||||||
|
settle = dt.timedelta(minutes=int(cfg["SETTLE_MINUTES"]))
|
||||||
|
now = time.time()
|
||||||
|
work = []
|
||||||
|
for p in top_level_sessions(cfg):
|
||||||
|
if args.session and p.stem != args.session:
|
||||||
|
continue
|
||||||
|
st = p.stat()
|
||||||
|
fp = f"{st.st_size}:{int(st.st_mtime)}"
|
||||||
|
key = f"claude-code:{p.stem}"
|
||||||
|
if not args.force and state.data.get(key, {}).get("fingerprint") == fp:
|
||||||
|
continue
|
||||||
|
if not (args.force or args.session) and now - st.st_mtime < settle.total_seconds():
|
||||||
|
continue # still being written
|
||||||
|
work.append((p, fp, key))
|
||||||
|
total = len(work)
|
||||||
|
cap = args.limit or int(cfg["MAX_PER_RUN"])
|
||||||
|
work = work[-cap:] if not args.oldest else work[:cap] # newest first by default
|
||||||
|
print(f"{total} session(s) pending, summarizing {len(work)} this run", flush=True)
|
||||||
|
if not work:
|
||||||
|
return
|
||||||
|
prefetch = getattr(args, "prefetch", False)
|
||||||
|
llm, vault = LLM(cfg), (None if args.dry_run or prefetch else Vault(cfg))
|
||||||
|
if vault:
|
||||||
|
vault.ping()
|
||||||
|
mode = "prefetch" if prefetch else args.dry_run
|
||||||
|
|
||||||
|
def one(p, fp, key):
|
||||||
|
meta = parse_claude_code(p)
|
||||||
|
if not meta["transcript"].strip():
|
||||||
|
if not args.dry_run and not prefetch:
|
||||||
|
state.record(key, {"note": None, "fingerprint": fp, "source": "claude-code"})
|
||||||
|
return SKIPPED
|
||||||
|
src = str(p).replace(str(HOME), "~")
|
||||||
|
return process(cfg, llm, vault, state, key, meta, "claude-code", "Code", fp, src, mode)
|
||||||
|
|
||||||
|
ok, failed, quota, skipped = run_jobs(cfg, [(w[0].name, *w) for w in work], lambda _l, p, fp, key: one(p, fp, key))
|
||||||
|
if vault and ok:
|
||||||
|
write_index(cfg, vault, state)
|
||||||
|
finish(ok, failed, quota, mode, total - ok - skipped, llm.ledger)
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_import(cfg, args) -> None:
|
||||||
|
_lock = run_lock(cfg)
|
||||||
|
state = State(cfg["STATE_FILE"])
|
||||||
|
convs = parse_claude_ai_export(pathlib.Path(args.export))
|
||||||
|
work = []
|
||||||
|
for m in convs:
|
||||||
|
key = f"claude-ai:{m['session_id']}"
|
||||||
|
fp = m.get("last") or str(len(m["transcript"]))
|
||||||
|
if args.force or state.data.get(key, {}).get("fingerprint") != fp:
|
||||||
|
work.append((m, fp, key))
|
||||||
|
total = len(work)
|
||||||
|
work = work[: args.limit or int(cfg["MAX_PER_RUN"])]
|
||||||
|
print(f"{len(convs)} conversations in export, {total} new or changed, summarizing {len(work)} this run", flush=True)
|
||||||
|
if not work:
|
||||||
|
return
|
||||||
|
llm, vault = LLM(cfg), (None if args.dry_run else Vault(cfg))
|
||||||
|
if vault:
|
||||||
|
vault.ping()
|
||||||
|
ok, failed, quota, skipped = run_jobs(cfg, [((m.get("title") or m["session_id"])[:50], m, fp, key) for m, fp, key in work],
|
||||||
|
lambda _l, m, fp, key: process(cfg, llm, vault, state, key, m, "claude-ai", "Web",
|
||||||
|
fp, None, args.dry_run))
|
||||||
|
if vault and ok:
|
||||||
|
write_index(cfg, vault, state)
|
||||||
|
finish(ok, failed, quota, args.dry_run, total - ok - skipped, llm.ledger)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
|
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||||
|
sub.add_parser("usage", help="model calls, tokens and spend from the local ledger")
|
||||||
|
s = sub.add_parser("stats", help="condensed sizes, no network")
|
||||||
|
s.add_argument("--top", type=int, default=15)
|
||||||
|
s = sub.add_parser("sync", help="summarize Claude Code sessions")
|
||||||
|
s.add_argument("--dry-run", action="store_true", help="print notes instead of writing them")
|
||||||
|
s.add_argument("--prefetch", action="store_true",
|
||||||
|
help="summarize into the local cache without touching the vault; a later sync writes them at no extra cost")
|
||||||
|
s.add_argument("--force", action="store_true", help="ignore state and the settle window")
|
||||||
|
s.add_argument("--session", help="only this session id")
|
||||||
|
s.add_argument("--limit", type=int, help="max sessions this run (default MAX_PER_RUN)")
|
||||||
|
s.add_argument("--oldest", action="store_true", help="take the oldest pending sessions instead of the newest")
|
||||||
|
s = sub.add_parser("import-claude-ai", help="import a claude.ai data export")
|
||||||
|
s.add_argument("export", help="the export .zip or its conversations.json")
|
||||||
|
s.add_argument("--dry-run", action="store_true")
|
||||||
|
s.add_argument("--force", action="store_true")
|
||||||
|
s.add_argument("--limit", type=int, help="max conversations this run (default MAX_PER_RUN)")
|
||||||
|
args = ap.parse_args()
|
||||||
|
cfg = load_config()
|
||||||
|
{"stats": cmd_stats, "usage": cmd_usage, "sync": cmd_sync, "import-claude-ai": cmd_import}[args.cmd](cfg, args)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# ~/.config/claude-obsidian/config.env (chmod 600; never commit the real one)
|
||||||
|
# Every key can also be set as an environment variable, which wins.
|
||||||
|
|
||||||
|
# Obsidian Local REST API plugin -- the plain-HTTP server (Settings -> Local REST API ->
|
||||||
|
# Enable HTTP server). Point this at wherever that port is reachable from this machine.
|
||||||
|
# Several URLs, comma-separated, are tried in order -- e.g. a permanent address, then a port-forward.
|
||||||
|
OBSIDIAN_API_URL=http://obsidian.example.internal:27123,http://127.0.0.1:27123
|
||||||
|
OBSIDIAN_API_KEY_FILE=~/.config/claude-obsidian/api-key
|
||||||
|
|
||||||
|
# Summarizer. claude-cli = headless `claude -p` on your own Claude plan (no API key needed).
|
||||||
|
LLM_BACKEND=claude-cli
|
||||||
|
LLM_MODEL=haiku # sonnet writes noticeably sharper notes for ~1-3x the usage; set prices to match below
|
||||||
|
# LLM_PRICE_IN_PER_M=1.0 # used only for the pre-call budget estimate (haiku list; sonnet 3.0 / 15.0)
|
||||||
|
# LLM_PRICE_OUT_PER_M=5.0
|
||||||
|
# CLAUDE_BIN=claude
|
||||||
|
|
||||||
|
# Or any OpenAI-compatible endpoint (LiteLLM, a local model):
|
||||||
|
# LLM_BACKEND=openai
|
||||||
|
# LLM_BASE_URL=https://litellm.example/v1
|
||||||
|
# LLM_MODEL=some-model
|
||||||
|
# LLM_API_KEY_FILE=~/.config/claude-obsidian/llm-key # or LLM_API_KEY_ENV=LITELLM_KEY
|
||||||
|
|
||||||
|
VAULT_FOLDER=Claude
|
||||||
|
SETTLE_MINUTES=30 # leave sessions alone until they have been idle this long
|
||||||
|
JOBS=1
|
||||||
|
MAX_PER_RUN=10 # summaries share your plan's usage limits; the timer drips a backlog
|
||||||
|
BUDGET_USD_PER_RUN=0.50 # list-price USD, refused pre-flight; see `claude-obsidian-sync usage`
|
||||||
|
BUDGET_USD_PER_DAY=2.00
|
||||||
Executable
+14
@@ -0,0 +1,14 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# Install claude-obsidian-sync for the current user: binary into ~/.local/bin, systemd user
|
||||||
|
# timer enabled. Idempotent. Does not create config or secrets -- see README.md.
|
||||||
|
set -eu
|
||||||
|
HERE="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
|
||||||
|
mkdir -p "$HOME/.local/bin" "$HOME/.config/systemd/user" "$HOME/.config/claude-obsidian"
|
||||||
|
chmod 700 "$HOME/.config/claude-obsidian"
|
||||||
|
ln -sf "$HERE/claude-obsidian-sync" "$HOME/.local/bin/claude-obsidian-sync"
|
||||||
|
cp "$HERE/systemd/claude-obsidian-sync.service" "$HERE/systemd/claude-obsidian-sync.timer" "$HOME/.config/systemd/user/"
|
||||||
|
[ -f "$HOME/.config/claude-obsidian/config.env" ] || { cp "$HERE/config.env.example" "$HOME/.config/claude-obsidian/config.env"; chmod 600 "$HOME/.config/claude-obsidian/config.env"; echo "created config.env from the example -- edit it"; }
|
||||||
|
systemctl --user daemon-reload
|
||||||
|
systemctl --user enable --now claude-obsidian-sync.timer
|
||||||
|
echo "installed. check: systemctl --user list-timers claude-obsidian-sync.timer"
|
||||||
|
echo "timers only run while logged in unless lingering is on: loginctl enable-linger \"\$USER\""
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Summarize Claude conversations into Obsidian
|
||||||
|
After=network-online.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
ExecStart=%h/.local/bin/claude-obsidian-sync sync
|
||||||
|
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
|
||||||
|
# 75 = usage limit reached. The run stopped cleanly and the next timer run resumes -- not a failure.
|
||||||
|
SuccessExitStatus=75
|
||||||
|
Nice=10
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Hourly Claude -> Obsidian conversation sync
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnCalendar=hourly
|
||||||
|
RandomizedDelaySec=10m
|
||||||
|
Persistent=true
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
@@ -1,114 +0,0 @@
|
|||||||
#!/usr/bin/env sh
|
|
||||||
# docker/install.sh — install full (rootful) Docker Engine and let the
|
|
||||||
# invoking user talk to it without sudo.
|
|
||||||
#
|
|
||||||
# What it does:
|
|
||||||
# 1. Installs Docker Engine from Docker's upstream package repo. Skipped when
|
|
||||||
# `docker` is already installed. Debian-family hosts go through Docker's
|
|
||||||
# convenience script (get.docker.com); RHEL-family hosts wire up the
|
|
||||||
# docker-ce repo directly — see install_rhel() for why.
|
|
||||||
# 2. Enables and starts the service (systemd hosts only).
|
|
||||||
# 3. Adds the invoking user to the `docker` group so the daemon socket is
|
|
||||||
# usable without sudo. Takes effect at next login.
|
|
||||||
#
|
|
||||||
# Deliberately no-ops when a rootless Docker daemon is detected (e.g. Kasm
|
|
||||||
# ships its own rootless setup) — installing rootful Docker there would fight
|
|
||||||
# the existing runtime. See docker-rootless-cgroup/ for the Kasm-side config.
|
|
||||||
#
|
|
||||||
# Idempotent and safe to re-run. Uses sudo.
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
|
|
||||||
REPO_ROOT="$(dirname -- "$SCRIPT_DIR")"
|
|
||||||
|
|
||||||
# shellcheck source=../lib/os-detect.sh
|
|
||||||
. "$REPO_ROOT/lib/os-detect.sh"
|
|
||||||
|
|
||||||
say() { printf '%s\n' "$*" >&2; }
|
|
||||||
|
|
||||||
DOCKER_PKGS="docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin"
|
|
||||||
|
|
||||||
# install_rhel — install docker-ce from Docker's EL repo.
|
|
||||||
#
|
|
||||||
# WHY NOT get.docker.com: the convenience script allowlists distros by
|
|
||||||
# /etc/os-release ID and aborts on anything it does not recognize —
|
|
||||||
# "ERROR: Unsupported distribution 'ol'" on Oracle Linux, and the same for
|
|
||||||
# other EL rebuilds it has not enumerated. On a match it does nothing more
|
|
||||||
# than what this function does: add the repo, then install the packages.
|
|
||||||
# Wiring the repo up directly drops the allowlist, so every EL rebuild
|
|
||||||
# (Oracle, Rocky, Alma, CentOS, RHEL) installs from one code path.
|
|
||||||
#
|
|
||||||
# The repo URL is keyed on $releasever/$basearch, which dnf expands per host,
|
|
||||||
# so EL9 and EL10 both resolve without a version table here.
|
|
||||||
install_rhel() {
|
|
||||||
say "docker: adding Docker CE repo (EL \$releasever/\$basearch)..."
|
|
||||||
sudo tee /etc/yum.repos.d/docker-ce.repo >/dev/null <<'REPO'
|
|
||||||
[docker-ce-stable]
|
|
||||||
name=Docker CE Stable - $basearch
|
|
||||||
baseurl=https://download.docker.com/linux/centos/$releasever/$basearch/stable
|
|
||||||
enabled=1
|
|
||||||
gpgcheck=1
|
|
||||||
gpgkey=https://download.docker.com/linux/centos/gpg
|
|
||||||
REPO
|
|
||||||
# --allowerasing: containerd.io conflicts with the distro's own runc/containerd
|
|
||||||
# on EL rebuilds that ship them (Oracle Linux does). Let dnf swap them out
|
|
||||||
# rather than dead-end on the conflict.
|
|
||||||
if command -v dnf >/dev/null 2>&1; then
|
|
||||||
# shellcheck disable=SC2086
|
|
||||||
sudo dnf install -y --allowerasing $DOCKER_PKGS
|
|
||||||
else
|
|
||||||
# shellcheck disable=SC2086
|
|
||||||
sudo yum install -y --allowerasing $DOCKER_PKGS
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
# install_debian — Docker's convenience script; it supports every Debian-family
|
|
||||||
# distro this repo targets and sets up the apt repo and keyring.
|
|
||||||
install_debian() {
|
|
||||||
say "docker: installing Docker Engine via get.docker.com..."
|
|
||||||
tmp="$(mktemp)"
|
|
||||||
if command -v curl >/dev/null 2>&1; then
|
|
||||||
curl -fsSL https://get.docker.com -o "$tmp"
|
|
||||||
elif command -v wget >/dev/null 2>&1; then
|
|
||||||
wget -qO "$tmp" https://get.docker.com
|
|
||||||
else
|
|
||||||
say "docker: need curl or wget"; rm -f "$tmp"; return 1
|
|
||||||
fi
|
|
||||||
sudo sh "$tmp"
|
|
||||||
rm -f "$tmp"
|
|
||||||
}
|
|
||||||
|
|
||||||
# --- skip on rootless-Docker hosts -------------------------------------------
|
|
||||||
if command -v docker >/dev/null 2>&1 &&
|
|
||||||
docker info -f '{{println .SecurityOptions}}' 2>/dev/null | grep -q 'name=rootless'; then
|
|
||||||
say "docker: rootless Docker daemon detected; leaving it alone."
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
# --- 1. install engine -------------------------------------------------------
|
|
||||||
if command -v docker >/dev/null 2>&1; then
|
|
||||||
say "docker: engine already installed ($(docker --version 2>/dev/null || echo 'version unknown'))."
|
|
||||||
else
|
|
||||||
case "$(detect_pkg_family)" in
|
|
||||||
rhel) install_rhel ;;
|
|
||||||
debian) install_debian ;;
|
|
||||||
*) say "docker: unsupported/undetected OS family; install Docker manually."; exit 1 ;;
|
|
||||||
esac
|
|
||||||
fi
|
|
||||||
|
|
||||||
# --- 2. enable + start the service (systemd only) ----------------------------
|
|
||||||
if [ -d /run/systemd/system ] && command -v systemctl >/dev/null 2>&1; then
|
|
||||||
sudo systemctl enable --now docker
|
|
||||||
else
|
|
||||||
say "docker: no systemd; not managing the service (start dockerd yourself)."
|
|
||||||
fi
|
|
||||||
|
|
||||||
# --- 3. group membership for the invoking user -------------------------------
|
|
||||||
user="$(id -un)"
|
|
||||||
if id -nG "$user" | tr ' ' '\n' | grep -qx docker; then
|
|
||||||
say "docker: $user already in the docker group."
|
|
||||||
else
|
|
||||||
sudo usermod -aG docker "$user"
|
|
||||||
say "docker: added $user to the docker group."
|
|
||||||
say " Log out and back in (or run 'newgrp docker') for it to take effect."
|
|
||||||
fi
|
|
||||||
+22
-1
@@ -30,4 +30,25 @@ say "fetch.prune = true (stale remote-tracking refs cleaned on every fetch)"
|
|||||||
git config --global credential.helper store
|
git config --global credential.helper store
|
||||||
say "credential.helper = store (~/.git-credentials, created on first auth)"
|
say "credential.helper = store (~/.git-credentials, created on first auth)"
|
||||||
|
|
||||||
say "Done. On the next HTTPS push/pull to a new host, git prompts once and caches the credential in ~/.git-credentials."
|
# Our own Gitea authenticates from $GITEA_KEY (exported by secrets/ from the
|
||||||
|
# vault), not from ~/.git-credentials.
|
||||||
|
#
|
||||||
|
# WHY: git never reads GITEA_KEY by itself. With only `store`, the token was
|
||||||
|
# whatever got typed at the first prompt, copied into ~/.git-credentials, and
|
||||||
|
# it silently went stale when the vault's key was rotated — or, on a box that
|
||||||
|
# never pushed, there was no credential at all and the push just failed. Now
|
||||||
|
# `opc GITEA_KEY` is the only step when the token changes.
|
||||||
|
#
|
||||||
|
# The empty helper first clears the inherited `store` for this host, so a
|
||||||
|
# rejected token is never cached and a stale cached one can never win. Gitea
|
||||||
|
# accepts the token as the password; the username is informational (GITEA_USER
|
||||||
|
# overrides it). With GITEA_KEY unset the helper prints nothing and git falls
|
||||||
|
# back to prompting, as before.
|
||||||
|
GITEA_URL="https://git.nic-oconnor.com"
|
||||||
|
git config --global --unset-all "credential.$GITEA_URL.helper" 2>/dev/null || true
|
||||||
|
git config --global --add "credential.$GITEA_URL.helper" ''
|
||||||
|
git config --global --add "credential.$GITEA_URL.helper" \
|
||||||
|
'!f() { test "$1" = get && test -n "${GITEA_KEY:-}" && printf "username=%s\npassword=%s\n" "${GITEA_USER:-$(id -un)}" "$GITEA_KEY"; }; f'
|
||||||
|
say "credential helper for $GITEA_URL = \$GITEA_KEY (from secrets.env)"
|
||||||
|
|
||||||
|
say "Done. $GITEA_URL uses \$GITEA_KEY; other HTTPS hosts prompt once and cache in ~/.git-credentials."
|
||||||
|
|||||||
+9
-1
@@ -26,6 +26,12 @@ REPO_ROOT="$(dirname -- "$SCRIPT_DIR")"
|
|||||||
|
|
||||||
say() { printf '%s\n' "$*" >&2; }
|
say() { printf '%s\n' "$*" >&2; }
|
||||||
|
|
||||||
|
# The installer links the CLI into ~/.local/bin, which is only on PATH in
|
||||||
|
# shells that sourced the managed ~/.zshrc. bootstrap run by a non-shell
|
||||||
|
# parent (the Coder agent's dotfiles script) has no such PATH, so the
|
||||||
|
# post-install `command -v hermes` failed on a successful install.
|
||||||
|
PATH="$HOME/.local/bin:$PATH"
|
||||||
|
|
||||||
# Hermes Agent version (synced with PyPI/npm).
|
# Hermes Agent version (synced with PyPI/npm).
|
||||||
HERMES_VERSION="v0.18.2"
|
HERMES_VERSION="v0.18.2"
|
||||||
|
|
||||||
@@ -74,7 +80,9 @@ if [ "$KEY_CHANGED" -eq 1 ] && [ -f "$HOME/.config/mcp-sync/key" ]; then
|
|||||||
fi
|
fi
|
||||||
say "hermes: Updated LITELLM_KEY in ~/.hermes/.env"
|
say "hermes: Updated LITELLM_KEY in ~/.hermes/.env"
|
||||||
else
|
else
|
||||||
# Create .env if it doesn't exist
|
# Create .env if it doesn't exist. On a fresh home this runs BEFORE the
|
||||||
|
# installer below has created ~/.hermes, so make the directory first.
|
||||||
|
mkdir -p "$HOME/.hermes"
|
||||||
printf 'LITELLM_KEY=%s\n' "$NEW_KEY" >"$HOME/.hermes/.env"
|
printf 'LITELLM_KEY=%s\n' "$NEW_KEY" >"$HOME/.hermes/.env"
|
||||||
say "hermes: Created ~/.hermes/.env with LITELLM_KEY"
|
say "hermes: Created ~/.hermes/.env with LITELLM_KEY"
|
||||||
fi
|
fi
|
||||||
|
|||||||
+30
-2
@@ -8,8 +8,8 @@
|
|||||||
# reset.
|
# reset.
|
||||||
#
|
#
|
||||||
# Source this (callers already run `set -eu`), then use fetch_arch /
|
# Source this (callers already run `set -eu`), then use fetch_arch /
|
||||||
# fetch_arch_deb, have_version + mark_version, install_tarball_bin, and
|
# fetch_arch_deb, have_version + mark_version, install_tarball_bin,
|
||||||
# install_raw_bin.
|
# install_tarball_tree, and install_raw_bin.
|
||||||
|
|
||||||
BIN_DIR="${BIN_DIR:-$HOME/.local/bin}"
|
BIN_DIR="${BIN_DIR:-$HOME/.local/bin}"
|
||||||
VERSION_DIR="$BIN_DIR/.versions"
|
VERSION_DIR="$BIN_DIR/.versions"
|
||||||
@@ -93,6 +93,34 @@ install_zip_bin() {
|
|||||||
fsay "installed $BIN_DIR/$3"
|
fsay "installed $BIN_DIR/$3"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# install_tarball_tree <url> <root-within-archive> <dest-dir> — download a
|
||||||
|
# .tar.gz that holds a whole application directory rather than one binary, and
|
||||||
|
# swap it into <dest-dir>. install_tarball_bin cannot be reused here: VS Code
|
||||||
|
# is an Electron bundle of thousands of files that only runs from an intact
|
||||||
|
# tree, not a single binary that can be picked out of the archive.
|
||||||
|
#
|
||||||
|
# The new tree is staged beside the target and moved into place only once the
|
||||||
|
# download and extraction have both succeeded, so an interrupted run leaves the
|
||||||
|
# previously installed version intact instead of a half-extracted directory.
|
||||||
|
install_tarball_tree() {
|
||||||
|
_dest="$3"
|
||||||
|
_tmp="$(mktemp -d)"
|
||||||
|
_fetch_dl "$1" "$_tmp/archive.tar.gz"
|
||||||
|
tar -xzf "$_tmp/archive.tar.gz" -C "$_tmp"
|
||||||
|
if [ ! -d "$_tmp/$2" ]; then
|
||||||
|
fsay "fetch: '$2' not found in archive"; rm -rf "$_tmp"; return 1
|
||||||
|
fi
|
||||||
|
mkdir -p "$(dirname "$_dest")"
|
||||||
|
rm -rf "$_dest.new" "$_dest.old"
|
||||||
|
mv "$_tmp/$2" "$_dest.new"
|
||||||
|
if [ -e "$_dest" ]; then
|
||||||
|
mv "$_dest" "$_dest.old"
|
||||||
|
fi
|
||||||
|
mv "$_dest.new" "$_dest"
|
||||||
|
rm -rf "$_dest.old" "$_tmp"
|
||||||
|
fsay "installed $_dest"
|
||||||
|
}
|
||||||
|
|
||||||
# install_raw_bin <url> <dest-name> — download a single raw binary.
|
# install_raw_bin <url> <dest-name> — download a single raw binary.
|
||||||
install_raw_bin() {
|
install_raw_bin() {
|
||||||
mkdir -p "$BIN_DIR"
|
mkdir -p "$BIN_DIR"
|
||||||
|
|||||||
+17
-5
@@ -120,11 +120,23 @@ say "Key stored at $KEYFILE (0600)."
|
|||||||
# bootstrap run — and then models and skills silently vanish, because sync.sh
|
# bootstrap run — and then models and skills silently vanish, because sync.sh
|
||||||
# never runs and OPENCODE_CONFIG is never exported. ~/.zshrc.local is the
|
# never runs and OPENCODE_CONFIG is never exported. ~/.zshrc.local is the
|
||||||
# unmanaged file the managed zshrc sources last, so the hook persists.
|
# unmanaged file the managed zshrc sources last, so the hook persists.
|
||||||
case "${SHELL:-}" in
|
#
|
||||||
*zsh) RC="$HOME/.zshrc.local" ;;
|
# Do NOT pick the file from $SHELL alone. On a fresh box bootstrap runs this
|
||||||
*bash) RC="$HOME/.bashrc" ;;
|
# from the user's ORIGINAL shell: zsh/ has just switched the login shell to zsh,
|
||||||
*) [ -f "$HOME/.zshrc" ] && RC="$HOME/.zshrc.local" || RC="$HOME/.bashrc" ;;
|
# but $SHELL in this process still says bash. The hook then went into ~/.bashrc,
|
||||||
esac
|
# below zsh/'s auto-exec hook, so every new terminal exec'd into zsh before
|
||||||
|
# reaching it, zsh never sourced sync.sh, and opencode came up unconfigured.
|
||||||
|
# The dotfiles-managed ~/.zshrc (or zsh as the account's login shell) wins.
|
||||||
|
_login_shell="$(getent passwd "$(id -un)" 2>/dev/null | cut -d: -f7)"
|
||||||
|
if grep -qs 'Managed by dotfiles/zsh/install.sh' "$HOME/.zshrc"; then
|
||||||
|
RC="$HOME/.zshrc.local"
|
||||||
|
else
|
||||||
|
case "${_login_shell:-${SHELL:-}}" in
|
||||||
|
*zsh) RC="$HOME/.zshrc.local" ;;
|
||||||
|
*bash) RC="$HOME/.bashrc" ;;
|
||||||
|
*) [ -f "$HOME/.zshrc" ] && RC="$HOME/.zshrc.local" || RC="$HOME/.bashrc" ;;
|
||||||
|
esac
|
||||||
|
fi
|
||||||
touch "$RC"
|
touch "$RC"
|
||||||
# It sits alongside the user's secrets; keep it non-readable by others.
|
# It sits alongside the user's secrets; keep it non-readable by others.
|
||||||
[ "$RC" = "$HOME/.zshrc.local" ] && chmod 0600 "$RC" 2>/dev/null || true
|
[ "$RC" = "$HOME/.zshrc.local" ] && chmod 0600 "$RC" 2>/dev/null || true
|
||||||
|
|||||||
+89
-20
@@ -6,19 +6,19 @@
|
|||||||
# field on EVERY new shell. That made each terminal wait on 1Password, broke
|
# field on EVERY new shell. That made each terminal wait on 1Password, broke
|
||||||
# every shell when the network or the service account was down, and meant each
|
# every shell when the network or the service account was down, and meant each
|
||||||
# box carried its own hand-written copy of the loop. Now the vault is read once
|
# box carried its own hand-written copy of the loop. Now the vault is read once
|
||||||
# here (at bootstrap, or on demand with `envs`), and shells read a local file.
|
# here (at bootstrap, or on demand with `ops`), and shells read a local file.
|
||||||
#
|
#
|
||||||
# Layout: one vault per user (default: the login name), holding one item
|
# Layout: one vault per user (default: the login name), holding one item
|
||||||
# (default: also the login name) whose field LABELS are the variable names —
|
# (default: also the login name) whose field LABELS are the variable names —
|
||||||
# GITEA_KEY, LITELLM_KEY, PROXMOX_TOKEN_SECRET, ... Every such field is
|
# GITEA_KEY, LITELLM_KEY, PROXMOX_TOKEN_SECRET, ... Every such field is
|
||||||
# exported. Add a secret by adding a field (`envc NAME`), not by editing
|
# exported. Add a secret by adding a field (`opc NAME`), not by editing
|
||||||
# this repo.
|
# this repo.
|
||||||
#
|
#
|
||||||
# Files, all under ~/.config/dotfiles (0700), none in the repo:
|
# Files, all under ~/.config/dotfiles (0700), none in the repo:
|
||||||
# op.env OP_SERVICE_ACCOUNT_TOKEN, OP_VAULT, OP_ITEM (0600). Kept so a
|
# op.env OP_SERVICE_ACCOUNT_TOKEN, OP_VAULT, OP_ITEM (0600). Kept so a
|
||||||
# refresh needs no prompt. Deliberately NOT sourced by shells:
|
# refresh needs no prompt. Deliberately NOT sourced by shells:
|
||||||
# the token stays out of every process's environment, and only
|
# the token stays out of every process's environment, and only
|
||||||
# the env* helpers in zsh/zshrc load it, in a subshell.
|
# the op* helpers in zsh/zshrc load it, in a subshell.
|
||||||
# secrets.env the exported fields (0600), replaced atomically, so a failed
|
# secrets.env the exported fields (0600), replaced atomically, so a failed
|
||||||
# fetch leaves the previous copy in place.
|
# fetch leaves the previous copy in place.
|
||||||
#
|
#
|
||||||
@@ -48,8 +48,17 @@ for _tool in op jq; do
|
|||||||
done
|
done
|
||||||
|
|
||||||
# --- 1. resolve the service account, vault and item -------------------------
|
# --- 1. resolve the service account, vault and item -------------------------
|
||||||
# Precedence: the environment, then op.env, then a prompt (token) / defaults.
|
# Precedence: the environment, then op.env, then a prompt / auto-detection.
|
||||||
|
#
|
||||||
|
# WHY each piece is checked on its own: the vault holds every key later
|
||||||
|
# modules need (LITELLM_KEY, GITEA_KEY, ...). A bad token paste used to fall
|
||||||
|
# through as a generic "could not read item", bootstrap carried on, and the
|
||||||
|
# first visible symptom was mcp-config/ asking for a LiteLLM key by hand —
|
||||||
|
# which hid the real problem. Now a rejected token, a wrong vault and a wrong
|
||||||
|
# item each get their own message, and on a TTY a re-prompt.
|
||||||
_env_token="${OP_SERVICE_ACCOUNT_TOKEN:-}"
|
_env_token="${OP_SERVICE_ACCOUNT_TOKEN:-}"
|
||||||
|
_token_src="from OP_SERVICE_ACCOUNT_TOKEN"
|
||||||
|
[ -n "$_env_token" ] || _token_src="stored in $OP_ENV"
|
||||||
_env_vault="${OP_VAULT:-}"
|
_env_vault="${OP_VAULT:-}"
|
||||||
_env_item="${OP_ITEM:-}"
|
_env_item="${OP_ITEM:-}"
|
||||||
if [ -f "$OP_ENV" ]; then
|
if [ -f "$OP_ENV" ]; then
|
||||||
@@ -61,6 +70,10 @@ OP_VAULT="${_env_vault:-${OP_VAULT:-}}"
|
|||||||
OP_ITEM="${_env_item:-${OP_ITEM:-}}"
|
OP_ITEM="${_env_item:-${OP_ITEM:-}}"
|
||||||
_user="$(id -un)"
|
_user="$(id -un)"
|
||||||
|
|
||||||
|
# Probe in a subshell: in POSIX sh a failed redirect on a special builtin
|
||||||
|
# like `:` exits the whole script.
|
||||||
|
have_tty() { ( : < /dev/tty ) 2>/dev/null; }
|
||||||
|
|
||||||
# ask <prompt> <default> — read one line from the TTY; echoes the default on empty.
|
# ask <prompt> <default> — read one line from the TTY; echoes the default on empty.
|
||||||
ask() {
|
ask() {
|
||||||
printf '%s [%s]: ' "$1" "$2" >&2
|
printf '%s [%s]: ' "$1" "$2" >&2
|
||||||
@@ -68,31 +81,85 @@ ask() {
|
|||||||
printf '%s' "${_ans:-$2}"
|
printf '%s' "${_ans:-$2}"
|
||||||
}
|
}
|
||||||
|
|
||||||
if [ -z "$OP_SERVICE_ACCOUNT_TOKEN" ]; then
|
# mask <secret> — first 10 and last 10 characters plus the length, so a
|
||||||
if [ -f "$SECRETS_ENV" ]; then
|
# truncated or doubled paste is visible without printing the whole token.
|
||||||
say "secrets: no service-account token; keeping existing $SECRETS_ENV."
|
mask() {
|
||||||
exit 0
|
printf '%s' "$1" | awk '{ n = length($0)
|
||||||
|
if (n <= 20) printf "%s (%d chars)", "(too short)", n
|
||||||
|
else printf "%s**********%s (first 10 + last 10 of %d chars)", substr($0, 1, 10), substr($0, n - 9), n }'
|
||||||
|
}
|
||||||
|
|
||||||
|
if [ -z "$OP_SERVICE_ACCOUNT_TOKEN" ] && [ -f "$SECRETS_ENV" ]; then
|
||||||
|
say "secrets: no service-account token; keeping existing $SECRETS_ENV."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# The token: `op whoami` is the cheapest call that fails on a bad token and
|
||||||
|
# nothing else. Up to three pastes before giving up.
|
||||||
|
_tries=0
|
||||||
|
while :; do
|
||||||
|
if [ -n "$OP_SERVICE_ACCOUNT_TOKEN" ]; then
|
||||||
|
export OP_SERVICE_ACCOUNT_TOKEN
|
||||||
|
op whoami >/dev/null 2>&1 && break
|
||||||
|
if [ "$_tries" -eq 0 ]; then
|
||||||
|
say "secrets: 1Password REJECTED the service-account token $_token_src (bad paste, revoked or expired?)."
|
||||||
|
say " token: $(mask "$OP_SERVICE_ACCOUNT_TOKEN")"
|
||||||
|
else
|
||||||
|
say "secrets: 1Password REJECTED that token — most likely a truncated or bad paste."
|
||||||
|
say " token: $(mask "$OP_SERVICE_ACCOUNT_TOKEN")"
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
# Probe in a subshell: in POSIX sh a failed redirect on a special builtin
|
if ! have_tty; then
|
||||||
# like `:` exits the whole script, which would skip the message below.
|
say "secrets: no valid OP_SERVICE_ACCOUNT_TOKEN and no TTY to prompt."
|
||||||
if ! ( : < /dev/tty ) 2>/dev/null; then
|
|
||||||
say "secrets: no OP_SERVICE_ACCOUNT_TOKEN and no TTY to prompt."
|
|
||||||
say " re-run as: OP_SERVICE_ACCOUNT_TOKEN=ops_... $0"
|
say " re-run as: OP_SERVICE_ACCOUNT_TOKEN=ops_... $0"
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
if [ "$_tries" -ge 3 ]; then
|
||||||
|
say "secrets: giving up after $_tries attempts. Every key in the vault (LITELLM_KEY,"
|
||||||
|
say " GITEA_KEY, ...) is missing until this works: re-run $0"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
_tries=$((_tries + 1))
|
||||||
printf '1Password service-account token (ops_..., input hidden): ' >&2
|
printf '1Password service-account token (ops_..., input hidden): ' >&2
|
||||||
stty -echo < /dev/tty 2>/dev/null || true
|
stty -echo < /dev/tty 2>/dev/null || true
|
||||||
IFS= read -r OP_SERVICE_ACCOUNT_TOKEN < /dev/tty || true
|
IFS= read -r OP_SERVICE_ACCOUNT_TOKEN < /dev/tty || true
|
||||||
stty echo < /dev/tty 2>/dev/null || true
|
stty echo < /dev/tty 2>/dev/null || true
|
||||||
printf '\n' >&2
|
printf '\n' >&2
|
||||||
OP_SERVICE_ACCOUNT_TOKEN="$(printf '%s' "$OP_SERVICE_ACCOUNT_TOKEN" | tr -d ' \t\r\n')"
|
OP_SERVICE_ACCOUNT_TOKEN="$(printf '%s' "$OP_SERVICE_ACCOUNT_TOKEN" | tr -d ' \t\r\n')"
|
||||||
[ -n "$OP_SERVICE_ACCOUNT_TOKEN" ] || { say "secrets: no token entered."; exit 1; }
|
[ -n "$OP_SERVICE_ACCOUNT_TOKEN" ] || say "secrets: no token entered."
|
||||||
[ -n "$OP_VAULT" ] || OP_VAULT="$(ask 'Vault' "$_user")"
|
done
|
||||||
[ -n "$OP_ITEM" ] || OP_ITEM="$(ask 'Item' "$_user")"
|
say "secrets: service-account token OK: $(mask "$OP_SERVICE_ACCOUNT_TOKEN")"
|
||||||
fi
|
|
||||||
OP_VAULT="${OP_VAULT:-$_user}"
|
# pick <kind> <current> <default> <check-fn> <list-cmd...>
|
||||||
OP_ITEM="${OP_ITEM:-$_user}"
|
# Resolve a vault or item name: keep <current> if it checks out; otherwise use
|
||||||
export OP_SERVICE_ACCOUNT_TOKEN
|
# the only one the token can see, or ask (TTY) showing what it can see.
|
||||||
|
pick() {
|
||||||
|
_kind="$1" _cur="$2" _def="$3"; shift 3
|
||||||
|
_check="$1"; shift
|
||||||
|
_names="$("$@" --format json 2>/dev/null | jq -r '.[] | .name // .title')"
|
||||||
|
if [ -n "$_cur" ]; then
|
||||||
|
$_check "$_cur" && { printf '%s' "$_cur"; return 0; }
|
||||||
|
say "secrets: $_kind '$_cur' not found. This token can see:"
|
||||||
|
printf '%s\n' "$_names" | sed 's/^/ - /' >&2
|
||||||
|
fi
|
||||||
|
if [ "$(printf '%s\n' "$_names" | grep -c .)" -eq 1 ]; then
|
||||||
|
say "secrets: using the only $_kind this token can see: $_names"
|
||||||
|
printf '%s' "$_names"; return 0
|
||||||
|
fi
|
||||||
|
have_tty || { say "secrets: set OP_$(printf '%s' "$_kind" | tr a-z A-Z) and re-run."; return 1; }
|
||||||
|
printf '%s\n' "$_names" | grep -Fxq -- "$_def" || _def="$(printf '%s\n' "$_names" | head -n 1)"
|
||||||
|
while :; do
|
||||||
|
_cur="$(ask "$_kind" "$_def")"
|
||||||
|
$_check "$_cur" && { printf '%s' "$_cur"; return 0; }
|
||||||
|
say "secrets: $_kind '$_cur' not found. This token can see:"
|
||||||
|
printf '%s\n' "$_names" | sed 's/^/ - /' >&2
|
||||||
|
done
|
||||||
|
}
|
||||||
|
vault_ok() { op vault get "$1" >/dev/null 2>&1; }
|
||||||
|
item_ok() { op item get "$1" --vault "$OP_VAULT" >/dev/null 2>&1; }
|
||||||
|
|
||||||
|
OP_VAULT="$(pick vault "$OP_VAULT" "$_user" vault_ok op vault list)" || exit 1
|
||||||
|
OP_ITEM="$(pick item "$OP_ITEM" "$_user" item_ok op item list --vault "$OP_VAULT")" || exit 1
|
||||||
|
|
||||||
# --- 2. fetch the item once and render it -----------------------------------
|
# --- 2. fetch the item once and render it -----------------------------------
|
||||||
# One `op item get` instead of one `op read` per field. Labels that are not
|
# One `op item get` instead of one `op read` per field. Labels that are not
|
||||||
@@ -115,7 +182,7 @@ _tmp="$(mktemp "$CONF_DIR/secrets.env.XXXXXX")"
|
|||||||
trap 'rm -f "$_tmp"' EXIT
|
trap 'rm -f "$_tmp"' EXIT
|
||||||
{
|
{
|
||||||
printf '# Generated by dotfiles secrets/install.sh from op://%s/%s — do not edit.\n' "$OP_VAULT" "$OP_ITEM"
|
printf '# Generated by dotfiles secrets/install.sh from op://%s/%s — do not edit.\n' "$OP_VAULT" "$OP_ITEM"
|
||||||
printf '# Refresh with: envs (change the vault with: envc NAME / envd NAME)\n'
|
printf '# Refresh with: ops (change the vault with: opc NAME / opd NAME)\n'
|
||||||
printf '%s' "$_json" | jq -r '
|
printf '%s' "$_json" | jq -r '
|
||||||
.fields[]
|
.fields[]
|
||||||
| select(.purpose != "NOTES")
|
| select(.purpose != "NOTES")
|
||||||
@@ -132,6 +199,8 @@ fi
|
|||||||
mv -f "$_tmp" "$SECRETS_ENV"
|
mv -f "$_tmp" "$SECRETS_ENV"
|
||||||
trap - EXIT
|
trap - EXIT
|
||||||
say "secrets: wrote $_count variables to $SECRETS_ENV (0600)."
|
say "secrets: wrote $_count variables to $SECRETS_ENV (0600)."
|
||||||
|
grep -q '^export LITELLM_KEY=' "$SECRETS_ENV" \
|
||||||
|
|| say "secrets: WARNING: op://$OP_VAULT/$OP_ITEM has no LITELLM_KEY field; mcp-config/ and hermes/ need it."
|
||||||
|
|
||||||
# Remember the account only after it has actually worked.
|
# Remember the account only after it has actually worked.
|
||||||
{
|
{
|
||||||
|
|||||||
Executable
+216
@@ -0,0 +1,216 @@
|
|||||||
|
#!/usr/bin/env sh
|
||||||
|
# vscode/install.sh — install Visual Studio Code into ~/.local/opt/vscode.
|
||||||
|
#
|
||||||
|
# WHY: VS Code is the one GUI app this repo installs, and the obvious way to get
|
||||||
|
# it — Microsoft's rpm/deb repo — needs root, writes outside $HOME, and is wiped
|
||||||
|
# by a Kasm redeploy, so the editor quietly disappears on every rebuild. The box
|
||||||
|
# this was written for had exactly that half-state: /etc/yum.repos.d/vscode.repo
|
||||||
|
# present, `code` never installed, and nothing in this repo recording that VS
|
||||||
|
# Code was meant to be there at all. Upstream's portable tarball is the same
|
||||||
|
# build with none of that: everything lands under $HOME, no sudo, and it
|
||||||
|
# survives a redeploy. The only thing given up is dnf/apt-driven updates, which
|
||||||
|
# is what the pinned VSCODE_VERSION below replaces.
|
||||||
|
#
|
||||||
|
# It is not in bin/install.sh because that script installs single self-contained
|
||||||
|
# binaries; VS Code is an Electron tree of thousands of files plus a menu entry
|
||||||
|
# and an icon. It follows node/ and ansible/ instead: unpack under ~/.local/opt
|
||||||
|
# and symlink the CLI into ~/.local/bin.
|
||||||
|
#
|
||||||
|
# The Chromium sandbox still works from a userland install: the bundled
|
||||||
|
# chrome-sandbox helper is not setuid here, so Electron falls back to the
|
||||||
|
# unprivileged user-namespace sandbox. That needs user namespaces enabled
|
||||||
|
# (user.max_user_namespaces > 0, the default on every distro this repo targets).
|
||||||
|
# Verified by launching without --no-sandbox. Nothing is given up for dropping
|
||||||
|
# root; if a host ever has userns disabled, VS Code says so on startup.
|
||||||
|
#
|
||||||
|
# Idempotent and safe to re-run. To upgrade, bump VSCODE_VERSION.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
|
||||||
|
REPO_ROOT="$(dirname -- "$SCRIPT_DIR")"
|
||||||
|
|
||||||
|
# shellcheck source=../lib/fetch.sh
|
||||||
|
. "$REPO_ROOT/lib/fetch.sh"
|
||||||
|
|
||||||
|
say() { printf '%s\n' "$*" >&2; }
|
||||||
|
|
||||||
|
# Pinned deliberately. VS Code auto-updates itself on other platforms; a tarball
|
||||||
|
# install does not, which is the point — the editor runs your code and holds
|
||||||
|
# your Git and extension credentials, so the version is a supply-chain decision
|
||||||
|
# rather than whatever upstream served today. `code --version` prints this
|
||||||
|
# string verbatim, so it must match with no leading "v".
|
||||||
|
VSCODE_VERSION="1.139.0"
|
||||||
|
|
||||||
|
PREFIX="$HOME/.local/opt/vscode"
|
||||||
|
DESKTOP_DIR="$HOME/.local/share/applications"
|
||||||
|
ARGV_JSON="$HOME/.vscode/argv.json"
|
||||||
|
|
||||||
|
# has_desktop — true when this host has a graphical session to run a GUI editor
|
||||||
|
# in. VS Code is ~400MB unpacked and $HOME is the thing that persists across
|
||||||
|
# redeploys, so it is not worth carrying on a box that will never display it.
|
||||||
|
# Desktop images ship session files under /usr/share/{x,wayland-}sessions.
|
||||||
|
has_desktop() {
|
||||||
|
[ -n "${XDG_CURRENT_DESKTOP:-}" ] && return 0
|
||||||
|
for d in /usr/share/xsessions /usr/share/wayland-sessions; do
|
||||||
|
[ -d "$d" ] && [ -n "$(ls -A "$d" 2>/dev/null)" ] && return 0
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- skip on headless hosts --------------------------------------------------
|
||||||
|
# DOTFILES_VSCODE=1 forces the install anyway (e.g. for `code tunnel` on a
|
||||||
|
# server); DOTFILES_VSCODE=0 skips it even on a desktop.
|
||||||
|
case "${DOTFILES_VSCODE:-auto}" in
|
||||||
|
0) say "vscode: DOTFILES_VSCODE=0; skipping."; exit 0 ;;
|
||||||
|
1) ;;
|
||||||
|
*) if ! has_desktop; then
|
||||||
|
say "vscode: no graphical session detected; skipping (DOTFILES_VSCODE=1 to force)."
|
||||||
|
exit 0
|
||||||
|
fi ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# Upstream keys its download URLs on x64/arm64 rather than amd64/arm64.
|
||||||
|
case "$(fetch_arch)" in
|
||||||
|
x86_64) VSCODE_ARCH=x64 ;;
|
||||||
|
aarch64) VSCODE_ARCH=arm64 ;;
|
||||||
|
*) say "vscode: unsupported arch $(fetch_arch)"; exit 1 ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# --- install the app tree ----------------------------------------------------
|
||||||
|
# Verify it RUNS, not just that the tree exists: a Kasm redeploy can leave a
|
||||||
|
# present-but-broken directory behind (the same failure node/ and ansible/
|
||||||
|
# guard against).
|
||||||
|
#
|
||||||
|
# The root fallback is not optional. Run as root, `code --version` refuses --
|
||||||
|
# it prints "You are trying to start Visual Studio Code as a super user"
|
||||||
|
# instead of the version, and still exits 0 -- so a bare run-check never
|
||||||
|
# matches and would re-download 325MB on every single bootstrap, silently.
|
||||||
|
# That is the forced-install path (DOTFILES_VSCODE=1 in a container or on a
|
||||||
|
# server), so for root only, fall back to the recorded version stamp.
|
||||||
|
installed_version="$("$PREFIX/bin/code" --version 2>/dev/null | head -1 || true)"
|
||||||
|
if [ "$installed_version" = "$VSCODE_VERSION" ]; then
|
||||||
|
say "vscode: $VSCODE_VERSION already present at $PREFIX."
|
||||||
|
elif [ "$(id -u)" = "0" ] && have_version code "$VSCODE_VERSION"; then
|
||||||
|
say "vscode: $VSCODE_VERSION already present at $PREFIX (stamped; root cannot query --version)."
|
||||||
|
else
|
||||||
|
say "vscode: installing $VSCODE_VERSION into $PREFIX (this is a ~325MB download)..."
|
||||||
|
install_tarball_tree \
|
||||||
|
"https://update.code.visualstudio.com/$VSCODE_VERSION/linux-$VSCODE_ARCH/stable" \
|
||||||
|
"VSCode-linux-$VSCODE_ARCH" "$PREFIX"
|
||||||
|
|
||||||
|
# bin/code is upstream's CLI wrapper (`code .`, `code --install-extension`);
|
||||||
|
# the Electron binary at the tree root is what the menu entry launches.
|
||||||
|
mkdir -p "$BIN_DIR"
|
||||||
|
ln -sf "$PREFIX/bin/code" "$BIN_DIR/code"
|
||||||
|
mark_version code "$VSCODE_VERSION"
|
||||||
|
say "vscode: installed $VSCODE_VERSION and linked $BIN_DIR/code."
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- menu entry --------------------------------------------------------------
|
||||||
|
# Icon= is an absolute path rather than a theme name: the bundled icon is a
|
||||||
|
# single 1024x1024 PNG and hicolor tops out at 512x512, so there is no correct
|
||||||
|
# themed directory to drop it in. The spec allows an absolute path, and it needs
|
||||||
|
# no icon-cache rebuild to take effect.
|
||||||
|
mkdir -p "$DESKTOP_DIR"
|
||||||
|
cat >"$DESKTOP_DIR/code.desktop" <<DESKTOP
|
||||||
|
[Desktop Entry]
|
||||||
|
Type=Application
|
||||||
|
Name=Visual Studio Code
|
||||||
|
GenericName=Text Editor
|
||||||
|
Comment=Code Editing. Redefined.
|
||||||
|
Exec=$PREFIX/code %F
|
||||||
|
Icon=$PREFIX/resources/app/resources/linux/code.png
|
||||||
|
Terminal=false
|
||||||
|
StartupNotify=true
|
||||||
|
StartupWMClass=Code
|
||||||
|
Categories=TextEditor;Development;IDE;
|
||||||
|
MimeType=text/plain;inode/directory;
|
||||||
|
Keywords=vscode;editor;
|
||||||
|
Actions=new-window;
|
||||||
|
|
||||||
|
[Desktop Action new-window]
|
||||||
|
Name=New Empty Window
|
||||||
|
Exec=$PREFIX/code --new-window %F
|
||||||
|
Icon=$PREFIX/resources/app/resources/linux/code.png
|
||||||
|
DESKTOP
|
||||||
|
say "vscode: wrote $DESKTOP_DIR/code.desktop"
|
||||||
|
|
||||||
|
# Refresh the per-user menu caches so the entry appears without a re-login.
|
||||||
|
if command -v update-desktop-database >/dev/null 2>&1; then
|
||||||
|
update-desktop-database "$DESKTOP_DIR" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
if command -v kbuildsycoca5 >/dev/null 2>&1; then
|
||||||
|
kbuildsycoca5 --noincremental >/dev/null 2>&1 || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- keyring prompt suppression ----------------------------------------------
|
||||||
|
# WHY: on first launch Electron's safeStorage reaches for the OS keyring through
|
||||||
|
# libsecret. On a KDE session that wakes kwalletd, which pops an unlock/create
|
||||||
|
# dialog before the editor is usable -- and since kwallet-pam is not installed
|
||||||
|
# on these hosts, nothing unlocks the wallet at login, so it prompts every
|
||||||
|
# session. "basic" keeps VS Code's secrets in its own store and never reaches
|
||||||
|
# for the wallet at all. Verified with dbus-monitor: no org.kde.KWallet traffic
|
||||||
|
# on launch.
|
||||||
|
#
|
||||||
|
# TRADE-OFF: basic is weaker than a real keyring. Secrets VS Code stores (Git
|
||||||
|
# and GitHub auth tokens, extension credentials) end up obfuscated on disk
|
||||||
|
# rather than sealed behind a wallet password. To move to the wallet later,
|
||||||
|
# install kwallet-pam so it unlocks at login, then change the value to
|
||||||
|
# "kwallet5" and restart VS Code.
|
||||||
|
#
|
||||||
|
# The path comes from product.json's dataFolderName (".vscode"), NOT the
|
||||||
|
# user-data-dir: argv.json lives at ~/.vscode/argv.json and a copy written
|
||||||
|
# anywhere else is read by nothing and silently ignored.
|
||||||
|
if [ ! -e "$ARGV_JSON" ]; then
|
||||||
|
mkdir -p "$(dirname "$ARGV_JSON")"
|
||||||
|
cat >"$ARGV_JSON" <<'ARGV'
|
||||||
|
// Runtime arguments for VS Code. See dotfiles vscode/install.sh.
|
||||||
|
{
|
||||||
|
// Keep secrets in VS Code's own store rather than the OS keyring, so KDE's
|
||||||
|
// kwallet does not prompt for an unlock on every launch.
|
||||||
|
"password-store": "basic"
|
||||||
|
}
|
||||||
|
ARGV
|
||||||
|
say "vscode: wrote $ARGV_JSON (password-store=basic)"
|
||||||
|
elif grep -q '"password-store"' "$ARGV_JSON"; then
|
||||||
|
say "vscode: $ARGV_JSON already sets password-store; leaving it alone."
|
||||||
|
else
|
||||||
|
# Merge rather than overwrite: this file is JSON-with-comments that VS Code
|
||||||
|
# and the user both own, and it carries a crash-reporter-id that should
|
||||||
|
# survive. Insert the key after the opening brace and keep the rest byte for
|
||||||
|
# byte.
|
||||||
|
cp -p "$ARGV_JSON" "$ARGV_JSON.bak"
|
||||||
|
if awk '
|
||||||
|
BEGIN { done = 0 }
|
||||||
|
{
|
||||||
|
print
|
||||||
|
if (!done && $0 ~ /^[[:space:]]*\{[[:space:]]*$/) {
|
||||||
|
print "\t// Added by dotfiles vscode/install.sh: keep secrets in VS Code'"'"'s"
|
||||||
|
print "\t// own store so kwallet does not prompt for an unlock on every launch."
|
||||||
|
print "\t\"password-store\": \"basic\","
|
||||||
|
done = 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
END { exit(done ? 0 : 1) }
|
||||||
|
' "$ARGV_JSON.bak" >"$ARGV_JSON.new"; then
|
||||||
|
mv "$ARGV_JSON.new" "$ARGV_JSON"
|
||||||
|
say "vscode: set password-store=basic in $ARGV_JSON (backup: $ARGV_JSON.bak)"
|
||||||
|
else
|
||||||
|
rm -f "$ARGV_JSON.new"
|
||||||
|
say "vscode: WARNING: could not find the object opening in $ARGV_JSON."
|
||||||
|
say " Add '\"password-store\": \"basic\"' by hand to stop the keyring prompt."
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# A system-wide VS Code (from Microsoft's rpm/deb repo) shadows this one in the
|
||||||
|
# menu and shows a duplicate entry. Flag it rather than touching it -- removing
|
||||||
|
# it needs the root this module deliberately avoids.
|
||||||
|
if [ -e /usr/share/applications/com.microsoft.VSCode.desktop ]; then
|
||||||
|
say ""
|
||||||
|
say "vscode: NOTE: a system-wide VS Code package is also installed."
|
||||||
|
say " You will see two menu entries until it is removed:"
|
||||||
|
say " sudo dnf remove code # or: sudo apt remove code"
|
||||||
|
say ""
|
||||||
|
fi
|
||||||
|
|
||||||
|
say "vscode: ready."
|
||||||
@@ -2,19 +2,6 @@
|
|||||||
# overwritten on the next bootstrap run. Edit zsh/zshrc in the dotfiles repo
|
# overwritten on the next bootstrap run. Edit zsh/zshrc in the dotfiles repo
|
||||||
# instead.
|
# instead.
|
||||||
|
|
||||||
# If this account was added to the docker group (dotfiles/docker/install.sh)
|
|
||||||
# but this shell's session predates that, re-exec via newgrp so docker works
|
|
||||||
# without a full logout. Guarded: fires only when the group is on the account
|
|
||||||
# yet missing from the running shell's groups, so the newgrp'd shell (which
|
|
||||||
# has the group) cannot loop.
|
|
||||||
# Only for real terminals ([ -t 0 ]): newgrp replaces the shell and would
|
|
||||||
# swallow the command in scripted `zsh -i -c ...` invocations.
|
|
||||||
if [ -t 0 ] && [ -S /var/run/docker.sock ] \
|
|
||||||
&& id -nG "${USER:-$(id -un)}" 2>/dev/null | grep -qw docker \
|
|
||||||
&& ! id -Gn 2>/dev/null | grep -qw docker; then
|
|
||||||
exec newgrp docker
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Userland tools installed by dotfiles/bin/install.sh live here.
|
# Userland tools installed by dotfiles/bin/install.sh live here.
|
||||||
case ":$PATH:" in
|
case ":$PATH:" in
|
||||||
*":$HOME/.local/bin:"*) ;;
|
*":$HOME/.local/bin:"*) ;;
|
||||||
@@ -76,57 +63,86 @@ grmb() {
|
|||||||
|
|
||||||
# Secrets from this user's 1Password vault, written by dotfiles/secrets/install.sh.
|
# Secrets from this user's 1Password vault, written by dotfiles/secrets/install.sh.
|
||||||
# Reading a local 0600 file keeps shell startup off the network: nothing here
|
# Reading a local 0600 file keeps shell startup off the network: nothing here
|
||||||
# calls op. envc/envd refresh the file themselves; after editing the vault
|
# calls op. opc/opd refresh the file themselves; after editing the vault
|
||||||
# anywhere else (the 1Password app, another box), run `envs`.
|
# anywhere else (the 1Password app, another box), run `ops`.
|
||||||
_DOTFILES_CONF="$HOME/.config/dotfiles"
|
_DOTFILES_CONF="$HOME/.config/dotfiles"
|
||||||
[ -f "$_DOTFILES_CONF/secrets.env" ] && source "$_DOTFILES_CONF/secrets.env"
|
[ -f "$_DOTFILES_CONF/secrets.env" ] && source "$_DOTFILES_CONF/secrets.env"
|
||||||
|
|
||||||
# envr NAME read one field straight from the vault
|
# opr NAME read one field straight from the vault
|
||||||
# envc NAME [VALUE] create or update a field (prompts, hidden, when VALUE is
|
# opc NAME=VALUE create or update a field, then refresh and reload this
|
||||||
# omitted, which keeps the secret out of shell history),
|
# opc NAME shell; with no VALUE (NAME or NAME=) it prompts, hidden,
|
||||||
# then refresh and reload this shell
|
# which keeps the secret out of shell history.
|
||||||
# envd NAME delete a field, refresh, and unset it in this shell
|
# `opc NAME VALUE` also still works.
|
||||||
# envl [VAULT [ITEM]] show every field and its current value (default: your
|
# opd NAME delete a field, refresh, and unset it in this shell
|
||||||
# vault and item from op.env)
|
# opl [VAULT [ITEM]] show every field and its current value (default: your
|
||||||
# envs re-fetch the vault into secrets.env and reload this shell
|
# vault and item from op.env)
|
||||||
|
# ops re-fetch the vault into secrets.env and reload this shell
|
||||||
#
|
#
|
||||||
# The service-account token lives in op.env and is loaded only inside these
|
# The service-account token lives in op.env and is loaded only inside these
|
||||||
# subshells, so it never lands in the environment of everything you run.
|
# subshells, so it never lands in the environment of everything you run.
|
||||||
_dotfiles_op() {
|
_dotfiles_op() {
|
||||||
[ -f "$_DOTFILES_CONF/op.env" ] || { print -u2 "no $_DOTFILES_CONF/op.env; run: envs"; return 1; }
|
[ -f "$_DOTFILES_CONF/op.env" ] || { print -u2 "no $_DOTFILES_CONF/op.env; run: ops"; return 1; }
|
||||||
( set -a; source "$_DOTFILES_CONF/op.env"; set +a; "$@" )
|
( set -a; source "$_DOTFILES_CONF/op.env"; set +a; "$@" )
|
||||||
}
|
}
|
||||||
_dotfiles_env_name() {
|
_dotfiles_env_name() {
|
||||||
[[ "$1" =~ '^[A-Za-z_][A-Za-z0-9_]*$' ]] && return 0
|
[[ "$1" =~ '^[A-Za-z_][A-Za-z0-9_]*$' ]] && return 0
|
||||||
print -u2 "$2: '$1' is not a valid variable name"; return 2
|
print -u2 "$2: '$1' is not a valid variable name"; return 2
|
||||||
}
|
}
|
||||||
envr() {
|
opr() {
|
||||||
_dotfiles_env_name "$1" envr || return
|
_dotfiles_env_name "$1" opr || return
|
||||||
_dotfiles_op sh -c 'op read "op://$OP_VAULT/$OP_ITEM/$1"' envr "$1"
|
_dotfiles_op sh -c 'op read "op://$OP_VAULT/$OP_ITEM/$1"' opr "$1"
|
||||||
}
|
}
|
||||||
envc() {
|
opc() {
|
||||||
emulate -L zsh
|
emulate -L zsh
|
||||||
local name=$1 value
|
local name=$1 value
|
||||||
_dotfiles_env_name "$name" envc || return
|
if [[ $1 == *=* ]]; then
|
||||||
if (( $# >= 2 )); then
|
# NAME=VALUE: split on the first '=' so values may contain '='
|
||||||
|
name=${1%%=*} value=${1#*=}
|
||||||
|
elif (( $# >= 2 )); then
|
||||||
value=$2
|
value=$2
|
||||||
else
|
fi
|
||||||
|
_dotfiles_env_name "$name" opc || return
|
||||||
|
if [[ -z $value ]]; then
|
||||||
read -rs "value?$name: " && print
|
read -rs "value?$name: " && print
|
||||||
|
[[ -n $value ]] || { print -u2 "opc: empty value for $name; nothing changed"; return 1; }
|
||||||
fi
|
fi
|
||||||
# [concealed] makes it a password field; an existing field of that name is updated.
|
# [concealed] makes it a password field; an existing field of that name is updated.
|
||||||
_dotfiles_op sh -c 'op item edit "$OP_ITEM" --vault "$OP_VAULT" "$1[concealed]=$2" >/dev/null' \
|
# op can exit non-zero with "(404) Not Found" after an edit it did save (seen with
|
||||||
envc "$name" "$value" && envs
|
# op 2.34.1 and 2.39.0), so its exit code is ignored: the field is read back instead.
|
||||||
|
_dotfiles_op sh -c '
|
||||||
|
clean() { printf "%s" "${1:-(no output)}" | sed "s/^\[ERROR\] [0-9/]* [0-9:]* //"; }
|
||||||
|
ref="op://$OP_VAULT/$OP_ITEM/$1"
|
||||||
|
err=$(op item edit "$OP_ITEM" --vault "$OP_VAULT" "$1[concealed]=$2" 2>&1 >/dev/null)
|
||||||
|
if [ "$(op read "$ref" 2>/dev/null)" = "$2" ]; then
|
||||||
|
printf "opc: saved %s\n" "$ref" >&2
|
||||||
|
else
|
||||||
|
printf "opc: could not save %s\n op said: %s\n check vault, item and service-account write access with: opl\n" \
|
||||||
|
"$ref" "$(clean "$err")" >&2
|
||||||
|
exit 1
|
||||||
|
fi' opc "$name" "$value" && ops
|
||||||
}
|
}
|
||||||
envd() {
|
opd() {
|
||||||
_dotfiles_env_name "$1" envd || return
|
_dotfiles_env_name "$1" opd || return
|
||||||
_dotfiles_op sh -c 'op item edit "$OP_ITEM" --vault "$OP_VAULT" "$1[delete]" >/dev/null' envd "$1" \
|
# As in opc, op's exit code is unreliable here: check the item's field labels instead.
|
||||||
&& unset "$1" && envs
|
_dotfiles_op sh -c '
|
||||||
|
clean() { printf "%s" "${1:-(no output)}" | sed "s/^\[ERROR\] [0-9/]* [0-9:]* //"; }
|
||||||
|
ref="op://$OP_VAULT/$OP_ITEM/$1"
|
||||||
|
err=$(op item edit "$OP_ITEM" --vault "$OP_VAULT" "$1[delete]" 2>&1 >/dev/null)
|
||||||
|
if ! item=$(op item get "$OP_ITEM" --vault "$OP_VAULT" --format json 2>&1); then
|
||||||
|
printf "opd: could not re-read %s to confirm the delete\n op said: %s\n" "$ref" "$(clean "$item")" >&2
|
||||||
|
exit 1
|
||||||
|
elif printf "%s" "$item" | jq -r ".fields[].label" | grep -qxF "$1"; then
|
||||||
|
printf "opd: could not delete %s\n op said: %s\n" "$ref" "$(clean "$err")" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
printf "opd: deleted %s\n" "$ref" >&2' opd "$1" \
|
||||||
|
&& unset "$1" && ops
|
||||||
}
|
}
|
||||||
envl() {
|
opl() {
|
||||||
_dotfiles_op sh -c '
|
_dotfiles_op sh -c '
|
||||||
vault=${1:-$OP_VAULT} item=${2:-$OP_ITEM}
|
vault=${1:-$OP_VAULT} item=${2:-$OP_ITEM}
|
||||||
printf "op://%s/%s\n" "$vault" "$item" >&2
|
printf "op://%s/%s\n" "$vault" "$item" >&2
|
||||||
op item get "$item" --vault "$vault" --reveal --format json' envl "$@" \
|
op item get "$item" --vault "$vault" --reveal --format json' opl "$@" \
|
||||||
| jq -r '
|
| jq -r '
|
||||||
[.fields[] | select(.purpose != "NOTES")] as $f
|
[.fields[] | select(.purpose != "NOTES")] as $f
|
||||||
| ([$f[] | .label // "" | length] | max // 0) as $w
|
| ([$f[] | .label // "" | length] | max // 0) as $w
|
||||||
@@ -136,7 +152,7 @@ envl() {
|
|||||||
| "\($name) \($lines[0] // "")",
|
| "\($name) \($lines[0] // "")",
|
||||||
($lines[1:][] | (" " * ($w + 2)) + .)'
|
($lines[1:][] | (" " * ($w + 2)) + .)'
|
||||||
}
|
}
|
||||||
envs() {
|
ops() {
|
||||||
"${DOTFILES_DIR:-$HOME/.dotfiles}/secrets/install.sh" \
|
"${DOTFILES_DIR:-$HOME/.dotfiles}/secrets/install.sh" \
|
||||||
&& source "$_DOTFILES_CONF/secrets.env"
|
&& source "$_DOTFILES_CONF/secrets.env"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -36,7 +36,7 @@
|
|||||||
# Nothing to set here. Do NOT loop over `op read` in this file: that ran on
|
# Nothing to set here. Do NOT loop over `op read` in this file: that ran on
|
||||||
# every shell. secrets/install.sh stores the service-account token in
|
# every shell. secrets/install.sh stores the service-account token in
|
||||||
# ~/.config/dotfiles/op.env and writes the vault's fields to secrets.env, which
|
# ~/.config/dotfiles/op.env and writes the vault's fields to secrets.env, which
|
||||||
# the managed ~/.zshrc loads. Use envr / envc / envd (see zsh/zshrc).
|
# the managed ~/.zshrc loads. Use opr / opc / opd (see zsh/zshrc).
|
||||||
|
|
||||||
# --- Proxmox -------------------------------------------------------------
|
# --- Proxmox -------------------------------------------------------------
|
||||||
# One pair of credentials, several consumers: the proxmox MCP server
|
# One pair of credentials, several consumers: the proxmox MCP server
|
||||||
|
|||||||
Reference in New Issue
Block a user