From 11138d90a99ed3fce86973b5280239191e272a93 Mon Sep 17 00:00:00 2001 From: Nic O'Connor Date: Sun, 13 Sep 2026 08:51:28 -0400 Subject: [PATCH] claude-obsidian: summarize Claude conversations into Obsidian notes One note per Claude Code session (incremental, hourly user timer) and a claude.ai data-export importer, written through the Obsidian Local REST API plugin -- whose built-in /mcp/ server is also how Claude reads them back. Transcripts are condensed (tool output dropped, one line per tool call, ~60x smaller), redacted, then summarized by headless `claude -p` on the user's own plan. Three things learned the hard way are encoded here: - --no-session-persistence, or every summary becomes a transcript that the next run summarizes. - A LiteLLM gateway passing through the caller's Claude subscription has no key of its own, so a background job cannot use its claude-* models. - Summaries share the plan's usage limits. A parallel ~100-session backfill exhausted a session limit in minutes. So one job, a per-run cap, and the run stops at the first usage-limit error (exit 75) instead of failing the whole queue; a stop flag, because Future.cancel() cannot stop work a thread already picked up. Covered by an offline test of stop / resume / idempotent rerun. Nothing environment-specific is committed: hosts, keys and paths live in ~/.config/claude-obsidian/ (this repo is public). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01KF322NzozeaFjpgWvh4F4g --- claude-obsidian/README.md | 91 +++ claude-obsidian/claude-obsidian-sync | 635 ++++++++++++++++++ claude-obsidian/config.env.example | 23 + claude-obsidian/install.sh | 14 + .../systemd/claude-obsidian-sync.service | 11 + .../systemd/claude-obsidian-sync.timer | 10 + 6 files changed, 784 insertions(+) create mode 100644 claude-obsidian/README.md create mode 100755 claude-obsidian/claude-obsidian-sync create mode 100644 claude-obsidian/config.env.example create mode 100755 claude-obsidian/install.sh create mode 100644 claude-obsidian/systemd/claude-obsidian-sync.service create mode 100644 claude-obsidian/systemd/claude-obsidian-sync.timer diff --git a/claude-obsidian/README.md b/claude-obsidian/README.md new file mode 100644 index 0000000..c7522d6 --- /dev/null +++ b/claude-obsidian/README.md @@ -0,0 +1,91 @@ +# claude-obsidian — Claude conversations → Obsidian notes + +Summarizes every Claude conversation into one searchable Obsidian note: what happened, what +was decided and why, the facts and gotchas worth remembering, and what was left open. + +- **Claude Code** sessions sync automatically and incrementally (hourly timer). +- **claude.ai** chats import from a data export (claude.ai has no API, so each export is manual). + +Notes are written through the [Obsidian Local REST API](https://github.com/coddingtonbear/obsidian-local-rest-api) +plugin, so the vault can live anywhere the plugin is reachable. The same plugin exposes a +**built-in MCP server** (`/mcp/`), which is how Claude reads the notes back — no separate MCP +process. + +``` +~/.claude/projects/*/*.jsonl ─┐ ┌─ Claude/Code/2026/…md +claude.ai export .zip ────────┼─ condense → redact → summarize → PUT ────┼─ Claude/Web/2026/…md + └─ (headless claude -p, no transcript) └─ Claude/Index.md +``` + +## Install + +1. In Obsidian, install **Local REST API**, enable its **HTTP server**, and make it reachable + from this machine. Put the API key in `~/.config/claude-obsidian/api-key` (`chmod 600`). +2. `./install.sh` — links the binary into `~/.local/bin`, enables the hourly user timer, and + creates `~/.config/claude-obsidian/config.env` from `config.env.example` if absent. +3. Edit `config.env` (at minimum `OBSIDIAN_API_URL`), then preview one note: + + ```bash + claude-obsidian-sync stats # sizes, no network + claude-obsidian-sync sync --dry-run --limit 1 # prints the note, writes nothing + ``` + +## Use + +```bash +claude-obsidian-sync sync # what the timer runs +claude-obsidian-sync sync --session # one session, now, ignoring the settle window +claude-obsidian-sync import-claude-ai ~/Downloads/data-2026-09-13.zip +``` + +Claude reads the vault through the plugin's MCP server: + +```bash +claude mcp add --scope user --transport http obsidian http://:27123/mcp/ \ + --header "Authorization: Bearer $(cat ~/.config/claude-obsidian/api-key)" +``` + +## How it behaves — and why + +- **Summaries run on your own Claude plan** via headless `claude -p` (`LLM_BACKEND=claude-cli`). + No API key to manage. An OpenAI-compatible backend (`LLM_BACKEND=openai`) exists for local + models. Note a LiteLLM gateway that fronts Claude by *passing through the caller's* + subscription has no key of its own, so a background job cannot use its `claude-*` models. +- **`--no-session-persistence` is load-bearing.** Without it every summary writes a new + transcript into `~/.claude/projects`, and the next run summarizes its own summarizing. +- **It shares your plan's usage limits.** A first backfill of ~100 sessions in parallel hit a + session limit within minutes — which also limits *interactive* use. So: one job at a time, + `MAX_PER_RUN` sessions per run (newest first), and the run **stops at the first usage-limit + error** (exit 75, which the systemd unit treats as success) instead of failing the rest of + the queue. The next hourly run resumes. A 100-session backlog drains in about ten hours. +- **Secrets are redacted before anything leaves the machine**, and again in the note: private + keys, bearer tokens, `password=`-style assignments, JWTs, provider tokens, ≥48-char hex, and + random mixed-case strings. It over-redacts on purpose; git SHAs, md5s and paths survive. + The summarizer is also told never to reproduce secrets. +- **Incremental and idempotent.** State (`~/.local/state/claude-obsidian/state.json`) records + each session's size+mtime. A resumed session is re-summarized into the *same* note path. A + session is left alone until idle for `SETTLE_MINUTES`, so a live conversation isn't + summarized mid-flight. Empty sessions are recorded and skipped. +- **Condensed, not truncated.** Tool output is dropped and each tool call becomes one line; + prose is kept. Typical compression is ~60× (271 MB of transcripts → 4.4 MB). Sessions over + 300 KB condensed are summarized in parts, then combined. +- **Subagent transcripts** (`/subagents/**`) are part of their parent session and are + not synced separately. + +## Note format + +```yaml +type: claude-conversation +source: claude-code # or claude-ai +session_id: 3f9c2a1e-… +title: Fix flaky CI cache after runner upgrade +project: my-service # basename of the working directory +branch: … +started / ended: … +prompts: 1 # user prompts, not tool-loop records +summarized_by: claude-haiku-4-5-20251001 +tags: [claude, claude-code, ci, caching, …] +``` + +followed by a summary callout and **Outcomes / Decisions / Facts & gotchas / Open threads**, +and a `claude --resume ` line. `Claude/Index.md` lists every note, newest first. diff --git a/claude-obsidian/claude-obsidian-sync b/claude-obsidian/claude-obsidian-sync new file mode 100755 index 0000000..5b0041d --- /dev/null +++ b/claude-obsidian/claude-obsidian-sync @@ -0,0 +1,635 @@ +#!/usr/bin/env python3 +"""claude-obsidian-sync -- summarize Claude conversations into an Obsidian vault. + +Reads Claude Code transcripts (~/.claude/projects//.jsonl) and claude.ai +data exports, condenses each conversation, redacts secrets, summarizes it with an +OpenAI-compatible LLM endpoint (e.g. LiteLLM), and writes one note per conversation into +Obsidian through the Local REST API plugin. Standard library only. + + claude-obsidian-sync stats # sizes and counts, no network + claude-obsidian-sync sync [--dry-run] # Claude Code sessions (incremental) + claude-obsidian-sync import-claude-ai EXPORT.zip|conversations.json + +Configuration comes from the environment, falling back to ~/.config/claude-obsidian/config.env +(KEY=VALUE lines). Nothing environment-specific lives in this file. See README.md. +""" +from __future__ import annotations + +import argparse +import concurrent.futures as cf +import datetime as dt +import hashlib +import json +import os +import pathlib +import re +import shutil +import subprocess +import sys +import threading +import time +import urllib.error +import urllib.parse +import urllib.request +import zipfile + +HOME = pathlib.Path.home() +CONFIG_DIR = HOME / ".config" / "claude-obsidian" + + +# --------------------------------------------------------------------------- config +def load_config() -> dict: + cfg: dict[str, str] = {} + f = CONFIG_DIR / "config.env" + if f.exists(): + for line in f.read_text().splitlines(): + line = line.strip() + if line and not line.startswith("#") and "=" in line: + k, v = line.split("=", 1) + cfg[k.strip()] = v.strip().strip('"').strip("'") + cfg.update({k: v for k, v in os.environ.items() if k in DEFAULTS or k.startswith("LLM_") or k.startswith("OBSIDIAN_")}) + for k, v in DEFAULTS.items(): + cfg.setdefault(k, v) + return cfg + + +DEFAULTS = { + "OBSIDIAN_API_URL": "http://127.0.0.1:27123", + "OBSIDIAN_API_KEY_FILE": str(CONFIG_DIR / "api-key"), + "LLM_BACKEND": "claude-cli", # claude-cli | openai + "CLAUDE_BIN": "claude", + "LLM_BASE_URL": "", + "LLM_API_KEY_FILE": "", + "LLM_API_KEY_ENV": "LITELLM_KEY", + "LLM_MODEL": "haiku", + "VAULT_FOLDER": "Claude", + "SETTLE_MINUTES": "30", + "CLAUDE_PROJECTS_DIR": str(HOME / ".claude" / "projects"), + "STATE_FILE": str(HOME / ".local" / "state" / "claude-obsidian" / "state.json"), + "JOBS": "1", + # Cap per run. Summaries draw on the same plan limits as interactive Claude use, so the + # timer drips a backlog instead of bursting it (a 107-session burst exhausted a session limit). + "MAX_PER_RUN": "10", +} + + +def secret(cfg: dict, file_key: str, env_key: str | None = None) -> str: + path = cfg.get(file_key) + if path: + p = pathlib.Path(os.path.expanduser(path)) + if p.exists(): + return p.read_text().strip() + if env_key and os.environ.get(cfg.get(env_key, "") or ""): + return os.environ[cfg[env_key]].strip() + raise SystemExit(f"missing secret: set {file_key} (or {env_key})") + + +# --------------------------------------------------------------------------- redaction +# Applied to everything sent to the LLM AND to the note written. Over-redaction is the +# accepted failure mode: a summary missing a token is fine, a vault holding one is not. +_REDACTIONS = [ + (re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----.*?-----END [A-Z ]*PRIVATE KEY-----", re.S), "[REDACTED PRIVATE KEY]"), + (re.compile(r"(?i)\b(bearer|token)\s+[A-Za-z0-9._~+/=-]{16,}"), r"\1 [REDACTED]"), + (re.compile(r"(?i)((?:password|passwd|pwd|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|client[_-]?secret|NB_API_KEY|PAT)[\"']?\s*[:=]\s*[\"']?)([^\s\"'`,;]{6,})"), r"\1[REDACTED]"), + (re.compile(r"\beyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}"), "[REDACTED JWT]"), + (re.compile(r"\b(?:sk|pk|rk)-[A-Za-z0-9_-]{20,}|\bgh[pousr]_[A-Za-z0-9]{30,}|\bglpat-[A-Za-z0-9_-]{20,}|\bxox[abpr]-[A-Za-z0-9-]{10,}"), "[REDACTED TOKEN]"), + # >=48 hex: API keys / sha256 tokens. Deliberately leaves 40-hex git SHAs and 32-hex md5s. + (re.compile(r"\b[a-fA-F0-9]{48,}\b"), "[REDACTED HEX]"), + # Random mixed-case+digit strings (generated passwords). No '/' so paths survive. + (re.compile(r"(? str: + for pat, repl in _REDACTIONS: + text = pat.sub(repl, text) + return text + + +# --------------------------------------------------------------------------- parsing +_NOISE_BLOCKS = re.compile( + r"<(system-reminder|local-command-stdout|local-command-stderr|local-command-caveat|task-notification)>.*?", + re.S, +) +_CMD = re.compile(r"(.*?).*?(?:(.*?))?", re.S) +MAX_MSG = 6000 + + +def _clip(text: str) -> str: + return text if len(text) <= MAX_MSG else text[:4000] + "\n[...clipped...]\n" + text[-1500:] + + +def _clean_user(text: str) -> str: + text = _NOISE_BLOCKS.sub("", text) + m = _CMD.search(text) + if m: + args = (m.group(2) or "").strip() + text = f"{m.group(1).strip()} {args}".strip() + text = re.sub(r"", "", text) + return text.strip() + + +def _tool_line(block: dict) -> str: + name = block.get("name", "tool") + inp = block.get("input") or {} + desc = inp.get("description") or inp.get("command") or inp.get("file_path") or inp.get("query") \ + or inp.get("url") or inp.get("prompt") or inp.get("pattern") or "" + desc = " ".join(str(desc).split())[:160] + return f"[{name}: {desc}]" if desc else f"[{name}]" + + +def parse_claude_code(path: pathlib.Path) -> dict: + meta = {"session_id": path.stem, "cwd": None, "branch": None, "first": None, "last": None, + "title": None, "version": None, "prompts": 0} + turns: list[list[str]] = [] + for line in path.open(encoding="utf-8", errors="replace"): + try: + r = json.loads(line) + except json.JSONDecodeError: + continue + t = r.get("type") + if t == "ai-title" and r.get("aiTitle"): + meta["title"] = r["aiTitle"] + if t == "custom-title" and r.get("customTitle"): + meta["title"] = r["customTitle"] + if t not in ("user", "assistant") or r.get("isSidechain") or r.get("isMeta"): + continue + ts = r.get("timestamp") + if ts: + meta["first"] = meta["first"] or ts + meta["last"] = ts + meta["cwd"] = meta["cwd"] or r.get("cwd") + meta["branch"] = r.get("gitBranch") or meta["branch"] + meta["version"] = r.get("version") or meta["version"] + content = (r.get("message") or {}).get("content") + if t == "user": + if isinstance(content, str): + text = content + elif isinstance(content, list): + text = "\n".join(b.get("text", "") for b in content if b.get("type") == "text") + else: + continue + text = _clean_user(text) + if text: + role = "SUMMARY-OF-EARLIER-CONTEXT" if r.get("isCompactSummary") else "USER" + turns.append([role, _clip(text)]) + meta["prompts"] += role == "USER" + else: + parts = [] + for b in content if isinstance(content, list) else []: + if b.get("type") == "text" and b.get("text", "").strip(): + parts.append(_clip(b["text"].strip())) + elif b.get("type") == "tool_use": + parts.append(_tool_line(b)) + if not parts: + continue + if turns and turns[-1][0] == "ASSISTANT": # a tool loop is many records + turns[-1][1] += "\n" + "\n".join(parts) + else: + turns.append(["ASSISTANT", "\n".join(parts)]) + meta["transcript"] = "\n\n".join(f"{role}: {text}" for role, text in turns) + return meta + + +def parse_claude_ai_export(src: pathlib.Path) -> list[dict]: + """claude.ai 'Export data' -> conversations.json. Tolerant of field-name drift.""" + if src.suffix == ".zip": + with zipfile.ZipFile(src) as z: + name = next((n for n in z.namelist() if n.endswith("conversations.json")), None) + if not name: + raise SystemExit(f"no conversations.json inside {src}") + data = json.loads(z.read(name)) + else: + data = json.loads(src.read_text()) + out = [] + for c in data if isinstance(data, list) else data.get("conversations", []): + turns, count = [], 0 + for m in c.get("chat_messages") or c.get("messages") or []: + text = m.get("text") or "" + if not text and isinstance(m.get("content"), list): + text = "\n".join(b.get("text", "") for b in m["content"] if b.get("type") == "text") + for a in m.get("attachments") or []: + if a.get("extracted_content"): + text += f"\n[attachment {a.get('file_name', '')}]: {a['extracted_content'][:1500]}" + if not text.strip(): + continue + role = "USER" if (m.get("sender") or m.get("role")) in ("human", "user") else "ASSISTANT" + turns.append(f"{role}: {_clip(text.strip())}") + count += role == "USER" + if not turns: + continue + out.append({ + "session_id": c.get("uuid") or hashlib.sha1(json.dumps(c, sort_keys=True).encode()).hexdigest(), + "title": c.get("name") or None, "first": c.get("created_at"), "last": c.get("updated_at"), + "cwd": None, "branch": None, "version": None, "prompts": count, + "transcript": "\n\n".join(turns), + }) + return out + + +# --------------------------------------------------------------------------- LLM +SYSTEM = """You write concise knowledge-base notes summarizing a conversation between a user and Claude. +The reader is the same user, months later, searching their notes. Capture what happened and +what was learned, not a play-by-play. Be specific: name the systems, files, commands, errors +and decisions. Never include passwords, API keys, tokens or other secrets, even if present; +refer to them generically ("the API key"). Text marked [REDACTED] must stay redacted. + +Reply with ONLY a JSON object, no prose, with these keys: + "title": short specific title (<= 70 chars) + "summary": 3-6 sentences + "outcomes": list of what was actually built, fixed or answered + "decisions": list of choices made and why + "facts": list of reusable facts, gotchas, root causes worth remembering + "open_threads": list of unfinished work or follow-ups + "tags": 3-8 lowercase kebab-case topic tags +Use empty lists where nothing applies.""" + +CHUNK_SYSTEM = """You are condensing one part of a long conversation between a user and Claude so that a +later step can summarize the whole. Output dense bullet notes: goals, actions taken, results, +errors and their causes, decisions, open questions. Be specific. Never reproduce secrets.""" + +CHUNK_CHARS = 300_000 + + +class QuotaExhausted(RuntimeError): + """The plan/API usage limit is hit. Every further call would fail the same way -- stop the run.""" + + +_QUOTA = re.compile(r"(?i)hit your (?:session |usage |weekly )?limit|usage limit|quota|credit balance|resets? (?:at )?\d") + + +class LLM: + """Two backends. + + claude-cli (default): headless Claude Code on the user's own plan -- no API key to manage. + --no-session-persistence is load-bearing: without it every summary writes a new transcript + into ~/.claude/projects and the next sync would summarize its own summarizing. + openai: any OpenAI-compatible /chat/completions endpoint (LiteLLM, local models). + """ + + def __init__(self, cfg: dict): + self.backend = cfg["LLM_BACKEND"] + self.model = cfg["LLM_MODEL"] + self.resolved = cfg["LLM_MODEL"] + if self.backend == "claude-cli": + self.bin = shutil.which(os.path.expanduser(cfg["CLAUDE_BIN"])) + if not self.bin: + raise SystemExit(f"claude CLI not found ({cfg['CLAUDE_BIN']})") + self.workdir = pathlib.Path(os.path.expanduser(cfg["STATE_FILE"])).parent # no project CLAUDE.md here + self.workdir.mkdir(parents=True, exist_ok=True) + elif self.backend == "openai": + if not cfg["LLM_BASE_URL"]: + raise SystemExit("LLM_BASE_URL is not set (see config.env.example)") + self.url = cfg["LLM_BASE_URL"].rstrip("/") + "/chat/completions" + self.key = secret(cfg, "LLM_API_KEY_FILE", "LLM_API_KEY_ENV") + else: + raise SystemExit(f"unknown LLM_BACKEND {self.backend!r}") + + def chat(self, system: str, user: str, max_tokens: int = 2500) -> str: + return self._chat_cli(system, user) if self.backend == "claude-cli" else self._chat_openai(system, user, max_tokens) + + def _chat_cli(self, system: str, user: str) -> str: + cmd = [self.bin, "-p", "--model", self.model, "--no-session-persistence", "--tools", "", + "--strict-mcp-config", "--output-format", "json", "--system-prompt", system] + last = "" + for attempt in range(5): + try: + proc = subprocess.run(cmd, input=user, capture_output=True, text=True, timeout=900, cwd=self.workdir) + except subprocess.TimeoutExpired: + last = "timeout" + else: + try: + d = json.loads(proc.stdout) + except json.JSONDecodeError: + d = {"is_error": True, "result": (proc.stdout or proc.stderr)[-300:]} + if not d.get("is_error") and d.get("result"): + used = list((d.get("modelUsage") or {}).keys()) + if used: + self.resolved = used[0] + return d["result"] + last = str(d.get("result") or proc.stderr)[-300:] + if _QUOTA.search(last): + raise QuotaExhausted(last.strip()) + if not re.search(r"(?i)rate.?limit|overloaded|429|529|timeout|temporar", last): + break # not transient -- do not hammer + time.sleep(2 ** attempt * 10) + raise RuntimeError(f"claude -p failed: {last}") + + def _chat_openai(self, system: str, user: str, max_tokens: int) -> str: + body = json.dumps({"model": self.model, "temperature": 0.2, "max_tokens": max_tokens, + "messages": [{"role": "system", "content": system}, + {"role": "user", "content": user}]}).encode() + for attempt in range(5): + req = urllib.request.Request(self.url, data=body, method="POST", headers={ + "Authorization": f"Bearer {self.key}", "Content-Type": "application/json"}) + try: + with urllib.request.urlopen(req, timeout=600) as r: + return json.load(r)["choices"][0]["message"]["content"] + except urllib.error.HTTPError as e: + detail = e.read()[:300].decode(errors="replace") + if _QUOTA.search(detail): + raise QuotaExhausted(detail) from None + if e.code in (429, 500, 502, 503, 504, 529) and attempt < 4: + time.sleep(2 ** attempt * 5) + continue + raise RuntimeError(f"LLM HTTP {e.code}: {detail}") from None + except (urllib.error.URLError, TimeoutError) as e: + if attempt < 4: + time.sleep(2 ** attempt * 5) + continue + raise RuntimeError(f"LLM unreachable: {e}") from None + raise RuntimeError("LLM retries exhausted") + + def summarize(self, meta: dict) -> dict: + text = redact(meta["transcript"]) + if len(text) > CHUNK_CHARS: # map-reduce for long sessions + chunks, cur = [], [] + size = 0 + for part in text.split("\n\n"): + if size + len(part) > CHUNK_CHARS and cur: + chunks.append("\n\n".join(cur)); cur, size = [], 0 + cur.append(part); size += len(part) + 2 + chunks.append("\n\n".join(cur)) + notes = [self.chat(CHUNK_SYSTEM, f"Part {i + 1} of {len(chunks)}:\n\n{c}", 3000) + for i, c in enumerate(chunks)] + text = "Condensed notes from each part of a long conversation, in order:\n\n" + \ + "\n\n---\n\n".join(notes) + header = f"Working directory: {meta.get('cwd')}\nGit branch: {meta.get('branch')}\n" \ + f"Existing title: {meta.get('title')}\n\n" + raw = self.chat(SYSTEM, header + text) + m = re.search(r"\{.*\}", raw, re.S) + if not m: + raise RuntimeError(f"LLM did not return JSON: {raw[:200]!r}") + out = json.loads(m.group(0)) + for k in ("outcomes", "decisions", "facts", "open_threads", "tags"): + out[k] = [redact(str(x)) for x in (out.get(k) or []) if str(x).strip()] + out["summary"] = redact(str(out.get("summary", ""))) + out["title"] = redact(str(out.get("title") or meta.get("title") or meta["session_id"])) + return out + + +# --------------------------------------------------------------------------- vault +class Vault: + def __init__(self, cfg: dict): + self.base = cfg["OBSIDIAN_API_URL"].rstrip("/") + self.key = secret(cfg, "OBSIDIAN_API_KEY_FILE") + + def put(self, path: str, content: str) -> None: + req = urllib.request.Request(f"{self.base}/vault/{urllib.parse.quote(path)}", + data=content.encode(), method="PUT", headers={ + "Authorization": f"Bearer {self.key}", + "Content-Type": "text/markdown"}) + with urllib.request.urlopen(req, timeout=60) as r: + if r.status not in (200, 201, 204): + raise RuntimeError(f"vault PUT {path}: HTTP {r.status}") + + def ping(self) -> None: + """Checked BEFORE any summarizing, so an unreachable vault never spends usage limits.""" + req = urllib.request.Request(f"{self.base}/", headers={"Authorization": f"Bearer {self.key}"}) + try: + with urllib.request.urlopen(req, timeout=15) as r: + info = json.load(r) + except (urllib.error.URLError, TimeoutError, OSError) as e: + raise SystemExit(f"Obsidian REST API unreachable at {self.base}: {getattr(e, 'reason', e)}") from None + if not info.get("authenticated"): + raise SystemExit("Obsidian REST API reachable but the API key was rejected") + + +_BAD = re.compile(r'[\\/:*?"<>|#^\[\]]+') + + +def note_path(cfg: dict, kind: str, meta: dict, title: str) -> str: + when = (meta.get("first") or dt.datetime.now(dt.timezone.utc).isoformat())[:10] + slug = " ".join(_BAD.sub(" ", title).split())[:60].strip() or "untitled" + return f"{cfg['VAULT_FOLDER']}/{kind}/{when[:4]}/{when} {slug} ({meta['session_id'][:8]}).md" + + +def render(meta: dict, s: dict, source: str, model: str, src_file: str | None) -> str: + project = pathlib.Path(meta["cwd"]).name if meta.get("cwd") else None + tags = ["claude", source] + [t for t in s["tags"] if t not in ("claude", source)] + fm = { + "type": "claude-conversation", "source": source, "session_id": meta["session_id"], + "title": s["title"], "project": project, "cwd": meta.get("cwd"), "branch": meta.get("branch"), + "started": meta.get("first"), "ended": meta.get("last"), "prompts": meta.get("prompts"), + "summarized_by": model, "synced": dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), + "tags": tags, + } + lines = ["---"] + [f"{k}: {json.dumps(v)}" for k, v in fm.items() if v not in (None, "")] + ["---", "", + f"# {s['title']}", "", "> [!summary]"] + [f"> {l}" for l in s["summary"].splitlines() or [""]] + for heading, key in (("Outcomes", "outcomes"), ("Decisions", "decisions"), + ("Facts & gotchas", "facts"), ("Open threads", "open_threads")): + if s[key]: + lines += ["", f"## {heading}", ""] + [f"- {x}" for x in s[key]] + lines += ["", "---"] + if source == "claude-code": + lines.append(f"Resume: `claude --resume {meta['session_id']}`" + (f" · transcript `{src_file}`" if src_file else "")) + else: + lines.append(f"claude.ai conversation `{meta['session_id']}`") + return "\n".join(lines) + "\n" + + +# --------------------------------------------------------------------------- state +class State: + def __init__(self, path: str): + self.path = pathlib.Path(os.path.expanduser(path)) + self.lock = threading.Lock() + self.data = json.loads(self.path.read_text()) if self.path.exists() else {} + + def save(self) -> None: + self.path.parent.mkdir(parents=True, exist_ok=True) + tmp = self.path.with_suffix(".tmp") + tmp.write_text(json.dumps(self.data, indent=1, sort_keys=True)) + tmp.replace(self.path) + + def record(self, key: str, value: dict) -> None: + with self.lock: + self.data[key] = value + self.save() + + +def write_index(cfg: dict, vault: Vault, state: State) -> None: + rows = sorted((v for v in state.data.values() if v.get("note")), key=lambda v: v.get("started") or "", reverse=True) + out = ["---", 'type: "claude-index"', "---", "", "# Claude conversations", "", + f"{len(rows)} conversations. Regenerated by claude-obsidian-sync.", "", + "| Date | Title | Project | Source |", "|---|---|---|---|"] + for v in rows: + link = v["note"][:-3] + title = v.get("title", "").replace("|", "-") + out.append(f"| {(v.get('started') or '')[:10]} | [[{link}\\|{title}]] | {v.get('project') or ''} | {v.get('source')} |") + vault.put(f"{cfg['VAULT_FOLDER']}/Index.md", "\n".join(out) + "\n") + + +# --------------------------------------------------------------------------- commands +def top_level_sessions(cfg: dict) -> list[pathlib.Path]: + root = pathlib.Path(os.path.expanduser(cfg["CLAUDE_PROJECTS_DIR"])) + return sorted(root.glob("*/*.jsonl"), key=lambda p: p.stat().st_mtime) # nested = subagents + + +def process(cfg, llm, vault, state, key, meta, source, kind, fingerprint, src_file, dry_run) -> str: + s = llm.summarize(meta) + prev = state.data.get(key, {}) + path = prev.get("note") or note_path(cfg, kind, meta, s["title"]) # stable across re-syncs + body = render(meta, s, source, llm.resolved, src_file) + if dry_run: + print(f"\n===== {path} =====\n{body}") + return path + vault.put(path, body) + state.record(key, {"note": path, "title": s["title"], "fingerprint": fingerprint, "source": source, + "started": meta.get("first"), + "project": pathlib.Path(meta["cwd"]).name if meta.get("cwd") else None}) + return path + + +SKIPPED = "(empty, skipped)" +EX_TEMPFAIL = 75 # "try again later" -- systemd logs it, the next timer run retries + + +def run_jobs(cfg, work, fn) -> tuple[int, int, str | None]: + ok = failed = 0 + quota = None + stop = threading.Event() # Future.cancel() cannot stop a job a worker already picked up + + def guarded(*w): + if stop.is_set(): + raise cf.CancelledError() + try: + return fn(*w) + except QuotaExhausted: + stop.set() + raise + + with cf.ThreadPoolExecutor(max_workers=int(cfg["JOBS"])) as ex: + futs = {ex.submit(guarded, *w): w for w in work} + for fut in cf.as_completed(futs): + label = futs[fut][0] + try: + result = fut.result() + if result == SKIPPED: + print(f"skip {label} (no conversation content)", flush=True) + else: + print(f"ok {label} -> {result}", flush=True); ok += 1 + except QuotaExhausted as e: + if quota is None: + quota = str(e) + for f in futs: # don't launch the rest + f.cancel() + except cf.CancelledError: + pass + except Exception as e: # keep going; next run retries + print(f"FAIL {label}: {e}", file=sys.stderr, flush=True); failed += 1 + return ok, failed, quota + + +def finish(ok: int, failed: int, quota: str | None, dry_run: bool, remaining: int) -> None: + verb = "rendered (dry run, nothing written)" if dry_run else "written" + print(f"done: {ok} {verb}, {failed} failed, {remaining} still queued") + if quota: + print(f"stopped: usage limit reached -- {quota}", file=sys.stderr) + sys.exit(EX_TEMPFAIL) + sys.exit(1 if failed else 0) + + +def cmd_stats(cfg, args) -> None: + total = 0 + rows = [] + for p in top_level_sessions(cfg): + m = parse_claude_code(p) + n = len(redact(m["transcript"])) + total += n + rows.append((n, p.stat().st_size, m["prompts"], m["title"] or p.stem)) + rows.sort(reverse=True) + for n, raw, msgs, title in rows[: args.top]: + print(f"{raw / 1e6:7.2f} MB raw -> {n / 1e3:7.1f} KB condensed {msgs:4d} prompts {title[:60]}") + print(f"\n{len(rows)} sessions, {total / 1e6:.1f} MB condensed (~{total / 4 / 1e6:.2f}M tokens in), " + f"{sum(1 for r in rows if r[0] > CHUNK_CHARS)} need map-reduce") + + +def cmd_sync(cfg, args) -> None: + state = State(cfg["STATE_FILE"]) + settle = dt.timedelta(minutes=int(cfg["SETTLE_MINUTES"])) + now = time.time() + work = [] + for p in top_level_sessions(cfg): + if args.session and p.stem != args.session: + continue + st = p.stat() + fp = f"{st.st_size}:{int(st.st_mtime)}" + key = f"claude-code:{p.stem}" + if not args.force and state.data.get(key, {}).get("fingerprint") == fp: + continue + if not (args.force or args.session) and now - st.st_mtime < settle.total_seconds(): + continue # still being written + work.append((p, fp, key)) + total = len(work) + cap = args.limit or int(cfg["MAX_PER_RUN"]) + work = work[-cap:] if not args.oldest else work[:cap] # newest first by default + print(f"{total} session(s) pending, summarizing {len(work)} this run", flush=True) + if not work: + return + llm, vault = LLM(cfg), (None if args.dry_run else Vault(cfg)) + if vault: + vault.ping() + + def one(p, fp, key): + meta = parse_claude_code(p) + if not meta["transcript"].strip(): + if not args.dry_run: + state.record(key, {"note": None, "fingerprint": fp, "source": "claude-code"}) + return SKIPPED + src = str(p).replace(str(HOME), "~") + return process(cfg, llm, vault, state, key, meta, "claude-code", "Code", fp, src, args.dry_run) + + ok, failed, quota = run_jobs(cfg, [(w[0].name, *w) for w in work], lambda _l, p, fp, key: one(p, fp, key)) + if vault and ok: + write_index(cfg, vault, state) + finish(ok, failed, quota, args.dry_run, total - ok) + + +def cmd_import(cfg, args) -> None: + state = State(cfg["STATE_FILE"]) + convs = parse_claude_ai_export(pathlib.Path(args.export)) + work = [] + for m in convs: + key = f"claude-ai:{m['session_id']}" + fp = m.get("last") or str(len(m["transcript"])) + if args.force or state.data.get(key, {}).get("fingerprint") != fp: + work.append((m, fp, key)) + total = len(work) + work = work[: args.limit or int(cfg["MAX_PER_RUN"])] + print(f"{len(convs)} conversations in export, {total} new or changed, summarizing {len(work)} this run", flush=True) + if not work: + return + llm, vault = LLM(cfg), (None if args.dry_run else Vault(cfg)) + if vault: + vault.ping() + ok, failed, quota = run_jobs(cfg, [((m.get("title") or m["session_id"])[:50], m, fp, key) for m, fp, key in work], + lambda _l, m, fp, key: process(cfg, llm, vault, state, key, m, "claude-ai", "Web", + fp, None, args.dry_run)) + if vault and ok: + write_index(cfg, vault, state) + finish(ok, failed, quota, args.dry_run, total - ok) + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + sub = ap.add_subparsers(dest="cmd", required=True) + s = sub.add_parser("stats", help="condensed sizes, no network") + s.add_argument("--top", type=int, default=15) + s = sub.add_parser("sync", help="summarize Claude Code sessions") + s.add_argument("--dry-run", action="store_true", help="print notes instead of writing them") + s.add_argument("--force", action="store_true", help="ignore state and the settle window") + s.add_argument("--session", help="only this session id") + s.add_argument("--limit", type=int, help="max sessions this run (default MAX_PER_RUN)") + s.add_argument("--oldest", action="store_true", help="take the oldest pending sessions instead of the newest") + s = sub.add_parser("import-claude-ai", help="import a claude.ai data export") + s.add_argument("export", help="the export .zip or its conversations.json") + s.add_argument("--dry-run", action="store_true") + s.add_argument("--force", action="store_true") + s.add_argument("--limit", type=int, help="max conversations this run (default MAX_PER_RUN)") + args = ap.parse_args() + cfg = load_config() + {"stats": cmd_stats, "sync": cmd_sync, "import-claude-ai": cmd_import}[args.cmd](cfg, args) + + +if __name__ == "__main__": + main() diff --git a/claude-obsidian/config.env.example b/claude-obsidian/config.env.example new file mode 100644 index 0000000..7531d71 --- /dev/null +++ b/claude-obsidian/config.env.example @@ -0,0 +1,23 @@ +# ~/.config/claude-obsidian/config.env (chmod 600; never commit the real one) +# Every key can also be set as an environment variable, which wins. + +# Obsidian Local REST API plugin -- the plain-HTTP server (Settings -> Local REST API -> +# Enable HTTP server). Point this at wherever that port is reachable from this machine. +OBSIDIAN_API_URL=http://obsidian.example.internal:27123 +OBSIDIAN_API_KEY_FILE=~/.config/claude-obsidian/api-key + +# Summarizer. claude-cli = headless `claude -p` on your own Claude plan (no API key needed). +LLM_BACKEND=claude-cli +LLM_MODEL=haiku +# CLAUDE_BIN=claude + +# Or any OpenAI-compatible endpoint (LiteLLM, a local model): +# LLM_BACKEND=openai +# LLM_BASE_URL=https://litellm.example/v1 +# LLM_MODEL=some-model +# LLM_API_KEY_FILE=~/.config/claude-obsidian/llm-key # or LLM_API_KEY_ENV=LITELLM_KEY + +VAULT_FOLDER=Claude +SETTLE_MINUTES=30 # leave sessions alone until they have been idle this long +JOBS=1 +MAX_PER_RUN=10 # summaries share your plan's usage limits; the timer drips a backlog diff --git a/claude-obsidian/install.sh b/claude-obsidian/install.sh new file mode 100755 index 0000000..407ee70 --- /dev/null +++ b/claude-obsidian/install.sh @@ -0,0 +1,14 @@ +#!/bin/sh +# Install claude-obsidian-sync for the current user: binary into ~/.local/bin, systemd user +# timer enabled. Idempotent. Does not create config or secrets -- see README.md. +set -eu +HERE="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)" +mkdir -p "$HOME/.local/bin" "$HOME/.config/systemd/user" "$HOME/.config/claude-obsidian" +chmod 700 "$HOME/.config/claude-obsidian" +ln -sf "$HERE/claude-obsidian-sync" "$HOME/.local/bin/claude-obsidian-sync" +cp "$HERE/systemd/claude-obsidian-sync.service" "$HERE/systemd/claude-obsidian-sync.timer" "$HOME/.config/systemd/user/" +[ -f "$HOME/.config/claude-obsidian/config.env" ] || { cp "$HERE/config.env.example" "$HOME/.config/claude-obsidian/config.env"; chmod 600 "$HOME/.config/claude-obsidian/config.env"; echo "created config.env from the example -- edit it"; } +systemctl --user daemon-reload +systemctl --user enable --now claude-obsidian-sync.timer +echo "installed. check: systemctl --user list-timers claude-obsidian-sync.timer" +echo "timers only run while logged in unless lingering is on: loginctl enable-linger \"\$USER\"" diff --git a/claude-obsidian/systemd/claude-obsidian-sync.service b/claude-obsidian/systemd/claude-obsidian-sync.service new file mode 100644 index 0000000..168a384 --- /dev/null +++ b/claude-obsidian/systemd/claude-obsidian-sync.service @@ -0,0 +1,11 @@ +[Unit] +Description=Summarize Claude conversations into Obsidian +After=network-online.target + +[Service] +Type=oneshot +ExecStart=%h/.local/bin/claude-obsidian-sync sync +Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin +# 75 = usage limit reached. The run stopped cleanly and the next timer run resumes -- not a failure. +SuccessExitStatus=75 +Nice=10 diff --git a/claude-obsidian/systemd/claude-obsidian-sync.timer b/claude-obsidian/systemd/claude-obsidian-sync.timer new file mode 100644 index 0000000..03cc583 --- /dev/null +++ b/claude-obsidian/systemd/claude-obsidian-sync.timer @@ -0,0 +1,10 @@ +[Unit] +Description=Hourly Claude -> Obsidian conversation sync + +[Timer] +OnCalendar=hourly +RandomizedDelaySec=10m +Persistent=true + +[Install] +WantedBy=timers.target