commit 84afdc1f56d07f83cdeb518b79a48c609927f244 Author: Atlas Date: Wed Jul 1 21:25:02 2026 +0200 SecondBrain v0.1 — local-first, privacy-scrubbing RAG memory Generalized + redeployable: config via env, local Ollama embeddings, single portable sqlite-vec db, hybrid dense+FTS5 recall, mandatory credential scrub. diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..ed91ee3 --- /dev/null +++ b/.env.example @@ -0,0 +1,22 @@ +# SecondBrain configuration — copy to `.env` and adjust. All values optional. +# `.env` is git-ignored; never commit real values. + +# Where the single portable database lives. +BRAIN_HOME=~/.secondbrain +# BRAIN_DB=/custom/path/brain.db # overrides BRAIN_HOME for the db file + +# Local embedding backend (Ollama). Point this at any reachable Ollama: +# local: http://localhost:11434 +# a GPU box on your LAN / tailnet: http://100.x.y.z:11434 +OLLAMA_URL=http://localhost:11434 +EMBED_MODEL=nomic-embed-text +EMBED_DIM=768 + +# Privacy: 1 = keep emails/IPs/hostnames (operational knowledge), +# 0 = also strip emails. Secrets are ALWAYS scrubbed regardless. +SCRUB_KEEP_PII=1 + +# Chunking (advanced; defaults are sensible) +# BRAIN_MAX_CHARS=1600 +# BRAIN_MIN_CHARS=40 +# BRAIN_MAX_FILE_BYTES=400000 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..6230af5 --- /dev/null +++ b/.gitignore @@ -0,0 +1,34 @@ +# ── data & secrets — NEVER commit ────────────────────────────── +*.db +*.db-wal +*.db-shm +*.sqlite +*.sqlite3 +data/ +store/ +corpus/ +logs/ +*.log +.env +.env.* +!.env.example +secrets/ +*.age +*.pem +*.key + +# ── python ───────────────────────────────────────────────────── +__pycache__/ +*.py[cod] +.venv/ +venv/ +env/ +*.egg-info/ +.pytest_cache/ +.mypy_cache/ + +# ── os / editor ──────────────────────────────────────────────── +.DS_Store +Thumbs.db +.idea/ +.vscode/ diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..a2e3c4c --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Atlas Corporation + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 index 0000000..9f810e0 --- /dev/null +++ b/README.md @@ -0,0 +1,155 @@ +
+ +# 🧠 SecondBrain + +**A local-first, privacy-scrubbing personal memory you can point at everything you produce.** + +*Notes · code · agent transcripts · event logs → one portable file → instant hybrid recall.* +*No cloud. No API keys. No data egress. Secrets scrubbed before anything is stored.* + +
+ +--- + +SecondBrain is a tiny (~500-line, single-file) knowledge engine. You feed it the things you +already generate — documents, source code, AI-agent chat transcripts, structured event logs — +and it gives you back one command: + +```bash +brain recall "what did we decide about the pricing model" --hybrid +``` + +Everything runs on your own machine. Embeddings are computed locally by +[Ollama](https://ollama.com) (`nomic-embed-text`), and the entire brain is a **single +portable `brain.db`** (SQLite + [`sqlite-vec`](https://github.com/asg017/sqlite-vec)) you can +copy to a USB stick. There is no server to run, no account to create, and **no credential +ever leaves your box** — because SecondBrain redacts them *before* embedding. + +## Why it exists + +Vector-RAG demos are easy; a memory you'd actually trust with your real life is not. SecondBrain +is opinionated about the three things that usually break: + +| Problem | SecondBrain's answer | +|---|---| +| **Secrets leak into your index** (and then into an LLM's context) | A mandatory scrub pass strips PEM keys, JWTs, `sk-ant-…`/`sk-…`/`ghp_…`/AWS keys, and any `password: / token= / bearer …` pattern **before** a chunk is stored. IPs, hostnames and emails are kept as operational knowledge (toggle with `SCRUB_KEEP_PII=0`). | +| **Dense-only search misses exact strings** (invoice numbers, IBANs, container names, file paths) | Hybrid recall fuses dense KNN with FTS5/BM25 lexical search via **reciprocal-rank-fusion** — meaning *and* identifiers both hit. | +| **Cloud RAG = data egress + lock-in + cost** | 100% local. Ollama for embeddings, SQLite for storage. Portable, free, offline-capable, redeployable on any device in minutes. | + +## The senses + +SecondBrain ingests from pluggable "senses" — each strictly **read-only**: + +- **`ingest-files `** — code, docs and config (60+ text extensions; skips `node_modules`, `.git`, binaries, and files > 400 KB). +- **`ingest-transcripts `** — `*.jsonl` AI-agent sessions (Claude Code–style event logs); tool spam is collapsed to signal. +- **`ingest-events --db `** — any SQLite with an `events(id, ts, sense, subject, payload)` table (e.g. email, chat, calendar, finance). Turns real activity into recallable memory. + +Add your own sense in ~10 lines: chunk → `scrub()` → `_stage()`. That's the whole contract. + +## Quickstart + +```bash +# 0. prerequisites: python3, and Ollama running (https://ollama.com) +ollama pull nomic-embed-text + +# 1. install +git clone https://git.atlascorporation.nl/atlas/secondbrain +cd secondbrain +bash deploy/install.sh # venv + deps + schema +. .venv/bin/activate + +# 2. feed it something +python3 brain.py ingest-files ~/notes +python3 brain.py ingest-files ~/code/my-project + +# 3. embed (local, free) +python3 brain.py embed + +# 4. recall +python3 brain.py recall "how does the auth flow work" --hybrid +python3 brain.py stats +``` + +Output: + +``` +#1 score=0.72 via=both rrf=0.0312 [file/my-project] auth.py + def login(user, pw): # verifies against argon2 hash, issues a signed … +``` + +## Configuration + +All via environment (or a `.env` next to `brain.py`). Copy `.env.example` → `.env`: + +| Var | Default | Meaning | +|---|---|---| +| `BRAIN_HOME` | `~/.secondbrain` | where `brain.db` lives | +| `BRAIN_DB` | `$BRAIN_HOME/brain.db` | explicit db path (overrides `BRAIN_HOME`) | +| `OLLAMA_URL` | `http://localhost:11434` | any reachable Ollama — including a **GPU box on your LAN/tailnet** | +| `EMBED_MODEL` | `nomic-embed-text` | embedding model | +| `EMBED_DIM` | `768` | must match the model | +| `SCRUB_KEEP_PII` | `1` | `0` also strips emails (secrets always scrubbed) | + +> **Tip — offload embeddings to a GPU:** point `OLLAMA_URL` at another machine's Ollama +> (`http://100.x.y.z:11434`). SecondBrain will embed there and store locally. This is how you +> keep a laptop's memory current using a desktop GPU, with zero extra infrastructure. + +## Deploy anywhere + +**systemd (continuous refresh):** set your sources in `.env` (`SB_FILE_DIRS`, `SB_TRANSCRIPTS`, +`SB_EVENTS_DB`), then: + +```bash +sudo cp deploy/secondbrain-refresh.* /etc/systemd/system/ +sudo systemctl enable --now secondbrain-refresh.timer # ingest+embed every 15 min +``` + +**Docker:** + +```bash +docker build -t secondbrain -f deploy/Dockerfile . +docker run --rm -v sbdata:/data -e OLLAMA_URL=http://host.docker.internal:11434 \ + secondbrain recall "quarterly numbers" --hybrid +``` + +## Use it as a library + +```python +import brain +for hit in brain.brain_recall_hybrid("open invoices for June", k=8): + print(hit["score"], hit["ref"], hit["text"][:120]) +``` + +`brain_recall_hybrid()` degrades gracefully to dense-only if the lexical index is empty, so it's +a safe drop-in for `brain_recall()`. + +## How it works + +``` + sources ──▶ scrub() ──▶ chunk ──▶ chunks table ──┬─▶ FTS5 (BM25, via triggers) + (senses) (redact) (≤1600 ch) └─▶ vec_chunks (nomic 768-dim, Ollama) + │ + recall("q") ──▶ embed query ──▶ dense KNN ┐ │ + └▶ FTS5 lexical ─────────────┴─ RRF fuse ─┴─▶ ranked hits +``` + +See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the full design, the scrub guarantees, +and the reciprocal-rank-fusion math. + +## Privacy & security + +- **Scrub-before-store** is not optional and runs on every chunk from every sense. `rescrub` + re-applies a hardened scrub to an existing db in place. +- The database **never leaves your machine** unless you copy it. `.gitignore` blocks `*.db`, + `.env`, `secrets/`, `*.age`, `*.pem`, `*.key` so you can't accidentally commit data. +- No telemetry. No network calls except to your configured Ollama endpoint. + +## Related components + +SecondBrain is the memory layer of a larger local-first autonomy stack. Sister components +(separate repos) include the **policy-gate** (an approval brake that parks money/comms/ +irreversible actions for a human) and fleet/ingest tooling. See the wiki. + +## License + +MIT © 2026 Atlas Corporation. Contributions welcome. diff --git a/brain.py b/brain.py new file mode 100755 index 0000000..cf4f944 --- /dev/null +++ b/brain.py @@ -0,0 +1,519 @@ +#!/usr/bin/env python3 +""" +SecondBrain — a local-first, privacy-scrubbing personal RAG memory +================================================================== +An additive knowledge layer you can point at anything you already produce — +notes, code, chat/agent transcripts, structured event logs — that: + + * SCRUBS credentials (keys/tokens/passwords/PEM/JWT) out of every chunk before + it is ever stored or embedded — privacy is the default, not an add-on; + * embeds locally with Ollama (nomic-embed-text, 768-dim) — no cloud, no API + keys, no data egress; + * stores everything in a single portable sqlite-vec file (`brain.db`); + * answers `recall("...")` with hybrid retrieval — dense KNN + FTS5/BM25 fused + by reciprocal-rank-fusion, so both meaning ("what did we decide about X") + and exact identifiers (invoice numbers, IBANs, container names, paths) hit. + +It never writes back to your sources — ingestion is strictly read-only. + +Configuration is entirely via environment variables (see `.env.example`): + BRAIN_HOME base dir for the DB (default: ~/.secondbrain) + BRAIN_DB explicit db path (default: $BRAIN_HOME/brain.db) + OLLAMA_URL Ollama endpoint (default: http://localhost:11434) + EMBED_MODEL embedding model (default: nomic-embed-text) + EMBED_DIM embedding dimension (default: 768) + SCRUB_KEEP_PII 1=keep emails, 0=strip (default: 1) + +The "senses" (ingestion sources): + ingest-files code / docs / config (60+ text extensions) + ingest-transcripts *.jsonl agent sessions (Claude Code style) + ingest-events --db structured events table (email/chat/finance/...) + +Then: + embed [--limit N --batch B] embed the un-embedded backlog (local) + recall "" [--hybrid] retrieve (dense, or dense+lexical fused) + stats counts + coverage + +License: MIT. Project home: https://git.atlascorporation.nl/atlas/secondbrain +""" +import os, sys, json, re, sqlite3, hashlib, time, glob, argparse +import sqlite_vec +import requests + +# ---------------------------------------------------------------------------- +# Config — all via environment, with sane local-first defaults +# ---------------------------------------------------------------------------- +BRAIN_HOME = os.environ.get("BRAIN_HOME", os.path.expanduser("~/.secondbrain")) +DB_PATH = os.environ.get("BRAIN_DB", os.path.join(BRAIN_HOME, "brain.db")) +OLLAMA = os.environ.get("OLLAMA_URL", "http://localhost:11434") +EMBED_MODEL = os.environ.get("EMBED_MODEL", "nomic-embed-text") +DIM = int(os.environ.get("EMBED_DIM", "768")) +MAX_CHARS = int(os.environ.get("BRAIN_MAX_CHARS", "1600")) # ~400 tokens per chunk +MIN_CHARS = int(os.environ.get("BRAIN_MIN_CHARS", "40")) # drop trivially short chunks +MAX_FILE = int(os.environ.get("BRAIN_MAX_FILE_BYTES", "400000")) + +# ---------------------------------------------------------------------------- +# CREDENTIAL SCRUB (mandatory before any text is embedded/stored) +# Strategy: remove SECRETS (keys/tokens/passwords/JWT/PEM), but KEEP IPs, +# hostnames and (by default) emails — those are operational knowledge the brain +# needs. Set SCRUB_KEEP_PII=0 to also strip emails. +# ---------------------------------------------------------------------------- +KEEP_PII = os.environ.get("SCRUB_KEEP_PII", "1") == "1" +_SCRUB = [ + # PEM private key blocks (do first, multi-line) + (re.compile(r'-----BEGIN [A-Z ]*PRIVATE KEY-----.*?-----END [A-Z ]*PRIVATE KEY-----', re.S), ''), + # labelled secrets: password: xxx DB_PASSWORD=xxx "token": "xxx" bearer xxx + # NOTE: [A-Za-z_]* prefix catches DB_PASSWORD / MYSQL_ROOT_PASSWORD / etc. (the '_' blocks a plain \b) + (re.compile(r'(?i)([A-Za-z_]*(?:password|passwd|passphrase|pwd|secret|api[_-]?key|apikey|access[_-]?token|auth[_-]?token|token|bearer|client[_-]?secret|private[_-]?key))\b\s*["\']?\s*(?:is|=|:)\s*["\']?([^\s"\'<>]{3,})'), '\\1='), + # credentials embedded in URLs / connection strings: scheme://user:PASS@host + (re.compile(r'(://[^/\s:@]+:)([^@\s/]{3,})(@)'), '\\1\\3'), + # user@host/PASS or user:PASS pairs + (re.compile(r'([A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+[/:])([A-Za-z0-9][A-Za-z0-9._@#%!-]{7,})'), '\\1'), + # generic "20YY!" password shapes (e.g. Company2026!) + (re.compile(r'\b[A-Za-z][A-Za-z0-9_.-]{2,}20\d\d[!@#%]'), ''), + # vendor key shapes + (re.compile(r'\bgsk_[A-Za-z0-9]{20,}'), ''), + (re.compile(r'\bsk-ant-[A-Za-z0-9_-]{20,}'), ''), + (re.compile(r'\bsk-[A-Za-z0-9]{20,}'), ''), + (re.compile(r'\bAKIA[0-9A-Z]{16}\b'), ''), + (re.compile(r'\bghp_[A-Za-z0-9]{30,}'), ''), + (re.compile(r'\bgithub_pat_[A-Za-z0-9_]{30,}'), ''), + (re.compile(r'\bxox[baprs]-[A-Za-z0-9-]{10,}'), ''), + # JWT + (re.compile(r'\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{6,}'), ''), +] +_SCRUB_PII = [ + (re.compile(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b'), ''), +] +def scrub(text: str) -> str: + if not text: + return "" + for pat, repl in _SCRUB: + text = pat.sub(repl, text) + if not KEEP_PII: + for pat, repl in _SCRUB_PII: + text = pat.sub(repl, text) + return text + +# ---------------------------------------------------------------------------- +# DB +# ---------------------------------------------------------------------------- +def db(): + os.makedirs(os.path.dirname(DB_PATH) or ".", exist_ok=True) + conn = sqlite3.connect(DB_PATH) + conn.enable_load_extension(True) + sqlite_vec.load(conn) + conn.enable_load_extension(False) + return conn + +def init_schema(): + conn = db() + conn.execute("""CREATE TABLE IF NOT EXISTS chunks( + id INTEGER PRIMARY KEY, + sha TEXT UNIQUE, -- dedup key + source TEXT, -- 'file' | 'transcript' | 'events' + project TEXT, + path TEXT, + ref TEXT, -- session id / symbol + ts TEXT, + role TEXT, + text TEXT, + embedded INTEGER DEFAULT 0, + created REAL + )""") + conn.execute("CREATE INDEX IF NOT EXISTS ix_chunks_emb ON chunks(embedded)") + conn.execute(f"CREATE VIRTUAL TABLE IF NOT EXISTS vec_chunks USING vec0(emb float[{DIM}])") + # optional learning layer: durable facts distilled by an external consolidate loop + conn.execute("""CREATE TABLE IF NOT EXISTS facts( + id INTEGER PRIMARY KEY, fact TEXT UNIQUE, topic TEXT, weight REAL DEFAULT 1.0, + provenance TEXT, created REAL)""") + # lexical channel (hybrid recall): FTS5 over chunks.text. + # external-content => stores ONLY the BM25 index, NOT a 2nd copy of the text + # (disk-safe). Triggers keep it in sync with chunks. + conn.execute("CREATE VIRTUAL TABLE IF NOT EXISTS fts_chunks USING fts5(" + "text, content='chunks', content_rowid='id', tokenize='unicode61')") + conn.execute("""CREATE TRIGGER IF NOT EXISTS chunks_ai AFTER INSERT ON chunks BEGIN + INSERT INTO fts_chunks(rowid, text) VALUES (new.id, new.text); END""") + conn.execute("""CREATE TRIGGER IF NOT EXISTS chunks_ad AFTER DELETE ON chunks BEGIN + INSERT INTO fts_chunks(fts_chunks, rowid, text) VALUES('delete', old.id, old.text); END""") + conn.execute("""CREATE TRIGGER IF NOT EXISTS chunks_au AFTER UPDATE ON chunks BEGIN + INSERT INTO fts_chunks(fts_chunks, rowid, text) VALUES('delete', old.id, old.text); + INSERT INTO fts_chunks(rowid, text) VALUES (new.id, new.text); END""") + conn.commit(); conn.close() + print(f"[init] schema ready at {DB_PATH}") + +def build_fts(): + """(Re)build the FTS5 lexical index from existing chunks. Idempotent. + Needed once for a DB that pre-dates the FTS table; triggers keep it fresh after.""" + init_schema() + conn = db() + conn.execute("INSERT INTO fts_chunks(fts_chunks) VALUES('rebuild')") + conn.commit() + n = conn.execute("SELECT count(*) FROM fts_chunks").fetchone()[0] + conn.close() + print(f"[build-fts] lexical index rebuilt: {n} rows") + +# ---------------------------------------------------------------------------- +# chunk helpers +# ---------------------------------------------------------------------------- +def _sha(*parts) -> str: + return hashlib.sha1("\x1f".join(p or "" for p in parts).encode("utf-8", "ignore")).hexdigest() + +def _segments(text: str): + """Split a long blob into <= MAX_CHARS segments on paragraph boundaries.""" + text = text.strip() + if len(text) <= MAX_CHARS: + if len(text) >= MIN_CHARS: + yield text + return + buf = "" + for para in re.split(r'\n\s*\n', text): + if len(buf) + len(para) + 2 > MAX_CHARS: + if len(buf) >= MIN_CHARS: + yield buf.strip() + buf = para + else: + buf += "\n\n" + para + if len(buf.strip()) >= MIN_CHARS: + yield buf.strip() + +def _stage(conn, source, project, path, ref, ts, role, text): + text = scrub(text) + n = 0 + for seg in _segments(text): + sha = _sha(source, path, ref, seg[:120], str(n)) + try: + conn.execute( + "INSERT OR IGNORE INTO chunks(sha,source,project,path,ref,ts,role,text,created)" + " VALUES(?,?,?,?,?,?,?,?,?)", + (sha, source, project, path, ref, ts, role, seg, time.time())) + if conn.total_changes: + n += 1 + except Exception: + pass + return n + +# ---------------------------------------------------------------------------- +# SENSE: transcripts (agent .jsonl: one JSON event per line, Claude Code style) +# ---------------------------------------------------------------------------- +def _event_text(ev): + """Reduce a transcript event to signal text + role. Tool spam collapsed.""" + role = ev.get("type") or (ev.get("message") or {}).get("role") or "" + msg = ev.get("message") or ev + content = msg.get("content") if isinstance(msg, dict) else None + out = [] + if isinstance(content, str): + out.append(content) + elif isinstance(content, list): + for b in content: + if not isinstance(b, dict): + continue + t = b.get("type") + if t == "text" and b.get("text"): + out.append(b["text"]) + elif t == "tool_use": + inp = b.get("input") or {} + desc = inp.get("description") or inp.get("command") or inp.get("file_path") or "" + out.append(f"[tool:{b.get('name','?')}] {str(desc)[:160]}") + elif t == "tool_result": + c = b.get("content") + if isinstance(c, list): + c = " ".join(x.get("text","") for x in c if isinstance(x, dict)) + c = str(c or "") + if len(c) > 300: + c = c[:200] + " … " + c[-80:] + out.append(f"[result] {c}") + return role, "\n".join(s for s in out if s).strip() + +def ingest_transcripts(root): + conn = db() + files = glob.glob(os.path.join(root, "**", "*.jsonl"), recursive=True) + print(f"[ingest-transcripts] {len(files)} session files under {root}") + total = 0 + for i, fp in enumerate(files): + project = os.path.basename(os.path.dirname(fp)) + sess = os.path.basename(fp).replace(".jsonl", "") + buf, last_ts = [], "" + try: + with open(fp, "r", encoding="utf-8", errors="ignore") as fh: + for line in fh: + line = line.strip() + if not line: + continue + try: + ev = json.loads(line) + except Exception: + continue + ts = ev.get("timestamp") or ev.get("ts") or "" + role, txt = _event_text(ev) + if not txt: + continue + last_ts = ts or last_ts + buf.append(f"{role}: {txt}") + if sum(len(x) for x in buf) > MAX_CHARS: + total += _stage(conn, "transcript", project, fp, sess, last_ts, "exchange", "\n".join(buf)) + buf = [] + except Exception as e: + print(f" ! {fp}: {e}") + continue + if buf: + total += _stage(conn, "transcript", project, fp, sess, last_ts, "exchange", "\n".join(buf)) + if (i+1) % 100 == 0: + conn.commit(); print(f" .. {i+1}/{len(files)} files, {total} chunks") + conn.commit(); conn.close() + print(f"[ingest-transcripts] staged {total} new chunks") + +# ---------------------------------------------------------------------------- +# SENSE: files / code / docs +# ---------------------------------------------------------------------------- +FILE_GLOBS = ("*.py","*.sh","*.md","*.markdown","*.txt","*.rst","*.yml","*.yaml","*.toml","*.ini","*.cfg","*.conf","*.env","*.properties","*.json","*.jsonl","*.xml","*.csv","*.tsv","*.sql","*.js","*.mjs","*.cjs","*.ts","*.tsx","*.jsx","*.vue","*.svelte","*.html","*.htm","*.css","*.scss","*.sass","*.less","*.ps1","*.psm1","*.bat","*.cmd","*.go","*.rs","*.java","*.kt","*.c","*.h","*.cpp","*.hpp","*.cs","*.rb","*.php","*.pl","*.lua","*.r","*.R","*.swift","*.scala","*.service","*.timer","*.socket","*.ipynb","*.tf","*.hcl","*.gitignore","*.editorconfig") +SKIP_DIRS = (".git","node_modules","venv",".venv","__pycache__","vendor",".obsidian","dist","build",".next",".cache") +def ingest_files(root): + conn = db() + n_files = total = 0 + for dirpath, dirs, names in os.walk(root): + dirs[:] = [d for d in dirs if d not in SKIP_DIRS] + for name in names: + if not any(glob.fnmatch.fnmatch(name, g) for g in FILE_GLOBS): + continue + fp = os.path.join(dirpath, name) + try: + if os.path.getsize(fp) > MAX_FILE: + continue + with open(fp, "r", encoding="utf-8", errors="ignore") as fh: + txt = fh.read() + except Exception: + continue + n_files += 1 + total += _stage(conn, "file", os.path.basename(root), fp, name, "", "code", txt) + conn.commit(); conn.close() + print(f"[ingest-files] {n_files} files -> staged {total} new chunks") + +# ---------------------------------------------------------------------------- +# SENSE: structured events (any sqlite with an events table) +# Expected schema (columns; extra columns ignored): +# events(id, ts, sense, subject, payload) -- payload = JSON string +# `sense` is a free-text channel label (e.g. email, chat, finance, calendar). +# This lets the brain answer questions about real activity, not just documents. +# ---------------------------------------------------------------------------- +def ingest_events(events_db, senses=None): + import sqlite3 as _sq, json as _json + conn = db() + k = _sq.connect(f"file:{events_db}?mode=ro", uri=True) + if senses: + placeholders = ",".join("?" for _ in senses) + rows = k.execute(f"SELECT id, ts, sense, subject, payload FROM events " + f"WHERE sense IN ({placeholders}) ORDER BY id", tuple(senses)).fetchall() + else: + rows = k.execute("SELECT id, ts, sense, subject, payload FROM events ORDER BY id").fetchall() + total = 0 + for eid, ts, sense, subject, payload in rows: + try: p = _json.loads(payload or "{}") + except Exception: p = {} + ref = p.get("from") or p.get("from_name") or p.get("counterparty") or sense or "?" + body = p.get("text") or p.get("info") or p.get("body") or "" + txt = f"[{sense}] {subject or ''} {body}".strip() + if txt: + total += _stage(conn, "events", sense, f"event-{eid}", str(ref)[:80], ts, sense, txt) + conn.commit(); conn.close(); k.close() + print(f"[ingest-events] {len(rows)} events -> staged {total} new chunks") + +# ---------------------------------------------------------------------------- +# embedding (local nomic-embed-text via Ollama) +# ---------------------------------------------------------------------------- +import numpy as np +def _l2norm(v): + a = np.asarray(v, dtype=np.float32); n = float(np.linalg.norm(a)) + return (a / n) if n else a # unit vector -> L2 ranking == cosine ranking + +EMBED_CAP = int(os.environ.get("EMBED_CHAR_CAP", "6000")) # ~1500 tok + +def _embed_one(text, prefix="search_document: "): + r = requests.post(f"{OLLAMA}/api/embeddings", + json={"model": EMBED_MODEL, "prompt": prefix + text, "keep_alive": "10m"}, timeout=120) + r.raise_for_status() + return _l2norm(r.json()["embedding"]).tolist() + +def _embed_batch(texts, prefix="search_document: "): + """Batch embed via /api/embed; falls back to the single endpoint per-chunk.""" + try: + r = requests.post(f"{OLLAMA}/api/embed", + json={"model": EMBED_MODEL, "input": [prefix + (t or "").replace(chr(0), " ")[:EMBED_CAP] for t in texts], "keep_alive": "10m"}, + timeout=600) + if r.status_code == 404: + raise RuntimeError("no /api/embed") + r.raise_for_status() + return [_l2norm(e) for e in r.json().get("embeddings", [])] + except Exception: + return [_l2norm(_emb_raw(t, prefix)) for t in texts] + +def _emb_raw(text, prefix): + r = requests.post(f"{OLLAMA}/api/embeddings", + json={"model": EMBED_MODEL, "prompt": prefix + (text or "").replace(chr(0), " ")[:EMBED_CAP], "keep_alive": "10m"}, timeout=120) + r.raise_for_status(); return r.json()["embedding"] + +def embed(limit=None, batch=48): + conn = db() + q = "SELECT id,text FROM chunks WHERE embedded=0 ORDER BY id" + if limit: + q += f" LIMIT {int(limit)}" + rows = conn.execute(q).fetchall() + print(f"[embed] {len(rows)} chunks to embed via {EMBED_MODEL} (batch={batch})") + done = 0; t0 = time.time() + for i in range(0, len(rows), batch): + grp = rows[i:i+batch] + try: + vecs = _embed_batch([t for _, t in grp]) + for (cid, _), v in zip(grp, vecs): + conn.execute("INSERT OR REPLACE INTO vec_chunks(rowid,emb) VALUES(?,?)", + (cid, sqlite_vec.serialize_float32(v))) + conn.execute("UPDATE chunks SET embedded=1 WHERE id=?", (cid,)) + conn.commit(); done += len(grp) + rate = done/(time.time()-t0+1e-9) + print(f" .. {done}/{len(rows)} {rate:.1f}/s") + except Exception: + bad = 0 + for cid, t in grp: + try: + v = _l2norm(_emb_raw(t or "", "search_document: ")) + conn.execute("INSERT OR REPLACE INTO vec_chunks(rowid,emb) VALUES(?,?)", + (cid, sqlite_vec.serialize_float32(v))) + conn.execute("UPDATE chunks SET embedded=1 WHERE id=?", (cid,)) + done += 1 + except Exception: + conn.execute("UPDATE chunks SET embedded=-1 WHERE id=?", (cid,)) + bad += 1 + conn.commit() + print(f" ~ batch@{i} fell back per-chunk: {bad} unembeddable (embedded=-1)") + conn.commit(); conn.close() + print(f"[embed] embedded {done} chunks in {time.time()-t0:.0f}s") + +# ---------------------------------------------------------------------------- +# recall +# ---------------------------------------------------------------------------- +def brain_recall(query, k=6): + conn = db() + qv = _embed_one(query, prefix="search_query: ") + rows = conn.execute(""" + SELECT c.project, c.source, c.path, c.ref, c.ts, c.text, v.distance + FROM vec_chunks v JOIN chunks c ON c.id = v.rowid + WHERE v.emb MATCH ? AND k = ? + ORDER BY v.distance + """, (sqlite_vec.serialize_float32(qv), k)).fetchall() + conn.close() + return [{"project":p,"source":s,"path":pa,"ref":rf,"ts":ts, + "score":round(1-(d*d)/2,3),"text":tx} for (p,s,pa,rf,ts,tx,d) in rows] + +def _fts_sanitize(q): + """Turn arbitrary user text into a safe FTS5 MATCH string: alphanumeric + terms, each quoted (neutralizes FTS5 operators), joined by OR for recall.""" + terms = [t for t in re.findall(r"[A-Za-z0-9_]+", (q or "")) if len(t) > 1] + return " OR ".join(f'"{t}"' for t in terms) + +def brain_recall_hybrid(query, k=6, pool=50, rrf_k=60): + """Hybrid recall: dense KNN + FTS5 BM25 fused by reciprocal-rank-fusion. + Same dict shape as brain_recall plus 'rrf' and 'via'. Degrades to dense-only + if the lexical channel is empty, so it is a safe drop-in for brain_recall.""" + conn = db() + qv = _embed_one(query, prefix="search_query: ") + qblob = sqlite_vec.serialize_float32(qv) + dense = conn.execute( + "SELECT v.rowid, v.distance FROM vec_chunks v WHERE v.emb MATCH ? AND k = ? ORDER BY v.distance", + (qblob, pool)).fetchall() + dense_rank = {cid: i for i, (cid, _) in enumerate(dense)} + dense_cos = {cid: 1-(d*d)/2 for cid, d in dense} + lex_rank = {} + m = _fts_sanitize(query) + if m: + try: + lex = conn.execute( + "SELECT rowid FROM fts_chunks WHERE fts_chunks MATCH ? ORDER BY rank LIMIT ?", + (m, pool)).fetchall() + lex_rank = {cid: i for i, (cid,) in enumerate(lex)} + except Exception: + lex_rank = {} + ids = set(dense_rank) | set(lex_rank) + fused = {} + for cid in ids: + s = 0.0 + if cid in dense_rank: s += 1.0/(rrf_k + dense_rank[cid]) + if cid in lex_rank: s += 1.0/(rrf_k + lex_rank[cid]) + fused[cid] = s + top = sorted(ids, key=lambda c: fused[c], reverse=True)[:k] + out = [] + for cid in top: + row = conn.execute("SELECT project,source,path,ref,ts,text FROM chunks WHERE id=?", (cid,)).fetchone() + if not row: + continue + p, s, pa, rf, ts, tx = row + cos = dense_cos.get(cid) + if cos is None: + try: + d = conn.execute("SELECT vec_distance_L2(emb, ?) FROM vec_chunks WHERE rowid=?", + (qblob, cid)).fetchone() + cos = 1-(d[0]*d[0])/2 if d and d[0] is not None else 0.0 + except Exception: + cos = 0.0 + via = "both" if (cid in dense_rank and cid in lex_rank) else ("dense" if cid in dense_rank else "lexical") + out.append({"project":p,"source":s,"path":pa,"ref":rf,"ts":ts, + "score":round(cos,3),"rrf":round(fused[cid],4),"via":via,"text":tx}) + conn.close() + return out + +def rescrub(): + """Re-apply the (possibly hardened) scrub to every staged chunk in place.""" + conn = db() + rows = conn.execute("SELECT id,text FROM chunks").fetchall() + changed = 0 + for cid, text in rows: + s = scrub(text) + if s != text: + conn.execute("UPDATE chunks SET text=? WHERE id=?", (s, cid)) + changed += 1 + conn.commit(); conn.close() + print(f"[rescrub] re-scrubbed {changed}/{len(rows)} chunks") + +def stats(): + conn = db() + tot = conn.execute("SELECT count(*) FROM chunks").fetchone()[0] + emb = conn.execute("SELECT count(*) FROM chunks WHERE embedded=1").fetchone()[0] + bysrc = conn.execute("SELECT source,count(*) FROM chunks GROUP BY source").fetchall() + proj = conn.execute("SELECT project,count(*) c FROM chunks GROUP BY project ORDER BY c DESC LIMIT 10").fetchall() + facts = conn.execute("SELECT count(*) FROM facts").fetchone()[0] + conn.close() + print(f"chunks: {tot} embedded: {emb} facts: {facts}") + print("by source:", dict(bysrc)) + print("top projects:", proj) + +# ---------------------------------------------------------------------------- +def main(): + ap = argparse.ArgumentParser(prog="brain", description="SecondBrain — local-first RAG memory") + sub = ap.add_subparsers(dest="cmd", required=True) + sub.add_parser("init") + sub.add_parser("build-fts") + p = sub.add_parser("ingest-transcripts"); p.add_argument("dir") + p = sub.add_parser("ingest-files"); p.add_argument("dir") + p = sub.add_parser("ingest-events"); p.add_argument("--db", required=True); p.add_argument("--sense", action="append", help="restrict to these sense labels") + p = sub.add_parser("embed"); p.add_argument("--limit", type=int); p.add_argument("--batch", type=int, default=64) + p = sub.add_parser("recall"); p.add_argument("query"); p.add_argument("-k", type=int, default=6) + p.add_argument("--hybrid", action="store_true", help="dense + FTS5 lexical, RRF-fused") + sub.add_parser("rescrub") + sub.add_parser("stats") + a = ap.parse_args() + if a.cmd == "init": init_schema() + elif a.cmd == "build-fts": build_fts() + elif a.cmd == "rescrub": rescrub() + elif a.cmd == "ingest-transcripts": ingest_transcripts(a.dir) + elif a.cmd == "ingest-files": ingest_files(a.dir) + elif a.cmd == "ingest-events": ingest_events(a.db, a.sense) + elif a.cmd == "embed": embed(a.limit, a.batch) + elif a.cmd == "recall": + fn = brain_recall_hybrid if a.hybrid else brain_recall + for i, r in enumerate(fn(a.query, a.k), 1): + tag = f" via={r['via']} rrf={r['rrf']}" if "via" in r else "" + print(f"\n#{i} score={r['score']}{tag} [{r['source']}/{r['project']}] {r['ref']}") + print(" " + r["text"][:500].replace("\n", "\n ")) + elif a.cmd == "stats": stats() + +if __name__ == "__main__": + main() diff --git a/deploy/Dockerfile b/deploy/Dockerfile new file mode 100644 index 0000000..bc240a5 --- /dev/null +++ b/deploy/Dockerfile @@ -0,0 +1,18 @@ +# SecondBrain — engine image. Bring your own Ollama (set OLLAMA_URL). +FROM python:3.12-slim + +WORKDIR /opt/secondbrain +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt +COPY brain.py . +COPY deploy/ ./deploy/ + +# The db lives on a mounted volume so it survives container rebuilds. +ENV BRAIN_HOME=/data +VOLUME ["/data"] + +# Default: show stats. Override the command, e.g.: +# docker run --rm -v sbdata:/data -e OLLAMA_URL=http://host.docker.internal:11434 \ +# secondbrain recall "what did we decide about pricing" --hybrid +ENTRYPOINT ["python3", "brain.py"] +CMD ["stats"] diff --git a/deploy/install.sh b/deploy/install.sh new file mode 100755 index 0000000..00c9cdd --- /dev/null +++ b/deploy/install.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# SecondBrain — one-shot local install. Idempotent. +# curl -fsSL https://git.atlascorporation.nl/atlas/secondbrain/raw/branch/main/deploy/install.sh | bash +# or, from a clone: bash deploy/install.sh +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "$DIR" + +echo "==> python deps" +python3 -m venv .venv 2>/dev/null || true +. .venv/bin/activate +pip install -q --upgrade pip +pip install -q -r requirements.txt + +echo "==> config" +[ -f .env ] || cp .env.example .env +echo " edit .env to point OLLAMA_URL / sources" + +echo "==> ollama + embed model" +if command -v ollama >/dev/null 2>&1; then + ollama pull nomic-embed-text >/dev/null 2>&1 || true +else + echo " NOTE: ollama not found. Install from https://ollama.com and run: ollama pull nomic-embed-text" +fi + +echo "==> init schema" +python3 brain.py init + +cat <<'EOF' + +SecondBrain installed. Next: + . .venv/bin/activate + python3 brain.py ingest-files /path/to/your/notes-or-code + python3 brain.py embed + python3 brain.py recall "your question" --hybrid + +Continuous refresh (Linux): edit deploy/secondbrain-refresh.service paths, then + sudo cp deploy/secondbrain-refresh.* /etc/systemd/system/ + sudo systemctl enable --now secondbrain-refresh.timer +EOF diff --git a/deploy/refresh.sh b/deploy/refresh.sh new file mode 100755 index 0000000..ca135ee --- /dev/null +++ b/deploy/refresh.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# SecondBrain — continuous refresh (niced, locked, idempotent). +# Re-ingests your configured sources, re-scrubs, and embeds a bounded backlog. +# Configure via environment (or a .env next to brain.py): +# +# BRAIN_DIR dir containing brain.py (default: script's parent) +# BRAIN_HOME db location (default: ~/.secondbrain) +# SB_FILE_DIRS colon-separated dirs to ingest as files +# SB_TRANSCRIPTS dir of *.jsonl agent sessions (optional) +# SB_EVENTS_DB sqlite events db (optional) +# SB_EMBED_LIMIT chunks to embed per run (default: 6000) +# +# Run from cron/systemd-timer, e.g. every 15 min. +set -euo pipefail +BRAIN_DIR="${BRAIN_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" +BRAIN_HOME="${BRAIN_HOME:-$HOME/.secondbrain}" +PY="${SB_PYTHON:-python3}" +LOGDIR="$BRAIN_HOME/logs"; mkdir -p "$LOGDIR" +LOCK="$LOGDIR/.refresh.lock" + +exec 9>"$LOCK" +flock -n 9 || { echo "[refresh] $(date -Is) busy, skip" >> "$LOGDIR/refresh.log"; exit 0; } + +cd "$BRAIN_DIR" +[ -f .env ] && set -a && . ./.env && set +a +{ + echo "[refresh] $(date -Is) start" + $PY brain.py init + IFS=':' read -ra DIRS <<< "${SB_FILE_DIRS:-}" + for d in "${DIRS[@]}"; do [ -n "$d" ] && [ -d "$d" ] && $PY brain.py ingest-files "$d"; done + [ -n "${SB_TRANSCRIPTS:-}" ] && [ -d "$SB_TRANSCRIPTS" ] && $PY brain.py ingest-transcripts "$SB_TRANSCRIPTS" + [ -n "${SB_EVENTS_DB:-}" ] && [ -f "$SB_EVENTS_DB" ] && $PY brain.py ingest-events --db "$SB_EVENTS_DB" + $PY brain.py rescrub + nice -n 15 $PY brain.py embed --limit "${SB_EMBED_LIMIT:-6000}" --batch 64 + echo "[refresh] $(date -Is) done" +} >> "$LOGDIR/refresh.log" 2>&1 diff --git a/deploy/secondbrain-refresh.service b/deploy/secondbrain-refresh.service new file mode 100644 index 0000000..c7b861a --- /dev/null +++ b/deploy/secondbrain-refresh.service @@ -0,0 +1,11 @@ +[Unit] +Description=SecondBrain refresh (ingest + embed) +After=network-online.target + +[Service] +Type=oneshot +# Adjust WorkingDirectory to where brain.py lives, and EnvironmentFile to your .env +WorkingDirectory=/opt/secondbrain +EnvironmentFile=-/opt/secondbrain/.env +ExecStart=/usr/bin/env bash /opt/secondbrain/deploy/refresh.sh +Nice=15 diff --git a/deploy/secondbrain-refresh.timer b/deploy/secondbrain-refresh.timer new file mode 100644 index 0000000..1dcb70a --- /dev/null +++ b/deploy/secondbrain-refresh.timer @@ -0,0 +1,10 @@ +[Unit] +Description=Run SecondBrain refresh every 15 minutes + +[Timer] +OnBootSec=3min +OnUnitActiveSec=15min +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..a4f7b84 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,74 @@ +# SecondBrain — Architecture + +SecondBrain is deliberately small: one Python file (`brain.py`), one SQLite database, one +external dependency you run yourself (Ollama). This document explains the design decisions. + +## Data model + +A single SQLite file (`brain.db`) with: + +| Object | Role | +|---|---| +| `chunks` | one row per stored text segment: `sha` (dedup), `source`, `project`, `path`, `ref`, `ts`, `role`, `text`, `embedded`, `created` | +| `vec_chunks` | `sqlite-vec` virtual table holding the 768-dim embedding per chunk (`rowid` = `chunks.id`) | +| `fts_chunks` | FTS5 **external-content** index over `chunks.text` — stores only the BM25 index, not a second copy of the text (disk-safe). Kept in sync by triggers | +| `facts` | optional distilled facts (for an external consolidation loop) | + +Because `vec_chunks` and `fts_chunks` both key on `chunks.id`, the two retrieval channels can be +fused cheaply. + +## Ingestion pipeline + +``` +source text ─▶ scrub() ─▶ _segments() ─▶ _stage() ─▶ chunks (INSERT OR IGNORE by sha) +``` + +1. **`scrub()`** — mandatory. Applies a battery of regexes that redact secrets (PEM blocks, + JWTs, vendor key shapes, labelled `password/token/secret/api_key…`, URL-embedded creds, + `Company2026!`-style passwords). Optionally strips emails. **Nothing is stored un-scrubbed.** +2. **`_segments()`** — splits long text on paragraph boundaries into ≤ `MAX_CHARS` (~400-token) + chunks; drops sub-`MIN_CHARS` noise. +3. **`_stage()`** — computes a content `sha` for dedup and does `INSERT OR IGNORE`, so + re-ingesting the same source is a cheap no-op. + +Ingestion is **strictly read-only** — SecondBrain never writes back to your sources. + +## Embedding + +`embed()` selects `WHERE embedded=0` and calls Ollama's batch `/api/embed` (falling back to +per-chunk `/api/embeddings` on older Ollama). Vectors are **L2-normalized** so that vec0's L2 +distance ranking is equivalent to cosine similarity (`cos = 1 − d²/2`). A chunk that genuinely +can't be embedded is marked `embedded=-1` and skipped on subsequent runs (reset to `0` to retry). + +Embedding is idempotent and resumable — kill it any time; the next run continues the backlog. + +## Retrieval + +`brain_recall()` is pure dense KNN. `brain_recall_hybrid()` is the recommended path: + +1. Embed the query (`search_query:` prefix, per nomic's asymmetric convention). +2. **Dense channel** — top-`pool` by vector distance. +3. **Lexical channel** — FTS5 BM25 over a sanitized MATCH string (alphanumeric terms, each + quoted to neutralize FTS5 operators, OR-joined for recall). +4. **Reciprocal-rank-fusion** — each candidate scores `Σ 1/(rrf_k + rank_in_channel)` across the + channels it appears in; top-`k` by fused score. + +RRF needs no score calibration between the two very different channels, and the hybrid degrades +to dense-only when the lexical index is empty — so it is a safe drop-in for the dense function. + +Why hybrid matters: pure embeddings are great at *meaning* but weak at *exact tokens*. An invoice +number like `INV/2026/00037` or an IBAN often ranks poorly by cosine yet is an exact lexical hit. +Fusion recovers those without hurting semantic queries. + +## Configuration & portability + +Every path, endpoint and model is an environment variable with a local-first default +(`~/.secondbrain/brain.db`, `http://localhost:11434`, `nomic-embed-text`). Nothing is hard-coded +to a host. To move a brain between machines, copy `brain.db`. To offload compute, point +`OLLAMA_URL` at any reachable Ollama (e.g. a GPU box on your tailnet). + +## What SecondBrain is not + +- Not a chat UI — it's a retrieval library + CLI you wire into your own tools/agents. +- Not a reasoning loop — that lives in sister components (see the wiki). +- Not multi-tenant — it's a *personal* brain; run one per person/box. diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..6dcf501 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,3 @@ +sqlite-vec>=0.1.6 +requests>=2.31 +numpy>=1.24