atlas/secondbrain
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

SecondBrain v0.1 — local-first, privacy-scrubbing RAG memory

Browse source 
Generalized + redeployable: config via env, local Ollama embeddings, single
portable sqlite-vec db, hybrid dense+FTS5 recall, mandatory credential scrub.
This commit is contained in:
Atlas 2026-07-01 21:25:02 +02:00
commit 84afdc1f56
12 changed files with 943 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

22
.env.example Normal file
View file

@ -0,0 +1,22 @@
# SecondBrain configuration — copy to `.env` and adjust. All values optional.
# `.env` is git-ignored; never commit real values.
# Where the single portable database lives.
BRAIN_HOME=~/.secondbrain
# BRAIN_DB=/custom/path/brain.db # overrides BRAIN_HOME for the db file
# Local embedding backend (Ollama). Point this at any reachable Ollama:
# local: http://localhost:11434
# a GPU box on your LAN / tailnet: http://100.x.y.z:11434
OLLAMA_URL=http://localhost:11434
EMBED_MODEL=nomic-embed-text
EMBED_DIM=768
# Privacy: 1 = keep emails/IPs/hostnames (operational knowledge),
# 0 = also strip emails. Secrets are ALWAYS scrubbed regardless.
SCRUB_KEEP_PII=1
# Chunking (advanced; defaults are sensible)
# BRAIN_MAX_CHARS=1600
# BRAIN_MIN_CHARS=40
# BRAIN_MAX_FILE_BYTES=400000

34
.gitignore vendored Normal file
View file

@ -0,0 +1,34 @@
# ── data & secrets — NEVER commit ──────────────────────────────
*.db
*.db-wal
*.db-shm
*.sqlite
*.sqlite3
data/
store/
corpus/
logs/
*.log
.env
.env.*
!.env.example
secrets/
*.age
*.pem
*.key
# ── python ─────────────────────────────────────────────────────
__pycache__/
*.py[cod]
.venv/
venv/
env/
*.egg-info/
.pytest_cache/
.mypy_cache/
# ── os / editor ────────────────────────────────────────────────
.DS_Store
Thumbs.db
.idea/
.vscode/

21
LICENSE Normal file
View file

@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Atlas Corporation
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

155
README.md Normal file
View file

@ -0,0 +1,155 @@
<div align="center">
# 🧠 SecondBrain
**A local-first, privacy-scrubbing personal memory you can point at everything you produce.**
*Notes · code · agent transcripts · event logs → one portable file → instant hybrid recall.*
*No cloud. No API keys. No data egress. Secrets scrubbed before anything is stored.*
</div>
---
SecondBrain is a tiny (~500-line, single-file) knowledge engine. You feed it the things you
already generate — documents, source code, AI-agent chat transcripts, structured event logs —
and it gives you back one command:
```bash
brain recall "what did we decide about the pricing model" --hybrid
```
Everything runs on your own machine. Embeddings are computed locally by
[Ollama](https://ollama.com) (`nomic-embed-text`), and the entire brain is a **single
portable `brain.db`** (SQLite + [`sqlite-vec`](https://github.com/asg017/sqlite-vec)) you can
copy to a USB stick. There is no server to run, no account to create, and **no credential
ever leaves your box** — because SecondBrain redacts them *before* embedding.
## Why it exists
Vector-RAG demos are easy; a memory you'd actually trust with your real life is not. SecondBrain
is opinionated about the three things that usually break:
| Problem | SecondBrain's answer |
|---|---|
| **Secrets leak into your index** (and then into an LLM's context) | A mandatory scrub pass strips PEM keys, JWTs, `sk-ant-…`/`sk-…`/`ghp_…`/AWS keys, and any `password: / token= / bearer …` pattern **before** a chunk is stored. IPs, hostnames and emails are kept as operational knowledge (toggle with `SCRUB_KEEP_PII=0`). |
| **Dense-only search misses exact strings** (invoice numbers, IBANs, container names, file paths) | Hybrid recall fuses dense KNN with FTS5/BM25 lexical search via **reciprocal-rank-fusion** — meaning *and* identifiers both hit. |
| **Cloud RAG = data egress + lock-in + cost** | 100% local. Ollama for embeddings, SQLite for storage. Portable, free, offline-capable, redeployable on any device in minutes. |
## The senses
SecondBrain ingests from pluggable "senses" — each strictly **read-only**:
- **`ingest-files <dir>`** — code, docs and config (60+ text extensions; skips `node_modules`, `.git`, binaries, and files > 400 KB).
- **`ingest-transcripts <dir>`** — `*.jsonl` AI-agent sessions (Claude Code–style event logs); tool spam is collapsed to signal.
- **`ingest-events --db <sqlite>`** — any SQLite with an `events(id, ts, sense, subject, payload)` table (e.g. email, chat, calendar, finance). Turns real activity into recallable memory.
Add your own sense in ~10 lines: chunk → `scrub()` → `_stage()`. That's the whole contract.
## Quickstart
```bash
# 0. prerequisites: python3, and Ollama running (https://ollama.com)
ollama pull nomic-embed-text
# 1. install
git clone https://git.atlascorporation.nl/atlas/secondbrain
cd secondbrain
bash deploy/install.sh # venv + deps + schema
. .venv/bin/activate
# 2. feed it something
python3 brain.py ingest-files ~/notes
python3 brain.py ingest-files ~/code/my-project
# 3. embed (local, free)
python3 brain.py embed
# 4. recall
python3 brain.py recall "how does the auth flow work" --hybrid
python3 brain.py stats
```
Output:
```
#1 score=0.72 via=both rrf=0.0312 [file/my-project] auth.py
def login(user, pw): # verifies against argon2 hash, issues a signed …
```
## Configuration
All via environment (or a `.env` next to `brain.py`). Copy `.env.example` → `.env`:
| Var | Default | Meaning |
|---|---|---|
| `BRAIN_HOME` | `~/.secondbrain` | where `brain.db` lives |
| `BRAIN_DB` | `$BRAIN_HOME/brain.db` | explicit db path (overrides `BRAIN_HOME`) |
| `OLLAMA_URL` | `http://localhost:11434` | any reachable Ollama — including a **GPU box on your LAN/tailnet** |
| `EMBED_MODEL` | `nomic-embed-text` | embedding model |
| `EMBED_DIM` | `768` | must match the model |
| `SCRUB_KEEP_PII` | `1` | `0` also strips emails (secrets always scrubbed) |
> **Tip — offload embeddings to a GPU:** point `OLLAMA_URL` at another machine's Ollama
> (`http://100.x.y.z:11434`). SecondBrain will embed there and store locally. This is how you
> keep a laptop's memory current using a desktop GPU, with zero extra infrastructure.
## Deploy anywhere
**systemd (continuous refresh):** set your sources in `.env` (`SB_FILE_DIRS`, `SB_TRANSCRIPTS`,
`SB_EVENTS_DB`), then:
```bash
sudo cp deploy/secondbrain-refresh.* /etc/systemd/system/
sudo systemctl enable --now secondbrain-refresh.timer # ingest+embed every 15 min
```
**Docker:**
```bash
docker build -t secondbrain -f deploy/Dockerfile .
docker run --rm -v sbdata:/data -e OLLAMA_URL=http://host.docker.internal:11434 \
secondbrain recall "quarterly numbers" --hybrid
```
## Use it as a library
```python
import brain
for hit in brain.brain_recall_hybrid("open invoices for June", k=8):
print(hit["score"], hit["ref"], hit["text"][:120])
```
`brain_recall_hybrid()` degrades gracefully to dense-only if the lexical index is empty, so it's
a safe drop-in for `brain_recall()`.
## How it works
```
sources ──▶ scrub() ──▶ chunk ──▶ chunks table ──┬─▶ FTS5 (BM25, via triggers)
(senses) (redact) (≤1600 ch) └─▶ vec_chunks (nomic 768-dim, Ollama)
│
recall("q") ──▶ embed query ──▶ dense KNN ┐ │
└▶ FTS5 lexical ─────────────┴─ RRF fuse ─┴─▶ ranked hits
```
See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the full design, the scrub guarantees,
and the reciprocal-rank-fusion math.
## Privacy & security
- **Scrub-before-store** is not optional and runs on every chunk from every sense. `rescrub`
re-applies a hardened scrub to an existing db in place.
- The database **never leaves your machine** unless you copy it. `.gitignore` blocks `*.db`,
`.env`, `secrets/`, `*.age`, `*.pem`, `*.key` so you can't accidentally commit data.
- No telemetry. No network calls except to your configured Ollama endpoint.
## Related components
SecondBrain is the memory layer of a larger local-first autonomy stack. Sister components
(separate repos) include the **policy-gate** (an approval brake that parks money/comms/
irreversible actions for a human) and fleet/ingest tooling. See the wiki.
## License
MIT © 2026 Atlas Corporation. Contributions welcome.

519
brain.py Executable file
View file

@ -0,0 +1,519 @@
#!/usr/bin/env python3
"""
SecondBrain — a local-first, privacy-scrubbing personal RAG memory
==================================================================
An additive knowledge layer you can point at anything you already produce —
notes, code, chat/agent transcripts, structured event logs — that:
* SCRUBS credentials (keys/tokens/passwords/PEM/JWT) out of every chunk before
it is ever stored or embedded — privacy is the default, not an add-on;
* embeds locally with Ollama (nomic-embed-text, 768-dim) — no cloud, no API
keys, no data egress;
* stores everything in a single portable sqlite-vec file (`brain.db`);
* answers `recall("...")` with hybrid retrieval — dense KNN + FTS5/BM25 fused
by reciprocal-rank-fusion, so both meaning ("what did we decide about X")
and exact identifiers (invoice numbers, IBANs, container names, paths) hit.
It never writes back to your sources — ingestion is strictly read-only.
Configuration is entirely via environment variables (see `.env.example`):
BRAIN_HOME base dir for the DB (default: ~/.secondbrain)
BRAIN_DB explicit db path (default: $BRAIN_HOME/brain.db)
OLLAMA_URL Ollama endpoint (default: http://localhost:11434)
EMBED_MODEL embedding model (default: nomic-embed-text)
EMBED_DIM embedding dimension (default: 768)
SCRUB_KEEP_PII 1=keep emails, 0=strip (default: 1)
The "senses" (ingestion sources):
ingest-files <dir> code / docs / config (60+ text extensions)
ingest-transcripts <dir> *.jsonl agent sessions (Claude Code style)
ingest-events --db <sqlite> structured events table (email/chat/finance/...)
Then:
embed [--limit N --batch B] embed the un-embedded backlog (local)
recall "<query>" [--hybrid] retrieve (dense, or dense+lexical fused)
stats counts + coverage
License: MIT. Project home: https://git.atlascorporation.nl/atlas/secondbrain
"""
import os, sys, json, re, sqlite3, hashlib, time, glob, argparse
import sqlite_vec
import requests
# ----------------------------------------------------------------------------
# Config — all via environment, with sane local-first defaults
# ----------------------------------------------------------------------------
BRAIN_HOME = os.environ.get("BRAIN_HOME", os.path.expanduser("~/.secondbrain"))
DB_PATH = os.environ.get("BRAIN_DB", os.path.join(BRAIN_HOME, "brain.db"))
OLLAMA = os.environ.get("OLLAMA_URL", "http://localhost:11434")
EMBED_MODEL = os.environ.get("EMBED_MODEL", "nomic-embed-text")
DIM = int(os.environ.get("EMBED_DIM", "768"))
MAX_CHARS = int(os.environ.get("BRAIN_MAX_CHARS", "1600")) # ~400 tokens per chunk
MIN_CHARS = int(os.environ.get("BRAIN_MIN_CHARS", "40")) # drop trivially short chunks
MAX_FILE = int(os.environ.get("BRAIN_MAX_FILE_BYTES", "400000"))
# ----------------------------------------------------------------------------
# CREDENTIAL SCRUB (mandatory before any text is embedded/stored)
# Strategy: remove SECRETS (keys/tokens/passwords/JWT/PEM), but KEEP IPs,
# hostnames and (by default) emails — those are operational knowledge the brain
# needs. Set SCRUB_KEEP_PII=0 to also strip emails.
# ----------------------------------------------------------------------------
KEEP_PII = os.environ.get("SCRUB_KEEP_PII", "1") == "1"
_SCRUB = [
# PEM private key blocks (do first, multi-line)
(re.compile(r'-----BEGIN [A-Z ]*PRIVATE KEY-----.*?-----END [A-Z ]*PRIVATE KEY-----', re.S), '<PRIVATE_KEY>'),
# labelled secrets: password: xxx DB_PASSWORD=xxx "token": "xxx" bearer xxx
# NOTE: [A-Za-z_]* prefix catches DB_PASSWORD / MYSQL_ROOT_PASSWORD / etc. (the '_' blocks a plain \b)
(re.compile(r'(?i)([A-Za-z_]*(?:password|passwd|passphrase|pwd|secret|api[_-]?key|apikey|access[_-]?token|auth[_-]?token|token|bearer|client[_-]?secret|private[_-]?key))\b\s*["\']?\s*(?:is|=|:)\s*["\']?([^\s"\'<>]{3,})'), '\\1=<REDACTED>'),
# credentials embedded in URLs / connection strings: scheme://user:PASS@host
(re.compile(r'(://[^/\s:@]+:)([^@\s/]{3,})(@)'), '\\1<REDACTED>\\3'),
# user@host/PASS or user:PASS pairs
(re.compile(r'([A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+[/:])([A-Za-z0-9][A-Za-z0-9._@#%!-]{7,})'), '\\1<REDACTED>'),
# generic "<word>20YY!" password shapes (e.g. Company2026!)
(re.compile(r'\b[A-Za-z][A-Za-z0-9_.-]{2,}20\d\d[!@#%]'), '<SECRET>'),
# vendor key shapes
(re.compile(r'\bgsk_[A-Za-z0-9]{20,}'), '<GROQ_KEY>'),
(re.compile(r'\bsk-ant-[A-Za-z0-9_-]{20,}'), '<ANTHROPIC_KEY>'),
(re.compile(r'\bsk-[A-Za-z0-9]{20,}'), '<OPENAI_KEY>'),
(re.compile(r'\bAKIA[0-9A-Z]{16}\b'), '<AWS_KEY>'),
(re.compile(r'\bghp_[A-Za-z0-9]{30,}'), '<GH_TOKEN>'),
(re.compile(r'\bgithub_pat_[A-Za-z0-9_]{30,}'), '<GH_PAT>'),
(re.compile(r'\bxox[baprs]-[A-Za-z0-9-]{10,}'), '<SLACK_TOKEN>'),
# JWT
(re.compile(r'\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{6,}'), '<JWT>'),
]
_SCRUB_PII = [
(re.compile(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b'), '<EMAIL>'),
]
def scrub(text: str) -> str:
if not text:
return ""
for pat, repl in _SCRUB:
text = pat.sub(repl, text)
if not KEEP_PII:
for pat, repl in _SCRUB_PII:
text = pat.sub(repl, text)
return text
# ----------------------------------------------------------------------------
# DB
# ----------------------------------------------------------------------------
def db():
os.makedirs(os.path.dirname(DB_PATH) or ".", exist_ok=True)
conn = sqlite3.connect(DB_PATH)
conn.enable_load_extension(True)
sqlite_vec.load(conn)
conn.enable_load_extension(False)
return conn
def init_schema():
conn = db()
conn.execute("""CREATE TABLE IF NOT EXISTS chunks(
id INTEGER PRIMARY KEY,
sha TEXT UNIQUE, -- dedup key
source TEXT, -- 'file' | 'transcript' | 'events'
project TEXT,
path TEXT,
ref TEXT, -- session id / symbol
ts TEXT,
role TEXT,
text TEXT,
embedded INTEGER DEFAULT 0,
created REAL
)""")
conn.execute("CREATE INDEX IF NOT EXISTS ix_chunks_emb ON chunks(embedded)")
conn.execute(f"CREATE VIRTUAL TABLE IF NOT EXISTS vec_chunks USING vec0(emb float[{DIM}])")
# optional learning layer: durable facts distilled by an external consolidate loop
conn.execute("""CREATE TABLE IF NOT EXISTS facts(
id INTEGER PRIMARY KEY, fact TEXT UNIQUE, topic TEXT, weight REAL DEFAULT 1.0,
provenance TEXT, created REAL)""")
# lexical channel (hybrid recall): FTS5 over chunks.text.
# external-content => stores ONLY the BM25 index, NOT a 2nd copy of the text
# (disk-safe). Triggers keep it in sync with chunks.
conn.execute("CREATE VIRTUAL TABLE IF NOT EXISTS fts_chunks USING fts5("
"text, content='chunks', content_rowid='id', tokenize='unicode61')")
conn.execute("""CREATE TRIGGER IF NOT EXISTS chunks_ai AFTER INSERT ON chunks BEGIN
INSERT INTO fts_chunks(rowid, text) VALUES (new.id, new.text); END""")
conn.execute("""CREATE TRIGGER IF NOT EXISTS chunks_ad AFTER DELETE ON chunks BEGIN
INSERT INTO fts_chunks(fts_chunks, rowid, text) VALUES('delete', old.id, old.text); END""")
conn.execute("""CREATE TRIGGER IF NOT EXISTS chunks_au AFTER UPDATE ON chunks BEGIN
INSERT INTO fts_chunks(fts_chunks, rowid, text) VALUES('delete', old.id, old.text);
INSERT INTO fts_chunks(rowid, text) VALUES (new.id, new.text); END""")
conn.commit(); conn.close()
print(f"[init] schema ready at {DB_PATH}")
def build_fts():
"""(Re)build the FTS5 lexical index from existing chunks. Idempotent.
Needed once for a DB that pre-dates the FTS table; triggers keep it fresh after."""
init_schema()
conn = db()
conn.execute("INSERT INTO fts_chunks(fts_chunks) VALUES('rebuild')")
conn.commit()
n = conn.execute("SELECT count(*) FROM fts_chunks").fetchone()[0]
conn.close()
print(f"[build-fts] lexical index rebuilt: {n} rows")
# ----------------------------------------------------------------------------
# chunk helpers
# ----------------------------------------------------------------------------
def _sha(*parts) -> str:
return hashlib.sha1("\x1f".join(p or "" for p in parts).encode("utf-8", "ignore")).hexdigest()
def _segments(text: str):
"""Split a long blob into <= MAX_CHARS segments on paragraph boundaries."""
text = text.strip()
if len(text) <= MAX_CHARS:
if len(text) >= MIN_CHARS:
yield text
return
buf = ""
for para in re.split(r'\n\s*\n', text):
if len(buf) + len(para) + 2 > MAX_CHARS:
if len(buf) >= MIN_CHARS:
yield buf.strip()
buf = para
else:
buf += "\n\n" + para
if len(buf.strip()) >= MIN_CHARS:
yield buf.strip()
def _stage(conn, source, project, path, ref, ts, role, text):
text = scrub(text)
n = 0
for seg in _segments(text):
sha = _sha(source, path, ref, seg[:120], str(n))
try:
conn.execute(
"INSERT OR IGNORE INTO chunks(sha,source,project,path,ref,ts,role,text,created)"
" VALUES(?,?,?,?,?,?,?,?,?)",
(sha, source, project, path, ref, ts, role, seg, time.time()))
if conn.total_changes:
n += 1
except Exception:
pass
return n
# ----------------------------------------------------------------------------
# SENSE: transcripts (agent .jsonl: one JSON event per line, Claude Code style)
# ----------------------------------------------------------------------------
def _event_text(ev):
"""Reduce a transcript event to signal text + role. Tool spam collapsed."""
role = ev.get("type") or (ev.get("message") or {}).get("role") or ""
msg = ev.get("message") or ev
content = msg.get("content") if isinstance(msg, dict) else None
out = []
if isinstance(content, str):
out.append(content)
elif isinstance(content, list):
for b in content:
if not isinstance(b, dict):
continue
t = b.get("type")
if t == "text" and b.get("text"):
out.append(b["text"])
elif t == "tool_use":
inp = b.get("input") or {}
desc = inp.get("description") or inp.get("command") or inp.get("file_path") or ""
out.append(f"[tool:{b.get('name','?')}] {str(desc)[:160]}")
elif t == "tool_result":
c = b.get("content")
if isinstance(c, list):
c = " ".join(x.get("text","") for x in c if isinstance(x, dict))
c = str(c or "")
if len(c) > 300:
c = c[:200] + " … " + c[-80:]
out.append(f"[result] {c}")
return role, "\n".join(s for s in out if s).strip()
def ingest_transcripts(root):
conn = db()
files = glob.glob(os.path.join(root, "**", "*.jsonl"), recursive=True)
print(f"[ingest-transcripts] {len(files)} session files under {root}")
total = 0
for i, fp in enumerate(files):
project = os.path.basename(os.path.dirname(fp))
sess = os.path.basename(fp).replace(".jsonl", "")
buf, last_ts = [], ""
try:
with open(fp, "r", encoding="utf-8", errors="ignore") as fh:
for line in fh:
line = line.strip()
if not line:
continue
try:
ev = json.loads(line)
except Exception:
continue
ts = ev.get("timestamp") or ev.get("ts") or ""
role, txt = _event_text(ev)
if not txt:
continue
last_ts = ts or last_ts
buf.append(f"{role}: {txt}")
if sum(len(x) for x in buf) > MAX_CHARS:
total += _stage(conn, "transcript", project, fp, sess, last_ts, "exchange", "\n".join(buf))
buf = []
except Exception as e:
print(f" ! {fp}: {e}")
continue
if buf:
total += _stage(conn, "transcript", project, fp, sess, last_ts, "exchange", "\n".join(buf))
if (i+1) % 100 == 0:
conn.commit(); print(f" .. {i+1}/{len(files)} files, {total} chunks")
conn.commit(); conn.close()
print(f"[ingest-transcripts] staged {total} new chunks")
# ----------------------------------------------------------------------------
# SENSE: files / code / docs
# ----------------------------------------------------------------------------
FILE_GLOBS = ("*.py","*.sh","*.md","*.markdown","*.txt","*.rst","*.yml","*.yaml","*.toml","*.ini","*.cfg","*.conf","*.env","*.properties","*.json","*.jsonl","*.xml","*.csv","*.tsv","*.sql","*.js","*.mjs","*.cjs","*.ts","*.tsx","*.jsx","*.vue","*.svelte","*.html","*.htm","*.css","*.scss","*.sass","*.less","*.ps1","*.psm1","*.bat","*.cmd","*.go","*.rs","*.java","*.kt","*.c","*.h","*.cpp","*.hpp","*.cs","*.rb","*.php","*.pl","*.lua","*.r","*.R","*.swift","*.scala","*.service","*.timer","*.socket","*.ipynb","*.tf","*.hcl","*.gitignore","*.editorconfig")
SKIP_DIRS = (".git","node_modules","venv",".venv","__pycache__","vendor",".obsidian","dist","build",".next",".cache")
def ingest_files(root):
conn = db()
n_files = total = 0
for dirpath, dirs, names in os.walk(root):
dirs[:] = [d for d in dirs if d not in SKIP_DIRS]
for name in names:
if not any(glob.fnmatch.fnmatch(name, g) for g in FILE_GLOBS):
continue
fp = os.path.join(dirpath, name)
try:
if os.path.getsize(fp) > MAX_FILE:
continue
with open(fp, "r", encoding="utf-8", errors="ignore") as fh:
txt = fh.read()
except Exception:
continue
n_files += 1
total += _stage(conn, "file", os.path.basename(root), fp, name, "", "code", txt)
conn.commit(); conn.close()
print(f"[ingest-files] {n_files} files -> staged {total} new chunks")
# ----------------------------------------------------------------------------
# SENSE: structured events (any sqlite with an events table)
# Expected schema (columns; extra columns ignored):
# events(id, ts, sense, subject, payload) -- payload = JSON string
# `sense` is a free-text channel label (e.g. email, chat, finance, calendar).
# This lets the brain answer questions about real activity, not just documents.
# ----------------------------------------------------------------------------
def ingest_events(events_db, senses=None):
import sqlite3 as _sq, json as _json
conn = db()
k = _sq.connect(f"file:{events_db}?mode=ro", uri=True)
if senses:
placeholders = ",".join("?" for _ in senses)
rows = k.execute(f"SELECT id, ts, sense, subject, payload FROM events "
f"WHERE sense IN ({placeholders}) ORDER BY id", tuple(senses)).fetchall()
else:
rows = k.execute("SELECT id, ts, sense, subject, payload FROM events ORDER BY id").fetchall()
total = 0
for eid, ts, sense, subject, payload in rows:
try: p = _json.loads(payload or "{}")
except Exception: p = {}
ref = p.get("from") or p.get("from_name") or p.get("counterparty") or sense or "?"
body = p.get("text") or p.get("info") or p.get("body") or ""
txt = f"[{sense}] {subject or ''} {body}".strip()
if txt:
total += _stage(conn, "events", sense, f"event-{eid}", str(ref)[:80], ts, sense, txt)
conn.commit(); conn.close(); k.close()
print(f"[ingest-events] {len(rows)} events -> staged {total} new chunks")
# ----------------------------------------------------------------------------
# embedding (local nomic-embed-text via Ollama)
# ----------------------------------------------------------------------------
import numpy as np
def _l2norm(v):
a = np.asarray(v, dtype=np.float32); n = float(np.linalg.norm(a))
return (a / n) if n else a # unit vector -> L2 ranking == cosine ranking
EMBED_CAP = int(os.environ.get("EMBED_CHAR_CAP", "6000")) # ~1500 tok
def _embed_one(text, prefix="search_document: "):
r = requests.post(f"{OLLAMA}/api/embeddings",
json={"model": EMBED_MODEL, "prompt": prefix + text, "keep_alive": "10m"}, timeout=120)
r.raise_for_status()
return _l2norm(r.json()["embedding"]).tolist()
def _embed_batch(texts, prefix="search_document: "):
"""Batch embed via /api/embed; falls back to the single endpoint per-chunk."""
try:
r = requests.post(f"{OLLAMA}/api/embed",
json={"model": EMBED_MODEL, "input": [prefix + (t or "").replace(chr(0), " ")[:EMBED_CAP] for t in texts], "keep_alive": "10m"},
timeout=600)
if r.status_code == 404:
raise RuntimeError("no /api/embed")
r.raise_for_status()
return [_l2norm(e) for e in r.json().get("embeddings", [])]
except Exception:
return [_l2norm(_emb_raw(t, prefix)) for t in texts]
def _emb_raw(text, prefix):
r = requests.post(f"{OLLAMA}/api/embeddings",
json={"model": EMBED_MODEL, "prompt": prefix + (text or "").replace(chr(0), " ")[:EMBED_CAP], "keep_alive": "10m"}, timeout=120)
r.raise_for_status(); return r.json()["embedding"]
def embed(limit=None, batch=48):
conn = db()
q = "SELECT id,text FROM chunks WHERE embedded=0 ORDER BY id"
if limit:
q += f" LIMIT {int(limit)}"
rows = conn.execute(q).fetchall()
print(f"[embed] {len(rows)} chunks to embed via {EMBED_MODEL} (batch={batch})")
done = 0; t0 = time.time()
for i in range(0, len(rows), batch):
grp = rows[i:i+batch]
try:
vecs = _embed_batch([t for _, t in grp])
for (cid, _), v in zip(grp, vecs):
conn.execute("INSERT OR REPLACE INTO vec_chunks(rowid,emb) VALUES(?,?)",
(cid, sqlite_vec.serialize_float32(v)))
conn.execute("UPDATE chunks SET embedded=1 WHERE id=?", (cid,))
conn.commit(); done += len(grp)
rate = done/(time.time()-t0+1e-9)
print(f" .. {done}/{len(rows)} {rate:.1f}/s")
except Exception:
bad = 0
for cid, t in grp:
try:
v = _l2norm(_emb_raw(t or "", "search_document: "))
conn.execute("INSERT OR REPLACE INTO vec_chunks(rowid,emb) VALUES(?,?)",
(cid, sqlite_vec.serialize_float32(v)))
conn.execute("UPDATE chunks SET embedded=1 WHERE id=?", (cid,))
done += 1
except Exception:
conn.execute("UPDATE chunks SET embedded=-1 WHERE id=?", (cid,))
bad += 1
conn.commit()
print(f" ~ batch@{i} fell back per-chunk: {bad} unembeddable (embedded=-1)")
conn.commit(); conn.close()
print(f"[embed] embedded {done} chunks in {time.time()-t0:.0f}s")
# ----------------------------------------------------------------------------
# recall
# ----------------------------------------------------------------------------
def brain_recall(query, k=6):
conn = db()
qv = _embed_one(query, prefix="search_query: ")
rows = conn.execute("""
SELECT c.project, c.source, c.path, c.ref, c.ts, c.text, v.distance
FROM vec_chunks v JOIN chunks c ON c.id = v.rowid
WHERE v.emb MATCH ? AND k = ?
ORDER BY v.distance
""", (sqlite_vec.serialize_float32(qv), k)).fetchall()
conn.close()
return [{"project":p,"source":s,"path":pa,"ref":rf,"ts":ts,
"score":round(1-(d*d)/2,3),"text":tx} for (p,s,pa,rf,ts,tx,d) in rows]
def _fts_sanitize(q):
"""Turn arbitrary user text into a safe FTS5 MATCH string: alphanumeric
terms, each quoted (neutralizes FTS5 operators), joined by OR for recall."""
terms = [t for t in re.findall(r"[A-Za-z0-9_]+", (q or "")) if len(t) > 1]
return " OR ".join(f'"{t}"' for t in terms)
def brain_recall_hybrid(query, k=6, pool=50, rrf_k=60):
"""Hybrid recall: dense KNN + FTS5 BM25 fused by reciprocal-rank-fusion.
Same dict shape as brain_recall plus 'rrf' and 'via'. Degrades to dense-only
if the lexical channel is empty, so it is a safe drop-in for brain_recall."""
conn = db()
qv = _embed_one(query, prefix="search_query: ")
qblob = sqlite_vec.serialize_float32(qv)
dense = conn.execute(
"SELECT v.rowid, v.distance FROM vec_chunks v WHERE v.emb MATCH ? AND k = ? ORDER BY v.distance",
(qblob, pool)).fetchall()
dense_rank = {cid: i for i, (cid, _) in enumerate(dense)}
dense_cos = {cid: 1-(d*d)/2 for cid, d in dense}
lex_rank = {}
m = _fts_sanitize(query)
if m:
try:
lex = conn.execute(
"SELECT rowid FROM fts_chunks WHERE fts_chunks MATCH ? ORDER BY rank LIMIT ?",
(m, pool)).fetchall()
lex_rank = {cid: i for i, (cid,) in enumerate(lex)}
except Exception:
lex_rank = {}
ids = set(dense_rank) | set(lex_rank)
fused = {}
for cid in ids:
s = 0.0
if cid in dense_rank: s += 1.0/(rrf_k + dense_rank[cid])
if cid in lex_rank: s += 1.0/(rrf_k + lex_rank[cid])
fused[cid] = s
top = sorted(ids, key=lambda c: fused[c], reverse=True)[:k]
out = []
for cid in top:
row = conn.execute("SELECT project,source,path,ref,ts,text FROM chunks WHERE id=?", (cid,)).fetchone()
if not row:
continue
p, s, pa, rf, ts, tx = row
cos = dense_cos.get(cid)
if cos is None:
try:
d = conn.execute("SELECT vec_distance_L2(emb, ?) FROM vec_chunks WHERE rowid=?",
(qblob, cid)).fetchone()
cos = 1-(d[0]*d[0])/2 if d and d[0] is not None else 0.0
except Exception:
cos = 0.0
via = "both" if (cid in dense_rank and cid in lex_rank) else ("dense" if cid in dense_rank else "lexical")
out.append({"project":p,"source":s,"path":pa,"ref":rf,"ts":ts,
"score":round(cos,3),"rrf":round(fused[cid],4),"via":via,"text":tx})
conn.close()
return out
def rescrub():
"""Re-apply the (possibly hardened) scrub to every staged chunk in place."""
conn = db()
rows = conn.execute("SELECT id,text FROM chunks").fetchall()
changed = 0
for cid, text in rows:
s = scrub(text)
if s != text:
conn.execute("UPDATE chunks SET text=? WHERE id=?", (s, cid))
changed += 1
conn.commit(); conn.close()
print(f"[rescrub] re-scrubbed {changed}/{len(rows)} chunks")
def stats():
conn = db()
tot = conn.execute("SELECT count(*) FROM chunks").fetchone()[0]
emb = conn.execute("SELECT count(*) FROM chunks WHERE embedded=1").fetchone()[0]
bysrc = conn.execute("SELECT source,count(*) FROM chunks GROUP BY source").fetchall()
proj = conn.execute("SELECT project,count(*) c FROM chunks GROUP BY project ORDER BY c DESC LIMIT 10").fetchall()
facts = conn.execute("SELECT count(*) FROM facts").fetchone()[0]
conn.close()
print(f"chunks: {tot} embedded: {emb} facts: {facts}")
print("by source:", dict(bysrc))
print("top projects:", proj)
# ----------------------------------------------------------------------------
def main():
ap = argparse.ArgumentParser(prog="brain", description="SecondBrain — local-first RAG memory")
sub = ap.add_subparsers(dest="cmd", required=True)
sub.add_parser("init")
sub.add_parser("build-fts")
p = sub.add_parser("ingest-transcripts"); p.add_argument("dir")
p = sub.add_parser("ingest-files"); p.add_argument("dir")
p = sub.add_parser("ingest-events"); p.add_argument("--db", required=True); p.add_argument("--sense", action="append", help="restrict to these sense labels")
p = sub.add_parser("embed"); p.add_argument("--limit", type=int); p.add_argument("--batch", type=int, default=64)
p = sub.add_parser("recall"); p.add_argument("query"); p.add_argument("-k", type=int, default=6)
p.add_argument("--hybrid", action="store_true", help="dense + FTS5 lexical, RRF-fused")
sub.add_parser("rescrub")
sub.add_parser("stats")
a = ap.parse_args()
if a.cmd == "init": init_schema()
elif a.cmd == "build-fts": build_fts()
elif a.cmd == "rescrub": rescrub()
elif a.cmd == "ingest-transcripts": ingest_transcripts(a.dir)
elif a.cmd == "ingest-files": ingest_files(a.dir)
elif a.cmd == "ingest-events": ingest_events(a.db, a.sense)
elif a.cmd == "embed": embed(a.limit, a.batch)
elif a.cmd == "recall":
fn = brain_recall_hybrid if a.hybrid else brain_recall
for i, r in enumerate(fn(a.query, a.k), 1):
tag = f" via={r['via']} rrf={r['rrf']}" if "via" in r else ""
print(f"\n#{i} score={r['score']}{tag} [{r['source']}/{r['project']}] {r['ref']}")
print(" " + r["text"][:500].replace("\n", "\n "))
elif a.cmd == "stats": stats()
if __name__ == "__main__":
main()

18
deploy/Dockerfile Normal file
View file

@ -0,0 +1,18 @@
# SecondBrain — engine image. Bring your own Ollama (set OLLAMA_URL).
FROM python:3.12-slim
WORKDIR /opt/secondbrain
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY brain.py .
COPY deploy/ ./deploy/
# The db lives on a mounted volume so it survives container rebuilds.
ENV BRAIN_HOME=/data
VOLUME ["/data"]
# Default: show stats. Override the command, e.g.:
# docker run --rm -v sbdata:/data -e OLLAMA_URL=http://host.docker.internal:11434 \
# secondbrain recall "what did we decide about pricing" --hybrid
ENTRYPOINT ["python3", "brain.py"]
CMD ["stats"]

40
deploy/install.sh Executable file
View file

@ -0,0 +1,40 @@
#!/usr/bin/env bash
# SecondBrain — one-shot local install. Idempotent.
# curl -fsSL https://git.atlascorporation.nl/atlas/secondbrain/raw/branch/main/deploy/install.sh | bash
# or, from a clone: bash deploy/install.sh
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$DIR"
echo "==> python deps"
python3 -m venv .venv 2>/dev/null || true
. .venv/bin/activate
pip install -q --upgrade pip
pip install -q -r requirements.txt
echo "==> config"
[ -f .env ] || cp .env.example .env
echo " edit .env to point OLLAMA_URL / sources"
echo "==> ollama + embed model"
if command -v ollama >/dev/null 2>&1; then
ollama pull nomic-embed-text >/dev/null 2>&1 || true
else
echo " NOTE: ollama not found. Install from https://ollama.com and run: ollama pull nomic-embed-text"
fi
echo "==> init schema"
python3 brain.py init
cat <<'EOF'
SecondBrain installed. Next:
. .venv/bin/activate
python3 brain.py ingest-files /path/to/your/notes-or-code
python3 brain.py embed
python3 brain.py recall "your question" --hybrid
Continuous refresh (Linux): edit deploy/secondbrain-refresh.service paths, then
sudo cp deploy/secondbrain-refresh.* /etc/systemd/system/
sudo systemctl enable --now secondbrain-refresh.timer
EOF

36
deploy/refresh.sh Executable file
View file

@ -0,0 +1,36 @@
#!/usr/bin/env bash
# SecondBrain — continuous refresh (niced, locked, idempotent).
# Re-ingests your configured sources, re-scrubs, and embeds a bounded backlog.
# Configure via environment (or a .env next to brain.py):
#
# BRAIN_DIR dir containing brain.py (default: script's parent)
# BRAIN_HOME db location (default: ~/.secondbrain)
# SB_FILE_DIRS colon-separated dirs to ingest as files
# SB_TRANSCRIPTS dir of *.jsonl agent sessions (optional)
# SB_EVENTS_DB sqlite events db (optional)
# SB_EMBED_LIMIT chunks to embed per run (default: 6000)
#
# Run from cron/systemd-timer, e.g. every 15 min.
set -euo pipefail
BRAIN_DIR="${BRAIN_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}"
BRAIN_HOME="${BRAIN_HOME:-$HOME/.secondbrain}"
PY="${SB_PYTHON:-python3}"
LOGDIR="$BRAIN_HOME/logs"; mkdir -p "$LOGDIR"
LOCK="$LOGDIR/.refresh.lock"
exec 9>"$LOCK"
flock -n 9 || { echo "[refresh] $(date -Is) busy, skip" >> "$LOGDIR/refresh.log"; exit 0; }
cd "$BRAIN_DIR"
[ -f .env ] && set -a && . ./.env && set +a
{
echo "[refresh] $(date -Is) start"
$PY brain.py init
IFS=':' read -ra DIRS <<< "${SB_FILE_DIRS:-}"
for d in "${DIRS[@]}"; do [ -n "$d" ] && [ -d "$d" ] && $PY brain.py ingest-files "$d"; done
[ -n "${SB_TRANSCRIPTS:-}" ] && [ -d "$SB_TRANSCRIPTS" ] && $PY brain.py ingest-transcripts "$SB_TRANSCRIPTS"
[ -n "${SB_EVENTS_DB:-}" ] && [ -f "$SB_EVENTS_DB" ] && $PY brain.py ingest-events --db "$SB_EVENTS_DB"
$PY brain.py rescrub
nice -n 15 $PY brain.py embed --limit "${SB_EMBED_LIMIT:-6000}" --batch 64
echo "[refresh] $(date -Is) done"
} >> "$LOGDIR/refresh.log" 2>&1

11
deploy/secondbrain-refresh.service Normal file
View file

@ -0,0 +1,11 @@
[Unit]
Description=SecondBrain refresh (ingest + embed)
After=network-online.target
[Service]
Type=oneshot
# Adjust WorkingDirectory to where brain.py lives, and EnvironmentFile to your .env
WorkingDirectory=/opt/secondbrain
EnvironmentFile=-/opt/secondbrain/.env
ExecStart=/usr/bin/env bash /opt/secondbrain/deploy/refresh.sh
Nice=15

10
deploy/secondbrain-refresh.timer Normal file
View file

@ -0,0 +1,10 @@
[Unit]
Description=Run SecondBrain refresh every 15 minutes
[Timer]
OnBootSec=3min
OnUnitActiveSec=15min
Persistent=true
[Install]
WantedBy=timers.target

74
docs/ARCHITECTURE.md Normal file
View file

@ -0,0 +1,74 @@
# SecondBrain — Architecture
SecondBrain is deliberately small: one Python file (`brain.py`), one SQLite database, one
external dependency you run yourself (Ollama). This document explains the design decisions.
## Data model
A single SQLite file (`brain.db`) with:
| Object | Role |
|---|---|
| `chunks` | one row per stored text segment: `sha` (dedup), `source`, `project`, `path`, `ref`, `ts`, `role`, `text`, `embedded`, `created` |
| `vec_chunks` | `sqlite-vec` virtual table holding the 768-dim embedding per chunk (`rowid` = `chunks.id`) |
| `fts_chunks` | FTS5 **external-content** index over `chunks.text` — stores only the BM25 index, not a second copy of the text (disk-safe). Kept in sync by triggers |
| `facts` | optional distilled facts (for an external consolidation loop) |
Because `vec_chunks` and `fts_chunks` both key on `chunks.id`, the two retrieval channels can be
fused cheaply.
## Ingestion pipeline
```
source text ─▶ scrub() ─▶ _segments() ─▶ _stage() ─▶ chunks (INSERT OR IGNORE by sha)
```
1. **`scrub()`** — mandatory. Applies a battery of regexes that redact secrets (PEM blocks,
JWTs, vendor key shapes, labelled `password/token/secret/api_key…`, URL-embedded creds,
`Company2026!`-style passwords). Optionally strips emails. **Nothing is stored un-scrubbed.**
2. **`_segments()`** — splits long text on paragraph boundaries into ≤ `MAX_CHARS` (~400-token)
chunks; drops sub-`MIN_CHARS` noise.
3. **`_stage()`** — computes a content `sha` for dedup and does `INSERT OR IGNORE`, so
re-ingesting the same source is a cheap no-op.
Ingestion is **strictly read-only** — SecondBrain never writes back to your sources.
## Embedding
`embed()` selects `WHERE embedded=0` and calls Ollama's batch `/api/embed` (falling back to
per-chunk `/api/embeddings` on older Ollama). Vectors are **L2-normalized** so that vec0's L2
distance ranking is equivalent to cosine similarity (`cos = 1 − d²/2`). A chunk that genuinely
can't be embedded is marked `embedded=-1` and skipped on subsequent runs (reset to `0` to retry).
Embedding is idempotent and resumable — kill it any time; the next run continues the backlog.
## Retrieval
`brain_recall()` is pure dense KNN. `brain_recall_hybrid()` is the recommended path:
1. Embed the query (`search_query:` prefix, per nomic's asymmetric convention).
2. **Dense channel** — top-`pool` by vector distance.
3. **Lexical channel** — FTS5 BM25 over a sanitized MATCH string (alphanumeric terms, each
quoted to neutralize FTS5 operators, OR-joined for recall).
4. **Reciprocal-rank-fusion** — each candidate scores `Σ 1/(rrf_k + rank_in_channel)` across the
channels it appears in; top-`k` by fused score.
RRF needs no score calibration between the two very different channels, and the hybrid degrades
to dense-only when the lexical index is empty — so it is a safe drop-in for the dense function.
Why hybrid matters: pure embeddings are great at *meaning* but weak at *exact tokens*. An invoice
number like `INV/2026/00037` or an IBAN often ranks poorly by cosine yet is an exact lexical hit.
Fusion recovers those without hurting semantic queries.
## Configuration & portability
Every path, endpoint and model is an environment variable with a local-first default
(`~/.secondbrain/brain.db`, `http://localhost:11434`, `nomic-embed-text`). Nothing is hard-coded
to a host. To move a brain between machines, copy `brain.db`. To offload compute, point
`OLLAMA_URL` at any reachable Ollama (e.g. a GPU box on your tailnet).
## What SecondBrain is not
- Not a chat UI — it's a retrieval library + CLI you wire into your own tools/agents.
- Not a reasoning loop — that lives in sister components (see the wiki).
- Not multi-tenant — it's a *personal* brain; run one per person/box.

3
requirements.txt Normal file
View file

@ -0,0 +1,3 @@
sqlite-vec>=0.1.6
requests>=2.31
numpy>=1.24