secondbrain/docs/ARCHITECTURE.md

SecondBrain — Architecture

SecondBrain is deliberately small: one Python file (brain.py), one SQLite database, one external dependency you run yourself (Ollama). This document explains the design decisions.

Data model

A single SQLite file (brain.db) with:

Object	Role
chunks	one row per stored text segment: sha (dedup), source, project, path, ref, ts, role, text, embedded, created
vec_chunks	sqlite-vec virtual table holding the 768-dim embedding per chunk (rowid = chunks.id)
fts_chunks	FTS5 external-content index over chunks.text — stores only the BM25 index, not a second copy of the text (disk-safe). Kept in sync by triggers
facts	optional distilled facts (for an external consolidation loop)

Because vec_chunks and fts_chunks both key on chunks.id, the two retrieval channels can be fused cheaply.

Ingestion pipeline

source text ─▶ scrub() ─▶ _segments() ─▶ _stage() ─▶ chunks (INSERT OR IGNORE by sha)

1. scrub() — mandatory. Applies a battery of regexes that redact secrets (PEM blocks, JWTs, vendor key shapes, labelled password/token/secret/api_key…, URL-embedded creds, Company2026!-style passwords). Optionally strips emails. Nothing is stored un-scrubbed.
2. _segments() — splits long text on paragraph boundaries into ≤ MAX_CHARS (~400-token) chunks; drops sub-MIN_CHARS noise.
3. _stage() — computes a content sha for dedup and does INSERT OR IGNORE, so re-ingesting the same source is a cheap no-op.

Ingestion is strictly read-only — SecondBrain never writes back to your sources.

Embedding

embed() selects WHERE embedded=0 and calls Ollama's batch /api/embed (falling back to per-chunk /api/embeddings on older Ollama). Vectors are L2-normalized so that vec0's L2 distance ranking is equivalent to cosine similarity (cos = 1 − d²/2). A chunk that genuinely can't be embedded is marked embedded=-1 and skipped on subsequent runs (reset to 0 to retry).

Embedding is idempotent and resumable — kill it any time; the next run continues the backlog.

Retrieval

brain_recall() is pure dense KNN. brain_recall_hybrid() is the recommended path:

1. Embed the query (search_query: prefix, per nomic's asymmetric convention).
2. Dense channel — top-pool by vector distance.
3. Lexical channel — FTS5 BM25 over a sanitized MATCH string (alphanumeric terms, each quoted to neutralize FTS5 operators, OR-joined for recall).
4. Reciprocal-rank-fusion — each candidate scores Σ 1/(rrf_k + rank_in_channel) across the channels it appears in; top-k by fused score.

RRF needs no score calibration between the two very different channels, and the hybrid degrades to dense-only when the lexical index is empty — so it is a safe drop-in for the dense function.

Why hybrid matters: pure embeddings are great at meaning but weak at exact tokens. An invoice number like INV/2026/00037 or an IBAN often ranks poorly by cosine yet is an exact lexical hit. Fusion recovers those without hurting semantic queries.

Configuration & portability

Every path, endpoint and model is an environment variable with a local-first default (~/.secondbrain/brain.db, http://localhost:11434, nomic-embed-text). Nothing is hard-coded to a host. To move a brain between machines, copy brain.db. To offload compute, point OLLAMA_URL at any reachable Ollama (e.g. a GPU box on your tailnet).

What SecondBrain is not

• Not a chat UI — it's a retrieval library + CLI you wire into your own tools/agents.
• Not a reasoning loop — that lives in sister components (see the wiki).
• Not multi-tenant — it's a personal brain; run one per person/box.