1 Architecture

atlas edited this page 2026-07-01 19:28:14 +00:00

Unescape Escape

Table of Contents

• SecondBrain — Architecture
• Data model
• Ingestion pipeline
• Embedding
• Retrieval
• Configuration & portability
• What SecondBrain is not

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

SecondBrain — Architecture

SecondBrain is deliberately small: one Python file (brain.py), one SQLite database, one external dependency you run yourself (Ollama). This document explains the design decisions.

Data model

A single SQLite file (brain.db) with:

Object	Role
chunks	one row per stored text segment: sha (dedup), source, project, path, ref, ts, role, text, embedded, created
vec_chunks	sqlite-vec virtual table holding the 768-dim embedding per chunk (rowid = chunks.id)
fts_chunks	FTS5 external-content index over chunks.text — stores only the BM25 index, not a second copy of the text (disk-safe). Kept in sync by triggers
facts	optional distilled facts (for an external consolidation loop)

Because vec_chunks and fts_chunks both key on chunks.id, the two retrieval channels can be fused cheaply.

Ingestion pipeline

source text ─▶ scrub() ─▶ _segments() ─▶ _stage() ─▶ chunks (INSERT OR IGNORE by sha)

1. scrub() — mandatory. Applies a battery of regexes that redact secrets (PEM blocks, JWTs, vendor key shapes, labelled password/token/secret/api_key…, URL-embedded creds, Company2026!-style passwords). Optionally strips emails. Nothing is stored un-scrubbed.
2. _segments() — splits long text on paragraph boundaries into ≤ MAX_CHARS (~400-token) chunks; drops sub-MIN_CHARS noise.
3. _stage() — computes a content sha for dedup and does INSERT OR IGNORE, so re-ingesting the same source is a cheap no-op.

Ingestion is strictly read-only — SecondBrain never writes back to your sources.

Embedding

embed() selects WHERE embedded=0 and calls Ollama's batch /api/embed (falling back to per-chunk /api/embeddings on older Ollama). Vectors are L2-normalized so that vec0's L2 distance ranking is equivalent to cosine similarity (cos = 1 − d²/2). A chunk that genuinely can't be embedded is marked embedded=-1 and skipped on subsequent runs (reset to 0 to retry).

Embedding is idempotent and resumable — kill it any time; the next run continues the backlog.

Retrieval

brain_recall() is pure dense KNN. brain_recall_hybrid() is the recommended path:

1. Embed the query (search_query: prefix, per nomic's asymmetric convention).
2. Dense channel — top-pool by vector distance.
3. Lexical channel — FTS5 BM25 over a sanitized MATCH string (alphanumeric terms, each quoted to neutralize FTS5 operators, OR-joined for recall).
4. Reciprocal-rank-fusion — each candidate scores Σ 1/(rrf_k + rank_in_channel) across the channels it appears in; top-k by fused score.

RRF needs no score calibration between the two very different channels, and the hybrid degrades to dense-only when the lexical index is empty — so it is a safe drop-in for the dense function.

Why hybrid matters: pure embeddings are great at meaning but weak at exact tokens. An invoice number like INV/2026/00037 or an IBAN often ranks poorly by cosine yet is an exact lexical hit. Fusion recovers those without hurting semantic queries.

Configuration & portability

Every path, endpoint and model is an environment variable with a local-first default (~/.secondbrain/brain.db, http://localhost:11434, nomic-embed-text). Nothing is hard-coded to a host. To move a brain between machines, copy brain.db. To offload compute, point OLLAMA_URL at any reachable Ollama (e.g. a GPU box on your tailnet).

What SecondBrain is not

• Not a chat UI — it's a retrieval library + CLI you wire into your own tools/agents.
• Not a reasoning loop — that lives in sister components (see the wiki).
• Not multi-tenant — it's a personal brain; run one per person/box.