# SecondBrain — Architecture

SecondBrain is deliberately small: one Python file (`brain.py`), one SQLite database, one
external dependency you run yourself (Ollama). This document explains the design decisions.

## Data model

A single SQLite file (`brain.db`) with:

| Object | Role |
|---|---|
| `chunks` | one row per stored text segment: `sha` (dedup), `source`, `project`, `path`, `ref`, `ts`, `role`, `text`, `embedded`, `created` |
| `vec_chunks` | `sqlite-vec` virtual table holding the 768-dim embedding per chunk (`rowid` = `chunks.id`) |
| `fts_chunks` | FTS5 **external-content** index over `chunks.text` — stores only the BM25 index, not a second copy of the text (disk-safe). Kept in sync by triggers |
| `facts` | optional distilled facts (for an external consolidation loop) |

Because `vec_chunks` and `fts_chunks` both key on `chunks.id`, the two retrieval channels can be
fused cheaply.

## Ingestion pipeline

```
source text ─▶ scrub() ─▶ _segments() ─▶ _stage() ─▶ chunks (INSERT OR IGNORE by sha)
```

1. **`scrub()`** — mandatory. Applies a battery of regexes that redact secrets (PEM blocks,
   JWTs, vendor key shapes, labelled `password/token/secret/api_key…`, URL-embedded creds,
   `Company2026!`-style passwords). Optionally strips emails. **Nothing is stored un-scrubbed.**
2. **`_segments()`** — splits long text on paragraph boundaries into ≤ `MAX_CHARS` (~400-token)
   chunks; drops sub-`MIN_CHARS` noise.
3. **`_stage()`** — computes a content `sha` for dedup and does `INSERT OR IGNORE`, so
   re-ingesting the same source is a cheap no-op.

Ingestion is **strictly read-only** — SecondBrain never writes back to your sources.

## Embedding

`embed()` selects `WHERE embedded=0` and calls Ollama's batch `/api/embed` (falling back to
per-chunk `/api/embeddings` on older Ollama). Vectors are **L2-normalized** so that vec0's L2
distance ranking is equivalent to cosine similarity (`cos = 1 − d²/2`). A chunk that genuinely
can't be embedded is marked `embedded=-1` and skipped on subsequent runs (reset to `0` to retry).

Embedding is idempotent and resumable — kill it any time; the next run continues the backlog.

## Retrieval

`brain_recall()` is pure dense KNN. `brain_recall_hybrid()` is the recommended path:

1. Embed the query (`search_query:` prefix, per nomic's asymmetric convention).
2. **Dense channel** — top-`pool` by vector distance.
3. **Lexical channel** — FTS5 BM25 over a sanitized MATCH string (alphanumeric terms, each
   quoted to neutralize FTS5 operators, OR-joined for recall).
4. **Reciprocal-rank-fusion** — each candidate scores `Σ 1/(rrf_k + rank_in_channel)` across the
   channels it appears in; top-`k` by fused score.

RRF needs no score calibration between the two very different channels, and the hybrid degrades
to dense-only when the lexical index is empty — so it is a safe drop-in for the dense function.

Why hybrid matters: pure embeddings are great at *meaning* but weak at *exact tokens*. An invoice
number like `INV/2026/00037` or an IBAN often ranks poorly by cosine yet is an exact lexical hit.
Fusion recovers those without hurting semantic queries.

## Configuration & portability

Every path, endpoint and model is an environment variable with a local-first default
(`~/.secondbrain/brain.db`, `http://localhost:11434`, `nomic-embed-text`). Nothing is hard-coded
to a host. To move a brain between machines, copy `brain.db`. To offload compute, point
`OLLAMA_URL` at any reachable Ollama (e.g. a GPU box on your tailnet).

## What SecondBrain is not

- Not a chat UI — it's a retrieval library + CLI you wire into your own tools/agents.
- Not a reasoning loop — that lives in sister components (see the wiki).
- Not multi-tenant — it's a *personal* brain; run one per person/box.