Atlas
84afdc1f56
Generalized + redeployable: config via env, local Ollama embeddings, single portable sqlite-vec db, hybrid dense+FTS5 recall, mandatory credential scrub.
74 lines
3.7 KiB
Markdown
74 lines
3.7 KiB
Markdown
# SecondBrain — Architecture
|
||
|
||
SecondBrain is deliberately small: one Python file (`brain.py`), one SQLite database, one
|
||
external dependency you run yourself (Ollama). This document explains the design decisions.
|
||
|
||
## Data model
|
||
|
||
A single SQLite file (`brain.db`) with:
|
||
|
||
| Object | Role |
|
||
|---|---|
|
||
| `chunks` | one row per stored text segment: `sha` (dedup), `source`, `project`, `path`, `ref`, `ts`, `role`, `text`, `embedded`, `created` |
|
||
| `vec_chunks` | `sqlite-vec` virtual table holding the 768-dim embedding per chunk (`rowid` = `chunks.id`) |
|
||
| `fts_chunks` | FTS5 **external-content** index over `chunks.text` — stores only the BM25 index, not a second copy of the text (disk-safe). Kept in sync by triggers |
|
||
| `facts` | optional distilled facts (for an external consolidation loop) |
|
||
|
||
Because `vec_chunks` and `fts_chunks` both key on `chunks.id`, the two retrieval channels can be
|
||
fused cheaply.
|
||
|
||
## Ingestion pipeline
|
||
|
||
```
|
||
source text ─▶ scrub() ─▶ _segments() ─▶ _stage() ─▶ chunks (INSERT OR IGNORE by sha)
|
||
```
|
||
|
||
1. **`scrub()`** — mandatory. Applies a battery of regexes that redact secrets (PEM blocks,
|
||
JWTs, vendor key shapes, labelled `password/token/secret/api_key…`, URL-embedded creds,
|
||
`Company2026!`-style passwords). Optionally strips emails. **Nothing is stored un-scrubbed.**
|
||
2. **`_segments()`** — splits long text on paragraph boundaries into ≤ `MAX_CHARS` (~400-token)
|
||
chunks; drops sub-`MIN_CHARS` noise.
|
||
3. **`_stage()`** — computes a content `sha` for dedup and does `INSERT OR IGNORE`, so
|
||
re-ingesting the same source is a cheap no-op.
|
||
|
||
Ingestion is **strictly read-only** — SecondBrain never writes back to your sources.
|
||
|
||
## Embedding
|
||
|
||
`embed()` selects `WHERE embedded=0` and calls Ollama's batch `/api/embed` (falling back to
|
||
per-chunk `/api/embeddings` on older Ollama). Vectors are **L2-normalized** so that vec0's L2
|
||
distance ranking is equivalent to cosine similarity (`cos = 1 − d²/2`). A chunk that genuinely
|
||
can't be embedded is marked `embedded=-1` and skipped on subsequent runs (reset to `0` to retry).
|
||
|
||
Embedding is idempotent and resumable — kill it any time; the next run continues the backlog.
|
||
|
||
## Retrieval
|
||
|
||
`brain_recall()` is pure dense KNN. `brain_recall_hybrid()` is the recommended path:
|
||
|
||
1. Embed the query (`search_query:` prefix, per nomic's asymmetric convention).
|
||
2. **Dense channel** — top-`pool` by vector distance.
|
||
3. **Lexical channel** — FTS5 BM25 over a sanitized MATCH string (alphanumeric terms, each
|
||
quoted to neutralize FTS5 operators, OR-joined for recall).
|
||
4. **Reciprocal-rank-fusion** — each candidate scores `Σ 1/(rrf_k + rank_in_channel)` across the
|
||
channels it appears in; top-`k` by fused score.
|
||
|
||
RRF needs no score calibration between the two very different channels, and the hybrid degrades
|
||
to dense-only when the lexical index is empty — so it is a safe drop-in for the dense function.
|
||
|
||
Why hybrid matters: pure embeddings are great at *meaning* but weak at *exact tokens*. An invoice
|
||
number like `INV/2026/00037` or an IBAN often ranks poorly by cosine yet is an exact lexical hit.
|
||
Fusion recovers those without hurting semantic queries.
|
||
|
||
## Configuration & portability
|
||
|
||
Every path, endpoint and model is an environment variable with a local-first default
|
||
(`~/.secondbrain/brain.db`, `http://localhost:11434`, `nomic-embed-text`). Nothing is hard-coded
|
||
to a host. To move a brain between machines, copy `brain.db`. To offload compute, point
|
||
`OLLAMA_URL` at any reachable Ollama (e.g. a GPU box on your tailnet).
|
||
|
||
## What SecondBrain is not
|
||
|
||
- Not a chat UI — it's a retrieval library + CLI you wire into your own tools/agents.
|
||
- Not a reasoning loop — that lives in sister components (see the wiki).
|
||
- Not multi-tenant — it's a *personal* brain; run one per person/box.
|