Atlas
84afdc1f56
Generalized + redeployable: config via env, local Ollama embeddings, single portable sqlite-vec db, hybrid dense+FTS5 recall, mandatory credential scrub.
155 lines
6.3 KiB
Markdown
155 lines
6.3 KiB
Markdown
<div align="center">
|
||
|
||
# 🧠 SecondBrain
|
||
|
||
**A local-first, privacy-scrubbing personal memory you can point at everything you produce.**
|
||
|
||
*Notes · code · agent transcripts · event logs → one portable file → instant hybrid recall.*
|
||
*No cloud. No API keys. No data egress. Secrets scrubbed before anything is stored.*
|
||
|
||
</div>
|
||
|
||
---
|
||
|
||
SecondBrain is a tiny (~500-line, single-file) knowledge engine. You feed it the things you
|
||
already generate — documents, source code, AI-agent chat transcripts, structured event logs —
|
||
and it gives you back one command:
|
||
|
||
```bash
|
||
brain recall "what did we decide about the pricing model" --hybrid
|
||
```
|
||
|
||
Everything runs on your own machine. Embeddings are computed locally by
|
||
[Ollama](https://ollama.com) (`nomic-embed-text`), and the entire brain is a **single
|
||
portable `brain.db`** (SQLite + [`sqlite-vec`](https://github.com/asg017/sqlite-vec)) you can
|
||
copy to a USB stick. There is no server to run, no account to create, and **no credential
|
||
ever leaves your box** — because SecondBrain redacts them *before* embedding.
|
||
|
||
## Why it exists
|
||
|
||
Vector-RAG demos are easy; a memory you'd actually trust with your real life is not. SecondBrain
|
||
is opinionated about the three things that usually break:
|
||
|
||
| Problem | SecondBrain's answer |
|
||
|---|---|
|
||
| **Secrets leak into your index** (and then into an LLM's context) | A mandatory scrub pass strips PEM keys, JWTs, `sk-ant-…`/`sk-…`/`ghp_…`/AWS keys, and any `password: / token= / bearer …` pattern **before** a chunk is stored. IPs, hostnames and emails are kept as operational knowledge (toggle with `SCRUB_KEEP_PII=0`). |
|
||
| **Dense-only search misses exact strings** (invoice numbers, IBANs, container names, file paths) | Hybrid recall fuses dense KNN with FTS5/BM25 lexical search via **reciprocal-rank-fusion** — meaning *and* identifiers both hit. |
|
||
| **Cloud RAG = data egress + lock-in + cost** | 100% local. Ollama for embeddings, SQLite for storage. Portable, free, offline-capable, redeployable on any device in minutes. |
|
||
|
||
## The senses
|
||
|
||
SecondBrain ingests from pluggable "senses" — each strictly **read-only**:
|
||
|
||
- **`ingest-files <dir>`** — code, docs and config (60+ text extensions; skips `node_modules`, `.git`, binaries, and files > 400 KB).
|
||
- **`ingest-transcripts <dir>`** — `*.jsonl` AI-agent sessions (Claude Code–style event logs); tool spam is collapsed to signal.
|
||
- **`ingest-events --db <sqlite>`** — any SQLite with an `events(id, ts, sense, subject, payload)` table (e.g. email, chat, calendar, finance). Turns real activity into recallable memory.
|
||
|
||
Add your own sense in ~10 lines: chunk → `scrub()` → `_stage()`. That's the whole contract.
|
||
|
||
## Quickstart
|
||
|
||
```bash
|
||
# 0. prerequisites: python3, and Ollama running (https://ollama.com)
|
||
ollama pull nomic-embed-text
|
||
|
||
# 1. install
|
||
git clone https://git.atlascorporation.nl/atlas/secondbrain
|
||
cd secondbrain
|
||
bash deploy/install.sh # venv + deps + schema
|
||
. .venv/bin/activate
|
||
|
||
# 2. feed it something
|
||
python3 brain.py ingest-files ~/notes
|
||
python3 brain.py ingest-files ~/code/my-project
|
||
|
||
# 3. embed (local, free)
|
||
python3 brain.py embed
|
||
|
||
# 4. recall
|
||
python3 brain.py recall "how does the auth flow work" --hybrid
|
||
python3 brain.py stats
|
||
```
|
||
|
||
Output:
|
||
|
||
```
|
||
#1 score=0.72 via=both rrf=0.0312 [file/my-project] auth.py
|
||
def login(user, pw): # verifies against argon2 hash, issues a signed …
|
||
```
|
||
|
||
## Configuration
|
||
|
||
All via environment (or a `.env` next to `brain.py`). Copy `.env.example` → `.env`:
|
||
|
||
| Var | Default | Meaning |
|
||
|---|---|---|
|
||
| `BRAIN_HOME` | `~/.secondbrain` | where `brain.db` lives |
|
||
| `BRAIN_DB` | `$BRAIN_HOME/brain.db` | explicit db path (overrides `BRAIN_HOME`) |
|
||
| `OLLAMA_URL` | `http://localhost:11434` | any reachable Ollama — including a **GPU box on your LAN/tailnet** |
|
||
| `EMBED_MODEL` | `nomic-embed-text` | embedding model |
|
||
| `EMBED_DIM` | `768` | must match the model |
|
||
| `SCRUB_KEEP_PII` | `1` | `0` also strips emails (secrets always scrubbed) |
|
||
|
||
> **Tip — offload embeddings to a GPU:** point `OLLAMA_URL` at another machine's Ollama
|
||
> (`http://100.x.y.z:11434`). SecondBrain will embed there and store locally. This is how you
|
||
> keep a laptop's memory current using a desktop GPU, with zero extra infrastructure.
|
||
|
||
## Deploy anywhere
|
||
|
||
**systemd (continuous refresh):** set your sources in `.env` (`SB_FILE_DIRS`, `SB_TRANSCRIPTS`,
|
||
`SB_EVENTS_DB`), then:
|
||
|
||
```bash
|
||
sudo cp deploy/secondbrain-refresh.* /etc/systemd/system/
|
||
sudo systemctl enable --now secondbrain-refresh.timer # ingest+embed every 15 min
|
||
```
|
||
|
||
**Docker:**
|
||
|
||
```bash
|
||
docker build -t secondbrain -f deploy/Dockerfile .
|
||
docker run --rm -v sbdata:/data -e OLLAMA_URL=http://host.docker.internal:11434 \
|
||
secondbrain recall "quarterly numbers" --hybrid
|
||
```
|
||
|
||
## Use it as a library
|
||
|
||
```python
|
||
import brain
|
||
for hit in brain.brain_recall_hybrid("open invoices for June", k=8):
|
||
print(hit["score"], hit["ref"], hit["text"][:120])
|
||
```
|
||
|
||
`brain_recall_hybrid()` degrades gracefully to dense-only if the lexical index is empty, so it's
|
||
a safe drop-in for `brain_recall()`.
|
||
|
||
## How it works
|
||
|
||
```
|
||
sources ──▶ scrub() ──▶ chunk ──▶ chunks table ──┬─▶ FTS5 (BM25, via triggers)
|
||
(senses) (redact) (≤1600 ch) └─▶ vec_chunks (nomic 768-dim, Ollama)
|
||
│
|
||
recall("q") ──▶ embed query ──▶ dense KNN ┐ │
|
||
└▶ FTS5 lexical ─────────────┴─ RRF fuse ─┴─▶ ranked hits
|
||
```
|
||
|
||
See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the full design, the scrub guarantees,
|
||
and the reciprocal-rank-fusion math.
|
||
|
||
## Privacy & security
|
||
|
||
- **Scrub-before-store** is not optional and runs on every chunk from every sense. `rescrub`
|
||
re-applies a hardened scrub to an existing db in place.
|
||
- The database **never leaves your machine** unless you copy it. `.gitignore` blocks `*.db`,
|
||
`.env`, `secrets/`, `*.age`, `*.pem`, `*.key` so you can't accidentally commit data.
|
||
- No telemetry. No network calls except to your configured Ollama endpoint.
|
||
|
||
## Related components
|
||
|
||
SecondBrain is the memory layer of a larger local-first autonomy stack. Sister components
|
||
(separate repos) include the **policy-gate** (an approval brake that parks money/comms/
|
||
irreversible actions for a human) and fleet/ingest tooling. See the wiki.
|
||
|
||
## License
|
||
|
||
MIT © 2026 Atlas Corporation. Contributions welcome.
|