secondbrain/README.md

<div align="center">

# 🧠 SecondBrain

**A local-first, privacy-scrubbing personal memory you can point at everything you produce.**

*Notes · code · agent transcripts · event logs → one portable file → instant hybrid recall.*
*No cloud. No API keys. No data egress. Secrets scrubbed before anything is stored.*

**An Atlas Corporation product · © 2026 Atlas Corporation · Proprietary**

</div>

---

SecondBrain is a tiny (~500-line, single-file) knowledge engine. You feed it the things you
already generate — documents, source code, AI-agent chat transcripts, structured event logs —
and it gives you back one command:

```bash
brain recall "what did we decide about the pricing model" --hybrid
```

Everything runs on your own machine. Embeddings are computed locally by
[Ollama](https://ollama.com) (`nomic-embed-text`), and the entire brain is a **single
portable `brain.db`** (SQLite + [`sqlite-vec`](https://github.com/asg017/sqlite-vec)) you can
copy to a USB stick. There is no server to run, no account to create, and **no credential
ever leaves your box** — because SecondBrain redacts them *before* embedding.

## Why it exists

Vector-RAG demos are easy; a memory you'd actually trust with your real life is not. SecondBrain
is opinionated about the three things that usually break:

| Problem | SecondBrain's answer |
|---|---|
| **Secrets leak into your index** (and then into an LLM's context) | A mandatory scrub pass strips PEM keys, JWTs, `sk-ant-…`/`sk-…`/`ghp_…`/AWS keys, and any `password: / token= / bearer …` pattern **before** a chunk is stored. IPs, hostnames and emails are kept as operational knowledge (toggle with `SCRUB_KEEP_PII=0`). |
| **Dense-only search misses exact strings** (invoice numbers, IBANs, container names, file paths) | Hybrid recall fuses dense KNN with FTS5/BM25 lexical search via **reciprocal-rank-fusion** — meaning *and* identifiers both hit. |
| **Cloud RAG = data egress + lock-in + cost** | 100% local. Ollama for embeddings, SQLite for storage. Portable, free, offline-capable, redeployable on any device in minutes. |

## The senses

SecondBrain ingests from pluggable "senses" — each strictly **read-only**:

- **`ingest-files <dir>`** — code, docs and config (60+ text extensions; skips `node_modules`, `.git`, binaries, and files > 400 KB).
- **`ingest-transcripts <dir>`** — `*.jsonl` AI-agent sessions (Claude Code–style event logs); tool spam is collapsed to signal.
- **`ingest-events --db <sqlite>`** — any SQLite with an `events(id, ts, sense, subject, payload)` table (e.g. email, chat, calendar, finance). Turns real activity into recallable memory.

Add your own sense in ~10 lines: chunk → `scrub()` → `_stage()`. That's the whole contract.

## Quickstart

```bash
# 0. prerequisites: python3, and Ollama running (https://ollama.com)
ollama pull nomic-embed-text

# 1. install
git clone https://git.atlascorporation.nl/atlas/secondbrain
cd secondbrain
bash deploy/install.sh          # venv + deps + schema
. .venv/bin/activate

# 2. feed it something
python3 brain.py ingest-files ~/notes
python3 brain.py ingest-files ~/code/my-project

# 3. embed (local, free)
python3 brain.py embed

# 4. recall
python3 brain.py recall "how does the auth flow work" --hybrid
python3 brain.py stats
```

Output:

```
#1  score=0.72  via=both rrf=0.0312  [file/my-project]  auth.py
   def login(user, pw):  # verifies against argon2 hash, issues a signed …
```

## Setup wizard — connect your apps

`brain.py` is the engine; the **setup wizard** is the product experience. Instead of manually
pointing at directories, run:

```bash
python3 setup.py            # pick apps → sign in → test → sync
python3 setup.py sync       # re-sync everything (put on a cron/timer)
python3 setup.py list       # available connectors
```

The wizard walks you through connecting the apps that hold your life, stores each app's login in
an **encrypted vault** (`connectors/vault.py` — Fernet/PBKDF2, `~/.secondbrain/vault.enc`, chmod
600), tests the connection, and syncs. **App credentials never enter `brain.db`** — connectors
pull data and only *scrubbed* text reaches the brain.

**Connectors today:** Files & folders · Email (IMAP) · Nextcloud / WebDAV.
**On the roadmap:** Google & Microsoft (OAuth) · WhatsApp (Evolution) · Calendar (CalDAV) ·
Paperless-ngx · Odoo · Obsidian · Git/GitHub. A connector is ~40 lines — subclass `Connector`,
declare its `fields`, implement `test()` + `sync()`; `feed()` runs it through the engine's scrub.

> **Architecture note:** the engine stays pure and connector-agnostic. Connectors + vault + wizard
> are an optional layer on top — SecondBrain still works as a bare `brain.py` library/CLI without them.

## Configuration

All via environment (or a `.env` next to `brain.py`). Copy `.env.example` → `.env`:

| Var | Default | Meaning |
|---|---|---|
| `BRAIN_HOME` | `~/.secondbrain` | where `brain.db` lives |
| `BRAIN_DB` | `$BRAIN_HOME/brain.db` | explicit db path (overrides `BRAIN_HOME`) |
| `OLLAMA_URL` | `http://localhost:11434` | any reachable Ollama — including a **GPU box on your LAN/tailnet** |
| `EMBED_MODEL` | `nomic-embed-text` | embedding model |
| `EMBED_DIM` | `768` | must match the model |
| `SCRUB_KEEP_PII` | `1` | `0` also strips emails (secrets always scrubbed) |

> **Tip — offload embeddings to a GPU:** point `OLLAMA_URL` at another machine's Ollama
> (`http://100.x.y.z:11434`). SecondBrain will embed there and store locally. This is how you
> keep a laptop's memory current using a desktop GPU, with zero extra infrastructure.

## Deploy anywhere

**systemd (continuous refresh):** set your sources in `.env` (`SB_FILE_DIRS`, `SB_TRANSCRIPTS`,
`SB_EVENTS_DB`), then:

```bash
sudo cp deploy/secondbrain-refresh.* /etc/systemd/system/
sudo systemctl enable --now secondbrain-refresh.timer   # ingest+embed every 15 min
```

**Docker:**

```bash
docker build -t secondbrain -f deploy/Dockerfile .
docker run --rm -v sbdata:/data -e OLLAMA_URL=http://host.docker.internal:11434 \
  secondbrain recall "quarterly numbers" --hybrid
```

## Use it as a library

```python
import brain
for hit in brain.brain_recall_hybrid("open invoices for June", k=8):
    print(hit["score"], hit["ref"], hit["text"][:120])
```

`brain_recall_hybrid()` degrades gracefully to dense-only if the lexical index is empty, so it's
a safe drop-in for `brain_recall()`.

## How it works

```
  sources ──▶ scrub() ──▶ chunk ──▶ chunks table ──┬─▶ FTS5 (BM25, via triggers)
 (senses)   (redact)   (≤1600 ch)                   └─▶ vec_chunks (nomic 768-dim, Ollama)
                                                          │
   recall("q") ──▶ embed query ──▶ dense KNN ┐            │
                └▶ FTS5 lexical ─────────────┴─ RRF fuse ─┴─▶ ranked hits
```

See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the full design, the scrub guarantees,
and the reciprocal-rank-fusion math.

## Privacy & security

- **Scrub-before-store** is not optional and runs on every chunk from every sense. `rescrub`
  re-applies a hardened scrub to an existing db in place.
- The database **never leaves your machine** unless you copy it. `.gitignore` blocks `*.db`,
  `.env`, `secrets/`, `*.age`, `*.pem`, `*.key` so you can't accidentally commit data.
- No telemetry. No network calls except to your configured Ollama endpoint.

## Related components

SecondBrain is the memory layer of a larger local-first autonomy stack. Sister components
(separate repos) include the **policy-gate** (an approval brake that parks money/comms/
irreversible actions for a human) and fleet/ingest tooling. See the wiki.

## License & ownership

**© 2026 Atlas Corporation. All Rights Reserved.**

SecondBrain is **proprietary, confidential software** owned by Atlas Corporation. It is provided
for reference and for use **only under a written agreement** with Atlas Corporation. No right to
use, copy, modify, redistribute, or create derivative works is granted absent such an agreement —
see [`LICENSE`](LICENSE) for the full terms. "Atlas Corporation" and its marks are the property of
Atlas Corporation.