> ## Documentation Index
> Fetch the complete documentation index at: https://gladlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Disaster recovery

# Disaster Recovery Runbook

**Last reviewed:** 2026-06-15
**Audience:** solo operator (Matt) at 2am during an incident
**Prereqs:** Local PC online, Docker running, gh CLI authed, poindexter CLI installed, `~/.poindexter/bootstrap.toml` accessible (or you have the `database_url` + `POINDEXTER_SECRET_KEY` somewhere safe)

This runbook covers catastrophic-loss scenarios — the kind where "just restart the container" is not enough. For per-alert triage and routing, see [`incident-response.md`](./incident-response). For known-pattern symptom debugging, see [`troubleshooting.md`](./troubleshooting).

***

## Scenarios this covers

* **DB-1.** PostgreSQL container is healthy but data is corrupted / wrong / missing rows
* **DB-2.** `poindexter-postgres-local` Docker volume was wiped or destroyed (full data loss)
* **DB-3.** A migration ran half-way, left the schema in an inconsistent state
* **HOST-1.** New machine — rebuild the entire local stack from scratch (laptop died, fresh OS install)
* **HOST-2.** Docker Desktop / WSL2 corrupted — need to reset Docker but keep the data
* **CONFIG-1.** `bootstrap.toml` lost — need to reconstruct it
* **CONFIG-2.** `POINDEXTER_SECRET_KEY` lost — every encrypted secret in `app_settings` is unreadable

For per-service recovery (worker crashed, image-gen degraded, Vercel down), see the **Per-service recovery** section at the bottom.

***

## Quick triage flowchart

```
Is the postgres container running?
  NO  -> Try DB-1 (restart)
  YES -> Can you connect with psql?
            NO  -> Container is broken: DB-1 / DB-2 (volume gone)
            YES -> Are the tables there? (check `app_settings`, `posts`, `pipeline_tasks`)
                     NO  -> DB-2 (volume wiped, fresh DB)
                     YES -> Does data look right? (post counts, recent rows)
                              NO  -> DB-1 (corruption, restore from backup)
                              YES -> Issue is downstream — check incident-response.md

Is `~/.poindexter/bootstrap.toml` present and readable?
  NO  -> CONFIG-1 (reconstruct bootstrap.toml)

Is `POINDEXTER_SECRET_KEY` set in env / bootstrap.toml?
  NO  -> CONFIG-2 (re-generate key + re-seed every secret)

Can you run `docker ps`?
  NO  -> HOST-2 (Docker broken)

Is this a brand new machine / fresh OS?
  YES -> HOST-1 (full rebuild)
```

***

## Procedure: DB-1 — Restore PostgreSQL from backup

**Symptoms.** Container is running, you can connect, but data is wrong (missing rows, bad migration, accidental DELETE, etc.).

### Step 1 — Stop the worker so it doesn't fight you

```bash theme={null}
docker stop poindexter-worker poindexter-brain-daemon
```

### Step 2 — List available backups

Two backup tiers write under `~/.poindexter/backups/` — see [`backups.md`](./backups) for the full design:

* **In-stack container tier (primary).** `backup-hourly` / `backup-daily` write `~/.poindexter/backups/auto/{hourly,daily}/poindexter_brain_*.dump` (last 24 hourly + 7 daily).
* **`DbBackupJob`.** The worker's scheduled job runs `scripts/db-backup-local.sh` into the flat `~/.poindexter/backups/` (`poindexter-db-*.dump`, 14-day retention).

```bash theme={null}
ls -lht ~/.poindexter/backups/auto/hourly/ ~/.poindexter/backups/auto/daily/ ~/.poindexter/backups/ 2>/dev/null | head -30
# Expected: poindexter_brain_2026-04-30T0312Z.dump (container tiers)
#       and/or poindexter-db-2026-04-30T0312Z.dump (DbBackupJob)
```

If every location is empty or stale, your backup automation is broken — go to **DB-2** and treat this as a volume-loss event (your last good state is whatever's in git/Vercel/R2). The restore steps below take whichever dump you picked here.

### Step 3 — Verify the backup is restorable BEFORE you drop the live DB

```bash theme={null}
pg_restore --list ~/.poindexter/backups/poindexter-db-<TIMESTAMP>.dump | head -20
# Expected: a list of TABLE / INDEX / SEQUENCE entries.
# If pg_restore errors, the backup is corrupt — try the next-oldest one.
```

### Step 4 — Drop and recreate the DB

```bash theme={null}
# Connect as superuser
docker exec -it poindexter-postgres-local psql -U poindexter -d postgres

# Inside psql:
DROP DATABASE poindexter_brain;
CREATE DATABASE poindexter_brain OWNER poindexter;
\q
```

### Step 5 — Restore from the backup

```bash theme={null}
docker exec -i poindexter-postgres-local pg_restore \
    -U poindexter \
    -d poindexter_brain \
    --no-owner --no-privileges \
    < ~/.poindexter/backups/poindexter-db-<TIMESTAMP>.dump
```

Expected output: a stream of `pg_restore: processing data for table "..."` lines, then exit 0. Some warnings about extensions (`pgcrypto`, `vector`) are normal — they'll be re-created.

### Step 6 — Re-apply any migrations newer than the backup

```bash theme={null}
poindexter migrate status
# Compare the "applied" list against what's on disk.
poindexter migrate up
```

### Step 7 — Bring the worker back

```bash theme={null}
docker start poindexter-brain-daemon poindexter-worker
sleep 10
curl -s http://localhost:8002/api/health | python -m json.tool
```

### Verification

```bash theme={null}
# Row counts should be in the right ballpark for the backup vintage
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT 'posts' AS tbl, COUNT(*) FROM posts UNION ALL
   SELECT 'app_settings', COUNT(*) FROM app_settings UNION ALL
   SELECT 'pipeline_tasks', COUNT(*) FROM pipeline_tasks;"
```

***

## Procedure: DB-2 — `poindexter-postgres-local` data volume was wiped

**Symptoms.** `docker ps` shows postgres is up, but `psql -c "\dt"` returns "Did not find any relations" — the database exists but is empty. Or the container won't even start because the volume mount is missing.

**Worst case** — you lost the volume AND have no backup. The static export on R2 / public-site frontend still has the published posts. The git history has the schema migrations. Everything else is gone.

### Step 1 — Confirm the volume is actually gone

```bash theme={null}
docker volume ls | grep postgres
# Expected names: poindexter-postgres-data, postgres-local-data, etc.

docker volume inspect <volume_name> 2>&1 | head -10
# If "No such volume", it's gone.
```

### Step 2 — Stop everything that talks to the DB

```bash theme={null}
docker stop poindexter-worker poindexter-brain-daemon poindexter-grafana \
  poindexter-langfuse-web poindexter-langfuse-worker 2>/dev/null
```

### Step 3 — IF you have a backup, restore it

Same as **DB-1** Steps 4-7. If you got here because backups were missing too, continue.

### Step 4 — IF no backup exists, rebuild the schema

```bash theme={null}
# Ensure the container has a fresh empty volume
docker compose -f docker-compose.local.yml up -d postgres-local
sleep 5

# Run all migrations from scratch
poindexter migrate up
# Expected: "Applied N migration(s)"
```

### Step 5 — Recreate the secrets that lived in `app_settings`

This is the painful part. Encrypted secrets are gone forever. You need to:

1. Go to [`secret-rotation.md`](./secret-rotation) and rotate **every** API key listed in the inventory (Telegram, Discord, OpenAI, Anthropic, Lemon Squeezy, Resend, Cloudinary, Pexels, etc.) — treat all of them as compromised because the encrypted blobs may have leaked with the volume backup.
2. Re-seed each one with `poindexter settings set <key> "<value>" --secret` (see secret-rotation.md "Re-seeding from scratch").

### Step 6 — Re-import published posts from R2

The static export on R2 (`static/posts/index.json` + `static/posts/<slug>.json`)
is the source of truth for what was live. `scripts/dr-reimport-posts-from-r2.py`
replays those JSON files back into the `posts` table.

```bash theme={null}
# Dry-run first — validates the R2 fetch and logs what would be written
python scripts/dr-reimport-posts-from-r2.py --dry-run

# Then write (idempotent — safe to re-run)
python scripts/dr-reimport-posts-from-r2.py
```

The script resolves the database URL from `bootstrap.toml` automatically. Pass
`--database-url` to override. Pass `--r2-url` if `NEXT_PUBLIC_STATIC_URL` env
var is unset and the hardcoded default is stale. The upsert is keyed on `slug`
so re-running fills gaps without overwriting rows already restored from a backup.

> **Note:** `pipeline_tasks` rows and `pipeline_versions` rows are NOT
> re-imported (they live only in the DB). Only the published `posts` rows
> surface in the frontend, so the site is functional after this step. The
> missing task history means the auto-publish gate and edit-distance tracker
> start fresh — that's acceptable for a DR scenario.

### Verification

```bash theme={null}
poindexter migrate status     # Should show all migrations applied.
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT COUNT(*) FROM app_settings;"   # Should be > 0 after migration seeds run.
```

***

## Procedure: DB-3 — Migration ran half-way, schema is inconsistent

**Symptoms.** A `poindexter migrate up` errored mid-run. `poindexter migrate status` shows a migration as "pending" but you can see partial DDL was already applied (e.g., a column exists that shouldn't be there until that migration completes).

### Step 1 — Capture the failure mode

```bash theme={null}
poindexter migrate status > /tmp/migrate-status-before.txt
# Note the LAST applied migration name. The pending one(s) after it are the suspects.
```

### Step 2 — Choose: roll forward or roll back?

**Roll forward** is preferred when:

* The failed migration is small and you can eyeball what didn't run
* Re-running it (after manual cleanup) is safe (the DDL is idempotent or you can drop the half-applied object)

**Roll back** is preferred when:

* The migration is complex
* You have a recent backup and can afford to lose a few minutes of activity

### Step 3a — Roll forward

```bash theme={null}
# Inspect the migration file
cat src/cofounder_agent/services/migrations/<NNNN>_<name>.py

# Manually drop / undo whatever DDL the failed migration partially applied
docker exec -it poindexter-postgres-local psql -U poindexter -d poindexter_brain
# Inside psql, e.g.:
#   DROP TABLE IF EXISTS half_created_table;
#   ALTER TABLE existing_table DROP COLUMN IF EXISTS half_added_col;
# \q

# Re-run
poindexter migrate up
```

### Step 3b — Roll back

```bash theme={null}
# Roll back to the migration immediately BEFORE the broken one
poindexter migrate down --to <NNNN_previous_known_good>
# Confirms with prompt; pass --yes to skip.

# Verify
poindexter migrate status
```

If the failing migration has no `down()`/`rollback_migration()`, the down command will skip it. In that case, restore from backup (**DB-1**) to the last good state.

### Step 4 — Bring services back up

```bash theme={null}
docker start poindexter-worker poindexter-brain-daemon
```

### Verification

```bash theme={null}
poindexter migrate status   # All applied or all pending — no half-states.
curl -s http://localhost:8002/api/health | python -m json.tool
```

***

## Procedure: DB-4 — Restore from the off-machine (Tier 2) restic repo

**Scenario.** The machine is gone — drive failure, theft, ransomware — so the
in-stack `~/.poindexter/backups/auto/` dumps went with it. You configured
Tier 2 ([`backups.md`](./backups)), so the daily dumps live in an
S3-compatible bucket as an encrypted restic repo. This restores from there.

> **Hard dependency: the restic password.** Tier 2's password is stored
> *encrypted in `app_settings`* — which is gone in this scenario along with
> `POINDEXTER_SECRET_KEY`. You restore from the **offline copy** the setup
> wizard told you to save (password manager / fireproof safe). Without it the
> repo is unrecoverable — restic encryption with a lost password is final. If
> you don't have it, this procedure cannot help; skip to DB-2 (rebuild from
> migrations + R2).

### Step 1 — Install restic on the recovery machine

```bash theme={null}
# Linux/macOS: package manager, or the static binary from
# https://github.com/restic/restic/releases
# Windows: `winget install restic.restic` or the release zip.
restic version
```

### Step 2 — Point restic at the repo with your offline credentials

You need four values: the repo URL (the `offsite_backup_repository` you chose —
`s3:https://<endpoint>/<bucket>/<path>`), the **offline** restic password, and
the S3 access key id + secret. On a fresh machine `app_settings` is gone, so
these come from your offline copy / the bucket provider's console.

```bash theme={null}
export RESTIC_REPOSITORY="s3:https://s3.us-west-002.backblazeb2.com/<bucket>/<path>"
export RESTIC_PASSWORD="<the password you saved offline>"
export AWS_ACCESS_KEY_ID="<s3 access key id>"
export AWS_SECRET_ACCESS_KEY="<s3 secret access key>"
```

### Step 3 — Confirm the repo is reachable and list snapshots

```bash theme={null}
restic snapshots
# Expected: one or more snapshots tagged `poindexter`, newest last.
# If this 403s, the key/password is wrong. If it hangs, check the endpoint.
```

### Step 4 — Restore the latest snapshot to a local directory

```bash theme={null}
mkdir -p ./offsite-restore
restic restore latest --target ./offsite-restore
# The Tier 1 daily dump lands under ./offsite-restore/daily/poindexter_brain_*.dump
ls -lht ./offsite-restore/daily/
```

### Step 5 — `pg_restore` the recovered dump

Bring up an empty Postgres (HOST-1 Steps 1-5 if this is a fresh machine),
then restore the recovered `.dump` exactly like DB-1 Step 5:

```bash theme={null}
docker exec -i poindexter-postgres-local pg_restore \
    -U poindexter -d poindexter_brain --no-owner --no-privileges \
    < ./offsite-restore/daily/poindexter_brain_<TIMESTAMP>.dump
```

Then re-apply any newer migrations (DB-1 Step 6) and bring services up
(DB-1 Step 7). Because the restored dump's `app_settings` still holds the
*old* encrypted secrets, you'll also need the original `POINDEXTER_SECRET_KEY`
to decrypt them — if that's lost too, finish with CONFIG-2 (rotate + re-seed).

### Verification

```bash theme={null}
restic check                    # repo integrity (optional, slow for full)
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT COUNT(*) FROM posts; SELECT COUNT(*) FROM app_settings;"
```

***

## Procedure: HOST-1 — Rebuild the entire local stack on a new machine

**Scenario.** Laptop died, fresh OS install, or you're moving to new hardware.

### Step 1 — Install prerequisites

| Tool                                   | Why                               |
| -------------------------------------- | --------------------------------- |
| Docker Desktop 4.26+ with WSL2 backend | Runs all containers               |
| Git + Git Bash (Windows)               | start-stack.sh uses bash          |
| Python 3.13+                           | poindexter CLI                    |
| Node.js 22+                            | Public site dev/build             |
| Ollama 0.1.40+                         | Local LLM inference               |
| NVIDIA driver supporting CUDA 12.8+    | image-gen + Wan video on RTX 5090 |

### Step 2 — Clone the repo

```bash theme={null}
git clone git@github.com:Glad-Labs/poindexter.git ~/glad-labs-stack
cd ~/glad-labs-stack
```

### Step 3 — Restore secrets

You need the following from a safe place (1Password, encrypted USB, second machine):

* `~/.poindexter/bootstrap.toml` — contains `database_url` and `POINDEXTER_SECRET_KEY`
* A recent backup from `~/.poindexter/backups/` (or the cloud copy if you set one up)

```bash theme={null}
mkdir -p ~/.poindexter/backups
cp /path/to/backup/bootstrap.toml ~/.poindexter/bootstrap.toml
chmod 600 ~/.poindexter/bootstrap.toml
cp /path/to/backup/poindexter-db-*.dump ~/.poindexter/backups/
```

If you don't have either, skip ahead — we'll generate a fresh bootstrap and rebuild from migrations + R2.

### Step 4 — Install poindexter

```bash theme={null}
pip install -e src/cofounder_agent
poindexter --help    # confirm install
```

### Step 5 — Bring up the stack

```bash theme={null}
bash scripts/start-stack.sh
# Expected: postgres-local, worker, brain-daemon, grafana, etc. all start.

docker ps --format "table {{.Names}}\t{{.Status}}"
```

### Step 6 — Restore data

If you have a backup → follow **DB-1** Steps 4-7 from "Drop and recreate the DB."

If no backup → run migrations to create empty schema, then go to **DB-2** Steps 5-6.

### Step 7 — Pull Ollama models

```bash theme={null}
ollama pull nomic-embed-text
ollama pull gemma3:27b           # critic model
ollama pull qwen3:8b
ollama pull gemma-4-31B-it-qat:latest  # writer model (bakeoff winner 2026-06-18)
ollama pull glm-4.7-5090:latest        # reviser model (qa.rewrite rescue cycle)
# Confirm with:
ollama list
```

### Step 8 — Verify everything

```bash theme={null}
curl -s http://localhost:8002/api/health | python -m json.tool
curl -s http://localhost:3000/api/health
curl -s http://localhost:9836/health     # image-gen
curl -s http://localhost:11434/api/tags  # Ollama
```

### Step 9 — Re-pair MCP / OpenClaw / Telegram / Discord

```bash theme={null}
# Telegram pairing — use the /telegram:configure skill in Claude Code
# Discord pairing — use the /discord:configure skill in Claude Code
# OpenClaw — re-install via the existing setup script + paste tokens
```

***

## Procedure: HOST-2 — Docker / WSL2 broken, keep the data

**Symptoms.** `docker ps` errors, Docker Desktop refuses to start, WSL2 is unresponsive. The data is fine — only the runtime is busted.

### Step 1 — Try a soft reset first

> **Note.** The **Docker Engine Watchdog** scheduled task (`scripts/docker-watchdog.ps1`,
> every 5 min) now performs this soft reset automatically. When the Docker Desktop
> process is alive but the engine is wedged (the WSL2-VM `HCS_E_CONNECTION_TIMEOUT`
> case), it re-checks after `-WedgeConfirmSeconds` (default 30) to rule out a transient
> blip, captures forensics to `~/.poindexter/logs/wedge-<timestamp>/` (host `nvidia-smi`,
> Hyper-V/`vmcompute` event channels, `docker diagnose` bundle), pings Telegram if
> `telegram_bot_token` is set in `bootstrap.toml`, then runs `wsl --shutdown` and
> restores the stack via `start-stack.sh`. The manual steps below are the fallback if
> the watchdog is disabled or the recycle doesn't take.

```powershell theme={null}
# PowerShell, as admin
wsl --shutdown
Restart-Service com.docker.service -Force
# Wait 30s, try `docker ps` again
```

### Step 2 — If that fails, reset Docker Desktop (preserves volumes by default)

In Docker Desktop GUI: Settings → Troubleshoot → "Reset to factory defaults" **WARNING — this nukes named volumes**. Use "Clean / Purge data" only if you have a current backup.

The safer path: **uninstall Docker Desktop, reinstall same version, restart**. Named volumes survive a reinstall as long as you don't delete the WSL2 distro `docker-desktop-data`.

### Step 3 — Restart the stack

```bash theme={null}
bash scripts/start-stack.sh
docker volume ls    # Confirm postgres volume still exists
```

### Step 4 — Verify data integrity

```bash theme={null}
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT COUNT(*) FROM posts; SELECT COUNT(*) FROM app_settings;"
# Compare against what you remember the counts being.
```

If the counts are way off, treat as **DB-2** (volume loss).

***

## Procedure: CONFIG-1 — Lost `bootstrap.toml`

**Symptoms.** Worker won't start. Logs say `notify_operator(): no DATABASE_URL resolved`, then `sys.exit(2)`.

### Step 1 — Reconstruct the file

```bash theme={null}
mkdir -p ~/.poindexter
cat > ~/.poindexter/bootstrap.toml <<'EOF'
database_url = "postgresql://poindexter:<password>@localhost:5433/poindexter_brain"

# Operator notification fallbacks (optional — used when DB is unreachable)
telegram_bot_token = ""
telegram_chat_id = ""
discord_ops_webhook_url = ""

# Symmetric key for app_settings encrypted secrets — REQUIRED if you
# have any is_secret=true rows.
POINDEXTER_SECRET_KEY = "<existing key>"
EOF
chmod 600 ~/.poindexter/bootstrap.toml
```

You need the **postgres password** (find it: `docker exec poindexter-postgres-local printenv POSTGRES_PASSWORD`) and the **`POINDEXTER_SECRET_KEY`** (this is the killer — see CONFIG-2 if lost).

### Step 2 — Restart worker

```bash theme={null}
docker restart poindexter-worker
sleep 5
curl -s http://localhost:8002/api/health
```

### Verification

```bash theme={null}
# Worker should now start without sys.exit(2)
docker logs poindexter-worker 2>&1 | tail -20
```

***

## Procedure: CONFIG-2 — Lost `POINDEXTER_SECRET_KEY`

**Symptoms.** Worker boots, but every `get_secret(...)` call raises `SecretsError: Could not decrypt`. Telegram/Discord/Vercel revalidation/Lemon Squeezy webhook verification all 401. `app_settings` rows with `is_secret=true` show `enc:v1:...` blobs that can never be decoded again.

**There is no recovery for the encrypted blobs.** Encryption with AES-256 + a lost key is final. You must rotate every secret.

### Step 1 — Generate a new key

```bash theme={null}
python -c "import secrets; print(secrets.token_urlsafe(48))"
# Copy the output, save it RIGHT NOW (1Password / encrypted USB).
```

### Step 2 — Update `bootstrap.toml`

```bash theme={null}
# Edit ~/.poindexter/bootstrap.toml — set POINDEXTER_SECRET_KEY = "<new key>"
chmod 600 ~/.poindexter/bootstrap.toml
```

### Step 3 — Wipe the dead encrypted rows

```bash theme={null}
docker exec -it poindexter-postgres-local psql -U poindexter -d poindexter_brain
```

```sql theme={null}
-- See what's gone
SELECT key, length(value) FROM app_settings
WHERE is_secret = TRUE AND value LIKE 'enc:v1:%';

-- Wipe them so they don't poison future reads
UPDATE app_settings SET value = '' WHERE is_secret = TRUE AND value LIKE 'enc:v1:%';
```

### Step 4 — Restart the worker so it picks up the new key

```bash theme={null}
docker restart poindexter-worker
```

### Step 5 — Re-seed every secret

Follow [`secret-rotation.md`](./secret-rotation) — go through the inventory and rotate / re-add each one.

### Verification

```bash theme={null}
# No more enc:v1: blobs with empty values
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT key FROM app_settings WHERE is_secret=TRUE AND value=''"
# Expected: empty after you've re-seeded all the secrets you actually use.

# Trigger a real downstream that uses a secret (e.g. Telegram alert).
# Mint a fresh JWT for the CLI client first — see `poindexter auth
# mint-token --client-id <pdx_xxx> --client-secret <secret>`.
curl -s -X POST http://localhost:8002/api/test/notify-operator \
  -H "Authorization: Bearer $JWT"
# Expected: Telegram message arrives.
```

***

## Per-service recovery (subordinate playbooks)

For when a single service is down but the rest of the stack is fine. These were the original disaster-recovery entries — kept here as a quick reference.

### Brain Daemon — no Telegram alerts, no `brain_decisions` rows

```bash theme={null}
docker ps | grep brain-daemon
docker compose -f docker-compose.local.yml up -d brain-daemon
cat ~/.poindexter/heartbeat   # Should be < 5 min old
```

The container has `restart: unless-stopped`, so Docker relaunches it automatically on crash. (The legacy OS-level watchdog was retired when the brain was containerized.) If it's *up* but the heartbeat is stale, it's hung — `docker restart poindexter-brain-daemon`.

### Content Worker (FastAPI) — pipeline stalled, `/api/health` fails

```bash theme={null}
curl -s http://localhost:8002/api/health
docker compose -f docker-compose.local.yml up -d worker
# Or full stack: bash scripts/start-stack.sh
```

### Ollama — content generation fails, "Ollama Unresponsive" alert

```bash theme={null}
curl -s http://localhost:11434/api/tags
ollama serve   # If not running as service
ollama list    # Verify required models present
```

### image-gen Server — posts publishing with Pexels stock images

```bash theme={null}
curl -s http://localhost:9836/health | python -m json.tool
docker compose -f docker-compose.local.yml up -d image-gen-server
# If torch/torchvision mismatch:
docker exec poindexter-image-gen-server pip install --upgrade torchvision peft
docker restart poindexter-image-gen-server
```

### Vercel — site returns 500 / blank page

```bash theme={null}
curl -sf https://www.gladlabs.io
gh run list --repo Glad-Labs/poindexter --limit 3
gh run rerun <run_id> --repo Glad-Labs/poindexter
```

### Grafana — dashboards down

```bash theme={null}
docker compose -f docker-compose.local.yml up -d grafana
curl -s http://localhost:3000/api/health
# Dashboards are file-provisioned and reload on restart.
```

The brain daemon alerts via Telegram **directly** — Grafana down does NOT mute alerts.

### OpenClaw Gateway — bot unreachable via Discord/Telegram

```bash theme={null}
curl -s http://127.0.0.1:18789/
# Windows restart:
start "OpenClaw Gateway" cmd /d /c "%USERPROFILE%\.openclaw\gateway.cmd"
```

`openclaw-watchdog.ps1` auto-restarts on health failure.

### GPU metrics stale ("GPU Metrics Stale" alert)

```bash theme={null}
curl -s http://localhost:9835/metrics | head -5
# Restart scraper (Windows):
taskkill /IM python.exe /FI "WINDOWTITLE eq nvidia-smi-exporter"
pythonw scripts/nvidia-smi-exporter.py
```

***

## Backup hygiene — preventing the next disaster

* Postgres is backed up automatically in tiers (see [`backups.md`](./backups)): the `backup-hourly` / `backup-daily` containers (24 hourly + 7 daily dumps under `~/.poindexter/backups/auto/`) plus the worker's `DbBackupJob` (`scripts/db-backup-local.sh`, flat 14-day dumps). The brain's `backup_watcher` restarts a stalled tier before paging, and a nightly `restore_test_probe` proves a dump actually restores. If you don't see fresh dumps every morning, that automation is broken — fix it before the next incident, not during.
* **`bootstrap.toml` should be in your password manager** as a secure note. If it dies, you're in CONFIG-1 + CONFIG-2 territory — minimum 1 hour of pain.
* **Enable off-machine (Tier 2) backups** so a drive failure / theft / ransomware event doesn't take your only copy with it. Run `poindexter backup setup` to ship the daily dumps to an S3-compatible bucket via restic (see [`backups.md`](./backups)); recover with **DB-4** above. Save the restic password offline when the wizard prints it.
* **`POINDEXTER_SECRET_KEY` is the doomsday key.** Print it on paper, put it in a fireproof safe. If you lose it AND the live `app_settings` table, you cannot recover the encrypted secrets — only re-issue them.

***

## See also

* [`incident-response.md`](./incident-response) — alert routing and triage
* [`secret-rotation.md`](./secret-rotation) — rotating individual secrets
* [`troubleshooting.md`](./troubleshooting) — known-symptom debugging entries
* [`local-development-setup.md`](./local-development-setup) — fresh setup walkthrough
* [`ci-deploy-chain.md`](./ci-deploy-chain) — how Vercel deploys are wired
* [`backups.md`](./backups) — backup tiers, retention, and the brain backup-watcher / restore-test probe
* `scripts/db-backup-local.sh` — the `DbBackupJob` backup script (flat-dir tier)
* `src/cofounder_agent/plugins/secrets.py` — encryption module reference
* `scripts/dr-reimport-posts-from-r2.py` — re-import published posts from R2 into a fresh DB (DB-2 Step 6)

## Contact

* Operator: configured via `app_settings.support_email` / `app_settings.privacy_email`
* Telegram: brain daemon sends alerts to configured `chat_id`
* Discord: ops channel receives Grafana alerts