Skip to main content

Disaster Recovery Runbook

Last reviewed: 2026-06-15 Audience: solo operator (Matt) at 2am during an incident Prereqs: Local PC online, Docker running, gh CLI authed, poindexter CLI installed, ~/.poindexter/bootstrap.toml accessible (or you have the database_url + POINDEXTER_SECRET_KEY somewhere safe) This runbook covers catastrophic-loss scenarios — the kind where “just restart the container” is not enough. For per-alert triage and routing, see incident-response.md. For known-pattern symptom debugging, see troubleshooting.md.

Scenarios this covers

  • DB-1. PostgreSQL container is healthy but data is corrupted / wrong / missing rows
  • DB-2. poindexter-postgres-local Docker volume was wiped or destroyed (full data loss)
  • DB-3. A migration ran half-way, left the schema in an inconsistent state
  • HOST-1. New machine — rebuild the entire local stack from scratch (laptop died, fresh OS install)
  • HOST-2. Docker Desktop / WSL2 corrupted — need to reset Docker but keep the data
  • CONFIG-1. bootstrap.toml lost — need to reconstruct it
  • CONFIG-2. POINDEXTER_SECRET_KEY lost — every encrypted secret in app_settings is unreadable
For per-service recovery (worker crashed, image-gen degraded, Vercel down), see the Per-service recovery section at the bottom.

Quick triage flowchart


Procedure: DB-1 — Restore PostgreSQL from backup

Symptoms. Container is running, you can connect, but data is wrong (missing rows, bad migration, accidental DELETE, etc.).

Step 1 — Stop the worker so it doesn’t fight you

Step 2 — List available backups

Two backup tiers write under ~/.poindexter/backups/ — see backups.md for the full design:
  • In-stack container tier (primary). backup-hourly / backup-daily write ~/.poindexter/backups/auto/{hourly,daily}/poindexter_brain_*.dump (last 24 hourly + 7 daily).
  • DbBackupJob. The worker’s scheduled job runs scripts/db-backup-local.sh into the flat ~/.poindexter/backups/ (poindexter-db-*.dump, 14-day retention).
If every location is empty or stale, your backup automation is broken — go to DB-2 and treat this as a volume-loss event (your last good state is whatever’s in git/Vercel/R2). The restore steps below take whichever dump you picked here.

Step 3 — Verify the backup is restorable BEFORE you drop the live DB

Step 4 — Drop and recreate the DB

Step 5 — Restore from the backup

Expected output: a stream of pg_restore: processing data for table "..." lines, then exit 0. Some warnings about extensions (pgcrypto, vector) are normal — they’ll be re-created.

Step 6 — Re-apply any migrations newer than the backup

Step 7 — Bring the worker back

Verification


Procedure: DB-2 — poindexter-postgres-local data volume was wiped

Symptoms. docker ps shows postgres is up, but psql -c "\dt" returns “Did not find any relations” — the database exists but is empty. Or the container won’t even start because the volume mount is missing. Worst case — you lost the volume AND have no backup. The static export on R2 / public-site frontend still has the published posts. The git history has the schema migrations. Everything else is gone.

Step 1 — Confirm the volume is actually gone

Step 2 — Stop everything that talks to the DB

Step 3 — IF you have a backup, restore it

Same as DB-1 Steps 4-7. If you got here because backups were missing too, continue.

Step 4 — IF no backup exists, rebuild the schema

Step 5 — Recreate the secrets that lived in app_settings

This is the painful part. Encrypted secrets are gone forever. You need to:
  1. Go to secret-rotation.md and rotate every API key listed in the inventory (Telegram, Discord, OpenAI, Anthropic, Lemon Squeezy, Resend, Cloudinary, Pexels, etc.) — treat all of them as compromised because the encrypted blobs may have leaked with the volume backup.
  2. Re-seed each one with poindexter settings set <key> "<value>" --secret (see secret-rotation.md “Re-seeding from scratch”).

Step 6 — Re-import published posts from R2

The static export on R2 (static/posts/index.json + static/posts/<slug>.json) is the source of truth for what was live. scripts/dr-reimport-posts-from-r2.py replays those JSON files back into the posts table.
The script resolves the database URL from bootstrap.toml automatically. Pass --database-url to override. Pass --r2-url if NEXT_PUBLIC_STATIC_URL env var is unset and the hardcoded default is stale. The upsert is keyed on slug so re-running fills gaps without overwriting rows already restored from a backup.
Note: pipeline_tasks rows and pipeline_versions rows are NOT re-imported (they live only in the DB). Only the published posts rows surface in the frontend, so the site is functional after this step. The missing task history means the auto-publish gate and edit-distance tracker start fresh — that’s acceptable for a DR scenario.

Verification


Procedure: DB-3 — Migration ran half-way, schema is inconsistent

Symptoms. A poindexter migrate up errored mid-run. poindexter migrate status shows a migration as “pending” but you can see partial DDL was already applied (e.g., a column exists that shouldn’t be there until that migration completes).

Step 1 — Capture the failure mode

Step 2 — Choose: roll forward or roll back?

Roll forward is preferred when:
  • The failed migration is small and you can eyeball what didn’t run
  • Re-running it (after manual cleanup) is safe (the DDL is idempotent or you can drop the half-applied object)
Roll back is preferred when:
  • The migration is complex
  • You have a recent backup and can afford to lose a few minutes of activity

Step 3a — Roll forward

Step 3b — Roll back

If the failing migration has no down()/rollback_migration(), the down command will skip it. In that case, restore from backup (DB-1) to the last good state.

Step 4 — Bring services back up

Verification


Procedure: DB-4 — Restore from the off-machine (Tier 2) restic repo

Scenario. The machine is gone — drive failure, theft, ransomware — so the in-stack ~/.poindexter/backups/auto/ dumps went with it. You configured Tier 2 (backups.md), so the daily dumps live in an S3-compatible bucket as an encrypted restic repo. This restores from there.
Hard dependency: the restic password. Tier 2’s password is stored encrypted in app_settings — which is gone in this scenario along with POINDEXTER_SECRET_KEY. You restore from the offline copy the setup wizard told you to save (password manager / fireproof safe). Without it the repo is unrecoverable — restic encryption with a lost password is final. If you don’t have it, this procedure cannot help; skip to DB-2 (rebuild from migrations + R2).

Step 1 — Install restic on the recovery machine

Step 2 — Point restic at the repo with your offline credentials

You need four values: the repo URL (the offsite_backup_repository you chose — s3:https://<endpoint>/<bucket>/<path>), the offline restic password, and the S3 access key id + secret. On a fresh machine app_settings is gone, so these come from your offline copy / the bucket provider’s console.

Step 3 — Confirm the repo is reachable and list snapshots

Step 4 — Restore the latest snapshot to a local directory

Step 5 — pg_restore the recovered dump

Bring up an empty Postgres (HOST-1 Steps 1-5 if this is a fresh machine), then restore the recovered .dump exactly like DB-1 Step 5:
Then re-apply any newer migrations (DB-1 Step 6) and bring services up (DB-1 Step 7). Because the restored dump’s app_settings still holds the old encrypted secrets, you’ll also need the original POINDEXTER_SECRET_KEY to decrypt them — if that’s lost too, finish with CONFIG-2 (rotate + re-seed).

Verification


Procedure: HOST-1 — Rebuild the entire local stack on a new machine

Scenario. Laptop died, fresh OS install, or you’re moving to new hardware.

Step 1 — Install prerequisites

Step 2 — Clone the repo

Step 3 — Restore secrets

You need the following from a safe place (1Password, encrypted USB, second machine):
  • ~/.poindexter/bootstrap.toml — contains database_url and POINDEXTER_SECRET_KEY
  • A recent backup from ~/.poindexter/backups/ (or the cloud copy if you set one up)
If you don’t have either, skip ahead — we’ll generate a fresh bootstrap and rebuild from migrations + R2.

Step 4 — Install poindexter

Step 5 — Bring up the stack

Step 6 — Restore data

If you have a backup → follow DB-1 Steps 4-7 from “Drop and recreate the DB.” If no backup → run migrations to create empty schema, then go to DB-2 Steps 5-6.

Step 7 — Pull Ollama models

Step 8 — Verify everything

Step 9 — Re-pair MCP / OpenClaw / Telegram / Discord


Procedure: HOST-2 — Docker / WSL2 broken, keep the data

Symptoms. docker ps errors, Docker Desktop refuses to start, WSL2 is unresponsive. The data is fine — only the runtime is busted.

Step 1 — Try a soft reset first

Note. The Docker Engine Watchdog scheduled task (scripts/docker-watchdog.ps1, every 5 min) now performs this soft reset automatically. When the Docker Desktop process is alive but the engine is wedged (the WSL2-VM HCS_E_CONNECTION_TIMEOUT case), it re-checks after -WedgeConfirmSeconds (default 30) to rule out a transient blip, captures forensics to ~/.poindexter/logs/wedge-<timestamp>/ (host nvidia-smi, Hyper-V/vmcompute event channels, docker diagnose bundle), pings Telegram if telegram_bot_token is set in bootstrap.toml, then runs wsl --shutdown and restores the stack via start-stack.sh. The manual steps below are the fallback if the watchdog is disabled or the recycle doesn’t take.

Step 2 — If that fails, reset Docker Desktop (preserves volumes by default)

In Docker Desktop GUI: Settings → Troubleshoot → “Reset to factory defaults” WARNING — this nukes named volumes. Use “Clean / Purge data” only if you have a current backup. The safer path: uninstall Docker Desktop, reinstall same version, restart. Named volumes survive a reinstall as long as you don’t delete the WSL2 distro docker-desktop-data.

Step 3 — Restart the stack

Step 4 — Verify data integrity

If the counts are way off, treat as DB-2 (volume loss).

Procedure: CONFIG-1 — Lost bootstrap.toml

Symptoms. Worker won’t start. Logs say notify_operator(): no DATABASE_URL resolved, then sys.exit(2).

Step 1 — Reconstruct the file

You need the postgres password (find it: docker exec poindexter-postgres-local printenv POSTGRES_PASSWORD) and the POINDEXTER_SECRET_KEY (this is the killer — see CONFIG-2 if lost).

Step 2 — Restart worker

Verification


Procedure: CONFIG-2 — Lost POINDEXTER_SECRET_KEY

Symptoms. Worker boots, but every get_secret(...) call raises SecretsError: Could not decrypt. Telegram/Discord/Vercel revalidation/Lemon Squeezy webhook verification all 401. app_settings rows with is_secret=true show enc:v1:... blobs that can never be decoded again. There is no recovery for the encrypted blobs. Encryption with AES-256 + a lost key is final. You must rotate every secret.

Step 1 — Generate a new key

Step 2 — Update bootstrap.toml

Step 3 — Wipe the dead encrypted rows

Step 4 — Restart the worker so it picks up the new key

Step 5 — Re-seed every secret

Follow secret-rotation.md — go through the inventory and rotate / re-add each one.

Verification


Per-service recovery (subordinate playbooks)

For when a single service is down but the rest of the stack is fine. These were the original disaster-recovery entries — kept here as a quick reference.

Brain Daemon — no Telegram alerts, no brain_decisions rows

The container has restart: unless-stopped, so Docker relaunches it automatically on crash. (The legacy OS-level watchdog was retired when the brain was containerized.) If it’s up but the heartbeat is stale, it’s hung — docker restart poindexter-brain-daemon.

Content Worker (FastAPI) — pipeline stalled, /api/health fails

Ollama — content generation fails, “Ollama Unresponsive” alert

image-gen Server — posts publishing with Pexels stock images

Vercel — site returns 500 / blank page

Grafana — dashboards down

The brain daemon alerts via Telegram directly — Grafana down does NOT mute alerts.

OpenClaw Gateway — bot unreachable via Discord/Telegram

openclaw-watchdog.ps1 auto-restarts on health failure.

GPU metrics stale (“GPU Metrics Stale” alert)


Backup hygiene — preventing the next disaster

  • Postgres is backed up automatically in tiers (see backups.md): the backup-hourly / backup-daily containers (24 hourly + 7 daily dumps under ~/.poindexter/backups/auto/) plus the worker’s DbBackupJob (scripts/db-backup-local.sh, flat 14-day dumps). The brain’s backup_watcher restarts a stalled tier before paging, and a nightly restore_test_probe proves a dump actually restores. If you don’t see fresh dumps every morning, that automation is broken — fix it before the next incident, not during.
  • bootstrap.toml should be in your password manager as a secure note. If it dies, you’re in CONFIG-1 + CONFIG-2 territory — minimum 1 hour of pain.
  • Enable off-machine (Tier 2) backups so a drive failure / theft / ransomware event doesn’t take your only copy with it. Run poindexter backup setup to ship the daily dumps to an S3-compatible bucket via restic (see backups.md); recover with DB-4 above. Save the restic password offline when the wizard prints it.
  • POINDEXTER_SECRET_KEY is the doomsday key. Print it on paper, put it in a fireproof safe. If you lose it AND the live app_settings table, you cannot recover the encrypted secrets — only re-issue them.

See also

  • incident-response.md — alert routing and triage
  • secret-rotation.md — rotating individual secrets
  • troubleshooting.md — known-symptom debugging entries
  • local-development-setup.md — fresh setup walkthrough
  • ci-deploy-chain.md — how Vercel deploys are wired
  • backups.md — backup tiers, retention, and the brain backup-watcher / restore-test probe
  • scripts/db-backup-local.sh — the DbBackupJob backup script (flat-dir tier)
  • src/cofounder_agent/plugins/secrets.py — encryption module reference
  • scripts/dr-reimport-posts-from-r2.py — re-import published posts from R2 into a fresh DB (DB-2 Step 6)

Contact

  • Operator: configured via app_settings.support_email / app_settings.privacy_email
  • Telegram: brain daemon sends alerts to configured chat_id
  • Discord: ops channel receives Grafana alerts