Operations
Disaster Recovery
Backup, restore, and what to do when things break.
#Disaster recovery
Restore procedures for each stateful component, plus what you can and can't recover automatically.
#Components
| Store | Hosts what | Backup |
|---|---|---|
| Convex | App state including Convex Auth users/sessions | Automatic snapshots |
| Cloudflare Worker | Deployed code | Git |
| AWS SES | Verified domains, Configuration Set, SNS topic | Re-bootstrap via in-app setup |
| Stripe | Customer + subscription state | Stripe is source of truth |
#Convex
#Hosted (convex.dev)
Convex takes automatic snapshots. To restore:
npx convex import snapshot.zip # if exported manually
# or restore from the Convex dashboard: Deployment → Backups
To export manually (e.g., monthly off-site copy):
npx convex export --path backups/$(date +%F).zip
#Self-hosted
Your Convex instance persists to whatever storage you configured. Back up that volume on the same cadence as any other stateful service.
#Re-bootstrapping SES
If AWS SES state is lost (rare - usually means an account change), the in-app Settings → AWS SES flow recreates the Configuration Set, SNS topic, and event subscription. Domains must be re-verified (DKIM TXT records are idempotent; the same records still work after re-initiating).
#Application code
Git. Every deployed version of the Cloudflare Worker is tagged with the git
sha via pnpm deploy. Roll back by checking out the previous sha and
re-deploying.
#Order of recovery
- Cloudflare Worker - restores the app surface
- Convex - restores project data
- Postgres - restores sessions (users will re-authenticate)
- AWS SES - re-verify domains if identities were deleted
- Stripe webhook - if the secret rotated, update
STRIPE_WEBHOOK_SECRET
#What you can't recover
- In-flight scheduled emails during a Convex snapshot restore
- the scheduler state is part of the Convex snapshot, so restoring to an older point rewinds scheduled sends. Anything scheduled after the snapshot is lost. Keep this window small with frequent exports.
- SES send-logs older than 14 days - CloudWatch retention default. Increase if you need longer.
#Testing restores
Once per quarter:
- Export a Convex snapshot.
- Create a staging deployment.
- Import the snapshot.
- Run the smoke tests from deploy.md.
If that passes, the production restore path is proven.