Operations

Disaster Recovery

Backup, restore, and what to do when things break.

#Disaster recovery

Restore procedures for each stateful component, plus what you can and can't recover automatically.

#Components

Store Hosts what Backup
Convex App state including Convex Auth users/sessions Automatic snapshots
Cloudflare Worker Deployed code Git
AWS SES Verified domains, Configuration Set, SNS topic Re-bootstrap via in-app setup
Stripe Customer + subscription state Stripe is source of truth

#Convex

#Hosted (convex.dev)

Convex takes automatic snapshots. To restore:

npx convex import snapshot.zip              # if exported manually
# or restore from the Convex dashboard: Deployment → Backups

To export manually (e.g., monthly off-site copy):

npx convex export --path backups/$(date +%F).zip

#Self-hosted

Your Convex instance persists to whatever storage you configured. Back up that volume on the same cadence as any other stateful service.

#Re-bootstrapping SES

If AWS SES state is lost (rare - usually means an account change), the in-app Settings → AWS SES flow recreates the Configuration Set, SNS topic, and event subscription. Domains must be re-verified (DKIM TXT records are idempotent; the same records still work after re-initiating).

#Application code

Git. Every deployed version of the Cloudflare Worker is tagged with the git sha via pnpm deploy. Roll back by checking out the previous sha and re-deploying.

#Order of recovery

  1. Cloudflare Worker - restores the app surface
  2. Convex - restores project data
  3. Postgres - restores sessions (users will re-authenticate)
  4. AWS SES - re-verify domains if identities were deleted
  5. Stripe webhook - if the secret rotated, update STRIPE_WEBHOOK_SECRET

#What you can't recover

  • In-flight scheduled emails during a Convex snapshot restore
    • the scheduler state is part of the Convex snapshot, so restoring to an older point rewinds scheduled sends. Anything scheduled after the snapshot is lost. Keep this window small with frequent exports.
  • SES send-logs older than 14 days - CloudWatch retention default. Increase if you need longer.

#Testing restores

Once per quarter:

  1. Export a Convex snapshot.
  2. Create a staging deployment.
  3. Import the snapshot.
  4. Run the smoke tests from deploy.md.

If that passes, the production restore path is proven.