Skip to content

Operations, backup & restore

What must be running

Component Required for How to check
Web app Status page, admin, public API, SLA exports GET /api/health returns 200
Worker E-mail, SMS, webhook and push delivery, uptime checks, maintenance start and automatic maintenance completion, warning on failed notifications, cleanup E-mail events in Information → Events move from pending to sent
PostgreSQL Everything Health check fails with 503 if the database is unreachable

Danger

Without the worker, the status page still works, but no notifications are delivered, uptime checks do not run, and maintenance windows past their end are only completed once the worker runs again. Monitor the worker process.

/api/health checks only the process and the database connection. It contains no business data and is suitable for external probes.

Startup requirements

  • The web app only starts with SESSION_SECRET set. If it is missing, startup aborts with an error. If it is shorter than 32 characters, a warning appears in the log. Generate it with openssl rand -hex 32.
  • The worker needs DATABASE_URL, the same SETTINGS_ENCRYPTION_KEY as the web app, and APP_URL for links in notifications.
  • All environment variables, including the rate-limit thresholds (AUTH_RATELIMIT_*, PUBLIC_FORM_RATELIMIT_*), are described in Deployment, migrations & CI.

Backup

What to back up: the PostgreSQL database. It contains the complete state: incidents, SLA reports with their confirmed snapshots, settings, subscribers, API tokens and the job queue tables.

What to back up separately: the environment secrets, because they are not stored in the database:

  • SETTINGS_ENCRYPTION_KEY. Without it, encrypted passwords and keys in the database cannot be read.
  • SESSION_SECRET
  • ADMIN_PASSWORD_HASH, if still used

Create a dump:

pnpm db:backup [target]    # writes backups/slatus-<database>-<timestamp>.dump

The pg_dump version must be at least the version of the database server. Keep several generations, store at least one copy off-site, and back up daily, for example with cron.

Introducing or rotating the encryption key

  • If SETTINGS_ENCRYPTION_KEY was only set later, run pnpm db:encrypt-secrets once. It encrypts the secrets that were stored as plaintext so far.
  • To rotate, set the previous key as SETTINGS_ENCRYPTION_KEY_OLD and the new one as SETTINGS_ENCRYPTION_KEY, run pnpm db:rotate-secret-key, then remove SETTINGS_ENCRYPTION_KEY_OLD.
  • Against a non-local database, both scripts only run with --yes-i-am-sure. Back up first.

Restore

Restore is destructive. pnpm db:restore asks before overwriting (type yes) and refuses a DATABASE_URL that contains "production" without --yes-i-am-sure. The pg_restore version must be at least the version of the database server.

  1. Stop the web app and the worker.
  2. Make sure PostgreSQL runs and an empty target database exists. Restore does not create the database.
  3. Restore the dump:

bash pnpm db:restore backups/slatus-<…>.dump

  1. Apply pending migrations: pnpm db:migrate
  2. Start the web app and the worker.
  3. Check the health endpoint and open the status page.

Warning

pnpm db:seed and pnpm db:seed:history delete data and are meant for development and tests only. They only run against a local host or a database whose name ends in _dev, _e2e or _test, otherwise only with --yes-i-am-sure. All db:* scripts read .env.

Migrations

Migrations are forward-only. They are applied with pnpm db:migrate before the new version starts. If a migration fails, the notes on recovery are in docs/context/migrations.md in the repository. Do not edit applied migrations.

Routine checks

Interval Check
Daily Failed events in Information → Events. Resend or investigate. The worker also reports failed e-mail events as a warning in the log every 10 minutes.
Daily Backup file exists and is recent
Weekly Expired API tokens and unused subscribers
Monthly Open SLA reports and outages waiting for confirmation
After each change The public page and one test notification

Logs

The application writes structured logs to standard output. In production they are JSON, in development readable text. Set the log level with LOG_LEVEL (debug to error) and the format with LOG_FORMAT (json or pretty). Use your container platform's log viewer to search them.

Warnings to watch for:

  • "notification events failed in the last window": e-mail notifications have failed.
  • "client ip is private or loopback; reverse proxy may not forward the client ip": the reverse proxy does not forward the client IP, see below.

Security checklist

  • Remove ADMIN_PASSWORD_HASH once a database admin exists. You can also turn the bootstrap access off from a given date with ADMIN_BOOTSTRAP_DISABLED_AFTER.
  • Set SESSION_SECRET (at least 32 characters) and SETTINGS_ENCRYPTION_KEY to random values, and store them safely.
  • Run behind a reverse proxy with HTTPS that forwards the client IP, and set TRUSTED_PROXY_IP_HEADER to match: x-real-ip if the proxy sets X-Real-IP, or x-forwarded-for with TRUSTED_PROXY_COUNT if it only appends to X-Forwarded-For. With a fixed header, a client cannot fake the other one. Details are in Deployment, migrations & CI.
  • Check whether the thresholds for the public forms (PUBLIC_FORM_RATELIMIT_*) fit your visitor traffic.
  • Remove STATUS_PAGE_EMAIL_DRIVER=capture from the production environment, otherwise no mails are sent.
  • Give each person the lowest role that fits their task. See Roles & login.