Workers & notifications

The worker process

scripts/worker.ts starts one pg-boss instance and registers all queues and schedulers. One instance is sufficient. Locally: pnpm worker. In a container: node --import tsx scripts/worker.ts, run directly with node so that SIGTERM reaches the graceful shutdown handler.

The worker needs DATABASE_URL and the same SETTINGS_ENCRYPTION_KEY as the web app, plus APP_URL, because it builds links to the status page when sending. It does not need SESSION_SECRET.

Queue / job	Purpose	Trigger
email-send	Sends e-mail (dispatch via lib/email/)	Event
sms-send	Sends SMS via Twilio	Event
webhook-dispatch	Delivers webhooks, signed with HMAC	Event
push-send	Sends web push notifications	Event
maintenance-transitions	Sends maintenance_started once for running windows without a start marker, and silently sets windows past their end to completed. The two sweeps run independently.	Every minute
uptime-check-sweep	Runs HTTP checks for due, unpaused components with checks enabled	Every minute
notification-event-reaper	Marks e-mail events stuck in pending/processing for more than 30 minutes as failed	Every 10 minutes
failed-notification-alert	Logs a warning "notification events failed in the last window" with the count and up to 50 event IDs, if e-mail events failed in the last 10 minutes. A basis for log-based alerting; nothing is sent.	Every 10 minutes
uptime-check-retention	Deletes uptime check results older than 30 days	Daily at 03:00
verify-token-cleanup	Clears the verification token of confirmed subscribers whose confirmation is older than 30 days	Daily at 03:00

On start, the worker also runs ensureNotificationTemplateRows(). If an event type has no template row, it is created with all channels enabled.

Dispatch pipeline

Every notification follows the same path:

1. A domain action calls dispatchNotification (lib/notifications/notify.ts) with an event type and the affected entity IDs.
2. Mute gates are evaluated: if all affected components have notifications disabled, nothing is sent (skipped_disabled). Per-channel flags on each component are checked separately.
3. Template gates are evaluated per channel. Each channel (mail, sms, webhook, push) has its own enable flag in notification_templates. Fail-closed: no row means no sending.
4. Mail-only types (MAIL_ONLY_EVENT_TYPES in lib/notifications/mail-only.ts, currently double_optin) are sent by e-mail only. Webhook, SMS and push are skipped for them, regardless of the flags.
5. Delivery rows are created per channel:
6. E-mail: one row in notification_events and one job, if a provider is configured and there are recipients.
7. Webhook, SMS, push: one row per endpoint, recipient or subscription in webhook_deliveries, sms_deliveries or push_deliveries, and one job per row.
8. The processor sends the message. Each delivery gets up to 3 attempts with a 2-second and then 4-second backoff. The result is stored in the channel's delivery table.

Channels are independent. A disabled e-mail channel does not block SMS or webhooks.

Channels

Channel	Recipients	Transport	Notes
E-mail	User accounts and confirmed subscribers (depending on subscription mode), per-component notification address	SMTP (nodemailer) or Resend	Templates are editable per event type in the admin UI. Subscriber mails use double opt-in. Sent in recipient chunks via BCC.
Webhook	Configured endpoints, optional filter by event type	HTTP POST, 10-second timeout per attempt	Headers X-SLAtus-Signature: sha256=<hex>, X-SLAtus-Event and X-SLAtus-Delivery. Target types generic, slack, teams, discord format the body for the target system. Secrets can be rotated.
SMS	Admin-managed phone numbers	Twilio REST API	The message is built from a fixed builder and truncated to 300 characters. No self-registration.
Web push	Browser subscriptions created by visitors	web-push with VAPID keys	Subscriptions that return HTTP 404 or 410 are removed automatically.

For webhooks, the HMAC signature is calculated over the formatted body that is actually sent.

Event types

The event types are defined in NOTIFICATION_EVENT_TYPES (lib/db/schema.ts) and include: double_optin, outage_started, service_operational, service_outage_change, incident_created, incident_updated, incident_resolved, incident_reviewed, incident_closed, maintenance_scheduled, maintenance_started, maintenance_completed and sla_report_started.

double_optin is mail-only. On the server, fanOutEventTypesSchema (lib/validations/event-types.ts, event filters of SMS, webhook and push) and setNotificationTemplateChannelEnabledAction (channel flags) reject this type.

Retries and failures

• Retries happen only in the processor, so there is exactly one retry layer. pg-boss uses retryLimit: 1 solely as crash protection.
• No processor throws. An unexpected exception sets the delivery to failed with lastError "unexpected error: …". For the e-mail processor, sent_chunks stays unchanged in that case.
• E-mails are sent in recipient chunks. notification_events.sent_chunks records how many chunks were already delivered. Every attempt, the pg-boss crash retry and the manual resend continue from there, so nobody gets a mail twice.
• E-mail events that failed all attempts can be resent from /admin/notification-events. Resetting sets the row back to pending and enqueues it again. For partially delivered events, the button reads Resend remaining.
• failed-notification-alert reports failed e-mail events in the log every 10 minutes (Issue #126).