Incidents & maintenance

Incidents

An incident is a communicated disruption. It has a title, an impact (none, minor, major, critical), the affected components with their status, and a timeline of public updates.

Lifecycle

investigating → identified → monitoring → resolved → reviewed → closed

• The first four statuses are live and can be set through timeline updates. Transitions can go in both directions. For example, monitoring back to identified is allowed.
• resolved sets resolved_at. An incident can be reopened by adding an update, and resolved_at is reset.
• reviewed and closed are frozen. They accept no more timeline updates. They are set by the review and close actions. The exact outage times can still be corrected.
• Every incident starts with an automatic timeline entry with status occurred, which carries the time the incident actually happened.

Timeline and internal notes

• Timeline updates (incident_updates) are public. Each update has its own occurred_at, which is shown instead of the creation time. The timeline is sorted by it.
• Internal notes (incident_notes) are private team notes. They are append-only, never shown publicly, and are separate from the timeline and from the review text.
• The review text (review_notes) documents root cause and measures after resolution.

Linking to existing outages

When an incident is created, open outages that are not yet linked can be selected. The incident is attached to these outage windows (component_status_events.source_incident_id), and the existing windows are kept as they are. The incident's occurred_at becomes the earliest of the form time and the earliest outage start (computeIncidentOccurredAt). Components of the selected outages are added automatically.

Impact and public display

The admin incident list groups entries: active incidents first, then resolved and reviewed ones. Closed incidents only appear after clicking Show all incidents (query parameter all=1). The public page lists incidents and outages together, sorted by severity (lib/domain/severity.ts). If show_resolved_minutes is greater than 0, a resolved incident stays visible for that many minutes, marked as resolved in green. The default is 0.

Status whitelist

Only investigating, identified, monitoring and resolved are accepted as starting status through the API and the form. This prevents an incident from being created directly as reviewed or closed.

Maintenance windows

A maintenance window has a title, description, a scheduled start and end, and the affected components. The database enforces scheduled_end > scheduled_start.

Phase	How it is reached
scheduled	Created in the admin UI or through the API
in_progress	Not stored; derived from the time window when reading, as soon as the start time is reached
completed	Manually, with "Mark complete", or automatically at the scheduled end (checked every minute)

• While a window is in progress, the affected components show under_maintenance.
• Maintenance does not count as an active outage. It appears in the "Active Maintenances" counter and the maintenance section of the public page, not in the outage counter.
• Creating a maintenance window sends maintenance_scheduled.
• A notification maintenance_started is sent once when the window begins (maintenances.notified_started_at makes this idempotent). The automatic end is silent. Only a manual completion sends maintenance_completed.
• If the worker was down when the window began, the start notification is sent later, as long as the window is still running. If sending fails, or there are no recipients or no enabled template, the marker is reset and the next run tries again. The notification is only lost if the window has already ended. The automatic completion at the scheduled end is always caught up.
• In the admin UI, admins and service managers can create, edit and complete maintenance (see Roles & login). A completed maintenance window can no longer be edited.