Thresholds, quiet hours and reminders

We write when something changes state, never on every check.

What triggers a message#

Message	Severity
An outage is confirmed	critical
A certificate or domain is about to expire, or has expired	warning or critical
A firewall blocks our checks	warning
A deploy is noticeably slower	warning
A new target has never answered	info
An incident is resolved	same channels as the first alert

Thresholds per channel#

Each channel receives everything, warnings and outages, or outages only. A channel that did not get the first alert of an incident does not get its reminders or its resolution either, and a channel that got the alert always gets the resolution.

Reminders#

While an incident is open, reminders slow down: 5 min → 30 min → 2 h → 6 h → 24 h, then stop. Acknowledging an incident stops the reminders without closing the incident. Everyone who got the alert also gets the resolution.

Quiet hours#

E-mail, Slack, Discord and Telegram channels can have quiet hours. During them, warnings and notices wait for one morning summary. Outages always come through, quiet hours or not.

Many alerts at once#

If one server takes down many sites, a channel gets the first alert right away and the rest grouped into one message within 5 minutes. A channel receives at most 60 messages an hour; anything beyond that is grouped too, never dropped.