Documentation / Observability

Job monitors

Watch a job in one repository and get a signed webhook or an email when a run fails and when it recovers; testing a monitor, muting it, and what it says when it cannot see.

A monitor watches one job in one repository and sends a message when a run of it fails, and another when it recovers. You set it up on the Monitors tab of Runners in the dashboard.

Monitors are not switched on for every organisation yet. Where they are not, the Monitors tab does not appear and nothing below does anything. Your jobs and your other notifications are unaffected either way.

You choose a repository, a workflow, a job and the branches to watch: the default branch, every branch, or branches matching a pattern. * stands for any workflow or any job. The monitor alerts on a failed run of that job.

When a run fails the monitor opens one incident and sends one message. Further failures of the same job while the incident is open are counted on it and do not send another message. The first run that passes closes the incident and sends a recovery message, unless you switched recovery messages off.

To watch one step of a job, fill in Step with the step's name. * works here too: Run tests (*) matches every step that starts with Run tests (. The monitor alerts only when that step ends in failure. A job that fails in a different step sends nothing, and a matching step that ended skipped does not count as a failure.

GitHub reports a job's steps a little after the job ends, so a step monitor can take a couple of minutes: the one we timed sent its message about two minutes after the job ended.

A monitor can instead alert when a phrase appears in a job's log. Choose A log pattern and give the words or phrase to look for. This option is offered only when the log search is switched on for your organisation (see Searching logs), because the monitor uses the log search to look. That means it looks at the error and warning lines the log search keeps, not every line of output, and a phrase your tool printed as ordinary output is not found.

The log search indexes a job about ten minutes after it finishes, so a log monitor takes longer to alert than a result monitor: the one we timed sent its message about 12 minutes after the job ended. The monitor waits up to 13 minutes after the job finished before it decides the phrase is not there. The message names the phrase you chose and links to the job; it does not copy any of the log.

This page does not yet describe how a log monitor closes its incident.

Choose one or more of the channels you have already added under Settings → Notifications: a signed webhook, a Slack channel or an email address. A webhook message is signed in the same way as every other notification, so the check on Notifications verifies it. A channel does not need to subscribe to any event for a monitor to use it, and a channel that is switched off cannot be chosen.

Every message names the monitor, the condition in words, the repository, the workflow, the job and the branch, and links to the job's page.

Test this monitor sends a made-up job through the same path a real one takes and sends the alert to every channel the monitor uses. The message starts with [DRILL] and says nothing is wrong. A monitor is marked as tested only when the channels accepted the test alert. Changing a monitor's job, condition or channels clears that mark.

Muting a monitor stops its messages without deleting it. Unmute it and it alerts again from the next failing run.

The engine reports finished jobs to the dashboard. If no report has arrived for more than 15 minutes, the page says so, with the time of the last report. A monitor in that state is not shown as healthy. The engine reports when a job finishes, so a fleet that ran no job reads the same way.