Notifications and Alerts

Purpose#

Three separate things tell you when something needs attention, and confusing them is the most common mistake on this surface. They are:

The distinction that matters: /notifications reacts to a run that happened. /alerts reacts to a run that did not happen — or to a pattern across runs. A pipeline that silently stops being scheduled never fails, so it never sends a run notification. An alert rule is what catches it.

Global settings are not an alerting surface at all; they are the plumbing the other two depend on. See Notification configuration.

Before you begin#

Warning. Email channels do nothing until an administrator has configured the global SMTP transport, and nothing at all is delivered while the master switch is off. If a channel tests fine but never delivers, check the master switch first. See Notification configuration.

Per-resource notifications#

A notification config attaches a channel to one resource and one or more run events.

Channel kinds#

Events#

Events are a multi-select and at least one is required; the default is run_failed alone, which is the right default for almost everything.

Tip. run_succeeded on a nightly job is a habit worth resisting — a daily "it worked" email trains people to ignore the channel. Use run_failed, and use an alert rule to catch the case where nothing ran at all.

Attach a config#

1. Go to /notifications and choose New.
2. Pick the resource type and the specific resource.
3. Pick the channel kind and enter the target.
4. Choose the events.
5. Save, then use Test to send a sample message.

Notification configs attach to:

Note. For SQL Pipelines, notifications attach to the job, not to an individual task. A task that runs inside a job is covered by the job's notifications. If you want to be told about one task specifically, put it in its own job. (Alert rules work the other way round — see below.)

Targets are write-only#

The channel target is a credential: a webhook URL is enough to post into your Slack, and a routing key is enough to page your on-call.

So targets are encrypted at rest and never returned. Once saved, you see only a masked preview — for a short target, dots; for a longer one, the first few characters, an ellipsis, and the last four. Enough to tell two channels apart, not enough to reuse.

Consequences when editing:

• Leave the target blank to keep the existing one. Editing the events on a config does not require re-entering the URL.
• Entering a new value rotates the credential, replacing what was stored.
• There is no way to read a target back. If you lose the original URL, get a new one from the target system and rotate.

Email is the one exception where recipients are visible: they are addresses, not secrets.

Delivery failures never fail a run#

If a channel is unreachable — a revoked webhook, an SMTP outage, a rate-limited endpoint — the failure is recorded and counted in metrics, and the run continues to its normal conclusion.

This is deliberate: an alerting problem must not become a data problem. It does mean a broken channel is quiet by nature, which is a good reason to press Test after any change and to watch the notification failure metrics. See Monitoring.

Alert rules#

An alert rule watches a resource's run history and fires when it looks wrong.

What a rule watches#

Note. Alert rules target ETL tasks, while notification configs target ETL jobs. This is not a mistake in the docs: staleness is a property of a table being loaded (a task), while run outcome is a property of the orchestration unit (a job). Set both if you want both.

The freshness metric#

Fires when the resource has not succeeded recently enough:

• No successful run on record at all, or
• The most recent success is older than the threshold.

This is the rule that catches silence — a paused schedule, a job that stopped being triggered, a stream whose endpoint quietly started returning nothing. No run failed, so no notification config fired, and without this rule nobody would know.

Set the threshold generously relative to the cadence: for an hourly load, three or four hours, not seventy minutes. A tight threshold produces alerts every time a run is a little late.

The failure_streak metric#

Fires when the resource has failed N times in a row.

The rule looks back over the recent run history and counts consecutive failures from the most recent run backwards. Any non-failure status breaks the streak — a success obviously, but also a skipped or a still-running run.

Use it to distinguish a transient failure from a real outage: a threshold of 3 ignores a one-off network blip but catches a credential that expired.

Fire behaviour#

Rules are evaluated every 60 seconds. Each rule holds a state, ok or firing.

A rule delivers only on the transition from ok to firing. A rule that stays firing for six hours pages you once, not 360 times. When the condition clears, the state returns to ok, and the next breach delivers again.

Each rule records when it was last evaluated and when it last fired, so you can tell "never fired" from "fired and recovered".

Create and test a rule#

1. Go to /alerts and choose New rule.
2. Pick the resource type, then the resource.
3. Pick the metric and set its threshold.
4. Set severity.
5. Pick the channel kind and enter the target.
6. Save.
7. Use Evaluate on the row to check it now.

Evaluate runs the rule immediately and reports whether it is firing, why, and whether delivery succeeded. It honours the same transition rule, so it will not spam a channel that is already firing — it is a safe probe.

Rules can be paused (toggle Enabled off) rather than deleted, which is what you want during planned maintenance.

Schema-drift alerts#

Schema drift has its own dispatch path, and it behaves differently from everything above.

When a run encounters drift, the alert goes to every enabled notification channel on that resource, regardless of which events those channels selected. A channel configured only for run_failed still receives drift alerts.

Severity depends on the outcome:

For ETL tasks that maps onto the two drift settings: block_and_alert produces critical alerts, proceed produces warnings. See SQL pipelines and Schema drift policies.

Warning. Email feeds have no drift channel. Drift on an email feed is recorded in the run and counted in metrics, but it is not pushed to any channel. If you ingest from email, check its run history rather than waiting to be told. See Email feeds.

Drift dispatch still honours the global master switch, and like everything else here, a failing channel never fails the run.

What global settings own#

Settings → Notifications is admin-only and owns three things the per-resource surfaces depend on:

• The SMTP transport — host, port, credentials, TLS mode, sender address, timeout. Every email channel in the product uses it. Without a configured host, email channels cannot deliver.
• The master on/off switch — one flag that suppresses all notification dispatch. Useful during maintenance; dangerous to forget.
• The license-expiry channel — a single dedicated channel for license warnings, separate from every per-resource config, because a license problem affects the whole deployment. See Expiry and clock.

Full configuration detail is in Notification configuration.

Field reference#

Notification config#

Alert rule#

Failure modes#

See also#

• Notification configuration — SMTP, master switch, license-expiry channel
• Executions — looking, rather than being told
• Jobs and SQL pipelines — the resources being watched
• Schema drift policies
• Monitoring