Jobs

Purpose#

A job runs a set of SQL Pipelines tasks together, in dependency order, as one unit with one status and one run history.

Running tasks one at a time is fine while you are building them. As soon as a staging table feeds a fact table which feeds a summary, you want the three to run in the right order, to stop sensibly when one fails, and to report as a single outcome. That is a job.

Jobs live at /jobs.

Before you begin#

• Tasks that already run cleanly on their own. A job is a bad place to debug a task. See SQL pipelines.
• One environment. A job is single-environment: every task in it must belong to the same environment as the job. Adding a task from a different environment is rejected. Promote the task into this environment first — see Environments and promotion.

Required role#

Both schedule.manage and task.run are environment-gated: a Developer holds them in unprotected environments only and is refused in a protected one with the reason protected_environment. The global admin role is an implicit Owner on every project.

Create a job and add tasks#

1. Go to /jobs and choose New job.
2. Name it and pick its environment.
3. Set its policies — failure, concurrency, retries, timeout (see Field reference).
4. Save.
5. Add at least one group.
6. Add tasks to the group.

Groups organise; they do not order#

This is the single most misunderstood thing about jobs, so it is worth being precise.

A group is a named bucket with a description and a display order. Groups exist so a forty-task job is readable — "staging", "facts", "marts" — and so the run detail is navigable.

Groups do not determine execution order. Neither does a task's position within a group.

Execution order is derived from the dependencies between the tasks themselves: Abrq DIP builds a DAG for the environment by reading the table references in each task's SQL, then runs the tasks in an order consistent with that DAG. A task that reads a table another task writes runs after it — regardless of which group either one is in.

Note. Group ordering and within-group sequence survive only as a tie-break for tasks the dependency graph cannot rank relative to each other. Do not use groups to express "this must run first". Express it in SQL: if task B reads what task A writes, the order is already correct and self- maintaining.

The practical consequence is a good one — the ordering cannot drift out of sync with reality, because it is reality. It also means a job's order changes when you edit a task's SQL, which is why the run detail records what actually ran.

If the tasks form a dependency cycle, the whole run is refused before any task executes, naming the cycle. Break the cycle — usually by splitting a task or introducing a staging table.

Execution today is sequential#

Tasks in a job run one after another. There is no parallel execution of independent branches yet.

Two settings relate to parallelism and are worth understanding honestly:

• max_parallelism on the job (0–64, default 0 meaning unbounded) and the deployment-wide cap ABRQ_JOB_MAX_PARALLELISM_CAP (default 0, also meaning no global cap). The intended semantics are smaller-wins: with a job set to 3 and a cap of 5 you get 3; with a job set to 8 and a cap of 5 you get 5; if either is 0 it does not constrain, and if both are 0 there is no bound.
• Today the job runner executes sequentially and does not read either value. The setting is stored, validated and displayed, but it does not change how a run behaves.

Warning. Set max_parallelism to record your intent if you like, but do not rely on it as a throttle — it does not currently limit anything. If you need to bound concurrent work, use the job's concurrency policy (below), which is enforced.

Failure and concurrency policies#

Failure policy#

Decides what happens to the rest of the job when a task fails.

With either of the continuing policies, a job that ends with both successes and failures reports partial_success. Under fail_job, any failure makes the job run failed.

Tip. fail_job is the right default for a nightly warehouse load, where a partial load is worse than no load. The continuing policies suit jobs that bundle genuinely independent pipelines for convenience.

Concurrency policy#

Decides what happens when a run is triggered while a previous run is still going — the common case for a schedule that fires faster than the job finishes.

Job parameters#

Job parameters let a job's tasks share values that you can change in one place — a cut-off date, a region code, a batch label.

Define a parameter#

1. Open the job and go to its Parameters tab.
2. Add a parameter with a name, a value, and a value type.

Use a parameter in SQL#

Reference it in a task's SQL with double braces:

SELECT *
FROM sales.orders
WHERE order_date >= '{{ cutoff_date }}'
  AND region = '{{ region_code }}';

Whitespace inside the braces is optional. There are no filters, no expressions, and no inline defaults — the name is replaced with the value, verbatim.

Things to know before you use them:

• Parameters apply to job runs only. Running that same task on its own, or from the task builder, performs no substitution — the placeholder is left as written and the SQL will fail. Test parameterised tasks by running the job.
• Values are read once per job run. The run snapshots every parameter at the start, so all tasks in one run see a consistent set.
• There are no per-run parameter overrides. Triggering a run takes no inputs. To run with a different value, change the parameter first, then run.
• An undefined placeholder fails that task, with an error naming the parameter it could not resolve. It does not silently render as empty.
• Substitution is plain text, not SQL binding. It is not protection against SQL injection. Treat parameter values as trusted operator input, never as something an end user supplies.
• Values are not secret. There is no masking or write-only handling. Do not put credentials in job parameters — use a connector.

Tip. In the builder, render preview on a task shows which placeholders remain unresolved. Job parameters legitimately appear there, because they are supplied at job-run time. See SQL pipelines.

Run a job#

1. Open the job and choose Run.
2. The run is accepted immediately and appears in the job's run list.

A job run can be triggered two ways:

• Manually, from the job page.
• By a schedule attached to the job — see Schedules.

Note. There is no webhook trigger. Abrq DIP does not expose a token-in-the-URL endpoint for starting a job from an external system. If you need to trigger a job from outside, call the authenticated jobs API with an API token — see API tokens and the API reference.

Cancel a run#

Cancelling is cooperative and best-effort: the run stops dispatching further tasks, but a task already executing is not killed mid-statement. A run that has already reached a terminal status cannot be cancelled.

Read the job run detail#

A job run is a three-level structure, and the run detail mirrors it:

Job run → group runs → task runs.

The job run header carries the trigger (manual or schedule), who started it, start and finish times, duration, and the error message when it failed.

Each task run row shows its status, start and finish times, rows read, rows written, and its error message.

Logs are split by level, deliberately:

• The job run log carries orchestration events — the order that was computed, which task started, policy decisions, refusals.
• Per-task detail is not duplicated here. Each task run links through to its real task run in SQL Pipelines, where you get the full log, the rendered SQL, the metrics, the attempt number, and any quarantined rows.

That link is the one to follow when a task failed. The job run tells you what failed; the task run tells you why.

Job runs also appear in the cross-family monitor at Executions under the etl_job kind, alongside their constituent etl_task runs.

Field reference#

Job fields#

Group fields#

Deployment setting#

Failure modes#

Protected-environment preflight refusal#

A job run targeting a protected environment is checked before any task executes. If any connector role the job's tasks require does not resolve in that environment, the run is created and immediately finalised as failed, with an error naming the environment and the missing roles (source, destination, or both) and pointing you at the project's Environments screen.

This is not an HTTP error you will see in a browser — with a scheduled trigger nobody is watching. It surfaces as a failed job run with no task runs underneath it, which is the tell.

Fix: bind the missing roles in that environment. See Environments and promotion.

License expired#

When the deployment's license has expired or is invalid, the platform enters read-only mode. Triggering a job run, editing a job, and adding a parameter all fail with 403 and the machine-readable code license_expired; the message explains that reads still work and writes are blocked.

Reading jobs, runs and logs continues to work throughout — this is deliberate, so you can still diagnose during an expiry. An already-dispatched run is not interrupted.

Fix: renew the license. See Expiry and clock.

Everything else#

See also#

• SQL pipelines — the tasks a job orchestrates
• Environments and promotion — protected environments and connector roles
• Schedules — running a job unattended
• Executions — job runs in the cross-family monitor
• Notifications and alerts — being told when a job fails