Stream destinations

The destination set#

A stream fans out to one or more destinations. Each destination drains independently, so a slow or broken sink does not stop the others.

Connection fields for each engine — including how host, database and the credential fields are repurposed for object stores and Kafka — are documented in Connectors.

Kafka#

Behaviour:

• Ordering is per-partition only. With a partition-key field set, all records sharing a key land on one partition and keep their relative order. Without a key, records round-robin across partitions and no ordering is preserved at all.
• Produced with acks=all, one record per send — the broker acknowledges every record before it is considered delivered.
• Each message carries an abrq_idempotency_key header so consumers can deduplicate a redelivered record.
• Kafka ignores truncate instructions and takes no part in schema evolution. It is a sink and nothing more.

Object and file destinations#

Applies to s3, azure_blob, gcs, ftp, ftps and local.

One object per batch. The object name is part-<idempotency-prefix>.<ext> — derived from the batch's idempotency key, with no timestamp component. That is deliberate: replaying a batch writes the identical object name, so a replay overwrites rather than duplicating.

exports/orders/dt=2026-07-30/part-8f2c41a9.jsonl.gz
exports/orders/dt=2026-07-30/part-8f2c41a9.jsonl.gz.manifest.json

• Date partitioning, when enabled, inserts a dt=YYYY-MM-DD/ directory keyed off the time the batch was staged, not the time it was written. A retried batch therefore lands in the same partition as its first attempt.
• The .manifest.json sidecar carries the object path, row count, byte count and a sha256 of the object. It is enabled by default for FTP and FTPS, where there is no server-side integrity check to lean on.
• FTP and FTPS write to a temporary name and then rename. A consumer polling the directory never observes a partially written object.

Relational destinations#

postgres and mysql only.

• Each batch is written as a batched INSERT inside one transaction. The batch lands completely or not at all.
• Relational destinations are the only stream destinations that participate in schema evolution — see Schema drift policies.

Delivery semantics#

Note. Stream delivery is at-least-once with idempotency keys. Exactly-once is not claimed and not implemented. Design consumers to deduplicate on the idempotency key.

How that guarantee is achieved:

• Records are staged into a spool inside the source transaction. The source cursor cannot advance without the corresponding records being staged, so nothing is read-and-lost.
• Each destination keeps its own high-water mark, and that mark only advances past a batch the destination durably accepted.
• Head-of-line consequence: a destination that cannot accept a batch does not skip it. Its high-water mark stays put and its backlog grows until the batch succeeds or is dead-lettered. The other destinations are unaffected.

Retries#

Full jitter means the actual delay is drawn uniformly between zero and the computed exponential ceiling, so a fleet of retrying workers spreads out instead of synchronising into one simultaneous retry surge.

Circuit breaker#

While the circuit is open the destination is skipped entirely — no attempts, no timeouts, no queue pressure on the worker. After the cooldown a single trial batch is allowed through; success closes the circuit, failure re-opens it for another cooldown.

An operator can force the issue with the Reset breaker action, which closes the circuit immediately without waiting out the cooldown. Use it after fixing the underlying problem — resetting into a still-broken sink just re-opens it.

The counter abrq_stream_fanout_circuit_opened_total{destination_kind} records every trip.

Dead letters#

A batch that exhausts its retry budget is dead-lettered for that destination only. Every dead letter carries a reason class:

Two operator actions exist:

• Replay — re-drive the dead-lettered batch to the one destination that dead-lettered it. Replay is targeted; it never re-sends to healthy sinks.
• Discard — drop the batch permanently. This is data loss and is recorded as such.

Spool retention#

The spool is garbage-collected by high-water mark: batches at or below the minimum high-water mark across all enabled destinations are deleted.

Warning. Only enabled destinations count. Disabling a destination removes it from the minimum, which lets the spool GC advance past batches it never received — re-enabling it later will not backfill them. A destination you intend to resume should be left enabled and allowed to fall behind, not disabled.

Metrics#

See Metrics.