Skip to content
ABRQ DATADocs Abrq DIP · latest
Product page Request a trial
On this page

File ingestions

What file ingestions do#

A file ingestion watches a landing path on a storage system and loads every file it finds there into a destination table, on a fixed schedule. One run processes many files; the run history records one row per file.

Sources are S3, FTP, and FTPS only. Azure Blob and GCS are not available as ingestion sources (they are available as export destinations — see Exports), and SFTP is not implemented.

Note. File ingestion is for recurring drops from a system that pushes files to you. For one-off loads, upload the file directly; for data behind an HTTP API, use Streams; for files that arrive as email attachments, use Email ingestion.

Before you begin#

  • Role. A viewer can read ingestions, run history, and per-file errors. An editor is required to create, edit, run, or delete one.
  • A source connector — S3, FTP, or FTPS — and a destination connector (Postgres, MySQL, MariaDB, CockroachDB, or SingleStore). See Data connectors.
  • Write access to the landing path when you use managed processed-file handling: the runner creates subfolders and moves files after each run.
  • A representative sample file — the schema is inferred from it.

The wizard#

Five steps: Source → File format → Sample → Destination → Review.

Step 1 — Source#

Field Notes
Name Identifies the ingestion in lists and run history.
Source connector S3, FTP, or FTPS.
Bucket S3 only.
Landing path The folder the sender drops files into, for example incoming/orders/.
Processed-file handling Managed (default) or unmanaged — see below.
Schedule Every minute / 5 minutes / 15 minutes / hour / 6 hours / Daily.
Bad-data handling Skip and count rejected rows, or fail the run.
Active Clear it to pause the ingestion.

Processed-file handling

Managed (default). The runner creates and uses three subfolders under the landing path:

<LANDING_PATH>/
├── completed/     # files that loaded successfully
├── failed/        # files that errored
└── logs/          # one JSON log per failed file

Moving the file out of the landing path is what prevents reprocessing. If the move fails — usually a permissions problem on the source — the same file is picked up again on the next run.

Unmanaged. Files stay where they are and nothing is moved. Deduplication comes from run history instead: a file already recorded as processed is skipped. One consequence to plan for — previously failed files are retried on every run, because they were never recorded as done.

Warning. Choose managed handling unless the source system forbids writes. Unmanaged mode leaves a growing landing path and re-attempts every past failure on every run.

Schedule

File ingestions use fixed intervals only: Every minute, Every 5 minutes, Every 15 minutes, Every hour, Every 6 hours, or Daily. There is no cron field here — unlike Exports, which accept a custom cron expression.

Step 2 — File format#

Format Options
CSV Delimiter, header row.
TSV Header row.
JSON Must be an array of objects.
JSONL One JSON object per line.
XML Requires record_xpath — the path to the repeating record element.
Excel (.xlsx) Sheet name or index.
Text Delimiter, header row.

For JSON and JSONL you can optionally explode an array, producing one output row per element of the chosen array field.

Step 3 — Sample file#

Upload one representative file. Abrq DIP infers the columns and types, shows them for review, and never stores the file. The upload cap is 10 MB — if your real files are larger, upload a trimmed copy with the same shape.

Before saving you can rename columns and change their types:

  • BIGINT
  • DOUBLE PRECISION
  • BOOLEAN
  • TIMESTAMP
  • TEXT

Tip. When a numeric-looking column contains leading zeros, product codes, or values that may later exceed integer range, set it to TEXT. Widening a type after the table exists means recreating the ingestion.

Step 4 — Destination#

Field Notes
Destination connector Postgres, MySQL, MariaDB, CockroachDB, or SingleStore.
Destination FQN Schema-qualified, for example raw.orders_drop.
Schema-drift policy Rescue, Evolve, or Reject — see Schema drift policies.

Step 5 — Review#

Confirm the source, format, columns, destination, and schedule, then save.

Limits you should design around#

Two behaviours regularly surprise people, so plan for them rather than around them:

  • There is no file-pattern or glob filter. Every file in the landing path is picked up. If the sender also drops unrelated files there, give this ingestion its own dedicated path.
  • There is no append/upsert control. Rows are always INSERTed. To deduplicate or merge, land the raw rows here and reconcile downstream with a SQL pipeline.

The maximum size of a single ingested file is 500 MB.

Runs and errors#

Run history shows one row per file:

Column Meaning
Started When the file began processing.
File The file's name and path.
Status Success or failure for that file.
Rows Rows inserted.
Rejected Rows dropped by the bad-data policy.
Error The failure message, when the file failed.

When a file fails under managed handling, two things happen: the file moves to failed/, and a JSON log is written to logs/<file>_<utc>.json with the details. Fix the cause, move the file back into the landing path, and it is picked up on the next run.

Error messages carry a prefix that tells you which stage failed:

Prefix Stage Typical cause
read failed: Fetching the file from the source Credentials, permissions, path, or connectivity.
parse failed: Decoding the file Wrong format, wrong delimiter, missing record_xpath, corrupt file.
write failed: Writing to the destination Destination unreachable, permission denied, constraint violation.
schema check failed: Comparing inferred columns to the table The file's columns do not line up with the destination.
bad data rejected (policy=fail): Row conversion A value did not fit its column type and the policy is fail the run.
schema drift blocked (on_schema_drift=…): Drift policy New or changed columns under a Reject policy.

Editing and pausing#

  • The source connector and landing path cannot be changed after creation. Delete the ingestion and create a new one.
  • Format options, columns' names/types, destination, schedule, and policies can be edited.
  • To pause, clear Active. The definition and history are kept.

Field reference#

Field Step Values / default
Source connector 1 S3, FTP, FTPS.
Bucket 1 S3 only.
Landing path 1 Required; immutable after creation.
Processed-file handling 1 Managed (default) or unmanaged.
Schedule 1 Minute / 5 min / 15 min / hour / 6 h / Daily. No cron.
Bad-data handling 1 Skip and count, or fail the run.
File format 2 CSV, TSV, JSON (array of objects), JSONL, XML, Excel (.xlsx), Text.
record_xpath 2 XML only, required.
Sheet 2 Excel only.
Array explosion 2 JSON / JSONL only, optional.
Sample file 3 10 MB cap; never stored.
Column types 3 BIGINT, DOUBLE PRECISION, BOOLEAN, TIMESTAMP, TEXT.
Destination connector 4 Postgres, MySQL, MariaDB, CockroachDB, SingleStore.
Destination FQN 4 Schema-qualified.
Schema-drift policy 4 Rescue, Evolve, Reject.
Max file size — 500 MB.

Failure modes#

What you see Cause Fix
read failed: … The runner cannot fetch the file. Re-test the source connector; check bucket/path and credentials.
parse failed: … Format mismatch. Confirm delimiter, header row, record_xpath, or sheet name.
write failed: … Destination rejected the insert. Check the destination connector's permissions and the table's constraints.
schema check failed: … Column set does not match the table. Re-upload a current sample, or align the file with the table.
bad data rejected (policy=fail): … A value did not fit its type. Widen the column type, or switch the policy to skip and count.
schema drift blocked (on_schema_drift=…): … Reject policy hit new or changed columns. Accept the change with Evolve, capture it with Rescue, or ask the sender to hold the schema.
The same file loads every run The move to completed/ is failing. Grant write access to the landing path, or switch to unmanaged handling.
A failed file keeps retrying (unmanaged) Unmanaged dedupe never records failures as done. Fix or remove the file; consider managed handling.
Unrelated files are being ingested There is no glob filter — everything in the path is read. Give the ingestion a dedicated landing path.
Duplicate rows in the destination Rows are always inserted; there is no upsert. Deduplicate downstream in a SQL pipeline.