Email ingestion (inboxes and feeds)

What email ingestion does#

Some data arrives as an attachment on a recurring email — a daily CSV from a supplier, a weekly export from a partner system. Abrq DIP turns that into a managed pipeline with two resources:

• Inboxes — the IMAP mailboxes Abrq DIP polls. One inbox can serve many feeds.
• Email feeds — a per-sender agreement: messages from this sender, with this subject, carrying a file in this format, land in this table, and are expected on this cadence. If a feed goes quiet past its grace period, Abrq DIP raises an overdue reminder.

The split matters: credentials and polling live on the inbox; matching, mapping, destination, and the delivery SLA live on the feed.

Before you begin#

• Licensed feature: email. Without the entitlement the nav item is hidden. See Entitlements.
• Role. A viewer can read inboxes, feeds, runs, and logs. An editor is required to create, edit, test, or delete either resource.
• A mailbox. Host, port, username, and password for an IMAP account. Only the imap protocol is supported — POP3 and Exchange Web Services are not.
• A destination connector — Postgres or MySQL. See Data connectors.
• A sample file matching what the sender actually attaches. You upload it during the wizard to infer columns; it is parsed in memory and never stored.

Step 1 — Add an inbox#

Passwords are encrypted at rest — see Secrets and encryption.

Tip. If your mail provider requires an app-specific password or OAuth-only access for regular sign-in, generate an app password for this inbox rather than reusing a human's credentials.

Test the connection#

Test poll runs one synchronous poll and reports counts:

Use it to verify two different things: that the credentials work at all (seen is greater than zero), and that routing works (processed grows rather than unmatched).

Deleting an inbox#

An inbox with feeds attached cannot be deleted. You will see:

inbox has N active feed(s); deactivate them or repoint them before deleting

Deactivate the feeds, or point them at another inbox, and retry.

Step 2 — Create an email feed#

The feed wizard has two steps.

Step 1 — Identify and map#

Then upload a sample file. Abrq DIP parses it in memory to infer the column names and types, shows them for review, and discards the file — sample files are never stored.

Metadata columns

You can add columns that are not in the file at all. Each one is stamped on every row of every delivery:

Step 2 — Destination and schedule#

Note. The connector error text mentions other engines, but the effective gate is postgres or mysql. Any other source engine will be rejected on save regardless of what the message lists.

How the overdue SLA works#

The cadence and grace period define a delivery contract, and the reminder is what tells you it was broken.

• A reminder fires once last_received_at + expected_interval + grace has passed. If nothing has ever arrived, the feed's creation time stands in for last_received_at, so a feed that never receives a first file still alerts.
• One reminder per breach. The feed does not re-notify every poll.
• The breach is cleared on the next successful delivery, which re-arms the reminder for the following cycle.
• last_received_at is stamped only if the delivery succeeded. A file that arrived but failed to load does not reset the clock — a broken feed keeps reminding you instead of going quiet.

Reminder settings are all-or-nothing:

reminder_kind and reminder_target must be set together (or both omitted to disable reminders)

Editing a feed#

Columns can be de-selected, but never added, after the feed exists:

cannot add new columns to an existing email feed (destination table is already created)… Delete and recreate the feed to change the schema.

The destination table is created from the feed's column list, so widening it means a new table. Plan the column set — including metadata columns — before you save.

Sender and subject patterns, policies, cadence, grace, and reminder settings can all be changed in place.

Logs and testing#

The Logs dialog shows per-delivery detail and carries Test feed now, which polls the feed's parent inbox immediately rather than waiting for the next minute-tick. Because it polls the whole inbox, other feeds on the same inbox may pick up messages during the same test.

When patterns are wrong, the dialog says so directly:

Messages were seen but none matched any feed's filters

That message means the connection is fine and the routing is not. Compare the real sender address and subject line against your patterns — a Contains subject match and a Domain match sender are the usual fix for senders whose subject lines carry a changing date.

Field reference#

Failure modes#

Related pages#

• Data connectors
• Notifications and alerts
• Schema drift policies
• File ingestions — for files dropped on S3 or FTP instead of emailed.