Preventing Duplicate Records When Parsing Email with POST /v1/parse

Duplicate processing can occur when a caller retries or a queue redelivers. Upstream mail servers retry transient handoff failures, mail transfer agents retransmit messages across network interruptions, and background worker queues redeliver unacknowledged tasks. When ingesting receipts, invoices, or order notifications into operational datastores, uncoordinated parsing pipelines inevitably produce duplicate database records.

MailFrame’s POST /v1/parse endpoint is a direct, synchronous parser. Callers submit raw RFC 822 MIME bytes via raw_mime alongside either a registered schema_id or an inline JSON Schema (schema), receiving typed, schema-validated JSON in the immediate HTTP response. The API reference does not document an idempotency key or automatic duplicate suppression, and callers should not rely on identical returned data on retries.

Deduplication must therefore be owned entirely by the consuming application. While schema-first validation ensures extracted data conforms to a strict structural contract, caller-side idempotency reduces duplicate business side effects when atomic claims and fencing are correctly implemented. For broader architecture patterns, see our guides on email parsing pipeline design and building a JSON Schema email parser.

Here is a 5-stage conceptual workflow for building robust caller-side deduplication around synchronous parsing.

Stage 1: Build a Stable, Scoped Deduplication Key

Never rely on a bare message identifier as a global deduplication key. Construct a composite key scoped across distinct operational boundaries:

• Tenant and Mailbox Isolation: Scope by tenant identifier and destination mailbox. Identical emails may legitimately arrive across separate tenants or different inboxes (such as billing@ and ap@) within the same organization.
• Schema Versioning: Incorporate schema versioning so evolving schema contracts can deliberately re-extract historical emails without triggering deduplication conflicts.
• Provider Identifier Nuances: Rely on provider identifiers only when they are genuinely immutable within your integration. For example, Microsoft Graph message IDs can change across folder moves unless explicitly requested with immutable ID preferences, and Gmail sequence indicators like historyId represent mailbox state updates rather than persistent message IDs.
• Unreliability of RFC 5322 Message-ID: Senders frequently omit Message-ID, duplicate it across distinct notifications, or emit unauthenticated values. Intermediate gateways and forwarding hops may also strip or alter header fields.
• Raw MIME Fingerprint Fallback: When a trustworthy upstream identifier is absent, compute a cryptographic digest (such as SHA-256) of the raw MIME. Note that email forwarding often alters headers or re-encodes bodies, changing byte hashes for what users perceive as the same message.

Stage 2: Persist an Atomic Claim and Compare Payload Fingerprints

Before dispatching an HTTP request to POST /v1/parse, record an atomic claim backed by a database unique constraint on your scoped key.

Store the raw payload fingerprint alongside the claim. If an incoming message matches an existing key, compare the incoming payload fingerprint against the recorded fingerprint before treating the message as a duplicate:

• Matching Fingerprints: A true duplicate delivery of identical raw content. If the prior parse succeeded, return or link the existing record.
• Divergent Fingerprints: The sender reused an identifier for different email content. Never silently drop or overwrite the existing record; flag the collision as a payload conflict for operator review.

Stage 3: Treat In-Progress Claims as Pending and Enforce Lease Fencing

When parallel workers consume from message queues, identical emails may be processed concurrently:

• Pending vs. Completed States: If a worker encounters an active claim that is still in progress, it must treat the job as pending rather than prematurely treating it as a completed duplicate or returning incomplete data.
• Worker Leases and Fencing: If using distributed workers with time-bounded leases, enforce lease ownership via fencing tokens or monotonic version checks. If a worker pauses and its lease expires, a new worker may safely reclaim the task; fencing prevents the stalled worker from committing late results or overwriting newer claims.

Stage 4: Differentiate Retryable Errors from Terminal Schema Errors

Distinguish transport-level network faults from contract-level extraction failures:

• Retryable Errors: For network drops, rate limits, or 5xx server responses, caller retry loops should use bounded exponential backoff with randomized jitter. Transient failures must release or expire claims so future attempts can proceed.
• Terminal Schema Rejections: When POST /v1/parse returns validation errors indicating an email does not satisfy your schema contract, do not blindly retry validation failures. Review the input and schema, and route to human review before deliberate retry.
• Isolate Failure Status: Never conflate operational execution failures with business data conflicts, and never mark an uncompleted failure permanently as a duplicate.

Stage 5: Commit Extracted Data and Business Side Effects Atomically

Once POST /v1/parse returns validated data, persist the extracted JSON and execute associated business side effects (such as updating account balances or creating order records) within a single atomic database transaction, or dispatch them through an idempotent transactional outbox pattern.

Caller-managed atomic persistence ensures your datastore remains consistent despite upstream re-deliveries. Refer to the API documentation for endpoint schemas and parameters.