Duplicate Detection and Idempotency

Every mature file pipeline eventually processes the same file twice. A retry fires after a timeout that actually succeeded, a partner resends yesterday's batch, an admin re-runs a job to be safe. The question is not whether duplicates will arrive — they will — but whether your pipeline shrugs them off or posts the same invoices twice. The property that makes reruns boring has a name: idempotency.

This series makes that property practical. It covers idempotency in plain words and where duplicate files actually come from. It covers detection using names, sizes, hashes, and a processed-files ledger. It translates the exactly-once idea from theory to file flows, and covers deliberately reprocessing old files without side effects. It closes with a worked example of retrofitting duplicate protection onto a real nightly flow.

Articles in This Series

• Idempotency in Plain Words for File Pipelines
  Run it twice, get the same result. This article explains what the property means for transfer jobs, and why reruns happen constantly - retries, resends, nervous admins. It shows what rerun-safe design looks like.
• Where Duplicate Files Actually Come From
  This article covers the taxonomy: sender resends, retry overlap after a timeout-that-succeeded, backfills gone wide, multiple delivery paths. It also covers the downstream cost that makes prevention worth it.
• Detecting Duplicates: Names, Sizes, Hashes, and Ledgers
  This article covers layered detection from cheap to certain: name-based checks, metadata comparison, content hashing, and the processed-files ledger. It includes honest notes on performance and ledger retention.
• Exactly-Once Thinking for File Pipelines
  This article translates the distributed-systems idea: at-least-once delivery plus idempotent processing equals effectively exactly-once. It covers where each mechanism lives in a file flow, and what to stop worrying about.
• Safe Reprocessing: Running Yesterday's Files Again
  This article covers the deliberate re-run: scoping what gets reprocessed, an explicit reprocess mode instead of hand-edits, and warning downstream. It covers the audit trail that records why the rerun happened.
• Building Duplicate Protection into a Real Flow
  A retrofit story: a nightly ingest gains a ledger, a hash check, and a rerun mode. The article covers the before-incident, the design decisions, the code shape, and the after.

Explore More Topics

This series is part of the Sysax file transfer topic library. The library covers the protocols, security practices, automation techniques, and operational skills behind reliable file transfer. The library pairs well with the practical tools we build. Sysax Multi Server is a secure FTP, FTPS, SFTP, and HTTPS server for Windows. Sysax FTP Automation schedules and scripts secure transfers so the routine ones run themselves.