The Anatomy of a Reliable Transfer: Integrity, Atomicity, and Delivery Guarantees

Somewhere right now, a business process is choking on a file that was "transferred successfully." Maybe the file is truncated — the connection died at 93% and the fragment on disk looks real. Maybe an import job grabbed it while it was still being written. Maybe a retry delivered it twice and two thousand invoices posted twice. None of these are exotic failures. They are the ordinary consequences of treating "the transfer finished" as the same thing as "the file arrived intact, complete, and exactly once."

This article is about the gap between those two statements and the small set of patterns that close it: end-to-end checksums, temp-name-and-rename atomicity, duplicate-tolerant retry design, and logging that can prove delivery after the fact. Every pattern here is protocol-agnostic — it works over FTP, SFTP, FTPS, or HTTPS — and none requires special software, though good tools make them easier. It is part of our File Transfer Fundamentals series and picks up exactly where how a file transfer actually works left off: at the completion message, and what it does not promise.

What “Delivered” Should Actually Mean

Before patterns, definitions. A file transfer worth trusting delivers four distinct properties, and it pays to name them separately because they fail separately:

• Complete — every byte of the file arrived. The opposite is truncation: a partial file left behind by an interrupted transfer.
• Intact — the bytes that arrived are the bytes that were sent, unaltered. The opposite is corruption: same length, wrong contents, or contents mangled in transit.
• Final — the file only becomes visible to consumers when it is finished. The opposite is the half-written file: technically neither corrupt nor incomplete, just read too early.
• Accounted for — the delivery happened exactly the intended number of times (once), and there is a record proving it. The opposites are the silent miss and the duplicate.

Here is the uncomfortable part: a standard protocol transfer, by itself, gives you a reasonable version of the first property, a weaker version of the second than most people assume, and nothing at all for the third and fourth. Those are the workflow's job — your job. The rest of this article covers each in turn.

Where Corruption Actually Sneaks In

"But TCP has checksums" is the standard objection, and it is true — every TCP segment carries a checksum, and segments that fail it are discarded and resent. So how do damaged files still happen? Several ways, each instructive:

• The TCP checksum is small and per-hop. It is a 16-bit check — strong enough to catch most random damage in one segment, weak enough that across billions of segments, an occasional double-bit error pattern slips through arithmetically clean. And it is recomputed at every middlebox that rewrites packets, so it verifies each hop, not the journey. On multi-gigabyte files moved daily, "vanishingly rare" multiplied by enormous volume becomes "eventually."
• The endpoints themselves. TCP verifies the network path. It knows nothing about the disk controller with failing memory, the filesystem error, or the RAM glitch on either machine that damages bytes before they were sent or after they arrived.
• Transfer-mode mangling. FTP's ASCII mode "helpfully" rewrites line endings in transit. Applied to a binary file — a ZIP, an image, a database export — it corrupts it thoroughly and reproducibly. Wrong-mode transfers remain a classic self-inflicted corruption, covered in this library's FTP protocol series.
• Truncation dressed as success. A dropped connection leaves a partial file on disk. The protocol knows the transfer failed — no completion message was exchanged — but the fragment does not know it is a fragment. Any process that judges files by their existence will treat it as real. (Interrupted FTP transfers and their zero-byte cousins have a diagnostic guide of their own in diagnosing FTP mode failures.)
• Well-meaning intermediaries. Content-scanning gateways, archive tools, and scripts that "clean up" files in the pipeline can alter bytes deliberately — re-encoding text, stripping content, recompressing archives. The transfer is flawless; the file still differs from the original. Only an end-to-end comparison notices.

The common thread: every check listed above is local to a layer or a hop. Reliability engineering has a name for the fix — the end-to-end principle. Verification that matters must compare what the sender meant to send with what the receiver finally has, at the application level, ignoring every intermediate assurance. Which brings us to checksums.

Checksums: Proving Both Ends Hold the Same Bytes

A checksum (or hash) is a short fingerprint computed from a file's entire contents. Change any single bit of the file and the fingerprint changes completely. Compute it on the source file, compute it again on the delivered file, compare: if the two strings match, the files are byte-for-byte identical for all practical purposes; if they differ, something changed the bytes, and you know before the file does damage.

The algorithms you will meet: SHA-256 is the modern default — use it when you have the choice. MD5 and SHA-1 are older; they are no longer trustworthy against a deliberate forger, but they still detect accidental corruption perfectly well, which is why partners still exchange MD5 values and why refusing them is rarely worth the argument. Detecting accidents needs any of these; resisting adversaries needs SHA-256.

Computing one is a single command on every platform:

# Windows — built-in certutil
certutil -hashfile report.csv SHA256

# Windows PowerShell
Get-FileHash report.csv -Algorithm SHA256

# Linux / macOS
sha256sum report.csv

# Create a sidecar checksum file to send along with the data file
sha256sum report.csv > report.csv.sha256

# Receiver verifies both in one step (Linux)
sha256sum -c report.csv.sha256

The sidecar file pattern in the last two commands is the workhorse of partner exchanges: send report.csv and a tiny report.csv.sha256 beside it, and the receiver can verify the delivery without any out-of-band conversation. Two refinements make it robust: send the checksum file after the data file (its arrival doubles as a "data file is complete" signal), and make sure both sides agree on which algorithm and format the sidecar uses before the first real exchange, not during the first incident.

The Half-Written File Problem

Now to the property nobody thinks about until it bites: finality. A file being written — by an upload, an export, a copy — exists on disk from its first byte. Its name is right, its location is right; only its contents are still arriving. Any consumer that selects files by name or presence can grab it mid-write: an import job reading a growing CSV gets the first half of the data and no error at all, because reading a file that is being appended to is not an error.

This race happens entirely on the receiving system, after the network did everything right, which is why no protocol setting fixes it. The fix is a workflow discipline: never let a file appear under its final name until it is finished. The diagram below contrasts the naive flow with the discipline.

Atomic Renames and Other “Finished” Signals

The pattern works because a rename is atomic — an operation that either has not happened yet or has fully happened, with no in-between state anyone can observe. Filesystems perform a rename within the same volume by updating the directory entry, not by copying data, so report.csv springs into existence complete. A consumer polling the folder sees either no file or the whole file. The race is not narrowed; it is gone.

The recipe, concretely: upload to report.csv.part (or into a tmp/ directory on the same volume), and after the transfer completes — and ideally after the checksum verifies — rename to report.csv (or move into the watched folder). Consumers ignore .part names by convention. Many transfer clients already upload through a temporary name; the pattern here is making it an explicit, verified contract between producer and consumer rather than a client default nobody documented.

Two companions to the rename pattern round out the toolbox. A marker file (like batch_0142.done, sent after all data files) signals that a whole set is complete — renames protect individual files, markers protect batches. And a settle delay — a monitor waiting until a file stops growing before touching it — is the defensive version consumers apply when they cannot change the producer's behavior. Folder-monitoring tools such as Sysax FTP Automation build in this wait-until-stable behavior precisely because so many producers upload straight to final names.

Retries and Duplicates: The Delivery-Guarantee Problem

Failures happen, so transfers get retried — automatically by a scheduler or manually by a human. Retries create a subtle question: if the first attempt might have succeeded (the transfer finished but the connection died before the confirmation arrived), a retry might deliver the file a second time. Distributed-systems people have crisp names for the three possible postures:

The mature position: run at-least-once with duplicate tolerance. Retry until the file is confirmed delivered, and make a duplicate delivery harmless rather than pretending it cannot happen. Making duplicates harmless is called idempotency — designing the receiving side so processing the same file twice has the same effect as processing it once. Practical idempotency for file flows is unglamorous and very effective:

• Deterministic, unique filenames — invoices_batch0142.csv, not invoices_new.csv. A re-delivered file overwrites its identical twin (harmless) instead of appearing as a new mystery file, and batch numbers make gaps and repeats visible.
• A processed log on the consumer side — before importing, check the filename (or better, its checksum) against the record of what was already processed; skip matches.
• Move-after-processing — the consumer moves each file to a processed/ folder as it ingests it, so the inbox only ever contains work not yet done, and a re-arrival is visibly new.

On the sending side, retry with backoff — spacing attempts progressively further apart, then escalating to a human — rather than hammering a down server every ten seconds. This is standard behavior in purpose-built schedulers (retry counts, intervals, and failure actions are per-job settings in Sysax FTP Automation) and worth replicating in any hand-rolled script.

Resume: Continuing Instead of Restarting

For large files on imperfect links, protocols offer resume — continuing an interrupted transfer from the byte where it stopped instead of starting over. FTP does this with the REST (restart) command; SFTP clients seek to an offset and append; HTTP uses range requests. Resume is a bandwidth feature, not an integrity feature, and it carries one sharp edge: the resuming client assumes the partial file at the destination still matches the beginning of the source file. If the source changed between attempts, resume welds the front of the old version to the back of the new one — a file that is complete, plausible, and wrong.

The rule that keeps resume safe is simple: resume, then verify. A checksum comparison after any resumed transfer costs seconds and catches the welded-file case (and any other mismatch) with certainty. Automation that retries interrupted large transfers should bake the verification in rather than trusting the seam. It is worth noting that delta-transfer tools take this philosophy furthest: the rsync family decides what to send by comparing checksums of file blocks, making verification part of the transfer itself — the mechanics are covered in our rsync and delta transfer series.

A Verification Recipe You Can Adapt

Assembled into one sequence, here is what a transfer worth trusting looks like from the producer's side. Treat it as a template — drop steps only when you can say who else guarantees that property:

1. Export/produce the file locally; compute its checksum.
     sha256sum invoices_batch0142.csv > invoices_batch0142.csv.sha256
2. Upload the data file under a temp name.
     put invoices_batch0142.csv  invoices_batch0142.csv.part
3. Compare sizes (cheap first-line check; a mismatch means stop).
4. Rename to the final name — the atomic "this file is complete" signal.
     rename invoices_batch0142.csv.part  invoices_batch0142.csv
5. Upload the checksum sidecar (its arrival = "verify me now").
     put invoices_batch0142.csv.sha256
6. Consumer verifies checksum before processing; quarantines on mismatch.
7. Log the outcome on both sides; alert on failure or absence.

The consumer's mirror-image duties: watch only for final names, verify the sidecar before ingesting, record what was processed, and alarm when an expected file has not appeared by its deadline — absence is a failure too, and detecting it belongs to whoever depends on the file.

Logging: Proving It Afterward

The final property — accounted for — lives in logs. When a delivery is disputed ("we never received it" / "we sent it twice"), the questions are always the same, so log the answers in advance: timestamp, direction, local and remote endpoint, account used, full filename, byte count, duration, outcome, and error detail on failure. Client-side logs tell you what your jobs attempted; server-side activity logs — such as those kept by Sysax Multi Server on the receiving end — tell you what actually arrived, from whom, and when. The two views together settle most disputes in minutes.

Logs also enable the cheapest reliability report there is: the reconciliation count. If the producer's log says twelve files were sent and the consumer's processed log says eleven were ingested, something was missed — and a scheduled comparison of those two numbers surfaces it the same day instead of at month-end. For recurring batch flows, this one automated count catches whole categories of silent failure that per-transfer checks cannot see.

When delivery proof needs to be contractual rather than merely convincing — signed, receipted, non-repudiable — that is the province of B2B protocols like AS2, whose receipts (MDNs) formalize exactly the guarantees this article builds by hand; see the AS2 and EDI series for that world.

Reliability Is a Workflow Property, Not a Protocol Feature

The one-paragraph summary: the network's own checks are per-hop and per-segment, so trust for the file as a whole comes only from end-to-end verification — a checksum computed where the file was made and checked where it will be used. Finality comes from the atomic-rename discipline, so nothing downstream can see an unfinished file. Delivery certainty comes from at-least-once retries paired with duplicate-tolerant consumers. And proof comes from logs on both ends. None of it is exotic; all of it is decidable at design time, which is the cheapest time.

To continue in this series: transfer vs sync vs share explains why these delivery semantics belong to the transfer model, the transfer census helps you find every flow that deserves this treatment, and the working glossary pins down the vocabulary — atomicity, idempotency, checkpoints — for the next vendor conversation.

Frequently Asked Questions