When the Transfer Itself Dies Midway

So far in this series, the writer creating a partial file has been a process that was still working. The consumer read too early, and patience — engineered as renames, markers, or settle checks — would have fixed it. This article covers the other way partial files are born: the transfer carrying the file dies partway through. The destination is left holding the first however-many bytes of something that will never finish arriving on its own. No amount of waiting makes that file complete.

Every network transfer is a writer, and networks fail — connections reset, links flap, sessions time out, sending machines reboot. What separates robust flows from fragile ones is not avoiding those failures but containing what they leave behind. By the end of this article you will know exactly what a dying transfer deposits at each stage. You will know why the receiving side cannot detect the truncation from the file alone. You will know how temp-name receiving and honest verification contain the damage. You will also know what your server may or may not be doing about uploads that die on it. It is part of our Partial-File Safety series.

What a Dying Transfer Leaves Behind, Stage by Stage

Walk through one failed upload from the receiver's point of view — a partner pushing a 52 MB file to your server over SFTP or FTP. The same anatomy applies to a download you are pulling; only the seats change.

1. The session opens. The client authenticates and asks to store feed_YYYYMMDD.csv. The destination file is created — visible immediately, size zero, exactly as any local write would be.
2. Bytes flow. The file grows in bursts as network buffers drain to disk. Ten minutes in, it holds 31 MB of the eventual 52.
3. The link dies. A wireless hop drops, a router reboots, an idle timeout fires somewhere in the path, the sending process is killed. The causes are legion — our tour of FTP failure modes catalogs the classics. But they share an ending: the connection is gone and no more bytes are coming.
4. The receiver gives up. After its timeout, the receiving side closes the destination file at whatever offset the last byte reached. The session ends abnormally, and the transfer log — if there is one — records that.
5. The remains. What sits in the folder is a perfectly ordinary-looking file: final name, plausible size, modified time equal to the moment of death. The filesystem has no "incomplete" flag to set. Nothing about the file itself distinguishes it from a smaller, successful delivery.

Here is that ending as the receiver's directory and log actually show it:

$ ls -l /srv/inbound/acme/
-rw-r-----  1 xferacme xfer  31457280  Mar 14 02:13  feed_YYYYMMDD.csv
                              ^^^^^^^^ 31 MB on disk; the sender meant to send 52 MB

--- server transfer log ---
Mar 14 02:10:02  [session 87] user xferacme: upload feed_YYYYMMDD.csv started
Mar 14 02:13:47  [session 87] connection lost; 31,457,280 bytes received; transfer INCOMPLETE

Everything you need to know is in that pairing: the file looks fine, and only the session record knows it is not. Hold onto that asymmetry — it drives every defense in this article.

Why the Receiver Cannot Tell on Its Own

It surprises people that the receiving side cannot simply notice the shortfall, so it is worth being precise about why. In FTP, a client storing a file opens a data connection and streams bytes until it is done. The protocol does not send "this file will be 52 MB" ahead of the data. Completion is signaled by closing the data connection cleanly and exchanging a final acknowledgment on the control channel. From the destination file's perspective, a clean close and a torn-down connection can leave identical bytes on disk. SFTP is similar in the way that matters. The client writes blocks at offsets and closes the remote file when it chooses. If the session evaporates mid-stream, the server simply stops receiving writes. In both protocols the session knows the transfer failed. The file records nothing.

Completeness is session knowledge, not file knowledge. That is the sentence to keep. It has two immediate consequences. First, any consumer judging arrival by the file alone — existence, name, even stable size — can be fooled by an abandoned partial. That is precisely the blind spot we flagged in the settle-checks article: a dead transfer holds perfectly still. Second, transfer logs are not bureaucratic exhaust; they are the only native record of whether a delivery finished. A server that logs sessions properly turns "is this file whole?" from a guess into a lookup. Sysax Multi Server, for example, logs every upload session to file and to a database, including how each session ended. That is exactly the evidence you want when a truncated delivery needs diagnosing after the fact.

Receive Into a Temp Name

The first containment is the same pattern this series keeps returning to. Now the receiving side applies it: let arriving bytes accumulate under a name no consumer watches, and grant the final name only after the transfer completes. When you control the pulling client, that is two lines of discipline — download to .part, rename on success, delete or quarantine on failure:

# Pull to a temp name; only success earns the real name
curl -o /data/in/.feed_YYYYMMDD.csv.part "sftp://host/outbox/feed_YYYYMMDD.csv" \
  && mv -- /data/in/.feed_YYYYMMDD.csv.part /data/in/feed_YYYYMMDD.csv \
  || echo "transfer failed; partial left under temp name for inspection"

A dead transfer now leaves .feed_YYYYMMDD.csv.part — self-describing wreckage that no import will touch — instead of a booby-trapped final name. Mature tools bake this in. The tool rsync receives each file into a hidden temporary name in the destination directory. It renames the file into place only when that file completes. That is one of the behaviors dissected in the rsync flags that matter. Web browsers do the same with their .part downloads. If a copy tool in your flow writes directly to the final name — and plenty do — wrap it in the rename yourself, using the mechanics from the atomic-rename article.

Pushing instead of pulling? The same shape works from the uploading side. Store to a temporary remote name, then issue the server-side rename that FTP and SFTP both support. That repertoire is covered in our guide to SFTP file operations. Either way, the commit point moves from "bytes started arriving" to "transfer verifiably finished," which is where it always belonged.

Resume, Honestly

Both major protocols can resume an interrupted transfer — continue from a byte offset instead of starting over. FTP does it with the REST command ("restart from offset N"). SFTP clients reopen the remote file and read or write from the saved offset. The tool rsync goes further and re-synchronizes content. Resume is genuinely valuable. On a big file over a slow link, not re-sending 31 MB you already have is the difference between a hiccup and a missed deadline.

But be precise about what resume is: a bandwidth optimization, not a correctness mechanism. Here is the honest ledger:

The middle rows deserve fear. Suppose attempt one dies at 31 MB, and overnight the sender's system regenerates the extract. Attempt two resumes at offset 31 MB — of the new file. The result is a seamless-looking hybrid: the head of yesterday's data stitched to the tail of today's, with a size that may even match expectations. No error will ever fire on its own. Here are the defensive rules. Record the source's size and modified time when a transfer starts. Resume only if they still match. When in doubt, restart from zero into a fresh temp name. Content-aware tools like rsync close this gap by re-checking data rather than trusting offsets. That is one more reason the offset-resume flows are the ones that most need the verification below.

Verifying Completeness After the Fact

Since the file cannot testify and resume cannot promise, completeness must be proved, with evidence that traveled separately from the bytes. In ascending order of strength:

• Expected size. The sender states the byte count — in a control file, a manifest, or a listing you query — and the receiver compares. Cheap, and catches every truncation, though not corruption or the hybrid-file trap.
• Record counts and trailers. For structured files, a stated row count or a mandatory end-of-file trailer record catches truncation even when nobody stated a byte size.
• Checksums. The sender publishes a cryptographic hash; the receiver recomputes it over the assembled file. This catches truncation, corruption, and the stitched hybrid in one test. The mechanics are in hashing, explained. The conventions for shipping hashes alongside files are in checksum files and manifests.

Stitched together with the temp-name pattern, verification gives you the receive pipeline that this whole article has been building toward. Download, verify, and only then commit the real name:

tmp="/data/in/.feed_YYYYMMDD.csv.part"
final="/data/in/feed_YYYYMMDD.csv"

sftp -b - partner@host <<EOF || { rm -f "$tmp"; exit 1; }
get outbox/feed_YYYYMMDD.csv $tmp
get outbox/feed_YYYYMMDD.csv.sha256 $tmp.sha256
EOF

want=$(cut -d" " -f1 "$tmp.sha256")
have=$(sha256sum "$tmp" | cut -d" " -f1)
if [ "$want" != "$have" ]; then
    mv -- "$tmp" /data/quarantine/ ; alert "hash mismatch on feed_YYYYMMDD" ; exit 1
fi
mv -- "$tmp" "$final"    # the commit: complete AND verified, in one atomic step

Note what the final rename now certifies: not merely "the transfer command returned," but "the bytes on disk match the bytes the sender published." That is the strongest arrival guarantee a receiver can manufacture, and the full argument for it is in verifying transfers end to end.

Server-Side Upload Behaviors Worth Knowing

When the receiver is your own server, one more variable enters: what the server software does with uploads in progress and uploads that die. As a category, servers differ — and the differences decide how much of this article's problem your downstream consumers inherit:

• Some can write incoming uploads under a temporary name and rename to the final name only on successful completion — producer-side protection applied for you.
• Some can hide in-progress uploads from directory listings until they finish, which protects polling consumers without changing names.
• Some keep partial files after an abnormal disconnect (enabling resume), others delete them, and either may be configurable.
• Many, by default, do none of the above. The upload grows under its final name and stays there if it dies — the worst case for anything sweeping that folder.

Treat these as behaviors to verify, not assume: they vary between products, versions, and configurations, and the documentation is not always explicit. If your server leaves dead partials under final names, your consumers need the defenses from the marker article or the settle-and-verify discipline. In that case, your operations also need a sweep that quarantines stale partials before someone processes them.

The Killed-Transfer Test

Ten minutes of deliberate failure answers every question this section has raised about your own estate. Run it once per flow, and again after any server change:

1. Prepare a test file large enough that its transfer takes at least thirty seconds.
2. Start uploading it to the real target folder with the real client or tool the flow uses.
3. Kill the transfer partway — terminate the client process, or drop the network it is using.
4. Immediately list the destination folder. What do you see: the final name at partial size, a temp name, or nothing?
5. Wait, then look again: does the server clean up, rename, or leave the remains untouched?
6. Read the server's log entry for the session: does it clearly record an incomplete transfer and the byte count?
7. Retry the upload: does it resume, overwrite cleanly, or fail because the corpse is in the way?

Write the answers into the flow's runbook. Every defense choice downstream follows from those seven observations. That includes whether consumers can trust final names, whether a partial sweep is needed, and whether resume is safe to enable.

Cleanup and Retry Around Failed Transfers

Finally, the job around the transfer has to close the loop. Three disciplines keep failure debris from becoming tomorrow's incident:

• Decide resume-versus-restart by rule, not mood. Resume when the source is provably unchanged (size and modified time match your record) and the tool or a final checksum will verify the result. Restart into a fresh temp name in every other case. Encode the rule in the script so it is applied at 03:00 exactly as it would be at 15:00.
• Sweep for stale partials. Any .part-style file older than your retry horizon is an abandoned attempt. Quarantine it and alert, rather than deleting silently — it is evidence, and occasionally it is resumable value.
• Retry with intent. Transient network failures deserve automatic retries with backoff; a transfer that fails identically three times deserves a human. That taxonomy and its budgets are the business of our retry and error handling series. And because retries re-send, receivers should be ready to see the same file twice — the reprocessing side is covered by our duplicate detection and idempotency series.

None of this needs exotic tooling. A scheduled-transfer harness that already does attempts, backoff, failure branching, and notification gives the rules above somewhere to live. Sysax FTP Automation is built around exactly that retry and error handling with email alerts. So a dead transfer at 03:00 becomes a retried success or a clear morning alert instead of a silent partial file.

The Receiver's Creed

Networks will keep dying mid-transfer; that part is not negotiable. What you control is the shape of the wreckage. Bytes accumulate under temp names, and final names are granted only after verification. Session logs are consulted rather than guessed around. Resume is trusted only when the source is provably unchanged. Stale partials are swept into quarantine, loudly. Do those five things and an in-flight death costs you a retry, not a dataset. The next and final article in the series, the partial-file safety checklist, folds these receiver-side controls into one audit matrix alongside everything the producer and consumer can do.

Frequently Asked Questions