Building Integrity Checks into Automated Jobs

Verifying a file by hand is easy when you are watching. Automation removes the watcher. A scheduled job runs at 2 a.m. with nobody at the keyboard, and whatever it decides to do with a corrupt file, it does silently and at scale — every night, on every file, until someone notices weeks later that the data has been quietly wrong. The difference between a job that protects you and a job that betrays you comes down to one design choice: what happens when verification fails.

This article is about wiring integrity checks into unattended jobs so they run themselves and fail loudly. You will learn the fail-closed principle, how to retry on a mismatch without shipping bad data, the write-then-rename pattern that keeps half-written files out of sight, and how to log the proof so nobody has to take your word for it. This is part of our Integrity Verification series and the automation payoff of everything in verifying a transfer end to end — the manual check, made durable.

Fail-Open vs Fail-Closed: The One Decision That Matters

Every automated check faces the same fork when something goes wrong, and the two paths have names worth knowing. A fail-open system, when it hits an error or an uncertain result, keeps going — it lets the file through and moves on. A fail-closed system, faced with the same uncertainty, stops — it refuses to deliver until the file is proven good. For integrity, fail-closed is the only safe default, because the whole point of verification is to prevent bad data from moving downstream, and a fail-open check does exactly the opposite of that.

The trap is that fail-open is the easier thing to build by accident. A job that computes a hash, writes the result to a log, and then unconditionally continues has technically "added verification" — and has protected nothing, because nothing acts on the result. The log fills with mismatches nobody reads while corrupt files flow on as if verified. Real verification is not computing the hash; it is letting the hash decide what happens next.

The diagram below shows a fail-closed job. The verified path is the only route to "delivered," and every other outcome loops to retry or stops at an alert — a corrupt file has no path forward.

The Anatomy of a Verified Job

Every self-verifying job, whatever tool runs it, has the same skeleton:

1. Record the reference. Compute the source digest before or during the transfer — the fingerprint the result must match.
2. Transfer to a staging name. Land the file under a temporary name the consumer is not watching, so a half-written or unverified file is never visible as "ready."
3. Verify. Hash the arrived file and compare to the reference.
4. Branch on the result. On a match, commit the file (rename it into place) and log the success. On a mismatch, quarantine the bad copy, retry within a limit, and alert if the retries run out.
5. Log the proof. Whatever happened, record the digests, the outcome, and the time.

The load-bearing idea is step 4: the branch. A job without that branch is not verifying; it is decorating a transfer with a hash it ignores.

Step 1 hides a question worth answering deliberately: where does the reference digest come from? When your job sends a file, you own the source, so you compute the reference yourself before the transfer — clean and trustworthy. When your job receives a file from a partner, you did not create it, so the reference has to come from them: a digest they publish, a sidecar they include, or a signed manifest. That distinction is not a detail. A reference you compute from the very file you are checking proves only that the file has not changed since you touched it — it cannot prove you received the right file, because a corrupted or substituted file would simply produce a corrupted or substituted reference. In a receive job, insist the partner supplies the reference through a channel you trust, and you are back to a meaningful check. This is the seam where integrity meets authenticity, drawn fully in integrity vs authenticity.

The Fail-Closed Pattern in a Script

Here is the pattern as a Bash script. The placeholder commands — transfer, remote_sha256, finalize, alert — stand in for whatever your environment uses; the control flow is the point:

#!/usr/bin/env bash
set -euo pipefail

SRC="/data/out/nightly.tar.gz"
DST_TMP="/incoming/nightly.tar.gz.part"    # staging name
DST="/incoming/nightly.tar.gz"             # final name
MAX_TRIES=3

# 1. Reference digest, taken at the true source
expected=$(sha256sum "$SRC" | awk '{print $1}')

attempt=1
while (( attempt <= MAX_TRIES )); do
  echo "attempt $attempt: transferring"
  transfer "$SRC" "$DST_TMP"               # your transfer command

  # 2. Digest of what actually landed
  actual=$(sha256sum "$DST_TMP" | awk '{print $1}')

  # 3. Compare and FAIL CLOSED
  if [[ "$actual" == "$expected" ]]; then
    mv -f "$DST_TMP" "$DST"                 # atomic commit into place
    logger "VERIFIED nightly.tar.gz $expected"
    exit 0
  fi

  logger "MISMATCH try $attempt expected=$expected got=$actual"
  rm -f "$DST_TMP"                          # never leave a bad file behind
  sleep $(( attempt * 10 ))                 # simple backoff
  (( attempt++ ))
done

logger "FAILED nightly.tar.gz after $MAX_TRIES tries - NOT delivered"
alert "Integrity check failed for nightly.tar.gz"
exit 1

Read the exits. The only path to exit 0 runs through a successful comparison and the rename. Every other path deletes the staging file, and the job ends in a non-zero exit and an alert. There is no branch where a mismatched file becomes the final file. That is fail-closed in code.

The same shape works in PowerShell for Windows jobs — compute, compare with the case-insensitive -eq, and let the result gate the rename:

$expected = (Get-FileHash $Src -Algorithm SHA256).Hash
for ($try = 1; $try -le 3; $try++) {
    Transfer-File $Src $DstTmp
    $actual = (Get-FileHash $DstTmp -Algorithm SHA256).Hash
    if ($actual -eq $expected) {
        Move-Item -Force $DstTmp $Dst        # commit only when verified
        Write-EventLog -Message "VERIFIED $expected" ; exit 0
    }
    Remove-Item -Force $DstTmp
    Start-Sleep -Seconds ($try * 10)
}
Send-Alert "Integrity check failed for $Src" ; exit 1

Retry-on-Mismatch, Done Right

Retrying is correct because most corruption is transient — a one-time glitch that a fresh transfer clears. But retries need discipline, or they turn a bad night into a worse one:

• Bound the attempts. Retry a small number of times, not forever. An unbounded retry loop against a systematic failure — text mode, a failing disk — hammers the network and never succeeds, because the same broken step repeats every time.
• Back off between tries. Wait a little longer after each failure. If the cause is a busy link or a transient network fault, spacing the attempts gives it time to clear instead of piling on.
• Re-transfer fresh, do not resume. On a verification failure, fetch the whole file again from the start. Resuming onto a suspect partial file risks stitching bad data, as covered in where corruption actually comes from.
• Give up loudly. When the retries are exhausted, stop and escalate. A job that silently abandons a file is just a slower kind of fail-open.

The retry loop doubles as a diagnostic. A file that fails once and verifies on retry was a transient blip — logged, handled, done. A file that fails identically on every attempt is systematic, and the persistence itself is the signal that this needs a human and a look at the corruption sources.

Write-Then-Rename: Never Expose a Half-Written File

A subtle failure has nothing to do with the network: a downstream consumer grabs the file while it is still being written, or before it is verified, and processes a partial or unproven copy. The transfer was fine; the timing was not. The fix is to make the final name appear only when the file is complete and verified.

Two conventions do this, and they combine well:

• Write-then-rename. Transfer to a staging name like file.part or file.tmp, verify, then rename to the real name. On a normal filesystem a rename within the same volume is atomic — the file flips from "not there" to "fully there" with no in-between state a reader can catch. Consumers watch only for the final name, so they never see a file that is not done.
• Marker files. Write the data under its real name, then write a tiny companion — file.done or file.ok — last, after verification. Consumers wait for the marker, not the data file, and only act once the marker appears. This suits partners whose tooling cannot easily watch for renames.

Both turn "the file exists" into "the file is complete and verified," which is the property a consumer actually needs. Pair either with the fail-closed check above and a partial or corrupt file is never even nameable downstream.

Make the Job Safe to Re-Run

Unattended jobs crash, get killed by reboots, and get launched twice by an anxious operator. A well-built integrity job is idempotent — running it again produces the same correct end state, never a double delivery or a corrupted half-step. The verification machinery you already have makes this almost free.

The key is that the reference digest lets a re-run recognize work that is already done. Before transferring, the job can check whether the final file exists and already matches the reference; if it does, the delivery succeeded on a previous run and there is nothing to do, so the job exits clean instead of sending the file a second time. The staging name helps too: because a file only reaches its final name after verification, a job that died mid-transfer left behind a .part file, not a delivered one, and the next run simply overwrites the staging file and tries again. No partial delivery was ever visible, and no consumer acted on it. Design the job so that "run it again" is always a safe instruction — because sooner or later, someone will.

Alerting Without the Fatigue

A fail-closed job that stops silently protects the data but strands it — someone has to know a delivery did not arrive. Good alerting on verification failure follows a few rules:

• Alert on the failure that survived retries, not on every transient blip the retry already fixed. Otherwise the signal drowns in noise and people learn to ignore it.
• Say what a responder needs: which file, which job, the expected and actual digests, how many attempts, and where the quarantined copy sits. An alert that just says "transfer failed" wastes the first ten minutes of the response.
• Route it where someone is actually looking — the channel your team watches at the hour the job runs, not an inbox nobody reads until morning if the delivery is time-critical.

Logging the Proof

The last step is evidence. For every file, record the reference digest, the computed digest, the verdict, the number of attempts, and the timestamp. This does three jobs at once: it lets you prove a file was verified when a partner asks months later, it gives you the history to spot a disk that is starting to fail (mismatches creeping up on one destination), and it means nobody has to take your word that verification happened — the log shows it did.

Those verification records belong with your transfer logs, where they become part of the audit trail. How to keep such logs readable, centralized, and tamper-resistant is the subject of our Transfer Logging and Audit Trails series; the integrity job's contribution is to write, for every single file, the one line that says "this exact content, verified, at this time."

Write that line the same way whether the file passed or failed. It is tempting to log only successes, but the failures are the more valuable record: they are what you will comb through when a partner disputes a delivery or when you are hunting the disk that keeps producing mismatches. A log that quietly omits the bad nights tells you a comforting story and hides the one you need. Record every verdict, good and bad, and let the pattern speak.

Where This Lives in a Transfer Tool

You can assemble all of this from scripts, and for many shops that is exactly right. A dedicated automation tool packages the same pattern so you configure it rather than code it. In Sysax FTP Automation, a post-processing step runs after each transfer — the natural place to hash the arrived file and compare — and its built-in retry and error handling supplies the bounded, backing-off retry loop without hand-rolled logic. Folder monitoring covers the write-then-rename side: the job waits for a file (or its marker) to appear, then acts, so partial files never trigger processing. The value is not that scripts cannot do this; it is that a configured pipeline makes the fail-closed behavior the default rather than something each script author must remember to build. Tools like rsync bring their own verification too, as our Group I piece on rsync dry runs and verification shows — the principle is the same whichever engine you drive.

The Version to Tell a Colleague

An unattended job must verify itself and fail closed: the only path to "delivered" runs through a passing hash comparison, and every other outcome quarantines the file, retries a bounded number of times with backoff, and alerts when the retries run out. Transfer to a staging name and rename into place only after verification, so consumers never see a half-written or unproven file. Log the reference digest, the result, and the time for every file, so the proof outlives the run. Adding a hash is the easy part; the discipline is letting the hash decide.

From here, verify whole batches this way with checksum files and manifests, and understand the limit of what any of this proves in integrity vs authenticity — because a fail-closed job defeats accidents, but a determined adversary needs one more layer.

Frequently Asked Questions