Freshness Checks: Alerting When Expected Files Do Not Arrive

Every monitoring system in your estate is built to notice events: a job failed, a login was refused, a disk filled up. But the most expensive problem in file transfer is not an event at all. It is an absence — the payroll file that is not there at six in the morning, the partner upload that never came, the nightly extract that simply did not happen. Absences write no log line, raise no error, and trip no alert, because on your side of the fence, nothing occurred.

The cure is a freshness check: a small, independent, scheduled test that asks "has the file I expect actually arrived by the time I expect it?" — and raises an alert when the answer is no. It is the cheapest high-value monitoring you can add to a transfer estate, and this article builds it properly: per-flow deadlines, grace windows that stop false alarms, weekend and holiday awareness that lives in configuration instead of code, and a complete worked monitor you can adapt this afternoon.

This is part of our Monitoring & Alerting series, and it picks up exactly where why jobs fail silently left off: the freshness check is the single test that catches most silent-failure modes at once.

The Absence Problem: What No Log Can See

Start with why this needs its own technique. Suppose a partner pushes an invoice file to your SFTP server every night. One night their export job dies. What happens on your side? Nothing — literally nothing. No connection is made, so your server logs no session. No job of yours failed, so no error fires. Every record on your side is consistent with a quiet, healthy night. The only place the problem exists is in the gap between what happened and what was supposed to happen, and no log file contains the word "supposed."

Pull flows are barely better off. If your scheduled job connects to the partner and downloads whatever is waiting, and nothing is waiting, the job runs cleanly, transfers zero files, and reports success — the "succeeded at nothing" mode from the silent failures article. Your logs now contain an event, but a misleading one: a success record for a night that failed.

In both cases the missing ingredient is the same: the expectation. Today, that expectation lives in someone's head — "the bank file shows up around two" — which means it is checked only when that someone happens to look, and it evaporates when they change roles. A freshness check is nothing more than that expectation written down in a checkable form and tested by a machine on a schedule. Everything else in this article is detail.

What a Freshness Check Actually Tests

The canonical freshness check asks one question per flow: does the newest file matching this pattern, in this location, carry a timestamp recent enough to be today's delivery? Three parameters define it:

• Where to look — the landing folder on your side: the directory your server receives into, or the local directory your pull job downloads into. Check the place the business consumes from, as late in the chain as you can, so the check covers as much of the pipeline as possible.
• What to match — a filename pattern such as invoice_*.csv, so an unrelated stray file cannot satisfy the check.
• How fresh is fresh — the deadline: "by 03:00 on weekdays," or equivalently "newest matching file must be less than a day old at check time."

Notice what the check does not ask. It does not ask whether any job succeeded, whether the scheduler fired, or whether the network was up. It looks only at the outcome, which is why one test catches so many causes: sender never ran, sender ran and failed, your pull job never ran, your pull job pulled nothing, the file went to the wrong folder — every one of those ends in the same observable fact, a stale landing directory. And because the check runs independently of the transfer jobs — its own schedule, ideally its own machine — it does not share their failure modes.

One boundary worth drawing honestly: job-level notifications and freshness checks are partners, not substitutes. A tool like Sysax FTP Automation will email you when a scheduled task fails — that is your fastest signal for the failures a job can see. But no job can report the run that never started or the pull that correctly found nothing, and that is the half the freshness check exists to cover.

Deadlines and Grace Windows on a Timeline

The tuning of a freshness check is all about time, and it is easier to see than to describe. The diagram below lays one flow on a clock: the file usually lands just after two, the deadline is set at three, a grace window runs to half past, and the business truly needs the file by six.

Each element earns its place. The expected-by deadline is not the average arrival time — it is set comfortably after the latest normal arrival you have observed, so ordinary jitter never crosses it. Watch a few weeks of history before you commit: if the file lands between 01:55 and 02:40 depending on the sender's load, an expected-by of 03:00 is honest and 02:15 is a false-alarm machine.

The grace window is the buffer between "late" and "alarm." Its job is to absorb the rare-but-normal slow night — the sender's monthly heavy run, a retry that succeeded on the second attempt — so that the alert, when it finally fires, means something a human should act on. A grace window of fifteen to forty-five minutes suits most nightly flows. Resist making it hours long to silence a flaky feed; that is treating the thermometer, not the fever. If a flow genuinely arrives anywhere in a four-hour spread, set the deadline after the spread and have the honest conversation with the sender about predictability.

The third mark is the one teams forget: the business deadline, the moment the absence starts costing something — payroll cutoff, warehouse opening, the batch window closing. Your alert must fire early enough to leave a repair window: time for a human to see the alert, diagnose, and rerun or escalate before the business deadline lands. Working backwards is the design method: business deadline at 06:00, minus ninety minutes of realistic repair time, means the alert must fire by 04:30 at the latest — so expected-by plus grace must sit no later than that.

Business-Day Awareness Without Hard-Coded Dates

The first Monday after you deploy a freshness check, it will teach you its next lesson: most business flows do not arrive seven days a week. A file that lands every weekday is absent every Saturday, and a naive checker will dutifully page someone about it. The fix is never to hard-code dates or special cases into the script — "skip the checks this weekend" patches rot instantly. The fix is to make expected days part of each flow's configuration, and keep the checker's logic generic.

Two rules cover almost everything:

• Weekday rules per flow. Each flow declares the days a fresh file is expected: Mon-Fri for the invoice feed, Mon only for the weekly price list, Daily for the replication drop. On a day outside the flow's list, the checker expects nothing and stays quiet. The checker computes "what weekday is it?" at run time — no dates ever appear in the config, so it is correct forever.
• A holiday file, maintained as data. Public holidays are the one thing that genuinely changes year to year, so isolate them in one small text file — one non-processing date per line, in YYYY-MM-DD form, refreshed once a year as a calendar task. The checker treats any date in the file like a weekend for flows marked "business days only." One file, shared by every flow, owned like any other piece of config.

Mind the Monday-morning subtlety: on a business-days-only flow, "the newest file should be less than a day old" is wrong on Monday, when the newest legitimate file is Friday's. The clean formulation is: on an expected day, a file for that day must arrive by the deadline; on a non-expected day, the check is skipped. Anchoring the check to "a file since midnight" (or since the start of the expected delivery window) rather than "a file in the last N hours" makes weekends and holidays fall out naturally. And if the sender delivers a catch-up double file after a holiday, that is a count expectation, which the per-flow config can also carry.

A Worked Freshness Monitor

Here is the whole idea assembled, sized for a small estate. Start from your flow inventory — the file flow census if you have done one — and pick the flows whose absence someone would phone you about. For each, fill in one row of the expectations table:

Notice the second row: for an outbound flow, you check the archive copy your own job writes as it sends — evidence on your side that the send happened. The table then becomes a plain configuration file the checker reads, one line per flow:

# flows.conf — name | directory | pattern | expected_by | grace_min | days
invoices_in  | D:\landing\invoices | invoice_*.csv     | 03:00 | 30 | Mon-Fri
payroll_out  | D:\sent\payroll     | pay_YYYYMMDD.txt  | 05:30 | 15 | Mon-Fri
db_extract   | /data/extracts      | extract_*.zip     | 04:15 | 45 | Daily
prices_wk    | D:\landing\prices   | prices_*.xml      | 07:00 | 60 | Mon

The checker itself is short. In sketch form — this is the logic, not polished production code:

# freshness check — run every 15 minutes from the scheduler
for each flow in flows.conf:
    if today's weekday not in flow.days:            skip   # not an expected day
    if today is listed in holidays.txt:             skip   # treated like a weekend
    if now is before flow.expected_by + flow.grace: skip   # too early to judge

    newest = most recent file in flow.directory matching flow.pattern
    if newest exists and newest.mtime is after last midnight:
        record "flow OK, arrived at newest.mtime"          # for the daily report
    else:
        raise alert:  "flow has not arrived; expected by flow.expected_by"

Every mainstream scripting stack expresses this in a few dozen lines — a shell loop over find results, a PowerShell script comparing LastWriteTime, a small Python program. The pattern matters more than the language. Three deployment notes: schedule it every ten to fifteen minutes across your delivery hours (a checker that runs once at 09:00 turns every overnight miss into a morning surprise); run it on a machine other than the one running the transfer jobs, if you can, so one dead host does not take out both the work and the watcher; and make it remember what it has already alerted on, so a missing file pages once and then holds, rather than re-paging every fifteen minutes — more on repeat-suppression in alert design.

Deciding What "Arrived" Means

The word "arrived" hides four traps, and each one has produced a false all-clear somewhere:

• The zero-byte file. A sender's failed export can leave an empty file with today's timestamp — present, fresh, and useless. Add a minimum-size condition to the check: for most feeds, "at least a few hundred bytes" separates real deliveries from stubs.
• The half-written file. A file still being uploaded has today's timestamp but not today's content. If your senders use temporary names or an atomic rename on completion, match only the final name and the problem disappears; if not, require the file to be a few minutes old before counting it. The full toolkit lives in our partial-file safety series.
• Yesterday's file, matched today. A pattern like invoice_*.csv is satisfied by an old file if the checker only asks "does a match exist?" — always compare timestamps, not mere existence. Better still, when filenames carry a datestamp token such as report_YYYYMMDD.csv, check for today's name specifically; naming conventions that make this trivial are covered in the file naming and datestamping series.
• The consumed file. If a downstream process sweeps the landing folder every few minutes, the file may be gone before the checker looks. Point the check at the archive copy the pipeline keeps (it should keep one), or at the processed-files ledger, rather than at a folder designed to be empty.

When the Alert Fires: The First Five Minutes

A freshness alert tells you something true and incomplete: the file is not there. The diagnosis is about locating the break in the chain, and there is an efficient order:

1. Establish which side is broken. For push flows, check your server's activity log: did the partner connect at all? A server that logs richly settles this in one query — Sysax Multi Server, for instance, records every session and transfer to both a log file and a database, so "show any logins from this partner account since midnight" is a single lookup, and its answer cleanly splits "their export never ran" from "they connected and something on our side went wrong."
2. Check your own job. For pull flows, did the scheduled task fire and what did it claim? A run that "succeeded" moving nothing points upstream; no run at all points at your scheduler or host.
3. Check the landing area itself. Full disk, changed permissions, or a renamed folder produce absences with purely local causes.
4. Then communicate. If the break is on the sender's side, the alert has done its finest work: you are calling them before their business day starts, instead of them calling you after yours has gone wrong.

Two closing disciplines keep the monitor trustworthy. Acknowledge alerts with an expiry — "known issue, sender rerunning, recheck at 05:00" — rather than disabling the check, because disabled checks are how the fourth silent-failure mode is born. And remember the checker is itself a scheduled job that can die quietly; it needs a heartbeat of its own, which is precisely the business of monitoring the monitoring.

Start Small, Then Trust It

A freshness monitor is one config file, one short script, and one scheduler entry — an afternoon of work that watches the outcomes your business actually cares about, independent of every job, sender, and scheduler that produces them. Begin with your three most critical flows, tune expected-by and grace against a few weeks of real history, add the weekday and holiday rules as configuration, and fold the results into your daily status report so the quiet days build confidence too.

From here, the natural companions are job status monitoring — the exit codes, logs, and heartbeats that tell you why a file is missing — and alert design, which makes sure the message the checker sends at three in the morning is one a groggy human can act on.

Frequently Asked Questions