The Arrival Contract: What a Watcher May Assume

Here is the trap at the center of every watch folder. A file does not appear on disk the way a letter appears in a mailbox — complete, sealed, done. It appears the way a house appears on a construction site: the name goes up first, and the contents arrive over the following seconds or minutes. A watcher that reacts to the name is reacting to the foundation pour. Whatever it does next — parse, transfer, move — it does to a fraction of a file.

Most teams discover this the expensive way. An order file is processed with the last two hundred rows missing. A zip forwarded to a partner fails to open. An import ran perfectly on Tuesday's small file and mangled Thursday's large one. The fix is not cleverer code first. It is an agreement between whoever produces the file and whatever watches for it. The agreement states how "this file is complete" gets signaled, and what the watcher is entitled to assume until it is. We call that agreement the arrival contract. This article of our watch folders and event-driven transfers series shows how to design one. It covers the strong conventions worth negotiating and the defensive checks for when you cannot negotiate. It gives you a written intake agreement you can hand to every sender.

Presence Is Not Completeness

Start with why the trap exists at all. Writing a file is a process, not an event. A local application opens the file (the name now exists, size zero), streams content into it, and closes it. That takes milliseconds for a small file, much longer for a large one on a busy machine. An upload over SFTP or FTPS stretches the same window across the network. A 500 MB file on a modest partner link can sit visibly in your inbox, growing, for ten minutes. During the whole window, the file is real, listable, openable — and wrong.

What makes the problem genuinely nasty is that partial files often look plausible. A half-written CSV is still a valid CSV as far as any parser knows. It simply ends early, and the import happily loads 6,100 of 11,000 rows. A truncated PDF or zip at least fails loudly. The quiet failures are the dangerous ones, and they are also the rare ones. Small files usually win the race, so the flow works in every test and fails only under size and load. That is why teams chase these bugs for months. The full anatomy of the race, along with the deep mechanics of the fixes, lives in our companion partial-file safety series. This article stays at the level where watch-folder design happens: who promises what, to whom.

Because that is the real insight. Detection — covered in polling vs filesystem events — can tell you a name exists. Nothing on the watcher's side can ever prove the producer has finished; only the producer knows that. So completeness is either signaled by the producer, or inferred by the watcher. Signals are reliable and inference is guesswork. The whole craft of intake design is moving as many flows as possible from guesswork to signal.

The Contract, and Its Three Strengths

An arrival contract answers one question: when may the watcher touch a file? Contracts come in three strengths, and it pays to know which one each of your flows actually has:

• Strong — the producer signals. The file is invisible or ignorable until a deliberate act by the producer marks it complete. That is a rename to the final name, or a companion marker file. The watcher acts on the signal, never on the data file's mere presence.
• Weak — the watcher infers. The producer writes the final name directly and offers no signal. The watcher defends itself with settle checks and age rules — reasonable inference, honestly fallible.
• None — the watcher hopes. The watcher grabs whatever it sees. Every flow starts here by default, and every partial-file incident is this contract collecting its fee.

Strength is set by the weaker party: you can only have a strong contract if the producer cooperates. So the practical sequence is always the same. Ask for a signal first, and fall back to inference only where the producer cannot or will not change. Let's take the two levels in that order.

Strong Contracts: Temp Names and Markers

Two conventions cover nearly every cooperative producer. Both are simple enough to describe in an email, which is exactly how they usually get agreed.

The temp-name convention

The producer writes the file under a working name that the watcher is instructed to ignore, then renames it to the final name as the last act:

producer:  upload as  acme_orders_YYYYMMDD.csv.part     (content streaming in)
producer:  rename to  acme_orders_YYYYMMDD.csv          (instant - this is the signal)
watcher:   ignores *.part; the final name only ever exists complete

The rename is the trick. Renaming within the same folder changes the directory entry, not the data, so it is effectively instantaneous and all-or-nothing. There is no moment when the final name exists half-written. From the watcher's point of view, complete files materialize out of nowhere, which is precisely the mailbox illusion we wanted. Common ignore patterns are a suffix (.part, .tmp, .filepart) or an upload subfolder the producer renames out of. Many transfer clients can do temp-name uploads natively, so the "cooperation" needed is often a checkbox on the sender's side. There are caveats: the rename must stay on one volume, and a crashed producer strands a .part file that needs cleanup. These are covered in depth in the partial-file safety pillar.

The marker-file convention

Sometimes the producer cannot rename — its software writes the final name directly and offers no hook. The fallback is a second, trivial file whose only job is to say "done":

producer:  upload  acme_orders_YYYYMMDD.csv        (data file - may take minutes)
producer:  upload  acme_orders_YYYYMMDD.csv.done   (zero bytes - written last)
watcher:   triggers on *.done, processes the matching .csv, removes both

The ordering rule carries the whole contract: the marker is written only after the data file is completely delivered. Because the marker is tiny, it cannot itself be meaningfully half-written. Markers also scale up nicely. A batch of five files plus one batch.done marker turns "process these together" into a clean trigger. Processing files together is normally an awkward fit for folder automation. A marker with content (row counts, checksums) becomes a small manifest the watcher can verify against. The discipline cost is cleanup. Orphaned markers, markers without data files, and data files whose marker never came all need a rule. That is one of the jobs of the error design in designing watch folders that handle failure.

Choosing between the two: prefer temp names when the producer's tooling supports a rename — one file, no cleanup, nothing to forget. Use markers when renames are impossible, when batches must travel as a unit, or when you want the signal to carry verification data. Both beat inference by a mile.

Weak Contracts: Settle Checks and Debouncing

Now the uncooperative case: the file appears under its final name, growing in place, and no signal is coming. The watcher must infer completeness from behavior, and the standard inferences are worth knowing precisely because their limits are part of the contract.

Debouncing comes first. The word is borrowed from electronics, where a pressed switch "bounces" and registers several contacts for one press. Files bounce too: a single delivery can raise a burst of changes — created, written, written again, attributes touched. An event-driven watcher that launches processing per change will start the machine five times for one file. Debouncing collapses the burst: after each change to a file, restart a quiet-period timer. Only when the timer expires with no further change does the file become a candidate. One decision per file, not per event.

The settle check is the completeness inference itself. Sample the file's size (and modification time), wait, sample again, and treat "no change across the window" as "probably complete." A minimum-age rule is the blunt cousin: touch nothing until it has existed untouched for N minutes. It trades latency for safety with no sampling logic at all. In the polling loop from the detection article, this is the size1/size2 comparison; in configurable tools it appears as a stability wait or delay setting.

The honesty section: settle checks fail in a specific, predictable way. They cannot distinguish finished from stalled. A producer that pauses mid-write longer than your settle window produces a file that holds still, passes the check, and is processed half-done. It might be a laptop upload riding a flaky connection, or a database export that thinks between chunks. Lengthening the window shrinks that risk while growing latency for every healthy file. No window eliminates the risk, because the pause can always be longer. That is why settle tuning is guesswork about sender behavior, and why generous windows are the rule for partner-facing intake. It is why an inferred contract should be treated as a stopgap while you lobby the producer for a real signal. The tuning craft is deep enough that the partial-file safety series devotes a full article to it. That includes choosing windows against real sender patterns, and what open-handle checks can and cannot add.

The Intake Agreement: What to Settle with Every Sender

Everything so far becomes real the day you write it down and hand it to the producer. For partners, this is one page attached to the onboarding exchange. The wider version of that conversation is our trading partner onboarding guide. For internal teams and applications, the agreement is a section in the runbook. The checklist below is the whole agreement — copy it, fill in the brackets, and you have an arrival contract in writing:

INTAKE AGREEMENT - [flow name], [producer] -> [watcher/owner]

1. Drop location   : only [path or account]; senders never write elsewhere
2. File naming     : [pattern, e.g. acme_orders_YYYYMMDD_NNN.csv]
                     nonconforming names are not processed (parked + alert)
3. Completeness    : [ ] upload as .part, rename when done   (preferred)
                     [ ] marker file [name].done written after data
                     [ ] none - we apply a [N min] settle window
4. Batches         : one file per drop / batch with manifest marker [name]
5. Size and format : expected [range]; [encoding]; [compressed? encrypted?]
6. Corrections     : resend uses a NEW name (never overwrite); note seq/number
7. Timing          : expected by [time, timezone]; late files still processed
8. On failure      : file parked in error area; [who] notified within [time]
9. Acknowledgment  : [none / done-marker back / email / listing visibility]
10. Contacts       : producer [name/address]  -  watcher owner [name/address]

Three of the lines earn a comment. Line 2 makes the naming pattern part of the contract. That lets the watcher treat the name as data — routing on it, datestamp-sorting on it, rejecting on it. Designing patterns that sort and parse cleanly is the subject of the file naming and datestamping series. Line 6 quietly prevents a whole incident class. A sender who overwrites a file you are mid-way through processing creates a race no contract survives. So corrections always arrive as new names. And line 9 makes explicit what folders do worst — telling the sender anything back — so nobody assumes an acknowledgment that does not exist.

Producers That Will Never Cooperate

Some producers cannot hold up any contract: the lab instrument that writes in place with no rename option, the ancient application nobody dares touch, the human. For these, combine defenses rather than choosing one:

• Give them a private staging folder, and put a small relay on their side, or on the same machine. That is a script or scheduled task that moves finished files into the real inbox using a proper rename. The relay speaks strong-contract; the awkward producer never has to.
• Set the settle window from evidence, not optimism. Watch the producer's real behavior for a week: largest file, slowest delivery, longest mid-write pause you can find in timestamps. Size the window beyond it, then accept the latency as the price of that producer.
• For humans, expect bursts and edits. People drop thirty files at once, save the same spreadsheet four times in two minutes, and occasionally drag in a folder. Debouncing absorbs the saves; a minimum-age rule absorbs the indecision; a written one-line instruction ("finish the file, then drop it") absorbs most of the rest.

And one architectural note outranks every workaround. If the files reach you as uploads to a server you operate, the server itself knows when each upload finishes — with certainty, not inference. Server-side event triggers turn that knowledge into action. In Sysax Multi Server, the Pro and Enterprise editions can run actions on server events such as a completed upload. That is the strongest arrival signal available anywhere, because it comes from the software that performed the delivery. Where you control the receiving server, prefer that fact to any filesystem guesswork.

Enforcing the Contract at the Edge

A contract nobody enforces decays into folklore. Enforcement lives in the watcher's first steps, before any real processing:

1. Filter by pattern. Process only names matching line 2 of the agreement. Ignore agreed temp patterns entirely. Park anything else where it will be seen. Never delete a stranger's file silently, and never process it either.
2. Apply the completeness rule for this producer: wait for the rename, wait for the marker, or run the settle window. Only then claim the file by moving it out of the inbox, per the hot folder pattern.
3. Validate after claiming. Cheap structural checks catch both contract violations and the settle check's rare losses while the file is still one move from the sender. Examples include checking for an expected header, checking for a plausible row count, and checking that the archive opens.
4. Log the arrival story. First seen, settled, claimed, validated — one line each, with the filename. When a partner asks "what happened to Thursday's file," these lines are the answer. Our guide on what to log sets the wider standard.

If you are configuring rather than scripting, the same enforcement maps onto tool settings. Folder monitoring in Sysax FTP Automation watches an intake folder and runs a transfer task when files arrive. Its surrounding machinery provides the enforcement scaffolding. That means retry and error handling for the steps after intake, and email notification when something needs a human. You supply the contract itself: which names are legal, and how completeness is signaled. A tool can watch a folder for you; only an agreement makes the folder trustworthy.

The Contract in One Breath

A file's presence is the weakest fact in file transfer, and everything in this article strengthens it. If the producer can signal — temp-name rename or marker file — take the signal and design the watcher to see only complete files. If the producer cannot, infer with debouncing, settle windows, and minimum ages, sized from real behavior. Back that inference with post-claim validation, because inference sometimes loses. Write the whole arrangement down as an intake agreement per sender and enforce it at the edge of the workflow. Escalate to server-side events where you own the receiving server. Do this once per flow and partial files stop being a mystery and become a contract violation — with a name, an owner, and a fix.

From here, a natural next step is designing watch folders that handle failure — where contract violations and validation rejects actually go. Another is the series capstone, building a watch folder workflow end to end. There, this contract appears as working folder layout and log lines. For the mechanics under the conventions — why renames are atomic, how settle tuning really behaves — the partial-file safety series is the deep dive.

Frequently Asked Questions