Evolving a File Interface Without Breaking the Other Side

"Can we add a column to the stock file?" "Which parsers depend on the column count?" "Nobody knows." "Then no." Every estate has one: the file interface nobody dares touch. It has run flawlessly for years, which is exactly why it is terrifying. Everyone suspects that somewhere, some consumer's parser depends on some detail nobody remembers agreeing to. The only way to find out is to change something and see who screams. So the extra field the business needs waits another quarter, and the format everyone admits is awkward becomes permanent. The column that finally does get added goes in on a Friday afternoon.

The fear is rational; the paralysis is optional. File interfaces can be changed safely, and banks, bureaus, and EDI networks do it constantly. But that takes machinery built for the purpose: a change taxonomy that says honestly what breaks and version markers that make change visible. It takes the parallel-run migration that removes the cliff, and a coordination checklist that scales to many consumers. This article, the closing piece of our Files as Integration Glue series, is that machinery.

Why File Interfaces Break So Easily

A file interface is two implementations that must mirror each other, with nothing enforcing the mirror. When the producer changes its half, no compiler flags the consumer's half. The mismatch is discovered at parse time, in production, usually at night. That alone would make change delicate. Two further facts make it dangerous.

First, the consumer's parser encodes assumptions the producer cannot see. Column count. Column order. Character positions. That the date is always eight digits. That the file is sorted by employee ID. That a name never contains a newline. None of these appear in any message the producer receives; they sit silently in the other team's code, waiting. In practice, every observable behavior of an interface eventually gets depended on. Someone parses the filename, someone keys a job on the 02:00 arrival time, and someone relies on the accidental sort order. The formal spec is what you promised; the effective interface is everything the other side ever noticed. Someone is keying on the accidental sort order right now.

Second, the failure you fear least is the worst one. A change that makes the consumer's parser crash is loud, immediate, and cheap — a rejected file and an incident, handled by the error-handling machinery. A change the parser survives but misreads is silent. The columns still count, the totals still balance, and a field now means something the consumer's code was never told about. Wrong data flows downstream wearing a green status. When you classify changes, you are sorting them into loud, silent, and safe — and "silent" outranks "loud" on the danger list every time. A crash, in this light, is a courtesy.

A Taxonomy of Changes

Whether a given change is compatible depends on the format family. It also depends, uncomfortably, on how strictly the other side's parser was written, which you do not control. The honest working rule: classify against the strictest consumer you might have, and treat anything unverified as breaking.

One escape hatch is worth negotiating before you need it: the tolerant-reader agreement. In this contract clause, consumers commit to ignoring unknown trailing columns (or unknown named fields in structured formats). Once both sides genuinely honor it, purely additive changes become routine instead of ceremonial. But it must be agreed and tested, never assumed — an undocumented hope that "their parser probably ignores extras" is how quarter-long paralysis begins. "Probably" is not a clause in any contract worth signing.

Version Markers: Making Change Visible

The formats article made the case for carrying a version marker from day one; evolution is where it pays off. Carry it in two places when you can. In the header record, where the consumer's parser checks it before reading a single detail record. And carry it in the filename — EMPL_ELIG_V04_YYYYMMDD_01.csv — where it is visible before parsing begins, where humans and monitors can see it. There, two versions can exist side by side in the same folder without colliding. That last property quietly enables the entire migration pattern below.

Two disciplines make markers meaningful. The producer bumps the version on any change to shape or semantics — including "harmless" ones, because the taxonomy above shows how often harmless is a guess. And the consumer rejects unknown versions loudly instead of guessing. A clean rejection of v05 is a five-minute conversation. A v05 parsed with v04 assumptions is a silent misread with a long tail. Each version maps to a dated entry in the spec's change log, kept inside the interface contract. So "what changed between v03 and v04?" has a written answer both teams can cite. Nobody has ever announced a harmful change; they are all harmless until Monday.

The Parallel Run: Migration Without a Cliff

For any genuinely breaking change, the safest migration is the parallel run. For an agreed window, the producer publishes both formats — old and new, distinct names, same data. Each consumer tests in a window of its own (arranging those is the subject of partner test windows). Each switches when it is ready. Nobody jumps on a fixed night. The old format retires only after the evidence says nobody reads it anymore. The timeline looks like this:

The overlap is the whole safety story: at no point is any consumer forced to change on a night chosen by someone else. The mechanics that make it work:

• One extraction, two renditions. Both files must come from the same snapshot of the source. Never run the export twice, because the data can change between runs and the two formats will disagree. Produce the new format and derive the old from it (or the reverse) with a transform step. If the delivery runs through Sysax FTP Automation, that conversion script slots naturally into the job's pre- or post-processing hooks. That way, one scheduled task extracts once and delivers both renditions.
• Cross-check the twins. Where fields overlap, the two files' trailer totals must match exactly. Automate that comparison every cycle of the window. It is a free proof that the transform is faithful, and it catches the transform bug before a consumer does.
• Budget the window honestly. Dual publishing doubles files, storage, monitoring entries, and questions. That is fine for a bounded window and corrosive forever — which is why the checklist below ends with a retirement, not a shrug.

Not every change needs the full apparatus. A change classified compatible under a tested tolerant-reader agreement can skip dual publishing and take the lighter lane. In that lane, bump the version marker anyway and notify consumers with the updated spec. Ship on a stated date, and watch the first cycle's acknowledgments closely. The version bump and the notice are not optional even here. They are what keeps "compatible" an auditable claim instead of a hope. They give you a clean line of retreat if one consumer's parser turns out stricter than advertised. The general discipline is in change rollout and rollback. Reserve the parallel run for genuine breaks; use the lighter lane often enough that change stays a habit rather than an event.

One Producer, Many Consumers

With one consumer, evolution is a conversation. With many, it is a program. It starts with a question that embarrasses more teams than admit it: do you actually know who consumes this file? Files get adopted quietly. A report team found the feed useful, a script started pulling it, nobody told the producer. The documented consumer list is a hypothesis; the download log is the truth. If the exchange point runs Sysax Multi Server, its activity logging — to file and to a database — lists every account that fetched the old format and when. That list is precisely your migration register's seed, unknown consumers included. (Reading transfer logs for answers like this is a skill worth practicing before you need it; see reading transfer logs.) Every register I have rebuilt from the logs was longer than the one on the wiki.

Northgate Retail's migration register started with four consumers and finished with seven. The producer team rebuilt the list from the exchange server's download log before sending the first notice, as the checklist says to. They found three accounts fetching the old-format stock file that nobody had documented. These belonged to a returns-analysis script, a store-planning spreadsheet refresh, and a job belonging to a project that had closed two years earlier. All three were registered and notified alongside the official four. The closed project's job was retired rather than migrated, which made one fewer consumer to wait for. The store-planning team received the first change notice anyone had ever sent them. The migration finished on the planned date with no surprises, because the surprises had all been found on day one.

From there, run it like the small program it is. Track each consumer through the states notified → testing → migrated → verified. Here, "verified" means their old-format downloads stopped in the log — their word plus the evidence. Send the notice at announce, at the halfway mark, and as a last call. The craft of notifying counterparts through a disruptive change is the same whether you are changing a format or retiring a protocol. Our guide to partner communications during a migration translates directly. For external partners, fold the change into the relationship machinery your partner exchange program already runs. Partner changes ride better on existing rails than on surprise emails.

Then there is the laggard — every migration has one. Escalate through owners rather than inboxes. Offer real help (their blocker is often one afternoon of parser work). Set a final date with consequences you are actually willing to enforce. Extending the window for a consumer who is genuinely working is generosity. Extending it for one who is ignoring you is just running two interfaces forever with extra steps. Either choice can be right, but make it visibly, as a decision with a date — never by drift. And when a previously unknown consumer surfaces mid-migration, resist the annoyance: register them, hand them the spec, and fold them into the tracking. They were always depending on you; now at least it is mutual.

Renaming, Relocating, Rescheduling

Not every change touches the format. Renaming the file, moving it to a new folder, or shifting the schedule breaks consumers at the delivery layer instead. Watchers stop matching, pullers poll an empty path, and downstream jobs miss a cutoff that silently moved. The good news: delivery changes migrate with the same pattern at a fraction of the cost. For a rename or move, deliver to both the old and new name or path for the window. That is a post-transfer copy step, trivial to automate. Then watch the logs until old-location pickups reach zero, and retire. If the rename is part of adopting a saner naming standard, design the target pattern once and properly; naming convention design is the guide.

Schedule changes have no parallel-run equivalent — a file cannot arrive at two times — so they lean entirely on notice and slack. Announce like a format change, confirm each consumer's downstream cutoffs still hold with honest margin, and move in one agreed step. Walk the dependency chain before you commit. A delivery shifted one hour later can land after a consumer's import window, whose output feeds a job with a cutoff of its own. The breakage then surfaces two systems away from the change that caused it. Time-zone honesty matters doubly here — restate the new time in the contract's declared zone, and check what it means on the days clocks shift. And occasionally a true flag-day cutover is unavoidable: a source system replacement, or a change like a delimiter swap where both sides must move together. Treat it like the small outage it is. That means a freeze on other changes and a joint checklist with named owners for each step. It means a first-file verification the moment the new shape flows (parse it, check its totals, confirm its ack). And it means a rollback path that keeps the old producer configuration warm until the evidence is in. Flag days are survivable; unplanned flag days are the thing this whole article exists to prevent.

The Coordination Checklist

Everything above compresses into one page. Copy it into the change ticket and work down:

INTERFACE CHANGE CHECKLIST — IF-042, v03 -> v04

[ ] Change classified: compatible / breaking / semantic (strictest-consumer rule)
[ ] Spec and contract updated; change log entry written; new version number set
[ ] Version marker present in header record and filename
[ ] Consumer register rebuilt from exchange-server download logs (not memory)
[ ] Every consumer notified, with spec attached; notice acknowledged and tracked
[ ] Parallel-run start agreed; both renditions produced from one extraction
[ ] Automated cross-total check between old and new outputs, every cycle
[ ] Per-consumer status tracked: notified / testing / migrated / verified-in-logs
[ ] Laggards escalated with a final date; window extension decided, not drifted
[ ] Old-format downloads at zero for N consecutive cycles (log evidence attached)
[ ] Deprecation date passed; old format retired; monitors and docs updated
[ ] Post-change review: what depended on the old shape that we did not predict?

The last line feeds the next change. Every migration teaches you one more thing consumers observably depended on — an arrival time, a sort order, a filename fragment. Each lesson recorded in the contract makes the following evolution one surprise shorter. There is always one more thing someone was parsing.

Interfaces Age Well When Change Is Routine

The interface nobody dares touch is not stable — it is brittle, waiting to fail on the day change becomes mandatory instead of optional. The machinery in this article is what converts that fear into maintenance. It includes a taxonomy that names what breaks, versions that make change visible, and parallel runs that remove the cliff. It includes a register that makes consumers knowable, and a checklist that makes the whole thing repeatable. Teams with this machinery ship format changes the way they ship patches: routinely, with evidence, and without meetings that need an apology first. The column goes in on a Tuesday morning, with a version number, and nobody screams.

That closes the loop this series opened. Files earn their place as integration glue — the honest case stands. But they hold that place through discipline: a written contract, formats that age, acknowledgments with real semantics, error handling by convention, and evolution without breakage. None of it is glamorous. All of it is why the nightly feed will still be running, correctly and unremarkably, long after everyone in the room has changed jobs.

Frequently Asked Questions