War Story: The Misdirected File

"Which account downloaded it?" Hana asked. The email Kofi had half-written to the customer, apologizing for a short delay, stopped being the right email. An analytics bureau had delivered one customer's weekly loyalty-member export into a different customer's pickup folder. The export had tens of thousands of rows with email addresses and postcodes. The recipient's automation collected it twelve minutes later. The cause was one row in a routing table, copied from the row above it and edited in two of its three columns. The job reported success. The mistake was found by the customer whose file never arrived, not by the customer who received it.

This postmortem is about the two hours after discovery as much as the mistake itself. It covers what containment means when the file has already been downloaded and how the server log settled who had it. It explains why the first instinct, fix the route quietly and re-send, was wrong. It also covers who owns the decision to disclose. Marrow Lane Analytics, its customers, and the people are composites with invented names. The mechanism is exact. This postmortem is part of our War Stories series. It ends with a misdelivery runbook and a checklist for your own routing.

The Estate, and Two Customers One Row Apart

Marrow Lane Analytics produces loyalty-program analytics for retailers. Each customer has an account on the bureau's SFTP server, sftp.example.com, confined to its own folder. Each customer collects finished exports from its outbox. A nightly job on jobs01.example.com generates the exports. A small router script then reads a routing table and copies each into the right customer's folder. Two customers matter here, and their names are the point: Alderby Stores and Altmore Foods. Their customer codes are ALD and ALT, adjacent in every list the bureau kept. (You can see it coming. So could everyone, afterward.)

D:\exports\ready\LOY-ALD-DAY_<day>.csv       export files, named by export code
D:\xfer\config\routes.csv                    export code  ->  customer  ->  destination folder
/customers/alderby/outbox                    Alderby's pickup folder on sftp.example.com
/customers/altmore/outbox                    Altmore's pickup folder

The routing table was a CSV on the job server, edited by hand and not kept in version control. The router validated only its shape: three columns per row, no duplicate export codes. Each customer's automation polled its outbox every fifteen minutes, downloaded everything present, and deleted it from the server. That last habit is normal and it matters. By the time anyone looked, the misdirected file was no longer on the bureau's server at all. The per-customer layout, one account confined to one tree with its own outbox, follows inbox and outbox conventions. That layout is why this story has one recipient rather than several.

The Timeline

• Sep 03 14:30 — Altmore has asked for a weekly export in addition to its daily one. Kofi, the transfer administrator, opens routes.csv and copies the existing weekly row for Alderby. He edits the export code to LOY-ALT-WK and the customer to Altmore Foods. The destination column keeps the pasted value. The router's validation passes. Ticket closed in nine minutes.
• Sep 05 06:00 — The weekly export job produces LOY-ALT-WK_w36.csv: Altmore's loyalty members, with member ID, email, postcode, spend band, and a churn score. The router delivers it to /customers/alderby/outbox and logs success.
• Sep 05 06:12 — Alderby's poller, as the account alderby, downloads every file in its outbox, the Altmore file included. It deletes them from the server.
• Sep 05 08:40 — Petra, at Altmore, emails: their new weekly file was due by 07:00 and has not arrived. Altmore has a freshness check. The bureau does not.
• Sep 05 09:05 — Kofi reads the job log: delivered LOY-ALT-WK_w36.csv to /customers/alderby/outbox OK. He understands immediately.
• Sep 05 09:08 — He corrects the row, re-runs the router for Altmore, and begins a reply to Petra apologizing for a delay. Hana, his team lead, reads the chat and asks one question: which account downloaded it?
• Sep 05 09:20 — The server's activity log answers: downloaded once, by alderby from 198.51.100.61 at 06:12:41. The file was deleted at 06:12:43, with no other access. The log excerpt is exported and saved before anything else is touched.
• Sep 05 09:30 — Hana calls Jonas, Alderby's integration analyst. Their poller drops everything into an import staging area. Their import matches on the prefix LOY-ALD-. So the Altmore file was shunted to an "unrecognized" folder, unopened. Jonas is asked not to open it and not to delete it yet.
• Sep 05 10:15 — The incident goes to Mei, the privacy officer, and the account managers for both customers. The disclosure decision leaves IT's hands.
• Sep 05 11:30 — Hana and Altmore's account manager call Petra with the facts. They explain what was sent, where, who received it, what the log shows, and what has been done. A written summary follows at 14:00.
• Sep 05 16:20 — Jonas confirms in writing that the file has been deleted from staging and from the unrecognized folder. He attaches their poller's log for the morning.
• Sep 08 — Altmore's privacy team weighs three facts. The recipient was a contractually bound party, the file was never opened, and the deletion is evidenced. On that basis, the team records the incident as low risk with no notification to individuals. That was their decision to make, in their jurisdiction, with their counsel. It could easily have gone the other way.
• Sep 09 10:00 — Postmortem.

Anatomy of the Edit

Here is the routing table after the change, with the row that caused the incident. It passed validation because validation checked only that the row had three columns and a unique code.

export_code,customer,destination
LOY-ALD-DAY,Alderby Stores,/customers/alderby/outbox
LOY-ALD-WK,Alderby Stores,/customers/alderby/outbox
LOY-ALT-DAY,Altmore Foods,/customers/altmore/outbox
LOY-ALT-WK,Altmore Foods,/customers/alderby/outbox        <-- copied from LOY-ALD-WK; two columns edited, one forgotten

Read the row on its own and the error is invisible: a plausible code, a real customer, a real folder. Read it against its own customer and it is obviously wrong. Altmore's file has no business in alderby's folder. But nothing read it that way. The router did precisely what it was told. The log recorded a success because, from the router's point of view, it was one:

Sep 05 06:00:41  router: LOY-ALT-WK_w36.csv (38,214 rows) -> /customers/alderby/outbox  OK
Sep 05 06:12:41  sftp:   download  alderby  198.51.100.61  /customers/alderby/outbox/LOY-ALT-WK_w36.csv  2.9 MB
Sep 05 06:12:43  sftp:   delete    alderby  198.51.100.61  /customers/alderby/outbox/LOY-ALT-WK_w36.csv

Two things about that excerpt shaped everything that followed. The second and third lines exist only because the SFTP server logs every download and delete with account and source address. Sysax Multi Server writes that activity log to file and, optionally, to a database. Those lines turned "we think Alderby probably got it" into "one account, one address, one download, one delete, nobody else." And the third line meant the file was gone from the bureau's server before anyone knew there was a problem. So containment had to happen at the customer's end.

Containment, and the Decision That Was Not IT's to Make

The quiet fix, and why Hana stopped it

Kofi's first three minutes were exactly what a good administrator does with an ordinary routing bug. That means fixing the row, re-running the delivery, and apologizing for the delay. The trouble is that this was not an ordinary routing bug. The file contained personal data belonging to Altmore's members, and it had been delivered to a third party. That fact does not become smaller by being unmentioned. It becomes a second incident, the concealment, layered on the first. Hana's question moved the response from "fix" to "contain." The postmortem's first action item was to write that question at the top of the runbook.

Evidence before cleanup

Before the correcting row was committed, before the re-delivery, before any call, the log lines above were exported with their timestamps. They were stored with the incident record. That is the discipline documenting a file journey teaches. Here it did three jobs at once. It defined the scope (one recipient). It gave Altmore's privacy team something to reason from. It gave Alderby a precise request, this file, this name, downloaded at this minute, instead of a vague one.

Retrieve where possible, verify where not

The bureau could not retrieve the file; the recipient's own poller had already deleted it from the server. Containment therefore meant confirming where the file went inside Alderby's estate. It meant asking for deletion from every location including backups where practical. It also meant getting that confirmation in writing with their own logs attached. Alderby's import design helped, because a prefix check quarantined the unfamiliar file. But the postmortem noted that a less careful importer would have loaded Altmore's members into Alderby's marketing database. Nothing on the bureau's side would have prevented it.

Whose decision is disclosure

Under most privacy regimes a processor that mishandles a customer's personal data must tell that customer promptly. What happens next, regulator, individuals, nothing, is the customer's call as the controller, made with their own counsel. So the administrator's job was not to decide whether this "counted." It was to hand precise facts to the privacy officer and account manager within the hour and let the people with the duty decide. Personal data transfer incidents lays out that hand-off and what the privacy team needs from IT.

Root Cause and Contributing Factors

• Routing was a hand-edited file with no review. A change deciding where customer data goes was made and deployed by one person in nine minutes. There was no second reader and no history to diff against.
• The destination was typed, not derived. The table carried a free-text path in every row. A path that can be pasted can be pasted wrong; a path computed from the customer code cannot.
• Validation checked shape, not meaning. Three columns and a unique code says nothing about whether the destination belongs to the customer named in the row. The one rule that mattered was not written.
• Adjacent, similar customers. ALD and ALT, one row apart. Copy-paste errors are attracted to near-neighbors.
• No test delivery for a new route. The first file down the new route was the real one, with real data.
• The bureau had no freshness check of its own, so the miss was found by the customer, on the customer's schedule.
• No misdelivery runbook. The first responder's instinct was a quiet fix, because nothing had ever told him otherwise.
• The export carried more personal data than the analytics needed. Email addresses and postcodes were in the file by habit. Hashed member IDs would have served the analysis and shrunk the incident.

What Was Actually Changed

The structural fix was to stop treating routing as data and start treating it as reviewed configuration. routes.csv went into version control. Every change needs a second person's approval. The router records the hash of the last approved table. It refuses to run if the file on disk differs from that table. So an unreviewed edit cannot take effect. The destination column was removed altogether. The router now derives /customers/<customer_code>/outbox from the row's customer code. So there is nothing left to paste wrong. A field that does not exist is the best-validated field there is. New routes deliver a synthetic zero-row file first, and go live only when the customer confirms receipt. The bureau added its own expected-deliveries check so it would notice a miss before a customer did. And the loyalty exports lost their email and postcode columns.

For estates that keep a destination column, the validator below is the copyable part of this article. It enforces the one rule that was missing: a row's destination must be its own customer's folder.

# validate-routes.ps1 -- refuse a routing table that could misdeliver; run before every router run
$Rows   = Import-Csv 'D:\xfer\config\routes.csv'
$Root   = 'D:\xfer\customers'                 # local view of /customers on the server
$errors = @(); $seen = @{}

foreach ($r in $Rows) {
    $code = $r.export_code
    if ($seen.ContainsKey($code)) { $errors += "$code: duplicate export code" }
    $seen[$code] = $true

    # The customer code is the middle token of the export code: LOY-ALT-WK -> ALT
    $cust = ($code -split '-')[1].ToLower()
    $expected = "/customers/$cust/outbox"
    if ($r.destination -ne $expected) {
        $errors += "$code: destination '$($r.destination)' is not this customer's own folder '$expected'"
    }
    if (-not (Test-Path (Join-Path $Root "$cust\outbox"))) { $errors += "$code: customer folder for '$cust' does not exist" }
}

if ($errors.Count -gt 0) { $errors | ForEach-Object { Write-Error $_ }; exit 1 }
Write-Output "routes.csv: $($Rows.Count) routes valid"

Run against the table above, it stops on the fourth row: LOY-ALT-WK: destination '/customers/alderby/outbox' is not this customer's own folder '/customers/altmore/outbox'. Nine minutes of editing, one line of refusal. The rule assumes a naming convention where the customer code lives inside the export code. If yours differs, the shape is the same. Derive what the destination should be from something the row already says, and refuse when the typed value disagrees.

The Lessons, and Where to Learn Each Fix

1. Routing is configuration, and configuration gets reviewed. Version it, diff it, approve it, and make the runtime refuse anything unapproved. Routing files to destinations covers the patterns, including derived destinations.
2. Derive, don't type. Anything that can be computed from a customer code should be. A field that has to be pasted correctly will eventually be pasted incorrectly.
3. Validate meaning, not just shape. The rule that matters is the one that says this data may only go to this party. Validating files before sending is the pre-flight discipline; apply it to the routing table as well as the file.
4. Canary every new route. The first file down a new path should contain nothing.
5. Keep the log that proves who got what. Downloads and deletes with account and address turned a guess into a fact. What to log and reading transfer logs cover what to keep and how to query it fast.
6. Contain, capture, escalate; never quietly fix. Personal data transfer incidents describes the hand-off to privacy and legal. The article on partner SLAs and expectations has the incident communication template both customer calls followed.
7. Carry less. Data minimization for transfers and recognizing personal data explain how to shrink the next incident before it happens.
8. Isolate customers on the server. Per-account folders held the blast radius to one recipient. Least privilege in practice is where that design comes from. The article on per-partner folder structures lays it out.

Check Your Estate

Two lists: one for the routing you run today, one for the hour after the next misdelivery. Print the second and keep it with the on-call notes. I have watched colleagues call that gloomy right up until the day they borrow it.

ROUTING CHECKLIST  (for every table, script, or job that decides where a file goes)
[ ] The routing source is in version control with a change history someone can diff
[ ] A routing change needs a second person's approval before it can take effect
[ ] Destinations are derived from a customer or partner code, not typed as free text
[ ] A validator enforces "destination belongs to this row's customer" and runs before every delivery
[ ] A new route carries a synthetic file first; real data waits for the recipient's confirmation
[ ] Similar customer codes are known, listed, and called out in the change template
[ ] You can answer "who downloaded this file, from where, when?" from the server log in a minute
[ ] Every export's personal-data columns have been questioned in the last year

MISDELIVERY RUNBOOK  (the first hour)
1. Stop the route: disable the job or the row so nothing further is delivered
2. Capture: export the delivery log and the server's download/delete log for the file; save with timestamps
3. Scope: which account(s) downloaded it, from which address, how many times; is a copy still on the server?
4. Retrieve or confirm: remove any copy still on the server; ask the recipient to locate, not open, and hold every copy
5. Classify: does the file contain personal or confidential data? If yes, notify the privacy owner NOW
6. Escalate: account manager for both parties; the disclosure decision belongs to them and to privacy/legal
7. Re-deliver correctly only after steps 1-6; keep the corrected route under review
8. Get written deletion confirmation from the recipient, with their logs; file it with the incident record

The Version to Tell a Colleague

A new weekly export was added to a routing table by copying the row above it and editing two of three columns. The destination stayed pointed at the neighbor's folder, and the neighbor's automation collected the file twelve minutes later. The customer whose file never arrived noticed; the customer who received it did not. The first instinct, fix the row, re-send, say nothing, was stopped by one question: which account downloaded it? The server log answered precisely. The recipient confirmed deletion in writing. The decision about disclosure went to the privacy officer and the customer who owned the data. The fix was to make routing reviewed configuration: versioned, approved, derived from customer codes, validated for meaning, and canaried before real data flows. The question is now the first line of the runbook.

The other story in this series where the log settled who touched a file is the leaked credential. The freshness check that made Altmore the first to notice is the subject of the file that never arrived. And the method behind all of these is in running a blameless postmortem.

Frequently Asked Questions