Migrating Folder Structures and In-Flight Files

The folder layer is the part of a platform swap everyone volunteers for. Copy the tree, copy the files, done: the one task on the plan that looks like an afternoon. It is simple until you remember that partners are writing into those folders while you copy them. A file half-uploaded at cutover exists on neither server in any usable form. And a dozen partner scripts depend on behaviors nobody ever wrote down. Those include the rename after every upload, the zero-byte marker that starts a downstream job, and the exact shape of a listing.

The folder layer is the directory tree, the rights attached to it, the files inside it, and the way it behaves when a partner uses it. This article moves all four: recreating the taxonomy exactly, translating access rules, and seeding and topping up contents. It also covers freezing live inboxes for a complete final delta, handling files caught mid-transfer, and preserving partner conventions. It is part of our Platform Migration Mechanics series. The accounts that own these folders were covered in migrating accounts and permissions. The bulk-copy tooling is in our seed-and-delta cutover article, which this one leans on.

Three Parts, Plus Behavior

The folder layer has three separable parts, and each moves differently. The taxonomy is the tree itself, every directory name and its position. It is recreated rather than copied, because the physical root and the jail mechanism differ between platforms. The access rules are the rights attached to each folder. They translate, using the neutral vocabulary from the accounts article. The contents are the files. They copy, in bulk, early, then repeatedly, until a final pass under a freeze catches the last arrivals. Wrapped around all three is behavior, what the folders do when a partner uploads, renames, lists, or deletes. Behavior is stored nowhere. It is verified by testing the new platform the way partners will use it.

Recreating the Taxonomy Exactly

"Exactly" is the operative word. A partner script that uploads to /inbox/daily/ will fail on a new platform that offers /inbox/Daily/, or /Inbox/daily/. It also fails with /inbox/ if the daily subfolder is missing because it happened to be empty on the day you copied. Start by dumping every directory on the old server, including empty ones, relative to each partner's root.

# Old server, Windows: every directory under the transfer root, one per line
cd /d D:\transfers
dir /s /b /ad > C:\migration\tree-old.txt

# Old server, Unix-like: same list
cd /srv/sftp && find . -type d | sort > /root/migration/tree-old.txt

# New server: recreate the list under the new root (Windows shape)
for /f "usebackq delims=" %d in ("C:\migration\tree-old.txt") do mkdir "E:\transfers\%d" 2>nul

# New server: recreate (Unix shape)
cd /srv/sftp && xargs -d '\n' mkdir -p < /root/migration/tree-old.txt

# Afterward, dump the new tree the same way and diff the two lists: it must be empty

Dump-and-diff is the whole method. Four details keep it honest.

• Empty folders count. Bulk-copy tools skip them by default in some modes. A missing empty archive/ folder breaks the partner script that moves files into it.
• Case must match letter for letter, even on a case-insensitive file system. The partner's client may be case-sensitive and the listing it receives must look identical.
• Links, junctions, and virtual folders are recreated as the new platform's native equivalent. Each is checked to stay inside the partner's jail. A shared exchange folder two partners reach through links is the classic case. The article on designing directory trees shows layouts that avoid needing links at all.
• Physical root and partner root are different things. The physical path can change freely; what the partner sees at / cannot. On platforms with per-account home directories (Sysax Multi Server sets the home on each account), the partner root is an account setting. It points at a folder in the recreated tree. Verify each partner root by logging in as the partner and listing /.

A migration is also the moment everyone wants to tidy the tree, and the moment they should not. Recreate what exists, warts included; reorganise later, on a separate ticket, towards per-partner folder structures. Two changes on one night means two suspects for every failure.

Translating Access Rules on Folders

The accounts article translated rights per account. This is the same translation seen from the folder: for each directory, who can do what. Where the old platform enforced rules through the operating system and the new one enforces them inside the server (or the reverse), the vocabulary shifts. The table shows the usual correspondences, with the trap in each row.

Two rows explain most post-migration surprises. On Unix-like systems, rename and delete are rights on the directory, not the file. So a partner who can write to a folder can rename and delete anything in it, including files you placed there for download. Under Windows or server-level flags, those are separate grants, so the same partner may suddenly be unable to rename their own upload. Neither model is wrong; the translation just has to be explicit per folder. The background is in permission models explained. The inheritance of newly created subfolders differs sharply between models. It is covered in umask and inheritance gotchas.

Seed, Delta, Verify

The contents move by the seed-and-delta pattern: one large seed copy days or weeks ahead, then repeated delta passes that copy only what changed. That makes the final pass on cutover night small enough to finish inside the freeze window. The tooling and its flags are in seed-and-delta cutover. The migration-specific settings are these.

• Preserve timestamps. Partner scripts that fetch "everything newer than my last run" will re-download the entire archive if every file carries tonight's date.
• Exclude temporary-name files — *.tmp, *.part, *.filepart, whatever your partners' clients use — from every pass. A temporary file copied to the new server becomes a permanent orphan there. One copied without its rename becomes a partial file that looks complete.
• Run the copy under an identity that can read everything on the old side and leaves ownership sensible on the new. Files owned by a migration account are a permission problem waiting for the first partner delete.
• Verify each delta with counts and sizes. Verify the final one with hashes on a sample or, for regulated flows, on everything. This is the method in verifying nothing left behind.

Live Inboxes: The Write Freeze and the Final Delta

Static folders (archives, outboxes you populate yourself) are finished after the seed and a delta or two. Live inboxes, where partners upload on their own schedule, are not finished until nobody can write to the old server any more. That is the purpose of the write freeze: a short window in which the old server accepts no new uploads. A final delta can then capture a complete, stable set of files. The new server can take over from exactly that state.

The freeze is imposed on the old server, and the mechanism matters. Stopping the service is the crude option; it also stops you verifying anything on the old side and makes rollback slower. Better mechanisms leave the service running and refuse writes. Partner accounts can be set read-only, or disabled while administrative ones stay active. Or inbound partner connections can be blocked at the firewall while administrative addresses stay allowed. Whichever you use, it must be reversible in one step, because it is also the first step of rollback. The night's sequence, in clock time:

FREEZE WINDOW -- live inboxes, sftp-old -> sftp-new

Mar 14 01:30  last routine delta pass completes (contents within minutes of current)
Mar 14 02:00  FREEZE: partner accounts on sftp-old set read-only / disabled
              admin access stays open; service keeps running
Mar 14 02:00  drain: list active sessions on sftp-old; wait for transfers in progress
              to finish -- typically a few minutes; do not kill them
Mar 14 02:12  drain complete: zero partner sessions; list temp-name files (see below)
Mar 14 02:13  snapshot: file list with sizes + mtimes of every inbox -> inbox-final.txt
Mar 14 02:15  FINAL DELTA: copy changes since 01:30, temp names excluded
Mar 14 02:28  final delta complete; verify counts/sizes against inbox-final.txt;
              hash the files that arrived since 01:30
Mar 14 02:35  new server opened to partners (address switch -- see cutover runbook)
              sftp-old stays FROZEN, not stopped, for rollback and late-arrival sweeps

Partners whose jobs run during the freeze get a refused login or a permission error. Almost all unattended jobs retry; every fifteen minutes is typical. By the time they retry, the address points at the new server and the upload lands there. That is the behavior you want. That is why the freeze should be as short as the final delta allows, not padded for comfort. Half an hour costs partners one or two retries. Four hours starts triggering their alerts, then their on-call, then yours. Partners without retry logic are known from the inventory and get a morning call.

Files caught mid-transfer

The drain step exists because a partner upload that began at 01:58 may still be running at 02:00. A file cannot be frozen halfway. While a session is alive, wait: watch the session list until every partner session has ended on its own. Then look for the partials the drain could not resolve. These are uploads whose session died before completion or whose client never sent the rename.

• Temporary names: anything matching the partners' temp-name patterns, still present after the drain.
• Unstable sizes: a file whose size differs between two listings a minute apart is still being written by something. This is the settle-check technique from size stability and settle checks.
• Zero-length non-marker files: a data file with no bytes is usually a failed start, not a real delivery.

These files are not migrated as final. They are listed by partner and folder and left in place on the frozen old server. Each partner gets a morning report: "these uploads did not complete before the migration; please re-send." Most partner automation will have retried them against the new server before anyone reads the message. The one thing never to do is rename a temporary file to its final name yourself. Not to be helpful, not because it looks complete, not because it is three in the morning. You do not know whether the bytes are all there. The whole point of the convention is that only the sender knows. Their origins are in why partial files happen.

Preserving the Conventions Partners Rely On

Behavior is where the two platforms differ most invisibly. Nothing in an export describes it; it is discovered by a partner script failing. Test each of these on the new platform, as a partner account, before cutover.

• Upload-then-rename. The partner uploads manifest.csv.tmp and renames to manifest.csv. The new platform must allow the rename (the rights table above). It must not hide or reject files with the temporary suffix. It must not treat the rename of an existing target name as an error if the old one did not. The convention itself is in temp names and atomic renames.
• Marker files. A zero-byte manifest.done or a small control file uploaded after the data. Some platforms reject zero-byte uploads or quarantine unknown extensions by default. Both silently break the downstream trigger. See marker and control files for the patterns partners use.
• Directory listing format. SFTP listings are structured and portable. FTP and FTPS listings are text, and old partner scripts parse that text. A Unix-style listing (-rw-r--r-- 1 ...) and a DOS-style listing (03-14-yy 02:10AM 12345 file) are parsed by different code. If the old server produced one style and the new one produces the other, every listing-parsing script on the FTP side breaks. That happens on its first run. Most servers can be set to either; match the old.
• Timestamps and timezone. Listings and the modification-time commands report either UTC or server-local time depending on platform. A partner script that filters by time will be off by the difference until the new platform is set to match.
• Resume and append. Partners who resume interrupted large uploads need the append right and a platform that supports the protocol's resume operation. Check both, with a deliberately interrupted test upload.
• Character sets in names. File names with accented or non-Latin characters travel as bytes, and the two platforms may interpret them under different encodings. A name that lists correctly on the old server and appears mangled on the new one is an encoding setting. Background in safe characters across platforms.
• Hidden files. Names beginning with a dot are hidden by convention on Unix-like systems and visible on Windows. A partner's dot-prefixed control file may or may not appear in the listing afterward, and their script may depend on either.

Acme learned the listing-format one from three partners at once. Their SFTP partners came through cutover without a murmur. Their three remaining FTP partners' scripts asked for a listing and received a perfectly valid one in the other style. They parsed it into nothing and reported nothing to fetch, which is not an error and so alarmed nobody. Two mornings later a partner's operations desk rang to ask why the confirmations had stopped. One server setting put the listing back; the two days of missed files took rather longer.

The test for all of them is the same: a real upload by a real partner, following their real convention, during their test window. The result is observed on the new server and in the downstream job that consumes it. Partner coordination in migrations explains how to get partners to actually use those windows. The article on partner test windows covers running one so it proves something.

The Folder Verification Pass

Before cutover night, and again after the final delta, run one pass that proves the folder layer as a whole. Three parts, each producing a file for the evidence pack.

# 1. Taxonomy: the two directory lists must be identical
diff tree-old.txt tree-new.txt              # expect: no output

# 2. Contents: counts and sizes per partner root, then hashes on the final delta
#    (old side, frozen)      find /srv/sftp/bayside -type f | wc -l ; du -sb /srv/sftp/bayside
#    (new side)              same commands under the new root -- numbers must match
#    hashes of files modified since the last routine delta, both sides, diffed:
find /srv/sftp -type f -newer /root/migration/delta-0130.stamp -exec sha256sum {} + | sort -k2 > hashes-old.txt
#    ...same on new, then:   diff hashes-old.txt hashes-new.txt   # expect: no output

# 3. Behavior: as each partner account, the convention probe
#    put data.csv.tmp ; rename data.csv.tmp data.csv ; put marker.done (0 bytes)
#    ls -l (compare format to old) ; confirm downstream job fired on data.csv

The taxonomy diff and the hash diff must be empty. Counts and sizes must match to the byte on frozen folders. The behavior probe must reproduce, for every partner convention, exactly what the old server did. Anything else is a finding, and findings in the folder layer are fixed before the address switches, never after.

Wrapping Up: Tree, Rules, Bytes, and Behavior

The folder layer moves in four motions. The tree is recreated from a dump and proven by a diff. The rules translate folder by folder, rename and inheritance watched closely. The bytes move by seed and delta until a write freeze makes the old inboxes stable enough for a final delta. That final delta is verified by count, size, and hash. And the behavior is proven by partners uploading the way they always have, before the address moves. Files caught mid-transfer are left behind and re-sent by their owners, never renamed into place by you. It is still an afternoon's work. It is just spread over three weeks.

The freeze and the final delta are the opening steps of the cutover night runbook, which puts them on the clock. The old server stays frozen rather than off until post-migration cleanup has swept it for late arrivals and the decommission evidence is complete.

Frequently Asked Questions