Syncing over WAN Links: Bandwidth-Efficient rsync Patterns

Syncing between two machines on a LAN is a solved problem — almost anything works. The character test comes when the two machines are separated by a WAN link: the branch office's modest broadband, a site-to-site VPN, a metered cloud egress path. Now the link is slow, shared with people trying to work, occasionally flaky, and possibly billed by the gigabyte. A sync job that was invisible on the LAN becomes the thing users complain about at 2 p.m.

rsync is the right tool for this terrain — delta transfer was invented for exactly it — but the defaults alone don't finish the job. This article covers the four disciplines that turn rsync into a good WAN citizen: compressing what's worth compressing, surviving interruptions without starting over, throttling so the link stays usable, and scheduling around the hours when humans need the bandwidth. Then, because claims deserve evidence, we end with how to measure what you actually saved. It is part of our rsync & Delta Transfer series.

Why rsync Is Already WAN-Native

Two built-in behaviors do most of the work before you add a single flag. The quick check skips every file whose size and modification time already match on both sides — in a typical tree, that is nearly everything. The delta algorithm then handles the files that did change, sending only the changed blocks rather than whole files; the full mechanism is in how the rsync algorithm works. Together they mean a recurring WAN sync moves a small fraction of the data a naive copy would.

Both savings share a precondition worth restating: they require an earlier copy at the destination. The first run of any job is a full transfer at full price. For a large initial dataset over a thin link, do the arithmetic before you start — 500 GB through 20 Mbit/s is roughly two and a half days of a saturated link. The time-honored answer is seeding: carry the first copy on a disk, ship it, or load it while the machine is still in the same building, and let the WAN handle only deltas from then on. Unglamorous, effective, and entirely respectable engineering.

Compression: -z and Its Dials

The -z flag compresses data in flight — the sender compresses the literal data it must transmit, the receiver decompresses on arrival, and files land unchanged. On a slow link with compressible content — text, logs, source code, CSV exports, database dumps — it routinely halves or better the bytes on the wire. Three dials matter:

• How hard to squeeze. --compress-level trades CPU for bandwidth. On a WAN the network is almost always the bottleneck, so higher effort usually wins; back off only if a slow CPU on either end becomes the new choke point. Newer rsync builds can also choose which compression algorithm to use (--compress-choice), including much faster modern compressors — when both ends support one, it is nearly free bandwidth.
• What not to squeeze. Compressing already-compressed data wastes CPU for zero gain. rsync knows this: it ships a default skip-compress list of extensions (gz, zip, jpg, mp4, and friends) that are sent uncompressed even under -z. Extend it with --skip-compress if your tree is full of some compressed format the default list doesn't name.
• How many layers are squeezing. If rsync runs over SSH and SSH's own Compression option is enabled in the SSH config, you are compressing twice — pure CPU waste. Pick one layer; -z is the better choice because it knows the skip list and compresses only literal data.

Keep expectations honest: -z shrinks the new bytes being sent; it does not create block matches where none exist. A tree of photos and zip archives gains almost nothing from -z — and, as the algorithm article explains, gains little from delta either once those files change. Compression is a text-and-data play.

Surviving Interruptions: Partial Transfers and Resume

WAN links drop. A VPN renegotiates, a line flaps, a laptop sleeps. What happens to the 9 GB file that was 80% transferred? By default, rsync deletes the incomplete temporary file, and the next run starts that file from zero. Over a link that drops nightly, a large file can fail to arrive forever — always 80% done, never finished.

The fix is the partial-transfer family:

# Keep interrupted files so the next run resumes instead of restarting
rsync -avz --partial-dir=.rsync-partial --timeout=120 \
      /data/exports/ sync@branch:/srv/exports/

# -P is the interactive shorthand: --partial plus --progress
rsync -avzP /data/exports/ sync@branch:/srv/exports/

--partial keeps the incomplete file; --partial-dir=.rsync-partial is the better version, parking it in a hidden subdirectory so nothing downstream mistakes a fragment for a finished file. On the next run, rsync finds the fragment and — this is the elegant part — uses it as the basis for delta transfer: the 80% already delivered becomes matched blocks, and only the missing tail plus changes cross the wire. Resume falls out of the algorithm for free.

Two companions round out the resilience story. --timeout=120 makes rsync give up if the connection stalls silently for two minutes, instead of hanging until someone notices the job has been "running" for a day. And since a timed-out run exits nonzero, a small retry loop turns flaky into eventually-reliable:

# Retry up to 5 times; only vanished-file warnings (24) count as success
n=0
until rsync -az --partial-dir=.rsync-partial --timeout=120 \
        /data/exports/ sync@branch:/srv/exports/; do
    rc=$?
    [ $rc -eq 24 ] && break          # source files vanished mid-run: benign
    n=$((n+1)); [ $n -ge 5 ] && exit $rc
    sleep 60
done

Exit code 24 deserves its footnote: it means some source files disappeared between listing and transfer — routine when syncing a live tree — and most jobs should treat it as a warning, not a failure. The full decoding habit lives in dry runs and verification.

One specialist tool with a warning label: --append (and its safer sibling --append-verify) resumes by assuming files only ever grow, sending just the tail beyond the destination's current length. For genuine append-only data — log shipping, capture files — it is efficient; for anything that can be edited in the middle, it will happily construct a corrupt file. Reach for it only when "append-only" is a fact about the producer, not a hope.

Throttling: Protecting the Link You Share

An unthrottled rsync will cheerfully consume every bit of bandwidth the path offers. On a dedicated line, fine. On the branch office's 50 Mbit/s connection at 10 a.m., your sync job just became everyone's video-call problem. The polite flag:

# Cap the transfer at roughly 5 MB/s (suffixes: K, M, G)
rsync -avz --bwlimit=5m /data/exports/ sync@branch:/srv/exports/

Details that keep --bwlimit honest: it caps the average rate of the whole session (both directions of the data flow, though in practice the payload direction dominates), enforced by pacing writes in bursts — short spikes above the cap are normal. It throttles this rsync only; three jobs at --bwlimit=5m each still add up to 15. And remember which direction is scarce: consumer and branch links are usually asymmetric, with upload a fraction of download, so a push from the branch bites harder than the same bytes pulled from headquarters.

A pattern that satisfies both the users and the sync deadline is the two-speed schedule: a strict cap during working hours, a generous or absent cap overnight. Two cron entries, one job script parameterized by limit — no cleverness required. (Throttling at the network layer with QoS is the heavier alternative when many applications share the link; rsync's flag is the right size when the sync job is the only elephant.)

Scheduling Around Business Hours

Throttling's twin discipline is simply not competing when it matters. The mechanics are ordinary cron plus the overlap lock from mirroring patterns (flock -n, so a slow night's run makes the next cycle skip rather than stack). The WAN-specific wrinkles:

• Mind whose night it is. Sites in different time zones don't share an "overnight"; schedule in the link's quiet window — the overlap of both sites' off-hours — not your own.
• Stagger multiple sites. Ten branches all starting at the same hour arrive at headquarters together and divide the head-office bandwidth ten ways. Offset start times, or let a lock on the receiving side serialize them.
• Bound the window. A job that starts at midnight should not still be saturating the link at 9 a.m. after an unusually large day. Wrapping the run in the standard timeout utility — timeout 6h rsync … — plus --partial-dir means it stops at the window's edge and resumes cleanly the next night.

Latency and the Many-Small-Files Problem

Bandwidth is only half a WAN link's personality; the other half is latency, and it punishes a different workload. Moving one big file is bandwidth-bound. Moving half a million tiny files is dominated by per-file bookkeeping, and no compression flag fixes it.

rsync is better positioned here than per-file tools: it reuses one connection for everything, builds and transfers its file list incrementally while transfers proceed, and pipelines work so it is not waiting a round trip per file. But physics still charges for metadata: enormous file counts cost memory, list-building time on both ends, and a long "examining files" phase before much data flows. Practical mitigations, in order of dignity: split monster trees into per-subdirectory jobs so the working set stays sane; prune with excludes (build caches and temp droppings rarely deserve WAN bandwidth — see the filter rules in flags that matter); and only as a last resort consolidate hordes of tiny files into container archives — accepting that, as the algorithm article explains, compressed archives largely forfeit delta transfer on their contents.

And when a link is both long and lossy — intercontinental paths, satellite hops — even a perfectly tuned TCP session may not fill the pipe, and rsync inherits that ceiling like every TCP tool. That is a different problem class with different remedies, mapped in our UDP-accelerated transfer series. Diagnose before prescribing: delta transfer fixes "too many bytes," not "too far, too lossy."

A Worked Example: The Branch Nightly in Numbers

To make the flags concrete, run the arithmetic on a typical case. A branch office holds a 120 GB working tree; on an average day, users touch about 900 files totaling 3 GB, and within those files perhaps a third of the bytes actually change. The link is 20 Mbit/s upstream — call it 2.2 MB/s of realistic throughput — and shared with the whole office.

• Naive full copy: 120 GB is roughly 15 hours of a fully saturated link. Not nightly-job material; not even weekend-job material once you add throttling.
• Quick check only (whole changed files): 3 GB is about 23 minutes at full speed, or an hour and a quarter under a polite 8 Mbit/s cap. Workable.
• Quick check + delta: if a third of the changed bytes really differ, about 1 GB crosses — under 10 minutes unthrottled, half an hour throttled.
• Plus -z on compressible data: office documents and exports often compress 2:1 or better; the wire cost drops toward 500 MB. The job now fits in any window you like, caps included.

The pattern generalizes: each mechanism buys roughly an order of magnitude, and the combination turns "impossible over this link" into "done before the first coffee." Your numbers will differ — which is exactly why the next section's --stats habit matters more than anyone's example arithmetic, including this one.

Measuring What You Actually Saved

Every claim above is measurable, and the meter is built in. Add --stats to a run and read three lines:

Total file size: 38,654,705,664 bytes   <- the data the job covers
Literal data: 214,748,364 bytes         <- changed bytes that had to be sent
Matched data: 38,439,957,300 bytes      <- reused from the destination's copies
Total bytes sent: 96,469,180            <- on-the-wire cost (after compression)
speedup is 400.71                       <- covered data / bytes actually moved

The ratio of total bytes sent to total file size is your delta-plus-compression dividend — here, a 38 GB tree serviced for under 100 MB of WAN traffic. Log these numbers every run (append --stats output to the job log) and glance at the trend: a mirror whose bytes-sent suddenly grows tenfold is telling you something upstream changed — someone started dropping nightly zip bundles into the tree, a database began rewriting files wholesale, an exclude rule stopped matching. The stats line is the cheapest monitoring you will ever deploy.

Putting It Together

The WAN-ready job, assembled from the parts above:

# crontab — nightly, throttled, resumable, bounded, non-overlapping
30 1 * * *  flock -n /run/lock/branch-sync.lock \
            timeout 6h rsync -az --partial-dir=.rsync-partial \
            --timeout=120 --bwlimit=8m --stats \
            /data/exports/ sync@branch:/srv/exports/ \
            >> /var/log/branch-sync.log 2>&1

Cron, flock, timeout, and a log file are honest tooling and enough for many sites. The setup outgrows shell scripting when requirements become organizational: retry policies someone can audit, alerts when a branch misses two nights, credentials managed outside a script, a console a colleague can read at 9 a.m. That is the job of scheduling software — on Windows, Sysax FTP Automation is built for precisely this shape of scheduled, monitored, retried transfer work over SFTP, FTPS, and FTP, trading script flexibility for operability. The strategy transfers wholesale: send less, send politely, resume instead of restart, and measure.

The Version to Tell a Colleague

rsync earns its WAN reputation twice over — the quick check skips the unchanged, delta transfer trims the changed — but the polished job adds four habits. Compress in flight with -z and let the skip-list leave compressed formats alone. Keep interruptions cheap with --partial-dir and a timeout, and let resume ride the delta algorithm. Throttle with --bwlimit during human hours, generously at night. Schedule in the link's quiet window, locked against overlap and bounded by timeout. Then let --stats prove the savings, every night, in two lines of log.

Deeper on each foundation: the delta algorithm itself, the mirroring patterns these jobs implement, and the transport choice underneath them all.

Frequently Asked Questions