lftp: The Power Tool for Scripted Transfers

Every command-line transfer client makes a trade. The sftp client is everywhere and simple, but it retries nothing and mirrors nothing. curl is a superb single-shot tool, but it thinks in one URL at a time. When the job is "keep this directory tree synchronized over a flaky line, retry until it works, and do four transfers at once," administrators reach for lftp. This client behaves less like a command and more like a small, scriptable transfer engine.

This article is a working tour of the features that earn lftp its reputation. It covers the retry-and-reconnect machinery you get without writing a single loop, and the mirror command that synchronizes whole trees in both directions. It covers queues and parallel transfers, and the script mode that turns all of it into scheduled jobs. It is equally honest about the costs. You must get credential handling right. Default persistence can hang a scheduled job. And there is the plain fact that lftp is not installed everywhere. This article is part of our command-line client mastery series.

What lftp Is, and Where It Actually Runs

lftp is a command-line client that speaks FTP, FTPS, SFTP, and HTTP/HTTPS through one consistent interface. Start it and you get a shell-like prompt with its own command set, its own job control, and its own settings system. Feed it a script and it becomes an unattended transfer engine. That shell-like design is the point. lftp sessions can run several transfers at once, queue work, and keep retrying in the background. A one-command-one-connection client cannot express those things at all.

Now the portability honesty, because it decides whether lftp can be your standard. On Linux, lftp is a standard distribution package — where it is not already installed, it is one command away (apt install lftp, dnf install lftp, or your platform's equivalent). BSD systems and macOS get it easily from their package systems. Windows is the exception: there is no lftp in a standard Windows install, and no official native build. On Windows it is realistically available only inside a Unix-like environment such as the Windows Subsystem for Linux. If your fleet is Windows-first, plan around that. Script the Windows side with the tools that are native there (see curl and the PowerShell transfer automation series). Let lftp shine on the Linux side.

Getting Connected Without Leaking Credentials

The lftp examples you find in old forum posts almost all share one bad habit:

lftp -u reports,S3cretPass ftp.partner.example      # do not do this

The problem is not lftp — it is that command-line arguments are public information on a shared system. Any user who runs ps while the job is active can read the password right out of the process list:

$ ps -ef | grep lftp
svc-xfer  4182  ... lftp -u reports,S3cretPass ftp.partner.example

The same applies to open -u user,pass inside a -c command string, and the password lands in shell history besides. Three clean alternatives, in order of preference:

• SSH keys, by using SFTP. Point lftp at sftp://transfer@host and there is no password at all. lftp runs your system's ssh program underneath (its sftp:connect-program setting, normally ssh -a -x). So the keys, ~/.ssh/config entries, and known_hosts trust you set up for the sftp client work identically here. This is the best answer whenever the server speaks SFTP.
• ~/.netrc for FTP and FTPS logins. lftp reads the classic netrc file automatically. It has one machine entry per host with login and password lines. The file is locked to chmod 600 so only its owner can read it. The secret still exists on disk, but it is out of the process list, out of history, and out of your scripts.
• Bookmarks for convenience. bookmark add partner saves the current connection so open partner reconnects later. By default lftp refuses to store passwords in bookmarks (the bmk:save-passwords setting is off, and should stay off). So treat bookmarks as a naming convenience layered over netrc or keys, not as a credential store.

# ~/.netrc — must be chmod 600
machine ftp.partner.example
login reports
password S3cretPass

One more FTPS note while we are here: for FTPS servers, tell lftp to insist on encryption rather than merely prefer it. set ftp:ssl-force yes refuses to log in over cleartext. set ftp:ssl-protect-data yes encrypts the file data as well as the login. Client-side FTPS quirks in general are covered in FTPS client compatibility.

The Settings Engine: Retries You Do Not Have to Write

lftp's defining feature is what happens when the network misbehaves: instead of failing, it reconnects and picks up where it left off. A dropped control connection mid-mirror, a timeout on file forty of ninety — lftp retries on its own, with a growing pause between attempts. Every knob lives in the set command:

set net:timeout 30                      # seconds before an idle operation is declared stuck
set net:max-retries 3                   # attempts per operation before giving up
set net:reconnect-interval-base 10      # first reconnect wait, in seconds
set net:reconnect-interval-multiplier 2 # each wait doubles: 10, 20, 40...

That multiplier is exponential backoff — retry pauses that grow geometrically so a struggling server is not hammered. lftp gives it to you as configuration rather than code you must write and test. (Why backoff matters, and how to choose budgets, is the subject of our retry and error handling series.)

The honest flip side: out of the box, lftp is famously persistent — left alone it will keep reconnecting and retrying far longer than any scheduled job should live. Interactively that tenacity is delightful; at 2 a.m. it means tonight's job is still "running" when tomorrow's starts. For unattended work, always cap the machinery explicitly as above. Add set cmd:fail-exit yes in script files so the run stops — with a nonzero exit code — at the first command that conclusively fails. Settings can live per-job in the script, or globally in ~/.lftprc, which the examples below assume you have left alone so every job states its own policy.

mirror: Directory Synchronization in One Command

The command that justifies the install. mirror compares a remote directory tree with a local one and transfers whatever it takes to make the target match — new files, changed files, subdirectories, the lot. Downward by default, upward with -R:

mirror --only-newer --verbose /outgoing/reports /srv/inbox/reports
mirror -R --only-newer /var/www/site /htdocs/site       # upload direction

The options that do the real work:

• --only-newer — skip files whose remote copy is not newer than the local one, judged by size and timestamp. This is what turns a full copy into an incremental one. Honesty note: over plain FTP the timestamps come from the server's listing and can be minutes or a timezone off; SFTP reports them reliably.
• --delete — remove files from the target that no longer exist on the source, making a true mirror. This flag deletes data by design; see below before using it.
• --dry-run — print what would be transferred or deleted without doing any of it.
• --parallel=N — transfer up to N files simultaneously, which can shorten a mirror of many small files dramatically.
• --exclude and --include — filter by regular expression, while --exclude-glob filters by shell-style pattern. Reaching for --exclude *.tmp and getting regex semantics is a classic surprise; --exclude-glob *.tmp is what most people mean.
• -c (continue) — resume a previously interrupted mirror, reusing partial transfers where the server allows.

With --verbose, mirror narrates its decisions, and the log becomes self-explanatory:

Transferring file `daily_a.csv'
Transferring file `daily_b.csv'
Removing old file `stale_export.csv'
Total: 1 directory, 2 files, 0 symlinks
New: 2 files, 0 symlinks
8462 bytes transferred

That closing summary is worth grepping for in job logs. A mirror might suddenly report zero new files for a flow that should produce them nightly. That is a freshness problem your monitoring should notice, even though the job "succeeded."

Pair --delete with --dry-run on the first run of any new mirror job, every time. A mirror pointed at the wrong directory with --delete active will faithfully erase everything that "should not" be there, and it will do it at full speed. If both ends of your sync are machines you control over SSH, also weigh rsync, which transfers only changed portions of files. The tradeoffs are in rsync mirroring patterns. lftp's mirror wins when one end is an FTP, FTPS, or SFTP server you do not control.

The supporting cast: cls, mget, and mrm

Three smaller commands round out daily lftp work. cls is lftp's own listing command, formatted client-side with ls-style options. cls -1 -t prints bare names sorted newest-first. That makes "fetch the latest export" a solvable scripting problem instead of a parsing exercise. mget and mput transfer sets of files by wildcard, and mrm deletes by wildcard. Note the m prefix, because plain rm takes literal names. It will cheerfully report success at removing nothing when handed an unexpanded pattern. Datestamped names pay off here. When archives are named backup_YYYYMMDD.tar.gz, a computed prefix plus mrm prunes a whole month in one line. The one-liner cookbook builds on that pattern.

pget and Parallelism, Honestly

lftp offers two very different kinds of "parallel," and conflating them causes disappointment. mirror --parallel=N moves N different files at once — safe, broadly compatible, and usually the speedup you actually want. pget -n N is the exotic one: it downloads one file in N simultaneous segments, each fetching a different byte range, and stitches the result together.

pget -n 4 /outgoing/archive_YYYYMMDD.tar.gz -o /srv/inbox/archive_YYYYMMDD.tar.gz

When does segmenting help? Mainly when something limits each individual connection — a per-connection throttle on the server or path, or a long high-latency route where one TCP stream cannot fill the pipe. On a healthy LAN or an unthrottled nearby server, pget adds complexity for little or no gain. It also requires the server to support resuming from an offset. And — bluntly — four connections for one file looks like abuse to some server operators and trips their connection limits. Use it deliberately for the occasional huge file over a long link (there is a --use-pget-n=N option to apply the same trick inside a mirror), not as a default. The networking behind why parallel streams sometimes help is explained in parallel streams.

Queues and Background Jobs

Because lftp is a shell, it has job control. Append & to run a transfer in the background, jobs lists what is running, and wait blocks until work finishes. The queue command turns this into an ordered work list: queued commands run one at a time, in sequence, while you keep typing — or while your script keeps adding work.

lftp sftp://transfer@backup.example.com
lftp transfer@backup.example.com:~> queue mirror -R /data/site_a site_a
lftp transfer@backup.example.com:~> queue mirror -R /data/site_b site_b
lftp transfer@backup.example.com:~> queue put /data/summary.csv
lftp transfer@backup.example.com:~> jobs
[0] queue (sftp://transfer@backup.example.com)
    Now executing: [1] mirror -R /data/site_a site_a
    Commands queued:
     1. mirror -R /data/site_b site_b
     2. put /data/summary.csv
lftp transfer@backup.example.com:~> wait all

In a script, ending with wait all is what makes the exit honest. It holds the session open until every queued job completes. So the script's exit code reflects the whole batch rather than just the enqueueing. The queue runs one entry at a time unless you raise cmd:queue-parallel, which is exactly the right default for jobs that must not compete with each other for bandwidth.

Script Mode for Real Jobs

lftp gives you three ways to hand it commands, and the distinction matters:

• -c "commands" — run the commands and exit, with a nonzero exit code if a command fails. The right form for one-liners and cron entries.
• -e "commands" — run the commands and stay at the prompt. For interactive sessions that start with setup; never for automation, because the job would hang waiting for input.
• -f scriptfile — execute a file of commands and exit. The right form for anything longer than one line, because the script can be reviewed and version-controlled.

A production-shaped script file, pulling nightly reports with capped retries:

# nightly_pull.lftp — run with: lftp -f /etc/transfer/nightly_pull.lftp
set cmd:fail-exit yes
set net:timeout 30
set net:max-retries 3
set net:reconnect-interval-base 10
set net:reconnect-interval-multiplier 2
open sftp://transfer@sftp.partner.example
mirror --only-newer --verbose /outgoing/reports /srv/inbox/reports
bye

And the wrapper the scheduler actually calls, following the same log-and-propagate pattern as any transfer job:

#!/bin/sh
LOG=/var/log/transfers/nightly_pull.log
if ! lftp -f /etc/transfer/nightly_pull.lftp >> "$LOG" 2>&1; then
    echo "nightly_pull FAILED" >> "$LOG"
    exit 1
fi

When a job misbehaves and the log is not enough, lftp can show you the protocol itself. Start it with -d, or type debug in a session. Every command and server reply scrolls past — the FTP dialogue or the SFTP exchange, timestamped and unabridged. It is the same troubleshooting instinct as reading a mail server conversation: one look at the raw exchange usually beats an hour of guessing. Turn it off for production runs; debug output is enormous.

Everything around this — cron's environment quirks, locking so tonight's run cannot overlap yesterday's stuck one, log rotation — is the province of our bash and cron automation series. The deeper unattended-input patterns are in batch modes, heredocs, and command files.

When lftp Is the Wrong Tool

A power tool is not a default tool. Three honest situations where you should not use lftp:

• The job is trivial. For one file, one direction, once a night, the plain sftp client or a curl one-liner is easier for the next administrator to read. It needs no extra package on a minimal host.
• The estate is Windows-native. If your transfer jobs run from Windows servers, building them on a tool that requires a Linux compatibility layer creates an odd dependency. Use what is native there instead.
• The flow has outgrown scripts entirely. When a mirror job accretes schedules, per-file accounting, encryption steps, and "email the team when it breaks," you are maintaining a small application. That is the point of a dedicated automation product. Sysax FTP Automation covers the same ground: scheduled SFTP, FTPS, and FTP transfers built through a wizard. It also covers folder monitoring, OpenPGP encryption, and email notifications on failure. The operational features are configuration instead of code.

One more practical suggestion: rehearse lftp against a server you own before aiming it at a partner. A trial of Sysax Multi Server (free from the download page) gives you a Windows SFTP/FTPS/FTP server to practice mirrors, deletes, and retry settings against. There, a wrong --delete costs you a test folder and nothing more.

The Takeaway

lftp earns its place in the toolbox the moment a transfer job needs resilience or bulk. It offers retries with exponential backoff as settings rather than code, and mirror for whole trees in either direction. It offers queues for ordered work and a script mode with honest exit codes. Its costs are equally clear. Credentials need the netrc-or-keys discipline, its patience must be capped for scheduled work, and it simply is not present on Windows systems.

Where it fits among its peers: reach for sftp when the job is simple and SSH-based. Reach for curl when you want one-shot transfers with fine-grained flags. Reach for lftp when the words "mirror," "retry," or "parallel" appear in the requirements. The one-liner cookbook at the end of this series puts all three side by side on the same jobs.

Frequently Asked Questions