curl for File Transfers: Beyond Web Requests

Ask most administrators what curl is for and they will say "testing web APIs." Fair — but incomplete in a way that costs them. curl is a general transfer tool that has spoken FTP since its earliest days and speaks FTPS and SFTP too. It is already installed on effectively every system you manage — Linux, macOS, the BSDs, and current Windows builds alike. It brings virtues to file transfer that the classic clients never had. Those include honest exit codes, built-in retry flags, resumable transfers, and a config-file system that keeps passwords out of sight.

This article is curl as a file transfer client. You will learn the URL syntax and its one famous trap, and uploads and downloads done properly. You will learn how to require real encryption on FTPS instead of merely requesting it, and the flags that make unattended jobs resilient. You will also learn the honest boundary where curl stops being the right tool. We stay off the web here: for HTTP and HTTPS work, our companion curl and wget cookbook covers that side. This piece is part of the command-line client mastery series.

One Tool, Many Protocols — Check Which Ones You Have

curl's model is simple and strict: one invocation, one job, described by a URL. The scheme at the front of the URL — ftp://, ftps://, sftp:// — selects the protocol; everything else is flags. There is no session to hold open, no prompt to interact with: curl connects, does the transfer, and exits with a status code. That single-shot shape is a limitation for complex work and a gift for scripting, because each transfer is one auditable line with one testable result.

Before planning anything, check what your curl can actually speak, because protocol support is decided when curl is built. The first lines of curl -V list it:

$ curl -V
curl ... (x86_64-pc-linux-gnu) ...
Protocols: dict file ftp ftps http https imap imaps mqtt pop3 pop3s
           rtsp scp sftp smb smbs smtp smtps telnet tftp

On most Linux builds you will find sftp in that list. On the curl bundled with Windows you usually will not — it typically ships with ftp and ftps but without SFTP support. That single fact shapes many Windows automation decisions, so verify it on the actual machine the job will run on, not on your workstation.

URL Syntax That Trips People Up

The most common curl-FTP surprise is not a flag — it is the path. In an ftp:// URL, the path is relative to the directory you land in after login, usually the account's home directory. It is not an absolute path from the server's root. To name an absolute path, the first slash of that path must be encoded as %2F:

curl ftp://ftp.example.com/reports/daily.csv           # reports/ under the login directory
curl ftp://ftp.example.com/%2Fsrv/exports/daily.csv    # absolute path /srv/exports/daily.csv
curl ftp://ftp.example.com/reports/                    # trailing slash: directory listing
curl --list-only ftp://ftp.example.com/reports/        # names only, one per line

SFTP URLs play by the opposite rule: the path after the host is absolute, and home-relative paths are written with an explicit /~/:

curl -u deploy: sftp://sftp.example.com/srv/exports/daily.csv -O   # absolute
curl -u deploy: sftp://sftp.example.com/~/daily.csv -O             # relative to home

Commit both rules to memory and half of all "curl says the file does not exist, but I can see it in my GUI client" tickets disappear. The GUI was showing you the login directory, and your URL was pointing somewhere else.

Moving Files in Both Directions

Downloads

By default curl writes the fetched bytes to standard output, which is rarely what a file job wants. Two flags fix that: -o name saves under a name you choose, and -O saves under the remote file's own name in the current directory. Add -s to silence the progress meter in scripts and -S so real errors still print, and you have the standard scripted download:

curl -sS --netrc -o /data/inbox/daily.csv ftp://ftp.partner.example/reports/daily.csv

For big files over unreliable links, -C - resumes an interrupted transfer. curl checks how much of the local file already exists and asks the server to continue from that offset. It works for FTP downloads and uploads, and for SFTP downloads in most builds. Pair it with a size or checksum verification at the end, because a resumed file deserves proof of completeness.

Uploads

Uploads use -T (long form --upload-file): send this local file to that URL. If the URL ends with a slash, curl appends the local filename. If it names a file, that becomes the remote name. This enables the upload-then-rename pattern below. Three upload flags matter for jobs:

• --ftp-create-dirs — create missing remote directories instead of failing, on FTP and SFTP both. Essential when paths include a datestamped folder that may not exist yet.
• -a (append) — append to the remote file rather than replace it, for log-shipping flows.
• -T - — read the upload from standard input, so a pipeline can stream straight to the server with no temporary file: tar czf - /data/site | curl --netrc -T - ftp://host/backups/site_backup_YYYYMMDD.tar.gz.

curl can also send raw server commands around the transfer with -Q. A plain command runs before the transfer, and one prefixed with - runs after (prefix * to tolerate its failure). That is exactly what a safe publish needs — upload under a temporary name, then rename in one server-side step so consumers never see a half-written file:

curl -sS --netrc --ssl-reqd \
     -T daily.csv ftp://ftp.partner.example/incoming/daily.csv.part \
     -Q "-RNFR incoming/daily.csv.part" \
     -Q "-RNTO incoming/daily.csv"

The same mechanism handles small housekeeping without a second tool. -Q "-DELE incoming/previous.csv" after an upload removes yesterday's file. Over SFTP the quoted commands use that protocol's vocabulary instead (rm, rename, mkdir). One transfer plus a line or two of server-side tidying is a whole job in a single invocation.

curl can also move several files in one command through its URL globbing. -T "{daily.csv,weekly.csv}" ftp://host/incoming/ uploads both. A download URL like ftp://host/reports/part[1-9].csv fetches a numbered series. Here, -o "part#1.csv" names each saved file from the matched value. Quote the globs so your shell does not eat them. It is a convenience, not a mirror — every file still transfers unconditionally — but it saves a loop when the file set is known and small.

RNFR/RNTO are FTP's rename-from and rename-to commands. The whole family is decoded in FTP commands and reply codes. The why of temp-name publishing is the subject of our partial-file safety series.

FTPS: Require the Encryption You Think You Have

curl reaches FTPS — FTP wrapped in TLS — two ways, matching the protocol's two flavors. For explicit FTPS (the common one: connect to port 21, then upgrade the connection to TLS) keep the ftp:// scheme and add a flag. For implicit FTPS (TLS from the first byte on port 990) use the ftps:// scheme. The difference between the two flavors is explained in explicit vs implicit FTPS.

Here is the trap: the flag that requests the upgrade comes in two strengths, and the weaker one is a security bug waiting to happen. --ssl means try to upgrade — and continue in cleartext if the server declines. --ssl-reqd means require the upgrade — and fail if it is not available. An unattended job must use --ssl-reqd, full stop. With --ssl, a misconfigured or downgraded server silently turns your "encrypted" job into a plaintext one, and nothing tells you.

curl --ssl-reqd --netrc -T daily.csv ftp://ftp.partner.example/incoming/   # explicit FTPS, required
curl --netrc -T daily.csv ftps://ftp.partner.example/incoming/             # implicit FTPS, port 990

TLS also means certificate verification, which curl performs by default against the system's trusted authorities. When a partner uses a private certificate authority, point curl at that CA's certificate with --cacert. What you should not do is reach for -k (--insecure), which accepts any certificate — including an attacker's. If -k appears in a production script, the job has encryption without identity, which is half a lock.

SFTP Through curl

Where your build supports it, curl speaks SFTP with the same one-shot grammar: sftp:// URLs, -T to upload, -O/-o to download. Authentication is SSH authentication: name a private key with --key, and depending on the build a running SSH agent can supply one too. For host identity, curl checks the server's key against ~/.ssh/known_hosts when that file exists and refuses a mismatch. So the cleanest workflow is to make first contact once with the sftp client. Verify the fingerprint out-of-band, and let curl inherit the recorded trust from then on. (Why that record matters is covered in host keys and known_hosts.)

curl -sS --key /etc/transfer/keys/transfer_key -u deploy: \
     -T results.csv sftp://sftp.partner.example/incoming/results.csv

Note the -u deploy: — username with an empty password, telling curl who to authenticate as while the key does the proving. Password-based SFTP works too (the transport is still encrypted). But then the password needs the same off-the-command-line handling as any other credential. That brings us to the section that matters most for production.

Credentials Off the Command Line

The naive form is -u reports:S3cretPass. Just as with every other client, that password is then visible to any local user in the process list for the life of the transfer. It is parked in shell history afterward. curl gives you two clean alternatives, both worth knowing.

The netrc file. --netrc (short -n) makes curl look up the machine name from the URL in ~/.netrc (on Windows, _netrc in the user's home folder) and use the credentials recorded there. --netrc-file /path points at a specific file, which is the right form for service accounts with their own credential files:

# /etc/transfer/netrc — chmod 600, owned by the job account
machine ftp.partner.example
login reports
password S3cretPass

# job invocation
curl -sS --netrc-file /etc/transfer/netrc -T daily.csv ftp://ftp.partner.example/incoming/

The config file. -K (long form --config) reads command-line options from a file — one option per line, long names without the dashes, values quoted. It can hold the whole job, not just the secret, which makes the invocation trivial and the job reviewable:

# /etc/transfer/push_daily.conf — chmod 600
url = "ftp://ftp.partner.example/incoming/"
upload-file = "/data/out/daily.csv"
netrc-file = "/etc/transfer/netrc"
ssl-reqd
ftp-create-dirs
silent
show-error

# invocation:  curl -K /etc/transfer/push_daily.conf

A last trick for environments with a secrets manager: -K - reads config from standard input. So a wrapper can fetch the credential at runtime and pipe user = "reports:$SECRET" straight into curl. Nothing secret ever touches disk or the argument list. Wherever the secret lives, file permissions (chmod 600) and a dedicated job account are the floor; the wider storage question is covered in our scheduled jobs series.

Retries, Timeouts, and Honest Exit Codes

Three flag families turn a curl line into something a scheduler can trust. First, timeouts: --connect-timeout 15 caps how long curl waits to establish the connection. --max-time caps the entire operation. Set the latter generously or not at all for large files, or you will kill healthy transfers that are merely big.

Second, retries: --retry 3 re-attempts the transfer on transient failures — timeouts and the retryable server responses — with a growing delay between attempts. --retry-delay 30 fixes that delay instead, and --retry-max-time 600 caps the total time spent trying. There are two sharp edges. A refused connection does not count as transient unless you add --retry-connrefused. And --retry-all-errors retries everything — including permanent failures like a bad password, which can hammer a partner's server and lock the account. Retry the failures that can heal, fail fast on the ones that cannot; the taxonomy is in our retry and error handling series.

Third, the exit code, curl's quiet gift to automation: it does not just succeed or fail, it tells you how it failed, stably and documented. The ones transfer scripts meet most:

A job wrapper can branch on these — retry a 28, alert on a 67, treat a 78 as "check again later". That is a level of scripted intelligence the classic ftp clients, with their permanent exit code 0, simply cannot offer.

And when the code alone does not explain a failure, -v replays the whole conversation. You see every FTP command curl sent, every numbered reply the server returned, and the TLS negotiation on FTPS sessions. Run the failing line once with -v, read the last few exchanges, and the mystery usually names itself. It might be a 550 permission refusal, a passive-mode address that goes nowhere, or a certificate the system does not trust. It is the same skill as reading any FTP session log, and it transfers directly to every other client you use.

Where curl Beats a Full Client — and Where It Loses

Reach for curl when the job is one transfer, known in advance: a nightly push of one file, a fetch of a known URL, a streamed backup. Its ubiquity means no install step, its exit codes make wrappers short, and one line in a scheduler is the whole deployment. It is also the natural bridge tool on Windows hosts, where it is present by default while richer Unix clients are not.

curl loses when the job needs session intelligence. It has no mirror mode: synchronizing a directory tree means listing, comparing, and looping yourself — work lftp does in one command. Parsing curl's directory listings to decide what to fetch is fragile compared to a client built for it. Interactive exploration is nicer in the sftp client. And when a flow grows past "one transfer" into schedules, monitoring, encryption steps, and notifications, the honest move is off hand-rolled wrappers entirely. That operational layer — wizard-built scheduled tasks, folder monitoring, email alerts on failure — is what Sysax FTP Automation provides as product rather than project. For experiments meanwhile, a trial of Sysax Multi Server gives you a local FTPS and SFTP server to test every flag in this article against. The trial is free from the download page. You can test before anything touches a partner system.

The Short Version

curl is the transfer tool you already have everywhere. It offers FTP, FTPS, and (on most builds) SFTP through one URL grammar. Use -T up and -O/-o down, and --ssl-reqd when encryption must be real. It offers netrc and config files to keep secrets out of process lists. It also offers retry, resume, and exit-code behavior that makes unattended jobs supervisable. Its boundary is session work — mirroring, listing logic, interactivity — where the fuller clients take over.

Next in the series: batch modes, heredocs, and command files for driving every client unattended. Then comes the one-liner cookbook, where curl's best recipes sit alongside sftp's and lftp's ready to copy.

Frequently Asked Questions