curl and wget for File Transfer: The Admin's Cookbook

Somewhere on every server you manage, a file needs to be fetched or delivered without a human clicking anything: an installer pulled during provisioning, a nightly export pushed to a partner's endpoint, a log bundle shipped to a support portal. The two tools that do this work almost everywhere are curl and wget — present or one package away on virtually every Linux box, and curl ships with modern Windows too. Most admins know one flag apiece (-O, maybe -c) and reinvent the rest under pressure.

This is the cookbook version: twenty worked one-liners covering downloads, resume and retry, uploads, authentication, and the error handling that separates a script you can trust at 3 a.m. from one that silently saves an HTML error page named backup.tar.gz. Every recipe states what it does and the trap it avoids. Copy them, adapt the hostnames, and keep the page bookmarked.

This article is part of our HTTP/HTTPS File Transfer series. If you want to understand what these commands actually put on the wire — request bodies, multipart boundaries, PUT versus POST — read how web upload works alongside this one.

Meet the Two Tools

curl is a general-purpose transfer client: it speaks HTTP and HTTPS plus a long list of other protocols (FTP, FTPS, and — when built with SSH support — SFTP and SCP), it can send any method with any body and any header, and it does exactly what you say and nothing more. By default it does not follow redirects, does not retry, and writes downloads to standard output. That obedience is a feature in scripts: no surprises.

wget is a downloader with opinions: it saves to a file by default, follows redirects by default, retries failed downloads by default (twenty attempts!), and can mirror whole directory trees recursively. It uploads poorly and speaks fewer protocols, but for "get this file, keep trying until you have it," it is superb out of the box.

A fair rule: wget for grabbing things, curl for everything else. Both are actively maintained, both are scriptable, and knowing both means never fighting a tool into a job the other does natively.

Ground Rules Before You Script

Four habits prevent most of the grief people have with these tools, so they come before the recipes.

Quote every URL. URLs are full of characters the shell treats as instructions: & backgrounds a command, ? and * trigger filename matching, and square brackets mean something to both the shell and curl. An unquoted URL "works" right up until the first query string, then fails in a way that looks like a server problem. Double quotes around every URL, every time, costs nothing.

Never "fix" a certificate error with -k. When curl refuses a connection because the certificate cannot be verified, it is doing its job: that verification is the part of HTTPS that proves you are talking to the real server. The -k/--insecure flag (wget: --no-check-certificate) switches that proof off, and it has a way of surviving from a quick test into production scripts. The correct fixes are boring — install the internal CA certificate into the system trust store, or repair the server's certificate chain — and they leave the protection intact.

Assume nothing about the working directory. Interactive shells start where you are; cron jobs start somewhere else entirely. A script that writes -o export.csv drops its file wherever the scheduler happened to launch it. Absolute paths for every input and output make scheduled runs boring, which is the goal.

When in doubt, add -v. curl's verbose flag narrates the whole conversation — resolution, TLS handshake, request headers, response headers. Ninety percent of "why doesn't this work" questions answer themselves in that output. Take the flag out once the script behaves.

Downloads

1. Download a file, keeping the server's name for it.

curl -O https://files.example.com/tools/agent.zip

Capital -O writes agent.zip to the current directory. Without it, the binary would pour into your terminal.

2. Download to a name you choose, following redirects.

curl -L -o agent-latest.zip https://get.example.com/agent/latest

-L matters more than people expect: download links are very often redirects to a storage backend, and without -L curl saves the tiny redirect page itself — the classic "why is my ISO 300 bytes" mystery.

3. The same job in wget.

wget https://files.example.com/tools/agent.zip

Redirects followed, file saved, transient failures retried up to twenty times — all default behavior. This is why wget remains the comfort tool for interactive downloads on shaky links.

4. Download only if the remote copy is newer than yours.

curl -z blocklist.txt -o blocklist.txt https://feeds.example.com/blocklist.txt

-z sends a conditional request ("only if modified since this file's timestamp"); the server answers 304 Not Modified and curl writes nothing if you are current. wget's equivalent is -N. Perfect for hourly feed pulls that usually haven't changed.

5. Be polite with bandwidth.

curl --limit-rate 5M -O https://files.example.com/images/vm-image.qcow2

Caps the transfer at 5 MB/s so a bulk pull does not crowd the office link during business hours. wget spells it --limit-rate=5m.

6. Fetch a numbered series in one command.

curl -f --remote-name-all "https://files.example.com/logs/app-[01-31].log"

curl expands the bracket range itself — thirty-one requests, thirty-one files, no shell loop. --remote-name-all applies -O naming to every URL in the set.

Resume and Retry

7. Resume an interrupted download (curl).

curl -C - -o vm-image.qcow2 https://files.example.com/images/vm-image.qcow2

-C - means "look at how much of the local file exists and ask the server for the rest." Under the hood this sends a Range request — the mechanism explained in large files over HTTP — so it only works when the server supports ranges; most static-file servers do.

8. Resume in wget.

wget -c https://files.example.com/images/vm-image.qcow2

Same idea, one flag. Run it again after any interruption and it picks up where the partial file ends.

9. Retry transient failures automatically.

curl --retry 5 --retry-delay 10 -fsS -o report.zip https://api.example.com/exports/report.zip

--retry 5 re-attempts up to five times on transient trouble (timeouts, connection resets, 5xx responses), waiting 10 seconds between tries. Add --retry-all-errors to retry on any failure — but only for operations that are safe to repeat.

10. The stubborn-network loop: resume plus retry until done.

until curl -fsS -C - -o big.iso https://files.example.com/images/big.iso; do
  echo "interrupted, resuming in 15s..." >&2; sleep 15
done

Each pass resumes from wherever the last one died, so progress is never lost. This little loop has finished multi-gigabyte downloads over hotel Wi-Fi that no single attempt could survive.

Uploads

11. Upload a file as a raw PUT.

curl -fsS -T backup.tar.gz https://files.example.com/drop/backup.tar.gz

-T sends the file as the entire request body — no form wrapper. End the URL with a slash (.../drop/) and curl appends the local filename for you. This is the natural verb for scripted, repeatable pushes, including to presigned URLs.

12. Upload like a browser form (multipart).

curl -fsS -F "file=@report.pdf" -F "comment=Q3 final" https://portal.example.com/upload

-F builds a multipart/form-data body — the @ marks a file, plain values become text fields. Use this when the endpoint was built for a web form and expects named fields.

13. POST a raw body with the right content type.

curl -fsS --data-binary @export.json -H "Content-Type: application/json" https://api.example.com/imports

--data-binary sends the file byte-for-byte (plain -d strips newlines — a subtle corrupter of uploads). Setting Content-Type yourself matters: APIs routinely reject bodies labeled with curl's default form type.

14. Push over SFTP with the same tool.

curl -u alex: --key ~/.ssh/id_ed25519 -T nightly.csv sftp://transfer.example.com/inbox/

A build of curl with SSH support (common, not universal — check curl -V for sftp) speaks SFTP with key authentication. Handy when a partner offers only SFTP and you want one tool in the script; the protocol itself is covered in our SFTP series.

Authentication

15. Basic auth without the password in the command.

curl -u alex -fsS -O https://files.example.com/private/policies.pdf

Give -u a username only and curl prompts for the password, which then never appears in shell history or the process list. -u alex:secret works but leaks the secret to anyone running ps — fine in a lab, not on a shared box.

16. Token auth, secret kept in the environment.

curl -fsS -H "Authorization: Bearer $API_TOKEN" -o export.csv https://api.example.com/files/export.csv

The standard pattern for REST endpoints (much more on those in files over REST APIs). Load API_TOKEN from a root-only file or a secrets manager at job start — never hard-code it in the script you will inevitably commit somewhere.

17. Store credentials in .netrc instead of scripts.

printf 'machine files.example.com\nlogin alex\npassword S3cret!\n' > ~/.netrc
chmod 600 ~/.netrc
curl -n -fsS -O https://files.example.com/private/policies.pdf

-n tells curl to look up the machine's entry in ~/.netrc; wget consults the same file automatically. One protected file, many scripts, no secrets inline — the traditional Unix answer, and still a good one.

Script-Safe Error Handling

18. Fail loudly instead of saving an error page.

curl -fsS -o agent.zip https://files.example.com/tools/agent.zip || { echo "download failed" >&2; exit 1; }

The flag trio to memorize: -f turns HTTP errors (404, 500) into a non-zero exit code instead of saving the server's error page as your file; -s silences the progress meter that pollutes cron mail; -S keeps real errors visible despite -s. Without -f, a script can "succeed" while writing an HTML apology into agent.zip — the single most common curl scripting bug.

19. Branch on the exact status code.

code=$(curl -s -o /tmp/resp.bin -w '%{http_code}' https://api.example.com/files/export.csv)
[ "$code" = "200" ] || { echo "got $code" >&2; exit 1; }

-w '%{http_code}' prints the status code after the transfer, so the script can distinguish "not ready yet" (404) from "credentials broken" (401) from "server down" (503) and react differently to each.

20. Put a clock on everything.

curl --connect-timeout 5 --max-time 900 -fsS -o feed.xml https://feeds.example.com/full.xml

--connect-timeout bounds how long curl waits to reach the server; --max-time bounds the whole transfer. Without them, one hung connection can wedge a scheduled job forever — and the next run piles on top of it. Every unattended transfer deserves both flags.

One companion habit rounds out the set: verify what you fetched. If the publisher provides a checksum file, sha256sum -c agent.zip.sha256 after the download proves the bytes arrived intact — cheap insurance that catches truncation and corruption that HTTP itself will not always surface.

Bonus: Scout an Endpoint Before You Commit

Before pointing a multi-gigabyte download or an automated job at a URL, spend one request learning what you are dealing with. A HEAD request asks the server for the response headers only — no body, no download:

curl -sSIL https://files.example.com/images/big.iso

HTTP/1.1 200 OK
Content-Length: 8589934592
Content-Type: application/octet-stream
Accept-Ranges: bytes

Three answers in one glance. Content-Length tells you the size before you commit the disk space — that is an 8 GB file. Accept-Ranges: bytes promises that resume (recipes 7–8) will actually work here. And because -L is in the flag set, any redirect chain gets followed and reported, so you learn now — not mid-transfer — that the "download link" really points at an object storage backend. Thirty seconds of scouting regularly saves an evening of debugging.

From One-Liner to Scheduled Job

A recipe becomes infrastructure the moment you put it in cron or Task Scheduler, and that transition deserves three habits. First, make failures visible: rely on exit codes (recipes 18–19), write a log line per run, and alert on non-zero exits rather than hoping someone reads the log. Second, prevent overlap: a slow transfer plus an hourly schedule equals two copies of the same job fighting over one file — a lock file or the timeout in recipe 20 keeps runs from stacking. Third, keep credentials out of the crontab itself (recipes 16–17). A minimal wrapper that honors all three looks like this:

#!/bin/sh
# nightly-push.sh — deliver the daily export, log every run, fail loudly
set -eu
. /etc/nightly-push.env          # provides API_TOKEN, readable by root only
exec >> /var/log/nightly-push.log 2>&1
echo "== $(date '+%F %T') starting"
curl -fsS --connect-timeout 5 --max-time 900 \
     -H "Authorization: Bearer $API_TOKEN" \
     -T /data/export/latest.csv \
     https://api.example.com/imports/latest.csv
echo "== $(date '+%F %T') done"

With set -eu, any failed command stops the script with a non-zero exit that the scheduler can flag; the log records both attempts and outcomes; the token lives in a root-only file rather than the command line. This is about as far as a shell wrapper should be stretched.

Be honest with yourself about the maintenance curve, too. One cron entry wrapping recipe 11 is fine. A dozen jobs with retries, sequencing, failure notifications, and file renaming to prevent half-written pickups is a part-time hobby written in shell. That is the point where a dedicated tool earns its keep: for the FTP, FTPS, and SFTP legs of scheduled work, Sysax FTP Automation handles the schedule, folder monitoring, retries, and error handling as configuration instead of code — the same jobs, minus the hand-rolled loops. And when you need something to practice these one-liners against, a trial install of Sysax Multi Server gives you a local HTTPS and SFTP endpoint you can hammer harmlessly.

The Habits Behind the Recipes

Twenty commands compress into four reflexes worth keeping. Use -L whenever a download link might redirect. Use -fsS in every script, so failures fail. Use resume (-C -, -c) and retry for anything large or any link that wobbles — and design uploads so a retry is harmless, which usually means PUT to a stable path, as explained in how web upload works. Keep secrets out of command lines, always.

When a job outgrows single files — mirroring trees, syncing only what changed — step up a tool: wget's recursive mode for one-way scrapes, or the delta-based approach in our rsync series for repeated synchronization. And for very large single files over unreliable paths, the chunking strategies in large files over HTTP pick up where -C - leaves off.

Frequently Asked Questions