Bash Transfer Scripts: The Fundamentals

Somewhere on almost every Linux server there is a file transfer script that began life as a command typed at a keyboard. It worked, so someone pasted it into a file. The file worked, so someone scheduled it. And now a script that was never really designed moves business data every night. Then comes the night a filename contains a space, or a variable comes up empty. Or a failed step goes unnoticed and the partner calls to ask where their files are.

The difference between that fragile script and one you can trust unattended is not cleverness. It is a small set of learnable habits. A strict mode turns silent failures into loud ones. Quoting survives hostile filenames. And a stranger — including you, a year from now — can read the structure top to bottom without guessing.

This article teaches those habits and assembles them into a complete, copyable skeleton for a transfer script. It is the foundation article of our Bash & Cron transfer automation series. Everything later in the series — scheduling, locking, logging, error handling, hardening — builds on the shape established here.

A Transfer Script Is a Contract, Not a Convenience

Why do transfer scripts deserve more care than the average bit of shell glue? Three reasons.

First, they run unattended. A script you run by hand has a human safety net: you see the error, you stop, you fix it. A script run by cron at 02:00 has no one watching. Whatever it does wrong, it does wrong completely, and the evidence is whatever it happened to write down. (The scheduling side of that story is the subject of Cron for File Transfers, Without the Surprises.)

Second, they cross organizational boundaries. A transfer script's failures are often visible to someone else — a partner, a customer, another department expecting a file by 06:00. An internal script that breaks costs you a morning; a transfer script that breaks can cost an apology.

Third, they handle credentials and data that matter. The script authenticates as a service account and touches files that exist precisely because somebody needs them. Sloppiness here is not a style problem; it is an operational and security problem.

None of this means transfer scripts need to be long or sophisticated. The move from "commands I run by hand" to "a script that earns trust" is the first real rung of the automation ladder. Our automation maturity series maps that climb. The habits below are what the rung is made of.

The First Line: Choosing Your Shell Honestly

Every script starts with a shebang — the #! line that tells the operating system which interpreter runs the file. For transfer scripts, write bash and say so:

#!/usr/bin/env bash

Two details hide in that one line. The first is bash, not sh. On a modern Linux distribution, /bin/sh is frequently a minimal POSIX shell, not bash. Several of the safety features this series depends on (pipefail, the [[ ]] test syntax, arrays, PIPESTATUS) are bash features that a minimal shell does not have. A script that says sh but assumes bash works on one machine and breaks on the next.

The second is /usr/bin/env bash versus a fixed path like /bin/bash. The env form finds bash wherever it lives on the system's search path, which travels better between systems. The fixed path is more predictable on a single server you control. Either is defensible. Pick one convention for your estate and use it everywhere — consistency is worth more than the difference between them.

Strict Mode, Explained Gently

By default, bash is dangerously forgiving. A command fails, and the script simply carries on to the next line as if nothing happened. An unset variable expands to an empty string instead of an error. For interactive use that forgiveness is friendly; for an unattended transfer job it is how small failures become large ones. The classic disaster shape:

cd /data/exprot        # typo -- this cd FAILS
rm -f *.csv            # ...and this rm runs anyway, in the wrong directory

The cd fails because the directory does not exist. Bash shrugs, and the cleanup step deletes CSV files from whatever directory the script happened to be in. Strict mode is the conventional name for the settings that close this class of hole. Put this immediately after the shebang:

set -euo pipefail

It is three switches in one line, and each is worth understanding on its own:

• set -e (exit on error): if a command fails — returns a nonzero exit code, the number every program hands back to say how it went — the script stops right there instead of continuing wounded. The typo above now halts the script at the cd.
• set -u (unset is an error): expanding a variable that was never set becomes a fatal error instead of a silent empty string. rm -rf "$STAGING_DIR/"* with a misspelled variable no longer turns into rm -rf /*.
• set -o pipefail: normally a pipeline like generate | compress reports only the exit code of the last command. So a failure in generate vanishes if compress succeeds. With pipefail, the pipeline fails if any stage fails.

Now the honest part, because strict mode has real gotchas and pretending otherwise produces admins who trust it too much:

• set -e has blind spots. It deliberately does not trigger for commands whose failure you are visibly testing. That means anything in an if or while condition, or on the left side of && or ||. Subtler: inside a function called from a condition (if my_func; then), -e is suspended for the whole function body. Strict mode complements explicit error checks; it does not replace them. Error Handling in Bash covers checking properly.
• set -u punishes optional values. A script that accepts an optional argument must write "${1:-}" — "the first argument, or empty if absent" — instead of "$1". The ${VAR:-default} form gives a real default. This is a feature: it forces you to say which values are allowed to be missing.
• pipefail changes what "normal" means. grep exits with code 1 when it finds no matches — often a perfectly normal outcome, like "no error lines in the log." Under pipefail plus -e, that normal outcome kills the script. Where empty results are legitimate, say so explicitly: grep -c ERROR "$log" || true — the || true marks "a nonzero here is fine" as a deliberate decision.

Quoting: The Habit That Prevents the Weirdest Bugs

When bash expands an unquoted variable, it does two things you did not ask for. It splits the value into separate words at spaces, tabs, and newlines (word splitting). And it treats wildcard characters like * and ? in the value as patterns to expand against filenames (globbing). Both are invisible until a value contains the wrong character. Transfer scripts handle filenames chosen by other people and other systems, so they meet wrong characters constantly.

file="march orders.csv"

rm $file      # bash splits first -- rm sees TWO arguments: "march" and "orders.csv"
rm "$file"    # rm sees one argument: march orders.csv

The unquoted version deletes the wrong things or errors out, depending on what else is in the directory. Neither happens on the machine where you tested, because your test filenames were tidy. The rule that ends the entire bug class is short enough to memorize:

• Double-quote every expansion, every time: "$file", "$SOURCE_DIR", and command substitutions too — "$(date '+%b %d')". Do not stop to reason about whether this particular value could contain a space. Quote it anyway; quoting a value that did not need it costs nothing.
• Pass arguments through as "$@" — the quoted form hands each original argument through intact. Its cousin $* unquoted mashes them into one splittable string; you essentially never want that.
• Leave a glob unquoted only when you mean it: for f in "$SOURCE_DIR"/*.csv quotes the variable but leaves *.csv bare because expanding it is the point. That is the main legitimate exception, and it is a visible, deliberate one.

When a command's options grow past one line, bash arrays keep the quoting airtight. Build the option list once as ssh_opts=(-i "$SSH_KEY" -o IdentitiesOnly=yes) and use it as "${ssh_opts[@]}". That expands to exactly one word per element no matter what the elements contain. You will see that pattern in the skeleton below.

Variables and Configuration That Stay Out of Trouble

A transfer script has two kinds of values, and keeping them visually distinct is half the readability battle. Configuration — hosts, paths, account names — belongs in one block at the top of the file, in UPPERCASE names. Mark them readonly so nothing can accidentally reassign them mid-run. Someone adapting the script for a new partner should be able to do it without reading past line twenty. Working variables — the current file, a counter, a temporary name — live in lowercase. Inside functions they are declared local so they cannot leak out and collide with something else.

Two parameter-expansion idioms appear in nearly every transfer script and are worth learning by sight. ${file##*/} strips everything up to the last slash — the pattern-matching way to get a bare filename from a full path. And ${VAR:-fallback}, met above, supplies a default when a value is unset or empty.

One thing does not belong among the configuration: secrets. A password pasted into a script travels with every copy, backup, and repository push the script ever makes. Transfer scripts should authenticate with SSH keys owned by a dedicated service account. The reasoning and setup are covered in our guides to service account hygiene and generating and storing SSH keys. Anything that must be a secret string belongs in a separate permission-locked file, a subject the hardening article treats in full.

Functions Give the Script a Shape

Even a short transfer script benefits from three small functions. Two are helpers you will reuse in every script you write. log() prints a timestamped line to the job's log file, so the unattended run leaves a trail. die() logs an error and exits nonzero — one word that means "stop, loudly." Note both use printf rather than echo. printf behaves identically everywhere, while echo varies between shells in how it treats options and backslashes. That is exactly the kind of variation you do not want interpreting your log messages.

The third is main(). Putting the actual sequence of work in a function called main, and invoking it with main "$@" as the file's last line, buys two things. Readability: the file reads as configuration, then vocabulary, then story. And there is a subtle safety property. Bash reads script files incrementally as it executes. So editing a script while cron is running it can make the running copy execute a half-old, half-new line. When everything is inside functions, the whole program is parsed before the first real action runs, and that failure mode disappears.

In longer jobs, give each stage its own function — export_files, upload_files, archive_files. Let main be the short list of calls that tells the story in order. That structure also pays off when you add per-step error handling and cleanup traps later in the series.

The Complete Skeleton, Line by Line

Here is the whole shape assembled: a real, runnable-shaped nightly upload script. It sends every CSV file from an export directory to a partner's SFTP server, using the safe-delivery pattern of uploading to a temporary name and renaming when complete.

#!/usr/bin/env bash
#
# push-orders.sh -- upload nightly order exports to the partner SFTP server.
# Runs from cron as the "xfer" service account. Runbook: ops wiki, "push-orders".
#
set -euo pipefail

# ---- configuration ----------------------------------------------------
readonly JOB_NAME="push-orders"
readonly SOURCE_DIR="/data/export/orders"
readonly REMOTE="orders@files.partner.example"
readonly REMOTE_DIR="incoming"
readonly SSH_KEY="/etc/transfer/keys/orders_ed25519"
readonly KNOWN_HOSTS="/etc/transfer/known_hosts"
readonly LOG_FILE="/var/log/transfer/${JOB_NAME}.log"

readonly SFTP_OPTS=(-i "$SSH_KEY" -o IdentitiesOnly=yes
                    -o UserKnownHostsFile="$KNOWN_HOSTS"
                    -o StrictHostKeyChecking=yes -o ConnectTimeout=30)

# ---- helpers ----------------------------------------------------------
log() {
    printf '%s %s: %s\n' "$(date '+%b %d %H:%M:%S')" "$JOB_NAME" "$*" >> "$LOG_FILE"
}

die() {
    log "ERROR: $*"
    exit 1
}

# ---- main -------------------------------------------------------------
main() {
    log "run started"

    [ -d "$SOURCE_DIR" ] || die "source directory missing: $SOURCE_DIR"

    local file name
    for file in "$SOURCE_DIR"/*.csv; do
        [ -e "$file" ] || { log "nothing to send; run finished"; return 0; }
        name="${file##*/}"
        log "uploading $name"
        sftp "${SFTP_OPTS[@]}" -b - "$REMOTE" >> "$LOG_FILE" 2>&1 <<EOF
put "$file" "$REMOTE_DIR/$name.part"
rename "$REMOTE_DIR/$name.part" "$REMOTE_DIR/$name"
EOF
        log "uploaded $name"
    done

    log "run finished ok"
}

main "$@"

A walkthrough of the decisions, top to bottom:

• The header comment answers the questions a stranger asks first: what is this, where does it run, who runs it, where is it documented. Thirty seconds of writing that saves an hour of archaeology later.
• The configuration block holds every value that could differ between environments — all readonly, all referenced in quotes below. The SFTP options live in an array so the quoting stays correct however long the list grows.
• The option list itself is the standard kit for unattended SFTP. It includes a specific key with IdentitiesOnly so no other keys get offered. It also includes a pinned known-hosts file with strict checking so the script only ever talks to the server it was told to trust. (See host keys and known_hosts for why the popular shortcut of disabling that check is a mistake.) And it includes a connection timeout so an unreachable server produces a fast failure instead of a hang. The full unattended-SFTP story, including batch mode's fail-don't-prompt behavior, is in our SFTP automation guide.
• The glob check — [ -e "$file" ] — handles a bash quirk. When *.csv matches nothing, the pattern stays in the loop as literal text. So the first (only) iteration sees a file named *.csv that does not exist. Testing for existence turns "no files tonight" into a logged, graceful non-event instead of an error.
• The upload-then-rename pair means the file never exists at its final remote name until it is complete. So a watcher on the far end can never collect half a file.
• Output goes to the log — both streams, via >> "$LOG_FILE" 2>&1. The exit behavior is honest: any sftp failure trips set -e, the script dies mid-loop, and cron sees a nonzero exit. Later articles refine both halves: the logging patterns article makes the trail properly answerable. The error-handling article adds traps and per-file accounting so one bad file does not abandon the rest.

Habits That Keep It Readable a Year From Now

A transfer script is read far more often than it is written — during incidents, mostly, by people in a hurry. A few habits keep future readers (including future you) on your side. Keep scripts in version control, so "what changed before this broke" is a one-command question. Run new scripts through shellcheck, the long-standing static analyzer for shell scripts; it catches unquoted expansions and strict-mode traps mechanically. Comment the why, not the what — # partner's server drops idle sessions, hence the timeout is worth ten comments that restate the code. And when a script grows past a couple hundred lines or starts parsing file contents, that is usually the sign to reach for a different tool. Our Python transfer automation series picks up exactly there.

One more reality of estate scripting deserves a mention: the far end of a bash transfer script is very often not another Linux box. Mixed estates are the norm. The partner or internal server your script uploads to is frequently a Windows machine running an SFTP or FTPS service. For example, Sysax Multi Server is a Windows file transfer server that speaks SFTP, FTPS, FTP, and HTTPS and runs as a Windows service. Your script neither knows nor cares: the OpenSSH sftp client in the skeleton talks to it exactly as it would to any other SFTP endpoint. The server's own activity logging gives the Windows side a matching record of every session your cron job opens.

Where to Go From Here

You now have the fundamentals: strict mode with its honest limits, quoting as a reflex, and configuration separated from logic. You have helpers and main() giving the file a shape, and a skeleton that puts them all in one place. What the skeleton does not yet have is a schedule, protection against overlapping runs, production-grade logging, or real error recovery — each of which is its own article in this series.

The natural next step is Cron for File Transfers, Without the Surprises, because a script this clean can still fail in cron's stripped-down environment. Knowing why is the difference between a five-minute fix and a lost evening. After that, the error-handling article upgrades die() into traps and cleanup that run no matter how the script ends. And when the Windows half of your estate needs the same discipline without hand-written scripts, Sysax FTP Automation covers that side with wizard-generated scheduled transfer tasks, folder monitoring, and email notifications. Those are the same principles, packaged for the platform where bash is not the native tongue.

Frequently Asked Questions