Batch Modes, Heredocs, and Command Files

Every command-line transfer client was born interactive: it prints a prompt, you type a command, it answers. Automation is the art of removing the human from that loop without removing the reliability. It means feeding the client its commands from somewhere else and capturing what it says. It turns "it looked fine when I typed it" into "the scheduler knows whether it worked."

There are exactly three ways to do that feeding, and this article covers all of them. First are the batch flags built into the clients (sftp -b, lftp -f, the old ftp -s:). Then comes the shell's heredoc mechanism with its famous quoting traps. Finally, there are standalone command files kept under version control. You will learn when each fits, and the failure modes each hides. You will learn how to capture per-command output so the log can answer questions at 8 a.m. about what happened at 2 a.m. This is part of our command-line client mastery series, and it builds directly on the sftp client, mastered.

Three Ways to Feed a Client

Strip away the syntax and every unattended transfer works the same way: a list of commands exists somewhere, and the client consumes it from top to bottom. The three techniques differ only in where the list lives.

• A batch flag with a file: the commands live in their own file, and the client is told to read it — sftp -b push.sftp, lftp -f pull.lftp, ftp -s:commands.txt. The file is a reviewable artifact; the invocation is short.
• A heredoc (or pipe): the commands live inside the calling script, delivered through standard input. Everything about the job sits in one file, and the shell can substitute variables into the commands — which is both the appeal and the trap.
• A generated command file: a hybrid for parameterized jobs — a template plus a few substitutions produces the batch file at run time, keeping review-ability and flexibility.

One principle governs all three, and it is worth stating before any syntax: the feeding mechanism does not create safety — the client's error behavior does. A perfect heredoc feeding a client that ignores failures is still an unsafe job. So we start with the client where the semantics are right, and work down to the one where they are wrong.

sftp -b: The Standard to Measure Against

The sftp client's batch mode has the three properties an unattended job needs. It is worth naming them because we will judge everything else against this list. The client aborts at the first failed command, so a failed upload never gets renamed, published, or deleted-from under. It lets you mark exceptions deliberately — prefix a single command with - and its failure is tolerated. And it returns an honest exit code: 0 only if everything succeeded.

# push_daily.sftp — comments and blank lines are allowed
cd /incoming
put /data/out/daily.csv daily.csv.part
rename daily.csv.part daily.csv
-mkdir archive

# invocation, with prompts disabled and a deadline:
sftp -b push_daily.sftp -i /etc/transfer/keys/transfer_key \
     -o BatchMode=yes -o ConnectTimeout=15 deploy@sftp.partner.example

Batch mode presumes nobody is present, so it pairs with authentication that needs nobody. That means an SSH key named with -i, plus -o BatchMode=yes. Any prompt that would have appeared — password, passphrase, host-key question — then becomes an immediate, visible failure. The job will not hang waiting for a keyboard that is not there.

The batch file may also be -, meaning "read commands from standard input" — the bridge between batch mode and heredocs that we will use shortly. Key authentication, host-key preparation, and the full flag set are covered in the sftp article; here it simply sets the bar: fail fast, fail loudly, report honestly. lftp's -f script mode reaches the same bar once you add set cmd:fail-exit yes, as shown in the lftp article.

The Windows Legacy: ftp -s: Told Straight

Now the other end of the spectrum. The Windows console ftp client's -s: mode reads a command file and types it at the session — and that is the whole feature. Measured against our list: it does not abort on failure. A failed cd is followed by a put into the wrong directory, executed with full confidence. The client has no way to mark tolerated exceptions, and its exit code is 0 essentially no matter what happened. Add that the protocol underneath is plain FTP — credentials and data in cleartext, no secure option at all. The command file cannot even carry comments (any unrecognized line just produces an error and moves on). The honest verdict is: ftp -s: is a legacy pattern to be recognized and retired, not written new. The full case lives in our built-in ftp clients article and why retire plain FTP.

While one still runs in your estate, two mitigations make it less blind. Capture the session and scan it for failure reply codes, because the log is the only truth the client offers:

ftp -n -i -s:commands.txt > C:\logs\ftp_last.log
findstr /r "^4[0-9][0-9] ^5[0-9][0-9]" C:\logs\ftp_last.log && echo JOB FAILED

And plan the replacement to be cheap. These jobs are defined by their command files. So the least disruptive migration is a client that consumes the same style of script but speaks secure protocols. That is precisely what sysaxftp.exe, the command-line client included with Sysax FTP Automation, is for. It works as a drop-in replacement for the Windows console ftp client and adds SFTP and FTPS. So an old -s: batch job can move to an encrypted protocol with minimal change instead of a ground-up rewrite.

Heredocs, Explained Gently

A heredoc ("here document") is the Unix shell's way of embedding a block of text in a script and feeding it to a command's standard input. The marker word after << names a terminator; everything until a line containing exactly that word becomes input:

sftp -b - -o BatchMode=yes deploy@sftp.partner.example <<EOF
cd /incoming
put /data/out/export_$(date +%Y%m%d).csv
EOF

Note the pieces working together. -b - keeps sftp's real batch semantics (abort on error, honest exit code) while the commands arrive from the script itself. The shell expands $(date +%Y%m%d) before sftp ever sees the line — so the uploaded name carries today's datestamp. That expansion is the heredoc's whole value, and also its trap, because it has two failure directions:

• Expansion you did not want. With an unquoted terminator (<<EOF), the shell processes $variables, $(commands), and backslashes inside the block. Feed the old ftp client a line like user reports pa$$word and the shell rewrites $$ (its own process ID) before ftp sees it. The login fails, and nothing in the log explains why. Any literal $ in a password, path, or filename is a landmine in an unquoted heredoc.
• Expansion you wanted but suppressed. Quote the terminator — <<'EOF' — and the block is perfectly literal: no variables, no substitutions. Safe for static command lists, wrong for that datestamped upload, which would now try to send a file literally named export_$(date +%Y%m%d).csv.

The rule to memorize: quote the terminator unless you specifically need substitution, and if you need substitution, keep literal $ out of the block. A third variant, <<-EOF, strips leading tab characters (tabs only, not spaces) so the block can be indented inside an if or loop. And bash offers the one-line cousin, the here-string: sftp -b - deploy@host <<< "ls -l /incoming". That is handy for a single command, though a plain printf ... | sftp -b - pipe does the same portably in any shell.

The classic Unix heredoc-driven ftp job, for recognition purposes (it shares every weakness of ftp -s:, cleartext password included):

ftp -inv ftp.partner.example <<'EOF' > /var/log/ftp_push.log
user reports S3cretPass
binary
cd /incoming
put daily_export.csv
bye
EOF

PowerShell readers: the concept ports directly. PowerShell's here-strings (@"..."@ expanding, @'...'@ literal) carry the same two-directional trap. The transfer-specific patterns live in our PowerShell transfer automation series.

Command Files Under Version Control

Once a job matters, move its command list out of the wrapper and into its own file. Put that file under version control next to the script that invokes it. The gains are the same ones code gets. There are diffs (what changed before the job broke Tuesday night). There is review (a second pair of eyes on a rename sequence before it runs in production). There are rollback and reuse (the same batch file driven by test and production wrappers against different hosts).

Parameterization is the usual objection: "our filenames change daily, so the commands must be built on the fly". The generated-file pattern answers it while keeping the artifact reviewable. Keep a template with obvious placeholders, render it to a temporary file at run time, and clean up after:

# push.sftp.tmpl — reviewed, version-controlled
cd /incoming
put __LOCALFILE__ __NAME__.part
rename __NAME__.part __NAME__

# wrapper excerpt
SRC=/data/out/daily_export.csv
BATCH=$(mktemp) || exit 1
trap 'rm -f "$BATCH"' EXIT
sed -e "s|__LOCALFILE__|$SRC|" -e "s|__NAME__|$(basename "$SRC")|" \
    push.sftp.tmpl > "$BATCH"
sftp -b "$BATCH" -o BatchMode=yes deploy@sftp.partner.example

Reuse across environments is where the discipline pays twice. Keep one reviewed command file, and let each environment's wrapper supply its own host, key, and paths. The test wrapper points the identical batch file at your staging server, and the production wrapper points it at the partner. When test and production run the same reviewed commands, "it worked in test" finally means something.

Two rules keep command files trustworthy. First, no credentials inside them, ever — the file is now in version control, backed up, and diffed in emails. Authentication belongs to SSH keys or a permission-locked netrc. The storage question is covered in our scheduled jobs series. Second, name them for their job (push_daily.sftp, not commands2.txt), because six months from now the filename is the documentation.

Capturing Output for the Log

An unattended run that leaves no record did not happen, as far as tomorrow's troubleshooting is concerned. The mechanics are one line of shell — redirect standard output and standard error into the job's log. But what lands in that log differs by client, and it pays to know what to expect:

• sftp in batch mode echoes each command as it executes, so the log interleaves commands with results and reads as a narrative. With -q the progress meter stays out of the way.
• The classic ftp clients print server replies when verbose (-v, or by default in -s: mode) — reply codes but no local context, which is why the findstr scan earlier works at all.
• lftp is quiet by default; give mirror the --verbose flag, and use its echo command to write your own milestones into the stream.

LOG=/var/log/transfers/push_daily.log
echo "=== run started Mar 14 02:10 ===" >> "$LOG"   # your wrapper stamps real dates
sftp -q -b push_daily.sftp deploy@sftp.partner.example >> "$LOG" 2>&1
echo "=== exit $? ===" >> "$LOG"

The resulting file answers the three questions that matter — what ran, what the server said, how it ended — one command at a time:

=== run started Mar 14 02:10 ===
sftp> cd /incoming
sftp> put /data/out/daily.csv daily.csv.part
sftp> rename daily.csv.part daily.csv
=== exit 0 ===

Two refinements cover the remaining cases. When you want to watch a run live and keep the record, pipe through tee. sftp -b push_daily.sftp host 2>&1 | tee -a "$LOG" shows the session on screen while appending it to the file. That is ideal for supervised first runs of a new job. And when a script captures output in order to parse it, keep the streams separate. One example is a directory listing that decides what to fetch next. Send standard output to the data file and standard error to the log (> listing.txt 2>> "$LOG"). That way, a stray warning never lands in the middle of the data your parser reads.

Timestamps on every run, rotation before logs eat the disk, and the discipline of checking $? belong to the wrapper script, and our bash and cron automation series treats them properly. The step beyond logging — retrying what failed — is a design topic of its own: see retry and error handling.

The Trap List: Quoting, Line Endings, and Friends

A short checklist collected from real 2 a.m. incidents. Run any new unattended job past it before its first scheduled night:

1. CRLF line endings. A batch file edited on Windows and run on Linux carries an invisible carriage return on every line. sftp then hunts for a file named daily.csv\r. It fails with a "not found" that looks insane in the log, since the name prints identically. Fix with dos2unix or tr -d '\r', and keep an eye on editors that "helpfully" convert.
2. Spaces in filenames. Inside sftp batch commands, quote them: put "Monthly Report.xlsx" "Monthly Report.xlsx". Better, keep automation filenames space-free entirely.
3. Who expands the wildcard? In a heredoc, put *.csv reaches the client untouched (the shell does not glob heredoc text) and the client expands it. On a normal command line, your shell may expand it first against the local directory. Know which one you are relying on.
4. Unquoted heredoc, literal dollar. The pa$$word trap from earlier. Any $ that must survive verbatim needs <<'EOF' or escaping.
5. The tolerated-failure creep. Every - prefix in an sftp batch file is a failure nobody will hear about. Re-read them quarterly and ask whether each is still genuinely optional.
6. Testing as yourself instead of the job account. Different home directory means different keys and known_hosts; rehearse under the account the scheduler will use.

Choosing Among the Three — and When to Graduate

That last row deserves its honest sentence. When the wrapper around a batch file has grown retry loops, freshness checks, alert emails, and an encryption step, you are maintaining an application. A product already meets its requirements. Sysax FTP Automation builds scheduled SFTP, FTPS, and FTP jobs through a wizard. It watches folders, encrypts with OpenPGP, and emails on failure. Those are the same outcomes with the machinery as configuration instead of code. The command-file skills stay valuable either way: they are how you will always test, diagnose, and understand what any automation layer does underneath.

Pulling It Together

Unattended transfer is a solved problem when three things line up. The first is a client whose batch semantics fail fast and report honestly (sftp's -b is the model; the legacy ftp -s: is the cautionary tale). The second is a feeding mechanism chosen deliberately (heredoc for embedded short jobs, version-controlled command files for stable ones, generated files for parameterized ones). The third is output captured so every run leaves a narrative. Add the trap list, and your batch jobs will be boring — the highest compliment automation gets.

Continue with the transfer one-liner cookbook, where these patterns compress into memorable single lines. Or back up to the built-in ftp clients if you are still deciding what to migrate away from.

Frequently Asked Questions