From Script to Schedule: Jobs That Run Themselves

sftp: command not found. That is the entire log from the first night your script ran alone, on a machine where sftp has been installed for years. The script works; it has worked every morning for a month. You typed the command, watched the log lines scroll, glanced at the result, and got on with your day. So you gave it to the scheduler, set it for 02:10, and went home. On its very first night alone it failed in a way it had never failed before. It was not the network, not the partner, but something that reads as if the machine had forgotten what it owns.

This is the most instructive jump on the automation ladder. A script that runs attended and a job that runs unattended are different animals, because the script was quietly leaning on you. It relied on your environment, your account, your ability to answer a prompt, your eyeballs on the output. It never mentioned any of this, and you never asked. This article catalogs everything that changes when nobody is watching. It shows how to hand a script to cron and to Task Scheduler correctly. It covers the two disciplines that stop being optional the moment you walk away — logging and monitoring. It is the fourth article in our Automation Ladder series, and it assumes you have a script worth scheduling, built the way your first scripted transfer describes.

The Gap Between "Works for Me" and "Works Alone"

Picture your interactive session as a warm, furnished room. It has your environment variables, your PATH, your home directory with your SSH keys and your known_hosts file. It has your mapped drives and your locale. And — most importantly — it has you, ready to notice anything odd and type answers to anything that asks. Your script grew up in that room and absorbed its comforts without either of you noticing. It has never once slept anywhere else.

A scheduler runs the same script in a bare concrete cell. The environment is minimal or different. The account is often not yours. There is no terminal, so anything that tries to prompt does not get an answer — it waits forever. And there is no human, so anything that goes wrong is gone, until someone looks. Nothing about the script changed; everything about its world did. The failures this produces are so consistent across platforms that you can list them in advance. That is exactly what the next section does, so you can fix each one before the first unattended night instead of after it. After is when most of us fixed them.

Bluewater Bank's statement-export job made the jump the usual way. The script had run by hand for a month without a hitch, so it went into the scheduler at 02:10 under a new service account. Its author went home. The next morning the scheduler showed the task as still running, and it stayed that way. The service account had never connected to the partner before. The job was sitting at a host-key confirmation prompt, waiting politely for a terminal that did not exist. The morning check caught it on day one, the file went by hand, and the account got its own verified known_hosts entry that afternoon. Nothing about the script had changed. Everything about the room it ran in had.

Everything That Breaks When Nobody Is Watching

The environment shrinks

Interactive shells load startup files that set dozens of environment variables; schedulers do not. Cron famously provides a near-empty environment with a minimal PATH — often just /usr/bin:/bin. That is why a script that calls a tool living in /usr/local/bin works for you and dies at 02:10 with "command not found." On Windows, a task running as another account gets that account's environment, not yours. It may run without loading a user profile at all, so variables and per-user settings you rely on are absent.

The fix is a habit, not a trick: a scheduled script assumes nothing from its environment. Call programs by absolute path or set PATH explicitly at the top of the script. Reference every file and folder by absolute path — never assume the working directory, because the scheduler decides that, not you. If the script needs a proxy variable or anything similar, set it in the script. The scheduler will not lend it one.

The account changes

By hand, the script ran as you. Scheduled, it should run as a dedicated service account — an identity created for the job, with only the permissions the job needs, per service account hygiene. That is the right design, and it breaks three things you set up as yourself. They include the SSH private key sitting in your home directory and the host key recorded in your known_hosts. The third is any folder permissions granted to your account. All three must be re-established for the service account. It needs its own key (registered with the partner). It needs its own verified known_hosts entry — see host keys and known_hosts. It also needs explicit permissions on every path the job touches.

Windows adds a classic trap of its own: mapped drive letters do not exist in non-interactive sessions. The X: drive you see is a convenience of your logged-in session; the scheduled task running as the service account has never heard of it. Use full UNC paths (\\fileserver\export\outbox) in anything a task will run.

Prompts become deadlocks

Every interactive question is now a hang. There may be a password prompt because the key was not found, or a host-key confirmation because this account has never connected before. There may be an "overwrite? (y/n)" from a copy command. With no terminal attached, each of these stops the job cold, silently, sometimes leaving it "running" in the scheduler for days. Hunt prompts down before scheduling. Connect once as the service account to settle the host key. Use BatchMode=yes so authentication fails loudly instead of prompting. Pass the force/quiet flags that tell each tool never to ask. A useful smoke test: run the script with its input redirected from nothing — ./upload_daily_report.sh < /dev/null. Anything that was going to prompt will fail immediately instead of waiting.

The world at 02:10 is different

Your morning runs enjoyed a fully awake network: VPN up, DNS warm, the export finished hours ago. The small hours have their own weather. The export job may still be running at 02:10 on month-end — so the file is missing or, worse, half-written. Backup jobs may be saturating the link. A nightly reboot window may have just restarted the machine, and the job fires before the network or a dependency service is ready. None of these are exotic; all of them are invisible until the job runs at the real hour. There are two defenses. Order the schedule around upstream reality (ask what time the export finishes on its slowest day, then add margin). Keep the preflight checks strict so a missing or empty file produces a clean, logged failure rather than a garbage transfer. If you also operate the receiving side on Windows, run the server as a service rather than an app in someone's session. A server like Sysax Multi Server runs as a Windows service precisely so it is listening after every reboot with nobody logged in.

Everything above compresses into one pre-flight checklist. Run it before the first scheduled night — every line is a real 2 a.m. failure someone has already had:

GOING-UNATTENDED CHECKLIST
[ ] Every program called by absolute path, or PATH set in the script
[ ] Every file and folder referenced by absolute path (UNC, not drive letters)
[ ] Script tested with: ./script.sh < /dev/null   (no hidden prompts)
[ ] Runs as the service account, not me (su/runas test performed)
[ ] Service account has: its own key, its own known_hosts entry,
    permissions on every path the job touches
[ ] BatchMode=yes (or equivalent): auth failure is loud, never a prompt
[ ] Schedule set AFTER the upstream file is reliably ready (worst day + margin)
[ ] Preflight check rejects missing/empty input with a logged error
[ ] Log file path is absolute, and the service account can write to it
[ ] Failure is visible: nonzero exit reaches the scheduler (next section)

Giving the Job a Clock: cron and Task Scheduler

With the script hardened, the scheduling itself is the easy part — one line or one command per platform. On Unix-like systems, the scheduler is cron, and jobs live in a crontab (the table of scheduled commands). Edit the service account's crontab with crontab -e and add:

10 2 * * 1-5  /usr/local/bin/upload_daily_report.sh >> /var/log/transfers/acme_daily.cron.log 2>&1

The five fields before the command are minute, hour, day of month, month, and day of week. So this reads "at minute 10 of hour 2, any date, any month, Monday through Friday" (days 1–5; 0 is Sunday). The redirection matters as much as the timing. The >> operator appends the job's standard output to a file, and 2>&1 sends standard error to the same place. So nothing the script or sftp prints can vanish. Without it, cron tries to mail the output to the local account — a destination nobody reads on most systems. Cron has more personality than one paragraph can cover (environment quirks, staggering, testing the way cron runs things); the bash and cron series gives it the full treatment.

On Windows, the scheduler is Task Scheduler, and the same job is created from an elevated prompt like this:

schtasks /create /tn "ACME daily report upload" ^
  /tr "C:\transfers\upload_daily_report.cmd" ^
  /sc weekly /d MON,TUE,WED,THU,FRI /st 02:10 ^
  /ru CORP\svc-transfer /rp *

Reading the flags: /tn names the task (make it say what and whose — this name is what a colleague sees in the task list). The /tr flag is the action to run. The /sc weekly flag with /d MON,TUE,WED,THU,FRI gives the weekday schedule. The /st flag sets the start time. The /ru flag sets the account the task runs as — the service account. And /rp * makes the command prompt for that account's password rather than putting it on the command line (where it would linger in shell history). Creating the task this way registers it to run whether or not anyone is logged on, which is exactly what a transfer job needs. The same options exist in the Task Scheduler GUI — the checkbox "Run whether user is logged on or not" is the one that matters. Scheduler mechanics on both platforms, including missed-run behavior after reboots and patches, get a full series of their own in scheduled jobs done right.

Why Logging Stops Being Optional

While the script was attended, the log was a courtesy; your eyes were the real record. Unattended, the log is the only witness. When Thursday's file is missing, the log is how you distinguish three different problems. They are "the job never ran," "it ran and found nothing to send," and "it sent the file and the partner lost it." They have three different fixes and are indistinguishable without evidence. Without the log, all three look like Thursday.

A scheduled transfer job's log has to answer four questions for every run. When did it start, what did it decide to transfer, what happened, and how did it end? Timestamps on every line, the file name and size, the client's own output captured, and the exit status recorded. The wrapper script from the previous article already does this. The crontab line above adds the belt-and-suspenders layer, catching anything that escapes to standard output or error. A healthy night and an unhealthy night should look different at a glance:

Mar 13 02:10:01 START uploading report_YYYYMMDD.csv to sftp.example.com
Mar 13 02:10:04 OK upload complete
Mar 13 02:10:04 DONE local copy moved to sent/

Mar 14 02:10:01 ERROR report_YYYYMMDD.csv missing or empty in
                /data/export/outbox - nothing sent

That Mar 14 entry is a success story, not a failure story: the job met a bad situation, refused to make it worse, and wrote down exactly what a responder needs. Deciding what belongs in transfer logs — and what must never be logged, like credentials — is covered in what to log. One more habit worth adopting on day one: pick a log location with room to grow and rotate the files. A job that runs nightly for years writes a lot of history. A full disk is itself a classic cause of 2 a.m. failures. The server-side version of the same problem is covered in log growth and service logs health. I have had that one; the log that would have explained it was the file that filled the disk.

The First Monitoring a Scheduled Job Needs

Logging records what happened; monitoring makes sure a human finds out. The minimum viable setup is three layers, in order of how little they cost.

Layer one: the exit code must reach the scheduler. Your script already exits nonzero on failure; make sure nothing swallows it. If a wrapper batch file or shell layer sits between the scheduler and the script, it must pass the status through. In a Windows .cmd wrapper, end with exit /b %ERRORLEVEL%. This is what makes the scheduler's own records — cron's mail, Task Scheduler's Last Run Result column — mean something.

Layer two: failure becomes a message. Nobody reads scheduler columns daily, so wire a notification. The simplest portable pattern is a final step that sends a short email when the run failed. Put the job name, host, and the last few log lines in the body. Ugly and effective. This is also where configuration-based tools earn their keep. Sysax FTP Automation pairs its built-in scheduler with email notifications. So a failed or successful task can mail the operators without you building the plumbing.

Layer three: watch for absence, not just failure. The sneakiest scheduled-job problem is the run that never happens. The task got disabled during maintenance, the machine was off, or the schedule was edited wrong. Or there is the run that "succeeds" while transferring nothing night after night. No failure alert will ever fire, because nothing failed. The antidote is a freshness check: something that independently asks "did today's file actually land where it should?" and alerts when the answer is no. Even a second tiny scheduled job that checks the sent folder's newest timestamp is enough to start. This absence-detection mindset is the heart of the transfer job monitoring series.

The Attended-to-Unattended Differences, Side by Side

The table condenses the whole transition into one reference you can review before any script meets any scheduler.

The First Unattended Week

Even a hardened job deserves a supervised transition, and a calm sequence beats a leap of faith.

1. Fire it through the scheduler while you watch. Use the scheduler's run-now facility, then read the entire log and check the scheduler's recorded result. This catches the environment and account problems in daylight.
2. Schedule the real time, and check the next morning. For the first few days, the morning check is part of the job: log says OK, file is in sent/, partner confirms arrival. You are doing manually what the freshness check will do forever.
3. Break it once, on purpose. Rename the source file some evening and confirm the whole failure chain works at night. Look for ERROR in the log, a nonzero result in the scheduler, and a notification received in the morning. A monitoring path that has never fired is a rumor, not a system.
4. Watch one boundary day. The first Monday (weekend gap), the first month-end (bigger file, slower export), the first patch-window night. Boundary days are where schedules meet reality.
5. Then stop checking. This is the point of the whole exercise. After a clean week including a proven failure drill, the morning ritual ends — the log, the alerts, and the freshness check have taken over. A job you still check by hand every day is not finished being scheduled.

One more thing before you file this flow under done: give a thought to what happens if a run overlaps itself. Tonight's job might still be running when tomorrow's fires, after a huge file or a slow network. For a nightly job with normal volumes the risk is low. But the fix (a lock that prevents a second copy from starting) is cheap insurance. The patterns live in the bash and cron series.

The Quiet Milestone

The first morning you realize the transfer happened without you — and that you can prove it from the log without leaving your chair — is a real milestone. It is also the last morning this job is interesting, which is the goal. The flow now runs on the business's clock instead of yours, fails loudly instead of silently, and no longer depends on anyone's memory. Record the job while the details are fresh: name, machine, schedule, account, script path, log path, one line of purpose. That entry is the seed of the estate-wide discipline in the automation inventory, and it costs two minutes now versus an afternoon of archaeology later.

From this rung, the ladder continues upward for the flows that need it. The next step, moving up to event-driven transfers, replaces the fixed clock with reaction to arriving files — worth it for some flows, overkill for many. And whichever rung each flow settles on, the placement logic in the maturity stages helps you decide deliberately rather than by momentum.

Frequently Asked Questions