The Anatomy of a Batch Night

"The morning numbers are missing." It is 07:40, the sentence comes from a director, and the true answer begins nine hours earlier with a file that never arrived. Somewhere in your organization a set of jobs runs every night while nobody watches. Partner files land, extracts leave for vendors, and long processing runs grind through the day's transactions. By morning the reports exist and the books balance. Most administrators inherit this arrangement rather than build it. I inherited mine as four sticky notes and a scheduler, and learned its shape the way everyone does, one bad morning at a time. The batch night does not introduce itself. It waits for you to notice, and it picks the moment.

This article maps a batch night end to end. We walk one realistic night hour by hour: the feeds coming in, the processing in the middle, the feeds going out. Then we draw the dependency chain hiding behind the clock times and mark the links most likely to snap. By the end you will be able to write the same map for your own night. It is the single most useful document a batch operator can own. It is also, in most shops, the one nobody has written.

This is the opening article of our Nightly Batch Ecosystems series. Later articles go deep on the cutoff times that rule the night. They go deep on mapping dependencies properly, and on recovering after a bad night. Here we build the mental model everything else stands on.

One Night, Many Systems, a Single Chain

A batch is work processed in accumulated groups rather than one item at a time. Think of the whole day's claims in one run, the whole day's payments in one file. A batch night is the stretch of hours in which that accumulated work is moved and processed. It typically runs from the evening close of business to the start of the next morning. The hours available are called the batch window: the time between the moment the day's data is complete and the moment the business needs the results. Every planning document calls it generous. No log ever has.

The word that matters most in this series is ecosystem. A batch night is almost never one system's schedule. It is several independent systems — yours, your partners', your vendors' — passing files to each other in sequence, each one consuming what an earlier step produced. A feed is one of those recurring file deliveries. Examples are the claims batch a partner sends you nightly, or the payroll extract you send a payroll bureau. Another is the settlement file a card processor drops at your door. The feeds and jobs form a dependency chain. Job B cannot start until feed A has landed, job C consumes B's output, and the morning reports sit at the far end consuming everything.

That chain structure explains the signature failure mode of batch ecosystems. Interactive systems fail loudly and locally — a user sees an error and calls. Batch ecosystems fail quietly and downstream. One late file at midnight becomes three empty tables at 04:00 and five wrong reports at 08:00. The symptom surfaces far from the cause, hours later, with the window already spent. Understanding the anatomy is how you learn to read the symptom back to the cause quickly. Better, you learn to see the failure at midnight instead of at breakfast. Breakfast is when the chain prefers to confess.

The Three Movements of Every Batch Night

Strip away the local details and nearly every batch night has the same three movements:

• Feeds in. External and internal producers deliver the night's raw material: partner files arriving on your transfer server, extracts pulled from daytime systems, files fetched from vendor servers. Each inbound feed has an expected arrival time and, usually, a deadline — a cutoff — after which the night proceeds without it.
• Processing. The long middle includes consolidation jobs that merge inbound files, and the big runs that apply the day's transactions. It includes posting jobs that update ledgers, and load jobs that fill the data warehouse. This is where most of the window's hours are spent, and where the chain is at its most sequential — each run consuming the previous run's output.
• Feeds out. The results leave: report files to the people who asked for them, and extracts to vendors and partners. Print files go to a mail house, and regulatory submissions go to whoever regulates you. Outbound feeds have cutoffs too — external ones, owned by other organizations, and usually harder than your internal deadlines.

Two layers run inside those movements, and they fail differently and are fixed by different tools. The transfer layer moves files between systems — the SFTP downloads, the uploads to a vendor, the copy from the staging server to the processing host. The processing layer transforms data — the adjudication run, the ledger posting. A batch night dies just as dead from a transfer that silently didn't happen as from a job that crashed. But the transfer failure is quieter: nothing errors, a file is just not there. That asymmetry is why the transfer layer deserves its own monitoring, a theme we return to below. Absence has never once raised a ticket about itself.

A Worked Night: Harborview Mutual

A concrete night teaches more than any abstraction, so here is one. Harborview Mutual is an invented mid-sized insurer, but its night is assembled from patterns you will recognize anywhere. Partner files come in, a long core run sits in the middle, vendor files go out, and reports appear at dawn. Three claims-processing partners send nightly claims batches; a card processor sends a settlement file; the bank sends a lockbox file of the day's mailed-in payments. In the middle sits the adjudication run — the core system deciding what each claim pays — followed by ledger posting and a warehouse load. Out the far side go a print file to a mail vendor with a hard 05:30 cutoff, and the morning reports the executives read at 07:30.

Here is the night sheet — the one-page timeline Harborview's operators keep. Every time is local; filenames carry the date as a YYYYMMDD token, following the sortable-naming conventions covered in datestamp formats that sort:

HARBORVIEW MUTUAL — BATCH NIGHT SHEET (all times local)

19:30  Policy admin end-of-day close          the night's data is now complete
20:05  policy_extract_YYYYMMDD.dat staged     internal extract, feeds warehouse
21:00  Claims window opens                    3 partners, cutoff 23:00
22:15  settle_YYYYMMDD.csv arrives            card processor settlement file
23:00  CLAIMS CUTOFF                          consolidation runs with what's in
23:30  Claims consolidation                   merges partner batches
00:15  Adjudication run starts                the long job: ~2h and growing
01:05  lockbox_YYYYMMDD.txt arrives           bank file; cash application follows
02:20  Adjudication ends -> GL posting        ledger consumes adjudication output
03:10  Warehouse load                         needs GL close + policy extract
04:35  Report generation                      morning pack + print file build
05:05  print_YYYYMMDD.zip sent to mail vendor HARD CUTOFF 05:30 (their presort)
06:10  Operator checks, exceptions review     first human eyes on the night
07:30  Morning reports distributed            the visible end of the night

Hour by Hour: How the Night Unfolds

The evening: raw material arrives (19:30–23:00)

The night begins when the day ends: at 19:30 the policy administration system closes out the business day, freezing the data the night will process. Shortly after, a scheduled task extracts the day's policy changes to a staging folder. Then the waiting starts. The three claims partners deliver into Harborview's transfer server on their own schedules. One lands reliably at 21:10, one drifts between 22:00 and 22:50, one is anyone's guess. The settlement file arrives at 22:15 with the punctuality you would expect from a payment processor. Nothing here is compute-heavy; the evening is a transfer-layer act, and its main risk is absence — a partner file that never comes. The evening is mostly the night waiting by the door.

The midnight turn: cutoff and the long run (23:00–02:30)

At 23:00 the claims cutoff falls. Whatever partner batches have arrived — and passed their completeness checks — get consolidated into one input. Ideally, those checks use marker and control files so a half-uploaded batch is never mistaken for a whole one. A partner who misses the cutoff is processed tomorrow. That unsentimental rule is what makes the rest of the night schedulable, and the reasoning behind it fills our cutoffs article. At 00:15 the adjudication run starts: the heaviest job of the night. It is currently about two hours and growing a little every quarter as claim volume rises. While it grinds, the bank's lockbox file arrives on a separate small branch of the chain and cash application posts the day's mailed payments.

The deep night: posting and loading (02:30–05:00)

Adjudication's output — what was paid, denied, adjusted — flows into general ledger posting, because the finance system must reflect the night's decisions before anything downstream reads balances. Once the ledger closes, the warehouse load begins, joining the ledger data with the policy extract staged back at 20:05. Notice the quiet trap: the warehouse load depends on a file produced seven hours earlier. If the 20:05 extract had failed, nothing would have complained until 03:10 — the failure would have aged silently for hours. I have made that walk backward, from a morning report to an evening extract, and it took until lunch. This is why absence-detection matters more in batch ecosystems than anywhere else, and why freshness checks on expected files are the cheapest insurance the night can buy.

The morning edge: outputs under deadline (05:00–07:30)

The last movement runs closest to the cliff. Report generation builds the morning pack and the print file of customer letters. The print file must reach the mail vendor by 05:30 — the vendor's presort deadline, entirely outside Harborview's control. Miss it and every letter slips a day, which for some regulated notices means a compliance clock starts ticking. At 06:10 the first human looks at the night: exceptions reviewed, alerts triaged. By 07:30 the reports are in inboxes, and the whole apparatus goes back to sleep. The window, measured honestly, ran from 19:30 to 05:30 — ten hours, of which the chain used nearly all. Nearly all is the batch night's idea of a comfortable margin.

The diagram below draws this night as a picture. Inbound feeds are on the top lane, the processing chain is in the middle, and outbound feeds are at the bottom. The two cutoffs are marked as vertical lines and the fragile links dashed in red.

The Chain Behind the Clock

Now read the night sheet again and notice what it does not say. It looks like a schedule — a list of clock times — but the clock times are an illusion of convenience. The real structure is the chain. Consolidation runs at 23:30 because the claims cutoff is 23:00. Adjudication starts at 00:15 because consolidation takes about forty minutes. The warehouse load sits at 03:10 because ledger posting usually ends around 03:05. Every time on the sheet after the first one is really a dependency wearing a clock costume.

This matters because schedules and chains fail differently. Everything may be triggered purely by clock. That is the common arrangement, since most shops grew up on schedulers like the ones described in Task Scheduler for transfers. In that case, a slow night does not shift the chain, it shears it. Adjudication overruns to 02:50. Ledger posting started at its usual 02:25 anyway against yesterday's data. The morning numbers are quietly wrong rather than loudly late. Wrong-but-on-time is the worst outcome a batch night can produce, and clock-coupled chains manufacture it, punctually. The alternatives — jobs that check for their inputs before running, arrival-triggered steps, explicit handoff markers — are the subject of much of this series.

Meridian Parts learned this from a quarter-end. Their inventory valuation run overran by forty minutes. The stock posting job started at its fixed clock time anyway and read the previous day's file. The morning valuation report arrived on time, to the minute, one day stale. A reconciliation total on the finance team's checklist caught it before the report went anywhere. The fix was one line: the posting job now refuses to start unless its input carries today's date token. The next quarter-end it started forty minutes late and nobody noticed, which was the point.

The Fragile Links

Every batch ecosystem has the same short list of places where nights actually break. On the Harborview sheet they are marked in red; in yours they will be somewhere close by.

• External arrivals. The drifting claims partner is the classic: an input owned by another organization, on another schedule, with another set of failure modes. You cannot make a partner punctual; you can only detect lateness early and design the cutoff honestly. Absence is silent by default — a monitoring gap explored in why jobs fail silently. So the expected-file check that pages someone at 22:30, not 06:00, is the fix.
• The long job in the middle. Adjudication is Harborview's pacing item: the single longest run, sitting on the critical path, growing with business volume. Whatever your equivalent is, its duration trend is the future of your whole night — the theme of the shrinking window.
• Hard outbound cutoffs. The 05:30 print deadline belongs to the vendor, not to Harborview. Internal deadlines flex under pressure; external ones do not. The gap between the chain's normal finish and the external cutoff is the night's total slack, and it is usually smaller than anyone admits.
• Handoffs between layers. A transfer hands a file to a job, or a job hands output to a transfer. Every such seam is a place where a partial file, a wrong name, or a timing mismatch can slip through. The staged extract consumed seven hours later is the archetype.

The transfer layer's contribution to fragility is the most fixable, because its needs are simple and mechanical. It needs transfers that run on schedule, notice their own failures, retry transient errors, and tell a human when retries run out. That is the job of a dedicated automation client — Sysax FTP Automation is built for exactly this slot. It runs scheduled transfer tasks with retry, error handling, and email notification. Its folder monitoring can make a step arrival-triggered instead of clock-guessed. The consolidation staging job then fires when the partner file actually lands, not when the schedule hopes it has.

Writing Your Own Night Sheet

The exercise that turns this article into an asset is writing the night sheet for your own ecosystem. It takes an evening of log reading and a week of observation, and it pays for itself the first bad morning. The method:

1. Pick one morning deliverable — the report or file whose absence would generate the angriest call — and work backward. Find what job builds it, what inputs that job reads, and what produces each input. Continue until you reach the systems where the day's data originates.
2. Gather the evidence. Three sources tell you when things really happen: the scheduler's task history, the processing jobs' own logs, and your file transfer server's activity log. The transfer log is the underrated one — it is the record of what landed and left, timestamped, independent of what any job believed. A server that logs to both file and database, as Sysax Multi Server does, gives you a queryable version of the night. It records every upload from every partner, every download by every job, with times you can sort and compare across weeks.
3. Record one full week. For each feed and job: earliest, typical, and latest observed time. One night lies; seven nights sketch the distribution. Note which arrivals drift and by how much.
4. Mark the deadlines. Which times are cutoffs owned by outsiders? Which are internal habits that merely look like deadlines? Write the owner next to each.
5. Mark the dependencies you can see — job B reads job A's output — and flag the ones you are guessing about. The guesses become the work list for proper dependency mapping.

Keep the result to one page, in plain text, next to the on-call runbook. Its audience is a person at 03:00 with a phone in one hand; density beats polish. Nobody has ever admired a font at 03:00.

The Night as a System

A batch night is a relay race run in the dark by teams that have never met. Feeds come in, processing runs through, and feeds go out, every runner taking the baton from the one before. Its schedule is a flattened dependency chain; its failures start small and surface late. Its fragile links cluster at external arrivals, the long middle run, and the hard outbound deadlines. Map it once — honestly, from logs, on one page — and every future incident starts with orientation instead of archaeology. The 07:40 question still comes. The answer is now on one page.

From here, the series follows the anatomy into its pressure points. Cutoff Times: The Deadlines That Rule the Night examines the deadlines that shape every batch schedule. Mapping Batch Dependencies turns the guessed chain into a known one. Catch-Up is the playbook for the morning the chain breaks anyway.

Frequently Asked Questions