Simple Visibility: Dashboards and Daily Reports

Every administrator who owns transfer jobs starts the day with the same unspoken question: did everything run last night? The difference between estates is only how long it takes to answer. In the worst ones, the answer is assembled by hand — a scheduler console here, a log file there, a folder listing on a third machine — or worse, assumed, until the phone rings. In the best ones, the answer is waiting: one page, one glance, every flow accounted for, before the first coffee is finished.

That one page is what this article builds. Not a wall of graphs, not a product purchase — a daily status report and, if you want it, a simple status page, both generated from evidence your jobs already produce. We will define the one-row-per-flow format, mock up a complete morning report you can copy, show three ways to generate it from what you have, and cover the quiet superpower of visibility: spotting the duration and volume trends that predict a miss weeks before it happens. It is part of our Monitoring & Alerting series.

Alerts Cover Exceptions. Visibility Covers Everything.

If you have followed this series so far, you have alerts: freshness checks for missing files, heartbeats and exit codes for dead jobs. So why build a report at all? Because alerting and visibility answer different questions in different directions. An alert is a push: the system decides something crossed a line and interrupts you. A report or dashboard is a pull: you decide to look, and see the whole estate — including everything that did not cross a line.

That whole-estate view does three jobs alerts cannot. First, it verifies silence. Silent failures taught us that a quiet night proves nothing by itself; a report that says "14 flows, 14 OK, here are the timestamps" converts silence from an assumption into a checked fact — and steadily builds justified trust in your own automation. Second, it catches drift: the job that still succeeds but takes three times longer than spring, the feed whose file counts have been sagging for a month. No single run crosses an alert threshold; the trend is invisible except side by side. Third, it gives everyone the same picture — when the helpdesk asks "did the retail files come in?", the answer is a glance, not an investigation.

The Unit of Visibility: One Row per Flow

The design rule that keeps dashboards honest is brutal simplicity: one row per flow, one page total. A flow — one logical movement of files from a source to a destination — gets exactly one line, and that line answers the morning question for that flow. The columns that earn their space:

• Flow name — the business name, matching the one your alerts use.
• Status — one of four words. OK: last run succeeded and the outcome checks passed. WARN: succeeded with a caveat — arrived inside the grace window, duration abnormal, volume odd. FAIL: failed, or the expected file is missing. UNKNOWN: the report generator could not determine status — a stale heartbeat, an unreachable folder. Honest reporting needs that fourth state; painting unknown as green is how dashboards lie.
• Last result — timestamp of the last completed run or arrival.
• Files and bytes — the headline numbers that make hollow successes visible: an OK with files=0 stands out immediately.
• Duration — how long the run took, feeding the trend habit below.
• Next expected — when the flow is due again, so a reader also sees what should happen tonight.

Sort by severity, never alphabetically: failures first, then warnings, then the OK block. The reader's eye should meet the worst news in the first second. If your estate has more flows than fit a page, summarize the healthy majority ("41 further flows OK") — detail is for the exceptions.

Behind each status word sits a small, explicit rule set the generator applies per flow. Writing the rules down — rather than leaving them implicit in the script — is what makes the page's colors mean the same thing to everyone:

• FAIL when the last run's verdict was failure, the exit code was non-zero, or the freshness check reports the expected file missing past its grace window.
• WARN when the run succeeded but a secondary check tripped: arrival inside the grace window, duration or volume outside its normal band, or success only after multiple retries.
• UNKNOWN when evidence is missing or stale: no heartbeat within the schedule interval, an unreadable status file, a folder the generator could not reach. Unknown must be visually loud — it usually means a host, a mount, or the monitoring itself has a problem.
• OK only when there is positive, fresh evidence of success. OK is a conclusion from data, never a default.

The Morning Report, Mocked Up

Here is the whole idea as one concrete artifact — a plain-text email, generated on a schedule, sized to be read in two minutes. Adapt the shape freely; the structure is the point:

TRANSFER STATUS - Fri Mar 14 06:45          2 need attention / 14 OK

NEEDS ATTENTION
----------------------------------------------------------------------
FAIL  acme_invoices    no invoice_*.csv since Thu 22:40 (due 03:30)
      first check: partner login query - runbook: acme-invoice-feed
FAIL  hr_extract       job exited 3 at 02:12 (auth failure after retries)
      first check: service account status - runbook: hr-extract

WARNINGS
----------------------------------------------------------------------
WARN  db_extract       arrived 04:51, 36 min late but inside grace
                       third late arrival this week - sender drifting?

OK (13 flows)
----------------------------------------------------------------------
flow               last result    files   bytes     took    next due
payroll_out        05:12 OK           1   2.1 MB    0:04    Mon 05:30
warehouse_sync     04:22 OK         118   310 MB    0:19    Sat 04:15
retail_pos_in      03:05 OK          24    88 MB    0:11    Sat 03:00
prices_wk          not due        (weekly - next expected Mon 07:00)
  ... 9 more flows OK ...

TRENDS
----------------------------------------------------------------------
warehouse_sync duration +41% vs 4-week average (19 min vs 13)

generated 06:45 on OPSSRV01. If this report is missing, treat that
as an alert: the status generator itself has failed.

Read the mock the way your team will: the subject-level summary ("2 need attention") triages the day instantly; each failure line carries its first diagnostic step, exactly as the alert design article prescribes for alerts; the warning section surfaces near-misses that no alert fired for; the OK block shows counts and sizes so hollow runs cannot hide; and the last two lines quietly turn the report into a monitoring signal in its own right — more on that at the end.

Building It from What You Already Have

The report is deliberately boring to generate, because all the hard work was done by the earlier articles. Three sources, in ascending order of estate size:

• The status directory. If your jobs write heartbeat files and RESULT: verdict lines as designed in status monitoring, the generator is a script that walks one directory, parses one line per job, merges the freshness checker's findings, and formats the page. A few dozen lines in any scripting language; this covers most small estates completely.
• The server's database log. Where flows land on a transfer server you run, the server's own records can drive the report without any log parsing at all — if they are queryable. This is a place a product choice quietly pays off: Sysax Multi Server logs all activity to a file and a database, with automatic rollover, so "sessions, files, and bytes per account since midnight" is a query with rows and columns, not a text-scraping exercise with the fragilities we covered honestly in the status article. A report built on queries survives message rewording; one built on scraping does not.
• The central log store. If you have centralized your logs, the same searches you run by hand during an incident become the scheduled queries behind the report — one generator, every host covered.

In sketch form, the small-estate generator is barely more than a loop — the value is in the conventions it reads, not in clever code:

# status report generator - runs 06:40 daily, mails 06:45
for each flow in flows.conf:
    read its heartbeat file and latest RESULT line
    read the freshness checker's verdict for the flow
    apply the status rules -> OK / WARN / FAIL / UNKNOWN
    append one formatted row, plus a history line to history.csv

sort rows: FAIL, WARN, UNKNOWN, then OK
prepend summary line ("2 need attention / 14 OK")
append trend flags (duration or volume vs 4-week average)
mail the page to the team; write status.html to the intranet share

Whichever source you use, the generator itself is a scheduled job — build it with the same care as any other, per our scheduled jobs series — that runs after the overnight window and mails the result at a fixed, promised time. The fixed time matters: "the status mail arrives at 06:45" is a contract, and a missing mail becomes information. Note what the report does not replace: per-task failure notifications — the immediate mail a tool like Sysax FTP Automation sends when a scheduled task fails — still do the urgent work overnight. The report is the aggregate layer above them: the notifications say "act now," the report says "here is everything, verified."

The Status Page: A Dashboard Without a Product

Some teams want the glance available all day, not just in the morning mail. The no-product version is a static status page: the same generator writes a simple HTML table to a file on an intranet share or internal web server every fifteen minutes, one row per flow, color by status. Bookmark it; put it on the wall screen if you have one. It is not interactive, it has no history browser — and for the morning question, none of that is missed. What it must have is one non-negotiable element: a large, prominent "generated at" timestamp.

Build notes from teams who run this pattern happily: keep the page a single static HTML table with inline styles and a meta-refresh tag, so any browser on any machine renders it with no dependencies; write it to a temporary name and rename into place, so a reader never catches a half-written page (the same atomic-rename habit your transfers use); make the share read-only to everyone but the generator's service account; and give WARN and FAIL rows the same first-diagnostic-step text as the mail, because the person staring at the wall screen at lunchtime deserves the same head start as the morning reader.

Resist the urge to grow the page into a control panel. The moment it needs logins, drill-downs, and per-user views, you have left "afternoon script" territory — at that scale, evaluate a proper monitoring or dashboard system and feed it the same status data. The one-row-per-flow discipline transfers directly; the tools change, the design does not.

Trends: Seeing the Miss Before It Happens

The report's daily job is triage; its compounding value is the history it accumulates. Have the generator append one line per flow per day to a history file — flow, date, result, files, bytes, duration, arrival time — a CSV nobody reads daily and everybody blesses quarterly. Three curves in that file predict failures while they are still cheap:

• Duration creep. The nightly sync that took thirteen minutes all spring now takes nineteen, then twenty-six. Nothing fails — until the runtime crosses the delivery deadline or collides with the next job in the chain, and a mysterious "sudden" outage arrives fully explained. Concretely: a sync starting at 04:00 with a 05:30 handoff has ninety minutes of headroom; at three extra minutes of growth a week, the headroom quietly expires in a season, and the first symptom anyone sees is a missed handoff on the month's biggest file. A one-line rule in the generator — flag any flow whose duration exceeds its rolling four-week average by half — surfaces the creep months early, while the fix (a growing table, an unindexed archive folder, a saturating link) can be scheduled instead of firefought.
• Volume drift. Byte and file counts sagging on a feed that "succeeds" every night usually mean an upstream filter, a dying export, or a partner quietly changing scope — the hollow-success failure mode in slow motion. Rising volume matters too: it forecasts duration creep and disk pressure.
• Arrival slippage. Files landing later and later inside the grace window — the WARN lines accumulating in the mock above — are a sender drifting toward your deadline. Three late arrivals in a week is a friendly email to the partner now, instead of a missed payroll cutoff later.

Keep the mechanism proportionate: a threshold against a rolling average in the generator, plus a human eye on the trend section weekly, is plenty. What crosses from "trend on the report" into "alert to a person" should follow the tiering rules in alert design — most trends are digest material until the day they threaten a deadline.

Ops Visibility Is Not Audit Reporting

One boundary keeps this artifact small and sharp. The daily report serves operations: is everything running, what needs action today, what is drifting. Auditors and compliance owners need something different — evidence over long periods, user access reviews, retention proof, tamper-resistant records — and bolting those onto the morning page bloats it past the two-minute read that makes it work. Keep the audiences separate: build the operational page from this article, and build the evidence products from our reports worth automating guide, even when both happen to draw on the same underlying logs and database. Same well, different buckets.

The Two-Minute Ritual

A report nobody reads is a cron job warming a mailbox, so finish by installing the human half. Make the morning read a named duty — on rotation if you have a team — with a concrete definition of "read": failures get an owner before standup, warnings get a triage decision (act, watch, or tune the threshold), and the trend line gets thirty seconds of skepticism. Two minutes on a good day, five on a bad one. Decide the weekend rule explicitly too: either the weekend reports are read Monday morning as a batch — fine for estates whose weekend flows are low-stakes — or a duty person glances at Saturday's and Sunday's mail, for estates where a weekend miss would hurt by Monday. What kills the ritual is not workload but ambiguity about whose turn it is.

And teach the team the report's final trick: its absence is an alarm. The generator sits downstream of the status directory, the checks, and the mail path — so the 06:45 mail failing to arrive means the monitoring stack itself has a problem, exactly the class of failure that otherwise goes unnoticed longest. That makes the daily report a human-checked heartbeat for your whole monitoring system: the cheapest possible bridge into monitoring the monitoring, which is where this series goes next.

Frequently Asked Questions