SSH Key Rotation and the Key Inventory

Ask an administrator "who can log into this transfer server?" and watch what happens. Password accounts they can list. But the keys — the lines sitting in authorized_keys files across every account — usually produce a pause, then an honest "I'd have to check," and then a check that raises more questions than it answers. Keys with comments like backup2 for jobs nobody remembers. Keys belonging to a contractor from three projects ago. Keys that are definitely used by something, which is exactly why nobody dares remove them.

That state has a name — key sprawl — and it is the default outcome of using SSH keys without two habits: an inventory that records who holds which key and where it grants access, and a rotation procedure that can replace any key without breaking the job that depends on it. This article builds both. You will also get the honest version of the "how often should we rotate?" question, which is asked constantly and usually answered with theater, and a procedure for rotating partner-facing keys — the scariest kind — without interrupting anyone's nightly feed.

This is part of our SSH Key Management series. It leans on vocabulary from SSH keys explained (especially fingerprints) and on the server-side file anatomy from distributing and controlling authorized_keys.

Why Keys Accumulate — and Why That Is Dangerous

Key sprawl is not carelessness; it is the natural physics of how keys work. Three properties guarantee accumulation unless something actively resists it:

• Keys have no built-in expiry. A password policy eventually forces attention. A key authorized once works forever — through job changes, project endings, and departures — until a human deliberately removes the line. Nothing ever forces that removal to happen.
• Grants are scattered, not centralized. Each server account has its own authorized_keys file. There is no master list to consult, so nobody sees the whole picture by default — the picture must be assembled.
• Setup pressure always points toward "add." When a job must run tonight, adding a key takes a minute and removal is somebody's someday problem. Every urgent onboarding deposits another line; nothing about ordinary operations ever deletes one.

Why this matters is blunt: every forgotten key is standing access with no owner. If a departed employee's laptop, an old backup, or a decommissioned server still holds a private key whose public half remains authorized, then access your organization believes is closed is in fact open. And because the grants are scattered, you cannot even quantify the exposure — you do not know what you do not know. The inventory exists to make that sentence false.

Honest Talk About Rotation

Rotation means replacing a key pair with a fresh one: generate new, authorize new, switch the client, revoke old. The common policy instinct is to copy password thinking — "rotate all keys every N days" — and it deserves honest scrutiny, because rotation has real costs (partner coordination, change windows, breakage risk) and its security value depends entirely on why you are rotating.

Here is the truthful hierarchy. Event-driven rotation is non-negotiable. When a machine holding a private key is compromised or lost, when a person with access leaves, when a key turns out to have been copied somewhere it should not be, or when a partner reports a breach on their side — the affected keys get rotated (or simply revoked) immediately, and no policy debate is needed. Calendar-driven rotation, by contrast, is mostly a fitness exercise. Rotating an uncompromised key does not make its cryptography stronger, and a schedule that exists only to satisfy a checkbox tends to produce shortcuts. The genuine value of a periodic rotation is different: it proves your inventory is accurate and your procedure works, so that on the day an event forces rotation, it is routine instead of an archaeology project. A key the team is afraid to rotate is the finding; the fear means the inventory has a hole.

The Inventory: One Row Per Key

The inventory is a table with one row per key pair. The tool is irrelevant — a spreadsheet, a page in your documentation system, a text file under version control all work. What matters is that the columns answer the questions an incident will ask, and that reality matches the table. The columns that earn their keep:

• Fingerprint — the key's identity; the join field for everything else.
• Comment — the label on the key material, which should agree with the row (disagreement is itself a finding).
• Owner — the named human, or the named job/system, responsible for the private key. "The team" is not an owner.
• Private key location — machine and account. Exactly one entry, per the one-home storage rule from earlier in this series.
• Authorized on — every server + account where the public half appears, including partner systems you do not control.
• Restrictions — the options on its authorized_keys lines (from=, forced command), so you know the containment story without logging in anywhere.
• Purpose and status — what it is for; active, in-rotation, or retired.

For keys involving partners, add the coordination metadata: their technical contact, and the fingerprint they have confirmed. This one table, kept truthful, converts every future question in this series — rotation, offboarding, audits — from investigation into lookup.

Building the Inventory From a Standing Start

Most teams start with existing sprawl, so the first inventory is assembled by discovery. Work from both ends toward the middle: servers tell you what is authorized; client machines tell you what private keys exist; logs tell you what is actually used.

Sweep the servers. On each server, fingerprint every line of every account's authorized_keys — then catch accounts living outside the usual paths:

# as root, on each server: fingerprint every authorized key, tagged by file
for f in /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys; do
  [ -f "$f" ] && echo "== $f" && ssh-keygen -lf "$f"
done

# catch nonstandard locations (service homes, relocated AuthorizedKeysFile paths)
find / -name authorized_keys -not -path "/proc/*" 2>/dev/null

# run the sweep across a small fleet from an admin workstation
for h in sftp1 sftp2 batch01; do
  echo "==== $h"
  ssh admin@"$h" 'for f in /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys; do
    [ -f "$f" ] && echo "== $f" && sudo ssh-keygen -lf "$f"; done'
done

Sweep the clients. Private keys announce themselves by their header text, so a recursive search finds them regardless of filename — including the strays in download folders that the storage rules warned about:

# list files containing private key material (any type: OPENSSH, RSA, EC...)
grep -rl "PRIVATE KEY-----" /home /root 2>/dev/null

# fingerprint each found key's public half for matching against the server sweep
ssh-keygen -y -f /home/svc-batch/.ssh/nightly-report | ssh-keygen -lf -

Ask the logs which keys are alive. Recent OpenSSH servers record the fingerprint of every accepted key, which lets you separate working keys from fossils (if your log shows no fingerprints, raise sshd's LogLevel to VERBOSE; the log file is auth.log or secure depending on the distribution, or the journal):

$ grep "Accepted publickey" /var/log/auth.log | tail -3
sshd[812]: Accepted publickey for transfer-acme from 203.0.113.40 port 52114 ssh2: ED25519 SHA256:Yq8kT3n0Wb1xR7pVd2mAzcE9fH6uJ4sLgN5oPiQwXk0

Two scoping reminders before you trust the results. First, sweep every machine that accepts SSH — the forgotten utility box and the bastion host included, since an intermediate hop holds keys and grants of its own (see bastion and jump-host transfers for why those deserve special attention). Second, remember that partner servers you connect to hold your public keys where you cannot scan; those rows come from your records and from asking partners, not from commands.

Now reconcile: every authorized fingerprint gets matched to a private key location and an owner. Whatever cannot be matched goes on an "unexplained" list — and an unexplained key with recent log activity is a security question to answer now, while an unexplained key with no activity is a removal candidate handled through the quarantine pattern in the offboarding article. Windows-based transfer infrastructure joins the same table: a server like Sysax Multi Server stores each account's public key in its configuration and records key logins in its activity log, so its accounts inventory exactly like authorized_keys entries do.

The Rotation Procedure That Never Breaks the Job

The secret to safe rotation is that it is not a swap — it is an overlap. Both keys work during the changeover, so there is no moment when the job can fail for lack of a valid credential. The sequence, for a nightly job's key:

1. Generate the replacement on the same machine, as the job account, with a fresh name and the same comment convention: ssh-keygen -t ed25519 -f ~/.ssh/nightly-report-new -C "nightly-report@batch01". Record the new fingerprint in the inventory with status "in rotation."
2. Authorize it alongside the old key — append the new public key to the server account's authorized_keys with the same restriction options as the old line (copy the restrict,from=...,command=... prefix exactly). Two lines now coexist; nothing has changed for the running job.
3. Switch the client to the new key. Point the job at the new private key path — the IdentityFile line in SSH configuration (see SSH config for transfers), the -i flag in a script, or the connection profile in a scheduling tool such as Sysax FTP Automation, where updating the profile's key path re-points every scheduled run at once.
4. Verify with evidence, not hope. Trigger a test run, then confirm in the server log that the login used the new fingerprint. The log line, not the job's green checkmark, is the proof — a job can succeed while quietly still using the old key.
5. Remove the old line from authorized_keys, update the inventory (old key retired, new key active), and after a soak period of successful runs, delete the old private key file.

The same shape scales down to a human rotating a laptop key and up to a fleet exercise. Steps 2 and 5 are the risk-free moments; step 3 is where mistakes happen, which is why step 4 demands log-level proof before anything is deleted.

Rotating Partner-Facing Keys Without Breaking Their Jobs

Partner rotations add a second organization, a communication lag, and someone else's change process — so the overlap window does the heavy lifting. Two directions to handle:

Keys partners use to reach your server. The partner generates their new pair (their private key never travels — if that rule is new, revisit the basics); they send you the new public key and quote its fingerprint through a second channel. You append it to their account beside the old key, with identical restrictions, and confirm it is live. They switch on their schedule; you watch your server's log until logins show the new fingerprint. Then — after an agreed deadline, announced in the first message, so the window cannot drift into permanence — you remove the old line. If their from= addresses are changing too, stage that the same overlapping way.

Keys you use to reach partner servers. Mirror image: you generate the replacement, send the public half plus fingerprint, and ask them to authorize it alongside your current key. When they confirm, switch your job's profile, verify a real transfer, then ask them to remove the old key and record the completion in your inventory. Rotation windows are also the natural moment to true-up the relationship record — current technical contacts, current source addresses, current restrictions — the credential lifecycle habits covered in our transfer authentication series.

Keeping the Inventory Alive

An inventory decays the moment grants happen around it, so the maintenance rules are mostly about routing:

• No grant outside the ledger. Every key added to any server gets its row first; every removal updates the row. The inventory is the change process, not a report about it.
• Reconcile on a schedule. Re-run the server sweep periodically and diff against the table. New unexplained lines mean the routing rule broke — or that someone granted themselves access, which is worth knowing urgently.
• Use last-seen evidence. Fold log fingerprints into the table as a "last used" column. Keys unused for a long stretch become review candidates — often the trailing edge of a job that was retired without telling anyone.
• Name owners, and re-confirm them. A yearly "does this key still need to exist?" question to each named owner catches purpose-drift that no scan can see.

The Version to Take Away

Keys accumulate because nothing stops them: no expiry, no central list, and every urgent day adds one more grant. The inventory — one row per key: fingerprint, owner, private-key home, authorized-where, restrictions — is what converts that entropy back into knowledge, and it is buildable in an afternoon with ssh-keygen -lf sweeps, a private-key search, and the server's own login log. Rotation is then just the overlap dance: authorize new beside old, switch, prove it in the log, remove old. Rotate immediately on events, periodically for fitness, and treat any key you are afraid to rotate as the most important finding of the exercise.

The companion skills are next door in the series: retiring keys when people and systems leave is the inventory's payoff under pressure, and host keys and known_hosts covers the other identity in every connection — your servers' own keys, which have a rotation story of their own.

Frequently Asked Questions