Self-Signed Certificates and Private CAs Done Responsibly

Every administrator eventually generates a self-signed certificate. Sometimes deliberately — a test lab, an internal tool, an automation link between two servers in the same rack. Sometimes by default, because the transfer software offered to make one during setup and the partner demo was in an hour. Either way, you now own a certificate that no authority vouches for, clients that complain about it, and a quiet question: is this fine, or is this the thing an auditor circles in red?

The honest answer is: it depends entirely on how the trust is handled — and that is a question with concrete, checkable criteria, not a vibe. This article lays them out: what self-signed actually means and what it does not, why the click-through-the-warning habit is the real damage, a matrix of where self-signed is acceptable and on what conditions, and the graduation path — a small private certificate authority — that keeps internal TLS clean once you have more than a handful of endpoints. It is part of our Certificate Management series and leans on the chain-of-trust model from certificates explained for transfer endpoints.

What Self-Signed Actually Means

A self-signed certificate is one whose issuer and subject are the same party: you generated a key pair, wrote out a certificate binding your name to your public key, and signed it with your own private key. Structurally it is a perfectly ordinary certificate — same fields, same formats, same SAN list, same expiry window. Cryptographically it is not one bit weaker: a TLS session under a self-signed certificate uses the same ciphers and the same encryption as one under a publicly issued certificate.

What is missing is exactly one thing: the vouching. Recall the core sentence of this series — a certificate binds a name to a public key, and a third party you already trust vouches for the binding. A self-signed certificate is that sentence with the third party deleted: "I assert that I am ftp.internal.example, signed, me." No chain leads from it to any root in any client's trust store, so every validating client correctly refuses it. That refusal is not a malfunction. The client is reporting the literal truth: nobody I trust vouches for this endpoint.

Which points at the real definition, the one that decides everything else in this article: self-signed means the vouching work moves to you. A public CA's job — verifying the binding and distributing trust to clients — does not disappear; it lands on your desk. Handled deliberately (trust installed on each client, on purpose, by an administrator), self-signed can be sound engineering in the right scenarios. Handled lazily (users told to click past the warning), it is a security control being dismantled one habit at a time.

The Real Danger: Training People to Click Through

Here is the paradox worth sitting with: the damage a careless self-signed certificate does is usually not at the endpoint that carries it. The certificate on the lab box is probably fine. The damage is in the habit it installs in people.

A validation warning is the client saying "I cannot confirm who this is — possibly an attacker in the middle." When the internal file server has shown that warning every morning for a year, and the sanctioned procedure is "click Advanced, click Proceed," the warning stops meaning anything. You have run a year-long training program teaching staff that certificate warnings are noise with a dismiss button. The day a warning is real — a hostile network, a redirected connection, an actual machine-in-the-middle as described in our encryption in transit series — those users will do what they were trained to do. Click. Proceed. The attacker inherits a workforce pre-trained to wave them through, and the corrosion is not confined to your endpoint: a user taught to dismiss warnings dismisses them everywhere, including on their bank.

Automation has the same disease in script form. The first time a job fails against a self-signed endpoint, someone discovers the verification-off flag — curl -k, verify=false, "accept any certificate" — and the job goes green. That flag then lives forever, silently, doing something worse than trusting one known certificate: it trusts every certificate, including the one an attacker presents. A script with verification disabled will encrypt your files beautifully to whoever answers the connection. And unlike a human clicking through, nobody even sees it happen.

Where Self-Signed Is Acceptable: The Matrix

With that principle fixed — no warnings, no disabled verification, ever — the acceptable-use question becomes mechanical. The matrix below is the reference version:

Notice the pattern down the middle column: self-signed survives exactly where the set of clients is small, machine-operated, and administered by you. The moment the audience includes many humans, or any machine you do not manage, the trust-distribution burden either becomes unpayable or gets paid in click-through habits — and the answer flips to a CA, private or public. The bottom rows have hard edges on purpose: partner-facing and internet-facing endpoints are where impersonation attacks actually happen, and where "we emailed them the certificate, they clicked accept" does not survive contact with an incident, an auditor, or a competent attacker.

Doing Self-Signed Properly: Distribute Trust, Then Pin It

For the scenarios where self-signed is legitimate, "properly" means the certificate is treated like a real one, because it is one:

1. Generate it with real hygiene. Correct hostname in the SAN list (clients match names on self-signed certificates too), a fresh key with tight file permissions, an expiry you record. A self-signed certificate for ftps-internal.example with an empty SAN list fails name checks forever and tempts someone toward the verification-off flag on day one.
2. Install trust on each client, out of band. Copy the certificate (the public file — never the key) to each connecting system by a channel you control, and register it in that client's trust configuration: the OS trust store, the application's CA-bundle setting, or the tool's trusted-certificate list. This is pinning in its simplest form — the client trusts exactly this certificate, warnings never appear, and verification stays fully on.
3. Confirm the failure case still fails. After setting it up, present the client with a different certificate (a second lab cert works) and verify it refuses. Trust configured so broadly that anything passes is the verification-off flag wearing a costume.
4. Plan the renewal. Pinned trust means a certificate swap is a coordinated change on every client that pins it — the price of skipping the CA. Put the expiry in the inventory from certificate expiry monitoring with a note listing every client that must be updated alongside the server.

This pattern — one endpoint, few clients, trust pre-installed, verification on — is common and sound for unattended internal flows: a nightly job pushing files between two servers you run, an internal FTPS endpoint receiving batches from a known application server. Server-side, the software does not care who signed the certificate: a Windows server such as Sysax Multi Server serves FTPS and HTTPS with whatever certificate you configure, self-signed, private-CA, or public — the trust decision lives entirely with the clients, which is exactly why this article is about managing them.

Outgrowing Self-Signed: A Small Private CA

Pinning scales badly on purpose. Three endpoints times four clients is twelve trust entries, every renewal touches all of them, and the thirteenth entry is the one that gets forgotten. The moment internal TLS covers more than a handful of endpoint-client pairs, the clean structure is a private certificate authority: one self-signed root you create and guard, which signs a leaf certificate for each internal endpoint. Now clients need exactly one trust entry — your root — installed once through your management tooling, and every internal endpoint validates cleanly forever after. New endpoint? Sign a new leaf; no client changes. Renewal? Sign a new leaf; no client changes. You have rebuilt the public CA model at office scale, with yourself as the authority.

The minimal, honest version fits in a few commands. Lifetimes are deliberately left as variables: set the root long (it is the stable anchor many leaves will outlive) and leaves short (they are meant to rotate — the same logic pushing the public world toward short lifetimes applies to you):

# --- once: create the root (do this on a protected machine) ---
openssl req -x509 -new -newkey rsa:4096 \
  -keyout ca.key -out ca.crt -days "$ROOT_DAYS" \
  -subj "/CN=Example Corp Internal Root CA/O=Example Corp"
# ca.key gets a passphrase prompt - accept it; signing is rare
# and attended, so a passphrase costs nothing here. Guard ca.key;
# distribute ca.crt to managed clients as a trusted root.

# --- per endpoint: key + CSR on the endpoint itself ---
openssl req -new -newkey rsa:3072 -nodes \
  -keyout ftps-internal.key -out ftps-internal.csr \
  -subj "/CN=ftps-internal.example.local"

# --- per endpoint: sign the CSR with your root ---
# san.cnf contains:  subjectAltName=DNS:ftps-internal.example.local
openssl x509 -req -in ftps-internal.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out ftps-internal.crt -days "$LEAF_DAYS" \
  -extfile san.cnf
# install ftps-internal.key + ftps-internal.crt on the endpoint,
# with ca.crt as the chain, exactly as in the installation article

The SAN sidecar file matters: signing tools do not automatically carry names from the CSR into the certificate, and a private-CA leaf without SANs recreates the name-mismatch problem on every modern client. After signing, decode the result and read the SAN line — the same read-back habit as with any CA. Installation and outside-in verification then proceed exactly as the installation article earlier in this series describes; an internal HTTPS file drop wired this way, for example, behaves precisely like the setup described in building an internal HTTPS file drop, with no warnings anywhere.

Rules That Keep a Private CA Honest

A private CA is real authority — every machine trusting your root will believe any certificate that root signs, for any name. The discipline list is short but non-negotiable:

• Guard the root key like the master credential it is. Passphrase-protected, on a machine that is hardened and ideally offline or powered down between signings — a stolen root key lets an attacker mint valid certificates for any internal name, silently. The server-hardening habits in our hardening transfer servers series apply doubly to the box that holds it.
• Install the root only where it is needed. Managed internal machines that use internal endpoints: yes, via your deployment tooling. Partner systems, personal devices, the public: never. Asking outsiders to install your root is asking them to let you impersonate anything — a request no careful partner should grant.
• Issue narrowly and keep the ledger. Internal server names only — never public domains you do not own, never certificates "just in case." Record every issuance (name, fingerprint, expiry, endpoint owner); that ledger is your inventory feed, and the certificates it lists expire like all others, including — eventually and catastrophically if unplanned — the root itself.
• Separate issuers for separate jobs. If you also issue client certificates for mutual TLS, use a distinct issuing CA, so "certificates that identify our servers" and "certificates that grant partners access" never blur into one trust anchor.
• Know the limits. A lightweight CA has no revocation infrastructure to speak of — if a leaf key leaks, you re-sign and redeploy, and short leaf lifetimes are what keep that window small. If you find yourself needing real revocation, delegated issuance, or hardware key protection, you have outgrown the openssl-and-discipline tier and should adopt a proper internal CA platform.

When the Answer Is a Public CA After All

The final honesty check runs the other direction. The traditional case for self-signed — public certificates cost money and effort — has mostly evaporated: automated issuance (the ACME model described in getting certificates) has made publicly trusted certificates free-to-cheap and renewal-free in labor. So for any endpoint with a public DNS name, the modern default is simply a public certificate, even if today's clients are all internal. And the hard edges from the matrix stand: partners and the public never install your root and never click through — those endpoints get public certificates, full stop. The private CA's rightful territory is what remains: internal names, managed clients, and the satisfying silence of TLS that just validates.

The Version to Tell a Colleague

Self-signed means nobody vouches, so the vouching becomes your job: the encryption is identical, and the entire question is how trust reaches the clients. Done responsibly — trust pre-installed on a small set of machines you manage, verification fully on, renewal planned — it is legitimate for labs, pinned automation links, and isolated appliances. Done lazily, it trains humans to click through warnings and scripts to run with verification off, which quietly disarms the one check that stops impersonation, everywhere, for everyone trained. When internal endpoints multiply, graduate to a small private CA — one guarded root, distributed once to managed clients, signing short-lived leaves per endpoint — and give partner-facing and public endpoints public certificates without debate. The test for every setup is the same: no warnings, no bypass flags, and a failure case you have proven still fails.

From here, certificate expiry monitoring keeps every certificate this article created off the surprise list, and client certificates and mutual TLS covers the other place a private CA earns its keep — identifying the clients instead of the servers.

Frequently Asked Questions