Active-Active and Load-Balanced Transfer Clusters

An active-passive pair keeps the service up, but it leaves half the hardware idle and gives you no more capacity than one machine. Sooner or later someone asks the obvious question: why not let both nodes serve at once? The answer is that you can, and many estates do. But the moment two nodes accept partner connections at the same time, a set of problems appears that a pair never had to face. Files uploaded to one node must be visible on the other immediately. FTP's second connection must find its way to the right machine. Counters that lived in one process now live in several.

This article walks through an active-active design honestly: how a load balancer spreads sessions, and why SFTP behaves well behind one and FTP does not. It covers what "consistent storage" demands, which pieces of state quietly stay per-node, and what the whole arrangement costs in complexity. This article is part of our High Availability for Transfer Services series and assumes you have read active-passive failover. Everything there about identical nodes still applies, only more so.

What Active-Active Means

In an active-active cluster, every node accepts partner connections all the time. There is no standby. A load balancer is a device or service that sits in front of the nodes. It owns the address partners connect to and forwards each new connection to one of the nodes behind it. When a node fails, the balancer stops sending it connections; when a node is added, the balancer starts. Partners still see one hostname and one address, exactly as with a virtual IP. But behind that address there are now several servers working at once.

Two things follow immediately. First, the balancer is now the component whose failure takes everything down. So it must itself be a redundant pair with its own floating address. The balancer tier is covered in our article on gateway resilience. This article stays on the server tier behind it. Second, any node may receive any partner at any time. So the nodes must be interchangeable in every respect a partner can observe: host key, certificate, accounts, folders, and the files inside them. The diagram shows the shape.

How a Load Balancer Spreads Sessions

For file transfer, the balancer works at the connection level. It accepts a TCP connection from a partner on the virtual IP and opens a matching connection to one node. Then it copies bytes in both directions without understanding the protocol inside. This is called layer-4 or TCP balancing, and it is the right mode for SFTP and FTPS. The traffic is encrypted and the balancer could not read it anyway. The balancer chooses the node with a balancing algorithm. That could be round robin (take turns) or least connections (send the new session to the node with the fewest open ones). Or it could be source hash (the same partner address always maps to the same node). Least connections is the sensible default for transfers, because sessions vary enormously in length. Round robin will happily pile three multi-gigabyte uploads onto one node.

The balancer also runs the health check, and the same rule from the active-passive article applies. A check that only opens the port is a check that will keep sending partners to a node with a full disk. A configuration in the style of HAProxy, the long-standing open-source balancer, looks like this for SFTP:

frontend sftp_in
    bind 10.20.0.50:22
    mode tcp
    default_backend sftp_nodes

backend sftp_nodes
    mode tcp
    balance leastconn
    option external-check
    external-check command /usr/local/bin/check_sftp.sh   # real login + upload
    server nodeA 10.20.0.51:22 check inter 5s fall 3 rise 2
    server nodeB 10.20.0.52:22 check inter 5s fall 3 rise 2
    server nodeC 10.20.0.53:22 check inter 5s fall 3 rise 2

The frontend listens on the virtual IP. The backend lists the nodes, balances by least connections, and every five seconds runs an external script against each node. It marks the node down after three failures and up again after two successes. The script is the same real-login check described in the active-passive article. It logs in with a probe account, uploads a small file, deletes it, and returns the exit code. A balancer that runs it will pull a node with a full disk out of rotation within fifteen seconds. That happens before more than a handful of partners have noticed.

To take a node out for patching, you mark it as draining. The balancer sends no new sessions but leaves existing ones alone. Once the last session ends, the node is free. On a Tuesday afternoon that might mean waiting twenty minutes for one partner's long download to finish, and then rebooting with nobody affected. That is the mechanism that makes maintenance genuinely invisible, and it is the one thing an active-passive pair cannot do. A pair has nowhere to send new sessions while the active node is being emptied.

One consequence of TCP balancing needs saying plainly. Unless the balancer is configured to preserve the partner's address, each node sees every connection as coming from the balancer's address. Server logs lose the real client IP, and any per-address allowlist on the nodes breaks. The fixes, transparent forwarding or a protocol that passes the original address along, are described in our reverse proxies for transfers article. Decide this before go-live, not after the first audit request for "who uploaded that."

SFTP Behind a Balancer: The Easy Case

SFTP is the protocol that balances well, and the reason is structural. An SFTP session is a single TCP connection to port 22 that carries commands and file data together. The balancer picks a node once, at connection time, and everything the client does afterwards stays on that connection and therefore on that node. No second connection has to find its way anywhere. Any of the three algorithms works, and no affinity rule is needed for the protocol's sake.

Two things still bite. First, every node must present the same SSH host key. The partner's client remembers one key for sftp.example.com and will refuse a node that shows a different one. A cluster with mismatched host keys fails randomly, which is the most confusing kind of failure. Our article on host keys and known_hosts explains the client's behavior. Second, a client that opens several sessions in parallel, which many do to speed up batches of files, may land on several nodes at once. That is harmless only if every node sees the same folders and files, which brings us to storage.

FTP and FTPS: Why Data Channels Complicate Balancing

FTP uses two connections: a control connection for commands and a separate, short-lived data connection for each file or listing. In passive mode the server tells the client "connect to me on port 50017," and the client opens a second connection to that port. Behind a balancer, that second connection arrives at the virtual IP. The balancer has to deliver it to the same node that said "50017," because only that node is listening there. The full mechanics, including the announced-address problem, are in our article on FTP load balancers and proxies. So here is only what the cluster designer needs.

The requirement is session affinity, sometimes called persistence or stickiness: all connections belonging to one client session must reach the same node. There are three ways to get it, and each has a price:

• Source-address affinity. The balancer maps each client IP to a node and keeps sending it there, so the data connection follows the control connection. Simple, and it works for FTPS too because it needs no protocol awareness. It fails when many partners share one address behind their own NAT, since they all pile onto one node. It also fails when a single partner's outbound address changes mid-session.
• Protocol-aware balancing. Some balancers read the FTP control connection, see the passive reply, and set up the data forwarding themselves. This works only for plain FTP; on FTPS the control channel is encrypted and the balancer cannot read it.
• Distinct passive ranges per node. Node A announces ports 50000 to 50099. Node B announces 50100 to 50199, and node C announces 50200 to 50299. The balancer forwards each range to the matching node by port number alone. No protocol awareness is needed, FTPS works, and the announced address on every node is the virtual IP. The cost is a firewall rule set per node and a passive range that must never overlap.

The third option is the one that works in the most estates, because it turns a hard balancing problem into a static port map. It depends on each node being able to fix its own passive range and announce the shared address. That is exactly the setting described in configuring passive port ranges. A server such as Sysax Multi Server exposes the passive range and the announced address as ordinary settings. So giving each node its own hundred-port slice is configuration rather than engineering.

The Consistent-Storage Requirement

Here is the rule that decides whether active-active is possible at all: every node must see the same files at the same moment. A partner uploads to whichever node the balancer chose. The downstream job, or the partner's own download five seconds later, may go to a different node. If that node has not yet received the file, the service is lying to someone.

Work through the numbers. Node A receives a 3 GB upload; it takes four minutes. When the upload finishes, node A has the file. Suppose the nodes are kept in step by a replication job that runs every two minutes. Node B and node C receive it somewhere between a few seconds and two minutes later. In the meantime a download request that lands on node B gets "file not found." Worse, if the replication job copies the file while it is still being written, node B may hold a 1.2 GB partial for a while. In that case, a downstream job that trusts the file's presence will process a truncated file. Two minutes of replication lag is an acceptable recovery point for an active-passive pair. Replication lag is the delay between a write on one node and the same write appearing on another. For an active-active cluster, that two-minute lag is a correctness bug.

The clean solution is shared storage: one volume, on a file server or storage array, that every node mounts. So there is exactly one copy of each file and no lag. The trade-offs between shared storage and replication, including what "one volume" does to your failure list, are the subject of the next article, shared storage vs replication. For active-active, the short version is that replication with lag is not enough on its own. To use replication with lag, you must also accept that a file is only "arrived" once it exists on every node, and design the downstream jobs around that. Whatever the storage, partners should upload to a temporary name and rename on completion. That way, a partial file is never visible under its final name on any node. That pattern is in temp names and atomic renames.

State That Quietly Stays on One Node

Files are the obvious shared state. The less obvious kind is the bookkeeping each transfer server keeps in memory or in local files. That does not follow the partner from node to node:

• Failed-login counters and lockouts. A rule of "five failures then lock for fifteen minutes" is enforced per node. Across three nodes an attacker gets fifteen attempts, and a partner locked out on node A can still log in on node B. Our article on lockout and throttling design covers the choices. In a cluster the practical answer is to enforce the limit at the balancer or gateway as well.
• Per-user connection limits. "Maximum three sessions per account" becomes nine across three nodes, because each node counts only its own.
• Active session lists. "Who is connected right now" is three questions with three answers.
• Logs. A partner's activity is scattered across nodes, and reconstructing one transfer means reading three logs with synchronized clocks. Centralize them; the method is in centralizing transfer logs. A server that can log straight to a database, as Sysax Multi Server can, lets every node write to one place from the start.

None of these stop you building the cluster. They stop you assuming that settings which meant one thing on a single server still mean the same thing on three. A useful habit is to go through the server's settings page once, line by line. Ask of each item, "is this counted per node or per cluster?" Anything counted per node either gets multiplied in your head, moved to the balancer, or documented as a known difference.

Troubleshooting changes in the same way. When a partner reports "the upload failed at 14:02," the first question is now which node they were on. The answer is in the balancer's log, not the server's. Give every node a distinguishable name in its own log lines. Keep the clocks synchronized, and make the balancer log the node it chose for each connection. That way, one partner's session can be traced from the front door to the file on disk without guessing.

The Honest Complexity Price

Active-active buys three things. It buys capacity, because sessions spread across nodes, and maintenance without any interruption, because draining a node drops nobody. And it buys tolerance of a node failure with only the sessions on that node lost. It charges for them in components you now own and understand. Those include a balancer pair, shared or tightly replicated storage, and per-node passive ranges if FTP is involved. They include a real-client-address strategy, cluster-wide lockout rules, and a log pipeline. Troubleshooting gains a permanent first question, "which node?", and every drill has more moving parts. The table puts the two designs side by side.

The honest advice: build the pair if one node handles your peak comfortably and a thirty-second reconnect during maintenance is acceptable. In that case, spend the saved effort on drills. Move to active-active when a single node genuinely cannot carry the load. That is a measurement to take rather than a feeling, and our Server Capacity and Concurrency series shows how to take it. Or move when partner agreements forbid even the brief reconnect a pair imposes.

The Short Version

An active-active cluster puts a load balancer in front of several identical transfer nodes and lets all of them serve. SFTP balances easily because each session is one connection. FTP and FTPS need session affinity, most practically by giving each node its own passive port range and forwarding by port. Every node must see the same files at the same instant, which in practice means shared storage or replication designed for it. And the per-node state that a single server took for granted, lockouts, limits, and logs, has to be rethought cluster-wide. The next article, shared storage vs replication, takes up the storage decision. And making redundancy invisible to partners covers the identity every node must share.

Frequently Asked Questions