A Firewall Troubleshooting Playbook for Transfers

"Connection timed out." That is the entire error message, and the job has been producing it since three in the morning. A transfer that fails because of a firewall almost never says so. It says "connection timed out," or "failed to retrieve directory listing," or nothing at all. The job does not finish, and the only evidence is its absence. The firewall may have logged a single line among thousands, or nothing, because dropping packets silently is what it is for. Between those two silences, a junior administrator can lose a day guessing. This playbook replaces the guessing with a fixed order of steps.

You will learn to read the three failure signatures TCP gives you before any tool is opened. You will learn to reproduce a failure from both ends and from a third vantage point. You will learn to find the one firewall log line that matters and take a packet-capture spot check that settles arguments. You will also learn to match the classic symptoms — listing hangs, timeouts after login, works inside but not outside — to causes and fixes. Each step shows the command, the output, and what the output means. The order matters more than the tools; I have watched the right tools used in the wrong order for an entire afternoon.

This is the fifth article in our Firewalls, NAT, and File Transfer series. It is the firewall-specific chapter of the general method in our troubleshooting failed transfers series. Use that series to decide that the problem is a firewall problem; use this one to find which firewall, which rule, and which fix.

Step 0: Learn the Three Signatures

Every failed TCP connection fails in one of three ways, and each points somewhere different. Learn to tell them apart before you touch anything.

• Refused — the client sent a SYN and got back a RST (reset) packet at once. Something at the destination answered "nothing is listening here." It is not a silent drop. It usually means the service is not running, listens on a different port, or a host firewall is set to reject rather than drop. It rarely means a network firewall; those drop silently.
• Timed out — the client sent a SYN, retried it several times, and heard nothing. The packet was dropped somewhere along the path, or the reply was dropped on the way back. This is the classic firewall signature, and also the signature of a missing NAT forward or a wrong announced address.
• Reset after connecting — the connection was established, worked for a while, then ended abruptly with a RST or "connection closed by peer." Something decided to end the session: an inspection device that disliked the content, an idle timeout, or an application that gave up. Firewall-related when it happens at a predictable point, such as the first encrypted command of an FTPS session.

The nc tool shows all three plainly:

$ nc -vz -w 5 198.51.100.20 22
Connection to 198.51.100.20 22 port [tcp/ssh] succeeded!

$ nc -vz -w 5 198.51.100.20 2222
nc: connect to 198.51.100.20 port 2222 (tcp) failed: Connection refused

$ nc -vz -w 5 198.51.100.20 50007
nc: connect to 198.51.100.20 port 50007 (tcp) failed: Operation timed out

Port 22 answered; port 2222 exists as a host but has nothing listening; port 50007 vanished into a firewall. Everything else in this playbook is about the third case.

Acme lost most of a day to the first signature before anyone read it. A new partner reported "connection refused" on port 22. Because the word firewall was in the ticket title, the morning went on tracing perimeter rules that turned out to be correct. Refused is rarely what a network firewall says; a network firewall says nothing. In the afternoon somebody ran ss -ltn on the server and found the SFTP service listening on port 2222. A hardening script had moved it there the week before without telling anyone. The rule was right, the partner was right, and the server had quietly moved house. The ticket template gained a first question that reads "refused, or timed out?"

Step 1: Gather the Facts

Five minutes of fact-gathering saves hours of testing. Before running anything, write down:

1. The two ends. Client address as the client sees it, and as the server sees it (the public address after NAT). Server address as the server sees it, and as the client sees it. If those four are not all known, that is your first finding.
2. Protocol, port, and mode. SFTP on 22? FTPS explicit on 21? Passive or active? The mode decides whether a passive range is involved at all.
3. Where it fails. Before login (control connection), after login (data connection), or mid-transfer (session drop). Login-then-hang means data connection; nothing else matters until that is fixed.
4. When it last worked and what changed. A firewall change, a new partner address, a server move, an OS patch that re-enabled a host firewall. Most firewall failures follow a change someone considered unrelated.
5. Who else is affected. One partner or all? One direction or both? "Only partner X" points at an allowlist; "everyone since Tuesday" points at a rule.

These are the same facts the shared connection sheet in the partner article records permanently. If you keep one, step 1 takes thirty seconds. If you do not, step 1 is four emails and a phone call.

Step 2: Reproduce From Both Sides

A firewall problem has a location, and the way to find it is to test from more than one place. Three vantage points, in order.

From the client

Test each port the protocol needs, separately, with a plain TCP connection: PowerShell's Test-NetConnection on Windows, nc elsewhere. For FTP, that means port 21 and a port from the passive range:

PS C:\> Test-NetConnection ftp.example.com -Port 21 | Select RemoteAddress,TcpTestSucceeded
RemoteAddress  TcpTestSucceeded
-------------  ----------------
203.0.113.80               True

PS C:\> Test-NetConnection ftp.example.com -Port 50007 | Select RemoteAddress,TcpTestSucceeded
WARNING: TCP connect to (203.0.113.80 : 50007) failed
203.0.113.80              False

Then run the real client with verbose output and read the negotiation. With curl, -v prints every reply, including the passive reply and the address the client then tries:

$ curl -v --user alex ftp://ftp.example.com/ 2>&1 | grep -E '227|Trying|failed'
< 227 Entering Passive Mode (203,0,113,80,195,87)
*   Trying 203.0.113.80:50007...
* connect to 203.0.113.80 port 50007 failed: Connection timed out

The 227 line answers two questions at once: is the announced address right (it is, here), and which port did the server pick (50007)? If the address were 10.0.5.20, you would stop and go to the announced-address fix in NAT types. Graphical clients show the same lines in their message log; diagnosing FTP mode failures walks through reading them.

From the server

Confirm the service is listening on the ports the firewall is supposed to deliver:

C:\> netstat -ano | findstr LISTENING | findstr ":21 :22 :990"
  TCP    0.0.0.0:21      0.0.0.0:0      LISTENING     2288
  TCP    0.0.0.0:22      0.0.0.0:0      LISTENING     2288

$ ss -ltn '( sport = :21 or sport = :22 )'
State   Recv-Q  Send-Q   Local Address:Port   Peer Address:Port
LISTEN  0       128            0.0.0.0:21          0.0.0.0:*
LISTEN  0       128            0.0.0.0:22          0.0.0.0:*

Passive ports appear only during a transfer — the server opens each on demand — so watch during a live attempt. Run the listing from the client while ss -tn on the server shows whether a connection to port 50007 ever arrives. If the server never sees it, the drop is upstream. If it sees a SYN-RECV that never completes, the reply is being dropped on the way back. That points at asymmetric routing or a NAT rule that translates one direction only. The server's own activity log gives the same answer from the application's side. A server such as Sysax Multi Server logs each session and transfer, so a login with no data connection is visible there too.

From a third place

Test from a machine that is neither the client nor the server. Examples are a workstation outside the office, a colleague at another site, or a cloud shell. This separates "works inside but not outside" from "works outside but not inside," which have opposite causes. Inside-only means the DNAT forward or perimeter rule is missing. Outside-only means the hairpin is missing. Both are rows in the symptom table below. A colleague at another site is the cheapest network tool there is.

Step 3: Read the Firewall Logs

With a time and a source address from step 2, the firewall log becomes a lookup rather than a search. You are looking for one of three things.

A deny line for the connection. The best outcome: the firewall saw the packet and refused it, and the line says why. On a Linux firewall with the logging deny described in the rule design article:

$ sudo grep XFER-DENY /var/log/kern.log | grep 'SRC=203.0.113.5' | tail -2
Mar 14 02:10:07 fw1 kernel: XFER-DENY IN=eth0 OUT=eth1 SRC=203.0.113.5 DST=10.0.5.20 PROTO=TCP SPT=49813 DPT=50007 SYN
Mar 14 02:10:08 fw1 kernel: XFER-DENY IN=eth0 OUT=eth1 SRC=203.0.113.5 DST=10.0.5.20 PROTO=TCP SPT=49813 DPT=50007 SYN

Field by field: IN=eth0 is the outside interface, and SRC is the client's public address. DST is the server (already translated by DNAT, so the forward exists). DPT=50007 is the passive port, and SYN is a new connection. Conclusion: the forward works; the filter rule for the passive range does not cover 50007. On Windows Defender Firewall the same line lives in pfirewall.log once dropped-packet logging is enabled with netsh advfirewall set allprofiles logging droppedconnections enable. After date and time, the columns are action, protocol, source, destination, source port, destination port:

02:10:07 DROP TCP 203.0.113.5 10.0.5.20 49813 50007 52 S 3141592 0 64240 - - - RECEIVE

No line at all. If the firewall logs denies on transfer ports and there is no line for the attempt, the packet never reached this firewall. Look upstream: a second NAT device, the client's own firewall, the provider. This is where the third-vantage-point test pays off. A firewall cannot deny what it never saw.

An accept line with no reply. Some firewalls log accepted new connections too. An accept followed by silence means the firewall passed the SYN and the server did not answer. That means the host firewall on the server, or the service not listening, or a reply routed around the firewall. Go back to the server-side checks.

Whatever the log says, note the rule it cites. Next-generation firewalls add the application and threat they identified. When that field says something unexpected for transfer traffic, you have found an inspection problem, covered in application helpers and ALGs. Reading transfer logs covers the server-side logs you read alongside.

Step 4: Packet-Capture Spot Checks

A packet capture is the evidence that ends debates between teams, and for firewall problems it needs only a handful of packets. The question is always the same: did the SYN arrive, and was it answered? Packets have no opinions, which is their main advantage in a meeting.

On the server (or on the firewall's inside interface), capture the client's traffic to the transfer ports and watch a listing attempt:

$ sudo tcpdump -ni eth0 'host 203.0.113.5 and (port 21 or portrange 50000-50100)'
02:10:05.201 IP 203.0.113.5.49812 > 10.0.5.20.21: Flags [S], seq 3141592, win 64240, length 0
02:10:05.201 IP 10.0.5.20.21 > 203.0.113.5.49812: Flags [S.], seq 2718281, ack 3141593, win 65160, length 0
02:10:05.225 IP 203.0.113.5.49812 > 10.0.5.20.21: Flags [.], ack 1, win 502, length 0
   ... login and PASV exchange ...
02:10:07.114 IP 203.0.113.5.49813 > 10.0.5.20.50007: Flags [S], seq 1618033, win 64240, length 0
02:10:08.130 IP 203.0.113.5.49813 > 10.0.5.20.50007: Flags [S], seq 1618033, win 64240, length 0
02:10:10.162 IP 203.0.113.5.49813 > 10.0.5.20.50007: Flags [S], seq 1618033, win 64240, length 0

The first three lines are a healthy three-way handshake on port 21: SYN ([S]), SYN-ACK ([S.]), ACK ([.]). The last three are a SYN to port 50007 repeated at one, two, and four seconds with no SYN-ACK. Here the SYN reached the server and the server did not answer. The network firewall is innocent; the host firewall or the service is the suspect. Had no SYN to 50007 appeared at all, the packet was dropped before the server, and the network firewall log is where to look. Two captures, one on each side of a firewall, bracket the drop exactly. At that point the conversation gets much shorter.

For readers of a graphical analyzer, these display filters are worth memorizing. In words, "SYN packets with no ACK flag" (tcp.flags.syn == 1 && tcp.flags.ack == 0) shows every connection attempt. "Retransmissions" (tcp.analysis.retransmission) highlights the ones that went unanswered. "Reset packets" (tcp.flags.reset == 1) finds who ended a session. "FTP passive replies" (ftp.response.code == 227) shows the announced address and port in one click. That is how you catch a helper rewriting it.

Keep captures short and specific — one host, one port range, one attempt. An hour of everything on the interface is where evidence goes to hide.

The Classic Symptom Table

Most firewall tickets fall into a dozen patterns. The table maps each to its most likely cause, the test that confirms it, and the fix.

A Worked Example

A partner reports: "Since Tuesday, our nightly pull logs in fine and then hangs on the directory listing." The server team says nothing changed. Both statements will turn out to be true. Apply the playbook.

Step 0. The partner's log shows a timeout, not a refusal. Silent drop. Step 1. Partner egress is 203.0.113.5; our server is 10.0.5.20 behind 203.0.113.80. The connection uses explicit FTPS in passive mode, range 50000–50100. It fails after login. "Tuesday" is the day the network team migrated to a new perimeter firewall. Step 2. From the partner, port 21 succeeds and port 50007 times out. From inside, both succeed. From a cloud shell, 21 succeeds, 50007 times out. Outside-only failure on the passive range: the perimeter, not the server. Step 3. The new firewall's log:

Mar 14 02:10:07 fw2 kernel: XFER-DENY IN=eth0 OUT=eth1 SRC=203.0.113.5 DST=10.0.5.20 PROTO=TCP SPT=49813 DPT=50037 SYN

Port 50037 denied. The migrated rule reads 50000-50010 — a transcription error. Ports 50000 to 50010 work, which is why the daytime tests by the network team passed. The nightly job, arriving when the server had a dozen sessions open, was handed a higher port. Step 4 was not needed, but a capture on the server would have shown no SYN to 50037 at all, confirming the drop was upstream. Fix: correct the rule to 50000-50100 and retest from the partner and the cloud shell. Then update the rule register with the ticket number and the reason. Total time, once the steps are followed in order: under an hour.

The Ten-Command Kit

Everything above, as a copyable sequence. Replace the addresses and ports with your own.

# From the client
Test-NetConnection ftp.example.com -Port 21            # control port reachable?
Test-NetConnection ftp.example.com -Port 50007         # a passive port reachable?
nc -vz -w 5 ftp.example.com 22                         # same on Linux/macOS
curl -v --user alex ftp://ftp.example.com/ 2>&1 | grep 227   # announced address and port
netstat -ano | findstr SYN_SENT                        # what is stuck, right now

# On the server
ss -ltn                                                # what is listening
ss -tn | grep 5000                                     # did a passive connection arrive?
sudo tcpdump -ni eth0 'host 203.0.113.5 and portrange 50000-50100'

# On the firewall
sudo grep XFER-DENY /var/log/kern.log | grep SRC=203.0.113.5 | tail
sudo conntrack -L | grep 203.0.113.5                   # what state does it hold?

The Short Version

Refused means something answered; timed out means something dropped; reset means something ended it. Gather the four addresses, the protocol and mode, the failure point, and what changed. Test each port separately from the client, confirm the server is listening, and test from a third place to split inside-only from outside-only. Find the deny line or the first device with no line. When teams disagree, capture on both sides of the firewall and look for the SYN with no SYN-ACK. Match the symptom to the table and apply the fix. Write the cause into the rule register so the next person starts from your answer instead of from zero. The firewall will never explain itself. The register can.

Most of the fixes lead back into the rest of this series. The announced address is in NAT types, and the helper in application helpers and ALGs. The rule and range are in rule design. The fix may belong on the partner's firewall rather than yours. In that case, the closing article, coordinating firewall changes with partners, is about getting it made without breaking anything else.

Frequently Asked Questions