Connection reset by peer — ECONNRESET in a Go, Node, Python or Java stack trace — means exactly one thing at the transport layer: a TCP RST segment arrived on a connection that was already established. It does not say who sent it, and every host on the path can send one. That is why the usual response — raising a timeout, adding a retry — so often changes nothing. This guide gives you a capture-and-classify procedure instead: localize the RST to the client, a middlebox, the server kernel, or the server application, confirm it against a kernel counter that names the cause, and only then change a setting.
What you’ll accomplish
- Read a TCP RST correctly, including the distinction between an RST that answers a SYN (
ECONNREFUSED) and one that lands mid-connection (ECONNRESET). - Capture on both ends simultaneously and determine which host emitted the reset, using the position in the flow and the TTL of the reset packet.
- Confirm the cause with named kernel counters —
ListenOverflows,ListenDrops,TCPBacklogDrop,TCPAbortOnClose,TCPAbortOnData,TCPAbortOnMemory— rather than by elimination. - Recognize the connection-pool versus idle-timeout mismatch that produces intermittent resets in front of nginx, HAProxy and managed database endpoints, and size the pool so it stops.
- Apply the fix that belongs to the cause you found, and prove it held.
Prerequisites
- Root or
sudoon at least the server, ideally on the client too. A single-ended capture can tell you a RST arrived; only a both-ended capture tells you where it was born. tcpdump, plusiproute2forssandnstat. On Debian and Ubuntunstatis in theiproute2package; on RHEL, Alma and Rocky it lives iniproutealongsidess.- A Linux kernel 4.x or newer. Counter names have been stable for years, but
net.core.somaxconnchanged its default from 128 to 4096 in Linux 5.4 — assume 128 until you have read the value on the host itself. - Knowing the full path in advance: every proxy, NAT gateway, sidecar and load balancer between the two processes. A reset you cannot explain is usually a hop you forgot about.
- If the traffic is TLS, a way to correlate the encrypted stream with application logs, since you will not be reading payloads. Request IDs work well.
How it works
A RST is the TCP abort primitive. Unlike FIN, which closes one direction and waits, RST tears the connection down immediately in both directions and discards anything still in flight. The receiving kernel drops the socket and the next read or write in the application returns ECONNRESET. Nothing is negotiated, so the segment carries no reason code — the reason has to be reconstructed from where and when it appeared.
The first split is what state the connection was in. An RST in response to a SYN means no socket was listening on that port; the application sees ECONNREFUSED, and the problem is a dead service, a wrong port or a firewall configured to reject rather than drop. An RST after the three-way handshake completed is the ECONNRESET case, and it has four realistic origins that behave quite differently.
Two of those origins are worth naming precisely, because they are easily confused. When an application calls close() while unread data is still sitting in the socket’s receive queue, the kernel cannot honour a graceful close — the peer sent bytes nobody will consume — so it sends an RST and increments TCPAbortOnClose. Separately, a socket closed with SO_LINGER set and a zero linger time deliberately skips the FIN exchange and resets immediately; that path is accounted under TCPAbortOnData. Both look identical to the client. Only the counters separate them.
The third origin, the accept queue, produces the most confusing symptom, because whether you get a reset or a stall depends on a single sysctl.
Step-by-step
1. Classify the error before capturing anything. Separate refused from reset, because they are different investigations. From the client host:
curl -sS -o /dev/null -w '%{http_code}\n' http://app.internal:8080/healthz
Connection refused means nothing is listening — check the service and the bind address with ss -lntp | grep 8080. Recv failure: Connection reset by peer, or a connect followed by a reset partway through, is the case this guide covers. Note also whether the failure hits every request or only some: reproducible points at the server or the listener, intermittent at a pool or a timeout.
2. Capture on both ends at the same time. Run this on the server and, in parallel, the same command on the client with the peer address swapped. Keep the snaplen small so you record headers without payloads, and write to a file rather than reading live — you want to replay the same capture with different filters.
sudo tcpdump -nn -i any -s 128 -w /tmp/reset-server.pcap 'host 10.20.0.14 and tcp port 8080'
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 128 bytes
Reproduce the failure, stop the capture with Ctrl-C, and check the packet count it reports. For the full filter syntax on a busy host, see capturing and filtering traffic with tcpdump.
3. Find the resets and read their direction. Filter the saved file for the RST flag. The tcp[tcpflags] form works for IPv4 only, because libpcap cannot compute a variable TCP offset behind IPv6 headers; for IPv6 without extension headers, filter the flags byte directly at offset 53.
tcpdump -nn -r /tmp/reset-server.pcap -v 'tcp[tcpflags] & tcp-rst != 0'
tcpdump -nn -r /tmp/reset-server.pcap -v 'ip6[53] & 4 != 0'
Read three things from each hit: the source address, the position in the flow, and the TTL. If the RST’s source is the server and you captured it on the server, the server genuinely sent it — go to step 4. If both captures show the RST arriving inbound, neither endpoint sent it and something in the middle injected it — go to step 6. The TTL corroborates that: a forged reset usually has a different TTL from the ordinary data packets of the same peer, because it was born fewer hops away.
4. Ask the kernel to name the cause. Counters are the fastest way to turn a guess into a fact. Take a baseline, reproduce, then read the delta:
nstat -n
# reproduce the failure now
nstat | grep -E 'ListenOverflows|ListenDrops|BacklogDrop|AbortOn|OutRsts|AttemptFails|EstabResets'
nstat -n writes the current values to its history file and prints nothing; the second call prints only what moved since. Interpret the deltas as follows:
TcpExtListenOverflowsandTcpExtListenDrops— the accept queue was full when a handshake completed. Step 5.TcpExtTCPBacklogDrop— packets discarded because a socket’s backlog was full while userspace held the socket lock; a sign of an application that is not reading fast enough.TcpExtTCPAbortOnClose— the application calledclose()with unread data in the receive queue. Step 7.TcpExtTCPAbortOnData— includes theSO_LINGERzero-timeout close path.TcpExtTCPAbortOnMemory— orphaned sockets exceedednet.ipv4.tcp_memortcp_max_orphans; the kernel killed connections to reclaim memory.TcpOutRsts— total resets this host emitted. If this stays flat while your application logs resets, your server is not the sender.
5. Check the accept queue if the resets cluster at connection setup. The queues in the second diagram are separate and separately sized. The SYN queue holds half-open handshakes, bounded by net.ipv4.tcp_max_syn_backlog; the accept queue holds completed connections waiting for accept(), bounded by the smaller of net.core.somaxconn and the backlog the program passed to listen(). Read both the current depth and the ceiling:
ss -lnt 'sport = :8080'
For a listening socket, Recv-Q is the number of established connections waiting to be accepted and Send-Q is the ceiling. A Recv-Q sitting at or near Send-Q under load is the diagnosis. Confirm the ceiling actually in force:
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog net.ipv4.tcp_abort_on_overflow
Raising somaxconn alone changes nothing unless the application also asks for a larger backlog — nginx needs listen 8080 backlog=4096;, gunicorn takes --backlog, Node’s server.listen() defaults to 511, and a bare Java ServerSocket defaults to 50. A full SYN queue is different again and announces itself in the kernel log:
sudo dmesg -T | grep -i 'possible SYN flooding'
6. Test the idle-timeout hypothesis when resets are intermittent. This is the most common cause behind a healthy-looking service that resets a few requests an hour. A pooled connection sits idle; a firewall, NAT gateway or load balancer expires its state entry; the next request on that connection gets an injected RST or vanishes. The tell is that failures correlate with idle time rather than load, and the failing request is always the first one after a quiet period. Instrument the age of the connection at failure time if you can; otherwise compare the timeouts directly. On the middle hop, if it is a Linux NAT box:
sudo sysctl net.netfilter.nf_conntrack_tcp_timeout_established
cat /proc/sys/net/netfilter/nf_conntrack_count /proc/sys/net/netfilter/nf_conntrack_max
The rule to enforce is one-directional: the client pool’s idle timeout must be strictly shorter than the shortest idle timeout anywhere along the path. Whoever closes first should be the pool, because it closes gracefully and knows to open a new connection. For nginx talking to an upstream, keep-alive is off unless you configure all three of these together:
upstream app {
server 10.20.0.14:8080;
keepalive 32;
keepalive_timeout 55s;
}
server {
location / {
proxy_pass http://app;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
}
sudo nginx -t && sudo systemctl reload nginx
nginx: configuration file /etc/nginx/nginx.conf test is successful
HAProxy’s equivalents are timeout http-keep-alive and timeout server; client pools have their own knob — IdleConnTimeout in Go’s http.Transport, maxLifetime in HikariCP, keepAliveTimeout for Node’s agent. Set whichever applies below the network timeout, not above it.
7. Apply the fix for the cause you confirmed. Accept-queue overflow needs a larger backlog and a faster accept() — more workers, less time spent holding the accept loop. Do not reach for tcp_abort_on_overflow=1; it converts a stall into a hard reset, which is useful for a few minutes of diagnosis and harmful as a standing setting. TCPAbortOnClose is an application bug: read the request body to completion, or shut down the read side explicitly, before closing. TCPAbortOnMemory means socket memory is running out, usually because orphaned sockets are accumulating. Middlebox resets are fixed by timeout alignment, never by retry logic alone.
Validate it works
Re-run the counter baseline and drive the same load that used to fail. The counters that were moving should now be flat:
nstat -n
# run the reproduction load again
nstat | grep -E 'ListenOverflows|ListenDrops|AbortOn'
No output from that grep means none of those counters moved, which is the success signal. Then confirm the accept queue is draining rather than filling under load:
watch -n1 "ss -lnt 'sport = :8080'"
Recv-Q should hover near zero and never approach Send-Q. For the idle-timeout fix, the honest test is time-based: hold a pooled connection idle for longer than the previous failure threshold, then issue a request on it. A loop that sends one request every two minutes for half an hour is enough:
for i in $(seq 1 15); do curl -sS -o /dev/null -w "%{http_code} %{time_total}\n" http://app.internal:8080/healthz; sleep 120; done
Every line should show 200 and a stable time. Finally, leave a reset-filtered capture running on the server for an hour and confirm it stays empty.
Troubleshooting
Both captures show the RST as inbound, but you have no access to the middle hop. Compare TTLs again, then compare the reset’s sequence number to what the sender had actually sent. An in-path device resets with a sequence number derived from the packet it just saw; an out-of-path injector often guesses. Either way, the evidence is enough to hand to whoever operates the hop.
The counters move but you cannot tell which socket. nstat is host-wide. Narrow it with per-socket state instead: ss -tmi state established '( dport = :8080 or sport = :8080 )' prints send and receive queue sizes, retransmits and RTT per connection, which will point at one peer.
Resets appear only under load, and the accept queue looks fine. Look at file descriptors before anything else. A process that hits its limit fails inside accept() and stops draining the queue, so the queue fills for a reason that has nothing to do with its size — see too many open files and ulimit. Ephemeral port exhaustion on a busy proxy produces a similar pattern; check net.ipv4.ip_local_port_range and the count of sockets in TIME-WAIT.
The application reports an unexpected EOF rather than a reset. That is a different failure. A clean TLS shutdown sends close_notify and then FIN, and the peer sees end-of-file; a truncated stream with no close_notify and no RST usually means a proxy gave up mid-response. Resets and EOFs need different fixes, so check the capture before assuming.
tcpdump shows nothing when the failure happens. Confirm the interface. -i any on Linux is a cooked-mode pseudo-interface and misses nothing, but a specific interface will miss loopback or container traffic. For container workloads, capture inside the network namespace, not on the host bridge.
When this is the wrong approach
If resets started the moment you deployed, skip all of this and read the diff. A new SO_LINGER option, a changed shutdown handler, or a health check that closes connections early produces textbook ECONNRESET and is far cheaper to find in the code than on the wire.
Packet capture is also the wrong first tool when the reset is a deliberate policy action. A firewall set to reject rather than drop, an IDS resetting on an SNI it dislikes, or a rate limiter tearing down abusive connections are all working as configured; the fix is a policy change, and kernel tuning will not help.
Finally, if you are tuning somaxconn because one instance cannot keep up with accept(), queue sizing is the wrong lever. A larger queue makes a saturated service fail later and with worse latency instead of failing fast. Add capacity or shed load.
Worth keeping open while you work: the kernel TCP sysctl reference for exact semantics of every value above, and the tcp(7) man page for how the socket API maps onto those states.
If this is the kind of work you’d rather never own again, it’s the daily job of Managed Cloud Ops — book an infrastructure review.