ESTABLISHED, TIME_WAIT, and CLOSE_WAIT are states a TCP connection can be in.
# Let's see a connection in ESTABLISHED state
tcpdump -i lo 'tcp port 8080 and tcp[tcpflags] & (tcp-ack | tcp-ack) != 0' -n -v
Here’s what’s happening:
- SYN: Client sends SYN.
- SYN-ACK: Server receives SYN, sends SYN-ACK.
- ACK: Client receives SYN-ACK, sends ACK.
- ESTABLISHED: Both sides acknowledge each other, data can flow.
Now, let’s imagine a scenario where the client crashes after the server has sent data but before the client has acknowledged it. The server is waiting for that final ACK to know the data arrived.
# Server's view (simplified)
# Local Address:Port Peer Address:Port State
192.168.1.10:8080 192.168.1.20:54321 ESTABLISHED
# Client's view (simplified, if it were still alive)
# Local Address:Port Peer Address:Port State
192.168.1.20:54321 192.168.1.10:8080 ESTABLISHED
The server has sent data and is now waiting for an ACK from the client. But the client is gone. The server will eventually time out and close the connection.
When a TCP connection closes, it goes through a formal handshake to ensure all data is delivered and acknowledged.
- FIN: One side (let’s say the client) decides to close. It sends a FIN packet.
- ACK: The other side (server) acknowledges the FIN.
- FIN: The server, now also ready to close, sends its own FIN.
- ACK: The client acknowledges the server’s FIN.
This is the graceful closure. However, things can get messy.
CLOSE_WAIT happens on the side that received the FIN. The application on that side needs to actually read from the socket and close it to send its own FIN. If the application is stuck, hung, or simply not reading, the connection will linger in CLOSE_WAIT.
# On the server, if the application is NOT reading from the socket
netstat -anp | grep CLOSE_WAIT
# Output might look like:
# tcp 123 0 192.168.1.10:8080 192.168.1.20:54321 CLOSE_WAIT 1234/java
TIME_WAIT happens on the side that sent the last ACK. This state is a safety net. It ensures that any delayed packets from the previous connection can be identified and discarded, preventing them from being misinterpreted as part of a new connection using the same IP addresses and ports. It lasts for a duration called the "2MSL" (Maximum Segment Lifetime), typically 30 seconds or 2 minutes, depending on the OS.
# On the client, after it sent the last ACK
netstat -anp | grep TIME_WAIT
# Output might look like:
# tcp 0 0 192.168.1.20:54321 192.168.1.10:8080 TIME_WAIT -
The key to understanding these states is that CLOSE_WAIT indicates a problem on the receiving end of a FIN (the application isn’t closing its socket), while TIME_WAIT is a normal, albeit sometimes resource-consuming, state on the sending end of the final ACK.
The most surprising thing about TIME_WAIT is that it can be a performance bottleneck if you have a very high rate of short-lived connections. While it’s designed for reliability, a server that opens and closes thousands of connections per second might find itself with a large number of sockets stuck in TIME_WAIT, consuming memory and port numbers.
Consider a web server handling many small requests. Each request might involve a new TCP connection, a few packets exchanged, and then the connection is closed. If the server is configured to aggressively close connections, it will quickly enter TIME_WAIT on its end for each of those connections.
# Example: A web server under load might look like this
netstat -an | grep TIME_WAIT | wc -l
# Output: 5500
If the server’s net.ipv4.tcp_fin_timeout (how long to wait for a FIN) is short, but net.ipv4.tcp_tw_reuse is not enabled, and the ephemeral port range is small, the server could run out of available ports for new connections. The TIME_WAIT state, while temporary, holds onto resources.
The tcp_tw_reuse kernel parameter allows new connections to reuse sockets in TIME_WAIT state if the timestamp of the new connection is newer than the connection in TIME_WAIT. This is generally safe for client-side connections or servers that don’t care about old duplicate packets.
If you see a massive number of CLOSE_WAIT states, it’s almost always an application bug – the app isn’t calling close() on its sockets. If you see a massive number of TIME_WAIT states, it could be a sign of a network churn problem or a misconfiguration around tcp_tw_reuse or the ephemeral port range, but it’s often just the natural consequence of a high-volume, short-lived connection pattern.
The next state you’ll encounter after understanding these three is often SYN_RECV, which is part of the initial connection establishment.