If you have ever stared at a monitoring dashboard during a DDoS attack, you know the sinking feeling of seeing thousands of connections stuck in a pending state. I remember a specific incident while working for a major fintech client in their Bangalore data center. Their load balancers started dropping legitimate traffic because the connection tables were completely saturated with half-open sessions. This was not a sophisticated application layer attack. It was a basic exploitation of the 3-way handshake. Understanding this process is not just for passing certifications. It is a fundamental skill for anyone responsible for defending enterprise infrastructure.
What is a 3-way handshake?
A 3-way handshake is the method Transmission Control Protocol (TCP) uses to establish a reliable connection between a client and a server. It ensures that both ends are ready to transmit data and have agreed on the initial sequence numbers. Without this process, your network traffic would be a chaotic mess of unordered packets with no way to verify if the recipient actually received them. TCP is a connection-oriented protocol that works on top of basic networking concepts. If you are new to networking and security fundamentals, start with this network security basics guide: Network Security Basics Guide
How does a 3-way handshake work?
The mechanism works through a precise exchange of packets that synchronize the state of two different machines. This is necessary because IP networks are inherently unreliable. Packets can arrive out of order, get duplicated, or disappear entirely. The 3-way handshake establishes a baseline of trust and tracking. Each side generates a random number to start their count. By acknowledging these numbers, both the client and the server confirm that the communication path is functional in both directions. It prevents old, delayed packets from accidentally opening new sessions and ensures that both parties have the resources available to maintain the connection.
Technical Flow and Data Exchange
The technical flow starts when an application, such as a web browser, requests a connection to a specific IP and port. The operating system kernel takes over and initiates the sequence.
First, the client sends a SYN packet. This is the first step of the 3-way handshake. The client chooses an Initial Sequence Number (ISN), let us call it $X$. At this point, the client enters the SYN-SENT state. It is waiting for the server to prove it is listening.
Second, the server receives the SYN. If the port is open and the server has capacity, it responds with a SYN+ACK. This packet does two things. It acknowledges the client’s sequence number by sending back $X+1$. It also sends its own random Initial Sequence Number, which we will call $Y$. Now, the server is in the SYN_RECV state. This is a critical state for security analysts to monitor. The server has now committed a small amount of memory to track this “embryonic” connection.
Third, the client receives the SYN+ACK and sends the final ACK. The client increments the server’s sequence number and sends back $Y+1$. The client’s own sequence number is now $X+1$. Once the server receives this packet, the connection moves to the ESTABLISHED state. Data transfer can finally begin.
Key Components of the Handshake
To understand why a connection is failing, you have to look at the specific parts of the TCP header. These fields tell you exactly what the protocol is trying to achieve at any given millisecond.
- SYN Flag: Short for Synchronize, this bit is set only in the first two packets of the handshake to coordinate sequence numbers.
- ACK Flag: The Acknowledgment bit indicates that the Acknowledgment Number field in the header is valid and should be read by the receiver.
- Sequence Number: A 32-bit value that tracks the byte order of the data being sent so the receiver can reassemble it correctly.
- Acknowledgment Number: This field tells the other party the next sequence number the sender expects to receive.
- Window Size: This informs the sender how many bytes the receiver is willing to accept before an acknowledgment is required.
- Maximum Segment Size (MSS): A TCP option negotiated during the handshake that defines the largest chunk of data a device can handle in a single packet.
Real-World Log Example
When you are troubleshooting, you will often use netstat or ss to see what is happening on the server. In an enterprise environment, a healthy server should have many ESTABLISHED connections and a relatively small number of SYN_RECV entries.
Plaintext
[admin@prod-web-01 ~]# netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 10.10.50.20:443 192.168.1.55:54321 ESTABLISHED 1204/nginx
tcp 0 0 10.10.50.20:443 172.16.10.12:33211 SYN_RECV –
tcp 0 0 10.10.50.20:443 172.16.10.15:44212 SYN_RECV –
tcp 0 0 10.10.50.20:443 172.16.10.18:55213 SYN_RECV –
In this output, you see three connections in the SYN_RECV state from different IPs in the same subnet. If you see hundreds of these from random external IPs, you are likely looking at a SYN flood attack. The server has sent its SYN+ACK and is waiting for the final ACK that will never come.
Practical Implementation of Network Security Controls
If you are setting up a new environment, you need to harden the TCP stack. This is how you prevent a simple 3-way handshake from becoming a point of failure for your entire service.
- Enable SYN Cookies in the kernel by editing /etc/sysctl.conf and setting net.ipv4.tcp_syncookies = 1.
- Increase the tcp_max_syn_backlog value to allow the server to handle more simultaneous half-open connections during traffic spikes.
- Reduce the tcp_synack_retries setting to ensure the server does not waste time re-sending SYN+ACK packets to dead or malicious clients.
- Configure your edge firewall to perform TCP Intercept, where the firewall completes the handshake with the client before passing the connection to the server.
- Set up rate limiting on your load balancer to drop excessive SYN packets coming from a single source IP address.
- Monitor for unusual patterns in the TCP Window Size, as very small windows can indicate a Sockstress attack trying to consume server resources.
Advantages and Limitations
The 3-way handshake is excellent for reliability. It ensures that both parties are synchronized before data moves. However, this reliability comes with a cost. The most obvious limitation is latency. Every new connection requires a full round-trip of packets before data can be sent. In high-latency environments, like satellite links or poorly routed international traffic between Dubai and New York, this adds significant delay.
Another limitation is the resource requirement. The server must remember the state of every connection attempt. This makes the protocol vulnerable to resource exhaustion by design. If an attacker can send SYNs faster than the server can time them out, the server will eventually run out of memory for its connection table. This is a fundamental trade-off of the TCP state machine.
Common Mistakes in Analysis
This is where most people get confused: they assume a RST (Reset) packet always means an attack is happening. In reality, a RST often indicates a simple configuration error. For example, if a client sends a SYN to a port that is not listening, the server will respond with a RST to tell the client to stop trying.
Another frequent error is ignoring sequence number randomization. Some junior analysts try to predict sequence numbers during manual packet injection tests. Modern operating systems use cryptographically secure random number generators for these values. If you see sequence numbers that look predictable (like incrementing by exactly 100 every time), you are likely dealing with an extremely old or misconfigured embedded device that is highly vulnerable to session hijacking.
The Security Analyst Perspective on the 3-way handshake
For a security analyst, the 3-way handshake is a telemetry source. You should be looking for anomalies in the flags. For instance, a packet with both the SYN and FIN flags set is illegal in standard TCP. Attackers use these “Christmas Tree” packets to bypass poorly configured firewalls or to fingerprint the operating system of a target.
In real-world enterprise environments, you must also consider the impact of Middleboxes. Firewalls, Proxies, and Load Balancers often modify the handshake. They might strip out certain TCP options or rewrite sequence numbers. If you are capturing traffic on the server and it does not match what the client claims to be sending, check the devices in between.
Troubleshooting Scenario: The “Silent Drop”
Imagine you have a user in a branch office in Riyadh trying to access a core banking application in Mumbai. The user reports that the application “just hangs.” You run a packet capture and see the client sending a SYN. You see the server receiving the SYN and sending a SYN+ACK. But the client never sends the final ACK. Instead, it just keeps re-sending the SYN.
The wrong assumption here is that the server is down. The server is clearly responding. The actual fix often lies in the network path. In real environments, it doesn’t work this cleanly because of MTU (Maximum Transmission Unit) mismatches or firewalls that drop packets with specific TCP options. In this case, a firewall in the middle was likely dropping the SYN+ACK because it was too large or contained a SACK (Selective Acknowledgment) option the firewall did not recognize. The fix was to adjust the MSS (Maximum Segment Size) on the router to ensure the packets could pass through the tunnel without fragmentation.
Interview Questions for Junior Engineers
Q: What happens if the final ACK in the 3-way handshake is lost?
A: The server will stay in the SYN_RECV state and eventually re-transmit the SYN+ACK packet. If the client tries to send data, the server will ignore it because it does not consider the connection established yet.
Q: Why can we not use a 2-way handshake instead?
A: A 2-way handshake cannot prevent old, delayed connection requests from creating “ghost” connections. The third step allows the client to verify that the server’s response is actually for the current request.
Q: How does a SYN Flood differ from a standard DDoS?
A: A SYN flood specifically targets the TCP stack’s memory (the backlog) rather than just trying to saturate the network bandwidth. You can have a SYN flood that uses very little bandwidth but still crashes a large server.
Q: What is the purpose of the Initial Sequence Number (ISN)?
A: The ISN provides a starting point for the byte count of the data stream. It must be random to prevent attackers from guessing it and injecting malicious data into an existing session.
Q: What does a SYN_SENT state on the client usually indicate?
A: It means the client has sent a request but has not received anything back. This usually points to a firewall dropping the packet, a routing issue, or the server being completely offline.
Future Trends: 2026 and Beyond
As we move further into 2026, the traditional 3-way handshake is seeing competition from newer protocols. QUIC, which runs over UDP, is becoming the standard for web traffic via HTTP/3. QUIC reduces the handshake overhead by combining the connection establishment with the TLS (Transport Layer Security) handshake. This is particularly relevant for mobile users in regions with fluctuating network quality, as it allows for much faster “0-RTT” (Zero Round Trip Time) reconnections.
Now here’s where it gets interesting: the shift toward Zero Trust Architecture (ZTA) in regions like the UAE and Saudi Arabia is changing how we monitor handshakes. Under NESA or SAMA guidelines, we are seeing more “Identity-Aware” proxies. These proxies do not even allow the 3-way handshake to reach the application server until the client has provided a valid identity token at the network edge. This effectively moves the “vulnerable window” of the handshake away from the critical data and onto a hardened gateway.
FAQ
What is the default timeout for a 3-way handshake?
Most modern Linux systems wait about 60 seconds before giving up on a half-open connection. This is controlled by the tcp_synack_retries setting, which determines how many times the server will re-send the SYN+ACK before timing out.
Can a 3-way handshake happen over UDP?
No, UDP is a connectionless protocol. It does not have flags like SYN or ACK and does not guarantee delivery. If an application needs a handshake over UDP, the developer must build that logic into the application layer itself, similar to how QUIC operates.
Is the 3-way handshake encrypted?
The handshake itself is not encrypted. The sequence numbers and flags are visible to anyone with a packet sniffer. Encryption usually happens after the handshake is complete, during the TLS negotiation phase.
How do I see the 3-way handshake in Wireshark?
You can use the filter tcp.flags.syn == 1 to see the initial connection attempts. To see a specific complete handshake, right-click a packet and select “Follow > TCP Stream.” This will show you the SYN, SYN+ACK, and ACK in sequence.
Does every TCP connection require a 3-way handshake?
Yes, every single new TCP connection must go through this process. Even if you are connecting to localhost on the same machine, the kernel still goes through the states of the handshake to ensure the TCB is properly initialized.
Conclusion
Understanding the 3-way handshake is the difference between guessing why a network is slow and actually knowing how to fix it. This process ensures that every byte of data has a tracked sequence and every connection is mutually agreed upon. As a security analyst, you should prioritize monitoring your SYN_RECV states and ensuring SYN cookies are active on all public-facing systems. Take ten minutes today to run ss -t -a on one of your production servers and look at the distribution of connection states. Knowing your baseline now will save you hours of panic during your next incident.
Reference: wikipedia
