A hand-drawn technical diagram showing the three steps of a TCP 3-way handshake.

3-Way Handshake Troubleshooting: Critical Guide for Security Analysts

by

If you have ever monitored a network during a Distributed Denial-of-Service (DDoS) attack, you know how quickly thousands of incomplete connections can overwhelm critical infrastructure. During one enterprise deployment, I investigated an incident where a load balancer began dropping legitimate user traffic because its connection table was filled with half-open TCP sessions. The root cause was not a complex application attack but an exploitation of the TCP 3-Way Handshake process.

For network engineers, cybersecurity analysts, and SOC professionals, understanding how TCP establishes a connection is an essential skill. Whether you are troubleshooting connectivity issues, analyzing packet captures in Wireshark, or defending against SYN flood attacks, knowing how the handshake works helps you identify problems faster and build more resilient networks.

In this guide, you’ll learn how the TCP 3-Way Handshake works, why each step matters, how attackers abuse the process, and how enterprise security teams detect and mitigate these threats in real-world environments.

What is a 3-way handshake?

The TCP 3-Way Handshake is the process that the Transmission Control Protocol (TCP) uses to establish a reliable connection between a client and a server before any data is transmitted. During this process, both devices confirm that they are ready to communicate and synchronize their initial sequence numbers to ensure accurate and ordered data delivery.

Without this handshake, network communication would be unreliable, with no guarantee that packets arrive in the correct order or that the receiving device is ready to accept them. This reliability is one of the key reasons TCP is widely used for web browsing, email, file transfers, and other critical internet services.

TCP is a connection-oriented protocol that builds upon fundamental networking concepts. If you’re new to networking or cybersecurity, we recommend reading our Network Security Basics Guide to understand how TCP fits into the overall network communication process before exploring the TCP 3-Way Handshake in detail.

How does a 3-way handshake work?

The TCP 3-Way Handshake works through a carefully coordinated exchange of packets that synchronizes the communication state between a client and a server before any application data is transmitted.

This process is essential because IP networks are inherently unreliable. Packets may arrive out of order, be duplicated, become delayed, or even be lost during transmission. TCP overcomes these challenges by establishing a reliable connection before data exchange begins.

During the handshake, both the client and the server generate random Initial Sequence Numbers (ISNs) and acknowledge each other’s values. This confirms that both systems can successfully send and receive packets, verifies that the communication path is working in both directions, and prepares each device to track every byte of data exchanged during the session.

The handshake also prevents delayed or duplicate packets from previous connections from being mistaken for new communication. By validating sequence numbers and allocating connection resources only after successful negotiation, TCP ensures reliable, ordered, and error-checked data transmission throughout the session.

Technical Flow and Data Exchange

Numbered sequence diagram of the 3-way handshake packets leading to an established state.

The TCP 3-Way Handshake begins when an application, such as a web browser, requests a connection to a specific IP address and port. The operating system’s TCP stack then initiates the connection establishment process before any application data is exchanged.

Step 1: SYN

The client starts the TCP 3-Way Handshake by sending a SYN (Synchronize) packet to the server.

This packet contains a randomly generated Initial Sequence Number (ISN), which we’ll call X. After sending the SYN packet, the client enters the SYN-SENT state and waits for a response from the server.

Step 2: SYN-ACK

When the server receives the SYN packet, it verifies that the requested port is open and that resources are available to create a new connection.

The server then responds with a SYN-ACK packet. This response serves two purposes:

  • It acknowledges the client’s sequence number by returning X + 1.
  • It generates its own Initial Sequence Number, represented as Y.

At this stage, the server enters the SYN-RECEIVED (SYN_RECV) state and temporarily allocates memory to track this pending connection.

Step 3: ACK

After receiving the SYN-ACK packet, the client sends the final ACK (Acknowledgment) packet.

This packet acknowledges the server’s sequence number by sending Y + 1, while the client’s next sequence number becomes X + 1.

Once the server receives this acknowledgment, the TCP 3-Way Handshake is complete, and both devices transition to the ESTABLISHED state. Application data can now be transmitted reliably between the client and the server.

Key Components of the TCP Header

To troubleshoot TCP connection problems effectively, it is important to understand the information carried within the TCP header.

SYN Flag

The SYN (Synchronize) flag is used during the first stage of connection establishment. It tells the receiving device that the sender wants to create a new TCP connection and synchronize sequence numbers.

ACK Flag

The ACK (Acknowledgment) flag indicates that the acknowledgment field contains a valid value confirming successful receipt of previous packets.

Sequence Number

The Sequence Number is a 32-bit value that identifies the position of data within the TCP stream. It enables the receiver to correctly reorder packets that arrive out of sequence.

Acknowledgment Number

The Acknowledgment Number informs the sender of the next sequence number the receiver expects to receive, confirming successful delivery of previous data.

Window Size

The Window Size specifies how much data the receiver can accept before another acknowledgment is required. This mechanism helps control network congestion and optimize throughput.

Maximum Segment Size (MSS)

The Maximum Segment Size (MSS) is negotiated during the handshake and defines the largest amount of TCP payload that can be transmitted in a single segment without fragmentation.

Real-World Example: Identifying a SYN Flood

When troubleshooting enterprise servers, administrators commonly use tools such as netstat or ss to inspect active TCP connections.

A healthy server typically contains many ESTABLISHED connections and only a small number of connections in the SYN_RECV state.

[admin@prod-web-01 ~]# netstat -antp

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address          Foreign Address         State         PID/Program

tcp        0      0 10.10.50.20:443        192.168.1.55:54321      ESTABLISHED   1204/nginx
tcp        0      0 10.10.50.20:443        172.16.10.12:33211      SYN_RECV      -
tcp        0      0 10.10.50.20:443        172.16.10.15:44212      SYN_RECV      -
tcp        0      0 10.10.50.20:443        172.16.10.18:55213      SYN_RECV      -

In this example, the server has one established connection and three connections waiting for the final acknowledgment.

A few SYN_RECV entries are completely normal because new connections are constantly being created.

However, if you observe hundreds or thousands of SYN_RECV connections from numerous external IP addresses without corresponding ESTABLISHED sessions, it may indicate a SYN flood attack. In this situation, attackers repeatedly send SYN packets but never complete the final acknowledgment, causing the server to allocate resources for incomplete connections until legitimate users can no longer connect.

Attack diagram showing an attacker overwhelming a server with SYN packets, causing legitimate traffic to drop.

Practical Implementation of Network Security Controls

When deploying production servers, properly configuring the TCP stack is essential for maintaining performance, availability, and resilience against network attacks. A well-configured TCP 3-Way Handshake process helps prevent connection-based attacks from affecting legitimate users.

Enable SYN Cookies

Enable SYN Cookies on Linux systems to protect against SYN flood attacks. This allows the operating system to validate connection requests without allocating resources until the handshake is successfully completed.

For Linux systems, configure the following setting in /etc/sysctl.conf:

net.ipv4.tcp_syncookies = 1

Increase the SYN Backlog Queue

Increase the tcp_max_syn_backlog value to allow the server to handle a larger number of pending connection requests during periods of heavy traffic or temporary connection spikes.

Reduce SYN-ACK Retries

Adjust the tcp_synack_retries parameter so the server does not repeatedly send SYN-ACK packets to unreachable or malicious clients. This helps release connection resources more quickly during attack scenarios.

Enable TCP Intercept on Network Devices

Many enterprise firewalls and load balancers support TCP Intercept. With this feature enabled, the security device completes the TCP handshake on behalf of the server before forwarding the connection. Invalid or incomplete sessions are discarded before reaching backend servers.

Apply Rate Limiting

Configure rate-limiting policies on firewalls or load balancers to restrict excessive SYN packets originating from the same source IP address. This reduces the effectiveness of automated denial-of-service attacks.

Monitor TCP Connection Metrics

Continuously monitor TCP connection statistics, including:

  • Large numbers of SYN_RECV connections
  • Abnormal TCP Window Size values
  • High connection failure rates
  • Unexpected retransmissions
  • Excessive half-open connections

These indicators may reveal SYN flood attacks, Sockstress attacks, or other network performance issues.

Advantages and Limitations

The TCP 3-Way Handshake provides reliable and ordered communication by ensuring both the client and server are fully synchronized before application data is exchanged.

Advantages

  • Establishes a reliable connection before data transfer begins.
  • Synchronizes sequence numbers to ensure ordered packet delivery.
  • Confirms that both endpoints can send and receive data successfully.
  • Detects communication problems early in the connection process.
  • Forms the foundation for reliable internet services such as HTTPS, SSH, email, and file transfers.

Limitations

The reliability of TCP comes with certain trade-offs.

Connection Latency

Every new TCP connection requires a complete handshake before data transmission begins. On high-latency networks, such as satellite links or long-distance international connections, this additional round trip can increase application response times.

Resource Consumption

During connection establishment, the server temporarily allocates memory and processing resources for each pending connection. If a large number of incomplete connections accumulate, legitimate users may experience delays or failed connections.

This characteristic makes TCP susceptible to SYN flood attacks, where attackers intentionally create numerous incomplete connections to exhaust server resources. Technologies such as SYN Cookies, connection rate limiting, TCP Intercept, and modern DDoS protection services help mitigate this risk.

Although these limitations exist, the TCP 3-Way Handshake remains one of the most reliable connection establishment mechanisms used across modern computer networks and continues to be the foundation of secure and dependable internet communication.

A radial map categorizing critical and moderate security risks for the 3-way handshake.

Common Mistakes When Analyzing the TCP 3-Way Handshake

Even experienced engineers can misinterpret packets when analyzing the TCP 3-Way Handshake. Understanding normal TCP behavior is essential before assuming malicious activity.

One common mistake is believing that every RST (Reset) packet indicates an attack. In reality, a reset packet often represents a normal response. For example, if a client sends a SYN packet to a port where no service is listening, the server responds with a RST packet to terminate the connection attempt. This behavior is part of the normal TCP 3-Way Handshake process.

Another frequent mistake is overlooking sequence number randomization. Some junior analysts attempt to predict sequence numbers during packet injection or penetration testing. Modern operating systems generate highly randomized Initial Sequence Numbers (ISNs) to prevent session hijacking. If sequence numbers appear predictable, the system may be outdated or improperly configured, creating a potential security risk.

Understanding these behaviors makes troubleshooting the TCP 3-Way Handshake much more accurate and helps distinguish legitimate network events from real attacks.

The Security Analyst’s Perspective on the TCP 3-Way Handshake

For cybersecurity professionals, the TCP 3-Way Handshake provides valuable visibility into network activity. Every connection begins with this exchange, making it an excellent source for detecting suspicious behavior.

Security analysts should carefully examine TCP flags during packet analysis. For example, a packet containing both the SYN and FIN flags simultaneously is not valid during a normal TCP 3-Way Handshake. Attackers sometimes generate these unusual packets to identify operating systems, evade poorly configured firewalls, or perform network reconnaissance.

Enterprise environments introduce another layer of complexity. Firewalls, proxies, load balancers, and cloud security gateways may inspect or modify packets as they pass through the network. These devices can rewrite sequence numbers, remove TCP options, or terminate and re-establish the TCP 3-Way Handshake before forwarding traffic to backend servers.

If packet captures collected on the client differ from those collected on the server, always consider whether an intermediate network device has modified the connection.

Understanding how enterprise security devices interact with the TCP 3-Way Handshake is an important skill for network engineers, SOC analysts, and incident responders.

Troubleshooting Scenario: The Silent Drop

Imagine a user in a branch office in Riyadh attempting to access a banking application hosted in Mumbai. The application never loads and appears to hang indefinitely.

A packet capture shows the client initiating the TCP 3-Way Handshake by sending a SYN packet. The server successfully receives the request and replies with a SYN-ACK packet. However, the client never sends the final ACK required to complete the TCP 3-Way Handshake. Instead, it repeatedly retransmits the SYN packet.

This behavior usually indicates that the SYN-ACK response is being dropped somewhere along the network path before reaching the client. Possible causes include firewall filtering, asymmetric routing, ISP packet loss, incorrect MTU settings, or load balancer configuration issues.

Following the TCP 3-Way Handshake step by step allows engineers to identify exactly where communication stops, making troubleshooting faster and more effective. In enterprise environments, analyzing each stage of the TCP 3-Way Handshake is often the quickest way to isolate connectivity problems and restore normal network communication.

A decision-tree flowchart for diagnosing 3-way handshake failures in a production network.

Troubleshooting Scenario: The Silent Drop

At first glance, many engineers assume the server is offline. However, the packet capture tells a different story. The server successfully receives the SYN packet and responds with a SYN-ACK, which means the TCP 3-Way Handshake has already started correctly.

The real problem often exists somewhere along the network path. In enterprise environments, issues such as MTU (Maximum Transmission Unit) mismatches, firewall policies, or unsupported TCP options can prevent the handshake from completing successfully.

In this example, an intermediate firewall was dropping the SYN-ACK packet because it exceeded the allowed packet size or contained a Selective Acknowledgment (SACK) option that the device did not process correctly. Since the client never received the SYN-ACK, it continued retransmitting the initial SYN packet, preventing the TCP 3-Way Handshake from reaching the ESTABLISHED state.

The solution was to adjust the Maximum Segment Size (MSS) on the router, allowing packets to pass through the VPN tunnel without fragmentation and enabling the TCP 3-Way Handshake to complete successfully.

Interview Questions

Q1. What happens if the final ACK in the TCP 3-Way Handshake is lost?

Answer:

The server remains in the SYN_RECV state and retransmits the SYN-ACK packet. Until the final ACK is received, the server does not consider the connection fully established and ignores application data from the client.

Q2. Why can’t TCP use a two-way handshake?

Answer:

A two-way handshake cannot reliably confirm that both devices are synchronized. The third acknowledgment ensures both the client and server agree on the connection parameters and prevents delayed packets from previous sessions from creating invalid or “ghost” connections.

Q3. How is a SYN flood different from a traditional DDoS attack?

Answer:

A SYN flood targets the TCP connection table by creating a large number of incomplete TCP 3-Way Handshake requests. Unlike bandwidth-based DDoS attacks, a SYN flood primarily exhausts server memory and connection resources rather than network bandwidth.

Q4. What is the purpose of the Initial Sequence Number (ISN)?

Answer:

The Initial Sequence Number (ISN) defines the starting point for data transmission during a TCP session. Modern operating systems generate random ISNs to reduce the risk of session hijacking and packet injection attacks.

Q5. What does the SYN_SENT state indicate?

Answer:

The SYN_SENT state indicates that the client has transmitted a SYN packet but has not yet received a SYN-ACK response. This often points to firewall filtering, routing problems, packet loss, or an unavailable server.

Future Trends

The TCP 3-Way Handshake continues to be the standard connection establishment mechanism for TCP applications, but newer protocols are reducing connection latency.

QUIC, which operates over UDP and powers HTTP/3, combines transport connection establishment with TLS encryption. This significantly reduces connection setup time and supports features such as 0-RTT (Zero Round Trip Time) session resumption, improving performance for mobile users and high-latency networks.

Enterprise security architectures are also evolving. Modern Zero Trust implementations increasingly validate user identity, device posture, and security policies before allowing application access. Identity-aware proxies and secure access gateways can inspect and authorize traffic before forwarding it to backend servers, providing an additional security layer around traditional network communication.

Frequently Asked Questions (FAQ)

What is the default timeout for a TCP 3-Way Handshake?

Most Linux systems wait approximately 60 seconds before abandoning an incomplete connection. The timeout behavior is controlled by parameters such as tcp_synack_retries, which determine how many times the server retransmits the SYN-ACK packet.

Can a TCP 3-Way Handshake occur over UDP?

No. UDP is a connectionless protocol and does not establish connections using SYN or ACK packets. Protocols such as QUIC implement their own connection establishment mechanisms while operating over UDP.

Is the TCP 3-Way Handshake encrypted?

No. The handshake itself is transmitted in clear text. Encryption is typically established afterward during the TLS handshake, where cryptographic keys are negotiated.

How can I capture the TCP 3-Way Handshake in Wireshark?

Use the display filter:

tcp.flags.syn == 1

To view a complete connection, right-click any TCP packet and select Follow → TCP Stream. This displays the SYN, SYN-ACK, and ACK packets in sequence.

Does every TCP connection require a TCP 3-Way Handshake?

Yes. Every new TCP connection begins with a TCP 3-Way Handshake, including connections established between applications running on the same computer. This process initializes the TCP connection state before reliable data transfer begins.

Conclusion

The TCP 3-Way Handshake is one of the most fundamental processes in computer networking. It establishes reliable communication, synchronizes sequence numbers, and ensures both endpoints are ready before any application data is exchanged.

For network engineers and cybersecurity professionals, understanding the TCP 3-Way Handshake makes troubleshooting much more efficient. Monitoring connection states such as SYN_SENT, SYN_RECV, and ESTABLISHED, enabling SYN Cookies, and regularly reviewing TCP connection statistics can help identify configuration problems and detect attacks before they affect production services.

Take a few minutes to inspect the active TCP connections on one of your systems using tools such as ss or netstat. Understanding your normal connection patterns today will make it much easier to recognize abnormal behavior during future incidents.

Continue learning with these networking and cybersecurity guides from Tech Naga:

References

1 thought on “3-Way Handshake Troubleshooting: Critical Guide for Security Analysts”

  1. Pingback: OSI Model Guide: 7 Powerful Layers Every Engineer Must Know

Leave a Comment

© 2026 technaga.com • designed by Mr.Tejaa