Perforce Public Knowledge Base - SYN Cookies and WSAECONNRESET Errors
Reset Search
 

 

Article

SYN Cookies and WSAECONNRESET Errors

« Go Back

Information

 
Problem
Perforce clients are seeing a lot of networking errors similar to:

Perforce client error:
TCP receive failed.
read: socket: WSAECONNRESET
error: TCP receive failed.
error: read: socket: WSAECONNRESET
exit: 1

The errors are intermittent, but seem to become more frequent when server load is high.

Solution

A common reason for this issue are routers or operating systems that use SYN "cookies" to prevent denial of service requests as the result of a "SYN flood".

What is a SYN Flood?

Normally when a client attempts to start a TCP connection to a server, the client and server exchange a series of messages which normally runs like this:

  1. The client requests a connection by sending a SYN (synchronize) message to the server.
  2. The server acknowledges this request by sending SYN-ACK back to the client.
  3. The client responds with an ACK, and the connection is established.

This is called the TCP three-way handshake, and is the foundation for every connection established using the TCP protocol. A SYN flood attack works by not responding to the server with the expected ACK code. The malicious client can either simply not send the expected ACK, or by spoofing the source IP address in the SYN, causing the server to send the SYN-ACK to a falsified IP address - which will not send an ACK because it "knows" that it never sent a SYN.

The server will wait for the acknowledgement for some time, as simple network congestion could also be the cause of the missing ACK, but in an attack increasingly large numbers of half-open connections will bind resources on the server until no new connections can be made, resulting in a denial of service to legitimate traffic. Some systems may also malfunction badly or even crash if other operating system functions are starved of resources in this way.

SYN Cookies

To combat SYN flooding many operating systems, firewalls and routers use SYN cookies. When the server sends a SYN-ACK, it encodes a special sequence, or "cookie" into the response that allows the server to reconstruct the SYN message. This way it can avoid filling the queue with SYN messages.

The problem is in high traffic situations the server may still interpret delays in SYN-ACK responses as potential flooding, which results in connections closing (being reset).

How to Identify SYN Related Failures (Linux)

Linux provides a log for SYN flooding warnings, located at:


/var/log/messages

Using 'grep' you can check for possible SYN related messages:


cat /var/log/messages | grep possible SYN flooding

Messages will look similar to:


[84440.731929] possible SYN flooding on port 1666. Sending cookies.

How to Turn Off SYN Cookies (Linux)

To disable SYN cookies under Linux and and to instruct Perforce to increase the accept bucket size, enter this command on the Perforce server:


sudo sysctl -w net.ipv4.tcp_syncookies=0

Edit /etc/sysctl.conf to include the line:


net.ipv4.tcp_syncookies=0

Then adjust the Perforce server net.backlog configurable to 50 or higher:


p4 configure set net.backlog=50

Note: The Perforce server must be restarted after making this change.

Also confirm that the operating systems's maximum backlog is also set at least as high as the value used in Perforce. You can check the current value using the command:


sysctl -a | grep -i backlog

Is Turning Off SYN Cookies Safe?

It can be argued that turning off SYN cookies could expose a server to a potential denial of service attack. However, SYN cookies were never meant to be a permanent fix for the issue, but a workaround that did not require low level changes to the TCP/IP stack of both the server and clients.

As long as the server is only available on the internal LAN, the risk of a true SYN flood attack is very small to begin with. Customers using this approach have solved most connection reset problems without negative side effects. 
Related Links

A good history lesson as to why SYN cookies were considered useful. Note that the time period for this article is over 17 years ago as of this writing (circa 1996). At that point in time a 100 base T network was fairly cutting edge, and most remote connections were 28Kbps.

A sysadmin with a SYN cookie issue in late 2011 as a result of legitimate traffic. This article provides a lot of the "meat" used in this article, including ways that the issue can be detected and fixed.

A lengthy RedHat discussion upon the reporting of SYN flooding as a potential bug (Spoiler: it's not), filed in September 2011. It covers much of the same configuration variables for Linux discussed above.

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255