A common reason for this issue are routers or operating systems that use SYN "cookies" to prevent denial of service requests as the result of a "SYN flood".
What is a SYN Flood?
Normally when a client attempts to start a TCP connection to a server, the client and server exchange a series of messages which normally runs like this:
- The client requests a connection by sending a SYN (synchronize) message to the server.
- The server acknowledges this request by sending SYN-ACK back to the client.
- The client responds with an ACK, and the connection is established.
This is called the TCP three-way handshake, and is the foundation for every connection established using the TCP protocol. A SYN flood attack works by not responding to the server with the expected ACK code. The malicious client can either simply not send the expected ACK, or by spoofing the source IP address in the SYN, causing the server to send the SYN-ACK to a falsified IP address - which will not send an ACK because it "knows" that it never sent a SYN.
The server will wait for the acknowledgement for some time, as simple network congestion could also be the cause of the missing ACK, but in an attack increasingly large numbers of half-open connections will bind resources on the server until no new connections can be made, resulting in a denial of service to legitimate traffic. Some systems may also malfunction badly or even crash if other operating system functions are starved of resources in this way.
To combat SYN flooding many operating systems, firewalls and routers use SYN cookies. When the server sends a SYN-ACK, it encodes a special sequence, or "cookie" into the response that allows the server to reconstruct the SYN message. This way it can avoid filling the queue with SYN messages.
The problem is in high traffic situations the server may still interpret delays in SYN-ACK responses as potential flooding, which results in connections closing (being reset).
How to Identify SYN Related Failures (Linux)
Linux provides a log for SYN flooding warnings, located at:
Using 'grep' you can check for possible SYN related messages:
cat /var/log/messages | grep possible SYN flooding
Messages will look similar to:
[84440.731929] possible SYN flooding on port 1666. Sending cookies.
How to Turn Off SYN Cookies (Linux)
To disable SYN cookies under Linux and and to instruct Perforce to increase the accept bucket size, enter this command on the Perforce server:
sudo sysctl -w net.ipv4.tcp_syncookies=0
Edit /etc/sysctl.conf to include the line:
Then adjust the Perforce server net.backlog configurable to 50 or higher:
p4 configure set net.backlog=50
Note: The Perforce server must be restarted after making this change.
Also confirm that the operating systems's maximum backlog is also set at least as high as the value used in Perforce. You can check the current value using the command:
sysctl -a | grep -i backlog
Is Turning Off SYN Cookies Safe?
It can be argued that turning off SYN cookies could expose a server to a potential denial of service attack. However, SYN cookies were never meant to be a permanent fix for the issue, but a workaround that did not require low level changes to the TCP/IP stack of both the server and clients.
As long as the server is only available on the internal LAN, the risk of a true SYN flood attack is very small to begin with. Customers using this approach have solved most connection reset problems without negative side effects.