Perforce Public Knowledge Base - Isolating Network Problems
Reset Search
 

 

Article

Isolating Network Problems

« Go Back

Information

 
Problem

How do I determine if slow Perforce response time is caused by network problems?


Solution

Start by reviewing Perforce log files and use standard system administration tools to diagnose slow response times. A network issue can be suspected if Perforce commands run quickly on the local machine but run slowly across a network. You can also compare the lapse time against the usage time, for example, if the information is available in the Perforce logs.

While it is possible that an extremely large user operation can be causing the Perforce Server to respond slowly, persistent slow responses to Perforce commands are usually due to network troubles. Any of the following can cause slow responses:

  1. Misconfigured domain name server (DNS)
  2. Misconfigured Windows name server (WINS) or Windows domain
  3. Slow network response

Although solving network problems is beyond the scope of this article, the following sections detail some suggestions for troubleshooting them.

p4 info

A good initial test is to run the p4 info command. If this command does not respond nearly immediately then there is a network related problem. The p4 info command uses the P4PORT setting to contact the Perforce Server. If the P4PORT setting uses the hostname of the server machine, a DNS lookup is required to fetch the server IP address. If the DNS server is failing or the network is slow, the lookup process takes time. You can use the IP address directly to avoid the DNS lookup.

Hostname vs. IP Address

On a client machine, try using the Perforce Server's IP address in the P4PORT setting. This avoids the DNS lookup used to convert the hostname to an IP address. Here is a P4PORT example which uses the server hostname:

P4PORT=hostname:1666

Using the IP address directly, the P4PORT setting then looks like this:

P4PORT=1.2.3.4:1666

If you do not have the Perforce Server machine's IP address handy, you can use the ping command to find it. Here is an example of how you would run the ping command:

ping hostnam

The output of the ping command lists the IP address for the hostname.

If the p4 info command responds immediately when you use the IP address in the P4PORT setting, you have a misconfigured DNS.

"p4 info" vs. P4Win

The p4 info command is processed by the Perforce Server. As the Perforce Server compiles the output information for the command, it does a reverse DNS lookup on the client's IP address. A forward DNS lookup might be fast, while a reverse DNS lookup is slow. One way to determine if a reverse DNS lookup is slow is by using P4Win, the Perforce Visual Client. The "Show Connection Info" operation in P4Win does not perform a reverse DNS lookup.

You can compare the response of P4Win's "Show Connection Info" with the response from the command line p4 info. If a "Show Connection Info" operation is fast while a p4 info is slow, there is a reverse DNS lookup problem on the Perforce Server machine.

The "hosts" file

To work around DNS problems, add hostname IP address entries to the hosts file. This task is typically performed by a Systems Administrator; be sure to follow your company's standard procedures.

Rather than using an IP address for the P4PORT setting, you can add a hostname IP address entry in your hosts file. The host file can be tricky to find. Here are a few places it can be:

  1. Windows Server 2000 (and later):
    C:\Winnt\System32\Drivers\etc\hosts
    
  2. Unix:
    /etc/hosts
    

The following is an example entry for the hosts file:

1.2.3.4		hostname

If you have determined there is a reverse DNS problem, you need to add an entry for your client hostname in the machine's hosts file where the Perforce Server resides.

Note: As of version 2010.1 Perforce no longer suffers performance problems as a result of reverse DNS issues.This includes an issue where form triggers would attempt to do a reverse DNS lookup for the %clienthost% trigger argument, even if the argument was not used as a part of the trigger. This resulted reverse lookup time outs that delayed the trigger firing.

Wildcards on Windows

In some cases, p4 commands using unquoted file patterns with a combination of depot syntax or client syntax and wildcards can cause delays.

p4 files //depot/*

p4 files //client/*

The delay occurs because the wild card expansion routines for Windows mistakes the depot name or client name for a network system name. You can prevent the delay by putting double quotes around the file pattern:

p4 files "//depot/*"

See KB Article: Using Special Characters for more information on special handling of wild cards.

Network Topology

Most networks use 100 Mbit technology. The maximum theoretical transfer rate of this type of network is 10 Mbytes per second. In most cases slower transfer rates are experienced. If a network is saturated, transfer rates can drop off significantly, maybe as low as 4 Mbytes per second. In cases of network saturation use of network routers or switches can help.

Client on a Network Filesystem

It is possible that the p4 executable itself is on a networked file system that is performing very poorly. To check for slow access to the executable, try running:

p4 -v

The p4 -v command simply prints out the version information and does not attempt to access the network. If you get a slow response, network access to the p4 executable itself may be the problem. Try copying or downloading the p4 executable onto a local filesystem.

Using a network filesystem for the Perforce client root can also cause slow command response. Every byte transferred in a Perforce client command must go through the network twice -- once from the Perforce Server to the Perforce client application, and once from the Perforce client application to the network filesystem. If the network is saturated and the client workspace contains large files or a large number of files, expect commands like p4 sync to be slow. Try moving the Perforce client workspace to a local filesystem.

Server on a Network Filesystem

Using a network filesystem for the Perforce Server must be thought through very carefully. Just as in the scenario above with the Perforce client stored on a network filesystem, data to and from the server application must go through the network twice. The Perforce Server also uses file locking on vital data files. Not all network filesystems have efficient locking implementations and some are buggy.

If the network is saturated and the transfer rate is down to about 4 Mbytes per second, the Perforce Server cannot satisfy client requests. Each client request uses part of the network, and the Perforce Server must share that resource in order to access the network filesystem. A cheap hard drive these days will provide a 20 Mbyte per second transfer rate. A good SCSI hard drive can transfer as much as 160 Mbytes per second. If possible, try to avoid a network filesystem related server bottle neck.

If you have a specialized network filesystem like a NetApp filer, consider using a private Gigabit fiber connection directly to the Perforce server machine. It is reasonable to expect transfer rates of 100 Mbytes per second on an isolated gigabit connection.

Disabling Remote Locking

If your Perforce server is the only machine mounting the NFS volume on a Network Appliance, consider disabling remote locking.  For example, see the "nolock" mount option below seen in the /etc/fstab

ex2:/vol/p4 /p4-1 nfs rw,soft,tcp,rsize=32768,wsize=32768,noatime,timeo=14,nolock 0 0
ex2:/vol/p4log1 /p4log1 nfs rw,soft,tcp,rsize=32768,wsize=32768,noatime,timeo=14,nolock 0 0

Note:
 

If remote locking is disabled, the filer must be solely dedicated to the Perforce server.  If two or more NFS clients mount the same NFS filesystem, those using the 'nolock' mount option will not see file locks from other NFS clients. If an NFS client takes an exclusive lock to ensure exclusive access prior to writing to a file on NFS filesystem mounted with the 'nolock' mount option, that NFS client has no way of knowing that some other NFS client is concurrently doing the same thing for the same file but writing different data. And other corruption scenarios are possible when using the 'nolock' mount option on NFS filesystems, such as reading incomplete data when writes are in-flight on another NFS client. If you are interested, there is a local-locking test suite available

ftp://ftp.ucsb.edu/pub/mirrors/procmail/procmail-3.22.tar.gz

where you decompress it, cd procmail-3.22, and edit Makefile so that:

LOCKINGTEST=001

Related Links

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255