Perforce Public Knowledge Base - Fixing a hung Helix Server
Downloads Blog Company Integrations Careers Contact Try Free
Menu Search
Reset Search



Fixing a hung Helix Server

« Go Back


What steps should I take if my Helix server is not responding?
If the Helix Server is not responding for a user, follow these steps to isolate the problem.  It is important to determine whether the Helix Server is down, slow, or hung.

1. Determine if the Helix server can be reached from the end user machine

First and foremost, have the end user run p4 info.   Have the user with the problem run

p4 -p <server IP>:1666 info

where 1666 is the port.  The IP address will bypass DNS.  If p4 is not installed have the user download the P4: Command-Line Client.  The p4 info command uses very few resources on the Perforce database, so it is a good indicator on whether the Helix server is up at all.

A. If the Helix client cannot connect to the Helix server

If a p4 info error comes back with a "check $P4PORT" error, double-check that you entered the proper IP address and port number after the -p flag.  A "check $P4PORT" error indicates a network connectivity issue, a firewall issue, or most likely, Perforce is down.  Proceed to step 2 to try to connect from the Helix server machine.

B. If the Helix client can connect to the Helix server

If p4 info does return, then the Helix Server is up.  The Helix Server may be slow, but it is not down.

If p4 info does return quickly, have the user run a command that uses more resources like

p4 changes -t -m 10
p4 changes -t -m 2000

Check how fast this runs, or if it hangs. If the larger command hangs but the smaller command works, there is a networking issue where large commands cannot be handled.  If both commands are slow, then there is a machine resource issue. Contact your IT administrator.

C. If p4 info returns quickly but other commands are not responding, terminate the process that is blocking the other commands.

You will be able to know if the Helix Server is up but waiting on the database if you can get output from

p4 lockstat
p4 lockstat -C

If locks are present, you may have to ask the client to stop their processes or perhaps reboot their workstation.

Alternatively, to stop the processes on the server side by following the section "Terminating locked processes" below. Assuming db.monitor.interval was already set, choose some of the oldest running processes listed in "p4 monitor show -ael" and terminate them until the Helix server is responsive

p4 monitor show -ael
p4 monitor terminate pid

2. Determine if the Helix server can be reached from the Helix server

Remote desktop or ssh into the Helix Server and run

p4 -p localhost:1666 info

where 1666 is the port number that runs the Helix Server

A.  If p4 info does not run on the Helix Server

If p4 info cannot run successfully, check if the Helix Server parent process is running.

On Unix, run

ps -elf | grep p4d 

and look for a running p4d process.  Ignore the "grep" line. 

On Windows, check that the Perforce service under Control Panel, Services, is running.

On either platform, if you cannot run p4 info on the Helix Server server, the server is basically unusable so you might as well stop the Helix Server. Kill the parent (not the child!) pid, then wait five minutes (running "ps -elf | grep p4d" frequently) and wait for all p4d processes to exit. On Unix, use the Unix "kill" or "kill -15" command on the parent Perforce pid. There is usually no need to run "kill -9". On Windows servers, kill the parent process by stopping the Perforce service or by using Task Manager or Process Explorer.

After five minutes after the parent pid has been killed, if the child processes are not being removed, make sure that your running journal (usually named "journal") is not growing, then kill some of the child p4d processes until the rest are terminating normally. Then wait until all p4d processes (or p4s threads on Windows) are gone and restart Perforce. If Perforce does not start, view the Perforce log for clues. 

But if you want to continue further, check whether CPU and memory are adequate. 

On Unix, run


and press the number 1 to see each processor.  Determine if any p4d process is consuming all the CPU or memory.

On Windows, run Task Manager and determine if the p4d or p4s process is consuming all the CPU or memory.  Note that overall CPU may be low, but a single processor may be at 100% CPU for minutes at a time which indicates not enough CPU to run a process.

B.  If  p4 info does run properly on the Helix Server

If p4 info does work properly on the Helix Server, the server is up.

If p4 info runs without errors on the server, but not at the end user, there is a firewall issue. But assuming p4 info runs on the server, then try a larger resource command like

p4 changes -t -m 10
p4 changes -t -m 2000

Check how fast this runs, or if it hangs. If the larger command hangs but the smaller command works, there is a networking issue where large commands cannot be handled.  If both commands are slow, then there is a machine resource issue. Contact your IT administrator.

In any case, run several times

p4 lockstat
p4 lockstat -C 

to see if the Helix Server database is locked.  

The "p4 lockstat" command will let you know if the Helix Server is currently processing commands that are currently locking the database.  If database tables are locked, run this command repeatedly to check whether the database locks are freeing up.

The "p4 lockstat -C" will check for client or global metadata locks as seen in Client Workspace and Global Metadata Locks

You may want to terminate processes that are blocking other users.

Terminating locked processes

Before running "p4monitor terminate", make sure the db.monitor.interval is set by running

p4 configure show db.monitor.interval

A setting of 0 means the feature is disabled; you can set db.monitor.interval by running

p4 configure set db.monitor.interval=30

which, for new commands processed by the server going forward, configures the server to check for and terminate processes that have had "p4 monitor terminate pid" run on them and are blocked on client input.

Assuming db.monitor.interval was set before the process was started, run

p4 monitor show -ael 

and look at some of the oldest commands that are not a form to fill out. (Ignore forms like "p4 client" or "p4 label" because these are just waiting for a user to complete the form).  While the oldest command times may not always be accurate, it provides candidates to kill.  To free up commands that are locking the Helix Server, contact the user to stop their command, run

p4 monitor terminate pid

where the pid is found from "p4 monitor show -ael".  Start killing the oldest pids first that are not fill-in form commands.

You can also optionally obtain Linux lock information by running

p4 configure set monitor=5

Optional Linux lock information

Even more optional information is available for Linux users by using the lsof utility if this has been installed.  Set monitor.lsof by running

p4 configure set monitor.lsof=/usr/bin/lsof -F pln

Then you can check what database files are being locked by running

p4 monitor show -aelL

If "p4 lockstat" is showing a Perforce db file locked, the -L flag in "p4 monitor" may provide a clue as to which process in "p4 monitor show" is locking that db file.   Look at the oldest processes that have been running "p4 monitor -ael".  Then you can use "p4 monitor terminate <pid>" to stop the process that is blocking other users.


Check the running journal

If the Perforce database is not locked per p4 lockstat, and the process cannot be killed with "p4 monitor terminate pid", check that the Helix Server journal is not growing.

tail -f <pathto>/journal 

If the journal is growing rapidly, then the Helix Server is processing commands.  Simply wait for the command running to complete.  Killing a sub-process that is accessing the Helix Server can corrupt the database files.

But if the journal seems sluggish and only showing occasional db.user and db.domain entries (logins), then check hardware resources for adequate CPU and memory. 
On Unix, use


and on Windows, use Task Manager.  If "top" or Task Manager shows a lack of memory or 100% CPU, find the process or thread that uses up the memory or CPU. 

In any case, you may want to use "p4 monitor terminate <pid>" to remove a process or thread as mentioned above.  If CPU and memory is at 100%, run "p4 monitor terminate <pid>" to kill the high resource command, then re-run the command with less files (such as running "p4 obliterate", "p4 integrate" or "p4 revert" on a smaller directory at a time).

3.  From here, you will have plenty to go on.
  1. From p4 lockstat and  p4 lockstat -C, you will know if the Helix Server is running, but a command is locking the database.
  1. From running p4 info commands on the same machine as the server, you will isolate network issues from your client to the server.
  1. From "top" or Task Manager, you will know whether you are out of CPU or memory.
  2. From "p4 monitor show -ael", you will know if the Helix Server is overwhelmed with processes and you can start guessing which process you can stop by running "p4 monitor terminate".
Feel free to run "p4 monitor terminate <pid>" anytime. This is the safest way to kill processes that will not harm the Perforce database. If you do this, let the end user know you terminated their command.

If you need off-hours support, note that each of our UK, Canada, US, and Australia offices will support you from Monday through Friday excluding holidays.  Use our support page and dial our international number, or send a new email to  If you have an existing case number, place this into the body of the message. 
Related Links



Was this article helpful?



Please tell us how we can make this article more useful.

Characters Remaining: 255