Perforce Public Knowledge Base - How To Confirm Shelved Archives for p4d versions prior to 2015/12084891
Reset Search
 

 

Article

How To Confirm Shelved Archives for p4d versions prior to 2015/12084891

« Go Back

Information

 
Problem

This article describes how to verify shelved changes for missing or damages archives for Perforce server versions prior to 2015.1/1204891.

When shelved changes were first introduced they were considered "temporary" objects that would not need to be verified. With Perforce server versions prior to 2015.1/1204891 they were skipped when running verify to reduce the impact on the p4 verify command run time. 

Perforce server version 2015.1/1204891 introduced the -S option to verify shelved archives, and the -t option to schedule the transfer of missing or damaged shelf archives. 

From the release notes:
 

Bugs fixed in 2015.1 PATCH6
	
    #1202710 (Bugs #44639, #63140, #74807) **
	    The verify command now supports the -S flag to specify that
	    shelved files are to be verified. This flag can also be combined
	    with the -t flag on a replica to schedule the transfer of any
	    missing or damaged shelf archives from the master server.


However, with earlier versions before this functionality was added, there are some issues that can result in missing archives, such as an incomplete move of the archives when moving the server.

There was also a serious bug introduced in Perforce server version 2012.2:

	#580188 (Bug #42670) **
	    Running concurrent 'p4 shelve' commands to both update and
	    delete the same shelved change could cause unexpected loss
	    of shelved data. This has been fixed.

When shelved archives are missing users might report shelved changes are showing errors similar to this one when they try to use p4 print or unshelve the file to their local workspace:

Librarian checkout depot/path/to/shelved.file failed. 
open for read: depot/path/to/shelved.file,d/1.1234.gz: No such file or directory

 

Solution

While prior to Perforce server 2015.1/1204891 there is no built in command to detect missing shelved archives, a relatively simple script can detect them for you.

The bash script example has been tested on Linux and on Windows using Cygwin, GIT Bash (which uses MINGW32) and GOW (GNU On Windows). If no Unix tools are available on Windows then a simplified Powershell script is also given that checks for the existence of the shelved archives but does not verify the stored digests for archives that do exist.

Note: While those scripts have been tested to work, they are offered AS-IS as examples only. Always test in a non-production environment and modify to suit your needs prior to running on a production environment.

Bash shell script example

  • Save the script below to a file, for example "verifyShelvedArchives.sh"
  • Set the executable bit on the file:
  chmod +x verifyShelvedArchives.sh
  • For usage notes and examples run the script with the -h option:
  verifyShelvedArchives.sh -h
  • For detailed notes on the script's operation see the comments in the script.
  • Execute the script as a Perforce super user
#!/bin/bash
#
# verifyShelvedArchives.sh
#
# p4 verify did not verify shelves archives files until 2015.1/1204891. For earlier versions we can achieve 
# the required result with some scripting as described below. See usage below for a fuller description.

# METHODOLOGY
# It's very straightforward really and the following could be adapted to any scripting lanugage. Bash can be cross platform 
# if using cygwin or a simlar set of unix tools and the pipefail functionality to help detect missing archives was a factor 
# in the decision to use it to allow a single execution of "p4 print"; if p4 print fails, without pipefail it's hard to detect
# as the following md5sum and awk commands in the pipe succeed even with an error and testing the status via $? doesn't work 
# without the pipefail.
#
#  - Take the output of "p4 changes -s shelved" to get a list changes with shelved files.
#
#  - Then feed that into "p4 files @=changenumber" to get a list of the files
#
#  - For each file in the shelved change, use fstat to get the recorded values for digest
#    (i.e. the hash recorded when the file was shelved) and the archive(library) file location as well as the filetype
#    to deal with text files on Windows where we need to strip out carriage returns before taking the checksum.
#
#   - Run md5sum on the output of p4 print file@=change. This step also detects if the archive file is missing:
#
#  The 2 digests should match (allowing for the value output by fstat being uppercase)

function usage () {
cat <<USAGE
PURPOSE
  This script verifies shelved archive files, both for digest (BAD) and existence (MISSING) in the same manner 
  as "p4 verify" does for standard archives. Intended for use in p4d versions prior to 2015.1/1204891 which introduced
  verification of shelved archives via the p4 verify -S option.

  Tested on Linux and on Windows using Cygwin, GIT Bash (which uses MINGW32) and GOW (GNU On Windows)
 
ARGUMENTS AND OPTIONS

  -q  :Optional 1st arg. Supress output of name each file/change as it is being processed, only reporting MISSING or BAD

   //depot/path/... : Optional depot path. Either as 2nd argument after -q or 1st argument if -q not used. 
                      *** You must enclose the path in double quotes if it contains spaces ***
 
EXAMPLES
 
  -Verify whole depot and print out every every file as it is procesed
 
    verifyShelveArchives.sh
 
  -Verify whole depot in quiet mode, only report MISSING or BAD 
 
    verifyShelveArchives.sh -q
 
  -Verify depot path //depot/Reg/... and print out every every file as it is procesed
 
    verifyShelveArchives.sh //depot/Reg/...

  -Verify depot path with spaces //depot/Reg/spaces in path/...  only report MISSING or BAD 
 
    verifyShelveArchives.sh "//depot/Reg/spaces in path/..."

USAGE
}

#
# Crude options handling..either -q or //depot/path/...
#

verbose=1

while getopts "qh" OPTION; do

    case $OPTION in
    q)
        verbose=0
    shift
        ;;
    h)
        usage
    exit 
        ;;
    esac
done

# Only argument left is depot path if given
if [ $# -gt 0 ] ; then
    depotPath=$*
    changes=$(p4 changes -s shelved "$depotPath" | cut -d' ' -f2)
else
    # Get empty string error if we leave in blank $depotPath with above changes command
    # so generate list of the changes without $depotPath
    changes=$(p4 changes -s shelved | cut -d' ' -f2)
fi

# Set pipefail option to set the exit status $? to the exit code of the last program to exit non-zero 
# (or zero if all exited successfully).
# Used in series of pipes after p4 print to evaluate if any commands in pipe sequence return non-zero; specificallly to detect
# in one pass if archives not found (MISSING)
set -o pipefail


#Loop over the list of shelved changes

for change in $changes
do
        # Cycle through the list of shelved changes looking for the files involved.
        # Using "while read file loop syntax as it handles spaces in filenames and doesn't
        # parse each part of the filename on spaces into separate elements

        # To make output like p4 verify report the revision as well as the file
        # Take entire line of p4 change and parse out file and rev in 2 steps
        # to make handling of spaces in filesnames simpler.

        p4 files @=$change | while read line
        do
        file=$(echo $line | awk  -F# '{print $1}')
         rev=$(echo $line | awk  -F# '{print $2}' | awk ' {print $1}')

                if [ $verbose -eq 1 ] ; then
                        echo Processing Change $change,file $file
                fi

                # Take md5sum of the archive file (what it actually is). Convert md5sum to uppercase to match value stored in Perforce
                # To make it work on windows using cygwin we need to strip the carriage returns in the output of p4 print for text
                # filetypes before calculating digest. Actually it works on Linux leaving in the "tr -d '\r' but to be efficient and
                # just to be on the safe side only strip cr's if on windows

                # Extract lbrFile,digest,headType with comma separator in one call to fstat and store into array 
                IFS=','
                fstat=($(p4 -ztag -F "%lbrFile%,%digest%,%headType%" fstat -Oclf "${file}@=$change"))

                 libFile="${fstat[0]}"
                dbDigest="${fstat[1]}"
                headType="${fstat[2]}"

                # More portable way to check if OS is one of the windows unix shells. Allows script to work with msys that comes with
        # GIT Bash and GOW (GNU On Windows) that doesn't recognise the "if [[ =~" regex syntax

                if echo "$OSTYPE" | egrep '(cygwin|msys|win32)' >/dev/null ; then

                        # ..and similarly, a more portable way to check if filetype contains "text" so we can test for ktext, xtext etc.
                        if echo $headType | grep 'text' >/dev/null ; then
                                archiveDigest=$(p4 print -q "${libFile}@=$change" | tr -d '\r' | md5sum | awk '{print toupper($1)}')
                        else
                                archiveDigest=$(p4 print -q "${libFile}@=$change" | md5sum | awk '{print toupper($1)}')
                        fi
                else
                        archiveDigest=$(p4 print -q "${libFile}@=$change" | md5sum | awk '{print toupper($1)}')
                fi

                # Check return status of pipe, $? gets set due to "set -o pipefail". If it failed then assume archive file not found
                # (see http://unix.stackexchange.com/questions/14270/get-exit-status-of-process-thats-piped-to-another)

                if [ $? -ne 0 ];then
                        echo "MISSING! Change $change, file ${file}#${rev}, archive $libFile"
                elif [ "$dbDigest" != "$archiveDigest" ]; then
                        echo BAD! ${file}#${rev} file digests do not match:change $change, archive $libFile, archiveDigest $archiveDigest, dbDigest $dbDigest
                fi
        done
done

 

Windows batch file example (reports missing archives, but no digest verification)

Save this file to chk_shelved.bat:

@echo off

echo Starting shelved change check: >shelved_err.log
echo -- >>shelved_err.log

for /F "tokens=2 delims= " %%a in ('"p4 changes -s shelved"') do (
	echo Check Shelved change %%a.>>shelved_err.log
	for /F "tokens=1 delims=#" %%b in ('"p4 files @=%%a"') do (
		p4 -Ztag print %%b@=%%a 1> NUL 2>>shelved_err.log
	)
	echo -- >>shelved_err.log
)

echo -- >>shelved_err.log
echo Ending shelved change check. >>shelved_err.log

To use this script, log in as a Perforce super user and enter chk_shelved.bat. When the script completes, the missing shelved files, if any, will be listed in the shelved_err.log file.

Note: Regardless of which script is used (Windows batch or Bash shell), large repositories may take a long time to scan.

Related Links

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255