Perforce Public Knowledge Base - Git Fusion Administrator Utilities
Perforce Software logo
Reset Search
 

 

Article

Git Fusion Administrator Utilities

« Go Back

Information

 
Problem
Git Fusion has many utilities that are useful to both the technical support and Git Fusion administrators.
Solution

Git Fusion Administrator Utilities 

These utilities are to be run by the git OS user on the Git Fusion server

** to display help message, use -h or --help
 
TypeUtilityPurpose
Statusp4gf_lock_status.pyReport the currently held locks in Git Fusion.
Statusp4gf_version.pyReports the Git Fusion, Helix and Component software versions.
Statusp4gf_repo_status.pyReport on the status of Git Fusion repositories.
Statusp4gf_config.pyDisplay the effective global or repository configuration.
Cron Jobp4gf_auth_update_authorized_keys.pyCopies SSH public keys from //.git-fusion/users/{userName}/keys/...
updates key and adds to the Git Fusion user's .ssh/authorized_keys
Cron Jobp4gf_poll.pyUpdate Git Fusion's internal repo(s) with recent changes from Perforce.
Maintenancep4gf_delete_repo.pyDeletes Git Fusion repositories and workspaces.
Maintenancep4gf_init_repo.pyConfigure and populate a new Git Fusion repo.
Maintenancep4gf_gc.pyScan Git Fusion's metadata storage and identify any unreachable objects
that could be garbage-collected via 'p4 obliterate' to reclaim P4D server space.
Diagnosticsgitlog_annotate_with_changenum.pyAnnotate a Git Fusion repo's 'git log' against its Helix changes.
Diagnosticsp4gf_repo_size.pyReport the estimated size for a newly created P4 based Git Fusion repo.
Repairp4gf_rollback.pyPermanently obliterates relevant history of repo at specified changelist. 
Diagnosticsp4gf_gitmirror.pyCheck for missing cache objects.
 

gitlog_annotate_with_changenum.py              

  • Annotate a Git Fusion repo's 'git log' against its Helix changes.
  • Report written to '/opt/perforce/git-fusion/home/perforce-git-fusion/annotate_gitlog_from_files.txt'.

Run with --repo option (16.1 and later)

gitlog_annotate_with_changenum.py --repo test
Gather data and then execute with files supplied as arguments  (15.4 and earlier)
~/.git-fusion/views/test/git$ git log --graph --decorate --oneline --all > ~/gitlog.txt
p4 -p HelixServer:PORT -u git-fusion-user files //.git-fusion/objects/repos/test/commits/... > p4files.txt
p4 changes -l -t //.git-fusion/branches/{repo}/... //depot/xx/... //depot/yy/... > p4changes.txt
Run with required files
gitlog_annotate_with_changenum.py gitlog.txt p4files.txt p4changes.txt

p4gf_auth_update_authorized_keys.py      [back to top]

  • Running configure-git-fusion.sh will add an entry to /etc/cron.d/perforce-git-fusion
  • This updates Git Fusion with new ssh keys that have been added to perforce
  • * * * * * git bash -l -c p4gf_auth_update_authorized_keys.py
  • Copies SSH public keys from Perforce depot //.git-fusion/users/{userName}/keys/... to current user's directory.
p4gf_auth_update_authorized_keys.py -v
NOP. Changes to //.git-fusion/users/*/keys/*=74731 < p4key git-fusion-auth-keys-last-changenum-gf=74731
--rebuild/-r, rebuilds the ~/.ssh/authorized_keys file
p4gf_auth_update_authorized_keys.py -v -r
rebuilding all keys through change 74731
adding file //.git-fusion/users/git/keys/id_rsa.pub
action rebuild, user git, key AAAAB3Nza<snip>RRDGr, FP df:72:95:ea:f3:d3:58:e9:e9:3c:4e:fd:a0:f9:84:d5

p4gf_config.py     [back to top]

Display the global configurations

p4gf_config.py
[repo-creation]
charset = utf8
git-autopack = yes

[git-to-perforce]
change-owner = pusher
changelist-date-source = perforce-submit-time
gitmirror-max-commits-per-submit = 10000

[perforce-to-git]
enable-stream-imports = no
enable-add-copied-from-perforce = yes

[@features]

[authentication]
email-case-sensitivity = no
author-source = git-email

[quota]
limit_commits_received = 0
limit_files_received = 0
Display repo specific configurations and the branch mappings
p4gf_config.py test
[@repo]
description = test repo
charset = utf8
depot-branch-creation-enable = no

[@features]

[main]
git-branch-name = main
view = //depot/test/main/... ...

[dev]
git-branch-name = dev
view = //depot/test/dev/... ...
--set, add repo specific configurations,
Supply section name , a slash (/), then the option name, then an equals sign (=), and then the value
p4gf_config.py test --set @repo/git-autopack=yes
--unset, unsets a configuration for the test repo
p4gf_config.py test --unset @repo/git-autopack

p4gf_delete_repo.py      [back to top]

NOTE:  Strongly recommended to run without the -y flag first to preview what will be deleted.
  • Deletes Git Fusion repositories and workspaces.
  • -a or --all option
    • Git Fusion finds and deletes the following for all repos on the current server disregarding specified views:
      • All git-fusion-view clients.
      • Client git-fusion--p4 workspace files.
      • Objects in //.git-fusion/objects/...
--verbose/-v, reports details of what will be deleted for the repo 'test'
p4gf_delete_repo.py -v test
Checking for client 'git-fusion-gf-test'...
Selecting cached commit objects for 'test'...
rm -rf /opt/perforce/git-fusion/home/perforce-git-fusion/.git-fusion/views/test
rm -rf /opt/perforce/git-fusion/home/perforce-git-fusion/.git-fusion/locks/test
p4 key -d git-fusion-test-gf-last-copied-changelist-number
p4 key -d git-fusion-test-gf-p4gf-config-rev
<snip>
Deleting 1 objects from //.git-fusion/objects/...
p4 group -a -d git-fusion-test-pull
p4 group -a -d git-fusion-test-push
p4 sync -f //.git-fusion/repos/test/p4gf_config*
p4 delete  //.git-fusion/repos/test/p4gf_config*
p4 submit -d "Delete repo config for test" //.git-fusion/repos/test/p4gf_config*
This was report mode. Use -y to make changes.
--execute/-y, Deletes the repo and displays the deletion steps
p4gf_delete_repo.py -y -v test
Checking for client 'git-fusion-gf-test'...
Selecting cached commit objects for 'test'...
Deleting repo test's directory /opt/perforce/git-fusion/home/perforce-git-fusion/.git-fusion/views/test...
Deleting repo test's lock directory /opt/perforce/git-fusion/home/perforce-git-fusion/.git-fusion/locks/test
Deleting group 'git-fusion-test-pull'...
Deleting group 'git-fusion-test-push'...
Deleted 46 files, 2 groups, 0 clients, and 101 p4keys.
Successfully deleted repos:
test

 Deleting the repo and not displaying full details

p4gf_delete_repo.py -y test
Deleted 46 files, 2 groups, 0 clients, and 101 p4keys.
Successfully deleted repos:
test

p4gf_gc.py   

  • Scan Git Fusion's metadata storage
  • Identifies any unreachable objects that could be garbage-collected via 'p4 obliterate' to reclaim P4D server space.
  • Report currently deleted commits, trees, and blobs in the object cache
  • This script is only useful after deleting a repo
  • If you deleted ALL Git Fusion repos, you can skip this script and obliterate all of //.git-fusion/objects/... .

This script is VERY expensive to run.

  • Takes hours to run as scans the ENTIRE Git Fusion object store
  • Takes multiple GB of disk space within current working directory <1GB of memory
  • Does NOT run `p4 obliterate`. Rather it reports files that can be obliterated.
  • The report file is '$PWD/p4gf_gc_unreachables'.
p4gf_gc.py

Perforce: Finding deleted commits : 30
Perforce: 100% (45/45) Syncing trees... --:--:--
Perforce: 100% (48/48) Syncing commits... --:--:--
Perforce: 100% (45/45) Moving cached trees to local git ... --:--:--
Perforce: 100% (48/48) Moving cached commits to local git ... --:--:--
Perforce: 100% (40/40) Traversing commit trees to find cached reachable trees and blobs ... --:--:--
Report of Git Fusion unreachable objects written to: '/opt/perforce/git-fusion/home/perforce-git-fusion/p4gf_gc_unreachables'.
30 deleted objects exist.
6 unreachable objects exist.
Deleting work directory: /opt/perforce/git-fusion/home/perforce-git-fusion/.p4gf_gc

Lapsed time: 0:00:00.283145

Sample output from the file output:  p4gf_gc_unreachables

# DELETED: These Git Fusion object cache files are already deleted.
# They are safe to 'p4 obliterate'.
//.git-fusion/objects/repos/test/commits/05/21/1274390fd71ed9be86ba8f09d81c4c5b677f-main,45
<snip>
# END DELETED: 30 deleted objects exist.
# UNREACHABLE: These existing Git Fusion object cache files are not reachable by any existing commit.
# They are safe to 'p4 obliterate'.
//.git-fusion/objects/trees/6f/c4/46d757b4fc5a9bedf3c96d82ad1291cba779
<snip>
# END UNREACHABLE: 6 unreachable objects exist.

p4gf_init_repo.py      [back to top]

  • Configure and populate a new Git Fusion repo.
  • Copies history from Perforce to Git, which can take a while for large histories, unless the --noclone option is used                  
  • This requires a repo-name to be supplied
    • if the --config is omitted, there must be an existing Perforce client spec of the same name of the repo or the repo config (p4gf_config) already in place.           
  • A default configuration file is located in /opt/perforce/git-fusion/libexec/p4gf_config.repo.txt
Creating Repo From a Configuration File
 
Example configuration file
p4gf_config.py test1
[@repo]
description = test1 repo
charset = utf8
depot-branch-creation-enable = all

[@features]

[master]
git-branch-name = master
view = //depot/test1/... ...
Example of the --config option
  • Cannot be used with --charset.
  • Cannot be used with --p4client.
p4gf_init_repo.py --config /opt/perforce/git-fusion/home/perforce-git-fusion/configs/test1.cfg test1
--start option
  • Start Git history at changelist <n>.
  • If omitted, Git history starts with the first changelist in this repo's view of Perforce.                 
  • Cannot be used with --noclone.
p4gf_init_repo.py --config /opt/perforce/git-fusion/home/perforce-git-fusion/configs/test1.cfg --start 74794 test1
--noclone option
  • Do not immediately populate the Git repo with content from Perforce
  • Cannot be used with --start.
p4gf_init_repo.py --config /opt/perforce/git-fusion/home/perforce-git-fusion/configs/test1.cfg --noclone test1
Initializing 'test1'...
Initialization complete.
Creating Repo From a Client Spec
  • Client Spec:  note only the client name and view are used by Git Fusion
  • creates a single Git branch 'master', associated with the view in Perforce client spec <repo-name>
  • Cannot be used with --charset or --p4client
Client Spec
Client: test1
Update: 2016/10/14 15:07:16
Access: 2016/05/24 10:57:41
Owner:  test1
Description:
        Created by test1.
Root:   /Users/workspace/test1
Options:        noallwrite noclobber nocompress unlocked nomodtime normdir
SubmitOptions:  submitunchanged
LineEnd:        local
View:
        //depot/test1/... //test1/...
Create the Repo from the client spec
p4gf_init_repo.py  test1
Initializing 'test1'...
Initialization complete.
Copying changes from '1'...
Perforce: Copying files: 2
Perforce: 100% (1/1) Copying changelists to git...
Perforce: Submitting new Git commit objects to Perforce: 2
Copying completed.
Git Clone to Create the Repo
git clone git@GFHost:test1
Cloning into 'test1'...
Perforce: Copying files: 2
Perforce: 100% (1/1) Copying changelists to git...
Perforce: Submitting new Git commit objects to Perforce: 2
remote: Counting objects: 4, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 4 (delta 0), reused 2 (delta 0)
Receiving objects: 100% (4/4), 291 bytes | 0 bytes/s, done.
Checking connectivity... done.

p4gf_lock_status.py            [back to top]

  • reports the currently held locks in Git Fusion
  • Please see KB article:  Git Fusion Repo Locks
    • p4 key based locks (instance and repo)
    • file based locks (read/write)
Locks held during a clone
p4gf_lock_status.py 
Connecting to P4PORT=HelixServer:1666 as P4USER=git-fusion-user 
File based locks for 'repo1'... 
  Writer User: test 
  Writer PID: 20199, status LIVE
Locks held during a push
p4gf_lock_status.py
Connecting to P4PORT=HelixServer:1666 as P4USER=git-fusion-user
***************** repo1 Status  *****************
Most recent push for repo 'repo1'... '1'
Most recent push status for repo 'repo1'... 'Push 1 started'
P4 key based locks for 'repo1'...
  Owning username: test
  Owning instance: gf
  Initial process: 20800
  Owner #1: PID 20800, started at 2016-10-13 12:37:37, status LIVE
  ==> Process 20800 is still running. Do not release this lock until that process ends.

File based locks for 'repo1'...
  Writer User: test
  Writer PID: 20800, status LIVE
 --reset, removes all stale locks for the specified repo using the --reset option (2016.2 and later)
p4gf_lock_status.py --reset test1
Connecting to P4PORT=HelixServer:1666 as P4USER=git-fusion-user
server_id

p4gf_poll.py      [back to top]

  • Update Git Fusion's internal repo(s) with recent changes from Perforce.
  • Generally added to the /etc/cron.d/perforce-git-fusion to update read-only Git Fusion instances
     --verbose/-v, list each repo updated
p4gf_poll.py -v test1
Updating: test1
     --all/-a, update all of the Git Fusion's internal repos with recent changes
p4gf_poll.py -v --all 
Updating: test1  
Updating: ...
     --repack/-p, Invoke git-repack -ad on the repository
p4gf_poll.py -v --repack test1
Updating: test1
Counting objects: 197, done.
Compressing objects: 100% (94/94), done.
Writing objects: 100% (197/197), done.
Total 197 (delta 64), reused 197 (delta 64)

p4gf_gitmirror.py      [back to top]

  • Check for missing mirror objects.
  • Repair the mirror.
     Report problems with the mirror
 p4gf_gitmirror.py -v <repo>
 Processing repository <repo>...
 missing commit 28fd38d591cccf88ca6ab4bd4472799f1798d10b-main,81190
 missing commit a15a5be1c2d458da8807e487c386f8bb01588de4-main,55806 missing commit 972b46679bd1f24fb2ccd5d9d10d837716aa09c3-rel16.1,55032
     --fix, repair problems in the mirror (2016.3+)
 p4gf_gitmirror.py -v --fix <repo>

p4gf_rollback.py      [back to top]

  • Reset Perforce and Git Fusion records to  an earlier time based on --change-num supplied.
  • Use to reset Git Fusion's copy of a repo to an earlier state.            
     Preview mode, -y not included
p4gf_rollback.py -u git-fusion-user --repo test --change-num 81118
Preview mode. Use --execute/-y to remove history.
# Delete commit object file(s) : 1
# Delete commit p4key(s) : 1
# Set last-change-num p4key(s) : 1
# Move/delete Git branch reference(s) : 1
     --verbose/-v, preview mode, suggest to write to file and STOUT as generates a large amount of details
p4gf_rollback.py -u git-fusion-user --repo test --change-num 81118 -v 2>&1 | tee test_rollback.out 
     --execute/-y, deletes only from Git Fusion
p4gf_rollback.py -u git-fusion-user -y --repo test --change-num 81101
      --obliterate, obliterates every changelist/commit after the specified -change-num.
p4gf_rollback.py -u git-fusion-user -y --obliterate --repo test --change-num 81101

p4gf_repo_size.py      [back to top]

  • Report the estimated size for a newly created P4 based Git Fusion repo.
  • Requires --config to supply a p4gf_config input data file.
  • Requires Helix user account -u p4user  (can run as the git-fusion-user)
p4gf_repo_size.py -u git-fusion-user --config ./configs/test1.cfg

Repo config: ./configs/test1.cfg
File revision count:                                 29

git clone size:
  Commit count:                                        20
  Commit objects size*:                            0.006M
  Tree count:                                          20
  Tree objects size*:                              0.010M
  Git total uncompressed file data:                0.001M
  Git total data:                                  0.016M

P4 data added:
  Commit count:                                        20
  Commit objects size*:                            0.006M
  Tree count:                                          20
  Tree objects size*:                              0.010M
  P4 key count:                                        40
  P4 key data:                                     0.002M
  P4 total data:                                   0.018M
                                           * using average commit size = 300
                                           * using average tree  size = 500
--start to start at a particular changelist
Updates all of the Git Fusion's internal repos with recent changes
p4gf_repo_size.py -u git-fusion-user --config ./configs/test1.cfg --start 65000

p4gf_repo_status.py      [back to top]

  • Report on the status of Git Fusion repositories.
  • Includes git fsck
Report on a specific repo
p4gf_repo_status.py test1
Processing repository test1...
/opt/perforce/git-fusion/home/perforce-git-fusion/.git-fusion/views/test1/git/.git
--all/-a option to report on all repos
p4gf_repo_status.py --all
Processing repository test1...
git-fusion-index-last-test4,master key appears to be unset
Push status appears incomplete: Copied 674 of 924 commits for push 4...
git-fusion-index-last-<repo>,<branch> appears out of sync with git-fusion-<repo>-<server>-last-copied-changelist-number
Processing repository test7...
Push status appears incomplete: Push 3 failed: 'NoneType' object has no attribute 'find_depot_branch'
p4gf_config/2 is missing branch named "version/16.1

p4gf_version.py                 [back to top]

  • Reports on the installed version of Git Fusion
  • Reports on the Helix database
  • Reports on the versions of the other required software components
p4gf_version.py
Perforce - The Fast Software Configuration Management System.
Copyright 2012-2016 Perforce Software.  All rights reserved.
Rev. Git Fusion/2016.2/1398420 (2016/06/07).
SHA1: 29b255f5ec92adb0685437883edadde194a6bc21
Git: git version 2.7.3
Python: 3.3.2
P4Python: Rev. P4PYTHON/LINUX32X86_64/2015.1/1199407 (2015.1/1127721 API) (2015/07/07).
uname: Linux Git_Fusion_Server 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Git Fusion path: /opt/perforce/git-fusion/libexec
Distributor ID:    Ubuntu
Description:    Ubuntu 14.04 LTS
Release:    14.04
Codename:    trusty
p4 info:
User name: git
Client name: GF_0xS_Sx0_FG
Client host: Git_Fusion_Server
Current directory: /opt/perforce/git-fusion/home/perforce-git-fusion
<snip>
Server version: P4D/LINUX26X86_64/2016.1/1396108 (2016/05/10)
ServerID: server.company.com
Server services: standard
Server license: 2000 users (expires 2016/12/29)
Server license-ip: 11.1.11.111
Case Handling: sensitive
Unicode: disabled
Related Links

Feedback

 

Was this article helpful?


   

Feedback

Please tell us how we can make this article more useful.

Characters Remaining: 255