Github: Difference between revisions
| Line 551: | Line 551: | ||
A leading slash indicates that the ignore entry is only to be valid with respect to the directory in which the .gitignore file resides. Specifying *.o would ignore all .o files in this directory and all subdirs, while /*.o would just ignore them in that dir, while again, /foo/*.o would only ignore them in /foo/*.o. | A leading slash indicates that the ignore entry is only to be valid with respect to the directory in which the .gitignore file resides. Specifying *.o would ignore all .o files in this directory and all subdirs, while /*.o would just ignore them in that dir, while again, /foo/*.o would only ignore them in /foo/*.o. | ||
=== | === Whitelist === | ||
https://stackoverflow.com/a/44568826 | <ul> | ||
<li>[https://www.cloudsavvyit.com/14444/how-to-set-up-gitignore-as-a-whitelist/ How to Set Up .gitignore As a Whitelist] | |||
<pre> | |||
* | |||
!*/ | |||
# track this file | |||
!.gitignore | |||
# whitelist everything in ./config/ | |||
!config/ | |||
</pre> | |||
</li> | |||
<li>[https://stackoverflow.com/a/44568826 Gitignore everything except a subfolder] | |||
<pre> | <pre> | ||
* | * | ||
| Line 559: | Line 572: | ||
!Also/This/Another/Folder/** | !Also/This/Another/Folder/** | ||
</pre> | </pre> | ||
</li> | |||
</ul> | |||
== git commit message == | == git commit message == | ||
Revision as of 20:53, 12 October 2021
Git
Resources
- Git Community Book
- Git Documentation & Pro Git book
- GitHub.com Help Documentation
- Github & Git Foundations Training from youtube.
- git/github guide a minimal tutorial from Karl Broman
- Brief Introduction to Git by Luke Tierney
- Some examples of projects hosted in github: RHadloop, shiny, Qt.
- http://www.atlassian.com/git/tutorial/undoing-changes
- Git reference (5 pages of summary)
- Distributed Version Control with Git (nice tutorial including Github, BitBucket. )
- Cheat sheet:
- github.com (2 pages)
- byte.kde.org (font too small)
- git-tower.com (1 page as an image)
- atlassian.com (immediate download)
- jan-krueger.net (cool too)
- patrickzahnd.ch (visual cheat sheet)
- https://www.kernel.org/pub/software/scm/git/docs/everyday.html
- http://haacked.com/archive/2012/05/21/introducing-github-for-windows.aspx
- Delete a repository
- A Quick Introduction to Version Control with Git and GitHub
Github profile
Customize Your GitHub Profile With a ReadMe
git config
This has been run the very first time you use git commit.
git commit (commit to the local repository) will ask username/password if we have not logged in (that is, this step has been done before we run git commit). The identity will be saved under ~/.gitconfig file.
git config --global user.name "Your Name Here" git config --global user.email [email protected]
OR to have a project specific git identity (this will write your identify in ./.git/config file)
git config user.name "Your Name Here" git config user.email [email protected]
Type
git config --list
to confirm the configuration.
Another way: Tweak your Git config for multiple user IDs
Privacy email
- Setting your commit email address
- Private emails, now more private
- using github’s noreply email address. <username>@users.noreply.github.com
git stage area
If you have run git add, git rm or git mv command, you put some files in the stage area. In other words git knows about the change, but it is not permanent in the repository. The next commit will include the changes staged.
Should you decide not to commit the change, the status command will remind you that you can use the git reset command to unstage these changes.
For example, if your IDE accidentally deletes a file for you, you can run git reset HEAD FILENAME and git checkout FILENAME to recover the file.
'git mv' vs 'mv'
If we use 'git mv SOMEFILE NEWILE', we will not be able to use 'git checkout SOMEFILE'.
If we try to do that, we will get a message "error: pathspec 'SOMEFILE' did not match any file(s) known to git".
git push, git clone and remote repository
For a new repository, use git remote add origin (not git remote set-url) before calling git push.
git remote add origin https://github.com/USERNAME/REPOSITORY.git # 'origin' is an alias of the long URL git remote add origin [email protected]:USERNAME/REPOSITORY.git
git push (push to a remote repository) will ask the username and/or password. To avoid asking the username part,
git remote set-url origin https://name:[email protected]/name/repo.git git remote set-url origin https://[email protected]/name/repo.git # skip password for security git remote -v cat .git/config
OR when you run git clone, use the format
git clone https://[email protected]/name/repos.git git clone https://[email protected]/name/repos.git repos2 # use 'repos2' as the directory name # git clone [email protected]:username/repos.git git remote -v
Either way, we can check the the remote repository by using git config --list or cat .git/config.
- When I commit (git commit) a directory containing resource file, it shows warning: LF will be replaced by CRLF in xxxx.qrc.
- Password cache
- github tutorial (for statistician)
- git add -A
git config --global credential.helper cache
git clone when no 'master' branch exists
Warning: remote HEAD refers to nonexistent ref, unable to checkout
In my case, I need to run
git clone ssh://[email protected]/home/git/seqtoolsweb.git --branch gh-pages --single-branch seqtoolsweb
A local copy of origin/master
When we do 'git push', git will automatically create a copy of origin/branchName on our local computer. So origin/branchName is a branch on our local machine that references the remote server branch.
Use 'git fetch' to sync our local copy of origin/branchName with remote's branchName.
Github wiki pages
Documenting your project with wikis from the official Github Help website.
See here for git clone the wiki repository.
git clone https://[email protected]/myusername/foobar.wiki.git git checkout $(git rev-list -n 1 HEAD -- yourPage)^ -- yourPage
It works on the motionEye wiki page. git clone https://github.com/ccrisan/motioneye.wiki.git
git fork
First example
My example of working on a new repository called 'network'.
- Follow https://help.github.com/articles/create-a-repo to create a new repository. For convenience, I also check the button to create README file.
- Click 'GitHub' icon on Windows Desktop. Look at the LHS and click on the word 'github'. Click 'clone' button (This can be accomplished by git clone https://github.com/arraytools/network.git). The new repository will appear under C:\Users\USERNAME\Documents\Github\ directory. Now Click 'Git Shell' icon on the Windows Desktop and go to C:\Users\USERNAME\Documents\Github\network directory where 'network' is my repository's name.
git config --global color.ui auto # colorize the output of git git init git add client.c git add server.c git add server2.c git commit -m 'first commit' git status # On branch master # Your branch is ahead of 'origin/master' by 1 commit. # git remote add origin https://github.com/arraytools/network.git # git push <remote> <branch> git push origin master # It seems 'git push' is the only chance we use the format 'origin master' instead of 'origin/master'. # If we modify any file, we need to go through git add, git commit and git push 3 commands. # get commit_id git log # get a specific version git checkout commit_id # You are in 'detached HEAD' state # after an examination, we want to get the latest version git pull origin master # If you do not want to merge the upstream changes wit your local repository, run git fetch to # fetch the changes and then git merge to merge the changes. # git pull is just a combination of fetch and merge. # To rename a file git mv originafile newfile git commit -m "rename orginalfile" git push # To remove a file rm myfile git add . -A git commit -m "remove a file" git push # Revert to origin's master branch's version of file # http://stackoverflow.com/questions/1817766/revert-to-origins-master-branchs-version-of-file # 1. Assuming you did not commit the file, or add it to the index, then: git checkout filename # 2. Assuming you added it to the index, but did not commit it, then: git reset HEAD filename git checkout filename
Check the https://github.com/arraytools/network. The commit goes to the repository!
In summary: add and commit are local operations, push, pull and fetch are operations that interact with a remote.
If we want to checkout a specific commit on a new computer, we can use (here we use Qt repository as an example)
git clone https://xxxxxx/xxxxxx/Qt.git cd Qt git log --oneline git checkout SHA1 (7 digits obtained from git log --oneline commandis enough)
After that we can run
# move HEAD to origin
git checkout origin/master # You are in 'detached HEAD' state
# I should use "git checkout master" to go back to my local master
# OR
# use something like "git checkout -b test origin/master"
# Visualize using text mode
git log --graph --oneline --date-order --decorate --color --all
Using GUI client: gitg
sudo apt-get install gitg
To switch among different branch (eg. github project page is located in gh-pages branch of a repository),
git checkout gh-pages git checkout master
GUI version of Git software
Should You Use a GUI Git Client?
Windows
Go to http://git-scm.com/download. The Windows version contain 'Git Bash', 'Git CMD', and 'Git GUI'. The 'Git GUI' software (based on Tcl/Tk) works pretty cool. It can 'Rescan' the project, compare the changed filefs and visualize the master's history too. The Git comes with 2 built-in tools: Git-gui is for committing and gitk is for browsing. Screenshots of gitk can be found below.
Note that Git-gui cannot run git pull directly. We have to go through two steps: Fetch and merge. See this message.
When installing Git for Windows, there's a tricky option you need to be careful with. You can configure Git to change line endings on text files from Linux (LF only) to/from Windows-DOS (CR and LF).
Linux
Use gitk or gitg (better than gitk on gnome based Ubuntu). For example,
gitk --all
If we run the gitk/gitg in background (gitg --all &), we will see the tree changed in real time when we run git commands in a terminal.
The problem with gitk is the font is awful.
gitg program (install it by sudo apt install gitg) looks nice.
GitKraken needs to sign in with Github or Gitkraken account.
git-cola: I am lost:(
Mac
GitHub Desktop (Electron based)
Gitk colors and bold
The missing gitk documentation.
- a yellow dot ○ marks the current HEAD (it seems the yellow dot and bold branch name are always together). Note that if we use "git log" to show the history in the command line, the HEAD is shown in cyan color. See Use gitk to understand git.
- local branch names are in a green background. See Use gitk to understand git.
- remote branch names are in a mixed orange/green background. See Use gitk to understand git.
- the currently checked out branch name is in bold
- tags are on a yellow background
See a screenshot at Branch - Pull.
ssh key
See Linux > Multiple ssh keys.
- Github
- Bitbucket. Remember to change the remote repository from https protocol to git protocol on local repository.
If you add a new key to your bitbucket account, you will receive an email about the change.
Github
Adding your SSH key to the ssh-agent
$ eval "$(ssh-agent -s)" $ ssh-add -K ~/.ssh/idName_rsa
Multiple git accounts
https://www.cloudsavvyit.com/2245/how-to-manage-multiple-git-accounts-on-one-system/ How to Manage Multiple Git Accounts on One System]
Use command line to create a remote repository
Is it possible to create a remote repo on GitHub from the CLI without opening browser?
curl -F 'login=username' -F 'token=API Token' \ https://github.com/api/v2/yaml/repos/create -F name=reponame
Set up a new local/remote repository
mkdir /path/to/your/project cd /path/to/your/project git init git remote add origin https://[email protected]/arraytools/REPOSITORYNAME.git # git remote add origin ssh://[email protected]/home/git/REPOSITORYNAME.git # git clone ssh://[email protected]/home/git/REPOSITORYNAME.git git config --global user.name "YOUR NAME" git config --global user.email "YOUR EMAIL ADDRESS" git config --list # confirm echo "arraytools" >> contributors.txt git add contributors.txt git commit -m 'Initial commit with contributors' git push -u origin master
Already has a git repository on my computer
cd /path/to/my/repo git remote add origin https://[email protected]/arraytools/REPOSITORYNAME.git git push -u origin --all # pushes up the repo and its refs for the first time git push -u origin --tags # pushes up any tags
Rename a remote repository
- Go to github.com, open the project and click Settings button on the left-bottom corner. Change the repository name on top.
- On local machine, rename the directory. Go to the directory. Issue
git remote -v
to get the ULR for the current working copy. Suppose the url is [email protected]:someuser/someproject.git. Now issue the following command to change to the new repository
git remote set-url origin [email protected]:someuser/newprojectname.git
Suppose we have added ssh key to git server and we want to use ssh key/protocol to automatically access the server instead of entering the password (https protocol). See
$ # the REMOTE below could be 'origin' or any name you define $ git remote show REMOTE $ git remote set-url REMOTE ssh://[email protected]/USERNAME/REPOS.git $ git remote show REMOTE $ git status # show local branch is update-to-date with REMOTE/BRANCH.
'master' and 'origin'
- master is a branch name. You can use git branch to find out all branches. The current branch has a asterisk in the command line output and has a bold font in the gitk program.
- origin is a repository name. You are free to create a new one and delete origin especially in situation that you are working with multiple remotes. That is, if you run git clone -o booyah instead, then you will have booyah/master as your default remote branch. See Remote Branches or Use gitk to understand git.
Change GitHub default branch from master to main
5 steps to change GitHub default branch from master to main
'origin master' vs 'origin/master' format
- origin master format: git push, git pull, git fetch where the branch name is optional.
- origin/master format: branch, diff, log, merge, reset but don't do it on checkout unless we create a new branch; eg git checkout -b newBranch origin/newBranch. Note that origin/master is a local copy of the branch named "master" on the remote named "origin".
See stackoverflow.
Multiple remotes
Keep track of multiple Git remote repositories
Suppose I have a remote at github.com. I add another remote (bitbucket.com) in my current project.
$ git status On branch master Your branch is up-to-date with 'origin/master'. $ git push -u bitbucket master $ git status On branch master Your branch is up-to-date with 'bitbucket/master'. $ git branch -a * master remotes/bitbucket/master remotes/origin/HEAD -> origin/master remotes/origin/master $ git checkout origin/master # seems not what I want to do Note: checking out 'origin/master'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b new_branch_name HEAD is now at 980fd0b... test fetch +1 $ git status HEAD detached at origin/master $ git checkout master Switched to branch 'master' Your branch is up-to-date with 'bitbucket/master'. $ git status On branch master Your branch is up-to-date with 'bitbucket/master'. nothing to commit, working directory clean $ nano linuxfile $ git commit -am "multiple remote +1" $ git push origin master $ git status On branch master Your branch is ahead of 'bitbucket/master' by 1 commit. (use "git push" to publish your local commits) nothing to commit, working directory clean $ # Question: How to switch back to origin repository to sync? $ # How do I change the remote a git branch is tracking? $ # Method 1: "-u" in git push; see next section about "-u" $ git push -u origin master Branch master set up to track remote branch master from origin. Everything up-to-date $ git status On branch master Your branch is up-to-date with 'origin/master'. nothing to commit, working directory clean $ # Method 2: "-u" in git branch; see next section about "-u" $ git branch master -u bitbucket/master Branch master set up to track remote branch master from bitbucket. $ git status On branch master Your branch is ahead of 'bitbucket/master' by 1 commit. (use "git push" to publish your local commits) nothing to commit, working directory clean
How to know which remote's master branch to pull/push
$ git status # On branch master # Your branch is up-to-date with 'bitbucket/master'
Import/move/migrate (not clone) a repository from one remote to another remote
NB: bitbucket allows to use its web interface to import a remote repository. We just need to give it the URL (eg from Github) and the new repository name. I can use this way to back up some public repositories from github.
Suppose the new remote is on bitbucket.
rsync -av DIRECTORY . cd SUBDIRECTORY git remote -v # Examine the current URL git remote set-url origin [email protected]:USERNAME/DIRECTORY.git git remote -v # Examine the new URL git push -u origin --all git push origin --tags
Other instructions
- https://stackoverflow.com/questions/1484648/how-to-migrate-git-repository-from-one-server-to-a-new-one
- https://www.smashingmagazine.com/2014/05/moving-git-repository-new-server/
# fetch all of the remote branches and tags $ git fetch origin $ git branch -a * master remotes/origin/develop remotes/origin/master remotes/origin/release/0.1 $ git checkout -b develop origin/develop $ git checkout -b release/0.1 origin/release/0.1 $ git branch -a develop master * release/0.1 remotes/origin/develop remotes/origin/master remotes/origin/release/0.1 # all branches in our repository are stored locally, # we are ready to move the repository to a new host # assume we have an SSH-cloned URL of our new repository $ git remote add new-origin [email protected]:manakor/manascope.git # This will give us two remotes for the existing repository: # the original one (named origin, connected to the existing remote host) and # a new one (named new-origin, connected to the new host). # pushing all branches at once $ git push --all new-origin $ git push --tags new-origin # make new-origin the default remote, $ git remote rm origin $ git remote rename new-origin origin
How to keep two Git repositories in sync
How to keep two Git repositories in sync
$ git clone --mirror https://primary_repo_url/primary_repo.git $ cd primary_repo.git $ git remote add --mirror=fetch secondary https://secondary_repo_url/secondary_repo.git $ git fetch origin $ git push secondary --all
-u option
The key is "argument-less git-pull". When you do a git pull from a branch, without specifying a source remote or branch, git looks at the branch.<name>.merge setting to know where to pull from. git push -u sets this information for the upstreaming branch you're pushing.
“git push -u origin master” vs “git push origin master”
What does the '-u' do in git push. The message: most of time we want to use "-u" option.
Adding '-u' means our local master is tracking remote's master branch. Run "cat .git/config" and look at the [branch 'master'] section -> the 'remote' key. It should have a value 'origin'. The 'merge' key has a value 'refs/head/master'. These information are added automatically when we clone a remote repository.
If we do not put "-u" in git push, it would not be a tracking branch. We can manually edit 'git/config' file or use git branch --set-upstream-to or just "-u" and the branch name (check out git help branch). For example, "git branch -u origin/newbranch newbranch" where newbranch is a local branch name.
Also we can untrack a remote branch by using "git branch --unset-upstream newbranch".
git branch
The git branch command can be used to connect a local branch and a remote branch.
$ git status # suppose the local master is connected to bitbucket/master $ git branch master -u origin/master # now the local master is connected to origin/master # the local branch name and the remote branch name can be different if you like! $ git status
Note that gitk can not see the difference when we change the connection of a local branch and a remote branch. Only the git status can reveal the connection.
Example
Setup editor
See man git-commit.
git config --global core.editor "nano" # OR export GIT_EDITOR=nano # OR for other programs to use too export EDITOR=nano
After that
$ cat ~/.gitconfig
[core]
editor = nano
Multiline commit messages
Once we issue "git commit -a", it opens a text editor. When we write the messages, we want the first line a simple description, the 2nd line empty and the rest of lines a more detailed message. With this way, when we use "git log --oneline", the message is short and we can still use "git log" to see the full message.
.gitignore file
- http://stackoverflow.com/questions/8527597/how-do-i-ignore-files-in-a-directory-in-git
- Use "!" to negate. For example, the following rule will ignore the directory 'packrat' but still keep the file 'packrat/packrat.lock'.
packrat/* !packrat/packrat.lock
- Microsoft Office temporary files. It works if we haven't made a mistake to run commit.
git help gitignore # or man gitignore
A leading slash indicates that the ignore entry is only to be valid with respect to the directory in which the .gitignore file resides. Specifying *.o would ignore all .o files in this directory and all subdirs, while /*.o would just ignore them in that dir, while again, /foo/*.o would only ignore them in /foo/*.o.
Whitelist
- How to Set Up .gitignore As a Whitelist
* !*/ # track this file !.gitignore # whitelist everything in ./config/ !config/
- Gitignore everything except a subfolder
* !*/ !That/Very/Folder/** !Also/This/Another/Folder/**
git commit message
- The seven rules of a great Git commit message
- Udacity Git Commit Message Style Guide
- How to Write Good Commit Messages: A Practical Git Guide
- How to Write a Git Commit Message, in 7 Steps
- Separate subject from body with a blank line
- Limit the subject line to 50 characters
- Capitalize the subject line
- Do not end the subject line with a period
- Use the imperative mood in the subject line
- Wrap the body at 72 characters
- Use the body to explain what and why vs. how
- A commit message should answer three primary questions;
- Why is this change necessary? (most important)
- How does this commit address the issue?
- What effects does this change have?
git status, multiple branches
Your branch is up-to-date with 'origin/master'
$ git status On branch master Your branch is up-to-date with 'origin/master'. nothing to commit, working directory clean
When git status says up to date, it means "up to date with the upstream status that was retrieved last time we did a fetch" which is not the same as "up to date with the latest live status of the upstream".
nothing to commit, working directory clean
If we have a new branch on local which does not exist on remote, it is useful to use the git remote show REMOTENAME command to find out what branches are tracked.
$ git status On branch newbranch nothing to commit, working directory clean $ git remote show bitbucket * remote bitbucket Fetch URL: ssh://[email protected]/XXXXXX/toy.git Push URL: ssh://[email protected]/XXXXXX/toy.git HEAD branch: master Remote branch: master tracked Local ref configured for 'git push': master pushes to master (up to date) $ git diff bitbucket/master diff --git a/linuxfile b/linuxfile index 894674f..b9f3a6f 100644 --- a/linuxfile +++ b/linuxfile @@ -1,6 +1,6 @@ test ssh key (mod in mint again) changed from 'newbranch' -another line +mod from newbranch $ git push bitbucket master Everything up-to-date
Note that
- The last command is NOT working since only the local/master can be pushed to remote/master. Local/newbranch cannot be pushed to remote/master.
- The local 'master' branch can be pushed to either remote1/master or remote2/master.
Remove untracked files
Go to the top directory of the local repository (or it won't work), then run
$ git clean -f -n # dry run $ git clean -f # real run
See the discussion.
Back up untracked files
7 Git tricks that changed my life
$ git ls-files --others --exclude-standard -z |\
xargs -0 tar rvf ~/backup-untracked.zip
Remove deleted files
git ls-files --deleted git rm `git ls-files --deleted`
Undo/Reset
https://git-scm.com/book/en/v2/Git-Basics-Undoing-Things
git commit --amend
If you commit too early and possibly forget to add some files, or you mess up your commit message.
Assumption: Commit has not been pushed online
$ git commit -m 'initial commit' $ git add forgotten_file $ git commit --amend # In your text editor, edit the commit message and save the commit.
You end up with a single commit.
If the commit has been pushed online, see https://help.github.com/articles/changing-a-commit-message/.
Unmodifying a Modified File (not staged)
$ git checkout -- Filename # no output $ git checkout -- Dirname # no output $ git checkout -- '*.R' $ git checkout -- . # All unstaged files $ git stash save --keep-index # http://stackoverflow.com/questions/52704/how-do-you-discard-unstaged-changes-in-git/52713#52713
See also git checkout --FileName.
Unstaging a Staged File
$ git reset HEAD Filename $ git reset # unstage all due changes
Undo git pull or merge
http://stackoverflow.com/questions/101752/i-ran-into-a-merge-conflict-how-can-i-abort-the-merge
$ git reset --hard HEAD $ git reset --hard HEAD^ # discard the last commit and will be abandoned eventually after git did a garbage collection $ git reset --hard <tree-ish>
Undo git commit
We can use git revert or git reset. See Undoing Commits & Changes from Atlassian.
- Best for public shared repositories
$ git revert --no-edit 6d8dafe # using the first 7 characters of the identifier is more than enough # Previous commits still be kept
- Best for local private changes
$ git reset --soft HEAD~1 # reset the HEAD (the last commit) to one commit before in the log history. $ git reset --hard a1e8fb5 # Some middle commits will no longer exist in the commit history $ git reset --soft HEAD~ # --soft means reset into staging # HEAD~ or HEAD^ means to move to commit before HEAD # http://stackoverflow.com/questions/927358/how-do-you-undo-the-last-commit $ git reset --soft HEAD^^ # move to the grandparent commit
Undo git push
http://stackoverflow.com/questions/1270514/undoing-a-git-push
git checkout master git reset --hard cc4b63bebb6 git push origin +master
change commit message
https://help.github.com/articles/changing-a-commit-message/#commit-has-not-been-pushed-online
To change the last committed message
git commit --amend # Enter and save git push --force origin master
Remove a file from a Git repository without deleting it from the local filesystem
git rm --cached mylogfile.log # file git rm --cached -r mydirectory # directory
Git reset revert rebase commands
- How to reset, revert, and return to previous states in Git
- Lynda.com -> Git: Branches, Merges and Remotes -> Reset Branches
Delete any untracked files or directories
How to remove local (untracked) files from the current Git working tree?
# Print out the list of files which will be removed (dry run) git clean -n # Delete the files from the repository git clean -f # To remove directories, run git clean -f -d # OR git clean -fd # To remove ignored files, run git clean -f -X # OR git clean -fX
git clone vs git pull
What is the difference between pull and clone in git?
git fetch vs git pull
git fetch origin master git merge origin/master # git push origin master
git fetch
Use git fetch as often as possible; before git push or before you go offline or before you work.
$ git fetch $ git log --oneline -5 origin/master $ git branch # local only $ git branch -r # remote only $ git checkout master $ git merge origin/master $ git log --oneline master
git fetch vs git pull
- If I am in the middle of committing something and get some work going on, I will do a git fetch to see what's on the server.
- If I am starting my work without making any changes/commits yet, I'll do a git pull. I want every thing on the server as my starting point.
git fetch and gikt
After we run git fetch origin master, we will see the remote commits from gitk if we add --all option.
gitk --all # OR gitg --all
git show: show details on a commit
git show SHA
Use 'f' to move forward and 'b' to move backward.
git log
log in a branch or origin/master
View log from origin/master (it's a tracking copy on our local computer; it's not the actual remote branch)
$ git log myBranch $ git log --oneline -5 origin/master # last 5 commits from origin/master
git log on a specific file
git log FILENAME
git show change history of a file
Terminal
git log -p FILENAME
GUI
gitk FILENAME
git log show changed files only
How to View Commit History With Git Log
git log --name-only git log --stat # show how many lines of changes in files
git log message is too long
Normally when we use 'git log' command, long log messages get truncated.
Use the following to view the complete log messages instead of truncated log messages.
git log | less
Common standard is 78 characters. See this and this posts.
filtering by a date
git log --since=2019-01-01 git log --until=2019-01-01 git log --until="3 days ago" git log --after=2.weeks --before=3.days
filter by a name
git log --author="xxx" # author string contains "xxx"
filter by a keyword in log
Search for a string in the log
git log --grep='String'
filter by commits
git log bd210165..HEAD git log <SHA>..<SHA>
Count number of commits
git rev-list --count master
Best ones: adding --graph --all
git log --graph --all --stat # just see lines of changes, sha1 is complete git log --graph --all --stat --abbrev-commit # same as above but sha1 is short git log --graph --all --oneline --decorate # sha1 is short, but the log message could be very very long
Search in the code?
- How to Search Through Recent Git Commit Changes
- Navigating your Git History
- git log -S 'some words' # search for the string in the code and show the logs
- Is it better to use git grep than plain grep if we want to search in versioned source code?
- You can search in Git even if you aren't sure which commit—or even branch—you made your changes. See 7 Git tricks that changed my life
$ git rev-list --all | xargs git grep -F 'KEYWORD'
branch - not the same as in CVS
- https://lostechies.com/joshuaflanagan/2010/09/03/use-gitk-to-understand-git/
- 15 Git Branch Command Examples to Create and Manage Branches
- https://www.atlassian.com/git/tutorials/using-branches. git branch is tightly integrated with the git checkout and git merge commands.
Gitk shows all of the commits as a single straight line. In git, a branch is a label for a commit. The label moves to new commits as they are created. When you create a git branch, you are not changing anything in the structure of the repository or the source tree. You are just creating a new label.
Below are some info borrowed from Pro Git.
list local and remote branches
mac$ git branch # list branches in local repo branch0323 * master mac$ git branch -a # list all remote branches branch0323 * master remotes/origin/HEAD -> origin/master remotes/origin/master
"--track" option
When you clone a repository, it generally automatically creates a master branch that tracks origin/master.
To set up other tracking branches, see Remote branches.
git checkout --track origin/serverfix
Create a new local or remote branch, checkout a remote branch?
We can use git branch or git checkout to create a new branch.
git branch testing # do not switch git log --oneline --decorate git checkout -b <new-branch> # creates and checks out <new-branch> off the current HEAD git checkout -b <new-branch> <existing-branch>
Pay attention to the keywords 'HEAD', 'master' and 'testing' in this case.
To create a new branch and switch to the new branch
git status git checkout -b myBranch # OR create a new branch starting at the commit with an identifier # git checkout -b myBranch XXXXXXX git status git log
To create a new branch based on a remote branch. Remote branch is just like any branch with one exception: you can't check them out. You can merge with them. You can see what's in them. But you can't check them out. Instead you need to create a new branch which tracks it.
# Method 1: it does not move HEAD. git branch myBranch origin/myBranch git log --oneline -5 myBranch # red color = a local copy of the remote repo, green color = local repo # Method 2: it switches to the new branch immediately git checkout -b myBranch origin/myBranch
To create a remote branch, git push origin MyBranch; see Git Branch from Atlassian.
Delete a branch locally or from the remote
How to Delete a Branch in Git Locally and Remotely
Cf. Push to the remote branch by git push -u origin newBranch; git branch -r.
Delete a local copy of a branch.
# Since you can't delete a branch you're on, switch to the master branch. git branch git checkout master git branch -d myBranch
To delete the branch from the remote:
git push origin :myBranch # OR git push -d origin myBranch # Checking git branch -r
Switch branches in local repository
git checkout testing git branch nano test.rb git commit -a -m 'made a change' git checkout master nano test.rb git commit -a -m 'made other changes' git log --oneline --decorate --graph --all
Note that when you run git pull without specify the branch name (eg gh-pages from the current local branch), it won't pull the latest from another branch (eg master).
git checkout
- https://guide.freecodecamp.org/git/git-checkout/. It compares "git checkout -b" and "git checkout -B".
- git checkout -b NEW-BRANCH-NAME will create and checkout out a new branch.
- git checkout -B BRANCH-NAME START-POINT will checkout a New Branch or Reset a Branch to a Start Point.
- Generally, Git won’t let you checkout another branch unless your working directory is clean, because you would lose any working directory changes that aren’t committed. You have three options to handle your changes: 1) trash them, 2) commit them, or 3) stash them.
Get the short Git version hash of HEAD
$ git rev-parse --short HEAD 38733fd # Compare to $ git log --oneline
Detached head/HEAD detached from xxxxxxx when you check out from a commit
- Better no to run "git checkout origin/XXX" or "git checkout SHA1" or "git checkout TAG". The right way is to create a local branch and then switch to that. git checkout -b branchXYZ origin/branchXYZ.
- HEAD detached from origin/master.
- Detached HEADS. When it points to a branch, Git doesn't complain, but when you check out a commit, it switches into a “detached HEAD” state. The point is, your development should always take place on a branch—never on a detached HEAD.
- How Do You Fix a “Detached HEAD” in a Git Repository?
(2020-03-23) I must made some mistake. See the following and the solution is by following the suggestion from the on-screen message.
git checkout XXXXXXX cp myfile /tmp/myfile git checkout origin/master # begin to mess up; why git allows this action? nano myfile # merge something from the old file '/tmp/myfile' git commit # repeat the commit action again for another change git status # HEAD detached from origin/master git checkout master # Warning: you are leaving 2 commits behind, not connected to any of your branches: # If you want to keep them by creating a new branch, this may be a good time to do so with: # git branch <new-branch-name> 854c654 # Switched to branch 'master' # Your branch is up to date with 'origin/master'. # Solution 1: git branch branch0323 854c654 git branch # check my current branch # Solution 2: git checkout -b branch0323 git merge branch0323 # Done
Good practice
See Get an old commit and merge some of its code to the current code.
Basic merging and conflicts
git checkout -b iss53 # shorthand for # git branch iss53 # git checkout iss53 nano index.html git commit -a -m 'added a new footer [issue 53]' git checkout master git checkout -b hotfix nano index.html git commit -a -m 'fixed the broken email address' git checkout master # step 1. Move back to master for merging git merge hotfix # step 2. Merge the hotfix branch into the master git branch -d hotfix # step 3. Delete the branch git checkout iss53 nano index.html git commit -a -m 'finished the new footer [issue 53]'
Basic Merging:
git checkout master git merge iss53
Basic merge conflicts:
git merge iss53 git status # Look at the standard conflict-resolution markers to the top of files git mergetool git status
push the new branch to remote
We can have a branch on our local computer only. We can also have a branch that is shared on the remote.
# make sure we are at the right branch $ git branch -a master * newbranch remotes/bitbucket/master remotes/origin/HEAD -> origin/master remotes/origin/master $ git push -u bitbucket newbranch # '-u' is to add the tracking information $ git status On branch newbranch Your branch is up-to-date with 'bitbucket/newbranch'. nothing to commit, working directory clean
If another machine runs git pull, it will get the new branch. The result can be seen by gitk --all.
pull
Suppose we are at the 'newbranch'. Some files are modified and committed to 'newbranch' (these actions are only done locally).
If someone modified files and committed to the 'master' branch, then when we run git pull (keep it in mind that we are still on newbranch) the files we just modified & committed OR even any files aren't affected since git pull is pulling files from the 'master' branch.
If we use the gitk --all or git log --graph --all --oneline --decorate command, we may see (remote has an orange background color, branch has a green background color, a yellow dot marks the current HEAD)
- remotes/bitbucket/newbranch
- remotes/bitbucket/master
- newbranch (local)
- master (local)
These 4 branches could be on different nodes. Note. gitk output is easy to read but git log gives the SHA information. The screenshot from git log below does NOT use the --decorate option.
Branch management
git branch git branch -v # see the last commit on each branch git branch -vv # see the last commit and what remote branch a local branch is tracking? git branch --merged # Filter the list to branches that you have merged into the branch you're currently on git branch --no-merged # See the branches that contain work you haven't yet merged in git branch -a # show all remotes' branches too git branch -d testing
Branch workflows
NA
Inspect a Remote
https://git-scm.com/book/ch2-5.html
brb@brb-P45T-A:~/github/SIK$ git remote origin brb@brb-P45T-A:~/github/SIK$ git remote -v origin https://[email protected]/arraytools/SIK.git (fetch) origin https://[email protected]/arraytools/SIK.git (push) brb@brb-P45T-A:~/github/SIK$ git remote add pb https://github.com/paulboone/ticgit brb@brb-P45T-A:~/github/SIK$ git remote -v origin https://[email protected]/arraytools/SIK.git (fetch) origin https://[email protected]/arraytools/SIK.git (push) pb https://github.com/paulboone/ticgit (fetch) pb https://github.com/paulboone/ticgit (push) brb@brb-P45T-A:~/github/SIK$ git fetch [remote-name] brb@brb-P45T-A:~/github/SIK$ git fetch pb warning: no common commits remote: Counting objects: 634, done. remote: Total 634 (delta 0), reused 0 (delta 0), pack-reused 634 Receiving objects: 100% (634/634), 109.18 KiB | 0 bytes/s, done. Resolving deltas: 100% (231/231), done. From https://github.com/paulboone/ticgit * [new branch] master -> pb/master * [new branch] ticgit -> pb/ticgit brb@brb-P45T-A:~/github/SIK$ git remote show origin * remote origin Fetch URL: https://[email protected]/arraytools/SIK.git Push URL: https://[email protected]/arraytools/SIK.git HEAD branch: master Remote branch: master tracked Local branch configured for 'git pull': master merges with remote master Local ref configured for 'git push': master pushes to master (up to date) brb@brb-P45T-A:~/github/SIK$ git remote show pb * remote pb Fetch URL: https://github.com/paulboone/ticgit Push URL: https://github.com/paulboone/ticgit HEAD branch: master Remote branches: master tracked ticgit tracked Local ref configured for 'git push': master pushes to master (local out of date) brb@brb-P45T-A:~/github/SIK$ git remote rm pb brb@brb-P45T-A:~/github/SIK$ git remote -v origin https://[email protected]/arraytools/SIK.git (fetch) origin https://[email protected]/arraytools/SIK.git (push)
So if we use gitg program, we will see there are following branches
- origin/master
- pb/master
- pb/ticgit
brb@brb-P45T-A:~/github/SIK$ git remote rm pb brb@brb-P45T-A:~/github/SIK$ git remote -v origin https://[email protected]/arraytools/SIK.git (fetch) origin https://[email protected]/arraytools/SIK.git (push)
Remote branches, tracking branches
https://git-scm.com/book/ch3-5.html
Remote references are references (pointers) in your remote repositories, including branches, tags, and so on.
Pushing:
git ls-remote (remoteName) git remote show (remoteName) # add RSA key fingerprint if the git is using the ssh protocol git push origin serverfix # serverfix is a branch name
Do not type your password every time: you can set up a credential cache. The simplest is to keep it in memory for a few minutes, which you can set up by running
git config --global credential.helper cache
One collaborator fetches from the server. They will get a reference to where the server's version of serverfix is under the remote branch origin/serverfix:
git fetch origin git checkout -b serverfix origin/serverfix
Tracking Branches:
Note: Tracking means that a local branch has its upstream set to a remote branch. Tracking can occur when we use clone or checkout commands. Tracking's advantage is to save our typing. Without tracking information, we need to specify the remote branch name when we want to interact with it (git fetch, git push, git pull).
When you clone a repository, it creates a master branch that tracks origin/master. However you can set up other tracking branches if you wish - ones that track branches on other remotes, or don't track the master branch.
If you create a new branch on your local repository, this new branch by default has no tracking information to a remote branch. So when we use "git push origin newBranch" (no "-u"), it does not add the tracking info. We can use --set-upstream-to or -u option when we use "git branch" to set up the tracking information; see "git help branch". One example: git branch -u origin/newBranch newBranch. Use cat .git/config to double check.
FAQ: git checkout --track origin/branch VS git checkout -b branch origin/branch. Basically '-b' allows a different branch name.
git checkout --track origin/serverfix # if the branch name you're trying to checkout (a) does not exist and (b) exactly # matches a name on only one remote, Git will create a tracking branch for you git checkout serverfix # set up a local branch with a different name than the remote branch git checkout -b sf origin/serverfix # your local branch sf will auto pull from origin/serverfix # if you ALREADY have a local branch and want to set it to a remote branch # you just pulled down, or want to change the upstream branch you're tracking. git branch -u origin/serverfix # see what tracking branches you have set up in your local repo git branch -vv
Pulling:
git pull
# Equivalent to two actions
git fetch origin
# get the contents of the remote repository (origin),
# but keep them under origin/branch branch
# requires the password
git merge origin/master # merge the master branch of the remote repository (origin) with your current branch
# no password required
Deleting Remote Branches:
git push origin --delete serverfix
Rebasing - integrate changes from 2 branches
In Git, there are two main ways to integrate changes from one branch into another: the merge and the rebase.
Finding Your Old Socks, Git hook
How To Make Life Easier When Using Git
# Checkout previous branch git checkout -
diff between two revisions
git diff <revision_1>:<file_1> <revision_2>:<file_2>
# Better to run git config --global diff.tool meld BEFORE
git difftool <revision_1> <revision_2> # Warning: it will open each file one by one.
# It is not a good idea if many files have changed.
# Better to use "-d" or "--dir-diff" option
git difftool <revision_1> <revision_2> FILENAME -y
# just compare the file <FILENAME>
git diff master..test # between the tips of the two branches
git diff origin/master..master # useful if origin/master is behind the local's master
git diff test # between your current working directory and the
# (remote) snapshot on the 'test' branch.
git diff --name-only SHA1 SHA2 # show only file names
git diff --name-only HEAD~10 HEAD~5 # between the tenth latest commit and the fifth latest
Monitor/find files that have been changed since last pull
git branch git fetch origin master # or git fetch on Windows git diff --name-only origin/master git log # local git log origin/master # remote repository $ git status On branch master Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded. (use "git pull" to update your local branch) nothing to commit, working directory clean
We can also use the gitk to view the log. The following is a screenshot from Window's git gui (Windows Start > Git > Git Gui. Then Repository > Visualize All Branch History). As you can see the local repository (master w/ yellow circle) is 2 commits behind the remote repository (remotes/origin/master w/ blue circle).
git difftool, mergetool & meld
Setting up and using Meld as your git difftool and mergetool
View differences of branches with meld?
git diff can only show the differences on the terminal. git difftool will show the difference on the GUI program.
git difftool FILENAME -y will launch 'meld' (if it has been installed before) to compare the file between revisions by using custom tools. It has to be run before we call git add. This is quite convenient since you can double check before running git commit. The '-y' argument is used to launch a diff tool without a prompt. See the documentation here.
To compare two revisions (sha1sum, branch name)
git difftool <revision_1> <revision_2>
It will open the first file in Meld. After we close the first file, it will launch the 2nd file in Meld, and so on.
The current meld has a new tool to compare git versions. New comparison > Version control. Select a git directory. It will show the differences of the local master and the current folder (left = local master, right = current dir). I can also launch the comparison from the command line:
git difftool -d master # '-d' means a directory comparison
# left = current master, right = current dir
# OR
git branch myBranch SHA1
git difftool -d myBranch # compare myBranch with my current branch
# left = myBranch, right = current master
To create a (temporary) branch, use git branch -b BRANCHNAME. To delete a (temporary) branch, use git branch -D BRANCHNAME.
It may be helpful to run the following too
git config --global diff.tool meld
Meld and Diffuse
How to build Meld 3.20 on Ubuntu. Note when we install meld using apt, it is installed under /usr/bin/meld.
git clone https://gitlab.gnome.org/GNOME/meld.git cd meld git checkout meld-3-20 python3 setup.py install --prefix=/usr sudo apt update sudo apt upgrade sudo apt install intltool sudo apt install libxml2-utils sudo apt install libglib2.0-dev-bin sudo apt install libglib2.0-dev sudo apt install libgirepository1.0-dev sudo apt install libcairo2-dev pip3 install pycairo pip3 install PyGObject sudo python3 setup.py install --prefix=/usr meld
To make meld to be in the right click menu, follow
- http://askubuntu.com/questions/112164/how-can-i-diff-two-files-with-nautilus
- http://superuser.com/questions/307927/right-click-files-to-meld
Another method of comparing two files without using the 'browse' button will be to use the command line.
The 'nautilus-compare' program does not work from my testing on Ubuntu 14.04.
Some thoughts
- I install kdiff3 (<2 MB to download) and the 'File' -> 'Reload' (F5) function there works though it shows an extra space on the place I modified.
- Beyond Compare (commercial $30/$60, trial version can be downloaded)
- diffuse. When I modified a file, diffuse can detect a change and ask me to reload the file. I am using the apt-get to install the software and the version number is 0.4.7 (2014). To copy lines from left panel to right panel, use 'Ctrl + Shift + >' or the Copy Selection Right icon. One drawback is it cannot save the history from the GUI though we can use the command line to include the file names in the arguments.
- Alternatively we can use WinMerge on Linux. To do that, install Wine on Ubuntu. Download Winmerge (I am using 2.14.0). Then on a terminal, run the following command. At the end, WinMerge will be launched. WinMerge can also be launched from Mint Menu -> Wine -> WinMerge. One problem is I cannot increase the font size (though acceptable) from View -> Select Font.
wine WinMerge-2.14.0-Setup.exe
git-split-diffs
git-split-diffs. This currently requires node version 12 or newer to run.
Pull and overwrite local files
If we want to run git pull and also overwrite possibly changed local file, we use (see stackoverflow)
git fetch --all git reset --hard origin/master
Your branch is ahead by X commits after running git pull
git status shows Your branch is Ahead by X commits after running git pull. See this post. The solution is to run git fetch after git pull.
This repository currently has approximately XXXX loose objects
git gc --aggressive
git commit -am
works only for simple edits. It does not work for renaming files, etc.
git diff --staged
git checkout -- FileName
to recover a modified/deleted file where -- is a linux notation, not specific to git. See Difference between "git checkout <filename>" and "git checkout -- <filename>".
rm FileName git pull # do nothing git status # still miss 'FileName' git checkout FileName
The following examples give a good clarification.
git checkout README # would normally discard uncommitted changes
# to the _file_ "README"
git checkout master # would normally switch the working copy to
# the _branch_ "master"
git checkout -- master # discard uncommitted changes to the _file_ "master"
git reset HEAD FileName
to un-stage a file
Rewrite a commit
ps. This is different from the case of Pull and overwrite local files
Suppose someone made a stupid commit. We want to get a previous version of the code and commit that one.
git checkout XXXXXX -- FileName git commit -am CommitMessage
origin and nickname
git remote add NickName https://XXX git remote git push -u NickName master
git merge
https://www.atlassian.com/git/tutorials/using-branches/git-merge
See an example of merging a temporary branch with the master branch in the local repository.
This is another example. It teaches how to make sure my master branch is in synch with the central repository on github (which I refer to using the remote “origin”) before I merge my changes into master.
Resolving merge conflict from lynda.com
Merge a local branch to the local master
git checkout master git merge newBranch
Merge a remote branch to the local master
git checkout master git fetch git merge origin/master
Merge conflict 1
Your branch is ahead of 'origin/master'
Merge conflict 2
Your branch and 'origin/master' have diverged
Manual merge and auto merge
If a file looks like
line1 line2 line3 line4
(Manual merge) Then suppose machine 1 modifies line3 and push to the remote. Machine 2 at the same time modifies line4 and tries to run git fetch & git merge from the remote. Then machine 2 will need to do merge manually (git mergetool) because line 3 and line 4 are next to each other. After fixing the conflict by git mergetool, we can run git commit & git push. A temporary file called FILENAME.orig will be created.
(Auto merge) On the other hand, if machine modifies line 2 and push to the remote. Machine 2 at the same time modifies line 4 and tries to run git fetch & git merge to the remote. Then auto merge will be done because line 2 and line 4 are separated. Machine 2 only needs to provide the commit message.
Whether there are conflicts or not, the practice is to run
- git fetch
- git diff master origin/master (- part is from master, + part is from origin/master)
- git merge origin/master (possibly need to run nano FILENAME + git add + git commit or run git mergetool + git commit if there is a conflict, OR a text editor will be opened to let you enter the commit message if there is no any conflict). Use git config --global core.editor nano to set the default text editor if this was not done yet; see Git configuration.
- git push.
Note that
- If we are using git pull instead of git fetch + git merge, we will be directed to the nano editor to enter a commit message if there are no conflicts; however, if we use git fetch, we will NOT be directed to nano editor).
- If we directly run git push (even on non-conflict file) without running git pull, we will be welcome with the message
brb@brb-P45T-A:~/github/toy$ git add . brb@brb-P45T-A:~/github/toy$ git commit -m "add a line from p45t-a" [master 3b69283] add a line from p45t-a 1 file changed, 1 insertion(+) brb@brb-P45T-A:~/github/toy$ git push To https://[email protected]/arraytools/toy.git ! [rejected] master -> master (fetch first) error: failed to push some refs to 'https://[email protected]/arraytools/toy.git' hint: Updates were rejected because the remote contains work that you do hint: not have locally. This is usually caused by another repository pushing hint: to the same ref. You may want to first integrate the remote changes hint: (e.g., 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details. brb@brb-P45T-A:~/github/toy$ git status On branch master Your branch is ahead of 'origin/master' by 1 commit. (use "git push" to publish your local commits) nothing to commit, working directory clean
This goes back to the same problem mentioned above in Merge conflict 2.
Summary:
- When we run git pull, if there is a conflict and that conflict can be resolved automatically, we will be directed to the text editor to enter the commit message.
- When we run git fetch, we will NOT be directed to the text editor regardless of conflicts. We need to enter the commit message when we run git merge origin/master. Also gitk will NOT show origin/master on graph BEFORE we run git merge origin/master for some reason.
Take home message:
- Before going to run git merge, run git checkout master first.
- Instead of running git pull, we'd better run git fetch & git merge origin/master if the bash code was run in a background.
See also http://www.gitguys.com/topics/merging-branches-without-a-conflict/
git stash
Practices/collaboration workflow
Collaboration workflow example from lynda.com
Create a branch from the current HEAD for experiment
Suppose we want to experiment on a new plan, we can create a new branch ('newbranch') in our local repository, merge newbranch to master and push to the remote.
git checkout master
git fetch
git merge origin/master
git branch newbranch # step 1 create a new branch called 'newbranch'
git checkout newbranch # step 2 switch to the 'newbranch' branch
# We can combine steps 1 & 2 by 'git checkout -b newbranch'
nano FILENAME # step 3 modify some files, create new files, ...
git add FILENAME
git commit # step 4 commit changes to newbranch
git checkout master # step 5 check out the master branch
git branch -a # (optional) verify we are on the master branch
git fetch # double check nothing has been committed to the remote
git merge newbranch # step 6 merge newbranch to master
git push -u origin master # step 7 push (local) master to remote/master
Another scenario is we push our local newbranch to the remote newbranch. Then we can skip steps 5 & 6 and do git push -u origin newbranch in step 7.
Question: If we have multiple remotes, how do we know if the local master is the same as the origin/master or origin2/master? Answer: git diff.
Get an old commit and merge some of its code to the current code
$ git checkout -b testBranch SHA1 (NB. if we want to checkout an old commit, always create a branch) $ nano someFile $ git checkout master # OR "git checkout -" $ git merge testBranch $ (optional) git branch -d testBranch
Or without using "git merge"
$ git checkout -b testBranch SHA1 (NB. if we want to checkout an old commit, always create a branch) $ cp someFile /tmp/ $ git checkout master # OR "git checkout -" $ meld /tmp/someFile someFile $ git commit $ (optional) git branch -d testBranch
git rebase
https://lostechies.com/joshuaflanagan/2010/09/03/use-gitk-to-understand-git-merge-and-rebase/
The simple git merge method can make the history very complicated; see and the following cons.
- Branching paths in the history can be unnecessarily complicated
- The extra merge commit.
- Your branch is now no longer a private, local concern. Everyone now knows that you worked in an issue123 branch. Why should they care?
git rebase can be used to avoid these issues.
The good approach (as in that example) is
git checkout issue123 git rebase master git checkout master git merge issue123 git branch -d issue123 git push origin master
.ref directory
brb@brb-P45T-A:~/github/toy$ ls -l .git total 60 drwxrwxr-x 2 brb brb 4096 May 6 11:55 branches -rw-rw-r-- 1 brb brb 13 May 7 16:12 COMMIT_EDITMSG -rw-rw-r-- 1 brb brb 273 May 6 11:55 config -rw-rw-r-- 1 brb brb 73 May 6 11:55 description -rw-rw-r-- 1 brb brb 95 May 7 16:32 FETCH_HEAD -rw-rw-r-- 1 brb brb 3676 May 6 21:44 gitk.cache -rw-rw-r-- 1 brb brb 23 May 6 11:55 HEAD drwxrwxr-x 2 brb brb 4096 May 6 11:55 hooks -rw-rw-r-- 1 brb brb 176 May 7 16:32 index drwxrwxr-x 2 brb brb 4096 May 6 11:55 info drwxrwxr-x 3 brb brb 4096 May 6 11:55 logs drwxrwxr-x 85 brb brb 4096 May 7 16:12 objects -rw-rw-r-- 1 brb brb 41 May 7 16:32 ORIG_HEAD -rw-rw-r-- 1 brb brb 107 May 6 11:55 packed-refs drwxrwxr-x 5 brb brb 4096 May 6 11:55 refs brb@brb-P45T-A:~/github/toy$ tree .git/refs .git/refs ├── heads │ └── master ├── remotes │ └── origin │ ├── HEAD │ └── master └── tags brb@brb-P45T-A:~/github/toy$ ls .git/objects/ 03 0f 28 39 47 52 5f 65 6e 8c ad b7 ce da e5 f2 ff 07 12 29 3b 49 53 60 67 6f 9a b1 bb cf dc e9 f5 info 08 13 2c 3d 4d 56 61 68 84 9e b2 bc d0 de ea f6 pack 09 16 36 43 4e 57 63 6c 89 a9 b3 c0 d5 e1 ef fc 0c 1b 37 45 51 58 64 6d 8a aa b5 c8 d6 e2 f1 fe brb@brb-P45T-A:~/github/toy$ ls .git/branches/ brb@brb-P45T-A:~/github/toy$ ls .git/info/ exclude brb@brb-P45T-A:~/github/toy$ ls .git/logs HEAD refs
Tagging a release
https://git-scm.com/book/en/v2/Git-Basics-Tagging
$ git tag # or add "--list" or add "-l" $ git tag -l "v2*" # add a filtering list tags starting with "v2" $ git tag -l -n # list tags with annotations # Create annotated tags $ git tag -a v1.4 -m "my version 1.4" $ git tag v1.4 $ git show v1.4 $ git diff v1.0..v1.1 # Tagging Later $ git log --pretty=online $ git tag -a v1.2 9fceb02 # Delete a tag $ git tag -d v1.1
By default, the git push command does not transfer tags to remote servers. You have to explicitly push tags to a shared server after you have created them.
$ git push origin v1.5 $ git push origin --tags # push up all tags at once # Delete remote tags like remote branches $ git push origin :v1.1 # OR $ git push -d origin v1.1
Check out tags just like any sha, but it has a profound effect if we later commit something (Detached HEAD state).
$ git checkout -b newBranch v1.1 $ git checkout v1.1
setting up SSH access on the server side
Follow the instruction on git-scm.com. It works.
I tested it by
- create a git account (called 'git') on my host machine.
- sudo to create a directory called /opt/git/project.git. Change the owner to 'git'. Cd to the directory and initialize it.
- Create two virtual machines (vb1 and vb2). Creating a username 'david' on vb1 and a user name 'joseph' on vb2.
- Create ssh key for both 'david' and 'joseph'. Ssh to copy their ssh keys to git account.
- Create a new directory on vb1 and initialize it. Run git commands to commit & push files to the server (no password is needed). Note that when we use 'git commit', git will ask to create a username and email by first running 'git config' command.
- Switch to vb2 and run git clone (no password is needed). The user can modify the code and commit & push files to the server.
- Run 'git log' to check if each user's name/email are shown on the log.
Some important points:
- There is no daemon to be installed. We only need to install the 'git' program on the server.
- When a client uses the 'git' command to communicate with the server, it is actually using the 'ssh' to access the server.
- We still need to create a user for this git server. Then the developers' rsa keys can be saved to the git user's <.ssh/authorized_keys> file.
- The instruction asks to chmod 700 (rwx------) for the .ssh directory and chmod 600 (-rw------) for the <.ssh/authorized_keys> file.
- If the git repository directory is not saved under the user's directory, we need to make sure the owner of the directory is the git user.
Create a git server
- How to Set Up a Private Git Server
- HOWTO setup a private git server on Ubuntu 18.04 (including to set the git user’s shell to the git-shell)
- Install git and git-shell
- Setup a dedicated (non-sudo) git user
- Create individual repo directories
- Use your private git repo
$ sudo apt update $ sudo apt install git $ sudo nano /etc/shells # ADD /usr/bin/git-shell
$ sudo adduser --disabled-password git
$ sudo su git
$ cd
$ mkdir ~/.ssh && chmod 700 ~/.ssh
$ touch ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys
$ nano ~/.ssh/authorized_keys # add the public keys of any users
# you want to access your private git server
$ sudo chsh git -s $(which git-shell)
$ cd /home/git $ sudo mkdir ios-backup-extractor.git $ cd ios-backup-extractor.git && sudo git init --bare && cd .. $ sudo chown -R git.git ios-backup-extractor.git/
$ git clone git@<IPADDRESS>:<repo-name> ~/git/ $ git remote add origin git@<IPADDRESS>:<repo-name>.git
Create a git server (github like w/ web interface)
If we like to create a github-like web interface, check out GitLab. See
- How to Install Gitlab on Debian 8 (Jessie),
- How to Run Your Own Git Server. This includes the installation of postfix for sending emails.
Below is my note
- https://about.gitlab.com/downloads/ contains steps of setting up Gitlab.
- By default, the domain name you have entered in setting up gitlab will be the URL you will use to access gitlab.
- Use the recommended method to install gitlab. Nginx will be installed as an http server.
- The root username and password is root and 5iveL!fe.
- When new users are created by root, we can put a faked email there (eg [email protected]). The root account can create password for the user.
- User's password is used to access GitLab web interface only. It is not used for pushing commits.
- After a new user is created, log out of root account and log in using the new user account. Click 'Profile setting' icon and then select SSH > Add SSH key. Copy your <id_rsa.pub> content there. To create your ssh key, use the command line "ssh-keygen -t rsa". The <id_rsa.pub> is located under ~/.ssh directory. The title should be auto populated. If ssh key is added successfully to gitLab, we won't get a pop-up asking password when we run 'git push'.
- A new project should be created by users (not root). If I create a project by root, I keep getting a permission issue when I run 'git push'.
- The username will affect path to all personal projects; e.g. [email protected]:newuser/test2.git.
How to Install Gitlab with PostgreSQL and Nginx on Ubuntu 15.04 from howtoforge.com.
Gitolite (favored by Ubuntu)
- https://help.ubuntu.com/lts/serverguide/git.html#git-installing-gitolite
- https://www.digitalocean.com/community/tutorials/how-to-use-gitolite-to-control-access-to-a-git-server-on-an-ubuntu-12-04-vps
This approach involves the following steps
- Installing a gitolite server
- Gitolite configuration
- Managing gitolite users and repositories
- Using your server
Not sure about any advantages of this approach?
Graphical tool, app
Command line approach
$ git log --oneline --decorate --graph --all * 63af56a (HEAD, origin/master, origin/HEAD, mybranch, master) merged in vm |\ | * 67d5aad modify linux by mint 4th line * | 5f3267a modify linuxfile by vm 4th line |/ * adf545a new modify linux by mint * f22932b resolve the conflict in vm |\ | * b2f1ee4 modify linuxfile by mint * | fe50c32 modify linuxfile by vm |/ * a95258a add linuxfile * b1a71a8 commit from linux * d02b570 2nd commit * 3700939 first commit
Tig
How to use Tig to browse Git logs
gitk and git-gui
- http://lostechies.com/joshuaflanagan/2010/09/03/use-gitk-to-understand-git/
- http://git-scm.com/download/gui/linux
sudo apt-get update; sudo apt-get install gitk git clone [email protected]:joshuaflanagan/gitk-demo.git cd gitk-demo gitk --all # show all refs (branches, tags, etc.)
What I find is gitk will take a little time to rebuild something because after I run git pull, it will show several files (if not all) are uncommitted.
Gitk worked with git-gui program (sudo apt-get install git-gui) which is a gui program to run rescan/stage/commit/push. You launch git-gui from gitk-File-Start git-gui.
It looks these 2 gui tools are sufficient enough.
GitKraken
#1 of Top 20 Dev Tools for 2019
Others
- QGit - QGit is a git GUI viewer built on Qt/C++.
- git-cola ???
- gitg
- Giggle - Similar to gitk
- SmartGit - Support push, pull, fetch
Git Tips
Permission denied (publickey)
The error happens on either github or bitbucket.
Some resources:
- Authenticating to GitHub
- Permission denied (publickey)
- Simple solution: add the ssh key in ~/.ssh/config file. There is no need to use ssh-add command. Tested on github.com and bitbucket.org. See
old mode new mode
See this post
git config core.filemode false
Temporarily move to a branch and then move back
# suppose we have 3 commits so far and we are at commit 3 now. git status git log # show 3 commits git checkout sha1_commit_1 # move to commit 1 git log # only show 1 commit git checkout master # go back to the original 3 commits git status
git push error (file is created by Windows but edited by Linux), Your branch is ahead of 'origin/master'
Suppose there are two users A & B (that me). A modified something, commits and pushes to the remote. I modified the same file, commit (so far OK since the actions are all local) and try to push to the remote.
If I forget to run git pull and after running git push, we will get something like if there is a conflict.
C:\Users\XXX\Documents\GitHub\toy [master]> git push To https://github.com/arraytools/toy.git ! [rejected] master -> master (non-fast-forward) error: failed to push some refs to 'https://github.com/arraytools/toy.git' hint: Updates were rejected because the tip of your current branch is behind hint: its remote counterpart. Merge the remote changes (e.g. 'git pull') hint: before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details. C:\Users\XXX\Documents\GitHub\toy [master]> git status # On branch master # Your branch is ahead of 'origin/master' by 1 commit. # nothing to commit, working directory clean
Follow the suggestion from this post to use git pull for git to do merge. It does not work in this case.
C:\Users\XXX\Documents\GitHub\toy [master]> git pull origin master
remote: Counting objects: 3, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 3 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), done.
From https://github.com/arraytools/toy
* branch master -> FETCH_HEAD
warning: Cannot merge binary files: README.md (HEAD vs. b1a71a846e17416a3b248dfd5829547f74fd812c)
Auto-merging README.md
CONFLICT (content): Merge conflict in README.md
Automatic merge failed; fix conflicts and then commit the result.
C:\Users\XXX\Documents\GitHub\toy [master +0 ~0 -0 !1 | +0 ~0 -0 !1]> dir
Directory: C:\Users\XXX\Documents\GitHub\toy
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 5/3/2016 4:33 PM 102 README.md
C:\Users\XXX\Documents\GitHub\toy [master +0 ~0 -0 !1 | +0 ~0 -0 !1]> git status
# On branch master
# Your branch is ahead of 'origin/master' by 1 commit.
#
# You have unmerged paths.
# (fix conflicts and run "git commit")
#
# Unmerged paths:
# (use "git add <file>..." to mark resolution)
#
# both modified: README.md
#
no changes added to commit (use "git add" and/or "git commit -a")
The README.md file was created on Windows/DOS. For some reason, it is recognized as binary files by git. If I look at the file on Linux, it shows the file is ASCII text but with CR line terminators.
$ file README.md README.md: ASCII text, with CR line terminators.
Since we already run git pull, it won't work to run git reset --soft HEAD~ to undo a commit. Two solutions
- git fetch origin; git reset --hard origin/master to throw away my local modified file OR
- git reset --hard HEAD to undo git pull; see this post. After that, we can run git reset --soft HEAD~ to undo a commit. See below.
C:\Users\XXX\Documents\GitHub\toy [master +0 ~0 -0 !1 | +0 ~0 -0 !1]> git reset --hard HEAD HEAD is now at cd2d105 commit from Windows C:\Users\XXX\Documents\GitHub\toy [master]> git status # On branch master # Your branch is ahead of 'origin/master' by 1 commit. # nothing to commit, working directory clean C:\Users\XXX\Documents\GitHub\toy [master]> git reset --soft HEAD~ C:\Users\XXX\Documents\GitHub\toy [master +0 ~1 -0]> git status # On branch master # Changes to be committed: # (use "git reset HEAD <file>..." to unstage) # # modified: README.md C:\Users\XXX\Documents\GitHub\toy [master +0 ~1 -0]> cat README.md # toy second commit 3rd commit (Windows MACHINE) C:\Users\XXX\Documents\GitHub\toy [master +0 ~1 -0]> git reset HEAD README.md Unstaged changes after reset: M README.md C:\Users\XXX\Documents\GitHub\toy [master +0 ~1 -0]> git status # On branch master # Changes not staged for commit: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # # modified: README.md # no changes added to commit (use "git add" and/or "git commit -a") C:\Users\XXX\Documents\GitHub\toy [master +0 ~1 -0]> git checkout -- README.md C:\Users\XXX\Documents\GitHub\toy [master]> cat README.md # toy second commit C:\Users\XXX\Documents\GitHub\toy [master]> git status # On branch master nothing to commit, working directory clean
Now the README.md file is the original one. If we want, we can make a copy of the modified file before unmodifying a modified file (git checkout -- Filename).
Summary
- (undo git pull) git fetch; git reset --hard origin/master # OR git reset --hard HEAD
- (undo commit) git reset --soft HEAD~
- (undo stage) git reset HEAD Filename
- (undo change) git checkout -- Filename
git push error (both clients are linux), Your branch and 'origin/master' have diverged
Two clients (mint and vm) running on Linux. The following is coming from vm machine.
brb@ubuntu:~/toy$ git push ! [rejected] master -> master (fetch first) error: failed to push some refs to 'https://[email protected]/arraytools/toy.git' hint: Updates were rejected because the remote contains work that you do hint: not have locally. This is usually caused by another repository pushing hint: to the same ref. You may want to first integrate the remote changes hint: (e.g., 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details. brb@ubuntu:~/toy$ git pull remote: Counting objects: 3, done. remote: Compressing objects: 100% (2/2), done. remote: Total 3 (delta 0), reused 3 (delta 0), pack-reused 0 Unpacking objects: 100% (3/3), done. From https://github.com/arraytools/toy a95258a..b2f1ee4 master -> origin/master Auto-merging linuxfile CONFLICT (content): Merge conflict in linuxfile Automatic merge failed; fix conflicts and then commit the result. brb@ubuntu:~/toy$ cat linuxfile first line <<<<<<< HEAD second line edited by vm ======= second line edited by mint >>>>>>> b2f1ee4b1f9c2a428a1af061c7b40316a8f85d3d brb@ubuntu:~/toy$ git status On branch master Your branch and 'origin/master' have diverged, and have 1 and 1 different commit each, respectively. (use "git pull" to merge the remote branch into yours) You have unmerged paths. (fix conflicts and run "git commit") Unmerged paths: (use "git add <file>..." to mark resolution) both modified: linuxfile no changes added to commit (use "git add" and/or "git commit -a")
master branch and 'origin/master' have diverged, how to 'undiverge' branches'?
The solution now is to run git mergetool (assume we have install something like meld, opendiff, kdiff3, tkdiff, or xxdiff). See Git Mergetool – Merging With a GUI from gitguys.com.
After we edit the file in the middle of meld, we save it and quit meld. Go back to the terminal. Strangely, git knows the conflict has been resolved (shown from git status).
brb@ubuntu:~/toy$ cat linuxfile first line second line. conflict resolved in vm brb@ubuntu:~/toy$ git status On branch master Your branch and 'origin/master' have diverged, and have 1 and 1 different commit each, respectively. (use "git pull" to merge the remote branch into yours) All conflicts fixed but you are still merging. (use "git commit" to conclude merge) Changes to be committed: modified: linuxfile Untracked files: (use "git add <file>..." to include in what will be committed) linuxfile.orig brb@ubuntu:~/toy$ git commit -m "resolve the conflict in vm" [master f22932b] resolve the conflict in vm brb@ubuntu:~/toy$ git status On branch master Your branch is ahead of 'origin/master' by 2 commits. (use "git push" to publish your local commits) Untracked files: (use "git add <file>..." to include in what will be committed) linuxfile.orig nothing added to commit but untracked files present (use "git add" to track) brb@ubuntu:~/toy$ git push
The repository now looks like:
warning: push.default is unset
Git 2.0 from 'matching' to 'simple'. To squelch this message ...
See stackoverflow.
It only affects what happens when you don't specify which branches you want to push; e.g. 'git push' or 'git push origin' instead of 'git push -u origin master'.
$ git push bitbucket warning: push.default is unset; its implicit value is changing in Git 2.0 from 'matching' to 'simple'. To squelch this message and maintain the current behavior after the default changes, use: git config --global push.default matching To squelch this message and adopt the new behavior now, use: git config --global push.default simple When push.default is set to 'matching', git will push local branches to the remote branches that already exist with the same name. In Git 2.0, Git will default to the more conservative 'simple' behavior, which only pushes the current branch to the corresponding remote branch that 'git pull' uses to update the current branch. See 'git help config' and search for 'push.default' for further information. (the 'simple' mode was introduced in Git 1.7.11. Use the similar mode 'current' instead of 'simple' if you sometimes use older versions of Git)
I guess 'simple' (the default) is what we/beginners usually want. Run
git config --global push.default simple
Search/find a deleted file from the commit history
For example, the noweb/code.pdf file was mentioned in the Readme file but it is missing from the CRAN's survival package.
$ git log --all --full-history -- noweb/code.pdf commit 1f0dbc538e9147ab264c5215cf205b9b3bc171cb Author: Terry M Therneau <[email protected]> Date: Sun Jun 26 14:19:51 2016 +0000 version 2.39-5 commit ec6b50c973b9298a5f1aeb839b2d04d3448e69de Author: Terry M Therneau <[email protected]> Date: Wed May 11 11:45:26 2016 +0000 version 2.39-4 commit 38a8ccc0fb014c2b2bfbb2b5153b419e615b01ba Author: Terry M Therneau <[email protected]> Date: Sat Apr 16 19:26:46 2016 +0000 version 2.39-2
Here we see code.pdf file existed in version 2.39-2 and was last seen in version 2.39-5.
To restore the deleted, we use
git checkout 38a8ccc -- code.pdf
Note that I follow the instruction to add the caret sign (^) at the end of SHA, I'll get an error pathspec 'code.pdf' did not match any file(s) known to git. My git version is 2.7.4.
However, for another file, I do need to use caret.
git log --all --full-history -- src/coxfit2.c git checkout 68e5adc1d1486e2b1daa7db9203fd9226e029992^ -- src/coxfit2.c
Removing files/a folder from a repository
To remove the file called Rakefile:
$ git filter-branch --force --index-filter \ 'git rm --cached --ignore-unmatch Rakefile' \ --prune-empty --tag-name-filter cat -- --all $ git push --all --force
Pull request
- How To Create a Pull Request on GitHub
- https://www.wikihow.com/Create-a-Pull-Request-on-Github
- A Newbie's Guide to Making A Pull Request (for an R package)
- Pull request helpers from the usethis package.
- What Are Git Pull Requests, And How Do You Use Them?
Cannot see new files added to my git working directory
Cannot see new files added to my git working directory. It happened when I include a new subdirectory which comes from other people's repository. The solution for my case is to rm .git/index and then run git add/commit/push. Next time I need to remove the .git from that subdirectory first.
Binary files and why is my .git folder so big?
https://www.reddit.com/r/git/comments/4xsh26/why_is_my_git_folder_so_big/
git gc --aggressive --prune
It seems there is no way I can remove binary files (history) from .git folder.
Large files and storage limit
- How to Store Large Files in Git
- The individual file size limit is 100MB on Github.
- The total repository on your Bitbucket is 1GB
Less than 100MB
$ git push -u origin master Counting objects: 115, done. Delta compression using up to 12 threads. Compressing objects: 100% (100/100), done. Writing objects: 100% (115/115), 97.54 MiB | 4.48 MiB/s, done. Total 115 (delta 16), reused 0 (delta 0) remote: warning: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com. remote: warning: See http://git.io/iEPt8g for more information. remote: warning: File GSE48215/breastcancer-bt20_raw.vcf is 79.57 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB To https://github.com/arraytools/seqtools_testdata.git * [new branch] master -> master Branch master set up to track remote branch master from origin.
It looks like all files are uploaded successfully.
One annoying thing about github is when we download the repository it automatically add a folder name (REPOSITORY-BRANCH) at the top. For example, it creates 'seqtools_testdata-master'.
$ unzip -l ~/Downloads/seqtools_testdata-master.zip
Archive: /home/odroid/Downloads/seqtools_testdata-master.zip
9f5e132bcfd2007f5fc1fb4ee465c5b307b3e85a
Length Date Time Name
--------- ---------- ----- ----
0 2016-06-01 21:05 seqtools_testdata-master/
0 2016-06-01 21:05 seqtools_testdata-master/GSE11209-master/
46737374 2016-06-01 21:05 seqtools_testdata-master/GSE11209-master/SRR002051.fastq
46675548 2016-06-01 21:05 seqtools_testdata-master/GSE11209-master/SRR002059.fastq
0 2016-06-01 21:05 seqtools_testdata-master/GSE11209-master/Saccharomyces_cerevisiae/
...
Larger than 100MB
remote: error: File hg19/chr1.fa is 242.46 MB; this exceeds GitHub's file size limit of 100.00 MB remote: error: File hg19/chr1.fa.bwt is 237.70 MB; this exceeds GitHub's file size limit of 100.00 MB remote: error: File hg19/chr1.fa.sa is 118.85 MB; this exceeds GitHub's file size limit of 100.00 MB To https://github.com/arraytools/GSE48215.git ! [remote rejected] master -> master (pre-receive hook declined)
git submodule
- Managing large binary files with Git
- https://git-scm.com/docs/git-submodule
- http://alx.github.io/gitbook/5_submodules.html
Import from CVS
- http://git-scm.com/docs/gitcvs-migration
- http://stackoverflow.com/questions/20869710/migrate-from-cvs-to-git-without-losing-history (see also comments after the suggestion using git-cvsimport)
sudo apt-get install git-cvs # cvsimport command tar xzvf tmp.tar.gz cd DirectoryName # The next command requires a connection to the CVS server even we have a copy of CVS in the local pc git cvsimport -C ~/Downloads/tmp ModuleName # SLOW & give up
Zip up the folder but exclude the .git subfolder
zip -r myproject.zip myproject -x *.git*
On Mac, we should add --exclude=*.DS_Store* to exclude Mac OS X directory display metadata files.
Preview PDF files
It has a built-in support.
Preview HTML files from Github and Bitbucket
Example: data.table cookbook
General resources
- https://www.atlassian.com/git/tutorial/remote-repositories
- http://www.sbf5.com/~cduan/technical/git/git-4.shtml
- http://www.acquia.com/blog/getting-started-collaborative-development-git
- http://documentup.com/skwp/git-workflows-book
- http://www.eqqon.com/index.php/Collaborative_Github_Workflow
Git hosting services
Self-hosted
Gitlab
- gitlabr R package
Compared to Github, Gitlab can
- host private projects for free
- host static web pages on http://pages.gitlab.io/.
Host web site
http://monitor.gitlab.net/
How to install and configure Gitlab on Ubuntu
- https://www.howtoforge.com/tutorial/how-to-install-and-configure-gitlab-on-ubuntu-16-04/
- How To Set Up a Personal Gitlab Server
Running an R Script on a Schedule
- Running an R Script on a Schedule: Overview
- Tweeting from gitlab actions
- Docker Containers on gitlab
Bitbucket
- New IP July 2, 2018
- Storage limit warning after 1GB, hard limit for 2GB.
Large files
Manage huge files in Bitbucket with Git LFS
Gitbucket
How to Install GitBucket with Nginx on Ubuntu 18.04 LTS
Github
Github Status
- https://twitter.com/githubstatus
- https://status.github.com/ (deprecate on Mar 3, 2019)
- https://www.githubstatus.com/
Large files (eg binary)
- Disk Quota. Repositories < 1GB and files < 100MB.
- Releases allow you to include binary files, such as compiled programs. See https://help.github.com/articles/distributing-large-binaries/
Create a new repository
After we use the web interface to create a new empty repository, we are instructed to do one of the following
- create a new repository on the command line
echo "# toy" >> README.md git init git add README.md git commit -m "first commit" git remote add origin https://github.com/arraytools/toy.git git push -u origin master
- Compare git remote add (set up remote url) and git remote set-url (switch remote url)
- push an existing repository from the command line
git remote add origin https://github.com/arraytools/toy.git git push -u origin master
- import code from another repository. You can initialize this repository with code from a Subversion, Mercurial, or TFS project.
Archive a repository
Host web site: pages and wiki
- https://pages.github.com/. Create a new repository "username.github.io". Fill out some files there. Go to https://username.github.io to see the result.
- Updating gh-pages (assuming your vignette file is named 'index.Rmd'). See a note.
# pip install ghp-import Rscript -e "devtools::build_vignettes()" ghp-import inst/doc git push
- Using GitHub Pages To Host Your Website using a Custom Domain. The IP address information for Github is changed to new ones.
# Create a new repository in Github using the web interface # clone it to the local pc git clone https://github.com/user/repository.git # create a new branch cd repository git checkout --orphan gh-pages git status # double check we are in the gh-pages branch git rm -rf . # create new files. touch index.html git add -a git commit -m "Adding pages" # push to github git push origin gh-pages # see the web page at # http://username.github.io/repository/.
Some Examples:
- PiScope
- dygraphs for R
- networkD3 for R
- BRB-DGE
- read depth calculator It can run some script!!!
- Wiki has its own git. See this example.
Create a documentation site with Docsify and GitHub Pages
How to create a documentation site with Docsify and GitHub Pages
Github Docker Container Registry
How to Get Started with Github’s New Docker Container Registry
Github Actions (GHA)
- https://github.com/features/actions
- https://docs.github.com/en/free-pro-team@latest/actions
- Specifications for GitHub-hosted runners. 2-core CPU, 7 GB of RAM memory, 14 GB of SSD disk space. For the free plan, 2,000 Actions minutes/month + 500MB of GitHub Packages storage.
- Workflow syntax: on, name, jobs, jobs.<job_id>.name, jobs.<job_id>.runs-on, jobs.<job_id>.env, jobs.jobs.<job_id>.steps, <job_id>.strategy, jobs.<job_id>.container.
- Lynda.com -> Learning Github Actions
- https://github.com/r-lib/actions
- R usethis package
- Rendering your README with GitHub Actions
- How to migrate from TravisCI to GitHub Actions for R packages (video). It teaches how to use the usethis::use_github_action_check_standard() function.
- Quick GitHub Actions for your R package. usethis::use_pkgdown() (set up a website for your package, not related to Github Action)
- biocthis
- Git hooks
- Tutorials
- Docker
- First Docker GitHub Action is here!
- Publishing Docker images
- BuildABiocWorkshop2020 works.
- Read the vignette How To Build A Workshop Package
- In repository setting, I need to enable Github Pages and use "/doc" as the folder in the "gh-pages" branch.
- In repository setting -> Secrets, I need to create two secrets. One name is DOCKER_USERNAME and the other is DOCKER_PASSWORD. Their values are from docker hub account info.
- After commit and push changes, we can go to github actions and click "update" (my commit message) -> "job1". It shows each step with running time listed on the yaml file.
- For a successful run, we will see docker hub page was created and a gh-page was created too.
- An example to monitor the number of Bioconductor package downloads. Note that the 'bioc' version needs to be updated once a new version of Bioconductor has been released.
- MLOPS with R and GitHub Actions
- Running an R Script on a Schedule: Gh-Actions
- Using Github actions to test your go code on ARM64
- Self-hosted runners.
- Self-hosted runners for GitHub Actions is now in beta
- setup-r@v1 stalls on a self-hosted ubuntu-18.04 runner? #173. It ask sudo password for apt command.
- https://github.com/tcardonne/docker-github-runner This Docker image allows you to create your own runners on Docker.
- Deploying Self-Hosted Github Actions Runners with Docker
- Google: githubactions self host apt command
- A working example (ga_readme private repo). Note the runner has an access all to my host OS's apps like R/Rscript and all R's libraries. We can put the GHA in a docker; see the next item for a step-by-step tutorial.
# For convenience I extract the actions-runner to the repo directory and I add actions-runner to .gitignore. $ ~/github/ga_readme/actions-runner$ ./run.sh √ Connected to GitHub 2020-11-22 13:09:50Z: Listening for Jobs 2020-11-22 13:09:53Z: Running job: build 2020-11-22 13:09:59Z: Job build completed with result: Succeeded
# The yml file contains several names: # CI - workflow name # build - job name # Run a one-line script - step name $ cat .github/workflows/main.yml name: CI on: push: branches: [ main ] # Allows you to run this workflow manually from the Actions tab workflow_dispatch: # A workflow run is made up of one or more jobs that can run sequentially or in parallel jobs: # This workflow contains a single job called "build" build: runs-on: self-hosted # Steps represent a sequence of tasks that will be executed as part of the job steps: # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it - uses: actions/checkout@v2 # Runs a single command using the runners shell - name: Run a one-line script run: Rscript --vanilla -e "library()" # Runs a set of commands using the runners shell - name: Render README run: Rscript -e 'rmarkdown::render("README.Rmd")' - name: Commit results run: | git config --local user.email "[email protected]" git config --local user.name "GitHub Actions" git commit README.md -m 'Re-build README.Rmd' || echo "No changes to commit" git push origin || echo "No changes to commit" - Ephemeral Self-Hosted Github Actions Runners: Dockerizing a runner
Workflow status badge, DOI
- Configuring a workflow
- Adding an action/workflow status badge to your GitHub repository
- Making Your Code Citable
- An example scheduling_r_scripts
- README badges from usethis R package
- How to write your own R package and publish it on CRAN
- Monitor your GitHub build with a Raspberry Pi pumpkin
12 件可以用 GitHub 完成的很酷的事情
12 cool things you can do with GitHub
Create Websites
Documentation generator
- Use a Static Site Generator to Build a Quick Website. Jekyll, Hugo, Hexo, Gitbook and Pelican. Jekyll was explained in detail.
- Comparison of documentation generators
Rmarkdown
Creating websites in R by Emily C. Zabor. The makes use of Jekyll theme. It covers
- Personal websites: index.Rmd, _site.yml. No special package is required except rmarkdown.
- Package websites: pkgdown package is used.
- Project websites: index.Rmd, _site.yml. No special package is required except rmarkdown.
- Blogs: blogdown package is used.
The tutorial's source code is also on github (Click the Github icon at the top).
Slides
An Introduction to Reproducible Analyses in R and its source
Math
- Problems with adding math formula in README.md
- https://youtu.be/dpVnmxpVdvg using the codecogs RESTful service
- How to support latex in github-pages?
Jekyll
- From help.github.com
- https://www.makeuseof.com/tag/static-site-generator-build-website/
- Get Started With GitHub Pages (Plus Bonus Jekyll)
- Cookbook for R powered by knitr and Jekyll.
- Slower than Hugo as pointed out by YiHui Xie.
- Easy Markdown to Github Pages. The key is go to Settings and pay attention to Github Pages.
- Create a new repository with a README
- Go to Settings - Github Pages. Source = main branch. Theme = Caymen or Architect. It will modify README file. Feel free to do any changes. Then click 'Commit Changes'.
- See my example at https://arraytools.github.io/test-site-Architect
- It does not work by just creating _config.yml file along with README.md without using 'Settings'. You have to change the setting.
- For the free plan, the repository has to be public in order to use Github Pages.
- Jekyll vs. GatsbyJS: Which Static Site Builder Builds the Best Website?
Install Jekyll
on Ubuntu 14.04 I use (the second command will take a while to start),
sudo apt-get install ruby1.9.1-dev sudo gem install jekyll
RStudio
R Blogging on a Chromebook. A simple approach using Jekyll, GitHub, and RStudio Cloud.
Example
- http://www.masalmon.eu/2017/11/16/wheretoliveus/ and its source code in Github.
- https://privefl.github.io/blog/one-year-as-a-subscriber-to-stack-overflow/ and its source
- Singularity-tutorial, source
Script
cd /tmp jekyll new MyNewSite # It will create a new folder 'MyNewSite' with about.md, _config.yml, css (folder), # _includes (folder), index.html, _layouts (folder), _posts (folder) and _sass (folder). cd MyNewSite jekyll serve --watch # It'll say the Server Address: http://127.0.0.1:4000/ # We can open a browser to see a template of html page created by '''jekyll new''' command.
hexo
Hexo is a fast, simple and powerful blog framework. You write posts in Markdown (or other languages) and Hexo generates static files with a beautiful theme in seconds.
Example
Sphinx & Read The Docs
Simple GitHub repo and ReadTheDocs set up
http://tutos.readthedocs.io/en/latest/source/git_rtd.html#creating-repository-on-github
Examples
- http://tutos.readthedocs.io/en/latest/index.html
- https://angus.readthedocs.io/en/2017/rmarkdown_rnaseq.html
- https://geoparse.readthedocs.org/en/latest/
- https://portcullis.readthedocs.io/en/latest/, Portcullis source code, Efficient and accurate detection of splice junctions from RNA-Seq with Portcullis
Raspberry Pi/Python
Creating a project website with Sphinx from Documenting your code.
Hugo
- R blogdown package depends on Hugo.
- Build and host a website with Git
- Install snap from the Snap Store.
Install
Start a blog in 30 minutes with Hugo, a static site generator written in Go (Unix and Windows). Note Hugo generates HTML for you. You then take that HTML and serve it on some web server, such as Apache HTTPD, or Nginx.
On Ubuntu
sudo apt-get install hugo
How To Install and Use Hugo, a Static Site Generator, on Ubuntu 14.04. See the latest binary version (32/64-bit or ARM/ARM64, Windows/Linux/Mac) at https://github.com/gohugoio/hugo/releases/.
wget https://github.com/gohugoio/hugo/releases/download/v0.31.1/hugo_0.31.1_Linux-64bit.deb sudo dpkg -i hugo*.deb # Install the Hugo Themes # SKIP FOR NOW # Install the Pygments Syntax Highlighter sudo apt-get install python-setuptools python-dev build-essential sudo easy_install pip pip install --user Pygments # https://github.com/pypa/pip/issues/4186
How to Install Hugo Site Generator On Ubuntu 18.04 LTS
Create a new site
Follow https://gohugo.io/getting-started/quick-start/
(trusty)brb@localhost:~/Downloads$ hugo new site quickstart
Congratulations! Your new Hugo site is created in /home/brb/Downloads/quickstart.
Just a few more steps and you're ready to go:
1. Download a theme into the same-named folder.
Choose a theme from https://themes.gohugo.io/, or
create your own with the "hugo new theme <THEMENAME>" command.
2. Perhaps you want to add some content. You can add single files
with "hugo new <SECTIONNAME>/<FILENAME>.<FORMAT>".
3. Start the built-in live server via "hugo server".
Visit https://gohugo.io/ for quickstart guide and full documentation.
(trusty)brb@localhost:~/Downloads$ cd quickstart;\
git init;\
git submodule add https://github.com/budparr/gohugo-theme-ananke.git themes/ananke;\
# Edit your config.toml configuration file
# and add the Ananke theme.
echo 'theme = "ananke"' >> config.toml
(trusty)brb@localhost:~/Downloads/quickstart$ hugo new posts/my-first-post.md
/home/brb/Downloads/quickstart/content/posts/my-first-post.md created
(trusty)brb@localhost:~/Downloads/quickstart$ ls -l content/posts
total 12
-rw-rw-r-- 1 brb brb 76 Dec 19 16:00 my-first-post.md
(trusty)brb@localhost:~/Downloads/quickstart$ ls -l
total 40
drwxrwxr-x 2 brb brb 4096 Dec 19 15:58 archetypes
-rw-rw-r-- 1 brb brb 91 Dec 19 16:03 config.toml
drwxrwxr-x 3 brb brb 4096 Dec 19 16:00 content
drwxrwxr-x 2 brb brb 4096 Dec 19 15:58 data
drwxrwxr-x 2 brb brb 4096 Dec 19 15:58 layouts
drwxrwxr-x 6 brb brb 4096 Dec 19 16:20 public
drwxrwxr-x 2 brb brb 4096 Dec 19 15:58 static
drwxrwxr-x 3 brb brb 4096 Dec 19 16:00 themes
(trusty)brb@localhost:~/Downloads/quickstart$ cat config.toml
baseURL = "http://example.org/"
languageCode = "en-us"
title = "TAICHIMD"
theme = "ananke"
# Start the server
(trusty)brb@localhost:~/Downloads/quickstart$ hugo server -D
# prompt will not be returned, Press Ctrl+C to stop
# -D/--buildDrafts: include content marked as draft
Started building sites ...
Built site for language en:
1 of 1 draft rendered
0 future content
0 expired content
1 regular pages created
8 other pages created
0 non-page files copied
1 paginator pages created
0 tags created
0 categories created
total in 27 ms
Watching for changes in /home/brb/Downloads/quickstart/{data,content,layouts,themes,static}
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
Press Ctrl+C to stop
Deploy Your Website
- https://gohugo.io/getting-started/usage/
- How To Deploy a Hugo Site to Production with Git Hooks on Ubuntu 14.04
After running hugo server for local web development, you need to do a final hugo run without the server part of the command to rebuild your site. You may then deploy your site by copying the public/ directory to your production web server.
(trusty)brb@localhost:~/Downloads/quickstart$ hugo Started building sites ... Built site for language en: 0 of 1 draft rendered 0 future content 0 expired content 0 regular pages created 6 other pages created 0 non-page files copied 0 paginator pages created 0 tags created 0 categories created total in 15 ms
Dev vs Deploy Destinations
Directory Structure
Configure Hugo
Hugo themes
- https://themes.gohugo.io/
- https://themes.gohugo.io/tags/documentation/
- https://themes.gohugo.io/slate/ & https://github.com/gesquive/hugo-slate-demo
(trusty)brb@localhost:~$ cd Downloads (trusty)brb@localhost:~$ git clone https://github.com/gesquive/hugo-slate-demo (trusty)brb@localhost:~$ cd hugo-slate-demo/ (trusty)brb@localhost:~/Downloads/hugo-slate-demo$ hugo server -t slate
Examples
Gitlab
roxygen
- https://cran.r-project.org/web/packages/roxygen2/
- http://roxygen.org/roxygen.pdf
- https://stackoverflow.com/questions/26798150/is-there-a-documentation-generator-for-r-like-there-is-for-java
Pelican
Git hooks
- Customizing Git - Git Hooks
- How To Deploy a Hugo Site to Production with Git Hooks on Ubuntu 14.04
- git hooks 实现自动部署
- Simple automated GIT Deployment using GIT Hooks
- Git Hooks Overview from bitbucket
- How To Use Git Hooks To Automate Development and Deployment Tasks on Ubuntu 14.04
- Using a Raspberry Pi as Your Development Server
- post-receive on a remote server (video). Apache web page is used as an example. Note one server-end repository directory and one git ‘local’ directory are used on the remote server.
- How to Use Git Hooks? (with some short examples)
- Git hooks, practical uses (yes, even on Windows)
- https://githooks.com/
- 3 Git hooks for continuous integration
- Using git hooks to improve your day-to-day workflow
- Simple git post-commit hook to copy committed files to a certain folder
- Git commit and other hooks
- Ten Simple Rules for Taking Advantage of Git and GitHub. Rule 5: Let GitHub Do Some Tasks for You: Integrate.
- useful commit hooks for R package dev
- Pre-Commit Hook for Processing README.Rmd
Markdown
- CommonMark - A strongly defined, highly compatible specification of Markdown
Cheat sheet
Software
- Editor/viewer on Ubuntu. See this discussion. Retext
- Multiple platforms.
- Haroopad free, not opensource
- Windows:
- Markdownpad nonfree.
- Texts notfree
- Python solution. Grip
- Online StackEdit
R markdown
- https://www.rstudio.com/wp-content/uploads/2015/02/rmarkdown-cheatsheet.pdf
- http://rmarkdown.rstudio.com/authoring_rcodechunks.html
For code chunks, use
```{r eval=FALSE echo=FALSE}
library(dplyr); library(ggplot2); library(ggthemes);
```
For internal links, see How to link to part of the same document in Markdown?
Tip: just one # for all heading sizes, no space between # and anchor name, anchor tag names must be lowercase, and delimited by dashes if multi-word.
[click on this link](#my-multi-word-header) ### My Multi Word Header
Github Markdown
- https://help.github.com/articles/about-writing-and-formatting-on-github/
- https://help.github.com/articles/basic-writing-and-formatting-syntax/
- Relative links
To show images on readme.md file use
<img src="https://raw.github.com/arraytools/arduino/master/images/%E5%9B%9B%E6%B5%B7%E4%B8%80%E5%AE%B6.jpg"/> # or relative paths (assume readme.md is at the root of your repository <img src="master/images/%E5%9B%9B%E6%B5%B7%E4%B8%80%E5%AE%B6.jpg"/>
For code, use
Some basic Git commands are: ``` git status git add git commit ```
Or
```R library(dplyr); library(ggplot2); library(ggthemes); ```
Bitbucket Markdown
To show images on readme.md, use for example
Note that it seems there is no way to resize the image if we use markdown files.
For code syntax highlight or preformatted text, use 3 ticks (```). See https://confluence.atlassian.com/bitbucketserver/markdown-syntax-guide-776639995.html. For example,
```c++ MainWindow w; ```
Continuous Integration, Travis-CI, Continuous Deployment
- A generic introduction to Gitlab CI
- Continuous integration for your private R projects with CircleCI
- A BEGINNER'S GUIDE TO TRAVIS-CI FOR R
- CI will automatically run R CMD check (along with your tests, etc.) every time you push a commit to GitHub
- Using Travis-CI is a way to check your package on a platform (Ubuntu) that you might not have access to otherwise, much like using build_win()
- Travis CI for R — Advanced guide
- Using Travis-CI to Create R-bloggers for Taiwan
- How to easily automate R analysis, modeling and development work using CI/CD, with working examples
- How to use CI/CD for your ML Projects? Aug 2020. Gitlab was used. R language was used.
- Wercker
- https://en.wikipedia.org/wiki/Wercker
- https://devcenter.wercker.com/
- Mentioned in Infrastructure and Tools for Teaching Computing Throughout the Statistical Curriculum Mine Çetinkaya-Rundel & Colin Rundel 2018
- How to Install Strider Continuous Integration Server on Ubuntu 18.04 LTS
- Building your CI/CD pipeline from lynda.com
- Continuous integration: Developers work locally and commit to a shared repo. That code is integrated with other code. The code base is tested.
- Continuous delivery: Partner to continuous integration. Allows for building, testing, and delivery with every code change.
- Continuous deployment: Deployment that's done automatically, without human intervention.
- Jenkins
- How to Automate Safe Lambda Deployments from Git
Mobile apps
- iOS: Working Copy works fine. It will create a new SSH key and added to your bitbucket account.
Software copyrighted by NIH
- BMDExpress-2.0 - analyze high dimensional dose-response data, in particular gene expression data.
- CBIIT https://github.com/CBIIT
- Genomic Data Commons https://github.com/NCI-GDC
- GDC Data Transfer Tool https://github.com/NCI-GDC/gdc-client
- SRA Toolkit https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=software