Linux

From 太極
Jump to navigation Jump to search

man

Navigation

Gentoo and fossbytes

  • Enter – Move down one line
  • Space – Move down one page
  • g – Move to the top of the page
  • G – Move to the bottom of the page
  • q – Quit

Search within a man page

Use / and type your search pattern.

Use 'n' for forward search and 'N' for reverse search.

The matched line will be moved to the top of the screen.

By default, the search is case insensitive or we can use man -i COMMAND.

Regular expression is supported. For example to find all of the long arguments with: /(--)[a-Z]

Colored man pages

This is a cool tip!

By default, the man program normally uses a terminal pager program such as less to format its output.

Add the following to "~/.bashrc" file 

export LESS_TERMCAP_mb=$'\e[1;32m'
export LESS_TERMCAP_md=$'\e[1;32m'
export LESS_TERMCAP_me=$'\e[0m'
export LESS_TERMCAP_se=$'\e[0m'
export LESS_TERMCAP_so=$'\e[01;33m'
export LESS_TERMCAP_ue=$'\e[0m'
export LESS_TERMCAP_us=$'\e[1;4;31m'

Navigate to another man page within a man page

See here.

When inside the man page, press ! followed by a valid shell command.

For example : !man cat

View a specific "Section"

$ whatis printf
printf (1)           - format and print data
printf (3)           - formatted output conversion
Printf (3o)          - Formatted output functions.
$ man 3 printf

Read man pages in vi without using temporary files

What is a way to read man pages in vim without using temporary files 

man find | vi -

Search man page referenced by

man -f KEYWORD

This command is equivalent to whatis -r KEYWORD

Search from all man pages

man -k KEYWORD will give you a list of all man pages which relate to 'KEYWORD'.

TLDR pages/cheat sheet: alternative to Man

$ tldr tar

A full list of commands is on TLDR Pages.

Cheat.sh (better than TLDR)

Cheat.sh Shows Cheat Sheets On The Command Line Or In Your Code Editor. There are different ways to use it. One way does not require to install anything as long as we have the curl command. 

curl cheat.sh/tar

curl cht.sh/python/random+list # Python programming language cheat sheet for random list

My test shows cheat.sh can find more commands and it gives colored output.

Some books

Beautiful desktop

.desktop file

This is not related to beautiful desktop. It is used to launch applications in Linux. Without the .desktop file, your application won’t show up in the Applications menu and you can’t launch it with third-party launchers such as Synapse and Albert Launcher.

The .desktop files are commonly saved in

  • ~/local/share/applications
  • /usr/share/applications

List of installed desktop environment

ls -l /usr/share/xsessions/

Themes

5 of the Best Linux Dark Themes that Are Easy on the Eyes

Virtual consoles/virtual terminals

Linux allows virtual consoles (aka virtual terminals) to be opened while an X Window System is executing.

Use Ctrl + Alt + FX to open a virtual console-- there are six virtual text-based consoles (F1 to F6). Use Alt + F7 (or possibly other keybinds) to return to the X Window System.

Managing devices in Linux -> Fun with device files.

Change/increase console fonts

Desktops/Workspaces

Ctrl + Alt + -> or Ctrl + Alt + <- to switch workspaces.

Ctrl + Alt + down can list the open applications on the current workspace.

Ctrl + Alt + up can show all workspaces and the open applications. We can use mouse to move an app to any workspace.

SuperKey + left tile a window to left. SuperKey + right tile a window to right.

Complete List of Linux Mint 18 Keyboard Shortcuts for Cinnamon for more examples.

Mouse

How To Bind Mouse Buttons To Keyboard Keys Or Commands (Linux Using X11)

Virtual memory

vmstat

hcache

A tool fork from pcstat, with a feature that showing top X biggest cache files globally

Memory: free command

Three types of memory reported by the free command.

  • Used: RAM that is currently in use by an application.
  • Available: RAM that may be in use for disk caching but can be freed up for applications. What is 'available' vs 'free' memory in free command?. Available = Estimation of how much memory is available for starting new applications, without swapping.
  • Free: RAM that is not in use by an application or disk caching.
  • Total = Used + Free + Buffers + Cache

How to Clear RAM Memory Cache and Buffer 

# To clear pagecache, enter the following command:
echo 1 > /proc/sys/vm/drop_caches

# To clear dentries and inodes, change the number to 2:
echo 2 > /proc/sys/vm/drop_caches

# To clear pagecache, dentries, and inodes all together, change the number to 3:
echo 3 > /proc/sys/vm/drop_caches

How do I determine the number of RAM slots in use

sudo dmidecode -t memory

Logging memory

ps_mem: Shows Per-Program Memory Usage On Linux

https://www.linuxuprising.com/2019/04/psmem-shows-per-program-memory-usage-on.html. In this case I have 6 tabs open in chrome and 12 tabs for firefox. 

$ pip install ps_mem
$ ps_mem -p $(pgrep -d, -u $USER)
 Private  +   Shared  =  RAM used	Program
...
249.9 MiB +  43.9 MiB = 293.9 MiB	firefox
549.7 MiB +  82.2 MiB = 631.9 MiB	Web Content (6)
  1.0 GiB + 149.6 MiB =   1.2 GiB	chrome (16)


                          3.2 GiB

Check RAM information

sudo dmidecode -t 17

Monitor Memory Utilization And Send an Email

Things to do after a fresh install of GNU/Linux

  1. Run upgrade such as apt-get update; apt-get upgrade. It helps to resolve the unmet dependencies issue too.
  2. Increase audio quality
  3. Make sure firewall is enabled.
  4. Disable any unnecessary services
  5. Install Timeshift
  6. Install ClamAV / Clamtk antivirus

Query whether the OS is 64-bit or 32-bit

SYSTEM_ARCH=getconf LONG_BIT
echo $SYSTEM_ARCH

Command line improved

https://remysharp.com/2018/08/23/cli-improved

Directory permission / attribute

See http://unix.stackexchange.com/questions/21251/how-do-directory-permissions-in-linux-work

When applying permissions to directories on Linux, the permission bits have different meanings than on regular files.

  • The write bit allows the affected user to create, rename, or delete files within the directory, and modify the directory's attributes
  • The read bit allows the affected user to list the files within the directory
  • The execute bit allows the affected user to enter the directory, and access files and directories inside

When we create a new directory, the attribute is 775. Some pre-created directories (Desktop, Documents, Music, Pictures, Public) have an attribute 755.

Making a new temporary directory

https://www.howtoforge.com/linux-mktemp-command/ 

mktemp  # temp directory is under /tmp
mktemp -d tempdirXXX # temp directory is under the current directory
mktemp tempfileXXX # temp file under the current directory

Special permissions

s bit - setuid, getuid

Shell

Login shell and non-login shell

Login Shell

  1. /etc/profile
  2. /etc/profile.d/*.sh
  3. ~/.bash_profile or ~/.profile (for example, environment variable like PATH)
  4. ~/.bashrc
  5. /etc/bashrc
  6. ~/.bashrc

Non-Login shell

  1. ~/.bashrc
  2. /etc/bash.bashrc or /etc/bashrc
  3. ~/.bashrc (bash-related settings, for example, prompt string, aliases)

Note: Bash only reads the first of the files in ~/ that it finds (and ignore the rest). rc means run commands for example, .nanorc.

Aliases and Functions for Individual Users

  1. /etc/profile (systemwide environment and shell variables)
  2. /etc/profile.d/*.sh (systemwide environment and shell variables)
  3. ~/.bash_profile (user environment and shell variables)
  4. ~/.bashrc (executes /etc/bashrc)
  5. /etc/bashrc (systemwide aliases and shell functions)
  6. ~/.bashrc (user aliases and shell functions)

Why does it take tens of seconds to get a shell prompt?

https://serverfault.com/a/722496 If your profile or bashrc have expensive things, consider trimming them back.

On raspbian commenting out some lines does help. Interestingly, the same lines does not make any difference on x86 server.

login shell (.bash_profile) vs interactive shell (.bashrc)

  1. login shell - non desktop environment. ~/.bash_profile is sourced for the bash shell. It is the shell you get when logging in or opening a new terminal session.
  2. interactive shell - Ctrl+Alt+t to open a terminal from a graphical mode (desktop environment) and also the ssh connection. ~/.bashrc is source. We usually edit ~/.bashrc to set up the environment to include fancy prompt, set aliases, set history options, or define custom shell functions. Bash Check If Shell Is Interactive or Not Under Linux / Unix Oses
  3. Non-interactive shell - instances of the shell you can't use interactively. Shells that are started to run a command or script.

To determine the shell type: echo $-

export environment variables

  1. Both a login shell and an interactive one. SSH (Putty) to connect to a remote machine.
  2. When a shell runs a script or a command passed on its command line, it's a non-interactive, non-login shell.

Change to root shell

The following command will switch to an environment similar to what the user would expect had the user logged in directly. 

sudo su -
# OR
sudo su
# OR
sudo -s

This can be useful when running 'su' or 'su -' failed because of an authentication failure error (note Ubuntu locked the root account).

See also

For sudo to work, my account ('debian' in this case) has to be included in the config file /etc/sudoers. 

debian  ALL=NOPASSWD: ALL

When sudo is invoked, it asks for the password of the user who started it.

pinky: find out about the people logged on to your Linux computer

How to Use the pinky Command on Linux

Switch user in command line

use 

su newusername

to switch to another user.

Switch to another user and run a command

runuser -l command 

runuser -l  userNameHere -c 'command'

su - command (keep the dash sign after su) 

su - username -c 'command'

sudo su -  # switch to root account
whoami

sudo su - -c "R -q -e \"install.packages('mypackage', repos='http://cran.rstudio.com/')\""
# OR
sudo su -c "COMMAND_REQUIRE_ROOT_ACCESS"
# OR
sudo "COMMAND_REQUIRE_ROOT_ACCESS"

man su

What’s the Difference Between Bash, Zsh, and Other Linux Shells

https://www.howtogeek.com/68563/htg-explains-what-are-the-differences-between-linux-shells/

sh (Bourne shell, 1977)

  • Basic shell
  • POSIX-compliant shell

Bash shell

  • Can be mostly POSIX compliant
  • Expose array indices
  • Regular expression conditionals
  • Increment assignment operator
  • Current version: Bash 4

Bash shell programming

Dash

  • Mostly compatible with Bash
  • POSIX compliant
  • Low-memory usage
  • Faster execution
  • Doesn't have a command history
  • Not easily installed in CentOS 7 (though popular in Debian OS)

zsh shell

sh -c "$(wget https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh -O -)"
nano .zshrc  # change ZSH_THEME to "agnoster"
             # add RPROMPT='%*'
     # add prompt_context(){} at the bottom of your ~/.zshrc to hide the “user@hostname” info
sudo apt install fonts-powerline
  • How to change from zsh to bash? Changing your shell, via sudo chsh --shell=/bin/bash $USER will change the shell field in /etc/passwd. This value is only consulted at login time, when the system has no idea of which shell to use for you. Running processes, whether started by zsh, bash apache or evil_overlord's_program won't be affected.
    • When I try to change the shell to zsh to bash, I use sudo usemod but choose the wrong location '/usr/bin/bash' (it should be /bin/bash). The result is when I try to reconnect to Pi, it always said the password is not correct. Fortunately I can shutdown Pi, move the SD card to chromebox and run sudo nano /var/host/media/removable/rootfs/etc/passwd to fix the bash path.
  • How can you export your .bashrc to .zshrc?
  • Use .zshrc to do things like
    • Export environment variables
    • Set command aliases
    • Set non-default Bash Options by using setopt
  • scp with zsh : no matches found. This is because the wildcard expansion is not enabled by default in zsh. One solution is to add a double quote. For example, scp "username@hostname:Downloads/*.txt" ~/Downloads. Another way is to add this to your .zshrc 
    alias scp='noglob scp'
# OR
setopt nonomatch

The more you customize, the less portable your config can become. The more you add to your shell, the more time it can take to start up and to run basic commands. Be selective when you customize.

Fish shell

  • Most compatible with Bash
  • Designed for interactivity
  • Syntax highlighting on the CLI
  • Interactive autosuggestions
  • Cleaner syntax for conditions and loops
  • Only available in third-party EPEL software repository

Oh My Fish! Make Your Shell Beautiful

Keyboard shortcut to move cursor in the terminal

  • Alt + b: go left (back) one word
  • Alt + f: go forward on word

Example: cd ~/bitbucket/gbmpdx/annovar_biowulf (Now press Alt+b to see the cursor moves)

Tools To Record Your Terminal And Generate Animated Gif or SVG Images

Record and Replay Linux Terminal Sessions Activity: script

Learn how to Record and Replay Linux Terminal Sessions Activity

Recording your terminal: asciinema

https://asciinema.org/

Redirect standard error

http://bash.cyberciti.biz/guide/Standard_error. Use 2> operator. 

command 2> errors.txt

Redirect standard output

This can be used in the cron job or displaying a clock on the desktop. 

$ cat ~/bin/clock
dclock -date "Today is %A %B %Y" -led_off black -bg black -fg yellow -geometry 577x194+119+139   &>/dev/null &

Quotes and asterisk

Combining these two will not work. For example 

brb@T3600 ~ $ ls -l ~/GSE48215/*.fastq
-rw-r--r-- 1 brb brb 16226673016 Jun 14 14:13 /home/brb/GSE48215/SRR925751_1.fastq
-rw-r--r-- 1 brb brb 16226673016 Jun 14 14:13 /home/brb/GSE48215/SRR925751_2.fastq
brb@T3600 ~ $ ls -l '~/GSE48215/*.fastq'
ls: cannot access ~/GSE48215/*.fastq: No such file or directory
brb@T3600 ~ $ ls -l "~/GSE48215/*.fastq"
ls: cannot access ~/GSE48215/*.fastq: No such file or directory

ls command

To use UID/GID instead of the user name and group name in ls -l, use the -n option. 

ls -n

To make a pretty output by showing selected columns (col 9 is the file name and col 5 is the file size) 

$ ls -nt bad |  grep -v ^total | awk '{ printf  "%-20s %15i\n", $9, $5}'
recal.bai                    8069704
recal.bam                12275091222
recal_data.table             1012453
realigned_reads.bai          8065496

Follow the symbolic link

Use -H option 

ls -lH myDir

List only directories

ls -d */
ls -l -d */

List only files

ls -l | egrep -v '^d'

Find and Delete Broken Symbolic Links

find /path/to/directory -xtype l -delete

ls | more without lose color

$ ls --color=auto
$ ls --color | more

Most likely your ls is aliased to ls --color=auto. If you do ls --color (which is morally equivalent to ls --color=always), that will force it to turn on colors.

ls output with color background

stackexchange or askubuntu.

In my case, after I apply chmod 755 -R XXXX, the weird green background color goes away.

ls output selected columns

The following example will sort the output by file size and output only the file size and file name columns. 

ls -lS  . | awk '{print $5 "\t" $9}'

ls -lhS  . | awk '{print $5 "\t" $9}'

# Mac
ls -lS  *.Rmd |head | awk '{print $6 "\t" $10}'
# a range of columns:
ls -lS  *.Rmd |head | awk -v f=6 -v t=10 \
    '{for(i=f;i<=t;i++) printf("%s%s",$i,(i==t)?"\n":OFS)}'

ls suddenly wrapping items with spaces in single quotes

Use ls -N to remove single quotes for files containing spaces. See Why is 'ls' suddenly wrapping items with spaces in single quotes?

Better to add export QUOTING_STYLE=literal to .bashrc

ls on BSD/macOS

Use the -G option to get a color output 

$ ls -G

realpath

realpath FILENAME to get the full path of a file.

tree command

tree -d: show directories only

cp command

Linux cp command tutorial for beginners (8 examples)

copy a directory

cp -avr Dir1 Dir2

where -a will preserve the attributes of files/directories, -v means verbally and -r means copy the directory recursively.

Copy a file with progress bar with pv (plus how to eject the USB drive)

http://www.tecmint.com/monitor-copy-backup-tar-progress-in-linux-using-pv-command/ 

sudo apt-get install pv
pv file1 > file2    # don't forget the ">" operator and the destination is a file, not a directory

After that, instead of clicking the reject icon from the file manager to eject it, it is better to use a command line to do that because there is no expect time for users to know when it will take for finish writing the data to a USB drive. 

sudo apt-get install udisks

sudo udisks --unmount /dev/sdb1   # /dev/sdb1 is the partition
sudo udisks --detach /dev/sdb     # /dev/sdb is the device

My testing shows this procedure works (tested by running md5sum after eject/plug-in) when I need to copy a 9GB file.

Reliable way: Split the large file and copy smaller chunks

# Use 'sudo iotop -o' to monitor the I/O
split -b 4G inputFile  # create xaa, xab, ... files
cat x* > outputFile    # merge them. md5sum check succeeds

type  x* > outputFile  # Windows OS. # https://stackoverflow.com/a/60254

# Use Prefix, and use numeric suffixes starting from 0
split -b 4M -d inputFile inputFile.part
md5sum inputFile
cat inputFile.part* > inputFile2
ech "LONG_MD5_SUM_From_inputFile inputFile2" | md5sum -c

It is interesting copying smaller files (eg 4GB) to USB drives is quite stable (just use the cp command). Even for a not-too large file (6.7GB), pv step looks OK but the unmount/detach step failed.

For a 6.7GB file, it will split it into a 4GB and 2.7GB files. Merge takes longer time if it is done on the USB drive. That is, it is best to do merge in the final destination (internal disk/storage).

  • split in the internal hdd: 1min 38sec
  • merge in the internal hdd: 37sec
  • merge in the USB 3.0 drive: 2min 17sec

Remember: Use a reliable USB drives.

The operation could not be completed because the volume is dirty

On a USB 2.0 drive, I can copy files to there but the drive cannot be rejected (Ubuntu has a pop-up showing it is still writing data to it).

When I forcibly rejects the drive and plug it in a Windows PC, Windows shows the message The operation could not be completed because the volume is dirty. This gives a way to run chkdsk (check and repair a file system).

  1. Open a Windows File Manager
  2. Right click the USB drive
  3. Properties
  4. Tools -> Check now... Start

Done. Now I can use the drive again.

The Linux equivalent to chkdsk is fsck. fsck is a front end that calls the appropriate tool (fsck.ex2, fsck.ex3, e2fsck, ...) for the filesystem in question. 

umount /dev/sdb1        # thumb drive
sudo fsck /dev/sdb1

sudo fsck -a /dev/sdb1  # auto repair

For the root disk, you have to use a live CD. Otherwise, you will see a message like 

$ fsck /dev/sdb1
fsck from util-linux 2.20.1
e2fsck 1.42.9 (4-Feb-2014)
/dev/sdb1 is mounted.

WARNING!!!  The filesystem is mounted.   If you continue you ***WILL***
cause ***SEVERE*** filesystem damage.

Do you really want to continue<n>? no

Files

List files by sorting according to the file size

Use the -S option in ls.

List files using wildcard without showing files under subdirectories

Use the -d option in ls. For example, the following command will not showing files under R-3.4.4 and R-3.5.3 

$ ls -d R*
drwxr-xr-x 15 brb brb     4096 Mar 14 09:48 R-3.4.4
-rw-rw-r--  1 brb brb 30474612 Mar 15  2018 R-3.4.4.tar.gz
drwxr-xr-x 15 brb brb     4096 Mar 14 09:31 R-3.5.3
-rw-rw-r--  1 brb brb 30205979 Mar 11 04:04 R-3.5.3.tar.gz

Delete multiple files

How to Remove Multiple Subdirectories with One Linux Command 

rm -r ~/Documents/htg/{done,ideas,notes}

Delete a certain type of files recursively under a directory

For example to delete *.o files under the current directory, 

find . -type f -name '*.o' -delete

Remove all files/directories except for one file/some file type

https://unix.stackexchange.com/a/153863 

find . ! -name 'file.txt' -type f -exec rm -f {} +

Create a new directory and cd to it

How to Make a New Directory and Change to It with a Single Command in Linux

chown and chmod recursively (-R)

Use -R (capital R). 

chmod -R u=rwx,go=rx /var/www/html

Note it is better not to remove 'x' on folders. Otherwise we will lose the permission to change file attributes for any file under the folder.

Get the chmod numerical value for a file/directory

https://unix.stackexchange.com/a/46921

  1. BSD/OS X: stat -f "%OLp" <file>
  2. Linux: stat --format '%a' <file>, stat -c %a <file>

Files under a directory have question mark attribute

Use sudo chmod -R a+x /some/directory to fix. This happened when I unzip a zip file compressed in a Windows OS.

Files have an integer owner in attributes

What does the “number” in the owner field of files signify in linux?

You probably did a copy that preserved the original group and owner of these files. Within linux internally the owner and group is basically just an id.

You can change the ower and group to an existing owner and group with the commands chown and chgrp respectively.

Uppercase S in permissions of a folder and setGID

I happen to create this case by chmod -R 760 ShareFolder. To make 'S' to become 's', I just need to use chmod -R 2770 ShareFolder.

immutable files

Manage file and directory attributes using chattr and lsattr command

Display files sorted by modified date in a directory recursively

stat --printf="%y %n\n" $(ls -tr $(find DIRNAME -type f))

find -type f -printf '%T+\t%p\n' | sort -n

Both of methods give the same output. Note the latest changed file is shown at the bottom of the output.

Sort files by their size

use the '-S' option. 

ls -lS

Files starting with a dash (meta-characters)

Move File Starting With A Dash 

$ > '-foo.txt'
$ rm "-foo.txt"
rm: invalid option -- 'o'
Try 'rm ./-foo.txt' to remove the file '-foo.txt'.
Try 'rm --help' for more information.
$ rm -- -foo.txt

alias

https://www.cyberciti.biz/faq/how-to-turn-on-or-off-colors-in-bash/ 

$ alias # list all aliases
$ alias | grep ls
$ unalias ls
$ alias ls='ls --color=auto' # save it in ~/.bash_profile or ~/.bashrc
$ alias server_name="ssh -v -l john 192.168.1.11" # or modify /etc/hosts
$ alias open='xdg-open'
$ alias sshnocheck='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'
$ alias sshserver='ssh [email protected]'
$ alias checkport='sudo lsof -i -P -n | grep LISTEN'

To avoid using the alias, use one of the following ways (eg use the command's full path) 

$ \ls
$ /bin/ls
$ command ls
$ 'ls'

Useful aliases (Added to ~/.bashrc) 

alias nano="nano -c --softwrap"

We can use the backslash to escape the double quote (or others like dollar sign).

If a command needs the sudo right, include the command inside alias. In the following example, if we skip "sudo" then running the command "sudo lsof2" will result in an error: sudo: lsof2: command not found. 

alias lsof2="sudo lsof -i -P | egrep \"PID|LISTEN\""

Use unalias [alias name] to remove an alias.

lolcat - bring color to text

sudo apt-get install ruby	
sudo gem install lolcat
lolcat -h
lolcat --version
fortune | lolcat

ps | lolcat
man ls | lolcat

lolcat test.R

sudo apt install figlet
figlet Merry Christmas | lolcat

alias lolls="ls -l | lolcat"
lolls

File manager

Cloud commander

diff

Run diff with large files

Meld freezes When I tested it with two large files (800k & 936k lines coming from human gtf files). Actually the whole linux system became unresponsive.

Actually Meld is sluggish when it is used in small files in Odroid XU4 running Ubuntu 16.04 MATE. I have used Meld 3.14.2 and the latest 3.16.2.

Kompare

https://apps.kde.org/en/kompare

Meld cannot change theme. So if my desktop has a dark theme, meld is hard to read.

Kompare still has a light them.

Directory

diff -qr dir1 dir2

where -q means to report only when files differ and -r is to recursively compare any subdirectories found.

diff & colordiff-color in terminal

PS. For a GUI version of diff, Meld works fine. Need to install first. apt-get install colordiff. http://www.cyberciti.biz/programming/color-terminal-highlighter-for-diff-files/ 

sudo apt-get install colordiff
diff -y file1 file2 | colordiff
# Ignore same rows (two ways):
# diff -C0 file1 file2 | colordiff
# diff -U0 file1 file2 | colordiff

# On systems that I have no root right, I need to install it from the source 
# (just need to run 'make')
$ diff file1 file2 | ~/bin/colordiff-1.0.18/colordiff.pl

where -y option means to show the output in two columns.

Colordiff.png

Interpretation of the diff output:

The first line of the diff output will contain:

  • line numbers corresponding to the first file,
  • a letter (a for add, c for change, or d for delete), and
  • line numbers corresponding to the second file.

In our output above, 2,4c2,4 means: "Lines 2 through 4 in the first file need to be changed in order to match lines 2 through 4 in the second file." It then tells us what those lines are in each file:

  • Lines preceded by a < are lines from the first file (color in red);
  • lines preceded by > are lines from the second file (color in green).
  • The three dashes ("---") merely separate the lines of file 1 and file 2.
2,4c2,4
< I need to run the laundry.
< I need to wash the dog.
< I need to get the car detailed.
---
> I need to do the laundry.
> I need to wash the car.
> I need to get the dog detailed.

colordiff -ur path1 path2

If you change -ur to -urN then that will also show the contents of files that are only present in one of the paths. Colordiff2.png

The meaning of colors can be found in /etc/colordiffrc (man colordiff or colordiff web site)

  • plain=off
  • newtext=darkgreen
  • oldtext=darkred
  • diffstuff=darkcyan
  • cvsstuff=cyan

gnome-terminal

Remember the session

The following is proved working on Ubuntu 18.04 

gnome-terminal --tab --working-directory=$HOME/Downloads \
               --tab --working-directory=$HOME/Documents

Fun: piano

Let Us Play Piano In Terminal Using Our PC Keyboard

Terminals in grids

See Terminal_multiplexer.

Tilix

Terminator

GNU screen

tmux*

Byobu

Guake

Guake / Yakuake / Tilda

Drop down terminals for the GNOME / KDE / GTK Environments. Great for quick access to a terminal!

System date/time, ntpd

$ timedatectl
      Local time: Mon 2019-06-10 08:37:09 EDT
  Universal time: Mon 2019-06-10 12:37:09 UTC
        RTC time: Mon 2019-06-10 12:37:09
       Time zone: America/New_York (EDT, -0400)
 Network time on: yes
NTP synchronized: yes
 RTC in local TZ: no

Linux file timestamps

Linux File Timestamps Explained: atime, mtime, and ctime

Change the date/timestamp of a file - touch

Modify the file relative to its existing modification time 

filename=MyFileName
touch -d "$(date -R -r $filename) - 2 hours" $filename # 2 hours before
touch -d "$(date -R -r $filename) + 2 hours" $filename # 2 hours later

See How can I change the date modified/created of a file?

Find binary file location

  • which - Display the full path of shell commands. See examples from cyberciti.biz.
$ which ls
/bin/ls
  • whereis - locate the binary, source, and manual page files for a command. See examples from cyberciti.biz.
$ whereis ls
ls: /bin/ls /usr/share/man/man1p/ls.1p.gz /usr/share/man/man1/ls.1.gz
  • type -a
$ type -a ls
ls is aliased to `ls --color=tty'
ls is /bin/ls
  • locate. Use locate command mindfully. It is used to find the location of files and directories. Note that locate does not search the files on disk rather it searches for file paths in a database. For example, the following command will search .png files over the system (not only the personal directory).
locate "*.png"
See How to use updatedb command as an ordinary user?, Use updatedb and locate to index files.

find: Find a file

-iname

Search by a file name 

$ find . -type f -name "abc*" # find a file starting with abc
$ find . -iname '*.txt'  # -iname or -name is necessary

It also works for searching files on subdirectories. 

$ find . -name transcripts.gtf
./RH_bio/transcripts.gtf
./dT_ori/transcripts.gtf
./dT_tech/transcripts.gtf
./dT_bio/transcripts.gtf
./RH_ori/transcripts.gtf
./RH_tech/transcripts.gtf

-mtime, -ctime, -atime argument

Find files modified in 10 days. 

$ find . -ctime -10 # include subdirectories
.
./mediawiki-1.32.0.tar.gz
./d
./d/deepSurv.pdf

Find files modified in one day and contain string 'est' 

$ find . -mtime -1 -exec grep --with-filename est {} \;

If the search directory is not the current directory, we need to add a forward slash to the directory name. 

$ find ~/Desktop -iname '*.txt'  # Not working
$ find ~/Desktop/ -iname '*.txt' # Working

The following example shows we can list multiple search criteria. The “‑r” option in tar appends files to an archive. xargs is a handy utility that converts a stream of input (in this case the output of find) into command line arguments for the supplied command (in this case tar, used to create a backup archive). 

find / -type f -mtime -7 | xargs tar -rf weekly_incremental.tar
gzip weekly_incremental.tar

What is the difference between mtime, atime and ctime?

  • mtime (modification time) indicates the time the contents of the file has been changed. Mind you, only the contents. Not the attributes.
  • ctime (change time) is the timestamp of a file that indicates the time that it was changed. Now, the modification can be in terms of its content or in terms of its attributes.
  • atime (access time) is the timestamp that indicates the time that a file has been accessed.

xargs

See Linux Programming

-exec COMMAND {} +

find . -exec grep chrome {} \;
# or
find . -exec grep chrome {} +

find will execute grep and will substitute {} with the filename(s) found. The difference between ; and + is that with ; a single grep command for each file is executed whereas with + as many files as possible are given as parameters to grep at once. The backslash before ; is to escape ; so linux won't interpret it directly.

  • Find files and execute something (google: find --exec)
$ find ./ -name "*.tar.gz" -exec tar zxvf {} \;
  • Find and move files to a new directory
find OLDDIR -type f -exec mv -t NEWDIR {} +

The following will find out the total file size of the 'accepted_hits.bam' file under all sub-directories. 

find ./ -iname "accepted_hits*" -exec du -ch {} + | grep total$

where '-c' produces a grand total, and will substitute {} with the filename(s) found in -exec.

How to find and delete directory recursively

How to find and delete directory recursively on Linux or Unix-like system. Application: recursively remove backups older than 30 days.

Recursive statistics on file types in directory?

You could use find and uniq for this. This is fast! 

$ find . -type f | sed 's/.*\.//' | sort | uniq -c

Exclude or Ignore Files

Find command Exclude or Ignore Files (e.g. Ignore All Hidden .dot Files )

Avoid Permission Denied Messages

How to fix find command permission denied messages

  • Redirecting ALL standard error (not only permission denied error): 2>/dev/null.
find . -iname "data*.txt" -print 2>/dev/null
  • Focus on the 'permission denied' message: grep -v "Permission denied" 
find / -name foo 2>&1 | grep -v "Permission denied"

Find Files That Have Been Modified Recently in Linux

Find Files That Have Been Modified Recently in Linux

grep: Find a file by searching contents

grep -r -i "Entering" ~/Downloads/R-3.0.0/

where -r means recursively searching the directory and -i means case insensitive.

Sometimes using -R is more effective because of the symbolic links issue. 

$ grep -r -i phpmyadmin /etc/apache2/  # nothing returned
$ grep -R -i phpmyadmin /etc/apache2/
/etc/apache2/conf-enabled/phpmyadmin.conf:# phpMyAdmin default Apache configuration
...
/etc/apache2/conf-available/phpmyadmin.conf:# phpMyAdmin default Apache configuration
...

We can also display the row numbers for matches by using the -n parameter in grep. 

# What variants appear in dbsnp
grep -n 'rs[0-9]' XXX.vcf

To exclude lines with a pattern, using the -v parameter. 

# How many variant were called
grep -v "^#" XXX.vcf | head

To exclude binary files, use -I parameter.

To show only matched filenames, using the -l parameter. 

grep -l "iterator" *.cpp
# if we add '-n', the '-n' option won't work.

To search with certain file extensions, use --include argument; see this post. 

grep -r -i --include \*.h --include \*.cpp KEYWORD ~/path[12345]  
# escape with \ just in case you have a directory with asterisks in the filenames

If the pattern is saved in a file, use the -f parameter 

grep -f PATTERNFILE INPUTFILE

If there are two keywords, use the following 

$ grep "begin\|completed" --color swarm_58606147_0.o  # needs an escape
begin 2018-01-12 14:46:05
alignment is completed 2018-01-12 16:45:24
marking duplication is completed 2018-01-12 17:52:01
assign read group is completed 2018-01-12 18:22:49
indel re-alignment is completed 2018-01-12 19:29:32
BQSR is completed 2018-01-12 22:26:22
GATK is completed 2018-01-12 23:43:3
$ egrep "begin|completed" --color swarm_58606147_0.o 
# no need an escape if we use extended regular expressions

We can use R to compute the time spent in each step; see Dealing with dates.

Check https://www.howtoforge.com/tutorial/linux-grep-command/ for more examples

  • Using grep to search only for words / exact match ("-w" option)
  • Using grep to search two different words (egrep -w 'word1|word2' /path/to/file)
  • Count line for matched words ("-c" option)
  • Grep invert match ("-v" option)
  • How to list only the names of matching files ("-l" option)

Preserve colouring after piping grep to grep

https://stackoverflow.com/a/2327216

Use grep --color=always .For example, grep --color=always KEYWORD Myfile | more.

GUI

A GUI version of a tool to search files is searchmonkey (open source, Linux, Windows). On Ubuntu, we install it by 

sudo apt-get install searchmonkey

It is also useful to change the settings so we can click a filename and open it in the desired text editor. To do that, go to Settings -> Preferences -> System Call -> Text Editor. I enter 'geany' since I want to use geany to open my C programs. Note. the v2.0 source code needs to be built using i386 gcc library and Qt 4.8.x. Still, I cannot get rid of some errors coming from the source code.

Summary of find and grep commands

Command Examples
find find [DIRECTORY] -iname '*.txt'

find [DIRECTORY] -maxdepth 2 -iname *.php

find -name '*.php' -o -name '*.txt' # OR operator

grep grep -r -i "check_samtools" DIRECTORY/

dpkg -l libgtk* | grep '^i'

Count number of columns: awk

The following command shows the number of columns for the first few rows of a text file. 

head MYFILE | awk '{ print NF}'

head MYFILE | awk -F '\t'  '{ print NF}'

Count number of rows in a file: wc

wc -l MYFILE

The source code of wc (or any Linux command) can be found by using this method 

brb@brb-T3500:~/Downloads$ which wc
/usr/bin/wc
brb@brb-T3500:~/Downloads$ dpkg -S /usr/bin/wc
coreutils: /usr/bin/wc
brb@brb-T3500:~/Downloads$ sudo apt-get source coreutils

As we can see from the coreutils-8.21/src directory, there are over 100 C programs including <cat.c>, <chmod.c>, <cp.c>, ...<wc.c>.

Print certain rows/lines of a text file

The following example will print out lines 10 to 60 of FILENAME. 

sed -n '10,60p' FILENAME

Or to print out line 60, 

sed -n '60p' FILENAME

It seems this method is not as fast as I expected. For example, the tail command will immediately print out the result without waiting!

Print a text file with line number: less

How to Use the less Command on Linux 

less -N myfile

output colored console to html

Use ansi2html.sh. It only requires gawk.

  1. Use wget to download it
  2. sudo apt-get install gawk
  3. chmod +x ansi2html.sh
  4. colordiff file1 file2 | ./ansi2html.sh > diff.html

using a the result of a diff in a if statement

ls -lR $dir > a
ls -lR $dir > b

DIFF=$(diff a b) 
if [ "$DIFF" != "" ] 
then
    echo "The directory was modified"
fi

Another example 

if [ "$(diff file1.html file2.html)" == "" ]; then echo Same; else echo Different; fi

Prompt

Colored prompt

For example, the following code will change the prompt to a light blue color. NOTE that we need ∖[ and ∖] in order to avoid a problem of miscalculating the cursor's starting position. 

# blue   
export PS1='\[\e[1;34m\]\u@\h:\w\$ \[\e[0m\]'     # bright blue (good)
export PS1='\[\e[0;34m\]\u@\h:\w\$ \[\e[0m\]'     # darker blue

# yellow
export PS1='\[\e[1;33m\]\u@\h:\w\$ \[\e[0m\]'     # bright yellow
export PS1='\[\e[0;33m\]\u@\h:\w\$ \[\e[0m\]'     # dark yellow (good)

# red
export PS1='\[\e[1;31m\]\u@\h:\w\$ \[\e[0m\]'     # bright red
export PS1='\[\e[0;31m\]\u@\h:\w\$ \[\e[0m\]'     # dark red (good)

# green
export PS1='\[\e[1;32m\]\u@\h:\w\$ \[\e[0m\]'     # bright green
export PS1='\[\e[0;32m\]\u@\h:\w\$ \[\e[0m\]'     # dark green

# cyan
export PS1='\[\e[1;36m\]\u@\h:\w\$ \[\e[0m\]'     # bright cyan
export PS1='\[\e[0;36m\]\u@\h:\w\$ \[\e[0m\]'     # dark cyan (good)

# purple
export PS1='\[\e[1;35m\]\u@\h:\w\$ \[\e[0m\]'     # bright purple (good)
export PS1='\[\e[0;35m\]\u@\h:\w\$ \[\e[0m\]'     # dark purple

Ps1tweak.png

To make a permanent change, we can add the line to ~/.bashrc file and (is it necessary) un-comment the following line 

force_color_prompt=yes


Some of my settings in .bashrc file 

# Office Linux Mint, bright blue, display current time
# Example: 12:45PM ~/Downloads$
PS1="\[\033[1;34m\]\$(date +%H:%M%p) \w$\[\033[0m\] "

# Biowulf and Helix, dark yellow
# Example: biowulf:~/R$
PS1='\[\e[0;33m\]\h:\w\$ \[\e[0m\]'
alias nano="nano -c --softwrap"

# Mac, light green
export PS1="\[\e[0;32m\]mac$\[\e[0m\] "

Add a timestamp to your Bash prompt

PS1 Prompt
default brb@p45t:~/Downloads$
PS1='[\D{%F %T}] \u@\h \W\$ ' [2016-07-08 16:56:48] brb@brb-P45T-A ~/Downloads$
PS1="\[\033[1;34m\]\$(date +%H:%M%p) \w$\[\033[0m\] " 10:54AM ~/Downloads$
From here, we can skip %F (not showing the date), \W (not showing the current directory) and change %T to %H:%M (not showing seconds).
export PROMPT_COMMAND="echo -n \[\$(date +%H:%M%p)\]\ "

and the output will be something like: 

[07:03AM] user@hostname:~$

zsh: display time to the right hand side

add a line RPROMPT='%*' to ~/.zshrc

Proxy

The 15 Best Web Proxies for Geo-Blocked Content and Online Privacy

Listen to pandora in Europe: install squid proxy

http://www.cyberciti.biz/faq/access-pandora-radio-using-proxy-server-outside-usa/

Interestingly, the firefox connection settings should choose HTTP Proxy instead of 'SOCKS host'.

SSH, scp

See ssh.

Graphical way to display disk usage

For example, to use xdiskusage, we run apt-get install xdiskusage and launch it by xdiskusage ~/.

  • Ubuntu has a built-in program called "Disk Usage Analyzer". Just search it from Dash. Looks useful!

df : Display disk space

df -h
df -h -T  # show the 't'ype of the file system like tmpfs, ext4, squashfs (snap), vfat
df -h -t ext4 # show file systems of given type (ext4 in this example)
df -a     # show all file system (include ones that have a size of zero blocks)

df -h | grep -v snap # ignore snap partitions
df -h | grep -v loop

Note for the NTFS type, it will be reported as fuseblk by mount or df command.

Disk encryption

LUKS

rm command and trash can

Make “rm” Command To Move The Files To “Trash Can” Instead Of Removing Them Completely

du/ncdu and block size: Display directory size with sorting and human readable

Use ncdu program (more interactive). Although it is a command line program, we can use the mouse to move through each directory to see its sub-directories.

ncdu can show the hidden directory size. This is useful. For example, ~/.local/share/Trash and ~/.singularity/docker can take a lot of space. 

sudo apt-get install ncdu
ncdu

And the du method. 

du -sh ~/*              # won't include hidden directories, Fast
du -h ~/ --max-depth=1  # include hidden directories, SLOW
du -h ~/ --max-depth=1 --exclude ".*" | sort -nr | cut -f2 | xargs -d '\n' du -sh
du -a -h ~/  # kilobytes will be used, '-a' is to see all files, not just directories.
du -a ~/ | sort -nr | head -n 10   # sort from the largest file size first

The --exclude is to hide hidden directories, '-n' is to compare according to string numerical value, and '-r' is to reverse the result.

Note that the 'du' commands may be cheating. See the following screenshot.

DiskUsage.png

The discrepancy is explained by 'sector'. See http://askubuntu.com/questions/122091/difference-between-filesize-and-size-on-disk. Note: it seems 4096 is what I see from all devices. 

$ sudo dumpe2fs /dev/sda1 | grep -i "block size"
dumpe2fs 1.41.14 (22-Dec-2010)
Block size:               4096

$ sudo tune2fs -l /dev/sda1 | grep -i "block size"

$ sudo fdisk -l | tail
$ sudo tune2fs -l /dev/mmcblk0p2 | grep -i "block size"

To show a file size in terms of blocks, we can use 

ls -s

So for example, if a file takes 150 blocks, and if a block takes 4096 bytes, then the file takes 150*4096/1024 KB on disk.

gdu

gdu Is A Fast Console Disk Usage Analyzer (Alternative To ncdu, du, Etc.)

Apache benchmark (ab) testing

ab -n 100 -c 10 http://taichimd.us/

Monitor progress of copying/transferring files: pv

How to monitor progress of Linux commands using PV and Progress utilities 

# Method 1: rsync
rsync --progress -a sourceDirectory destinationDirectory
rsync --info=progress2 source dest

# Method 2: pv
sudo apt-get install pv
## copy a single file
pv inputfile > outputfile

## multiple files or directories 
tar c sourceDirectory | pv | tar x -C destinationDirectory

## https://stackoverflow.com/a/26226261
docker save <image> | bzip2 | pv | \
     ssh user@host 'bunzip2 | docker load'

rsync

See Backup.

Wireless File Transfer Apps on Linux

The 7 Best Wireless File Transfer Apps on Linux

sudo

How to Control sudo Access on Linux

https://www.howtogeek.com/447906/how-to-control-sudo-access-on-linux/

How to Keep ‘sudo’ Password Timeout Session Longer in Linux

http://www.tecmint.com/set-sudo-password-timeout-session-longer-linux/

How to run multiple commands in sudo

https://www.cyberciti.biz/faq/how-to-run-multiple-commands-in-sudo-under-linux-or-unix/

How do I run specific sudo commands without a password?

https://askubuntu.com/questions/159007/how-do-i-run-specific-sudo-commands-without-a-password

Alternative browsers, text browsers

See Browser.

Filezilla

Keyboard shortcut. Especially, Alt+Down=Transfers the currently selected item to an item of the same name in the other pane.

The device is busy

brb@brb-P45T-A:~$ sudo umount /media/brb/TOSHIBA 
[sudo] password for brb: 
umount: /media/brb/TOSHIBA: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
brb@brb-P45T-A:~$ sudo umount /dev/sdc1
umount: /media/brb/TOSHIBA: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
brb@brb-P45T-A:~$ lsof /media/brb/TOSHIBA/
COMMAND  PID USER   FD   TYPE DEVICE   SIZE/OFF NODE NAME
VBoxSVC 5600  brb   18w   REG   8,33 4294967295    3 /media/brb/TOSHIBA/Windows 10.ova (deleted)
brb@brb-P45T-A:~$ kill -9 5600
brb@brb-P45T-A:~$ lsof /media/brb/TOSHIBA/
brb@brb-P45T-A:~$ sudo umount /dev/sdc1
brb@brb-P45T-A:~$ 

# fuser -m /dev/sdc1
/dev/sdc1: 538
# ps auxw|grep 538
donncha 538 0.4 2.7 219212 56792 ? SLl Feb11 11:25 rhythmbox

Another handy one is: 

umount -l /dev/sdwhatever

mkfs command

dd and mkfs

How to Use the mkfs Command on Linux 

dd if=/dev/zero of=~/howtogeek.img bs=1M count=250
mkfs.ext2 ~/howtogeek.img
sudo mkdir /mnt/geek
sudo mount ~/howtogeek.img /mnt/geek
sudo chown dave:users /mnt/geek/

cd /mnt/geek
cp ~/Documents/Code/*.? .

sudo umount /mnt/geek
cd /mnt
sudo rmdir geek

Format a USB drive

Easily Format A USB Flash Drive On Ubuntu 18.04 Using USB Stick Formatter (mintStick deb & source)

This is a GUI application. After the installation, search "USB Stick Formatter".

For some reason, it doesn't have the 'exFAT' option. My system has installed exFAT drivers. This post said installing exFAT related drivers only helps reading/writing but not formatting.

If I want exFAT format, I need to use the USB Stick Formatter to format the drive first (for example fat32), plug it and then using the following command to format it to exFAT. 

sudo mkfs.exfat -n Staples /dev/sdc1

Note that fdisk or sfdisk cannot differentiate NTFS/exFAT. But cfdisk or GParted can. 

sudo cfdisk /dev/sdX

Create an ext3/ext4 file system on a USB flash drive

Best reliable way is to use Ubuntu's Disks utility. 

umount /dev/sdb1   (depending on the device of course)
sudo mkfs.ext3 /dev/sdb1
sudo e2label /dev/sdb1 usbdrive   (change the label)

We can create MS-DOS file system by 

sudo mkfs.vfat /dev/sdb1

exFat - cross platform partition format

sudo apt-get install exfat-utils exfat-fuse
# Still need to create a partition (ex. FAT32) first using gparted in order to get it mounted
sudo fdisk -l 
sudo mkfs.exfat -n LABEL /dev/sd**  #  LABEL with whatever you want to label your drive

This should delivery a working exfat file system (read and write support, but not formatting the drives with exfat via Gnome Disks and GParted).

Add a new user with home directory

adduser xxx

adduser is better than useradd since useradd does not create home directory and it does not even ask the password for new user. adduser will interactively ask user information.

To delete the user and home directory, use 

deluser --remove-home xxx

To view the user information, type id USERNAME or cat /etc/passwd.

gzip with multi cores

Use pigz utility. It makes a lot of difference. For example for a 21GB file, gzip can't finish the job after 30 minutes. But pigz only took 7 minutes on a 12-core machine. 

sudo apt-get install pigz
pigz -9 FILENAME   # compress & convert the file to FILENAME.gz

tar cf - paths-to-archive | pigz -9 -p 12 > archive.tar.gz

There is no need to use pigz to un-compress the file. gunzip is fast enough and only takes 4 minutes to decompress.

The '-9' (best compression) option does not make difference (6.6G vs 6.5G).

Compress a folder without full path name

Suppose we want to compress the folder ~/Documents and its subfolders. We want to include Documents folder name but not /home/brb/Documents name. 

# Method 1. Include 'Documents' as the top folder name
cd ~/
tar -czvf tmp.tar.gz Documents
# Method 2. Mind the last dot. Not include 'Documents' as the top folder.
tar -czvf tmp.tar.gz -C /home/brb/Documents .

# Double check the tarball
tar -tzvf tmp.tar.gz

If we want to strip the upper directories when we uncompress a tar file, use --strip-components. For example, we can use --strip-components=1 to remove the Documents folder.

Fix mess created by accidentally untarred files in the current dir

Suppose I accidentally untar a tarball in /var/www/html/ directory instead of /home/projects/www/current. It created mess in /var/www/html/. The easiest way to fix this mess: 

cd /var/www/html/
/bin/rm -f "$(tar ztf /path/to/file.tar.gz)"
## or better ##
tar ztf /path/to/file.tar.gz | xargs -d'\n' rm -v

lzma

squashfs

squashfs is a heavy-compression based read-only filesystem that is capable of compressing 2 to 3 GB of data onto a 700MB. Linux liveCD are built using squashfs. These CDs make use of a read-only compressed filesystem which keeps the root filesystem on a compressed file. It can be loopback mounted and loads a complete Linux env. Thus when some file are required by processes, they are decompressed and loaded onto the RAM and used. 

# create a squashfs file
sudo mksquashfs /etc test.squashfs 

# mount the squashfs file 
mkdir /mnt/squash
mount -o loop compressedfs.squashfs /mnt/squash
# you can acess the contents at /mnt/squashfs

# exclude files 
sudo mksquashfs /etc test.squashfs -e /etc/passwd /etc/shadow
# or specify a list of exclude files given in a file
cat excludelist  # /etc/passwd
sudo mksquashfs /etc test.squashfs -ef excludelist

List contents of tar.gz or tar.bz2

tar -tzvf myfile.tar.gz

tar -tjvf myfile.tar.bz2  # replace z with j

gzip: stdin: not in gzip format

I got the following message when I try to run tar -xzvf or tar -tzvf command. 

$ tar -tzvf filename.tar.gz 
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
$ file filename.tar.gz 
filename.tar.gz: POSIX tar archive

The answer is How to solve: stdin: not in gzip format.

Solution: Since it was not a gzipped file, a simple tar is able to extract the file: tar xvf MyFile.tar.gz

Extract files, AVFS

See Extract files.

Show folder size for one level only

du --max-depth=1 -h

The graphical tool is called Disk Usage Analyze which is already available on Ubuntu.

Soft link and hard link

Soft link

ln -s /full/path/of/original/file /full/path/of/soft/link/file

Understanding Linux Links Part 1 & Part 2

Hard link

What's the difference between a hard links and copied files?

  • Why are hard links not allowed for directories?
  • Hard linked file looks the same as the source file when viewed by "ls"
  • Change in either of hard linked file and the source file will affect the other
  • Hard linked file and source file can be deleted separately without affecting the other
echo "abcd" > foo
ln foo foo2
ls -l foo*
echo "efgh" >> foo2
cat foo
rm foo
cat foo2

Self-hosted servers

  • https://github.com/Kickball/awesome-selfhosted This is a list of Free Software network services and web applications which can be hosted locally. Selfhosting is the process of locally hosting and managing applications instead of renting from SaaS providers.
  • Sovereign: A set of Ansible playbooks to build and maintain your own private cloud: email, calendar, contacts, file sync, IRC bouncer, VPN, and more.

Cockpit: manage and monitor my servers using just a web browser

DNS

DNS

Port number is 53. An example is Pi-hole.

Email server

See Mail_server.

Backup

See Backup.

Running a cron job as a user

Some examples

MIN HOUR DOM MON DOW CMD
30 08 10 06 * /home/ramesh/full-backup 
#   30 – 30th Minute
#   08 – 08 AM
#   10 – 10th Day
#   06 – 6th Month (June)
#   * – Every day of the week
  • Twice a day
00 11,16 * * * /home/ramesh/bin/incremental-backup 
#    00 – 0th Minute (Top of the hour)
#    11,16 – 11 AM and 4 PM
#    * – Every day
#    * – Every month
#    * – Every day of the week
  • Every 10 minutes
*/10 * * * * /home/ramesh/check-disk-space

# Will only run on odd days:
0 0 1-31/2 * * command

# Will only run on even days:
0 0 2-30/2 * * command

crontab

crontab SOME-CRON-FILE; crontab -l

Make sure the .sh file gives a complete path. For example, 

#!/bin/sh
R --vanilla  < arraytoolsip.R

does not work in cron job although it works perfect when we manually run it from the right path. The sh file should be 

#!/bin/sh
R --vanilla  < $HOME/Dropbox/scripts/arraytoolsip.R

To disable everything on crontab -l, use one of the following methods:

  • run crontab -e then comment out each line you don't want to run with #. OR
  • run crontab -r to empty the current crontab. OR
  • run crontab with no arguments, and then type Ctrl+D. It will create an empty crontab, overwriting your previous crontab.

GUI

Schedule Commands And Scripts In Linux With Zeit (GUI For Cron And At)

PATH and Shell

Cron knows nothing about your shell; it is started by the system, so it has a minimal environment. If you want anything, you need to have that brought in yourself. For example, to use 'ifconfig' command, I need to give it a complete path in my script file. 

$ cat syncIP 
/sbin/ifconfig eth0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}'

and the cron job 

06 15 * * * /home/MYUSERNAME/Ubuntu\ One/syncIP > $HOME/Ubuntu\ One/ip.txt 2>&1

See here on how to add environment variable into cron environment.

Disable mail alert

If something went wrong with executing a cron job, cron will output a message "You have new mail in /var/mail/$USER". You can open this file using a text editor. To disable this alert, run 'crontab -e (see this post 

0 1 5 10 * /path/to/script.sh >/dev/null 2>&1
# OR
0 1 5 10 * /path/to/script.sh > /dev/null

Run a command at boot

rc.local

How to enable rc.local shell script on systemd while booting Linux system

Running crontab as root

Use sudo crontab -e to edit. After saving it, no need to initialize it. Use sudo crontab -l to list the cron job.

Display and back up cron jobs

Linux List / Display and view all cron jobs

Check log

sudo grep CRON /var/log/syslog --color

Anacron

Anacron keeps track of the last time a task was run, and if it was missed, it runs it.

Anacron typically runs daily, while cron runs every minute. 

cat /etc/anacrontab

GUI cron

md5sum

Linux md5sum Command Explained For Beginners (5 Examples)

How to verify files? 

md5sum file1.txt file2.txt file3.txt > hashes
md5sum --check hashes

fsck

Fsck error on boot

fsck error on boot: /dev/sda6: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY

This happened when I resize an Ubuntu partition.

fsck -fy /dev/sda1

Force fsck on the Next Reboot or Boot Sequence

https://www.linuxuprising.com/2019/05/how-to-force-fsck-filesystem.html

https://www.cyberciti.biz/faq/linux-force-fsck-on-the-next-reboot-or-boot-sequence/ Create a blank file /forcefsck and fsck will check your drive next time your reboot. 

sudo touch /forcefsck

The fsck was used to fix a journal checksum error on a USB drive which has been formatted as Ext4 was used on a security camera application; see motionEyeOS.

Can I run fsck or e2fsck when Linux file system is mounted?

https://www.cyberciti.biz/faq/can-i-run-fsck-or-e2fsck-when-linux-file-system-is-mounted/

What is the difference between fsck and e2fsck?

https://superuser.com/a/19984

Swap partition

Swap file vs swap partition

Jetson Nano – Run on USB Drive

Mount drive, add a new hard drive

/etc/fstab and blkid

$ sudo blkid  #  list devices even not mounted yet

and the result should be compared with 

$ sudo fdisk -l
  • Run sudo mount -a to remount /etc/fstab without reboot, except the partitions with noauto option.

The following example shows a problem (as found from the output of df command) with </etc/fstab> where we use /dev/sdXY instead of UUID for specifying hard disks. 

$ sudo blkid
/dev/sda1: LABEL="WD640" UUID="d3a0a512-bf96-4199-9674-f410f22f0a92" TYPE="ext4"
/dev/sdb1: UUID="afaa4bde-1172-4c54-8b0a-a324ad855355" TYPE="ext4"
/dev/sdb5: UUID="fb2a4ada-d80a-4e23-b4a2-67376b8b7e72" TYPE="swap"

$ sudo fdisk -l
Disk /dev/sda: 640.1 GB, 640135028736 bytes
...
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048  1250263039   625130496   83  Linux

Disk /dev/sdb: 640.1 GB, 640135028736 bytes
...
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *        2048  1217761279   608879616   83  Linux
/dev/sdb2      1217763326  1250263039    16249857    5  Extended
/dev/sdb5      1217763328  1250263039    16249856   82  Linux swap / Solaris

$ cat /etc/fstab
proc            /proc                                 proc    nodev,noexec,nosuid 0 0
UUID=afaa4bde-1172-4c54-8b0a-a324ad855355 /           ext4    errors=remount-ro   0 1
UUID=fb2a4ada-d80a-4e23-b4a2-67376b8b7e72 none        swap    sw                  0 0
/dev/sdb1       /mnt/WD640                            ext4    rw,nosuid,nodev     0 2
UUID=fb2a4ada-d80a-4e23-b4a2-67376b8b7e70 /mnt/extUSB auto nosuid,nodev,nofail 0 2
UUID=2ab36808-038a-4dfa-ad52-c10944cf61f2 /mnt/hdd  ext4  defaults  0  2

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       572G  413G  130G  77% /
...
/dev/sdb1       572G  413G  130G  77% /mnt/WD640

To fix the error here, modify the line starting /dev/sdb1 in /etc/fstab and replace it with the UUID. Then run sudo umount /mnt/WD640 and sudo mount -a. Done! 

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       572G  413G  130G  77% /
...
/dev/sda1       587G  283G  275G  51% /mnt/WD640

autofs, /etc/auto.master

Building a network attached storage device with a Raspberry Pi

Fix a malfunctioning USB device or port

5 Ways to Fix a Malfunctioning USB Device or Port on Linux

Check the physical health of a USB stick

Linux check the physical health of a USB stick

USB drive

Run the following to confirm the USB device is detected. 

sudo fdisk -l
# OR
dmesg | grep -i "SCSI"

Now suppose the usb device is found in dev/sdb1. 

sudo mkdir /mnt/usb
sudo mount -t vfat -o rw,users /dev/sdb1 /mnt/usb

The above mount command assumes the usb drive has Windows vfat partition and users give non-root users the ability to unmount the drive. If the USB drive is partitioned linux ext2/3, we can merely run mount command as 

sudo mount /dev/sdb1 /mnt/usb

At the end, run umount command like 

sudo umount /mnt/usb

To make the mounting automatically, edit the file /etc/fstab. 

/dev/sdb1       /mnt/usb           vfat    defaults        0       0 
/dev/sdb2       /mnt/usb2          ext3    defaults        0       0
UUID=XXXXXXXXXX /mnt/usb3       ntfs-3g    rw              0       0

and run 

sudo mount -a

Mount an iso file

sudo mkdir -p /mnt/mount_point # create a mount point
sudo mount -o loop /home/user/disk.iso /mnt/mount_point
mount  # verify

Mount remote Windows share

Sharing files with Windows by using NitroShare

Simple way of Sharing files between Ubuntu 16.04 and Windows 10 by using open-source NitroShare which is based on Qt framework.


NTFS usb drive in xubuntu

http://xflinux.blogspot.com/2011/01/mount-ntfs-volumes-automatically-in.html 

sudo apt-get install ntfs-config

Now go to Applications>> System>> Ntfs Configuration Tool

Expand the "Advanced Configuration" and select all those partitions you want to be auto mounted and writable( The tool will detect all partitions at its startup).

Make sure the " Enable write support for internal devices" option is selected. Now click Close.

Many drives, one folder

Partition tables

Partition Tables and the Dangers of Editing Them

parted command

How to partition a disk in Linux

Recommended partition schemes

HOME /home directory

How to Move Your Linux home Directory to Another Drive

/var directory filled up

How to move /var directory to another partition 

blkid | grep sdc1  # get UUID 
mkdir /mnt/newvar
mount /dev/sdc1 /mnt/newvar
df -h /mnt/newvar
rsync -aqxP /var/* /mnt/newvar  # q=quiet,x=one-file-system, P=partial,progress
umount /mnt/newvar/  /mnt/var/
nano /etc/fstab
# UUID=XXXX   /var    ext4  defaults  0   2

Reboot

Why put things other than /home to a separate partition?

Why put things other than /home to a separate partition?

The /var partition is used by Docker and Apache.

/home, /boot and /var partitions can be separated.

Process/job

ps and top commands

How to Use the ps Command to Monitor Linux Processes, Linux how long a process has been running? 

ps -C shutter # Listing only a Process by Command
              # Adding the 'watch' command to show the process in real-time
ps -C dd --format pid,cmd,%cpu # Show PID, CMD and %CPU

sudo ps -p {PID} -o pid,cmd,lstart,etimes,etime

ps -e | less # Listing Process for All Users

ps -eH --forest | less # hierarchy

ps -e | grep firefox # Listing Processes by Name

ps -p 3403 # Listing Processes by Process ID

ps -u mary  # Listing Processes Owned by a User

sudo pkill top # Killing Processes by Name

sudo killall top # Killing Multiple Processes by Name

Kill a process and the pstree command

  • Killing a process and all of its descendants. This covers a PPID, PID and more importantly PGID, SID. Also ps j -A command can show these IDs for the running processes. 
    $ tail -f /var/log/syslog | grep "CRON" &
$ ps j
$ kill -SIGTERM -- -($Some_PGID)
  • pgrep & kill 
    # find the PID
pgrep ProgramName
# Kill the ProgramName process
kill -9 PID
  • killall. For example, if Firefox is acting up (as Firefox will do from time to time) simply type killall firefox and it should kill the application completely. In the rare circumstances that this doesn’t work you can always type xkill and then click on the window that won’t close; this will completely close a given window immediately. See this. To kill a privileges process, use for example sudo killall crond. 
    sudo killall -u USERNAME

How to Kill Zombie Processes on Linux

How to Kill Zombie Processes on Linux

Simulate/produce high cpu load

How can I produce high CPU load on a Linux server? 

# method 1:
sudo apt install stress
stress --cpu 3

# method 2:
for i in 1 2 3 ; do while : ; do : ; done & done
jobs   # list background jobs
for i in 1 2 3 4; do kill %$i; done  # kill "job" (not "PID") 1,2,3,4

ps, pgrep and pidof: How much resource is used by a process

Find the process ID first by ps -ef | grep APPLICATIONAME where "-e" is to show the running processes and "-f" is for a full listing. Then 

ps -p <pid> -o %cpu,%mem,cmd

For example, 

$ ps -ef | grep akregator
brb      15013  1942  1 10:41 ?        00:00:05 akregator --icon akregator -caption Akregator
brb      15186 24045  0 10:50 pts/11   00:00:00 grep --color=auto akregator
$ ps -p 15013 -o %cpu,%mem,cmd
%CPU %MEM CMD
 1.0  0.8 akregator --icon akregator -caption Akregator

pgrep 

08:49AM ~$ ps -ef | grep firefox
brb       7798  7778  0 08:49 pts/2    00:00:00 grep --color=auto firefox
brb      25486 24869  0 Sep10 ?        00:42:48 /usr/lib/firefox/firefox
brb      25612 25486  0 Sep10 ?        00:19:49 /usr/lib/firefox/firefox .....
08:49AM ~$ pgrep firefox
25486

pidof 

08:49AM ~$ pidof firefox
27951 25961 25612 25486
08:51AM ~$ pidof /usr/lib/firefox/firefox
27951 25961 25612 25486

$ kill $(pidof firefox)

All You Need To Know About Processes in Linux

http://www.tecmint.com/linux-process-management/

job: How do I send an already-running process into the background

Steps:

  1. 'Ctrl+Z' to stop (pause) the program and get back to the shell.
  2. bg to run it in the background.
  3. jobs -l to get the jobID and process ID
  4. disown -h [job-spec] where [job-spec] is the job number (like %1 for the first running job; find about your number with the jobs command) so that the job isn't killed when the terminal closes.

Stopped job

A stopped job is one that has been temporarily put into the background and is no longer running, but is still using resources (i.e. system memory). Because that job is not attached to the current terminal, it cannot produce output and is not receiving input from the user.

  • jobs -s showing stopped jobs
  • jobs -l showing the job PID

Send kill to a stopped job, it will do nothing but queue than bring it in in foreground, it will terminate. So don't repeatly sending a 'kill' command.

  • fg %1 move the stopped job ID #1 to the foreground (works)
  • kill %1 # kill job ID #1
  • kill 12345 # kill job PID 12345
  • kill -9 `jobs -ps` may not work
  • kill -9 $(jobs -p)

nice

How to Set Process Priorities With nice and renice on Linux

watch command

We can use the watch command to monitor a specific process such as the progress of the dd command.

Terminal 1 

watch ps -C dd --format pid,cmd,%cpu

Terminal 2 

dd if=/dev/zero of=/dev/null

Use --color for color output.

while + sleep

How To Repeat A Command Every X Seconds On Linux: watch, while + sleep,

ulimit

How to find ulimit for user on Linux

run commands in a background and allow log off

nohup /path/to/script >output 2>&1 &

Or to disable output and be more safe. It also explains the concept of file descriptor/fd in Unix. 

nohup command </dev/null >/dev/null 2>&1 &

See also Anonymous named pipe.

Notepadqq - Notepad++-like editor

Notepadqq. It is written using Qt. It does not have printing function:(

Note apps that can sync

Top 8 Notepad Apps for Linux That You Can Sync. Some are compatible with Evernote.

Evernote

Evernote alternative

How to Install Turtl Server - Evernote Alternative - on Ubuntu 16.04

Backup/restore Evernote

https://www.makeuseof.com/tag/backup-restore-evernote/

Markdown

Preview markdown/view markdown offline

sudo pip install grip
grip readme.md

# title 1
![screenshot](myfile.png?raw=true)
  • Chrome markdown preview plus extension does not show images from github.

Markdown editor

Text editor with navigation

Text editor with navigation

nano/pico editor

The nano editor is also called pico in R. See ?edit in R.

nano editor

vi editor

vi editor

Cloud

http://slidedecks.wilmoore.com/2012-confoo/diy-private-cloud-using-virtualBox-and-chef/#66

http://www.datacentermap.com/blog/cloud-software-389.html

Commercial cloud services

  • DigitalOcean - simple pricing. One droplet = one server.
  • Amazon EC2
  • Google cloud
  • Microsoft Azure

Manage all your cloud storage

http://www.makeuseof.com/tag/chrome-extensions-you-need-manage-cloud-storage/

Dropbox

Dbxfs – Mount Dropbox Folder Locally As Virtual File System In Linux. It is a command line dropbox client and requires no disk space for access.

Comparison of Linux VPS providers

https://www.ghacks.net/2017/06/15/a-comparison-of-linux-vps-providers-for-beginner-hosting/

Boot

U-boot

http://www.denx.de/wiki/U-Boot

Pandora linux client

COW (copy on write) file system

The cow filesystem was found on xubuntu live CD. See http://en.wikipedia.org/wiki/Copy-on-write

How To Protect Your Server Against the Dirty COW Linux Vulnerability (10/21/2016)

Apache redirection

http://cran.r-project.org/mirror-howto.html

Redirect a Website URL from One Server to Different Server in Apache

Important linux directories

Filesystem Hierarchy Standard, Wikipedia

  • /bin - executables used by the base system
  • /boot
  • /dev
  • /etc - configuration files
  • /media
  • /mnt
  • /opt - optional application packages
  • /proc - process information only
  • /sbin - critical executables for running the system, but should be used by superuser
  • /usr - non-critical files. For example /usr/bin contains most of the libraries used by apps. /usr/share/ contains Architecture-independent data (eg some pretty images are located in /usr/share/backgrounds)
  • /var - variable data such as databases, mails spools and system logs.

Difference of /bin, /sbin, /usr/local/bin, ...

  • /bin : For essential binaries; e.g. bash, cat, ls.
  • /sbin : is similar to /bin but for scripts with superuser (root) privileges required; e.g. shutdown command is located here. Local users have to use sudo to run binaries here.
  • /usr/bin : Same as first, but for general system-wide & non-essential binaries; e.g. grep, zip, docker, etc.
  • /usr/sbin : Same as above, but for scripts with superuser (root) privileges required.
  • /usr/local/bin or /usr/local/sbin for system-wide available (personal) scripts. For example, install docker-compose is merely to download the binary and place it under /usr/local/bin/ directory.

If you want to create your own scripts and make them available to all users, you’re pretty safe adding them to /usr/local/bin. Or to add my scripts to my local bin (~/bin) and then I create a symbolic link in /usr/local/bin to the commands I want to make public. As a result, I can manage all my scripts from the same directory but still make some of them publicly available since /usr/local/bin is added to $PATH. See this post.

DHCP

DHCP server IP

Linux find DHCP server IP address using CLI

Get a New IP Address

dhclient -r   #  release your IP Address

dhclient   #  get your DHCP to issue you a new IP Address based on how it’s been configured.

Open a file/URL using the default application from the command line

See also

Check a file's encoding

file -bi myfile

For example, 

file -bi Downloads/hmv_.rc 
# text/x-c++; charset=utf-16le

Know you system using the command line

Alerting and visualization tools

5 alerting and visualization tools for sysadmins

System monitor tools (TUI)

conky and autostart

For auto start on Lubuntu, see How can I add new autostart programs in Lubuntu?

On Lubuntu 18.04, add the path to the application to ~/.config/lxsession/Lubuntu/autostart

top and htop command

CPU frequency

What is the correct way to view your CPU speed on Linux? (x86) 

watch -n.1 "cat /proc/cpuinfo | grep \"^[c]pu MHz\""

How can I get the current CPU frequency of an ARM processor on Ubuntu? 

ls /sys/devices/system/cpu/cpu0/cpufreq/

glances command: more than htop

Glances is similar to htop but it provides network stats and disk usage too. It also supports web UI. Install it by sudo apt-get install glances.

scout_realtimep

This is used by Dataplicity

gtop command (100% Javascript)

https://www.cyberciti.biz/howto/gtop-awesome-system-monitoring-dashboard-for-terminal/.

Pros:

  • CPU history graph in time
  • Memory history graph in time (not useful)
  • Network bandwidth usage is real-time. It is accurate as what nload gives.
  • Percentage usage of memory, swap, disk usage
  • Top processes
$ docker run --rm -it \
    --name gtop \
    --net="host" \
    --pid="host" \
    aksakalli/gtop

$ sudo apt install npm nodejs
$ npm install gtop -g
$ gtop
  • Press p to sort by process ID (PID).
  • Press c to sort by CPU usage.
  • Press m to sort by memory usage.

It can be installed on Linux Mint 18.2 but not in Ubuntu 14.04 or raspbian (9 stretch). 

$ npm install gtop -g
npm http GET https://registry.npmjs.org/gtop
npm http GET https://registry.npmjs.org/gtop

npm http GET https://registry.npmjs.org/gtop
npm ERR! Error: CERT_UNTRUSTED
npm ERR!     at SecurePair.<anonymous> (tls.js:1370:32)
npm ERR!     at SecurePair.EventEmitter.emit (events.js:92:17)
npm ERR!     at SecurePair.maybeInitFinished (tls.js:982:10)
npm ERR!     at CleartextStream.read [as _read] (tls.js:469:13)
npm ERR!     at CleartextStream.Readable.read (_stream_readable.js:320:10)
npm ERR!     at EncryptedStream.write [as _write] (tls.js:366:25)
npm ERR!     at doWrite (_stream_writable.js:223:10)
npm ERR!     at writeOrBuffer (_stream_writable.js:213:5)
npm ERR!     at EncryptedStream.Writable.write (_stream_writable.js:180:11)
npm ERR!     at write (_stream_readable.js:583:24)
npm ERR! If you need help, you may report this log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <[email protected]>

npm ERR! System Linux 4.4.0-119-generic
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "gtop" "-g"
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.3.10

gotop

A terminal based graphical activity monitor inspired by gtop and vtop. It is quite beautiful.

Gotop – Yet Another TUI Graphical Activity Monitor, Written In Go

Compared to gtop, it has a temperature monitor. However, it can only show the average CPU usage (one line) on my Xeon computer. 

git clone --depth 1 https://github.com/cjbassi/gotop /tmp/gotop
/tmp/gotop/scripts/download.sh
sudo cp gotop /usr/local/bin; rm gotop
gotop

Note the temperatures do not show up in Raspbian (raspberry pi 3 b+).

termui: Golang terminal dashboard

https://github.com/gizak/termui

Bashtop

Bashtop Is A Cool Linux Resource Monitor Written In Bash

S-tui command

Monitor Linux CPU temperature, frequency, power in a graphical way

System monitor tools (GUI)

Comparisons:

Some lists:

Linux-Dash

https://github.com/afaqurk/linux-dash. Not working when I tested on RPi and Ubuntu.

Nagios

Zabbix

How to Install Zabbix on Debian 10

Munin and Monit

Server Monitoring with Munin and Monit on Ubuntu 16.04 LTS

Cacti

sysstat, sar

# CPU
sar 2 10 # every two seconds, 10 times
# Memory
sar -r  # look at the kbcommit and commit columns

sar -r -f /var/log/sysstat/sa02

Stacer

Linux System Optimizer and Monitoring

Curses and ncurses: TUI library

Calcurse

Keep up with your calendar and to-do list with Calcurse

Bitbucket (free for 5 users)

Bitbucket 101

  • Unlimited private repos
  • Code reviews
  • JIRA integration
  • REST API
  • Custom domains

See this post to know how to fix the problem of unknown author. In short, when I uncheck "Use global user setting" from Repository-> Repository Settings -> Advanced does the commit author change as expected.

See here for a list of Android apps related to bitbucket.

Image

See Images.

GIMP

Reload/Refresh .profile file

https://askubuntu.com/a/59127 

. ~/.profile

. is a bash builtin and a synonym for source, see man bash.

After changing the .profile file, you have to logout from your account and login, then it will be sourced once automatically.

History of commands

history command with date and time

Running the following code once and history will give date and time the next time you issue the history command. 

echo 'export HISTTIMEFORMAT="%y%m%d %T "' >> ~/.bashrc

Note that the original post asks to write the line to ~/.bash_profile but this is not working in the desktop environment.

Note that on zsh, the above method is not supported. We can use history -i instead. Type man zshoptions or man zshbuiltins for more information.

Bang bang - Run a command/Fetch parameters from previous history

  1. ^P: Move up through the command history list one command at a time.
  2. ^N: Move down through the command history list one command at a time.
  3. !!: Run the previous command. For example, we can run sudo !! in order to run the previous command with sudo.
  4. !n: Run command number n (useful)
  5. !string: Run most recent command starting with characters in string (useful). For example, !ls
  6. !?string: Run most recent command containing characters that match string
  7. !*: Fetch parameters from last command (useful). For example, if we run "ls /var" first. Then when we run stat !*, it would run stat /var. Or we can run cd !* and it will cd to /var directory.
  8. !_: Fetch the last parameter from last command. For example, if we run "ls /var/ /etc" first. Then when we run stat $_, it would run stat /etc. (zsh shell only)
  9. Ctrl + r and type a keyword (most useful). Press ctrl + r to scroll the match. This is called reverse i search.

For example, 

!-1
!4
!tail

Recall commands with reverse-i-search

Ctrl + r. See Bash bang commands: A must-know trick for the Linux command line

To continue with the search, just hit Ctrl + r keys again.

To run a forward search, hit Ctrl + s. How to cycle through reverse-i-search in BASH?

Increase history limit

http://unix.stackexchange.com/questions/17574/is-there-a-maximum-size-to-the-bash-history-file Unlimited Bash History 

$ echo $HISTSIZE  # $HISTSIZE variable controls how much history is displayed 
500
$ export HISTSIZE=1000
$ echo “HISTSIZE=1000” >> ~/.bashrc

# $HISTFILESIZE variable controls how many commands are retained in your .bash_history file.
$ echo $HISTFILESIZE
2000
$ wc -l .bash_history
2000 .bash_history

Not to add to bash history

Add a space after the command.

It is useful if there is a password in the command.

For example, 

$ ls ~/             # this won't be recorded in history
$  ls ~/Downloads/  # this will be recorded in history
$ history

Delete a single command from history

To stop adding history entries, you can place a space before the command, as long as you have ignorespace in your HISTCONTROL environment variable.

You can force Bash to exclude commands starting with empty space by placing this in your .bashrc file: 

export HISTCONTROL=$HISTCONTROL:ignorespace

How to Clear Bash History on Linux

$ cat /dev/null > ~/.bash_history && history -c && exit

multiple terminals

Use ‘’’history -a’’’ to write the history to the file if we do not plan to close the terminal. See How to use the history command on Linux.

Listen to HiChannel internet radio

Use Radio Tray

I use it to listen m3u file (VLC also supports it too).

Web Analytics Reporting Tools

Painting software

  • Pinta. It can be install by apt-get command. It works just line Window's paint. Ctr + v to paste an image and save to a file. To crop an image, click the selection tool on the most left hand side (it's a black color on v1.6 but a gray color on v1.7), then select a rectangle. Now click 'Image' > 'Crop to Selection' to finish. Pinta New Release After 5 Years. Here’s How to Get it! 
    sudo add-apt-repository ppa:pinta-maintainers/pinta-stable
sudo apt update
sudo apt install pinta
  • mtPaint. It is included in Odroid - xu4 - Lubuntu 14.04. To crop an image, just select an area and click Image > Crop. It can be used to take a screenshot from the desktop by using either the application or through the command line (mtpaint -s). It will then display the screenshot in the application if you use the command line.
  • MyPaint

Take a screenshot (and edit them)

See Take screenshots.

Cozy - audiobook player

ebook readers

7 Best eBook Readers for Linux: Calibre, FBReader, Okular, Lucidor, Bookworm, Easy Ebook Viewer and Buka.

Calibre - Read ebook in epub format

See Calibre

RSS reader

Some references:

  1. 5 Best Feed Reader Apps for Linux
  2. 14 Best RSS Feed Readers for Linux in 2018. It contains nice screenshots.

Some examples:

  • Akregator. KDE based. This is preinstalled in CentOS-KDE under the Internet category. It is also called 'Feed Reader'.
  • QuiteRSS. It works on Linux, Windows and MacOS.
  • Liferea. GTK based. It is considered one of the best RSS feed readers on Ubuntu Linux. It can synchronize with several online feed managers such as InoReader among others.
  • FeedReader. Looks nice. Works with several online feed managers.
  • Newsbeuter: RSS feed in terminal
  • Newsboat: terminal
  • RSSOwl. Depends on Java. Cross platform.
  • Firefox and Thunderbird have built-in support for RSS.

Clear gibberish all over the screen

Just type “reset”. See BASH Fix Display and Console Garbage and Gibberish on a Linux / Unix / macOS. It is useful, for example, accidentally I run cat command over binary file.

Display/screen

Turn off/on your monitor via command line

xset dpms force off # Press any key to turn it on 
xset dpms force on
xset -q # check the status of the X server settings

If we want to turn off/on the screen via ssh, add 

export DISPLAY=:0.0

first before calling the xset command, or use '-display' argument 

xset -display :0.0 dpms force off 
xset -display :0.0 dpms force on

autoxrandr

Plug your laptop into different monitor setups. https://www.donarmstrong.com/posts/autorandr/

Move a window without clicking the titlebar

Hold down the Alt key and then click in the window anywhere, and move your mouse.

Add new screen/display resolutions

xrandr | grep maximum
gtf 800 480 59.9  # give some output used in the following line
xrandr --newmode "800x480_59.90" 29.53 800 816 896 992 480 481 484 497 -HSync +Vsync
xrandr --addmode "DISP3 BG" 800x480_59.90
xrandr --output "DISP3 BG" --mode 800x480_59.90

I cannot find the file /etc/X11/xorg.conf in my UDOObuntu 2 beta 2. It seems this file does not exist anymore. See this post about how to re-create it.

Wayland

loginctl show-session $(awk '/tty/ {print $1}' <(loginctl)) -p Type | awk -F= '{print $2}'

export DISPLAY

If we want to run a GUI app on a remote computer (such as Raspberry Pi/Beaglebone Black) and show the GUI app on the remote computer's screen using ssh, we can issue the following command before running the app. 

export DISPLAY=:0.0

LVM Demystified

http://www.linuxjournal.com/content/lvm-demystified

See which groups you belong to, id & group commands

id <username>
groups 
groups <username>

finger: show user information

finger USERNAME

groupadd, chgrp, usermod, ACL (access control lists)

sudo mkdir -p /var/www/reports/
sudo groupadd project 

sudo usermod -a -G project tecmint 
sudo chgrp -R project /var/www/reports/
sudo chmod -R 2775 /var/www/reports/

create more system users and add them to the directory group as follows: 

sudo useradd -m -c "Aaron" -s/bin/bash -G project aaron
sudo useradd -m -c "John" -s/bin/bash -G project john
sudo useradd -m -c "Ravi" -s/bin/bash -G project ravi

sudo mkdir -p /var/www/reports/aaron_reports
sudo mkdir -p /var/www/reports/john_reports
sudo mkdir -p /var/www/reports/ravi_reports

Add a standard user to sudo group

This is useful on Debian distribution where a new user does not have the sudo power. First log in as root, 

# usermod -aG sudo username

Shared library management

http://www.cyberciti.biz/tips/linux-shared-library-management.html

  1. ldconfig : Updates the necessary links for the run time link bindings.
  2. ldd : Tells what libraries a given program needs to run.
  3. ltrace : A library call tracer.
  4. ld.so/ld-linux.so: Dynamic linker/loader.

Log files

DigitalOcean 

$ ls -lt /var/log

ssh log files: /var/log/auth.log

  • /var/log/syslog
  • /var/log/auth.log: it includes ssh log in information and lots of CRON sessions opened and closed every minutes.

And

Apache log

  • /var/log/apache2/error.log (small 83K). Useful to troubleshoot errors/crashes of Apache.
grep "May 08" /var/log/apache2/error.log
  • /var/log/apache2/access.log (large 10M)

mail

/var/log/maillog

uptime command

uptime
watch -n 60 uptime

Windows

Find out from the logs what caused system shutdown?

How to find out from the logs what caused system shutdown?

Get notified when a system is rebooted

Get Notified When Your Raspberry Pi is Booted with Pushbullet. This makes use of /etc/rc.local file. If it does not work, we can use cron to run a command at startup; see sending an email on boot.

timeout command

Linux command similar to top to show hard disk activity

Use iotop. On ubuntu, we can use sudo apt-get install to install it. Use sudo iotop to launch it. 

sudo apt-get install iotop

sudo iotop -o -u $USER

Another program is iostat and the -d (disk) option. The -x option will display extension I/O status. 

sudo apt-get install sysstat
iostat -dx 5 # every 5 seconds

24 iostat, vmstat and mpstat Examples for Linux Performance Monitoring

curl and wget

See Curl.

Torrent

Torrench

Torrench: How To Search And Download Torrent Files Using Terminal (Linux, Mac, Windows)

aria2 - command line downloader supports torrents and multi-connection

aria2 command examples

The -x argument helps a little bit. Download a file 112MB; see https://www.archlinux.org/download/ 

$ time aria2c  http://mirror.jmu.edu/pub/archlinux/iso/2016.11.01/archlinux-bootstrap-2016.11.01-i686.tar.gz # 16 seconds
$ time aria2c -x10 http://mirror.jmu.edu/pub/archlinux/iso/2016.11.01/archlinux-bootstrap-2016.11.01-i686.tar.gz # 11 seconds

Axel

It can create an unlimited number of worker threads to download any kind of data. See https://www.beginnersheap.com/top-5-command-line-download-accelerators-linux/

lftp

Apply a patch to source code

For example Tophat 2.0.12 compatibility with Samtools 1.0, 

$ ls
support_for_tophat_1.patch  tophat-2.0.12  tophat-2.0.12.tar.gz

$ grep -r -i "check_samtools" tophat-2.0.12/
tophat-2.0.12/src/tophat.py:def check_samtools():
tophat-2.0.12/src/tophat.py:        check_samtools()

$ cp support_for_tophat_1.patch tophat-2.0.12/src/
$ cd tophat-2.0.12/src/
$ patch tophat.py < support_for_tophat_1.patch 
patching file tophat.py
Hunk #1 succeeded at 1540 (offset 3 lines).
Hunk #2 succeeded at 1563 (offset 3 lines).

IP address fundamental

http://www.howtogeek.com/133943/geek-school-learning-windows-7-ip-addressing-fundamentals/.

There are three classes for private IP ranges.

  • 1-126.0.0.0 from Class A
  • 128-191.0.0.0 from Class B
  • 192-223.0.0.0 from Class C

Subnet

Get internal IP address

$ hostname -I

Get external IP address

Using a browser: Google.com and type "my ip"

How to find your IP address in Linux

Command for determining my public IP? wget -qO- https://ipecho.net/plain ; echo

5 Commands to Find the IP Address of a Domain in the Linux Terminal

https://www.iplocation.net or https://www.ipify.org/

https://github.com/jakewmeyer/Geo (one shell script)

It seems there is no way to get the external IP address without not using external services. 

# https://unix.stackexchange.com/a/81699
sudo apt-get install dnsutils
dig +short myip.opendns.com @resolver1.opendns.com
# OR
curl http://ipecho.net/plain; echo
# OR
curl ipv4.ipogre.com

To store my IP in a shell variable 

myip="$(dig +short myip.opendns.com @resolver1.opendns.com)"
echo "My WAN/Public IP address: ${myip}"

The above only gives the IP. The following method gives geo information too. 

curl ipinfo.io # ifconfig.me, icanhazip.com, ipecho.net/plain, ifconfig.co
# OR give a specific IP (domain name does not work)
curl ipinfo.io/216.58.194.46

{
  "ip": "216.58.194.46",
  "hostname": "dfw25s12-in-f14.1e100.net",
  "city": "Mountain View",
  "region": "California",
  "country": "US",
  "loc": "37.4192,-122.0574",
  "org": "AS15169 Google Inc.",
  "postal": "94043"
}

IP geolocation

Test url: ubuntu.mirrors.pair.com

Domain

  • WHOIS LOOKUP. For example, consider "r-pkg.org" domain,
    • The 'Sponsoring Registrar' shows who is the sponsoring registrar (eg GoDaddy.com).
    • The 'Registrant Name' shows who registered this domain.
    • Command Line Interface.
sudo apt-get install whois
whois r-pkg.org
  • IP-Lookup. For example, consider "r-pkg.org" domain.
    • The linux command line tool 'ping' shows the ip address.
    • When we use the ip address to search in the IP-Lookup, the IP owner info > 'Organization' or 'OrgName' field shows the owner of this IP (eg Digital Ocean, Inc).
    • The Domain owner info there gives the same (or less) information as WHOIS LOOKUP.
  • WhoIsHostingThis or webhostinghero (the returned result will be like Amazon, GoDaddy, CloudFlare, Github, Verizon, etc).

Subnet

IPv4

  • Class A: 255.0.0.0 or /8;
  • Class B, 255.255.0.0 or /16;
  • Class C, 255.255.255.0 or /24.

For example, in the subnet 192.168.5.0/255.255.255.0 (192.168.5.0/24) the identifier 192.168.5.0 commonly is used to refer to the entire subnet.

In the /16 subnet 192.168.0.0/255.255.0.0, which is equivalent to the address range 192.168.0.0–192.168.255.255

IPv6

Private use

https://en.wikipedia.org/wiki/IPv4#Private_networks

  • 10.0.0.0 – 10.255.255.255
  • 172.16.0.0 – 172.31.255.255
  • 192.168.0.0 – 192.168.255.255

ping command

ping uses the ICMP Echo Message to force a remote host to echo a packet back to the local host. If packets can travel to and from a remote host, it indicates that the two hosts can successfully communicate.

Check if a server is up/down if pining to that machine is not allowed

Hint: use nmap to scan open ports 

nmap -PS22 -p22 <hostname> # port 22 only
# OR
nc -z -v <hostname> 22

Shell script to check whether a server is reachable?. I found if a server is up and I am trying a closed port, it will take about 2 minutes before it returns. 

if nc -z $server 22 2>/dev/null; then
    echo "$server ✓"
else
    echo "$server ✗"
fi

6 Methods to Quickly Check if a Website is up or down from the Linux Terminal

Check if Sites are Online using a PHP script

How to Use Crontab to Automate Repetitive Tasks in Linux

Build a home network

IP Subnet Calculator

https://www.dan.me.uk/ipsubnets?ip=10.0.0.0

  • CIDR block IP range (network - broadcast) Subnet Mask IP Quantity
  • 10.0.0.0/24 10.0.0.0 - 10.0.0.255 255.255.255.0 256
  • 10.0.0.0/16 10.0.0.0 - 10.0.255.255 255.255.0.0 65536=256^2
  • 10.0.0.0/8 10.0.0.0 - 10.255.255.255 255.0.0.0 16777216=256^3

How Use Your Router and ISP’s Modem/Router Combo In Tandem

http://www.howtogeek.com/255206/how-use-your-router-and-isps-modemrouter-combo-in-tandem/

Troubleshoot and repair network problems

http://www.linuxuser.co.uk/features/troubleshot-repair-linux-networks

Computer 1 can ping Computer 2 but not reverse

Use the route command to solve. See also this post.

Simple Network Management Protocol (snmp)

Monitor network by Cacti (GUI)

Monitor network by command line

3 Simple, Excellent Linux Network Monitors: iftop, nethogs and vnstat.

bandwidth

bandwhich Shows What`s Taking Up Your Network Bandwidth On Linux And macOS

iftop

Use the interface top iftop command. On ubuntu, we need to use sudo apt-get install iftop and then run it by sudo iftop -i eth0. After that, we can press some keys to toggle options.

  • p: port
  • s: source
  • d: destination

See thegeekstuff.

It is strange that the output shows other devices names in my network. 

$ dig A pandora.com
$ ipcalc -b 208.85.40.20
$ sudo iftop -F 208.85.40.20/24 -i wlan0

nethogs

$ sudo nethogs wlan0

Why does the snapd service use so much data?

nload

nload -m

nload – Monitor Linux Network Bandwidth Usage in Real Time

The result is the same as gtop (gtop is cooler) gives.

bmon

https://www.tecmint.com/bmon-network-bandwidth-monitoring-debugging-linux/

vnstat for network traffic monitor

# 1. Install vnStat
sudo apt-get install vnstat

# 2. Pick a Interface to Monitor using vnStat
vnstat -u -i eth0
vnstat --iflist
vnstatd -d # start the daemon
ps -ef | grep vnst

# 3. vnStat Basic Usage
vnstat

# 4. vnStat hours, days, months, weeks Network Data
vnstat -d
vnstat -m

# 5. Export the data to Excel or other DB
vnstat --dumpdb

# 6. Display Live Network Statistics
vnstat -l

# 7. Change the default vnstat output format
vnstat -s (--short)
vnstat --style 0

# 8. Display Top 10 Traffic Days
vnstat --top10

Network related linux commands

nmcli and nmtui (useful for getting DNS IPs)

# Find the up network adaptors
$ nmcli dev status
DEVICE           TYPE      STATE        CONNECTION         
docker0          bridge    connected    docker0            
eno1             ethernet  connected    Wired connection 1 

# Find the DNS 
$ nmcli dev show | grep 'IP4.DNS'

iwconfig

8 Linux Commands: To Find Out Wireless Network Speed, Signal Strength And Other Information

Getting information (Link Quality, Signal level) about the wireless. 

sudo iwconfig

See link quality continuously on screen 

watch -n 1 cat /proc/net/wireless

(GUI) NetworkManager

ps -ef | grep NetworkManager

ifconfig - spoof the hardware address at the software level

To change the MAC address temporarily on a NIC (network interface controller), 

sudo ifconfig eth0 down
sudo ifconfig eth0 hw ether 00:11:22:33:44:55
sudo ifconfig eth0 up

And it seems there is no need to modify /etc/network/interfaces.

For wlan 

sudo ifconfig wlan0 down
sudo ifconfig wlan0 hw ether 00:11:22:33:44:55
sudo ifconfig wlan0 up

See

ip command

It is said ip is replacing the old ifconfig command on modern Linux distributions.

Linux ip Command Examples

http://www.makeuseof.com/tag/networking-commands-linux-terminal/ 

ip a
ip addr
ip address show

ip link set DEVICE down # eg ip link set eth0 down

ip link set DEVICE up

iptables

route

7 Linux Route Command Examples

  1. Display Existing Routes (route -n)
  2. Adding a Default Gateway (route add default gw 192.168.1.1)
  3. List Kernel’s Routing Cache Information (route -Cn)
  4. Reject Routing to a Particular Host or Network (route add -host 192.168.1.51 reject)
  5. Make 192.168.3.* Accessible from 192.168.1.* (route add -net 192.168.3.0 netmask 255.255.255.0 gw 192.168.3.10)
  6. Make 192.168.1.* Accessible from 192.168.3.* (route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.1.10)
  7. Allow Internet Access/External World (route add default gw 125.250.60.59)

On Ubuntu 16.04, it shows 

$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         FIOS_Quantum_Ga 0.0.0.0         UG    600    0        0 wlp3s0
link-local      *               255.255.0.0     U     1000   0        0 wlp3s0
192.168.1.0     *               255.255.255.0   U     600    0        0 wlp3s0
$ route -n   # showing numerical IP address instead of host name.
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    600    0        0 wlp3s0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 wlp3s0
192.168.1.0     0.0.0.0         255.255.255.0   U     600    0        0 wlp3s0

Flag value 'U' means up and 'G' means gateway'.

Connect two networks

The trick is explained in this post or the above route command.

For example, my network structure is

  • Modem/router: LAN IP 192.168.1.*/24
  • PC1: connect to Modem/router
  • Second router (ASUS) connect to Modem/router: its WAN IP is 192.168.1.ASUS. It's LAN IP 192.168.2.*/24
  • PC2 (raspberry pi): connect to the second router (ASUS): its IP is 192.168.1.212

By default, PC2 can ssh to PC1 but PC1 cannot access PC2.

The following command will solve the problem that PC1 cannot access PC2: 

# From PC1
$ sudo route add -net 192.168.2.0 netmask 255.255.255.0 gw 192.168.1.ASUS
$ ssh [email protected]    # 192.168.2.212 is the IP address for the Raspberry Pi

$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.2.0     192.168.1.ASUS   255.255.255.0   UG        0 0          0 eth0

One article from linux.com using the ip command.

traceroute

sudo apt-get install traceroute
traceroute 8.8.8.8

On Windows, we can use the tracert command. For example, tracert www.microsoft.com.

nslookup (convert between hostname and ip) and host

$ host google.com
google.com has address 172.217.5.238
google.com has IPv6 address 2607:f8b0:4004:802::200e
google.com mail is handled by 30 alt2.aspmx.l.google.com.
google.com mail is handled by 10 aspmx.l.google.com.
google.com mail is handled by 40 alt3.aspmx.l.google.com.
google.com mail is handled by 50 alt4.aspmx.l.google.com.
google.com mail is handled by 20 alt1.aspmx.l.google.com.

$ nslookup google.com
Server:         127.0.1.1
Address:        127.0.1.1#53

Non-authoritative answer:
Name:   google.com
Address: 172.217.7.238

mac$ nslookup cran.r-project.org
Server:		156.40.70.10
Address:	156.40.70.10#53

Non-authoritative answer:
cran.r-project.org	canonical name = cran.wu-wien.ac.at.
Name:	cran.wu-wien.ac.at
Address: 137.208.57.37

mac$ nslookup 137.208.57.37
Server:		156.40.70.10
Address:	156.40.70.10#53

Non-authoritative answer:
37.57.208.137.in-addr.arpa	name = cran.wu-wien.ac.at.

The first two lines show the IP address of my DNS. If we run the command inside a Docker container where DNS was specified manually, the IP address we specified will be shown here.

dig

Dig provided by Google G Suite Toolbox. If the office internet is interrupted, we can use a cell phone to search for the IP address of a website for trouble shooting. Other providers include kloth.net. 

$ dig world.std.com
; <<>> DiG 9.9.5-3ubuntu0.16-Ubuntu <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49227
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             130     IN      A       172.217.5.238

;; Query time: 11 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Fri Dec 01 17:32:37 EST 2017
;; MSG SIZE  rcvd: 55

arp (Address Resolution Protocol)

The arp command can be used to show the MAC addresss of all hosts in LAN 

arp -a

Find open ports

command example comment
lsof sudo lsof -i -P -n | egrep "PID|LISTEN" contain pid. Using egrep to include header in output. It includes the commands.
nmap (network mapper) nmap localhost (OR IP) shortest output and input. Need to install. No 'sudo' needed. Works on non local.
netstat netstat -tlpn | grep LISTEN | awk '{print $4 "\t" $7}' show pid and program name. Need to install netstat. Windows already has it.
ss ss -tulpn | grep LISTEN long output, contain pid, no program name
nc nc -zvw3 <hostname> <port> need to specify a port number. More accurate than nmap. "-w3" means time out after 3 seconds.

Socket

What Are Unix Sockets and How Do They Work

nmap/network mapper - port scanning & IPs in local network

nmap - Network exploration tool and security / port scanner

  • nmap does not show all open ports By default, Nmap scans the most common 1,000 ports for each protocol.
  • https://nmap.org/book/nmap-os-db.html. Local OS database is located at /usr/share/nmap/nmap-os-db. The 2nd line will show the revision number.
    • Modifying the nmap-os-db Database Yourself
    • Download the latest from https://svn.nmap.org/nmap/nmap-os-db. Note that the current revision number has to be found from the website. You can edit the file and insert the revision number on the 2nd line of your local copy.
    • Even I update the database, it cannot detect my Ubuntu 14.04 OS (it only shows OS details: Linux 3.8 - 4.9). For the Raspberry Pi, it can show information from the network adapter; e.g. MAC Address: AA:BB:CC:DD:EE:FF (Raspberry Pi Foundation) but not the OS name (OS details: Linux 3.2 - 4.8). 
      sudo mv /usr/share/nmap/nmap-os-db /usr/share/nmap/nmap-os-db-old

cd /usr/share/nmap
sudo wget https://svn.nmap.org/nmap/nmap-os-db
  • http://www.cyberciti.biz/networking/nmap-command-examples-tutorials/
  • http://bencane.com/2013/02/25/10-nmap-commands-every-sysadmin-should-know/
  • http://www.tecmint.com/nmap-command-examples/ 
    sudo apt-get install nmap

nmap 192.168.1.100   # does not require root privileges
                     # used to check open ports

nmap 192.168.1.*     # show IPs and ports in LAN

sudo nmap -sP 192.168.1.1/24 # show connected IPs (no hostnames?) and MAC addresses
                             # If you don't use 'sudo' only partial devices can be found
                             # The output may contains the hostname. For example,
                             # Nmap scan report for brb-P45T-A.fios-router.home (192.168.1.xxx)
nmap -sV 192.168.1.1 # show Daemon name (in VERSION column) together with port number

nmap -T4 -F 192.168.1.99-255 # show connected IPs and open ports
                             # -F means fast
nmap -F taichimd.us  # Note that domain name != server
nmap -v taichimd.us

nmap -A 192.168.1.1  # Aggressive scan (more output)

nmap -p http,ssh,mysql taichimd.us  # scan ports/services
                                    # note that mysql will be shown as closed
                                    # ssh port is not correct. 
                                    # Not sure how to get the correct ssh port using nmap
nmap --open taichimd.us   # scan open ports

sudo nmap -traceroute nih.gov

sudo nmap -sS -O 192.168.1.99 # -O shows operating system
                              # eth0 MAC

$ nmap localhost # showing the true ports from the server

Starting Nmap 7.01 ( https://nmap.org ) at 2017-10-09 15:01 EDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00016s latency).
Not shown: 996 closed ports
PORT    STATE SERVICE
22/tcp  open  ssh
25/tcp  open  smtp
80/tcp  open  http
631/tcp open  ipp

$ nmap localhost -p 3838-4000  # Shiny
  • A gui version of nmap is called Zenmap. How to install Zenmap Nmap GUI on Ubuntu 20.04 LTS

netstat: get a list of all open ports

How to use netstat in GNU/Linux

  • -l or --listening shows only the sockets currently listening for incoming connection.
  • -a or --all shows all sockets currently in use.
  • -e --show extended/additional information
  • -t or --tcp shows the tcp sockets.
  • -u or --udp shows the udp sockets.
  • -n or --numeric shows the hosts and ports as numbers, instead of resolving in dns and looking in /etc/services.
  • -s --Print network stats
  • -r --Print the network routing information
  • -p --Print PID and name of the program to which each socket belongs
netstat -l             # only listening ports
netstat -rn            # displays the system's routing table
netstat -at
netstat -ant           # For tcp
sudo netstat -pant     # show ports and programs (pant = 喘氣). Best of the best!!!
sudo netstat -peanut   # (output is too wide)
netstat -anp | grep 3306 | wc -l  # print # of connections for the port mysql port

Other commands 

sudo lsof -i -P -n | grep LISTEN
sudo ss -tulpn
sudo netstat -tulpn | grep LISTEN
sudo netstat -tulpn   # include dhcpcd & avahi-daemon

ss command

ss is one of 4 commands to check open ports for Linux. The 4 commands are netstat, nmap, ss and lsof.

How to Use the ss Command on Linux

lsof command

lsof -i   # this gives a more list than nmap command

Common ports

Copy text to a clipboard to be used in other apps

Install the xclip program. See here or here. 

sudo apt-get install xclip
# Examples
sort -n -k 3, -k 2 file.txt | xclip -selection clipboard

cat ~/.ssh/id_rsa.pub | xclip -sel clip

Works.

Start Emacs without X

Add -nw (no window) option. 

emacs -nw

Audio

Record audio from mic

How to Save Important Voicemails for Both iPhone and Android. Hint: use Audacity.

mp3 codecs

https://help.ubuntu.com/community/RestrictedFormats 

sudo apt-get install ubuntu-restricted-extras

Concatenate mp3 files

sudo apt-get install mp3wrap
mp3wrap output.mp3 *.mp3

Reduce the size of an mp3 file

Specify a new lower bitrate using the -b option in lame. For example if your starting mp3 has a quality of 256kbs you can lower its bitrate to 128kbps (or even lower like 64kbps) by: 

lame --mp3input -b 128 input.mp3 output.mp3

Convert ogg to mp3

ffmpeg is not included in Ubuntu repository. Use the avconv command. http://superuser.com/questions/15327/how-to-convert-ogg-to-mp3 

sudo apt-get install libav-tools
avconv -i input.ogg -c:a libmp3lame -q:a 2 output.mp3

Convert m4a/webm to mp3

avconv -i input.m4a output.mp3

Remove the vocals from any song using Audacity

https://www.makeuseof.com/tag/remove-vocals-song-audacity/

How to Remove Ambient Noise From Audio Files Using Audacity

How to Remove Ambient Noise From Audio Files Using Audacity

Normalize the volume of an audio file

  • Can You Losslessly Increase the Volume of MP3 Files?
  • Use Audacity. To raise (Amplify) volume:
    1. Edit > Select All.
    2. Effect > Amplify. Increase db. Adjust the sound until the highest peaks and lowest valleys of the waveform reach the top and bottom of the window.
    3. Check clip3. Export > MP3 or just start to listen.
  • Command line tool: avconv (replace ffmpeg program). See this post.
avconv -ss 00:00:10 -i OLD.mp3 -vol 2560 NEW.mp3

The anconv/ffmpeg -vol parameter amplifies the sound. The default value is 256 (no amplification), and you can adjust the number accordingly. Here it’s 2560, as it’s 10 times louder. Note that these are not decibel values or anything that sophisticated, but just an integer value. 512 equals to twice the volume, 768 three times, 1024 four times, etc. The -ss parameter specifies the start time offset. Here it will skip the first 10 seconds.

I found the converted file by sox has about one half file size compared to anconv/ffmpeg program (source file=47MB, anconv converted=135MB, sox converted file=54MB). 

sudo apt-get install sox libsox-fmt-all
sox --norm OLD.mp3 NEW.mp3 trim 10
sox --norm OLD.mp3 NEW.mp3 silence 1 0.1 1%
sox -v 4.0 OLD.mp3 NEW.mp3             # increase volume

where '--norm' will normalize the audio and the 'trim' option set to skip the first 10 seconds. The silence parameter allows to trim silence at the beginning without a need to specify the number of seconds.

cut, delete or trim an audio

  1. Open the audio file in audacity.
  2. select a region in the waveform area. Do not select in the time interval area (above the waveform).
    1. To precisely select a range from one position to the end. Click Zoom in several times. Click one position in the waveform and click Edit -> Select -> Cursor to the track end to select
    2. Similarly, if we want to precisely select a range from the start to some position, we can click one position in the waveform and then click Edit -> Select -> Track start to cursor.
    3. To move around the track, use the scrollbar (below the waveform and above the bottom toolbar, not quite clear in Ubuntu/Unity)
  3. Click Edit -> Remove Audio or labels -> Cut/Delete/Trim Audio
  4. play the new audio by clicking the green triangle.
  5. File -> Export -> mp3 format.

Helpful resource for Audacity.

  1. Different Toolbars
  2. Tutorial - Editing an Existing Audio File

Fade out at the end of an audio

  1. Select a region.
  2. Effect -> Fade out

Wireshark

sudo apt-get install wireshark
sudo chmod 4711 `which dumpcap`

Track the Time a Command Takes

How To Find The Execution Time Of A Command Or Process In Linux

time command

time COMMAND
time (COMMAND1; COMMAND2)
time (COMMAND1 && COMMAND2)

help time

When I run a set of 7 jobs using parallel, time command gives an output 

real  15m53.788s # the wall clock time the command took from execution till termination
user  95m20.238s # the time taken by the user space
sys   9m1.320s   # the time taken by kernel space

Here we see the real time is about 16m and the user time is about 6-7 times the real time. Indicating the parallel executing works.

/usr/bin/time command

/usr/bin/time provides more information then time command. 

man time

Magazines

Latex

Editors

Online editing

  • Latex Base. You can start to try it without registration. Free accounts cannot publish but still can download.
  • Overleaf. Free account for 1GB space.
  • ShareLatex

Missing cls

$ apt-cache search IEEEtran
texlive-publishers - TeX Live: Publisher styles, theses, etc.

sudo apt-get install texlive-publishers

Missing sty

$ apt-cache search pseudocode
gpt - G-Portugol is a portuguese structured programming language
libgportugol-dev - Development files for the G-Portugol library
libgportugol0 - G-Portugol library
texlive-science - TeX Live: Natural and computer sciences
$ sudo apt-get install texlive-science

PDF

See PDF.

Flow chart

  • LibreOffice Draw OR MS_PowerPoint (insert > shape). Check youtube.
  • yEd
  • Dia & wikipedia
  • (online) www.draw.io

Clock

xclock (analog)

oclock -geometry 500x500+100+0 &

oclock (analog)

oclock -bg blue -geometry 500x500+100+0 -bd purple -transparent &
oclock -bg blue -geometry 500x500+100+0 -bd purple -jewel green &

See oclock, X - a portable, network-transparent window system which includes an example of specifying the geometry parameter.

dclock (digital)

Digital clock for the X Window System with flexible display. 

sudo apt-get install dclock
dclock -h
dclock -d
dclock -date "Today is %A %B %Y" -led_off black -bg black -fg yellow -geometry 577x194+119+139         # 'q' to quit
dclock -date "Today is %m/%d/%y" -led_off black -bg black -fg yellow -geometry 400x150+0+0 # width x hight + X + Y

In practice, I create a shell script file <bin/clock> with the following content. The first ampersand sign is to hide warnings messages and the 2nd ampersand sign is to put the process in the background. 

dclock -date "Today is %A %B %Y" -led_off black -bg black -fg yellow -geometry 577x194+119+139 &>/dev/null &

Lubuntu digital clock format

http://netgator.blogspot.com/2012/09/change-edit-panel-digital-clock-format.html. My format is 

%a, %x, %r
# Tue, 05/17/2016, 09:42:27 PM

%a %m-%d-%y, %I:%M %p
# Mon 05-30-16, 08:31 AM

xdaliclock (digital)

https://www.mankier.com/1/xdaliclock

Scaling is not good. Colors is changing with time. 

xdaliclock -noseconds -cycle  # 'q' to quit

date command

sudo apt install toilet
sudo apt install figlet
watch -n 60 "date +'%m/%d/%y   %H:%M:%S' | toilet"

ls /usr/share/figlet # list of fonts
                     # looking for *.tlf
date +'%m/%d/%y%H:%M:%S' | toilet -f mono12
date +'%m/%d/%y%H:%M:%S' | toilet -f bigmono12   # good on 1024x600
date +'%m/%d/%y%H:%M:%S' | toilet -f bigascii12
date +'%m/%d/%y%H:%M:%S' | toilet -f ascii12

watch --color "date +'%m/%d/%y%H:%M:%S' | toilet -f bigmono12 -F metal"

while true; do echo "$(date '+%H:%M:%S' | toilet -f bigmono12 -F border --metal)"; sleep 1; done

Reminder take a break, relieve eye strain

Stretchly. It's open-source and cross-platform. Nodejs is required.

Workrave is another choice. The source code is available too.

Prevent Eye Strain While Working On Your Linux Desktop With Safe Eyes, Github source code

wine and winetricks

Running Linux in the AWS/Amazon Web Services

Forum software

RAID

Timer

How to track you laptop using Prey

https://www.howtoforge.com/tutorial/how-to-track-your-linux-laptop/

last command

Linux last Command Tutorial for Beginners (8 Examples)

Display a list of system shutdown/reboot date/time

Linux Find Out Last System Reboot Time and Date Command 

# Works on Linux and Mac
last shutdown
last reboot

Automatic reboot after power failure

It seems there is no reliable way to find out when the power failed.

The linux command 'last' can show some information about system reboot.

Another way is to modify the BIOS to select the option like 'Power off and Reboot'. This won't automatically boot your computer when it is shutdown normally.

How to restart/shutdown server safely

How to restart CentOS or RHEL server safely

Wake up and Shut Down Linux Automatically

Two best options

  • Bios: BIOS may have an easy-to-use wakeup scheduler
  • wakeonlan:
    • Eanble it: Check if it is enabled by default. If not, we can 1) enable it through a command (ethtool -s eth0 wol g) or 2) using the Network Manager
    • Send a wake up command: (from a second linux) /usr/bin/wakeonlan D0:50:99:82:E7:2B where D0:50:99:82:E7:2B is the IP on the machine you want to wake it up

How to update Lenovo BIOS from Linux without using Windows

https://www.cyberciti.biz/faq/update-lenovo-bios-from-linux-usb-stick-pen/

Internet speed test

Web

Speedtest-cli

See Track_Internet_Dropouts. 

sudo apt-get intall python-pip
sudo pip install speedtest-cli
# A slightly modified code that will create a one-line space/semi-colon 
# delimited result is 
git clone https://github.com/HenrikBengtsson/speedtest-cli-extras.git

speedtest-cli-extras/bin/speedtest-csv

works. But if I want to put it in cron, cron will issue an error speedtest-cli cannot be found. So I need to modify line 52 of the code in <speedtest-cli-extras/bin/speedtest-csv> to explicitly specify the location of speedtest-cli. 

    /usr/local/bin/speedtest-cli --share > $log

NOTE: 1. the results differ from the network connection. For example, the speed is good when I test it on the machine directly connected to the router. 2. It is helpful to modify the last line of the bash script to output what I need. 3. The separator is ";" in the output.

curl and wget

How to Run Speed Test from the Command Line to Check Internet Connection Speed

uname - Print system information

uname -a will give you

  • OS (uname = uname -s if you are under a Linux environment)
  • OS (uname -s) eg Linux
  • node name (uname -n=hostname)
  • kernel release (uname -r) eg 3.16.0-38-generic
  • kernel version (uname -v)
  • machine architecture (uname -m) eg x86_64
  • processor (uname -p)
  • hardware platform (uname -i)
  • operating system (uname -o)

How to check if running in Cygwin, Mac or Linux?

Hardware information

How To Check System Hardware Manufacturer, Model And Serial Number In Linux

Command Line

Linux commands to display your hardware information

python--hwinfo

https://www.2daygeek.com/python-hwinfo-check-display-system-hardware-configuration-information-linux/

dmesg command

How to Use the dmesg Command on Linux 

sudo dmesg -L -T
sudo dmesg -L -T --follow   # Watching Live Events
sudo dmesg -L -T | grep -i usb    # Search for a specific term
sudo dmesg | grep -E "memory|tty|dma"  # Search for multiple terms

where -L to is force color output and -T is to make timestamp human-readable.

Linux Logo and the current system information

odroid@odroid:~$ sudo apt-get install screenfetch
odroid@odroid:~$ screenfetch
                          ./+o+-       odroid@odroid
                  yyyyy- -yyyyyy+      OS: Ubuntu 15.10 wily
               ://+//////-yyyyyyo      Kernel: armv7l Linux 3.10.96-77
           .++ .:/++++++/-.+sss/`      Uptime: 4d 23h 8m
         .:++o:  /++++++++/:--:/-      Packages: 2000
        o:+o+:++.`..```.-/oo+++++/     Shell: 2263
       .:+o:+o/.          `+sssoo+/    Resolution: 1920x1080
  .++/+:+oo+o:`             /sssooo.   DE: MATE 1.10.2
 /+++//+:`oo+o               /::--:.   WM: Metacity (Marco)
 \+/+o+++`o++o               ++////.   GTK Theme: 'Ambiant-MATE' [GTK2/3]
  .++.o+++oo+:`             /dddhhh.   Icon Theme: Ambiant-MATE
       .+.o+oo:.          `oddhhhh+    Font: Ubuntu 10
        \+.++o+o``-````.:ohdhhhhh+     CPU: ARMv7 rev 3 (v7l) @ 1.4GHz
         `:o+++ `ohhhhhhhhyo++os:      GPU: Gallium 0.4 on llvmpipe (LLVM 3.6, 128 bits)
           .o:`.syhhhhhhh/.oo++o`      RAM: 537MiB / 1990MiB
               /osyyyyyyo++ooo+++/    
                   ````` +oo+++o\:    
                          `oo++.      

odroid@odroid:~$ screenfetch -h

Neofetch.png

Dictionary - Artha

  • Lifehacker. Once it is launched, it is sitting on the task bar. Press Ctrl+Alt+W after selecting a word to look it up in Artha (a balloon tip will pop up on the screen top-right). It also supports using regular expressions to search words.
sudo apt-get install artha

Translation

odroid@odroid:~/binary$ ./trans :zh-TW word
word
/wərd/

字
(Zì)

Definitions of word
[ English -> 正體中文 ]

noun
    字
        word, character, letter, calligraphy, symbol, style of writing
    詞
        word, term, speech, statement
    單詞
        word, individual word
    話
        words, word, dialect, saying, talk, speech
    言
        word, speech, character
    言辭
        words, word, what one says
    筆墨
        pen and ink, words, word, writings
    約言
        pledge, promise, word

verb
    為 ... 措辭
        word
odroid@odroid:~/binary$ time ./trans -brief :zh-TW word
字

real	0m4.249s
user	0m2.670s
sys	0m1.330s

ASCII art/ word art

  ____  ____  ____       ____            _____           _     
 | __ )|  _ \| __ )     / ___|  ___  __ |_   _|__   ___ | |___ 
 |  _ \| |_) |  _ \ ____\___ \ / _ \/ _` || |/ _ \ / _ \| / __|
 | |_) |  _ <| |_) |_____|__) |  __/ (_| || | (_) | (_) | \__ \
 |____/|_| \_\____/     |____/ \___|\__, ||_|\___/ \___/|_|___/
                                       |_|                     

  ____  _____  ____        _____         _______          _     
 |  _ \|  __ \|  _ \      / ____|       |__   __|        | |    
 | |_) | |__) | |_) |____| (___   ___  __ _| | ___   ___ | |___ 
 |  _ <|  _  /|  _ <______\___ \ / _ \/ _` | |/ _ \ / _ \| / __|
 | |_) | | \ \| |_) |     ____) |  __/ (_| | | (_) | (_) | \__ \
 |____/|_|  \_\____/     |_____/ \___|\__, |_|\___/ \___/|_|___/
                                         | |                    
                                         |_|                    

  ___ ___ ___     ___          _____         _    
 | _ ) _ \ _ )___/ __| ___ __ |_   _|__  ___| |___
 | _ \   / _ \___\__ \/ -_) _` || |/ _ \/ _ \ (_-<
 |___/_|_\___/   |___/\___\__, ||_|\___/\___/_/__/
                             |_|

Software that scan Malware and rootkits

Text to speech

VPN

Ubuntu -> VPN

Mono Project

Mono is a software platform designed to allow developers to easily create cross platform applications part of the .NET Foundation

Mono is required for Repetier-Host software for 3D printing.

NAS server

OpenMediaVault

OpenMediaVault

FreeNAS

ZFS system (FreeBSD-based).

10 Reasons Why You Should Store Your Data on a FreeNAS Box. Note With the current version of FreeNAS (FreeNAS 11) comes a hypervisor. See

Change detection

http://bhfsteve.blogspot.com/2013/03/monitoring-web-page-for-changes-using.html

3 command-line tools for feigning productivity

https://opensource.com/article/18/2/command-line-tools-productivity: Blessed-contrib (javascript), Genact, Hollywood.

Mind mapping

Diagram

Open source surveillance

ZoneMinder

Systemctl, systemd

Systemd vs SysVinit

Systemctl vs service commands

Difference between systemctl and service commands 

$ systemctl   # list all services
$ cat /lib/systemd/system/rsyslog.service
$ systemctl status rsyslog
$ cat /lib/systemd/system/ufw.service

How to Run a Linux Program at Startup with systemd

How to Run a Linux Program at Startup with systemd

How Long Does it Take To Boot Your Linux System

Find Out How Long Does it Take To Boot Your Linux System 

$ systemd-analyze       # total boot time along with the time taken by 
                        # firmware, boot loader, kernel and the userspace
$ systemd-analyze blame # breakdown the boot time into each unit

chkservice

chkservice Is A systemd Unit Manager With A Terminal User Interface

Kernel

Bitcoin

Game

Best Linux Adobe Alternatives You Need to Know

Linux distributions

The Top 10 Open Source Distros You Haven’t Heard About

Linux Distro for Your Cloud Server

How to Choose the Right Linux Distro for Your Cloud Server Fedora is the upstream source of Red Hat. CentOS is the downstream source of RHEL.

Small/lightweight Linux distributions

Live USB

See Live USB.

10 Best And Most Secure Linux Distributions

https://fossbytes.com/secure-linux-distros-privacy-anonymity/

Kids

How to Configure Linux for Children

chroot

How to Use the chroot Command on Linux

Retrieved from "https://wiki.taichimd.us/index.php?title=Linux&oldid=27804"