Linux: Difference between revisions

From 太極
Jump to navigation Jump to search
Line 4,628: Line 4,628:


=== Terminal ===
=== Terminal ===
==== Directory color on ssh ====
When I ssh to biowulf from Gnome/Terminal, the folder color became dark blue which is hard to read when the background color is black. A solution is https://serverfault.com/questions/137649/changing-terminal-colors-in-ubuntu-server.
1. append this to your ~/.bashrc
<pre>
    if [ "$TERM" != "dumb" ]; then
      [ -e "$HOME/.dir_colors" ] &&
      DIR_COLORS="$HOME/.dir_colors" [ -e "$DIR_COLORS" ] ||
      DIR_COLORS=""
      eval "`dircolors -b $DIR_COLORS`"
      alias ls='ls --color=auto'
    fi
</pre>
2. create/edit your ~/.dir_colors, for example with
<pre>
  $ dircolors --print-database > .dir_colors
</pre>
3. Then force a read of your .bashrc file with:
<pre>
  $ source ~/.bashrc
</pre>
Everything should be pretty.
==== Remember terminal tabs ====
==== Remember terminal tabs ====
The trick on [http://askubuntu.com/questions/310705/some-fast-way-to-save-and-restore-tabs-of-terminal this post] works for me on my GNOME Terminal 3.6.2 (gnome-terminal --version).  
The trick on [http://askubuntu.com/questions/310705/some-fast-way-to-save-and-restore-tabs-of-terminal this post] works for me on my GNOME Terminal 3.6.2 (gnome-terminal --version).  

Revision as of 10:39, 6 November 2017

Handy Linux tips

Some books from OReilly

  • UNIX in a nutshell
  • sed & awk
  • lex & yacc
  • bash
  • Linux Pocket Guide
  • bash cookbook: pdf is online
  • Classic Shell Scripting
  • GNU EMACS
  • Learning the vi and Vim editors 7th
  • UNIX POWER TOOLS
  • Bash Pocket Reference
  • System Administration 3rd
  • TCP/IP 3rd: pdf is online
  • DNS and BIND 5th
  • Network Troubleshooting Tools

Beautiful desktop

Virtual consoles/virtual terminals

Linux allows virtual consoles (aka virtual terminals) to be opened while an X Window System is executing.

Use Ctrl + Alt + FX to open a virtual console-- there are six virtual text-based consoles (F1 to F6). Use Ctrl + Alt + F7 (or Ctrl + Alt + F8 on Linux Mint) to return to the X Window System.

On X Window System, we can use Ctrl + Alt + -> or Ctrl + Alt + <- to switch workspaces.

Managing devices in Linux -> Fun with device files.

Virtual memory

vmstat – A Standard Nifty Tool to Report Virtual Memory Statistics

hcache

A tool fork from pcstat, with a feature that showing top X biggest cache files globally

How much resource is used by a process

Find the process ID first by ps -ef | grep APPLICATIONAME. Then 

ps -p <pid> -o %cpu,%mem,cmd

For example, 

$ ps -ef | grep akregator
brb      15013  1942  1 10:41 ?        00:00:05 akregator --icon akregator -caption Akregator
brb      15186 24045  0 10:50 pts/11   00:00:00 grep --color=auto akregator
$ ps -p 15013 -o %cpu,%mem,cmd
%CPU %MEM CMD
 1.0  0.8 akregator --icon akregator -caption Akregator

All You Need To Know About Processes in Linux

http://www.tecmint.com/linux-process-management/

5 Things to do after a fresh install of GNU/Linux

https://www.ghacks.net/2017/11/04/5-things-to-do-after-a-fresh-install-of-gnulinux/

  1. Increase audio quality
  2. Make sure firewall is enabled.
  3. Disable any unnecessary services
  4. Install Timeshift
  5. Install ClamAV / Clamtk antivirus

Query whether the OS is 64-bit or 32-bit

SYSTEM_ARCH=getconf LONG_BIT
echo $SYSTEM_ARCH

Switch user in command line

use 

su newusername

to switch to another user.

Directory permission / attribute

See http://unix.stackexchange.com/questions/21251/how-do-directory-permissions-in-linux-work

When applying permissions to directories on Linux, the permission bits have different meanings than on regular files.

  • The write bit allows the affected user to create, rename, or delete files within the directory, and modify the directory's attributes
  • The read bit allows the affected user to list the files within the directory
  • The execute bit allows the affected user to enter the directory, and access files and directories inside

When we create a new directory, the attribute is 775. Some pre-created directories (Desktop, Documents, Music, Pictures, Public) have an attribute 755.

IP address fundamental

http://www.howtogeek.com/133943/geek-school-learning-windows-7-ip-addressing-fundamentals/

Subnet

Change to root shell

The following command will switch to an environment similar to what the user would expect had the user logged in directly. 

sudo su -
# OR
sudo su
# OR
sudo -s

This can be useful when running 'su' or 'su -' failed because of an authentication failure error (note Ubuntu locked the root account).

See also

For sudo to work, my account ('debian' in this case) has to be included in the config file /etc/sudoers. 

debian  ALL=NOPASSWD: ALL

When sudo is invoked, it asks for the password of the user who started it.

Switch to another user and run a command using su -c

Example: 

sudo su - -c "R -q -e \"install.packages('mypackage', repos='http://cran.rstudio.com/')\""
# OR
sudo su -c "COMMAND_REQUIRE_ROOT_ACCESS"
# OR
sudo "COMMAND_REQUIRE_ROOT_ACCESS"

man su

What’s the Difference Between Bash, Zsh, and Other Linux Shells

https://www.howtogeek.com/68563/htg-explains-what-are-the-differences-between-linux-shells/

Bash shell programming

http://bash.cyberciti.biz/guide/Main_Page

Redirect standard error

http://bash.cyberciti.biz/guide/Standard_error. Use 2> operator. 

command 2> errors.txt

Quotes and asterisk

Combining these two will not work. For example 

brb@T3600 ~ $ ls -l ~/GSE48215/*.fastq
-rw-r--r-- 1 brb brb 16226673016 Jun 14 14:13 /home/brb/GSE48215/SRR925751_1.fastq
-rw-r--r-- 1 brb brb 16226673016 Jun 14 14:13 /home/brb/GSE48215/SRR925751_2.fastq
brb@T3600 ~ $ ls -l '~/GSE48215/*.fastq'
ls: cannot access ~/GSE48215/*.fastq: No such file or directory
brb@T3600 ~ $ ls -l "~/GSE48215/*.fastq"
ls: cannot access ~/GSE48215/*.fastq: No such file or directory

cp command

Linux cp command tutorial for beginners (8 examples)

copy a directory

cp -avr Dir1 Dir2

where -a will preserve the attributes of files/directories, -v means verbally and -r means copy the directory recursively.

Copy a file with progress bar with pv (plus how to eject the USB drive)

http://www.tecmint.com/monitor-copy-backup-tar-progress-in-linux-using-pv-command/ 

sudo apt-get install pv
pv file1 > file2    # don't forget the ">" operator and the destination is a file, not a directory

After that, instead of clicking the reject icon from the file manager to eject it, it is better to use a command line to do that because there is no expect time for users to know when it will take for finish writing the data to a USB drive. 

sudo apt-get install udisks

sudo udisks --unmount /dev/sdb1   # /dev/sdb1 is the partition
sudo udisks --detach /dev/sdb     # /dev/sdb is the device

My testing shows this procedure works (tested by running md5sum after eject/plug-in) when I need to copy a 9GB file.

Reliable way: Split the large file and copy smaller chunks

# Use 'sudo iotop -o' to monitor the I/O
split -b 4G inputFile  # create xaa, xab, ... files
cat x* > outputFile    # merge them. md5sum check succeeds

type  x* > outputFile  # Windows OS. http://stackoverflow.com/questions/60244/is-there-replacement-for-cat-on-windows

It is interesting copying smaller files (eg 4GB) to USB drives is quite stable (just use the cp command). Even for a not-too large file (6.7GB), pv step looks OK but the unmount/detach step failed.

For a 6.7GB file, it will split it into a 4GB and 2.7GB files. Merge takes longer time if it is done on the USB drive. That is, it is best to do merge in the final destination (internal disk/storage).

  • split in the internal hdd: 1min 38sec
  • merge in the internal hdd: 37sec
  • merge in the USB 3.0 drive: 2min 17sec

Remember: Use a reliable USB drives.

The operation could not be completed because the volume is dirty

On a USB 2.0 drive, I can copy files to there but the drive cannot be rejected (Ubuntu has a pop-up showing it is still writing data to it).

When I forcibly rejects the drive and plug it in a Windows PC, Windows shows the message The operation could not be completed because the volume is dirty. This gives a way to run chkdsk (check and repair a file system).

  1. Open a Windows File Manager
  2. Right click the USB drive
  3. Properties
  4. Tools -> Check now... Start

Done. Now I can use the drive again.

The Linux equivalent to chkdsk is fsck. fsck is a front end that calls the appropriate tool (fsck.ex2, fsck.ex3, e2fsck, ...) for the filesystem in question. 

 
umount /dev/sdb1        # thumb drive
sudo fsck /dev/sdb1

sudo fsck -a /dev/sdb1  # auto repair

Multiple files, new directory

rm -r ~/Documents/htg/{done,ideas,notes}

mkdircd MyNewDirectory

ls

Follow the symbolic link

Use -H option 

ls -lH myDir

ls | more without lose color

ls --color | more

Most likely your ls is aliased to ls --color=auto. If you do ls --color (which is morally equivalent to ls --color=always), that will force it to turn on colors.

ls output with color background

stackexchange or askubuntu.

In my case, after I apply chmod 755 -R XXXX, the weird green background color goes away.

Meld and Diffuse

To make meld to be in the right click menu, follow

Another method of comparing two files without using the 'browse' button will be to use the command line.

The 'nautilus-compare' program does not work from my testing on Ubuntu 14.04.

Refresh does not work

On Ubuntu 14, Meld version is 1.8.4. The current version is 1.16.2 (Jul 30 2016). The current version requires GTK+ 3.14 or higher. 

brb@brb-P45T-A:$ ~/binary/meld-3.16.2/bin/meld
Meld requires GTK+ 3.14 or higher.

Final though

  • I install kdiff3 (<2 MB to download) and the 'File' -> 'Reload' (F5) function there works though it shows an extra space on the place I modified.
  • Beyond Compare (commercial $60, trial version can be downloaded)
  • diffuse. When I modified a file, diffuse can detect a change and ask me to reload the file. I am using the apt-get to install the software and the version number is 0.4.7 (2014). To copy lines from left panel to right panel, use 'Ctrl + Shift + >' or the Copy Selection Right icon. One drawback is it cannot save the history from the GUI though we can use the command line to include the file names in the arguments.
  • Alternatively we can use WinMerge on Linux. To do that, install Wine on Ubuntu. Download Winmerge (I am using 2.14.0). Then on a terminal, run the following command. At the end, WinMerge will be launched. WinMerge can also be launched from Mint Menu -> Wine -> WinMerge. One problem is I cannot increase the font size (though acceptable) from View -> Select Font.
wine WinMerge-2.14.0-Setup.exe

diff

Run diff with large files

Meld freezes When I tested it with two large files (800k & 936k lines coming from human gtf files). Actually the whole linux system became unresponsive.

Actually Meld is sluggish when it is used in small files in Odroid XU4 running Ubuntu 16.04 MATE. I have used Meld 3.14.2 and the latest 3.16.2.

Directory

diff -qr dir1 dir2

where -q means to report only when files differ and -r is to recursively compare any subdirectories found.

diff & colordiff-color in terminal

PS. For a GUI version of diff, Meld works fine. Need to install first. apt-get install colordiff. http://www.cyberciti.biz/programming/color-terminal-highlighter-for-diff-files/ 

sudo apt-get install colordiff
diff -y file1 file2 | colordiff
# Ignore same rows (two ways):
# diff -C0 file1 file2 | colordiff
# diff -U0 file1 file2 | colordiff

where -y option means to show the output in two columns.

Colordiff.png

Interpretation of the diff output:

The first line of the diff output will contain:

  • line numbers corresponding to the first file,
  • a letter (a for add, c for change, or d for delete), and
  • line numbers corresponding to the second file.

In our output above, 2,4c2,4 means: "Lines 2 through 4 in the first file need to be changed in order to match lines 2 through 4 in the second file." It then tells us what those lines are in each file:

  • Lines preceded by a < are lines from the first file;
  • lines preceded by > are lines from the second file.
  • The three dashes ("---") merely separate the lines of file 1 and file 2.
2,4c2,4
< I need to run the laundry.
< I need to wash the dog.
< I need to get the car detailed.
---
> I need to do the laundry.
> I need to wash the car.
> I need to get the dog detailed.

colordiff -ur path1 path2
# If you change -ur to -urN then that will also show the contents of files that are only present in one of the paths.

Colordiff2.png

The meaning of colors can be found in /etc/colordiffrc (man colordiff)

  • plain=off
  • newtext=darkgreen
  • oldtext=darkred
  • diffstuff=darkcyan
  • cvsstuff=cyan

Terminator - terminals in grids

https://gnometerminator.blogspot.com/p/introduction.html

Guake / Yakuake / Tilda

Drop down terminals for the GNOME / KDE / GTK Environments. Great for quick access to a terminal!

System date/time

Install and configure Network Time Protocol (NTP) Server,Clients on Ubuntu 16.10 Server

Change the date/timestamp of a file - touch

Modify the file relative to its existing modification time 

filename=MyFileName
touch -d "$(date -R -r $filename) - 2 hours" $filename # 2 hours before
touch -d "$(date -R -r $filename) + 2 hours" $filename # 2 hours later

See How can I change the date modified/created of a file?

Find binary file location

  • which - Display the full path of shell commands. See examples from cyberciti.biz.
$ which ls
/bin/ls
  • whereis - locate the binary, source, and manual page files for a command. See examples from cyberciti.biz.
$ whereis ls
ls: /bin/ls /usr/share/man/man1p/ls.1p.gz /usr/share/man/man1/ls.1.gz
  • type -a
$ type -a ls
ls is aliased to `ls --color=tty'
ls is /bin/ls

Use locate command mindfully. It is used to find the location of files and directories. Note that locate does not search the files on disk rather it searches for file paths in a database. For example, the following command will search .png files over the system (not only the personal directory). 

locate "*.png"

find: Find a file based on file name

$ find . -iname '*.txt'  # -iname or -name is necessary

It also works for searching files on subdirectories. 

$ find . -name transcripts.gtf
./RH_bio/transcripts.gtf
./dT_ori/transcripts.gtf
./dT_tech/transcripts.gtf
./dT_bio/transcripts.gtf
./RH_ori/transcripts.gtf
./RH_tech/transcripts.gtf

Find files and execute something (google: find --exec) 

$ find ./ -name "*.tar.gz" -exec tar zxvf {} \;

Find files modified in one day. 

$ find . -mtime -1

Find files modified in one day and contain string 'est' 

$ find . -mtime -1 -exec grep --with-filename est {} \;

If the search directory is not the current directory, we need to add a forward slash to the directory name. 

$ find ~/Desktop -iname '*.txt'  # Not working
$ find ~/Desktop/ -iname '*.txt' # Working

The following example shows we can list multiple search criteria. The “‑r” option in tar appends files to an archive. xargs is a handy utility that converts a stream of input (in this case the output of find) into command line arguments for the supplied command (in this case tar, used to create a backup archive). 

find / -type f -mtime -7 | xargs tar -rf weekly_incremental.tar
gzip weekly_incremental.tar

xargs

See Linux Programming

-exec COMMAND {} +

The following will find out the total file size of the 'accepted_hits.bam' file under all sub-directories. 

find ./ -iname "accepted_hits*" -exec du -ch {} + | grep total$

where '-c' produces a grand total, and will substitute {} with the filename(s) found in -exec.

grep: Find a file by searching contents

grep -r -i "Entering" ~/Downloads/R-3.0.0/

where -r means recursively searching the directory and -i means case insensitive.

We can also display the row numbers for matches by using the -n parameter in grep. 

# What variants appear in dbsnp
grep -n 'rs[0-9]' XXX.vcf

To exclude lines with a pattern, using the -v parameter. 

# How many variant were called
grep -v "^#" XXX.vcf | head

To show only matched filenames, using the -l parameter. 

grep -l "iterator" *.cpp
# if we add '-n', the '-n' option won't work.

If the pattern is saved in a file, use the -f parameter 

grep -f PATTERNFILE INPUTFILE

https://www.howtoforge.com/tutorial/linux-grep-command/ gives more examples

  • Using grep to search only for words ("-w" option)
  • Using grep to search two different words (egrep -w 'word1|word2' /path/to/file)
  • Count line for matched words ("-c" option)
  • Grep invert match ("-v" option)
  • How to list only the names of matching files ("-l" option)

GUI

A GUI version of a tool to search files is searchmonkey (open source, Linux, Windows). On Ubuntu, we install it by 

sudo apt-get install searchmonkey

It is also useful to change the settings so we can click a filename and open it in the desired text editor. To do that, go to Settings -> Preferences -> System Call -> Text Editor. I enter 'geany' since I want to use geany to open my C programs. Note. the v2.0 source code needs to be built using i386 gcc library and Qt 4.8.x. Still, I cannot get rid of some errors coming from the source code.

Summary of find and grep commands

Command Examples
find find [DIRECTORY] -iname '*.txt'

find [DIRECTORY] -maxdepth 2 -iname *.php

find -name '*.php' -o -name '*.txt' # OR operator

grep grep -r -i "check_samtools" DIRECTORY/

dpkg -l libgtk* | grep '^i'

Count number of columns: awk

The following command shows the number of columns for the first few rows of a text file. 

head MYFILE | awk '{ print NF}'

head MYFILE | awk -F '\t'  '{ print NF}'

Count number of rows in a file: wc

wc -l MYFILE

The source code of wc (or any Linux command) can be found by using this method 

brb@brb-T3500:~/Downloads$ which wc
/usr/bin/wc
brb@brb-T3500:~/Downloads$ dpkg -S /usr/bin/wc
coreutils: /usr/bin/wc
brb@brb-T3500:~/Downloads$ sudo apt-get source coreutils
[sudo] password for brb: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Need to get 12.3 MB of source archives.
Get:1 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main coreutils 8.21-1ubuntu5.1 (dsc) [1,635 B]
Get:2 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main coreutils 8.21-1ubuntu5.1 (tar) [12.3 MB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main coreutils 8.21-1ubuntu5.1 (diff) [31.6 kB]
Fetched 12.3 MB in 22s (559 kB/s)                                              
gpgv: Signature made Tue 13 Jan 2015 10:33:04 PM EST using RSA key ID 9D8D2E97
gpgv: Cannot check signature: public key not found
dpkg-source: warning: failed to verify signature on ./coreutils_8.21-1ubuntu5.1.dsc
dpkg-source: info: extracting coreutils in coreutils-8.21
dpkg-source: info: unpacking coreutils_8.21.orig.tar.gz
dpkg-source: info: applying coreutils_8.21-1ubuntu5.1.diff.gz

As we can see from the coreutils-8.21/src directory, there are over 100 C programs including <cat.c>, <chmod.c>, <cp.c>, ...<wc.c>.

Print certain rows/lines of a text file

The following example will print out lines 10 to 60 of FILENAME. 

sed -n '10,60p' FILENAME

Or to print out line 60, 

sed -n '60p' FILENAME

It seems this method is not as fast as I expected. For example, the tail command will immediately print out the result without waiting!

output colored console to html

Use ansi2html.sh. It only requires gawk.

  1. Use wget to download it
  2. sudo apt-get install gawk
  3. chmod +x ansi2html.sh
  4. colordiff file1 file2 | ./ansi2html.sh > diff.html

using a the result of a diff in a if statement

ls -lR $dir > a
ls -lR $dir > b

DIFF=$(diff a b) 
if [ "$DIFF" != "" ] 
then
    echo "The directory was modified"
fi

Another example 

if [ "$(diff file1.html file2.html)" == "" ]; then echo Same; else echo Different; fi

Prompt

Colored prompt

http://www.tldp.org/HOWTO/Bash-Prompt-HOWTO/x329.html

For example, the following code will change the prompt to a light blue color. 

PS1="\[\033[1;34m\][\u@\h:\w]$\[\033[0m\] "

In Odroid running Ubuntu mate, we can modify ~/.bashrc and un-comment the following line 

force_color_prompt=yes

Add a timestamp to your Bash prompt

PS1='[\D{%F %T}] \u@\h \W\$ '

So the output will be like 

[2016-07-08 16:56:48] brb@brb-P45T-A ~$

instead of 

brb@brb-P45T-A:~$

From here, we can skip %F (not showing the date), \W (not showing the current directory) and change %T to %H:%M (not showing seconds). 

export PROMPT_COMMAND="echo -n \[\$(date +%H:%M:%S)\]\ "

So the output will be something like: 

[07:00:31] user@hostname:~$

instead of 

user@hostname:~$

To the right hand side/Aligned to right

Listen to pandora in Europe: install squid proxy

http://www.cyberciti.biz/faq/access-pandora-radio-using-proxy-server-outside-usa/

Interestingly, the firefox connection settings should choose HTTP Proxy instead of 'SOCKS host'.

scp

file path with spaces

Use double quotes around the full path and a backslash to escape any space. 

scp [email protected]:"web/tmp/Master\ File\ 18\ 10\ 13.xls" .

Recursive copying

Use -r parameter.

Preserve permissions and modes

Use -p parameter.

scp files through one intermediate host

http://stackoverflow.com/questions/9139417/how-to-scp-with-a-second-remote-host

The following command is tested. 

scp -o 'ProxyCommand ssh user@remote1 nc %h %p' user@remote2:path/to/file .

A second method which is useful for ssh and scp commands 

$ ssh -L 9999:host2:22 user1@host1 # leave the terminal
# Open a new terminal
$ scp -P 9999 fileName user2@localhost:/path/to/dest/fileName   # transfer from local to remote. Note: Upper P.
$ scp -P 9999 user2@localhost:/path/to/source/fileName fileName # transfer from remote to local. Note: Upper P.
# If we only want to use ssh
$ ssh -p 9999 user2@localhost # Note: lower p.

scp with non-standard port: -P (capital)

Use -P argument. 

scp -P 23 myfile user@remoteip:

scp or ssh without password

Steps:

  1. Verify that local-host and remote-host is running openSSH (ssh -V)
  2. Generate key-pair on the local-host using ssh-keygen (Enter a passphrase here, do not leave it empty. A passphrase should be at least several words long, something you can easily remember. It's a bad idea to use a single word as a passphrase.)
  3. Install public key on the remote-host
  4. Give appropriate permission to the .ssh directory on the remote-host (chmod 755 ~/.ssh; chmod 644 ~/.ssh/authorized_keys)
  5. Login from the local-host to remote-host using the SSH key authentication to verify whether it works properly
  6. Start the SSH Agent on local-host to perform ssh and scp without having to enter the passphrase several times (ssh-agent $SHELL)
  7. Load the private key to the SSH agent on the local-host (ssh-add, need to enter the passphrase 1 time only)
  8. Perform SSH or SCP to remote-home from local-host without entering the password. It works for all remote machines containing the key from local-local.

Another option is to use ssh -i IDENTITY_FILE. See superuser.com.

ssh with password on the command line

Install sshpass utility. See https://serverfault.com/questions/241588/how-to-automate-ssh-login-with-password

SSH

Way to avoid ssh connection timeout

https://superuser.com/questions/98562/way-to-avoid-ssh-connection-timeout-freezing-of-gnome-terminal

put the following in your ~/.ssh/config. 

Host remotehost
  HostName remotehost.com
  ServerAliveInterval 240

To enable it for all hosts use: 

Host *
  ServerAliveInterval 240

Also make sure to run chmod 600 ~/.ssh/config

Running commands on a remote host

ssh user@host 'COMMANDS'

ssh user@host "command1; command2; command3"

COMMANDS="command1; command2; command3"
ssh user@host "$COMMANDS"

A practical example 

#!/bin/bash

IP_LIST="192.168.0.1 192.168.0.5 192.168.0.9"
USER="test"

for IP in $IP_LIST;
do
  utime=$(ssh ${USER}@${IP} uptime  | awk '{ print $3 }' )
  echo $IP uptime:  $utime
done

ssh log files

It is also helpful to check /etc/hosts.allow and /etc/hosts.deny for any possible wrong configuration.

ssh key

SSH key is useful if you want a password-less login to a remote system. Some useful resources:

Also there are different kinds of keys (see for example <~/.ssh/known_hosts file>): RSA, DSA and ECDSA (newer). They're keys generated using different encryption algorithms. See SSH key-type, rsa, dsa, ecdsa, are there easy answers for which to choose when?

The steps are

  • Check if there is an existing key
ls -al ~/.ssh
  • Create a new RSA key pair:
ssh-keygen -t rsa
ssh-keygen -f ~/.ssh/personalid -C "bitbucket"

where the comment 'bitbucket' will appear at the end of <~/.ssh/personalid> file.

  • Copy the public key to a remote host ([email protected]) over ssh. The current user (eg brb) and the remote user (eg git)have not any relationship (they most likely have different user names):
ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected] # this will 'append' the key to the remote-host’s .ssh/authorized_key.

Or (may not work:() 

cat ~/.ssh/id_rsa.pub | ssh [email protected] "mkdir -p ~/.ssh && cat >>  ~/.ssh/authorized_keys"
  • Delete the authorized key. Open the text file '.ssh/authorized_keys' and remove the offending lines.
  • Test if this is working by trying 'ssh [email protected]'.
  • To disable the password for root login. Type sudo nano /etc/ssh/sshd_config
PermitRootLogin without-password

Then run the following to put the changes into effect: 

reload ssh
# Or service ssh restart

If we like to ask all users to use key-based to log in, we can modify the line 

PasswordAuthentication no

in sshd_config.

Multiple ssh keys

<Method 1> If we want to use a specific key in ssh, use 

ssh -i ~/.ssh/xxx_id_rsa [email protected]

<Method 2> Another way is to use ssh-add & ssh-agent to manager your keys. ssh-agent keeps your key in its memory and pulls it up whenever it is asked for it. 

$ ssh-keygen -f ~/.ssh/personalid -C "bitbucket"
$ eval $(ssh-agent -s)       # Ensure ssh-agent is enabled:
$ ssh-add ~/.ssh/personalid  # ssh-add program will ask you for your passphrase
$ ssh-add -l

<Method 3> <~/.ssh/config> file.

ssh key management

Copy ssh keys to another computer

http://askubuntu.com/questions/134975/copy-ssh-private-keys-to-another-computer 

$ chown brb:brb ~/.ssh/id_rsa*
$ chmod 600 ~/.ssh/id_rsa
$ chmod 644 ~/.ssh/id_rsa.pub

If we do not change the permission correctly in <id_rsa>, we will get a warning: Unprotected private key file. Permissions 0664 for '/home/USERNAME/.ssh/id_rsa' are too open.

Preserve ssh keys when upgrading computers

ls -l /etc/ssh/*key* > ~/key_list
mkdir ~/serverkeys && cp -p /etc/ssh/*key* ~/serverkeys/
cp -p ~/serverkeys/*key* /etc/ssh
ls -l /etc/ssh/*key* | diff - ~/key_list

If diff produces no output, you're finished.

The dates of the key files show the original date the Linux system was created.

Disable SSH host key checking

ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no [email protected]

To disable the checking for all hosts, in your ~/.ssh/config (if this file doesn't exist, just create it): 

Host *
    StrictHostKeyChecking no

Handling the ssh key change when connecting to a remote machine

An article from cybercitz.biz.

  • Method 1. Remove the key using ssh-keygen -R command.
$ ssh-keygen -R {server.name.com}
$ ssh-keygen -R {ssh.server.ip.address}
$ ssh-keygen -R server.example.com
  • Method 2. Add correct host key in /home/user/.ssh/known_hosts
  • Method 3. Just delete the known_hosts file If you have only used one ssh server

SSH Port forwarding

  • Chapter 9 Port forward. SSH Mastery OpenSSH, PuTTY, Tunnels and Keys by Michael W. Lucas

Verizon Quantum Gateway Router

User guide p98. Click 'Advanced' button first.

  • Source port: Any (this is the key)
  • Destination Ports: the port you want to use (connect from outside)
  • Forward to Port: Same as incoming port or the port used in the local computer

What is tunnel

https://www.howtogeek.com/299845/why-is-a-network-tunnel-called-a-tunnel/. A tunnel provides a direct path that avoids some type of complexity you would otherwise have to deal with.

Local port forwarding

This port forwarding involves three computers (local, remote and hostname) as you can see from the SSH syntax.

For example, we like to access home's router (192.168.1.1) information from an outsider computer. Suppose the host 'hostname' is one computer in the home network and it can be accessed from outside world. 

# ssh -L localhost:localport:remoteIP:remoteport hostname
# ssh -L localport:remoteIP:remoteport hostname
ssh -L 8080:192.168.1.1:80 username@hostname

The -L option specifies local port forwarding. In this case, port 8080 on the local machine was forwarded to port 80 on the remote machine. For the duration of the SSH session, pointing your browser at http://localhost:8080/ would send you to http://192.168.1.1/ as if you are in the same local network of 'hostname'.

The reason it works is because the 'ssh' trick. In addition to being able to make yourself in the home network environment, the traffic on http://localhost:8080 is encrypted too.

Note that this forwarding uses port 8080 on the client rather than port 80. Binding to port 80 would require using root privileges every time we SSH.

To stop the ssh session, use ps -ef to find the process id and kill it.

Remote port forwarding (Reverse port forwarding)

This is most useful in situations where you have a machine which isn't publicly accessible from the internet, but you want others to be able to access a service on this machine. In this case, if you have SSH access to a remote machine which is publicly accessible on the internet, you can set up a reverse port forward on that remote machine to the local machine which is running the service. 

ssh -R 8000:localhost:80 user@REMOTE_MACHINE

This will forward port 8000 on the remote machine to port 80 on the local machine. Using this method, if you browse to http://localhost on the remote machine, you will actually connected to a web server running on port 8000 of the local machine.

Example 2: Suppose you have two machine

  • machine A (userA): under firewall. cannot be directly accessed (like corporate machines)
  • machine B (userB): local machine (like home machines)

Our goal is to access machine A directly from machine B.

We can run the following on the machine A 

# ssh -R remoteIP:remoteport:localIP:localport hostname
# ssh -R remoteport:localIP:localport hostname
ssh -R 2222:localhost:22 userB@machineB_IP
ssh -i /path/to/priv/key/id_rsa -f -N -R 2222:localhost:22 userB@machineB_IP

Then we can access machine A from machine B by 

ssh -p 2222 userA@localhost

If you want remote port forwarding configured every time you connect to a host, use the RemoteForward option in ssh_config . 

LocalForward server-IP:server-port client-IP:client-port

Dynamic port forwarding, SOCKS proxy, bypass blocked websites from work computer

ssh -D 4096 user@remoteip
ssh -D 4096 -p 23 user@remoteip

This will require you to enter the password and leave you in the remote machine. If a nonstandard port is required, we can use -p option.

Now in the firefox, we need to go to Edit -> Preferences -> Advanced -> Network tab -> Settings... Check 'Manual proxy configuration' (The default is 'Use system proxy settings') and enter 'localhost' for SOCKS (SOCKS5 by default) Host and '4096' for the Port. Don't enter 'localhost' in the HTTP Proxy.

Note that in addition to the Firefox, we can use SeaMonkey (seems better than Firefox since the form works better on 1024x600 resolution). The network setting option in my 32-bit maxthon browser does not work (cannot show options). For the Opera browser, it cannot connect to Internet after I made a change to the network setting.

On Windows, we can use Putty. In short, in the left-hand panel, navigate through Connection > SSH > Tunnels. Enter 4096 in the Source Port box and select the Dynamic radio button. Click Add and “D4096″ will appear in the Forwarded Ports list. The setting in the firefox end is the same. See also my Windows wiki page.

Linux journal also put a video on youtube. We can use http://www.ipligence.com/geolocation to check the current location. The port number is 1080 in the example. The example actually also use '-N' option which means no interaction; i.e. ssh -N -D 1080 user@remoteip. So we won't see anything after we type our password. Once we want to stop SOCK proxy, we just need to hit Ctr+C on terminal.

Backgrounding OpenSSH Forwarding

Use the -N flag to tell ssh to not run anything, including a terminal, on the remote server, and the -f flag to tell ssh to go into the background on the client. 

ssh -fNL 2222:localhost:22 user@remotehost &

By backgrounding this command, you get your original terminal back.

ssh through an intermediate server

Simple method is 

$ ssh -tt vivek@Jumphost ssh -tt vivek@FooServer

Another method is to use ssh ProxyCommand to tunnel connections.

A third method is to 

$ ssh -L 9999:host2:22 user1@host1  # leave this terminal 
# open a new terminal tab
$ ssh -p 9999 user2@localhost

Graphical way to display disk usage

For example, to use xdiskusage, we run apt-get install xdiskusage and launch it by xdiskusage ~/.

  • Ubuntu has a built-in program called "Disk Usage Analyzer". Just search it from Dash. Looks useful!

Display files sorted by modified date in a directory recursively

stat --printf="%y %n\n" $(ls -tr $(find DIRNAME -type f))
find -type f -printf '%T+\t%p\n' | sort -n

Both of methods give the same output. Note the latest changed file is shown at the bottom of the output.

Sort files by their size

use the '-S' option. 

ls -lS

df: Display disk space

df -h
df -h -T  # show the 't'ype of the file system
df -h -t ext4 # show file systems of given type (ext4 in this example)
df -a     # show all file system (include ones that have a size of zero blocks)

du and ncdu : Display directory size with sorting and human readable

Use ncdu program (more interactive). Although it is a command line program, we can use the mouse to move through each directory to see its sub-directories. 

sudo apt-get install ncdu
ncdu

And the du method. 

du -sh ~/*              # won't include hidden directories, Fast
du -h ~/ --max-depth=1  # include hidden directories, SLOW
du -h ~/ --max-depth=1 --exclude ".*" | sort -nr | cut -f2 | xargs -d '\n' du -sh
du -a -h ~/  # kilobytes will be used, '-a' is to see all files, not just directories.
du -a ~/ | sort -nr | head -n 10   # sort from the largest file size first

The --exclude is to hide hidden directories, '-n' is to compare according to string numerical value, and '-r' is to reverse the result.

Note that the 'du' commands may be cheating. See the following screenshot.

DiskUsage.png

The discrepancy is explained by 'sector'. See http://askubuntu.com/questions/122091/difference-between-filesize-and-size-on-disk. 

$ sudo dumpe2fs /dev/sda1 | grep -i "block size"
dumpe2fs 1.41.14 (22-Dec-2010)
Block size:               4096

To show a file size in terms of blocks, we can use 

ls -s

So for example, if a file takes 150 blocks, and if a block takes 4096 bytes, then the file takes 150*4096/1024 KB on disk.

Apache benchmark (ab) testing

ab -n 100 -c 10 http://taichimd.us/

Monitor progress of running a command

How to monitor progress of Linux commands using PV and Progress utilities 

# Method 1: rsync
rsync --progress -a sourceDirectory destinationDirectory
rsync --info=progress2 source dest

# Method 2: pv
sudo apt-get install pv
## copy a single file
pv inputfile > outputfile

## multiple files or directories 
tar c sourceDirectory | pv | tar x -C destinationDirectory

rsync

Copy large file

If we need to copy large file (say > 4GB), we shall

  1. format USB drive to NTFS (exFat seems not work)
  2. Run rsync --progress source dest
  3. Run sync

The last step (rsync) is important. We can use sudo iotop to check if rsync is finished or not.

rsync with exclude files/directories

See http://www.thegeekstuff.com/2011/01/rsync-exclude-files-and-folders/. The key is excluded files are relative to the current directory even we specify the absolute path. For example /path1/path2/file does not mean the file is located under /path1/path2; it means the file is located under ./path1/path2. 

rsync -avz --exclude '/path1/path2/file' source/ destination/

We add add multiple --exclude to exclude more files/directories.

rsync to exclude hidden files/directories

--exclude=".*"  # exclude both hidden files and directories
--exclude ".*"  # same as above

--exclude ".*/" # exclude hidden directories ONLY

--exclude ".git" # exclude .git directory ONLY; relative to the directory to be synchronized.

rsync with -a option

The -a flag in there stands for “archive,” and it’s important to include. It makes sure that the sync command is recursive (meaning any sub-folders and files inside of old_movies are copied too) and it’s important for preserving all of those modification dates, symbolic links, permissions, and other goodies we talked about earlier.

rsync with non-standard port

Use -e option 

rsync -avz -e "ssh -p 23" mydir user@remoteip:

rsync with progress bar

Use --progress option. 

rsync -avz --progress file1 file2

The 'rsync' command works on transferring files local to local too.

Or it is better to use -P option which is the same as --partial --progress. When it is used you’ll get a progress dialog at the command line that shows you which file is currently transferring, what percentage of that transfer is complete, and how many more files are left to check. As each file completes, you’ll see an ever-growing list of completed file transfers, which is great for making sure everything transfers successfully. It also allows you to easily resume suspended or interrupted transfers. Combined, you can see how it’ll show you which file was the last one to go, where it failed, and if it failed, give you the option to resume. It’s a pretty powerful combination.

rsync on Windows

Download and install command line rsync from http://www.rsync.net/resources/howto/windows_rsync.html. The website also provides a documentation. Some people are concern about the license issue. The website here provides a link to the free, old but usable version 4.0.5 which is newer than I tested v3.1.0.

Below are my note by using cwrsync v3.1.0 installer got from http://www.rsync.net. 

cd C:\Program Files (x86)\cwRsync\bin
ssh-keygen -t rsa -N ''
rsync -av "/cygdrive/c/Users/brb/.ssh/id_rsa.pub" [email protected]:.ssh/authorized_keys

rsync -av "/cygdrive/c/Users/brb/Downloads/cytokineMC.txt" [email protected]:Downloads/

sudo

How to Keep ‘sudo’ Password Timeout Session Longer in Linux

http://www.tecmint.com/set-sudo-password-timeout-session-longer-linux/

How to run multiple commands in sudo

https://www.cyberciti.biz/faq/how-to-run-multiple-commands-in-sudo-under-linux-or-unix/

How do I run specific sudo commands without a password?

https://askubuntu.com/questions/159007/how-do-i-run-specific-sudo-commands-without-a-password

Text browser

Links

Alternative browsers

Midori

Qupzilla

QupZilla is a new and very fast QtWebEngine browser. It aims to be a lightweight web browser available through all major platforms.

Filezilla

Keyboard shortcut. Especially, Alt+Down=Transfers the currently selected item to an item of the same name in the other pane.

The device is busy

brb@brb-P45T-A:~$ sudo umount /media/brb/TOSHIBA 
[sudo] password for brb: 
umount: /media/brb/TOSHIBA: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
brb@brb-P45T-A:~$ sudo umount /dev/sdc1
umount: /media/brb/TOSHIBA: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
brb@brb-P45T-A:~$ lsof /media/brb/TOSHIBA/
COMMAND  PID USER   FD   TYPE DEVICE   SIZE/OFF NODE NAME
VBoxSVC 5600  brb   18w   REG   8,33 4294967295    3 /media/brb/TOSHIBA/Windows 10.ova (deleted)
brb@brb-P45T-A:~$ kill -9 5600
brb@brb-P45T-A:~$ lsof /media/brb/TOSHIBA/
brb@brb-P45T-A:~$ sudo umount /dev/sdc1
brb@brb-P45T-A:~$ 

# fuser -m /dev/sdc1
/dev/sdc1: 538
# ps auxw|grep 538
donncha 538 0.4 2.7 219212 56792 ? SLl Feb11 11:25 rhythmbox

Another handy one is: 

umount -l /dev/sdwhatever

Kill a process and the pstree command

# find the PID
pgrep ProgramName
# Kill the ProgramName process
kill -9 PID

Another command is killall. For example, if Firefox is acting up (as Firefox will do from time to time) simply type killall firefox and it should kill the application completely.

In the rare circumstances that this doesn’t work you can always type xkill and then click on the window that won’t close; this will completely close a given window immediately. See this.

How To kill An Inactive OR Idle SSH Sessions. The pstree -p command can show a tree diagram of all the processes.

Create an ext3 file system on a USB flash drive

umount /dev/sdb1   (depending on the device of course)
sudo mkfs.ext3 /dev/sdb1
sudo e2label /dev/sdb1 usbdrive   (change the label)

We can create MS-DOS file system by 

sudo mkfs.vfat /dev/sdb1

Add a new user with home directory

adduser xxx

adduser is better than useradd since useradd does not create home directory and it does not even ask the password for new user. adduser will interactively ask user information.

To delete the user and home directory, use 

deluser --remove-home xxx

gzip with multi cores

Use pigz utility. It makes a lot of difference. For example for a 21GB file, gzip can't finish the job after 30 minutes. But pigz only took 7 minutes on a 12-core machine. 

sudo apt-get install pigz
pigz -9 FILENAME   # compress & convert the file to FILENAME.gz

tar cf - paths-to-archive | pigz -9 -p 12 > archive.tar.gz

There is no need to use pigz to un-compress the file. gunzip is fast enough and only takes 4 minutes to decompress.

The '-9' (best compression) option does not make difference (6.6G vs 6.5G).

Compress a folder without full path name

Suppose we want to compress the folder ~/Documents and its subfolders. We want to include Documents folder name but not /home/brb/Documents name. 

# Method 1. Include 'Documents' as the top folder name
cd ~/
tar -czvf tmp.tar.gz Documents
# Method 2. Mind the last dot. Not include 'Documents' as the top folder.
tar -czvf tmp.tar.gz -C /home/brb/Documents .

# Double check the tarball
tar -tzvf tmp.tar.gz

If we want to strip the upper directories when we uncompress a tar file, use --strip-components. For example, we can use --strip-components=1 to remove the Documents folder.

squashfs

squashfs is a heavy-compression based read-only filesystem that is capable of compressing 2 to 3 GB of data onto a 700MB. Linux liveCD are built using squashfs. These CDs make use of a read-only compressed filesystem which keeps the root filesystem on a compressed file. It can be loopback mounted and loads a complete Linux env. Thus when some file are required by processes, they are decompressed and loaded onto the RAM and used. 

# create a squashfs file
sudo mksquashfs /etc test.squashfs 

# mount the squashfs file 
mkdir /mnt/squash
mount -o loop compressedfs.squashfs /mnt/squash
# you can acess the contents at /mnt/squashfs

# exclude files 
sudo mksquashfs /etc test.squashfs -e /etc/passwd /etc/shadow
# or specify a list of exclude files given in a file
cat excludelist  # /etc/passwd
sudo mksquashfs /etc test.squashfs -ef excludelist

List contents of tar.gz or tar.bz2

tar -tzvf myfile.tar.gz

tar -tjvf myfile.tar.bz2  # replace z with j

Extract files

Extract tar.gz or zip to a specified directory

  • tar xzvf XXXX.tar.gz -C DIRECTORY
  • unzip XXX.zip -d DIRECTORY

Extract gz file but keep the original gz file

gunzip -c x.txt.gz > x.txt

gunzip -c which simply writes the output stream to stdout

Extract .xz file

xz -d archive.xz

Extract tar.xz file

The bottomline is we don't need the 'z' parameter (used for gz ONLY but does not work for xz file) in the tar command for tar.xz files. And the method also works for tar.gz files. The argument '-f' means the archive file. Recall that the tar command can be used to store and extract files, so no default parameters. 

tar xf archive.tar.xz
tar xf archive.tar.gz

Extract tar.bz2 file

tar -xjvf archive.tar.bz2  # replace z with j as we compare it to tar.gz file

How To Extract and Decompress a .bz2/.tbz2 File

See this article from cyberciti.biz. 

bzip2 -d your-filename-here.bz2
# OR
bzip2 -d -v your-filename-here.bz2
# OR
bzip2 -d -k your-filename-here.bz2
# OR
bunzip2 filename.bz2

How to Encrypt and Decrypt Files and Directories Using Tar and OpenSSL

http://www.tecmint.com/encrypt-decrypt-files-tar-openssl-linux/

How to install and use 7zip file archiver

https://www.howtoforge.com/tutorial/how-to-install-and-use-7zip-file-archiver-on-ubuntu-linux/

Compare zip, tar.xz, tar.gz, 7z

The compression rate comparison is (from best to worst) 7z > tar.xz > tar.gz > zip.

For example, consider qt-everywhere-opensource-src-5.5.0 from http://download.qt.io/official_releases/qt/5.5/5.5.0/single/

  • zip 540M
  • tar.xz 305M
  • tar.gz 436M
  • 7z 297M

Extract one files from tar.gz

Extract a file called etc/default/sysstat from config.tar.gz tarball: 

$ tar -zxvf config.tar.gz etc/default/sysstat

Noe that a new directory etc/default will be created under the current directory if it does not exist.

Wildcard based extracting

You can also extract those files that match a specific globbing pattern (wildcards). For example, to extract from cbz.tar all files that begin with pic, no matter their directory prefix, you could type: 

$ tar -xf cbz.tar --wildcards --no-anchored 'pic*'

To extract all php files, enter: 

$ tar -xf cbz.tar --wildcards --no-anchored '*.php'

remove leading directory components on extraction with tar

--strip-components option

AVFS and Archivemount

If we want to extract certain files from a tarballj/archive, it is more efficient to use a virtual filesystem like AVFS. PS. for a large archive file, even extracting only a single file at the top directory it is terribly slow if we use the tar command directly.

Before we install the utility, let's look at the package dependecies of AVFS and Archivemount. 

$ apt-cache showpkg archivemount
Package: archivemount
Versions: 
0.8.1-1 (/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages)
 Description Language: 
                 File: /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages
                  MD5: d6302be9f06a91afa32326ab175e2086
 Description Language: en
                 File: /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_i18n_Translation-en
                  MD5: d6302be9f06a91afa32326ab175e2086


Reverse Depends: 
  archivemount:i386,archivemount
Dependencies: 
0.8.1-1 - libarchive13 (0 (null)) libc6 (2 2.4) libfuse2 (2 2.8.1) fuse (2 2.8.5-2) archivemount:i386 (0 (null)) 
Provides: 
0.8.1-1 - 
Reverse Provides: 
brb@T3600 ~ $ apt-cache showpkg avfs
Package: avfs
Versions: 
1.0.1-2 (/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages) (/var/lib/dpkg/status)
 Description Language: 
                 File: /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages
                  MD5: bce08fbc36fd7b8e3c454f36f0daf699
 Description Language: en
                 File: /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_i18n_Translation-en
                  MD5: bce08fbc36fd7b8e3c454f36f0daf699


Reverse Depends: 
  avfs:i386,avfs
  worker,avfs
Dependencies: 
1.0.1-2 - libc6 (2 2.14) libfuse2 (2 2.8.1) fuse (0 (null)) unzip (0 (null)) zip (0 (null)) arj (0 (null)) lha (0 (null))
 zoo (0 (null)) rpm (0 (null)) p7zip (16 (null)) p7zip-full (0 (null)) cdparanoia (0 (null)) 
wget (0 (null)) avfs:i386 (0 (null)) 
Provides: 
1.0.1-2 - 
Reverse Provides:

Install it now. 

sudo apt-get install avfs
mountavfs
# Assume MyFile.tar.gz exists in the current directory
ls ~/.avfs/$PWD/MyFile.tar.gz#       
# Alternatively, browse the content in Nautilus, but you need to add a trailing # character by hand to the path 
# (Ctrl-L to access the address bar).
...
cat ~/.avfs/$PWD/MyFile.tar.gz#/README
# another tarball
ls ~/.avfs/$PWD/MyFile2.tar.gz#       
umountavfs

For some reason, avfs sometimes does not work:( In this case, Ubuntu's Archive Manager does work. Maybe the file is too large. 

brb@T3600 ~/Downloads $ time ls ~/.avfs/$PWD/Homo_sapiens_UCSC_hg19.tar.gz#/
ls: cannot access /home/brb/.avfs//home/brb/Downloads/Homo_sapiens_UCSC_hg19.tar.gz#/nown	exact	1	SingleClassTriAllelic,InconsistentAlleles	2	1000GENOMES,SSMP,	2	A,T,	22.000000,2274.000: Input/output error
ls: cannot access /home/brb/.avfs//home/brb/Downloads/Homo_sapiens_UCSC_hg19.tar.gz#/chr12	25482890	rs544684287	G	A	0	.	molType=genomic;class=single
chr12	25482914	rs558575390	T	G	0	.	m: Input/output error
000,?0.999500,0.000500,??797?chr3?27877637?27877638?rs1478557?0?+?G?G?A
4?rs555100828?0?+?T?T?C
76?chr2?103777623?103777624?rs181283085?0?+?A?A?A
chr12?25482890?rs544684287?G?A?0?.?molType=genomic;class=single?chr12?25482914?rs558575390?T?G?0?.?m
G?A
Homo_sapiens
nown?exact?1?SingleClassTriAllelic,InconsistentAlleles?2?1000GENOMES,SSMP,?2?A,T,?22.000000,2274.000
README.txt
T?C

real	25m51.340s
user	0m0.000s
sys	0m0.003s
brb@T3600 ~/Downloads $ ls ~/.avfs/$PWD/annovar.latest.tar.gz#/
annovar

For archivemount, see Cool User File Systems: ArchiveMount 

archivemount files.tgz mntDir
umount mntDir

Show folder size for one level only

du --max-depth=1 -h

The graphical tool is called Disk Usage Analyze which is already available on Ubuntu.

Soft link

ln -s /full/path/of/original/file /full/path/of/soft/link/file

Self-hosted servers

  • https://github.com/Kickball/awesome-selfhosted This is a list of Free Software network services and web applications which can be hosted locally. Selfhosting is the process of locally hosting and managing applications instead of renting from SaaS providers.
  • Sovereign: A set of Ansible playbooks to build and maintain your own private cloud: email, calendar, contacts, file sync, IRC bouncer, VPN, and more.

DNS

setup

DNSmasq (DNS + DHCP server)

Local forwarding server

dnsmasq program is running on my Ubuntu and Linux/Mint machines.

See nameserver 127.0.1.1 in resolv.conf won't go away! 

$ ps -ef | grep -i dnsmasq

$ sudo netstat -anp | grep -i dnsmasq

Email server

POP, IMAP and Exchange

POP works by contacting your email service and downloading all of your new messages from it. Once they are downloaded onto your PC or Mac, they are deleted from the email service.

IMAP allows you to access your email wherever you are, from any device. When you read an email message using IMAP, you aren't actually downloading or storing it on your computer; instead, you're reading it from the email service. As a result, you can check your email from different devices, anywhere in the world: your phone, a computer, a friend's computer.

Exchange offers the same syncing capabilities as IMAP, plus much more. Exchange is a Microsoft product, giving you the ability to still use Outlook as your email service and benefit from Exchange functionality.

Configure Postfix to use Gmail as a Mail Relay

https://www.howtoforge.com/tutorial/configure-postfix-to-use-gmail-as-a-mail-relay/

How to Build an Email Server on Ubuntu Linux

sendemail package

sendEmail -f $USERNAME -s smtp.gmail.com:587 \
            -xu $USERNAME -xp $PASSWORD -t $USERNAME \
            -o tls=yes \
            -u "Web page changed" \
            -m "Visit it at $URL"

sendEmail -f [email protected] -t [email protected] \
            -s smtp.gmail.com:587 -xu [email protected] -xp YOURPASSWORD \
            -o tls=yes \
            -u "Hello from sendEmail" \
            -m "How are you? I'm testing sendEmail from the command line."

Kolab

Install and Configure Kolab Groupware on Ubuntu 16.04 LTS

Backup

  1. rdiff-backup. See
  2. Use a Dropbox folder.

If we don't want to install dropbox software, we can install bash dropbox uploader: http://www.andreafabrizi.it/?dropbox_uploader OR https://github.com/andreafabrizi/Dropbox-Uploader

It allows to upload/download/delete/list files and show info of user. The version I am using is v0.9.7. It works on linux, Windows/Cygwin, Raspberry Pi, etc.

I install it under ~/Downloads/andreafabrizi-Dropbox-Uploader-cdc2466 directory

Instruction with screenshots: http://www.jobnix.in/dropbox-command-line-interface-cli-client/

Sample usages: 

./dropbox_uploader.sh  list /
./dropbox_uploader.sh  upload ~/Desktop/ConfigurateNote.txt

Github, Bitbucket, Gitlab

We can use these git services to get real-time data (eg temperature, IP, etc).

Running a cron job as a user

Some examples

  • A specific time
MIN HOUR DOM MON DOW CMD
30 08 10 06 * /home/ramesh/full-backup 
#   30 – 30th Minute
#   08 – 08 AM
#   10 – 10th Day
#   06 – 6th Month (June)
#   * – Every day of the week
  • Twice a day
00 11,16 * * * /home/ramesh/bin/incremental-backup 
#    00 – 0th Minute (Top of the hour)
#    11,16 – 11 AM and 4 PM
#    * – Every day
#    * – Every month
#    * – Every day of the week
  • Every 10 minutes
*/10 * * * * /home/ramesh/check-disk-space

crontab

  • crontab cron-file-winter; crontab -l

Make sure the .sh file gives a complete path. For example, 

#!/bin/sh
R --vanilla  < arraytoolsip.R

does not work in cron job although it works perfect when we manually run it from the right path. The sh file should be 

#!/bin/sh
R --vanilla  < $HOME/Dropbox/scripts/arraytoolsip.R

To disable everything on crontab -l, run crontab -e then comment out each line you don't want to run with #. OR run crontab -r to empty the current crontab.

PATH and Shell

Cron knows nothing about your shell; it is started by the system, so it has a minimal environment. If you want anything, you need to have that brought in yourself. For example, to use 'ifconfig' command, I need to give it a complete path in my script file. 

$ cat syncIP 
/sbin/ifconfig eth0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}'

and the cron job 

06 15 * * * /home/MYUSERNAME/Ubuntu\ One/syncIP > $HOME/Ubuntu\ One/ip.txt 2>&1

See here on how to add environment variable into cron environment.

Disable mail alert

If something went wrong with executing a cron job, cron will output a message "You have new mail in /var/mail/$USER". You can open this file using a text editor. To disable this alert, run 'crontab -e (see this post 

0 1 5 10 * /path/to/script.sh >/dev/null 2>&1
# OR
0 1 5 10 * /path/to/script.sh > /dev/null

Running crontab as root

Use sudo crontab -e to edit. After saving it, no need to initialize it. Use sudo crontab -l to list the cron job.

Mount drive

/etc/fstab and blkid

sudo blkid

and the result should be compared with 

sudo fdisk -l
  • Run mount -a to remount /etc/fstab without reboot, except the partitions with noauto option.

The following example shows a problem (as found from the output of df command) with </etc/fstab> where we use /dev/sdXY instead of UUID for specifying hard disks. 

$ sudo blkid
/dev/sda1: LABEL="WD640" UUID="d3a0a512-bf96-4199-9674-f410f22f0a92" TYPE="ext4"
/dev/sdb1: UUID="afaa4bde-1172-4c54-8b0a-a324ad855355" TYPE="ext4"
/dev/sdb5: UUID="fb2a4ada-d80a-4e23-b4a2-67376b8b7e72" TYPE="swap"

$ sudo fdisk -l
Disk /dev/sda: 640.1 GB, 640135028736 bytes
...
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048  1250263039   625130496   83  Linux

Disk /dev/sdb: 640.1 GB, 640135028736 bytes
...
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *        2048  1217761279   608879616   83  Linux
/dev/sdb2      1217763326  1250263039    16249857    5  Extended
/dev/sdb5      1217763328  1250263039    16249856   82  Linux swap / Solaris

$ cat /etc/fstab
proc            /proc                          proc    nodev,noexec,nosuid 0 0
UUID=afaa4bde-1172-4c54-8b0a-a324ad855355 /    ext4    errors=remount-ro   0 1
UUID=fb2a4ada-d80a-4e23-b4a2-67376b8b7e72 none swap    sw                  0 0
/dev/sdb1       /mnt/WD640                     ext4    rw,nosuid,nodev     0 2

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       572G  413G  130G  77% /
...
/dev/sdb1       572G  413G  130G  77% /mnt/WD640

To fix the error here, modify the line starting /dev/sdb1 in /etc/fstab and replace it with the UUID. Then run sudo umount /mnt/WD640 and sudo mount -a. Done! 

~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       572G  413G  130G  77% /
...
/dev/sda1       587G  283G  275G  51% /mnt/WD640

usb drive

Run the following to confirm the USB device is detected. 

sudo fdisk -l
# OR
dmesg | grep -i "SCSI"

Now suppose the usb device is found in dev/sdb1. 

sudo mkdir /mnt/usb
sudo mount -t vfat -o rw,users /dev/sdb1 /mnt/usb

The above mount command assumes the usb drive has Windows vfat partition and users give non-root users the ability to unmount the drive. If the USB drive is partitioned linux ext2/3, we can merely run mount command as 

sudo mount /dev/sdb1 /mnt/usb

At the end, run umount command like 

sudo umount /mnt/usb

To make the mounting automatically, edit the file /etc/fstab. 

/dev/sdb1       /mnt/usb           vfat    defaults        0       0 
/dev/sdb1       /mnt/usb           ext3    defaults        0       0

and run 

sudo mount -a

Mount an iso file

sudo mkdir -p /mnt/mount_point # create a mount point
sudo mount -o loop /home/user/disk.iso /mnt/mount_point
mount  # verify

Mount remote Windows share

Sharing files with Windows by using NitroShare

Simple way of Sharing files between Ubuntu 16.04 and Windows 10 by using open-source NitroShare which is based on Qt framework.

exFat - cross platform partition format

sudo apt-get install exfat-utils exfat-fuse

sudo fdisk -l 
sudo mkfs.exfat -n NAME /dev/sd**  #  NAME with whatever you want to label your drive

This should delivery a working exfat file system (read and write support, but not formatting the drives with exfat via Gnome Disks and GParted).

NTFS usb drive in xubuntu

http://xflinux.blogspot.com/2011/01/mount-ntfs-volumes-automatically-in.html 

sudo apt-get install ntfs-config

Now go to Applications>> System>> Ntfs Configuration Tool

Expand the "Advanced Configuration" and select all those partitions you want to be auto mounted and writable( The tool will detect all partitions at its startup).

Make sure the " Enable write support for internal devices" option is selected. Now click Close.

Many drives, one folder

Partition tables

Partition Tables and the Dangers of Editing Them

Recommended partition schemes

How do I send an already-running process into the background

http://stackoverflow.com/questions/625409/how-do-i-put-an-already-running-process-under-nohup

  1. 'Ctrl+Z' to stop (pause) the program and get back to the shell.
  2. bg to run it in the background.
  3. disown -h [job-spec] where [job-spec] is the job number (like %1 for the first running job; find about your number with the jobs command) so that the job isn't killed when the terminal closes.

run commands in a background and allow log off

nohup /path/to/script >output 2>&1 &

Or to disable output and be more safe. It also explains the concept of file descriptor/fd in Unix. 

nohup command </dev/null >/dev/null 2>&1 &

See also Anonymous named pipe.

Notepadqq - Notepad++-like editor

Notepadqq. It is written using Qt. It does not have printing function:(

Note apps that can sync

Top 8 Notepad Apps for Linux That You Can Sync. Some are compatible with Evernote.

Markdown

Preview markdown/view markdown offline

sudo pip install grip
grip readme.md
# title 1
![screenshot](myfile.png?raw=true)
  • Chrome markdown preview plus extension does not show images from github.

Markdown editor

Text editor with navigation

7 Best Note-Taking Tools for Programmers

http://www.makeuseof.com/tag/best-note-taking-tools-programmers/

RStudio

Code -> Insert Section

We can create different levels of sections.

Netbeans and navigator

On ODroid (ARM works too!) Ubuntu 16.04 

# Note: OpenJDK 8 will not work
# We have to install Oracle Java
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

sudo apt-get install netbeans # version 8.1 in my case

See screenshots

Note:

  • Netbeans has a built-in support for HTML/XML files. XML has an advantage over HTML since HTML cannot have any tags you want.
  • We need to download a plugin for markdown file support. Go to Tools -> Plugins. In the 'Settings' tab make sure the 3 items are checked. Go to 'Available plugins' tab

search 'markdown'. Install 'Markdown support'. It works on Netbeans 8.0 on x64 Ubuntu 14.04 and Netbeans 8.1 on my ARM Ubuntu 16.04.

  • For Markdown or XML, the comment syntax can be found here.
  • For some reason, the order of headlines on the navigator pane is not the same as they appeared on the file. So it is better to use XML file format.
  • My hack on Netbeans options (change to use a dark color on background).
    • Profile: NetBeans
    • Syntax. Default: Foreground=White, Background=Dark Gray. Comment: Foreground=cyan. Keyword: Foreground=Orange.
    • Highlighting. Highlight Caret Row: Foreground=Dark Gray. Background=Pink.
  • For choosing colors, go to Google: rgb to hex
  • For some reason, it makes my graphical Mint desktop unstable. I have to use Ctrl + Alt + F1 and Ctrl + Alt + F8 to fix it temporarily. Deal breaker!

Netbeans.png NetbeansMarkdown.png

IntelliJ IDEA

It requires JDK. The community version is free. Download the tarball. Extract it and run bin/idea.sh. It even identifies a mismatch in my XML documentation that netbeans does not find.

  • Viewing Structure of a Source File or Alt + 7
  • Open a terminal at the bottom; Alt + F12
  • SOLARIZED color. Copy icls file to ~/.IdeaIC2016.2/config/colors directory. Restart Intellij IDEA. Go to File -> Settings -> Editor -> Colors & Fonts -> Font.
  • To deactivate spelling checking: Ctrl + Alt + s -> Editor -> Inspections -> Spelling -> Typo. Uncheck it.
  • We need to create a project for IntelliJ IDEA to automatically open the file we are working on. IDEA will create a hidden folder call .idea. For git repository, we shall create the .gitignore file contains
.idea/workspace.xml
.idea/misc.xml

Zim

  • Available in Linux, Windows & Mac.
  • The side panel contains a hierarchical view of the pages.
  • Right click on the page tab, we can create a new page or sub page.
  • On Ubuntu, the title bar is on the top of the desktop.
  • Auto save. Auto re-open the last session.
  • Handles several types of markup, like headings, bullet lists and of course bold, italic and highlighted. This markup is saved as wiki text so you can easily edit it with other editors.
  • Toggle notebook editable.
  • Insert image (cannot resize)
  • Plugins, e.g. Equation editor, R plot,
  • The default folder for storing the notes is ~/Notebook. Each page has its own file in storage.
~/Notebooks/Notes/
~/Notebooks/Notes/notebook.zim
~/Notebooks/Notes/Home.txt

If we create a 2nd page called 'Home2' with sub pages 'Subpage1' and 'subpage2' we will have 

~/Notebooks/Notes/Home2.txt
~/Notebooks/Notes/Home2
~/Notebooks/Notes/Home2/subpage1.txt
~/Notebooks/Notes/Home2/subpage2.txt

Cherrytree - a hierarchical note taking application

featuring rich text and syntax highlighting, storing data in a single xml or sqlite file.

Vim

with the Tagbar plugin.

The instruction works for cpp file.

Unfortunately xml files are not supported from my testing. See its wiki for supported filetypes.

Lime

Maybe

nano editor

Permission denied and sudoedit command

When I run 'nano tmp', I got a message: Error reading /home/odroid/.nano/search_history: Permission denied. Press Enter to continue. 

odroid@odroid:~$ ls -ld /home/odroid/.nano
drwxr-xr-x 2 root root 4096 Feb 12 08:01 /home/odroid/.nano
odroid@odroid:~$ ls -l /home/odroid/.nano
total 4
-rw------- 1 root root 15 Feb 12 08:01 search_history

A simple solution is 

sudo chown -R odroid:odroid /home/odroid/.nano  # note '-R' has to be capital

This seems to be a bug in nano after we use 'sudo nano [file]' (eg 'sudo nano /etc/chromium-browser/default') when the 'nano' program has not been run before.

The bottom line is use something like below for editing system files 

EDITOR=nano sudoedit /etc/chromium-browser/default

show line number/cursor position

Use the -c option for cursor position. 

nano -c FILENAME

Enable soft line wrapping

nano -$ FILENAME
nano --softwrap FILENAME

syntax highlight

Add a syntax highlight support for some languages 

$ ls /usr/share/nano/
asm.nanorc    html.nanorc  mutt.nanorc    perl.nanorc    ruby.nanorc
c.nanorc      java.nanorc  nanorc.nanorc  pov.nanorc     sh.nanorc
groff.nanorc  man.nanorc   patch.nanorc   python.nanorc  tex.nanorc
$ cat ~/.nanorc
include /usr/share/nano/sh.nanorc
include /usr/share/nano/c.nanorc
include ~/r.nanorc

R syntax highlight file r.nanorc. Note that I have to comment out line 29 starting with 'header'. A personal copy is saved in github.

To disable syntax highlight (useful if we use a terminal app on an Android ebook reader such as Boox), add -Ynone parameter.

vi keys

  • ctrl +f: page down. Ctrl +b: page up.
  • ^: beginning of a line. $: end of a line.
  • command mode : this is the default when you run vi. Hit Esc key to return to the command mode. Command mode is right for moving about a file, copying or deleting a line, saving a file, etc.
  • Insert/edit mode : hit "i" (insert text before cursor position) or "a" (add text after cursor position) to enter the edit mode. The screen will show the text -- INSERT -- on the last line of the vi editor.
  • Command mode/leave edit mode: "ESC". In this mode, you can search, navigate or enter an insert model.
  • Last line mode: Press ':', vi editor will show ':' on the last line. If we continue to type 'q[Enter]' we will quit vi. If we continue to type 'wq', it will write the file and quits.
  • delete entire line: "dd"
  • undo: "u"
  • search forward: "/pattern" (case sensitive). Hit "n" to repeat search.
  • search backward: "?pattern" (case sensitive). Hit "n" to repeat search.
  • save: ":w"
  • quit: ":q"
  • quit without saving: ":q!"
  • save and quit: ":wq"

Some helps

Cloud

http://slidedecks.wilmoore.com/2012-confoo/diy-private-cloud-using-virtualBox-and-chef/#66

http://www.datacentermap.com/blog/cloud-software-389.html

Commercial cloud services

  • DigitalOcean - simple pricing. One droplet = one server.
  • Amazon EC2
  • Google cloud
  • Microsoft Azure

Manage all your cloud storage

http://www.makeuseof.com/tag/chrome-extensions-you-need-manage-cloud-storage/

Comparison of Linux VPS providers

https://www.ghacks.net/2017/06/15/a-comparison-of-linux-vps-providers-for-beginner-hosting/

U-boot

http://www.denx.de/wiki/U-Boot

Pandora linux client

COW (copy on write) file system

The cow filesystem was found on xubuntu live CD. See http://en.wikipedia.org/wiki/Copy-on-write

How To Protect Your Server Against the Dirty COW Linux Vulnerability (10/21/2016)

Apache redirection

http://cran.r-project.org/mirror-howto.html

Redirect a Website URL from One Server to Different Server in Apache

Important linux directories

  • /bin - executables used by the base system
  • /boot
  • /dev
  • /etc - configuration files
  • /media
  • /mnt
  • /opt - optional application packages
  • /proc - process information only
  • /sbin - critical executables for running the system, but should be used by superuser
  • /usr - non-critical files. Inside is /usr/bin, which contains most of the libraries used by apps.
  • /var - variable data such as databases, mails spools and system logs.

Difference of /bin, /sbin, /usr/local/bin, ...

  • /bin : For essential binaries; e.g. bash, cat, ls.
  • /sbin : is similar to /bin but for scripts with superuser (root) privileges required; e.g. shutdown command is located here. Local users have to use sudo to run binaries here.
  • /usr/bin : Same as first, but for general system-wide & non-essential binaries; e.g. grep, zip, docker, etc.
  • /usr/sbin : Same as above, but for scripts with superuser (root) privileges required.
  • /usr/local/bin or /usr/local/sbin for system-wide available (personal) scripts.

If you want to create your own scripts and make them available to all users, you’re pretty safe adding them to /usr/local/bin. Or to add my scripts to my local bin (~/bin) and then I create a symbolic link in /usr/local/bin to the commands I want to make public. As a result, I can manage all my scripts from the same directory but still make some of them publicly available since /usr/local/bin is added to $PATH. See this post.

DHCP lease time

On Windows, it is 8 days.

Get a New IP Address

dhclient -r   #  release your IP Address

dhclient   #  get your DHCP to issue you a new IP Address based on how it’s been configured.

Check a file's encoding

file -bi myfile

For example, 

file -bi Downloads/hmv_.rc 
# text/x-c++; charset=utf-16le

Know you system using the command line

System monitor tools

Git and Github

Check out the Github page.

Bitbucket (free for 5 users)

Bitbucket 101

  • Unlimited private repos
  • Code reviews
  • JIRA integration
  • REST API
  • Custom domains

See this post to know how to fix the problem of unknown author. In short, when I uncheck "Use global user setting" from Repository-> Repository Settings -> Advanced does the commit author change as expected.

See here for a list of Android apps related to bitbucket.

Image

ImageMagick 入门：使用命令行来编辑图片

Convert a color image to black and white

https://linux.cn/article-8851-1.html

Convert an image file to a different format (eg icon)

Using the imagemagic program. 

convert winamp-ncrow.png -resize 32x32 winamp-ncrow.ico

4 Ways to Batch Convert Your PNG to JPG and Vice-Versa

Rotate an image

http://www.imagemagick.org/script/command-line-options.php#rotate 

convert winamp-ncrow.png -rotate 45 winamp-ncrow2.png  # 45 degrees

One problem with this simple approach is the picture size (not image file) changed (become smaller) if the degree is not one of 90,180 or 270.

Create an animated gif file

Use the script here. See the last example on here. The rotation speed looks good too! Just change the source image file in the script. 

#!/bin/sh
#
# Create a rotating figure using Distort SRT transformations
#
command='convert -delay 10 koala.gif -virtual-pixel white'

for i in `seq 5 5 360`; do
  command="$command \\( -clone 0 -distort SRT $i \\)"
done

command="$command -delete 0 -loop 0 animate_distort_rot.gif"

eval $command

chmod 644 animate_distort_rot.gif

Tool to convert a sequence of numbered PNG files to an animated GIF?. Convert command line option. The option '-loop 0' means repeats infinitely and '-delay 200' means 2 seconds delay between each frame. 

convert -delay 200 -loop 0 file_1.png file_2.png file_3.png animated.gif

Edit gif file

gifsicle package

Replace transparency in PNG images with white background

http://stackoverflow.com/questions/2322750/replace-transparency-in-png-images-with-white-background 

convert image.png -background white -alpha remove white.png

Remove GPS metadata from jpg files - exiftool

sudo apt-get install libimage-exiftool-perl
exiftool -gps:all= -xmp:geotag= image.jpg

The image file will be updated. To check the current metadata, use 

exiftool image.jpg

Note that the above command only remove gps information. The other information like date/time of creation, camera model are not changed.

Edit Svg image

Animated gif

GIMP

login shell (.bash_profile) vs interactive shell (.bashrc)

8 http://serverfault.com/questions/8882/what-is-the-difference-between-a-login-and-an-interactive-bash-shell

  1. login shell - non desktop environment. ~/.bash_profile is sourced for the bash shell.
  2. interactive shell - Ctrl+Alt+t to open a terminal from a graphical mode (desktop environment). ~/.bashrc is source. We usually edit ~/.bashrc to set up the environment to include fancy prompt, set aliases, set history options, or define custom shell functions.

export environment variables

  1. Both a login shell and an interactive one. SSH (Putty) to connect to a remote machine.
  2. When a shell runs a script or a command passed on its command line, it's a non-interactive, non-login shell.

History of commands

history command with date and time

Running the following code once and history will give date and time the next time you issue the history command. 

echo 'export HISTTIMEFORMAT="%d/%m/%y %T "' >> ~/.bashrc

Note that the original post asks to write the line to ~/.bash_profile but this is not working in the desktop environment.

Bang bang - Run a command/Fetch parameters from previous history

  1. ^P: Move up through the command history list one command at a time.
  2. ^N: Move down through the command history list one command at a time.
  3. !!: Run the previous command.
  4. !n: Run command number n
  5. !string: Run most recent command starting with characters in string
  6. !?string: Run most recent command containing characters that match string
  7. !!*: Fetch parameters from last command

For example, 

!-1
!4

Increase history limit

http://unix.stackexchange.com/questions/17574/is-there-a-maximum-size-to-the-bash-history-file

Listen to HiChannel internet radio

Use Radio Tray

I use it to listen m3u file (VLC also supports it too).

Web Analytics Reporting Tools

Painting software

  • Krita - professional painting program made by artists that want to see affordable art tools for everyone.
  • Pinta. It can be install by apt-get command. It works just line Window's paint. Ctr + v to paste an image and save to a file. To crop an image, click the selection tool (1st one) on the left hand side, then select a rectangle. Now click 'Image' > 'Crop to Selection' to finish.
  • mtPaint. It is included in Odroid - xu4 - Lubuntu 14.04. To crop an image, just select an area and click Image > Crop. It can be used to take a screenshot from the desktop by using either the application or through the command line (mtpaint -s). It will then display the screenshot in the application if you use the command line.
  • MyPaint

Take a screenshot (and edit them)

See Take screenshots.

Calibre - Read ebook in epub format

$ sudo -v && wget -nv -O- https://github.com/kovidgoyal/calibre/raw/master/setup/linux-installer.py | \
  sudo python -c "import sys; main=lambda:sys.stderr.write('Download failed\n'); exec(sys.stdin.read()); main()"

2014-03-19 15:54:28 URL:https://raw.github.com/kovidgoyal/calibre/master/setup/linux-installer.py [25423/25423] -> "-" [1]
Installing to /opt/calibre
Downloading tarball signature securely...
Will download and install calibre-1.28.0-x86_64.tar.bz2 
                                Downloading calibre-1.28.0-x86_64.tar.bz2                                
100% [===============================================================================================]
                                                                                                         Downloaded 63255897 bytes 
Checking downloaded file integrity... 
Extracting files to /opt/calibre ...
Extracting application files... 
Creating symlinks...
	Symlinking /opt/calibre/fetch-ebook-metadata to /usr/bin/fetch-ebook-metadata
	Symlinking /opt/calibre/lrf2lrs to /usr/bin/lrf2lrs
	Symlinking /opt/calibre/ebook-convert to /usr/bin/ebook-convert
	Symlinking /opt/calibre/ebook-meta to /usr/bin/ebook-meta
	Symlinking /opt/calibre/ebook-edit to /usr/bin/ebook-edit
	Symlinking /opt/calibre/lrfviewer to /usr/bin/lrfviewer
	Symlinking /opt/calibre/calibre to /usr/bin/calibre
	Symlinking /opt/calibre/markdown-calibre to /usr/bin/markdown-calibre
	Symlinking /opt/calibre/calibre-debug to /usr/bin/calibre-debug
	Symlinking /opt/calibre/calibre-parallel to /usr/bin/calibre-parallel
	Symlinking /opt/calibre/web2disk to /usr/bin/web2disk
	Symlinking /opt/calibre/calibre-server to /usr/bin/calibre-server
	Symlinking /opt/calibre/calibre-customize to /usr/bin/calibre-customize
	Symlinking /opt/calibre/ebook-polish to /usr/bin/ebook-polish
	Symlinking /opt/calibre/ebook-viewer to /usr/bin/ebook-viewer
	Symlinking /opt/calibre/calibre-smtp to /usr/bin/calibre-smtp
	Symlinking /opt/calibre/lrs2lrf to /usr/bin/lrs2lrf
	Symlinking /opt/calibre/ebook-device to /usr/bin/ebook-device
	Symlinking /opt/calibre/calibredb to /usr/bin/calibredb
Setting up command-line completion...
Installing bash completion to /etc/bash_completion.d/calibre
Setting up desktop integration...
Creating un-installer: /usr/bin/calibre-uninstall
Run "calibre" to start calibre

Create ebooks

https://itsfoss.com/create-ebook-calibre-linux

Fetch News

Several built-in sources like TIME, US NEWS report do not work. It'll show 'http error 404 not found'. The self added RSS feed method (see below) does work!

GNU screen

How to do it...

  • Run screen command first (run sudo apt-get install screen if necessary). You are now inside of a window within screen. This functions just like a normal shell except for a few special characters.
  • Create screen windows: Ctrl + a, then c. To close a screen window: exit. Once you close all screen windows, you shall see a message [screen is terminating] on the terminal.
  • View a list of open windows: Ctrl + a, then ".
  • Switch between windows: Ctrl + a and n for the next window and Ctrl +a and p for the previous window.
  • Attaching to and detaching screens: To detach (save) from the current screen session, Ctrl +a, and d (these keyboard shortcuts won't affect current execution). This will drop you into your shell. This is useful when you need to run a time-consuming job or your connection is dropped. To attach to an existing screen, use:
screen -r -d
  • Split screen:
    • To split the screen horizontally, Ctrl +a and S (capital).
    • To unsplit the screen, Ctrl +a and Q (capital).
    • To switch from one to the other: Ctrl +a and TAB.
    • Note: After splitting, you need to go into the new region and start a new session via Ctrl + a then c before you can use that area.

Display/screen

Turn off/on your monitor via command line

xset dpms force off # Press any key to turn it on 
xset dpms force on
xset -q # check the status of the X server settings

If we want to turn off/on the screen via ssh, add 

export DISPLAY=:0.0

first before calling the xset command, or use '-display' argument 

xset -display :0.0 dpms force off 
xset -display :0.0 dpms force on

Add new screen/display resolutions

xrandr | grep maximum
gtf 800 480 59.9  # give some output used in the following line
xrandr --newmode "800x480_59.90" 29.53 800 816 896 992 480 481 484 497 -HSync +Vsync
xrandr --addmode "DISP3 BG" 800x480_59.90
xrandr --output "DISP3 BG" --mode 800x480_59.90

I cannot find the file /etc/X11/xorg.conf in my UDOObuntu 2 beta 2. It seems this file does not exist anymore. See this post about how to re-create it.

export DISPLAY

If we want to run a GUI app on a remote computer (such as Raspberry Pi/Beaglebone Black) and show the GUI app on the remote computer's screen using ssh, we can issue the following command before running the app. 

export DISPLAY=:0.0

LVM Demystified

http://www.linuxjournal.com/content/lvm-demystified

groupadd, chgrp, usermod, ACL (access control lists)

sudo mkdir -p /var/www/reports/
sudo groupadd project 

sudo usermod -a -G project tecmint 
sudo chgrp -R project /var/www/reports/
sudo chmod -R 2775 /var/www/reports/

create more system users and add them to the directory group as follows: 

sudo useradd -m -c "Aaron" -s/bin/bash -G project aaron
sudo useradd -m -c "John" -s/bin/bash -G project john
sudo useradd -m -c "Ravi" -s/bin/bash -G project ravi

sudo mkdir -p /var/www/reports/aaron_reports
sudo mkdir -p /var/www/reports/john_reports
sudo mkdir -p /var/www/reports/ravi_reports

Shared library management

http://www.cyberciti.biz/tips/linux-shared-library-management.html

  1. ldconfig : Updates the necessary links for the run time link bindings.
  2. ldd : Tells what libraries a given program needs to run.
  3. ltrace : A library call tracer.
  4. ld.so/ld-linux.so: Dynamic linker/loader.

Install binary software using sudo

One example (Calibre) is like 

sudo -v && wget -nv -O- https://raw.githubusercontent.com/kovidgoyal/calibre/master/setup/linux-installer.py | \
sudo python -c "import sys; main=lambda:sys.stderr.write('Download failed\n'); exec(sys.stdin.read()); main()"

Note that in wget the option "-O-" means writing to standard output (so the file from the URL is NOT written to the disk) and "-nv" means no verbose.

If the option "-O-" is not used, we'd better to use "-N" option in wget to overwrite an existing file.

See the Logging and Download options in wget's manual. 

       -O file
       --output-document=file
           The documents will not be written to the appropriate files, but all
           will be concatenated together and written to file.  If - is used as
           file, documents will be printed to standard output, disabling link
           conversion.  (Use ./- to print to a file literally named -.)

Log files

DigitalOcean 

ls -lt /var/log

uptime command

uptime
watch -n 60 uptime

htop command

Linux command similar to top to show hard disk activity

Use iotop. On ubuntu, we can use sudo apt-get install to install it. Use sudo iotop to launch it. 

sudo apt-get install iotop

sudo iotop -o -u $USER

Install Apache HBase

Follow the Quick Start to downloaded hbase tar ball. Suppose we save the tar ball under ~/Downloads folder and extract it in the same directory. We shall edit conf/hbase-site.xml file according to their instruction. The following is my case. 

$ tar xzvf hbase-0.98.3-hadoop2-bin.tar.gz 
$ cd hbase-0.98.3-hadoop2/
$ cat conf/hbase-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>file:///home/brb/Downloads/hbase-0.98.3-hadoop2/hbase</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/home/brb/Downloads/hbase-0.98.3-hadoop2/zookeeper</value>
  </property>
</configuration>

Before we follow the getting started guide to launch HBase, we shall make sure JAVA_HOME environment variable is created. 

$ ls /usr/lib/java
$ export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-amd64

Note that the last line may be replaced by 

export JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:bin/javac::")

Then we can launch HBase, 

$ ./bin/start-hbase.sh 

starting master, logging to /home/brb/Downloads/hbase-0.98.3-hadoop2/bin/../logs/hbase-brb-master-brb-P45T-A.out
brb@brb-P45T-A:~/Downloads/hbase-0.98.3-hadoop2$ ./bin/hbase shell
2014-07-06 09:51:34,621 INFO  [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.98.3-hadoop2, rd5e65a9144e315bb0a964e7730871af32f5018d5, Sat May 31 19:56:09 PDT 2014

hbase(main):001:0> create 'test', 'cf'
2014-07-06 09:51:49,510 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
0 row(s) in 2.0770 seconds

=> Hbase::Table - test
hbase(main):002:0> list 'test'
TABLE                                                                           
test                                                                            
1 row(s) in 0.0530 seconds

=> ["test"]
hbase(main):003:0> exit

brb@brb-P45T-A:~/Downloads/hbase-0.98.3-hadoop2$ ./bin/hbase 

shell2014-07-06 09:53:37,480 INFO  [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.98.3-hadoop2, rd5e65a9144e315bb0a964e7730871af32f5018d5, Sat May 31 19:56:09 PDT 2014

hbase(main):001:0> list 'test'
TABLE                                                                           
2014-07-06 09:53:44,373 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
test                                                                            
1 row(s) in 1.4800 seconds

=> ["test"]
hbase(main):002:0> put 'test', 'row1', 'cf:a', 'value1'
0 row(s) in 0.4460 seconds

hbase(main):003:0> put 'test', 'row2', 'cf:b', 'value2'
0 row(s) in 0.0140 seconds

hbase(main):004:0> put 'test', 'row3', 'cf:c', 'value3'
0 row(s) in 0.0050 seconds

hbase(main):005:0> scan 'test'
ROW                   COLUMN+CELL                                               
 row1                 column=cf:a, timestamp=1404654837532, value=value1        
 row2                 column=cf:b, timestamp=1404654856976, value=value2        
 row3                 column=cf:c, timestamp=1404654866298, value=value3        
3 row(s) in 0.0560 seconds

hbase(main):006:0> get 'test', 'row1'
COLUMN                CELL                                                      
 cf:a                 timestamp=1404654837532, value=value1                     
1 row(s) in 0.0280 seconds

hbase(main):007:0> disable 'test'
0 row(s) in 1.6050 seconds

hbase(main):008:0> drop 'test'
0 row(s) in 0.2290 seconds

hbase(main):009:0> exit
brb@brb-P45T-A:~/Downloads/hbase-0.98.3-hadoop2$

curl vs wget

sudo apt-get install curl

For example, the Download link at the National Geographic Travel Photo Contest 2014 works for curl but not wget. I can use curl with -o option but wget with -o will not work in this case. Note with curl, we can also use the -O (capital O) option which will write output to a local file named like the remote file. 

curl \
 http://travel.nationalgeographic.com/u/TvyamNb-BivtNwcoxtkc5xGBuGkIMh_nj4UJHQKuoXEsSpOVjL0t9P0vY7CvlbxSYeJUAZrEdZUAnSJk2-sJd-XIwQ_nYA/ \
 -o owl.jpg

Should I Use Curl Or Wget? and curl vs Wget

  • The main benefit of using the wget command is that it can be used to recursively download files.
  • The curl command lets you use wildcards to specify the URLs you wish to retrieve. And curl supports more protocols than wget (HTTP, HTTPS, FTP) does.
  • The wget command can recover when a download fails whereas the curl command cannot.

Actually curl supports continuous downloading too. But not all FTP connection supports continuous downloading. The following examples show it is possible to use the continuous downloading option in wget/curl for downloading file from ncbi FTP but not from illumina FTP. 

$ wget -c ftp://igenome:[email protected]/Drosophila_melanogaster/Ensembl/BDGP6/Drosophila_melanogaster_Ensembl_BDGP6.tar.gz
--2017-04-13 10:46:16--  ftp://igenome:*password*@ussd-ftp.illumina.com/Drosophila_melanogaster/Ensembl/BDGP6/Drosophila_melanogaster_Ensembl_BDGP6.tar.gz
           => ‘Drosophila_melanogaster_Ensembl_BDGP6.tar.gz’
Resolving ussd-ftp.illumina.com (ussd-ftp.illumina.com)... 66.192.10.36
Connecting to ussd-ftp.illumina.com (ussd-ftp.illumina.com)|66.192.10.36|:21... connected.
Logging in as igenome ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /Drosophila_melanogaster/Ensembl/BDGP6 ... done.
==> SIZE Drosophila_melanogaster_Ensembl_BDGP6.tar.gz ... 762893718
==> PASV ... done.    ==> REST 1706053 ... 
REST failed, starting from scratch.
 
==> RETR Drosophila_melanogaster_Ensembl_BDGP6.tar.gz ... done.
Length: 762893718 (728M), 761187665 (726M) remaining (unauthoritative)
 
 0% [                                                                                                                   ] 374,832     79.7KB/s  eta 2h 35m ^C
 
$ curl -L -O -C - ftp://igenome:[email protected]/Drosophila_melanogaster/Ensembl/BDGP6/Drosophila_melanogaster_Ensembl_BDGP6.tar.gz
** Resuming transfer from byte position 1706053
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  727M    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
curl: (31) Couldn't use REST

$ wget -c ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b147_GRCh37p13/VCF/common_all_20160601.vcf.gz
--2017-04-13 10:52:02--  ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b147_GRCh37p13/VCF/common_all_20160601.vcf.gz
           => ‘common_all_20160601.vcf.gz’
Resolving ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)... 2607:f220:41e:250::7, 130.14.250.10
Connecting to ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)|2607:f220:41e:250::7|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /snp/organisms/human_9606_b147_GRCh37p13/VCF ... done.
==> SIZE common_all_20160601.vcf.gz ... 1023469198
==> EPSV ... done.    ==> RETR common_all_20160601.vcf.gz ... done.
Length: 1023469198 (976M) (unauthoritative)
 
24% [===========================>                                                                                       ] 255,800,120 55.2MB/s  eta 15s    ^C
 
$ wget -c ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b147_GRCh37p13/VCF/common_all_20160601.vcf.gz
--2017-04-13 10:52:11--  ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b147_GRCh37p13/VCF/common_all_20160601.vcf.gz
           => ‘common_all_20160601.vcf.gz’
Resolving ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)... 2607:f220:41e:250::7, 130.14.250.10
Connecting to ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)|2607:f220:41e:250::7|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /snp/organisms/human_9606_b147_GRCh37p13/VCF ... done.
==> SIZE common_all_20160601.vcf.gz ... 1023469198
==> EPSV ... done.    ==> REST 267759996 ... done.    
==> RETR common_all_20160601.vcf.gz ... done.
Length: 1023469198 (976M), 755709202 (721M) remaining (unauthoritative)
 
47% [++++++++++++++++++++++++++++++========================>                                                            ] 491,152,032 50.6MB/s  eta 12s    ^C

$ curl -L -O -C - ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b147_GRCh37p13/VCF/common_all_20160601.vcf.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 65  976M   65  639M    0     0  83.7M      0  0:00:11  0:00:07  0:00:04 90.4M^C

curl man page, supported protocols

https://curl.haxx.se/docs/manpage.html

wget and username/password

http://www.cyberciti.biz/faq/wget-command-with-username-password/

Download and Un-tar(Extract) in One Step

If we don't want to avoid saving a temporary file, we can use one piped statement. 

curl http://download.osgeo.org/geos/geos-3.5.0.tar.bz2 | tar xvz
# OR
wget http://download.osgeo.org/geos/geos-3.5.0.tar.bz2 -O - | tar jx

See shellhacks.com. Note that the magic part of the wget option "-O -"; it will output the document to the standard output instead of a file.

Download and execute the script in one step

See Execute bash script from URL. 

curl -s http://server/path/script.sh | sudo bash /dev/stdin arg1 arg2

curl and POST request

Website performance

httpstat – A Curl Statistics Tool to Check Website Performance

wget to download a website

To download a copy of a complete web site, use the recursive option ('-r') By default it will go up to five levels deep. You can change the default level by using the '-l' option.

All files linked to in the documents are are downloaded to enable complete offline viewing ('-p' and '--convert-links' options). Instead of having the progress messages displayed on the standard output, you can save it to a log file with the -o option. 

wget -p --convert-links -r -l2 linux.about.com -o logfile
wget -p --convert-links -r -l1 https://csgillespie.github.io/efficientR # create csgillespie/efficientR

aria2 - command line downloader supports torrents and multi-connection

aria2 command examples

The -x argument helps a little bit. 

# Download a file 112MB; see https://www.archlinux.org/download/
$ time aria2c  http://mirror.jmu.edu/pub/archlinux/iso/2016.11.01/archlinux-bootstrap-2016.11.01-i686.tar.gz # 16 seconds
$ time aria2c -x10 http://mirror.jmu.edu/pub/archlinux/iso/2016.11.01/archlinux-bootstrap-2016.11.01-i686.tar.gz # 11 seconds

Axel

It can create an unlimited number of worker threads to download any kind of data. See https://www.beginnersheap.com/top-5-command-line-download-accelerators-linux/

lftp

Apply a patch to source code

For example Tophat 2.0.12 compatibility with Samtools 1.0, 

brb@brb-VirtualBox:~/Downloads$ ls
support_for_tophat_1.patch  tophat-2.0.12  tophat-2.0.12.tar.gz

brb@brb-VirtualBox:~/Downloads$ grep -r -i "check_samtools" tophat-2.0.12/
tophat-2.0.12/src/tophat.py:def check_samtools():
tophat-2.0.12/src/tophat.py:        check_samtools()

brb@brb-VirtualBox:~/Downloads$ cp support_for_tophat_1.patch tophat-2.0.12/src/
brb@brb-VirtualBox:~/Downloads$ cd tophat-2.0.12/src/
brb@brb-VirtualBox:~/Downloads/tophat-2.0.12/src$ patch tophat.py < support_for_tophat_1.patch 
patching file tophat.py
Hunk #1 succeeded at 1540 (offset 3 lines).
Hunk #2 succeeded at 1563 (offset 3 lines).
brb@brb-VirtualBox:~/Downloads/tophat-2.0.12/src$ ls

Get external IP address

https://www.iplocation.net

https://github.com/jakewmeyer/Geo (one shell script)

It seems there is no way to get the external IP address without not using external services. 

# http://unix.stackexchange.com/questions/22615/how-can-i-get-my-external-ip-address-in-a-shell-script
sudo apt-get install dnsutils
dig +short myip.opendns.com @resolver1.opendns.com
# OR
curl http://ipecho.net/plain; echo
# OR
curl ipv4.ipogre.com

The above only gives the IP. The following method gives geo information too. 

curl ipinfo.io
# OR give a specific IP (domain name does not work)
curl ipinfo.io/216.58.194.46

{
  "ip": "216.58.194.46",
  "hostname": "dfw25s12-in-f14.1e100.net",
  "city": "Mountain View",
  "region": "California",
  "country": "US",
  "loc": "37.4192,-122.0574",
  "org": "AS15169 Google Inc.",
  "postal": "94043"
}

Domain

  • WHOIS LOOKUP. For example, consider "r-pkg.org" domain,
    • The 'Sponsoring Registrar' shows who is the sponsoring registrar (eg GoDaddy.com).
    • The 'Registrant Name' shows who registered this domain.
    • Command Line Interface.
sudo apt-get install whois
whois r-pkg.org
  • IP-Lookup. For example, consider "r-pkg.org" domain.
    • The linux command line tool 'ping' shows the ip address.
    • When we use the ip address to search in the IP-Lookup, the IP owner info > 'Organization' or 'OrgName' field shows the owner of this IP (eg Digital Ocean, Inc).
    • The Domain owner info there gives the same (or less) information as WHOIS LOOKUP.
  • WhoIsHostingThis or webhostinghero (the returned result will be like Amazon, GoDaddy, CloudFlare, Github, Verizon, etc).

Subnet

IPv4

  • Class A: 255.0.0.0 or /8;
  • Class B, 255.255.0.0 or /16;
  • Class C, 255.255.255.0 or /24.

For example, in the subnet 192.168.5.0/255.255.255.0 (192.168.5.0/24) the identifier 192.168.5.0 commonly is used to refer to the entire subnet.

In the /16 subnet 192.168.0.0/255.255.0.0, which is equivalent to the address range 192.168.0.0–192.168.255.255

IPv6

ping command

ping uses the ICMP Echo Message to force a remote host to echo a packet back to the local host. If packets can travel to and from a remote host, it indicates that the two hosts can successfully communicate.

Build a home network

IP Subnet Calculator

https://www.dan.me.uk/ipsubnets?ip=10.0.0.0

  • CIDR block IP range (network - broadcast) Subnet Mask IP Quantity
  • 10.0.0.0/24 10.0.0.0 - 10.0.0.255 255.255.255.0 256
  • 10.0.0.0/16 10.0.0.0 - 10.0.255.255 255.255.0.0 65536=256^2
  • 10.0.0.0/8 10.0.0.0 - 10.255.255.255 255.0.0.0 16777216=256^3

How Use Your Router and ISP’s Modem/Router Combo In Tandem

http://www.howtogeek.com/255206/how-use-your-router-and-isps-modemrouter-combo-in-tandem/

Troubleshoot and repair network problems

http://www.linuxuser.co.uk/features/troubleshot-repair-linux-networks

Computer 1 can ping Computer 2 but not reverse

Use the route command to solve. See also this post.

Monitor network by Cacti (GUI)

Monitor network by command line

iftop

Use the interface top iftop command. On ubuntu, we need to use sudo apt-get install iftop and then run it by sudo iftop -i eth0. After that, we can press some keys to toggle options.

  • p: port
  • s: source
  • d: destination

See thegeekstuff.

vnStat

http://www.thegeekstuff.com/2011/11/vnstat-network-traffic-monitor/ 

# 1. Install vnStat
sudo apt-get install vnstat

# 2. Pick a Interface to Monitor using vnStat
vnstat -u -i eth0
vnstat --iflist
vnstatd -d # start the daemon
ps -ef | grep vnst

# 3. vnStat Basic Usage
vnstat

# 4. vnStat hours, days, months, weeks Network Data
vnstat -d
vnstat -m

# 5. Export the data to Excel or other DB
vnstat --dumpdb

# 6. Display Live Network Statistics
vnstat -l

# 7. Change the default vnstat output format
vnstat -s (--short)
vnstat --style 0

# 8. Display Top 10 Traffic Days
vnstat --top10

Network related linux commands

ifconfig - spoof the hardware address at the software level

To change the MAC address temporarily on a NIC (network interface controller), 

sudo ifconfig eth0 down
sudo ifconfig eth0 hw ether 00:11:22:33:44:55
sudo ifconfig eth0 up

And it seems there is no need to modify /etc/network/interfaces.

For wlan 

sudo ifconfig wlan0 down
sudo ifconfig wlan0 hw ether 00:11:22:33:44:55
sudo ifconfig wlan0 up

See

ip

It is said ip is replacing the old ifconfig command on modern Linux distributions.

http://www.makeuseof.com/tag/networking-commands-linux-terminal/ 

ip address show

ip link set DEVICE down # eg ip link set eth0 down

ip link set DEVICE up

iptables

See the ufw command which provides an easy way to configure iptables.

route

7 Linux Route Command Examples

  1. Display Existing Routes (route -n)
  2. Adding a Default Gateway (route add default gw 192.168.1.1)
  3. List Kernel’s Routing Cache Information (route -Cn)
  4. Reject Routing to a Particular Host or Network (route add -host 192.168.1.51 reject)
  5. Make 192.168.3.* Accessible from 192.168.1.* (route add -net 192.168.3.0 netmask 255.255.255.0 gw 192.168.3.10)
  6. Make 192.168.1.* Accessible from 192.168.3.* (route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.1.10)
  7. Allow Internet Access/External World (route add default gw 125.250.60.59)

On Ubuntu 16.04, it shows 

$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         FIOS_Quantum_Ga 0.0.0.0         UG    600    0        0 wlp3s0
link-local      *               255.255.0.0     U     1000   0        0 wlp3s0
192.168.1.0     *               255.255.255.0   U     600    0        0 wlp3s0
$ route -n   # showing numerical IP address instead of host name.
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    600    0        0 wlp3s0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 wlp3s0
192.168.1.0     0.0.0.0         255.255.255.0   U     600    0        0 wlp3s0

Flag value 'U' means up and 'G' means gateway'.

Connect two networks

The trick is explained in this post or the above route command.

For example, my network structure is

  • Modem/router: LAN IP 192.168.1.*/24
  • PC1: connect to Modem/router
  • Second router (ASUS) connect to Modem/router: its WAN IP is 192.168.1.ASUS. It's LAN IP 192.168.2.*/24
  • PC2 (raspberry pi): connect to the second router (ASUS): its IP is 192.168.1.212

By default, PC2 can ssh to PC1 but PC1 cannot access PC2.

The following command will solve the problem that PC1 cannot access PC2: 

# From PC1
$ sudo route add -net 192.168.2.0 netmask 255.255.255.0 gw 192.168.1.ASUS
$ ssh [email protected]    # 192.168.2.212 is the IP address for the Raspberry Pi

$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.2.0     192.168.1.ASUS   255.255.255.0   UG        0 0          0 eth0

traceroute

sudo apt-get install traceroute
traceroute 8.8.8.8

On Windows, we can use the tracert command. For example, tracert www.microsoft.com.

netstat

  • -l or --listening shows only the sockets currently listening for incoming connection.
  • -a or --all shows all sockets currently in use.
  • -t or --tcp shows the tcp sockets.
  • -u or --udp shows the udp sockets.
  • -n or --numeric shows the hosts and ports as numbers, instead of resolving in dns and looking in /etc/services.
netstat -ant           # For tcp
netstat -peanut        # Easier to remember

nmap - port scanning & IPs in local network

nmap - Network exploration tool and security / port scanner

  • https://nmap.org/book/nmap-os-db.html. Local OS database is located at /usr/share/nmap/nmap-os-db. The 2nd line will show the revision number.
    • Modifying the nmap-os-db Database Yourself
    • Download the latest from https://svn.nmap.org/nmap/nmap-os-db. Note that the current revision number has to be found from the website. You can edit the file and insert the revision number on the 2nd line of your local copy.
    • Even I update the database, it cannot detect my Ubuntu 14.04 OS (it only shows OS details: Linux 3.8 - 4.9). For the Raspberry Pi, it can show information from the network adapter; e.g. MAC Address: AA:BB:CC:DD:EE:FF (Raspberry Pi Foundation) but not the OS name (OS details: Linux 3.2 - 4.8).
sudo mv /usr/share/nmap/nmap-os-db /usr/share/nmap/nmap-os-db-old

cd /usr/share/nmap
sudo wget https://svn.nmap.org/nmap/nmap-os-db
sudo apt-get install nmap

nmap 192.168.1.99    # does not require root privileges
                     # used to check open ports

nmap 192.168.1.*     # show IPs and ports in LAN

sudo nmap -sP 192.168.1.1/24 # show connected IPs (no hostnames?) and MAC addresses
                             # If you don't use 'sudo' only partial devices can be found
                             # The output may contains the hostname. For example,
                             # Nmap scan report for brb-P45T-A.fios-router.home (192.168.1.xxx)
nmap -sV 192.168.1.1 # show Daemon name (in VERSION column) together with port number

nmap -T4 -F 192.168.1.99-255 # show connected IPs and open ports
                             # -F means fast
nmap -F taichimd.us
nmap -v taichimd.us

nmap -A 192.168.1.1  # Aggressive scan (more output)

nmap -p http,ssh,mysql taichimd.us  # scan ports/services
                                    # note that mysql will be shown as closed
nmap --open taichimd.us   # scan open ports

sudo nmap -traceroute nih.gov

sudo nmap -sS -O 192.168.1.99 # -O shows operating system
                              # eth0 MAC

A gui version of nmap is called Zenmap.

arp (Address Resolution Protocol)

The arp command can be used to show the MAC addresss of all hosts in LAN 

arp -a

Copy text to a clipboard to be used in other apps

Install the xclip program. See here or here. 

sudo apt-get install xclip
# Examples
sort -n -k 3, -k 2 file.txt | xclip -selection clipboard

cat ~/.ssh/id_rsa.pub | xclip -sel clip

Works.

Start Emacs without X

Add -nw (no window) option. 

emacs -nw

Audio

mp3 codecs

https://help.ubuntu.com/community/RestrictedFormats 

sudo apt-get install ubuntu-restricted-extras

Concatenate mp3 files

sudo apt-get install mp3wrap
mp3wrap output.mp3 *.mp3

Reduce the size of an mp3 file

Specify a new lower bitrate using the -b option in lame. For example if your starting mp3 has a quality of 256kbs you can lower its bitrate to 128kbps (or even lower like 64kbps) by: 

lame --mp3input -b 128 input.mp3 output.mp3

Convert ogg to mp3

ffmpeg is not included in Ubuntu repository. Use the avconv command. http://superuser.com/questions/15327/how-to-convert-ogg-to-mp3 

sudo apt-get install libav-tools
avconv -i input.ogg -c:a libmp3lame -q:a 2 output.mp3

Convert m4a to mp3

avconv -i input.m4a output.mp3

Normalize the volume of an audio file

avconv -ss 00:00:10 -i OLD.mp3 -vol 2560 NEW.mp3

The anconv/ffmpeg -vol parameter amplifies the sound. The default value is 256 (no amplification), and you can adjust the number accordingly. Here it’s 2560, as it’s 10 times louder. Note that these are not decibel values or anything that sophisticated, but just an integer value. 512 equals to twice the volume, 768 three times, 1024 four times, etc. The -ss parameter specifies the start time offset. Here it will skip the first 10 seconds.

I found the converted file by sox has about one half file size compared to anconv/ffmpeg program (source file=47MB, anconv converted=135MB, sox converted file=54MB). 

sudo apt-get install sox libsox-fmt-all
sox --norm OLD.mp3 NEW.mp3 trim 10
sox --norm OLD.mp3 NEW.mp3 silence 1 0.1 1%
sox -v 4.0 OLD.mp3 NEW.mp3             # increase volume

where '--norm' will normalize the audio and the 'trim' option set to skip the first 10 seconds. The silence parameter allows to trim silence at the beginning without a need to specify the number of seconds.

cut, delete or trim an audio

  1. Open the audio file in audacity.
  2. select a region in the waveform area. Do not select in the time interval area (above the waveform).
    1. To precisely select a range from one position to the end. Click Zoom in several times. Click one position in the waveform and click Edit -> Select -> Cursor to the track end to select
    2. Similarly, if we want to precisely select a range from the start to some position, we can click one position in the waveform and then click Edit -> Select -> Track start to cursor.
    3. To move around the track, use the scrollbar (below the waveform and above the bottom toolbar, not quite clear in Ubuntu/Unity)
  3. Click Edit -> Remove Audio or labels -> Cut/Delete/Trim Audio
  4. play the new audio by clicking the green triangle.
  5. File -> Export -> mp3 format.

Helpful resource for Audacity.

  1. Different Toolbars
  2. Tutorial - Editing an Existing Audio File

Fade out at the end of an audio

  1. Select a region.
  2. Effect -> Fade out

Wireshark

sudo apt-get install wireshark
sudo chmod 4711 `which dumpcap`

Track the Time a Command Takes

time command

Use time command. 

time COMMAND
time (COMMAND1; COMMAND2)
time (COMMAND1 && COMMAND2)

help time

When I run a set of 7 jobs using parallel, time command gives an output 

real  15m53.788s
user  95m20.238s
sys   9m1.320s

Here we see the real time is about 16m and the user time is about 6-7 times the real time. Indicating the parallel executing works.

/usr/bin/time command

/usr/bin/time provides more information then time command. 

man time

Magazines

PDF reader

The default one Evince seems slow when I try to view odroid magazine. I installed and tried MuPDF (github source code). It seems faster and I don't see blank pages when I view one odroid magazine. In terms of speed, mupdf >> xpdf >> okular >> Evince. 

sudo apt-get install mupdf

Keyboard shortcuts for mupdf (man mupdf) or http://mupdf.com/docs/manual: 

W    - fit to width
H    - fit to height
L    - rotate page left (clockwise)
R    - rotate page right (counter-clockwise)
12g  - go to page 12
>,<  - go to the next or previous page
+,-  - zoom in or out
/    - search for text
n,N  - Find the next or previous search result.

To copy a text, use right mouse button to select a text. Then use Ctrl+c to copy it.

Other pdf viewer choices are

  • acroread
  • xpdf
  • okular (KDE/Qt application), allow to change its background color
  • kpdf
  • gv
  • qpdfview
  • Foxit or PDF-XChange Viewer(needs wine)

PDF highlight and annotation

Install Okular by 

sudo apt-get install okular

To highlight a line, click F6 (Tools -> Review) to turn on the annotation tool bar (it will be shown on the left hand side of the documentation). You can then click

  1. the 4th icon to highlight a line (it may not be able to select the right texts we want. But when it works the result is nice)
  2. the last icon to draw an ellipse or a rectangle (to change from an ellipse to a rectange you can click Settings -> configure Okular... -> annotation)

Another method is to use a windows program and run it using Wine. See the discussion here.

Flow chart

  • LibreOffice Draw OR MS_PowerPoint (insert > shape). Check youtube.
  • yEd
  • Dia & wikipedia
  • (online) www.draw.io

Clock

xclock

oclock -geometry 500x500+100+0 &

oclock

oclock -bg blue -geometry 500x500+100+0 -bd purple -transparent &
oclock -bg blue -geometry 500x500+100+0 -bd purple -jewel green &

See oclock, X - a portable, network-transparent window system which includes an example of specifying the geometry parameter.

dclock

Digital clock for the X Window System with flexible display. 

sudo apt-get install dclock
dclock -h
dclock -d
dclock -date "Today is %A %B %Y" -geometry 577x194+119+139

Lubuntu digital clock format

http://netgator.blogspot.com/2012/09/change-edit-panel-digital-clock-format.html. My format is 

%a, %x, %r
# Tue, 05/17/2016, 09:42:27 PM

%a %m-%d-%y, %I:%M %p
# Mon 05-30-16, 08:31 AM

wine and winetricks

Running Linux in the AWS/Amazon Web Services

Forum software

RAID

Timer

How to track you laptop using Prey

https://www.howtoforge.com/tutorial/how-to-track-your-linux-laptop/

Automatic reboot after power failure

It seems there is no reliable way to find out when the power failed.

The linux command 'last' can show some information about system reboot.

Another way is to modify the BIOS to select the option like 'Power off and Reboot'. This won't automatically boot your computer when it is shutdown normally.

Speedtest-cli

See Track_Internet_Dropouts. 

sudo apt-get intall python-pip
sudo pip install speedtest-cli
# A slightly modified code that will create a one-line space/semi-colon 
# delimited result is 
git clone https://github.com/HenrikBengtsson/speedtest-cli-extras.git

speedtest-cli-extras/bin/speedtest-csv

works. But if I want to put it in cron, cron will issue an error speedtest-cli cannot be found. So I need to modify line 52 of the code in <speedtest-cli-extras/bin/speedtest-csv> to explicitly specify the location of speedtest-cli. 

    /usr/local/bin/speedtest-cli --share > $log

NOTE: 1. the results differ from the network connection. For example, the speed is good when I test it on the machine directly connected to the router. 2. It is helpful to modify the last line of the bash script to output what I need. 3. The separator is ";" in the output.

uname - Print system information

https://www.lifewire.com/display-system-information-uname-command-3964321

uname -a will give you

  • OS (uname = uname -s if you are under a Linux environment)
  • OS (uname -s) eg Linux
  • node name (uname -n=hostname)
  • kernel release (uname -r) eg 3.16.0-38-generic
  • kernel version (uname -v)
  • machine architecture (uname -m) eg x86_64
  • processor (uname -p)
  • hardware platform (uname -i)
  • operating system (uname -o)

How to check if running in Cygwin, Mac or Linux?

Linux Logo and the current system information

odroid@odroid:~$ sudo apt-get install screenfetch
odroid@odroid:~$ screenfetch
                          ./+o+-       odroid@odroid
                  yyyyy- -yyyyyy+      OS: Ubuntu 15.10 wily
               ://+//////-yyyyyyo      Kernel: armv7l Linux 3.10.96-77
           .++ .:/++++++/-.+sss/`      Uptime: 4d 23h 8m
         .:++o:  /++++++++/:--:/-      Packages: 2000
        o:+o+:++.`..```.-/oo+++++/     Shell: 2263
       .:+o:+o/.          `+sssoo+/    Resolution: 1920x1080
  .++/+:+oo+o:`             /sssooo.   DE: MATE 1.10.2
 /+++//+:`oo+o               /::--:.   WM: Metacity (Marco)
 \+/+o+++`o++o               ++////.   GTK Theme: 'Ambiant-MATE' [GTK2/3]
  .++.o+++oo+:`             /dddhhh.   Icon Theme: Ambiant-MATE
       .+.o+oo:.          `oddhhhh+    Font: Ubuntu 10
        \+.++o+o``-````.:ohdhhhhh+     CPU: ARMv7 rev 3 (v7l) @ 1.4GHz
         `:o+++ `ohhhhhhhhyo++os:      GPU: Gallium 0.4 on llvmpipe (LLVM 3.6, 128 bits)
           .o:`.syhhhhhhh/.oo++o`      RAM: 537MiB / 1990MiB
               /osyyyyyyo++ooo+++/    
                   ````` +oo+++o\:    
                          `oo++.      

odroid@odroid:~$ screenfetch -s  # take a screenshot and auto save it to ~/ directory.

odroid@odroid:~$ sudo apt-get install linuxlogo
odroid@odroid:~$ linuxlogo
        _,met$$$$$gg.                                                           
     ,g$$$$$$$$$$$$$$$P.                                                        
   ,g$$P""       """Y$$.".                                                      
  ,$$P'              `$$$.                                                      
',$$P       ,ggs.     `$$b:                                                     
`d$$'     ,$P"'   .    $$$                               ,#.                    
 $$P      d$'     ,    $$P      ##:          :##        :###:                   
 $$:      $$.   -    ,d$$'      ##'          `##         `#'                    
 $$;      Y$b._   _,d$P'    __  ##     __     ##  __      _     __          _   
 Y$$.    `.`"Y$$$$P"'     ,####:##  ,######.  ##.#####. :### ,######. ###.####: 
 `$$b      "-.__         ,##' `###  ##:  :##  ###' `###  ##' #:   `## `###' `##:
  `Y$$b                  ##    `##  ##    ##  ##'   `##  ##    ___,##  ##:   `##
   `Y$$.                 ##     ##  #######:  ##     ##  ##  .#######  ##'    ##
     `$$b.               ##     ##  ##'       ##     ##  ##  ##'  `##  ##     ##
       `Y$$b.            ##.   ,##  ##        ##    ,##  ##  ##    ##  ##     ##
         `"Y$b._         :#:._,###  ##:__,##  ##:__,##' ,##. ##.__:##. ##     ##
             `""""       `:#### ###  ######'  `######'  #### `#####"## ##     ##

 Linux Version 3.10.96-77, Compiled #1 SMP PREEMPT Fri Feb 5 04:47:32 BRST 2016
             Eight ARM  Processors, 2GB RAM, 456.00 Bogomips Total
                                     odroid

odroid@odroid:~$ linuxlogo -f -L list
odroid@odroid:~$ linuxlogo -f -L ubuntu
              .-. 
        .-'``(|||) 
     ,`\ \    `-`.                 88                         88 
    /   \ '``-.   `                88                         88 
  .-.  ,       `___:      88   88  88,888,  88   88  ,88888, 88888  88   88 
 (:::) :        ___       88   88  88   88  88   88  88   88  88    88   88 
  `-`  `       ,   :      88   88  88   88  88   88  88   88  88    88   88 
    \   / ,..-`   ,       88   88  88   88  88   88  88   88  88    88   88 
     `./ /    .-.`        '88888'  '88888'  '88888'  88   88  '8888 '88888' 
        `-..-(   ) 
              `-` 


 Linux Version 3.10.96-77, Compiled #1 SMP PREEMPT Fri Feb 5 04:47:32 BRST 2016
             Eight ARM  Processors, 2GB RAM, 192.00 Bogomips Total
                                     odroid

odroid@odroid:~$ screenfetch -h
odroid@odroid:~$ linuxlogo -h

Neofetch.png

Dictionary - Artha

  • Lifehacker. Once it is launched, it is sitting on the task bar. Press Ctrl+Alt+W after selecting a word to look it up in Artha (a balloon tip will pop up on the screen top-right). It also supports using regular expressions to search words.
sudo apt-get install artha

Translation

  • Translate Shell. No installation is needed. It is just a bash script (4990 lines) so it works on ODroid SOC.
odroid@odroid:~/binary$ ./trans :zh-TW word
word
/wərd/

字
(Zì)

Definitions of word
[ English -> 正體中文 ]

noun
    字
        word, character, letter, calligraphy, symbol, style of writing
    詞
        word, term, speech, statement
    單詞
        word, individual word
    話
        words, word, dialect, saying, talk, speech
    言
        word, speech, character
    言辭
        words, word, what one says
    筆墨
        pen and ink, words, word, writings
    約言
        pledge, promise, word

verb
    為 ... 措辭
        word
odroid@odroid:~/binary$ time ./trans -brief :zh-TW word
字

real	0m4.249s
user	0m2.670s
sys	0m1.330s

ASCII art

  ____  ____  ____       ____            _____           _     
 | __ )|  _ \| __ )     / ___|  ___  __ |_   _|__   ___ | |___ 
 |  _ \| |_) |  _ \ ____\___ \ / _ \/ _` || |/ _ \ / _ \| / __|
 | |_) |  _ <| |_) |_____|__) |  __/ (_| || | (_) | (_) | \__ \
 |____/|_| \_\____/     |____/ \___|\__, ||_|\___/ \___/|_|___/
                                       |_|                     

  ____  _____  ____        _____         _______          _     
 |  _ \|  __ \|  _ \      / ____|       |__   __|        | |    
 | |_) | |__) | |_) |____| (___   ___  __ _| | ___   ___ | |___ 
 |  _ <|  _  /|  _ <______\___ \ / _ \/ _` | |/ _ \ / _ \| / __|
 | |_) | | \ \| |_) |     ____) |  __/ (_| | | (_) | (_) | \__ \
 |____/|_|  \_\____/     |_____/ \___|\__, |_|\___/ \___/|_|___/
                                         | |                    
                                         |_|                    

  ___ ___ ___     ___          _____         _    
 | _ ) _ \ _ )___/ __| ___ __ |_   _|__  ___| |___
 | _ \   / _ \___\__ \/ -_) _` || |/ _ \/ _ \ (_-<
 |___/_|_\___/   |___/\___\__, ||_|\___/\___/_/__/
                             |_|

Software that scan Malware and rootkits

Text to speech

VPN

OpenVPN

Mono Project

Mono is a software platform designed to allow developers to easily create cross platform applications part of the .NET Foundation

Mono is required for Repetier-Host software for 3D printing.

NAS server

OpenMediaVault

4 easy Linux projects for newbies and intermediate users. OpenMediaVault is a linux-based system.

Docker container for OpenMediaVault. OpenMediaVault插件之Docker教程

FreeNAS

ZFS system (FreeBSD-based).

10 Reasons Why You Should Store Your Data on a FreeNAS Box. Note With the current version of FreeNAS (FreeNAS 11) comes a hypervisor. See

Change detection

http://bhfsteve.blogspot.com/2013/03/monitoring-web-page-for-changes-using.html

Open source surveillance

ZoneMinder

Systemctl, systemd

Best Linux Adobe Alternatives You Need to Know

Small Linux distributions

Install Linux to Save Space! These Tiny Linux Distros Are Super Small 10/10/2017

Debian/Ubuntu/Bio-Linux

Debian

Download Debian

Go to http://www.debian.org/distrib/ and download "Small CDs or USB sticks", for example debian-8.1.0-amd64-netinst.iso (Jessie, released June 2015). It is about 280 MB.

Screenshots of step-by-step installation can be found in here.

At the end of installation, it will offer a collection of software to install. Below 'Debian desktop environment', it has a selection of GNOME, Xface, KDE, Cinnamon, MATE and LXDE (new in Jessie/8.0). Note that the cd images download page only provides a selection of kde, lxde and xfce desktop. The default is 'GNOME' in Jessie.

DebianInstall.png

After installation, you got a desktop environment of Debian based on GNOME 3 (Virtual machine will use recovery mode, but still works. Some people suggest to install the guest additions (in the guest) and make sure that you enable the 3D acceleration in the guest settings.). Also 'free -m' command shows it uses 202 MB memory and the whole system takes up 3.3 GB. I am testing on a Chinese desktop environment.

When Debian is in recovery mode, the desktop interface is like old fashion. Application and Place on top of the screen. When GNOME 3 is working (in my test of Debian 7.1.0, I cannot run VBoxLinux.run, but it still works after I did other steps), the interface is sort of Ubuntu with application launched from the left hand side. It differs from Ubuntu because the side bar appears by clicking a 'preview' button on top left corner.

The default browser in Debian is Iceweasel with AdBlock Plus preinstalled.

I don't know why the default user does not have root privilege.

When I installed the Chinese version, the keyboard switch icon (SCIM) is automatically available. However once the desktop is in regular GNOME 3, the switch icon disappeared. Fortunately, we can use Ctrl + Space to switch languages. Thanks to the hint there.

Server version

There is no a server version of iso to download. At the end of installation, it will ask what software to install: Debian desktop environment, Web server, SSH server, Laptop, SQL database, ... We can uncheck 'Debian desktop environment' item.

Compared to Desktop version, the server version takes 1.3GB space and 33MB memory.

The server version does not have 'sudo' command. Use 'su' to switch to 'root' user.

Note that even we installed 'sudo', we can not use 'sudo' from the default user. It will complain xxx is not in the sudoers file. This incident will be reported..

Virtualbox guest addition installation

See Virtualbox -> Debian.

Browse iso files

Note that if we want to download the iso image, we should consider using the torrent method. The can see a variety of download options from

http://www.debian.org/CD/ > Downloading Debian CD/DVD images via HTTP/FTP

Permission denied

http://roger.steneteg.org/blog/virtualbox-guest-additions-on-debian/

The script uses /bin/sh as shell and on Debian Wheezy/Jessie /bin/sh is symlinked to /bin/dash. Dash is a more light-weight replacement for Bash, and it turns out that the VirtualBox script does not work as it should when run with "dash".

An easy workaround is to explicitly run the script with "bash" with the following command: 

sudo bash ./VBoxLinuxAdditions.run

Browse source code

Ubuntu/Kubuntu/Lubuntu/Xubuntu

Download links for all versions from wiki.ubuntu.com > releases.ubuntu.com.

Ubuntu flavors and derivatives

For some reason, when I try the Ubuntu (13.04) live CD, the screen resolution looks perfect. But when I installed the OS, the screen resolution is always too low. The propriety graphics driver cannot be installed successfully. Fortunately, when I try the Kubuntu (13.04), the display resolution problem automatically works!

Update: Kubuntu failed to respond after I install SCIM related programs. A freshly installed linuxmint OS also has a similar problem that the desktop does not respond to mouse or keyboard. Luckily, the Xubuntu works fine and the Chinese input works out of box if I choose Chinese as desktop environment (339MB was used).

Download mirror

https://launchpad.net/ubuntu/+cdmirrors

For me, mirrors.acm.jhu.edu (only visible from Ubuntu's Software & Updates -> Download from, Ubuntu 16.04) or mirror.umd.edu are closest.

Installation

Installing Ubuntu (or xubuntu, Mint) still requires an internet connection for downloading language packs. This could be very time consuming. However, in the installation process I can click the 'skip' button to skip downloading language packs. This saves a lot of time when the internet connection is slow. After ubuntu desktop appears, it still pops up a message to give an instruction to install language packs.

The installation takes about 10 minutes when I installed ubuntu 14.04 (unity) on virtualBox.

How to upgrade

End of life date of Ubuntu release

https://wiki.ubuntu.com/Releases

Each time I log into my Ubuntu 12.04.5 LTE, I'll receive a message 

New release '14.04.2 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Your current Hardware Enablement Stack (HWE) is no longer supported
since 2014-08-07.  Security updates for critical parts (kernel
and graphics stack) of your system are no longer available.

For more information, please see:
http://wiki.ubuntu.com/1204_HWE_EOL

There is a graphics stack installed on this system. An upgrade to a 
supported (or longer supported) configuration will become available
on 2014-07-16 and can be invoked by running 'update-manager' in the
Dash.
Ubuntu 12.04

As of Dec 1, 2015, Chrome for 32-bit Ubuntu 12.04 is not supported anymore.

Black screen on boot Ubuntu 14.04

Press e when you see the first menu.

Server version

UbuntuServerInstall.png

AV Linux

AV Linux features a complete customized Debian Linux XFCE4 4.10 Desktop Environment with the added bonus of a handpicked selection of pre-tested and pre-configured Audio, Graphics and Video content creation software demonstrating the excellence of Open-Source and also includes many unique Commercial Demos.

Kali Linux

Create customized ubuntu iso

  • Ubuntu Mini Remix (~200MB). Note that this is a live ubuntu which can't be installed even we can remaster it to include Desktop Environment, packages, et al. See this FAQ.
    • Ubuntu customization kit - linux.com howtogeek. The project has not been updated since 2013-01-16.
    • Ubuntu Builder - lifehacker. It looks the project is abandoned.
    • Customizer - quite information from its website. The manuals/user guide 3.x p46 talks about how to make the iso installable instead of just a live CD (e.g. apt­-get install ubiquity ubiquity-­frontend-­gtk). N.B. Installing ubiquity should be run once we have installed all software we want; i.e. if we want to install xfce4 we should install xfce4 before we install ubiquity. Also for some reason, Customizer crashed when I tried to create an iso if I have installed xubuntu-desktop, ubiquity and ubiquity-­frontend-­gtk.

Note that the Ubuntu Mini Remix by default contains only 3 repositories. We may want to add some more. 

deb http://archive.ubuntu.com/ubuntu/ trusty main restricted
deb http://security.ubuntu.com/ubuntu/ trusty-security main restricted
deb http://archive.ubuntu.com/ubuntu/ trusty-updates main restricted

while for example an official v14.04 xubuntu contains 22 sources, 

# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://us.archive.ubuntu.com/ubuntu/ trusty main restricted
deb-src http://us.archive.ubuntu.com/ubuntu/ trusty main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://us.archive.ubuntu.com/ubuntu/ trusty-updates main restricted
deb-src http://us.archive.ubuntu.com/ubuntu/ trusty-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://us.archive.ubuntu.com/ubuntu/ trusty universe
deb-src http://us.archive.ubuntu.com/ubuntu/ trusty universe
deb http://us.archive.ubuntu.com/ubuntu/ trusty-updates universe
deb-src http://us.archive.ubuntu.com/ubuntu/ trusty-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu 
## team, and may not be under a free licence. Please satisfy yourself as to 
## your rights to use the software. Also, please note that software in 
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://us.archive.ubuntu.com/ubuntu/ trusty multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ trusty multiverse
deb http://us.archive.ubuntu.com/ubuntu/ trusty-updates multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ trusty-updates multiverse

## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb http://us.archive.ubuntu.com/ubuntu/ trusty-backports main restricted universe multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ trusty-backports main restricted universe multiverse

deb http://security.ubuntu.com/ubuntu trusty-security main restricted
deb-src http://security.ubuntu.com/ubuntu trusty-security main restricted
deb http://security.ubuntu.com/ubuntu trusty-security universe
deb-src http://security.ubuntu.com/ubuntu trusty-security universe
deb http://security.ubuntu.com/ubuntu trusty-security multiverse
deb-src http://security.ubuntu.com/ubuntu trusty-security multiverse

## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
# deb http://archive.canonical.com/ubuntu trusty partner
# deb-src http://archive.canonical.com/ubuntu trusty partner

## This software is not part of Ubuntu, but is offered by third-party
## developers who want to ship their latest software.
deb http://extras.ubuntu.com/ubuntu trusty main
deb-src http://extras.ubuntu.com/ubuntu trusty main

Create your own Debian iso

MultiSystem – Create a MultiBoot USB from Linux

Minimal Ubuntu

The minimal ubuntu iso is about 50MB. It will download files when we install the Ubuntu.

In the halfway of installing the minimal Ubuntu, there is a dialog called 'Software selection'. It says At the moment, only the core of the system is installed. To tune the system to your needs, you can choose to install one or more of the following predefined collections of software. Choose software to install:.

In addition to some default selections (like 'standard system utilities'), I choose Ubuntu MATE minimal installation (not 'Ubuntu MATE desktop'). This action will retrieve about 1228 files from the internet. After finishing install them, the installer also installed GRUB and set up system clock. Then the installation was complete. We have to reboot the system (for virtual machine case we need to power off the guest machine and remove the virtual drive).

For the Ubuntu MATE minimal installation selection, it still includes several software. The 'df' command shows 3.3GB space was used in this minimal Ubuntu MATE 16.04.

  • Accessories: Character Map, Engrampa Archive Manager, Calculator, MATE Search Tools, Passwords and Keys, Pluma Text Editor, Take Screenshot
  • Graphics: Eye of MATE image Viewer, MATE Color Selection, Simple Scan
  • Internet: Firefox
  • Office: Atril Document Viewer, MATE Dictionary
  • Sound & Video: Sound
  • System Tools:Avahi Zeroconf Browser, Caja, dconf Editor, GDebi Package Installer, Log File Viewer, MATE Disk Usage Analyzer, MATE System Monitor, MATE Terminal, Power Statistics
  • Universal Access: Onboard, Screen Magnifier, Screen Reader

Actually, if we do not select Ubuntu MATE minimal installation but rather choose to install it later on from the command line (sudo apt-get install --no-install-recommends ubuntu-mate-core) we still end up with the same Ubuntu MATE desktop environment (3.3GB).

The Perfect Server

Live USB with persistent storage

Linux Live USB Creator

Selection of desktop environment

echo $DESKTOP_SESSION

On Ubuntu, it returns

  1. 'ubuntu' gnome 3+unity
  2. 'mate' ubuntu Mate on Odroid xu4
  3. 'default' on Debian 8.0
  4. 'LXDE' on Debian's BBB
  5. 'Lubuntu' on UDOObuntu 2

Unity

Unity as the default user interface instead of GNOME Shell, beginning April 2011, with Ubuntu 11.04 according to the wikipedia.

Use 'unity --version' to check the unity version. If something was screwed up (eg after we remove gnome-desktop), we can reinstall the unity desktop by 

sudo apt-get install ubuntu-desktop

Don't forget Unity Tweak Tool. That is required if you want to install, saying, Arc theme. I install it on Ubuntu 16.04 by running sudo apt install unity-tweak-tool.

Unity tweak tool.png

Note that there is no screensaver anymore starting with Ubuntu 12.04. Read this post. If we want to add a screensaver program, read How to Add Screensavers to Ubuntu 12.04

GNOME

Ubuntu GNOME (GNOME 3). The build-in screensaver is a digital clock showing the current time & date. Cool! This seems to be a new feature in GNOME 3.6 optimized for touch screen devices. See this and this.

UbuntuGnome1404.png UbuntuGnome1404b.png

Note that we can install the gnome desktop by using the command line. It will keep the current wallpaper. The clock in screensaver will not be shown until we shake the mouse or keyboard. 

sudo apt-get install gnome-shell
# Choose 'gdm' (Gnome Desktop Manager) as the display manager instead of 'lightdm' the Ubuntu's default
#    when it is configuring gdm as only GDM offers GNOME-specific features such as lock-screen notifications.
# See the screenshot at 
#    https://ideasnet.wordpress.com/2013/05/11/ides-desktop-how-to-replace-unity-with-gnome-3-8-in-ubuntu-13-04-desktop-edition/
# If messed up, run "sudo dpkg-reconfigure gdm"
sudo apt-get install ubuntu-gnome-desktop

If the lock screen does not work, use Settings > Brightness and Lock, or use the command line 

gsettings set org.gnome.desktop.lockdown disable-lock-screen 'false'

If the screensaver is not working, try 

sudo apt-get install gnome-screensaver

In my case, the screen turns off (black). But if we wake the PC up, the time and date screen shows up.

Some differences (inconvenience): 1. No maximize, minimize windows buttons 2. Have to click 'Activities' button (or 'Windows' key) to switch applications. These complains also appeared in other review.

KDE

Kubuntu

Xfce

Xfce Explained: A Look at One of Linux’s Speediest Desktops "Xfce doesn’t support HiDPI, which can be a deal breaker on newer machines. Even on a laptop with a 1920 x 1080 screen resolution, I find Xfce’s interface and windows to be absolutely tiny."

Xubuntu. The response is quicker when I compare the speed by clicking the top-left icon (app menu) in Xfce and Acitvities button in ubuntu-GNOME. This is tested when both Xubuntu and ubuntu-GNOME are installed in VirtualBox. 

# Note the sources.list should contain 'universe' repositories.
# https://help.ubuntu.com/community/Repositories/Ubuntu
#
# Install XFCE alone, without Xubuntu, with this command:
sudo apt-get install xfce4
# Install the entire Xubuntu package, which includes a full suite of software and a lot of improvements:
sudo apt-get install xubuntu-desktop
#
# Note that installed terminal is XTerm and UXTerm which looks awful. We will want 
# xfce4-terminal, a modern, lightweight and low memory cost terminal emulator for X11, 
# which was included in the Xubuntu desktop.
sudo apt-get install xfce4-terminal

The default display manager (used e.g. log in screen) can be found by 

cat /etc/X11/default-display-manager

To install lightdm display manager 

sudo apt-get install lightdm

After running the above command, I found 1. a GUI login screen came out, but login failed (my case). Use Ctrl+Alt+F1 to switch to the command line approach. 2. startx failed.

LXDE

Lubuntu, LXLE and LXQt. LXDE is the default desktop environment for Raspbian, LXLE, BBB, and Lubuntu.

The default browser in LXLE is SeaMonkey (Debian Jessie has Iceweasel which is even similar to Firefox; see Odroid). In the Internet category, it also includes FireFTP (SeaMonkey extension), Transmission, Gitso (VNC), uGet. In the Sound/Video category, it includes Arista transcoder, Audacity, Guaydadeque Music Player, guncviewer, Pithos, RecordMyDesktop, Videos/Totem. Gedit, GParted, KeepassX and LibreOffice are standard. There is also a Games category.

The LXLE LS version provides several business apps to anyone on the network.

LXQt is the Qt port and the upcoming version of LXDE, the Lightweight Desktop Environment.

The download links (if we want to skip answering the question) are https://sourceforge.net/projects/lxle/files/Final/OS/.

LXLE key features

One cool thing in lxle is the random wallpaper. On regular Ubuntu, we can install a wallpaper changer program - Wallch or Variety. See also OMGUBUNTU.

Variety pref.png

In Lxle, the random wallpaper is done through a simple command 

dash -c 'pcmanfm -w "$(find ~/Pictures/Wallpapers -type f | shuf -n1)"'

A list of keyboard shortcuts for Lubuntu.

The file manager in LXLE is PCManFM (not to confuse with the package manager pacman in ArchLinux). It does not support drag and drop to copy a file name to a terminal but it can execute a file in a terminal by PCManFM -> Tools -> Run a Command in Current Folder.

Cinnamon

Linux Mint. A GTK+-based desktop environment. Note that Mint releases only LTS versions (5 years support as Ubuntu).

To check the desktop environment, use echo $DESKTOP_SESSION. To check the cinnamon version, use cinnamon --version.

Note

  1. To change to other workspace, use Ctrl + Alt + Up or Alt + F1 keybind to enter Expo mode and then select one. To directly switch to the next workspace, use Ctrl + Alt + Right/Left arrow key.
  2. To change the title bar color from gray to black follow this instruction. Go to Preferences -> Themes and click 'Add/Remove desktop themes'. Type the keyword 'nightfall' and install the theme. Close the 'Themes' window and open it again. Click 'Windows borders' and select 'nightfall'.
  3. Mint has its own package update manager. Click Menu > Administration > Update Manager. Security updates are level 5. Kernel updates are level 4/5. See their meaning on the official user's guide.
    • Level 1 and Level 2 updates are risk-free and you should always apply them. Level 3 updates “should be safe” but, although we recommend you take them, make sure you look over them on the list of updates.
    • Level 4 is "Unsafe packages". It could potentially affect the stability of the system". Level 5 is "Dangerous packages".
  4. We don't have to install Linux Mint in order to get cinnamon desktop. How To Install Cinnamon 3.0 In Ubuntu 16.04 Or 15.10 Via PPA
  5. this Cinnamon desktop has a digital clock as a screen saver while the MATE desktop only has a black screen as a screen saver. I still get the digital clock as a screen saver when I install cinnamon desktop on top of regular Ubuntu.

MintCinnamonSS.png

MATE

Ubuntu Mate. The MATE Desktop Environment is the continuation of GNOME 2. 

odroid@odroid:~/$ cat /etc/X11/default-display-manager
/usr/sbin/lightdm

Pantheon

Elementary OS. I cannot make the Chinese input to work (I can install ibus-chewing but cannot switch input methods?).

How to Refresh Your Linux Desktop Without Rebooting

http://www.makeuseof.com/tag/refresh-linux-desktop-without-rebooting/

Ubuntu Software Repository

The repository components are:

  1. Main - Officially supported software.
  2. Restricted - Supported software that is not available under a completely free license.
  3. Universe - Community maintained software, i.e. not officially supported but enabled by default software.
  4. Multiverse - Software that is not free.

See the pitfall in the PCWorld article.

Slow download

Check if a repository exists

For example, consider the CRAN repository at cran.rstudio.com server. 

if grep -q "deb http://cran.rstudio.com/bin/linux/ubuntu" /etc/apt/sources.list; then
  echo http://cran.studio.com/bin/linux/ubuntu was found
else
  add-apt-repository "deb http://cran.rstudio.com/bin/linux/ubuntu $codename/"
  gpg --keyserver keyserver.ubuntu.com --recv-key E084DAB9
  gpg -a --export E084DAB9 | apt-key add -
fi

Now run nano /etc/apt/sources.d to check if duplicated repository has been added.

GPG/Authentication key

gpg (GNU Privacy Guard) is the tool used in secure apt to sign files and check their signatures. See https://help.ubuntu.com/community/SecureApt or https://wiki.debian.org/SecureApt.

apt-key is a program that is used to manage a keyring of gpg keys for secure apt. Note The keyring is kept in the file /etc/apt/trusted.gpg. Not to be confused with the related but not very interesting /etc/apt/trustdb.gpg. 

brb@ubuntu16041:~$ gpg --keyserver keyserver.ubuntu.com --recv-keys E084DAB9
gpg: directory `/home/brb/.gnupg' created
gpg: new configuration file `/home/brb/.gnupg/gpg.conf' created
gpg: WARNING: options in `/home/brb/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/home/brb/.gnupg/secring.gpg' created
gpg: keyring `/home/brb/.gnupg/pubring.gpg' created
gpg: requesting key E084DAB9 from hkp server keyserver.ubuntu.com
gpg: /home/brb/.gnupg/trustdb.gpg: trustdb created
gpg: key E084DAB9: public key "Michael Rutter <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
brb@ubuntu16041:~$ ls -l ~/.gnupg
total 20
-rw------- 1 brb brb 9398 Nov 27 09:31 gpg.conf
-rw------- 1 brb brb 1531 Nov 27 09:31 pubring.gpg
-rw------- 1 brb brb    0 Nov 27 09:31 pubring.gpg~
-rw------- 1 brb brb    0 Nov 27 09:31 secring.gpg
-rw------- 1 brb brb 1200 Nov 27 09:31 trustdb.gpg

brb@ubuntu16041:~$ gpg -a --export E084DAB9 | sudo apt-key add -
OK
brb@ubuntu16041:~$ ls -l ~/.gnupg
total 20
-rw------- 1 brb brb 9398 Nov 27 09:31 gpg.conf
-rw------- 1 brb brb 1531 Nov 27 09:31 pubring.gpg
-rw------- 1 brb brb    0 Nov 27 09:31 pubring.gpg~
-rw------- 1 brb brb    0 Nov 27 09:31 secring.gpg
-rw------- 1 brb brb 1200 Nov 27 09:31 trustdb.gpg

brb@ubuntu16041:~$ apt-key list
/etc/apt/trusted.gpg
--------------------
pub   1024D/437D05B5 2004-09-12
uid                  Ubuntu Archive Automatic Signing Key <[email protected]>
sub   2048g/79164387 2004-09-12

pub   4096R/C0B21F32 2012-05-11
uid                  Ubuntu Archive Automatic Signing Key (2012) <[email protected]>

pub   4096R/EFE21092 2012-05-11
uid                  Ubuntu CD Image Automatic Signing Key (2012) <[email protected]>

pub   1024D/FBB75451 2004-12-30
uid                  Ubuntu CD Image Automatic Signing Key <[email protected]>

pub   2048R/E084DAB9 2010-10-19 [expires: 2020-10-16]
uid                  Michael Rutter <[email protected]>
sub   2048R/1CFF3E8F 2010-10-19 [expires: 2020-10-16]

/etc/apt/trusted.gpg.d/peterlevi_ubuntu_ppa.gpg
-----------------------------------------------
pub   1024R/A546BE4F 2012-06-28
uid                  Launchpad PPA for Peter Levi

Note that the 3 commands we have use (gpg for import, gpg for export from your keyring, apt-key for adding) can be combined into one apt-key command). See R installation.

GPG key error: If the machine is behind a proxy, we may get the following error. See this post. 

gpg: requesting key E084DAB9 from hkp server keyserver.ubuntu.com
?: keyserver.ubuntu.com: Connection refused
gpgkeys: HTTP fetch error 7: couldn't connect: Connection refused
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0

Check if a ppa repository exists

For example, consider the ppa:webupd8team repository. 

if [ $codename == "trusty" && ! find /etc/apt/sources.list.d/* -iname *.list | xargs cat | grep webupd8team ]; then
  add-apt-repository ppa:webupd8team/java
fi

How to know if there are updates available?

http://askubuntu.com/questions/457874/how-to-know-if-there-are-updates-available

Type 'software update' in the Dash. It will launch Software Updater and also check for updates.

"Failed to download Package Files" - Software Updater

One solution is to run the following command first 

sudo apt-get update && sudo apt-get upgrade

and then run the software updater. Usually it requires the computer to restart.

I personally adjust the frequency of notification by choosing 'Every two weeks' for Automatically check for updates, etc.

apt-get upgrade vs apt-get dist-upgrade

My experience aligns with the statement: dist-upgrade is more likely to break stuff badly than upgrade.

Troubleshooting

Grub2 cannot boot after timeout

There is not /etc/default/grub with GRUB2. 

sudo apt-get --reinstall install grub-pc

The disk drive for /tmp is not ready yet

sudo mv /tmp /tmp_old
sudo mkdir /tmp
sudo chmod 1777 /tmp

Software updater - “Failed to download package files” error?

Change the download location to 'Main Server' or another server close to the country you live in and try to sudo apt-get update again.

Use parted command to format a new hard disk

sudo apt-get install parted 

sudo fdisk -l /dev/sdb  # find out the disk size

sudo parted
  print
  select /dev/sdb
  mklabel myLabel
  mkpart primary ext4 0GB 128GB
  print
  quit
lsblk
sudo mkfs.ext4 /dev/sdb1

mkdir /mnt/newdisk
sudo mount /dev/sdb1 /mnt/newdisk
df -h

Gparted

It is best to use the latest release of GParted https://github.com/GNOME/gparted/.

The gparted version on Mint 17.2 is 0.18.0 (Feb 19, 2014) while the current one is 0.28.1 (Feb 17, 2017).

After I create a bootable USB drive, the partition format is fat32 (I use an old version of GParted to format the drive as fat32, and then I use unetbootin to create the bootable USB drive).

If we want to use unetbootin, the USB drive has better to be pre-formatted as fat32 first for unetbootin to detect it. Once unetbootin detect the drive, we can use gparted to reformat it as NTFS before going to burn the Windows bootable USB drive. Read How do I use Unetbootin to make a bootable Windows USB installer?.

gparted on a 4TB disk

Get an error when I tried to create partitions on a 4T disk. Search 4TB gparted sectors exceeds the msdos-partition-table-imposed maximum of 4294967295

It seems it is necessary to use GPT instead of MBR/msdos as a partition table to overcome 2TB limitions.

It is also a good idea to use a live gparted os since the one in Ubuntu may not be up-to-date and gparted takes forever to scan devices. When boot from live USB, we need to turn off the Ext hard disk first.

Step1. Device -> Create a partition table -> GPT

Step2. Create a new ext4 partition as you want.

Note:

  • The 4TB hard disk can be recognized and used normally in an internal hd in Ubuntu 12.04.
  • The 4TB hard disk can also be used in Windows 7 as an external hd if I formatted it (e.g. from gparted in Linux) as an NTFS partition. The Disk Management (command prompt -> diskmgnt.msc) shows it has 3726.02 GB (3726.02 * 1024 * 1024 * 1024 = 4.000784e12 Bytes) capacity and the windows manager shows it is 3.63TB (3726.02/1024=3.638) total space.
  • It cannot be used as 4TB in the case when I use a docking station in Dell Precision T3500. Unsolved problem:
    • Running the command chown from root to user takes forever on 4T partition.
    • The ext dock station (StarTech) will halt the shutdown until I power off the station?

Conclusion:

  • 4TB using NTFS works on Ubuntu.
  • rsync will not stop spinning for some reason even the command is finished (through StarTech dock station).

Below is a screenshot I got from Gparted on a pre-formatted (NTFS) 4T portable drive from Seagate. It seems not harmful because I can still umount, change labels, etc on the disk.

Gparted gpt.png

tracker-miner-fs

See http://askubuntu.com/questions/346211/tracker-store-and-tracker-miner-fs-eating-up-my-cpu-on-every-startup how to disable it.

Health check of the hdd

I got an input/output error when I use sudo rm, sudo reboot or Ctrl + Del commands.

When I use the power button to force shutdown, I could not boot again. The BIOS does find the hdd and the Ubuntu Live USB does find the internal hdd too.

To force to reboot/shutdown, follow the suggestion here

dmesg command shows there are a few bad sectors on that hdd.

http://www.howtogeek.com/howto/37659/the-beginners-guide-to-linux-disk-utilities/ shows a few ways to run a health check on the hdd. The gnome disk utility cannot run S.M.A.R.T. on the external hdd. 

sudo badblocks -v /dev/sdb1
sudo badblocks /dev/sdb > /home/zainul/bad-blocks
sudo fsck -l bad-blocks /dev/sdb

This article http://linoxide.com/linux-how-to/how-to-fix-repair-bad-blocks-in-linux/ talks about how to fix/repair bad blocks in Linux .

This article http://unix.stackexchange.com/questions/25902/what-does-this-hard-disk-error-message-mean-current-pending-sector-count talks about how to do with bad sectors.

It is an indicator that hdd is going to die http://www.linuxquestions.org/questions/linux-hardware-18/34-bad-blocks-what-should-i-do-927224-print/.

Burn/Write an iso or img file to a USB flash drive

Official instruction on www.ubuntu.com and from wikipedia.

Use dd

First, get to know the USB drive device name like /dev/sdb. When using 'dd', the USB drive has to be unmounted (using 'umount' command, not click 'reject' button in File Manager). Note that this will irrevocably destroy all data on /dev/sdX.

The instruction can be found in a lot of places like Archlinux wiki page. 

sudo fdisk -l
sudo dd bs=4M if=xxx.img of=/dev/sdb && sync

where /dev/sdb is a device name, not a partition name. We can also adjust bs to a smaller value like 1M, 4m. 

sudo pkill -USR1 -n -x dd

For some reason when I use dd to create ubuntu 14.04 on usb drive, sudo gparted also gives me a Libparted warning /dev/sdc contains GPT signature, indicating that it has a GPT table. However, it does not have a valid fake msdos partition table, as it should... Is it a GPT partition table? messsage. If I click 'Yes', Gparted shows no partition on the usb drive??? Nevertheless, the usb drive can be used to boot into ubuntu 14.04.

In another case, the gparted compalins the usb drive "Invalid partition table - recursive partition on /dev/sdb". Someone suggests to issue a command 

sudo dd if=/dev/zero of=/dev/sdb bs=4M

Etcher - cross platform

For Ubuntu, there is no need to install the program. Just run the binary file (.AppImage).

See https://www.raspberrypi.org/magpi/pi-sd-etcher/

Rufus - Windows

UNETBOOTIN - cross platform

For creating a Windows bootable USB drive, we cannot use Etcher program. On UDoo-X86 Get started site, it suggests to use Unetbootin if the host machine is Ubuntu. NOTE. the USB drive has to be formatted as FAT32 (this can be done by GParted program); otherwise Unetbootin cannot recognize the drive. Unfortunately the USB drive can not be booted from UDoo-X86. See also the trick by this post.

  • The command sudo mount /dev/sdc1 /mnt should be sudo mkdir /media/$USER/usb; sudo mount /dev/sdc1 /media/$USER/usb where /dev/sdc1 should be changed appropriately
  • This approach works
  • Rufus is good but not always (eg the USB drive is not bootable??)

The GUI is written by Qt so the program is cross-platform. See its wiki.

The following is a screenshot of the contents of xubuntu 12.04. The usb drive needs to be formatted to fat32 on Windows OS to repair partition table error. The partition table error was discovered when I use sudo gparted program to view the USB drive.

Note that Unetbootin (Windows & Linux) and Universal USB installer (Windows only) are quite similar although Universal USB installer provides more options in its interface while Unetbootin does not have any other options.

XUbuntu12044 USBdrive.png XUbuntu gparted.png

Universal USB Installer/UUI

http://www.pendrivelinux.com/universal-usb-installer-easy-as-1-2-3/

I first used dd command to successfully created a xubuntu 14.04 usb drive but there seems to be a problem with the partition since the gparted program will give a warning message about that and also the whole 16GB was used when I insert the drive onto a Windows machine.

Note that the fdisk utility cannot handle this new partition format so we have to use the gparted program.

Open the gparted program. Use it MULTIPLE TIMES to create a partition table (Device -> Create Partition Table...). Then we create a FAT32 partition with all of the space. After that, we can use the UUI program to create an Ubuntu USB drive.

The following screenshots are from a 16GB USB drive.

UUIa.png UUIb.png

Multiple boot USB

YUMI

YUMI works much like Universal USB Installer, except it can be used to install more than one distribution to run Live from your USB.

It can be used to create a Multiboot USB Flash Drive containing multiple operating systems, antivirus utilities, disc cloning, diagnostic tools, and more.

MultiBootUSB

The program is included by LXLE.

It supports persistence up to 4GB for ubuntu and its derivatives.

It also supports multi-thread (check by top or htop). The %cpu > 100.

It will take space as needed. So we can still use the USB drive to write data.

For some reason, the USB drive could not be boot after I use the program. The ubuntu does not show/recognize it though gparted still finds it. Maybe it is because the partition format (ext4 by gparted) is right. For YUMI program, it says to use fat16/fat32/NTFS; otherise syslinux will fail. But it seems not to help:( Maybe it is the partition table (I choose gpt instead of the default msdos).

Determine/install/switch Window Manager

sudo apt-get install wmctrl
wmctrl -m           

sudo apt-get install <pkg-name>
<pkg-name> --replace

On Ubuntu the default window manager is Compiz, for xubuntu it is Xfwm4 and for BBB it is Openbox.

Add date and time to the clock indicator on the top panel

Method 1. GUI approach. Right click the clock indicator and choose Time and Date Settings. Click on 'clock' tab and check 'date and month' option.

Method 2. Command line approach. 

gsettings set com.canonical.indicator.datetime show-date true

Jenkins

How to Install Jenkins Automation Server with Apache on Ubuntu 16.04

Automatic update

http://www.howtogeek.com/228690/how-to-enable-automatic-system-updates-in-ubuntu/

After running update/upgrade in Ubuntu

Virtualbox

Virtualbox does not work. After initial fix, the guest machine cannot connect to internet:(

Crashes network manager (no internet connection, no applet)

The solution on here works on my Ubuntu 14.04.4. Download 3 deb files and install them (downgrade packages).

Graphics driver

GPU info 

$ lspci -vnn | grep VGA -A 12
# OR
$ lshw -numeric -C display

To check hardware acceleration 

# If glxinfo is not find, run sudo apt-get install mesa-utils
$ glxinfo | grep OpenGL

Under Ubuntu-Unity, we can search Additional Drivers to install propriety party drivers instead of using X.org.

On Dell T3600, it shows (pay attention to the line driver=nouveau). See also Installing Nouveau on your Linux computer. 

$ sudo lshw -C video
[sudo] password for brb:
  *-display
    description: VGA compatible controller
    product: GF108GL [Quadro 600]
    vendor: NVIDIA Corporation
    physical id: 0
    bus info: pci@0000:03:00.0
    version: a1
    width: 64 bits
    clock: 33MHz
    capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
    configuration: driver=nouveau latency=0
    resources: irq:82 memory:ee000000-eeffffff memory:f0000000-f7ffffff memory:f8000000-f9ffffff ioport:e000(size=128) memory:ef000000-ef07ffff

If I change to use the nvidia driver on T3600 which has NVIDIA Quadro 600, the GLX error will happen (OpenGL related. Some programs like Qt Creator, Blender will not be able to run). See the detail.

AMD vs NVIDIA

Should You Use AMD or NVIDIA GPUs on Linux?

System program problem detected

sudo rm /var/crash/*

Change value of "enabled" from 1 to 0. Then reboot.

Autostart

http://www.howtogeek.com/228467/how-to-make-a-program-run-at-startup-on-any-computer/

Look at the hidden directory ~/.config/autostart/. Add .desktop files to here to have programs start automatically at startup. These .desktop files are application shortcuts — you can often create them by dragging-and-dropping an application onto your desktop or even just into the ~/.config/autostart/ window.

If you’re not using a desktop environment check out ~/.bash_profile file.

Take screenshots (and edit them)

https://wiki.archlinux.org/index.php/taking_a_screenshot

gnome-screenshot (preinstalled in Ubuntu)

A comprehensive guide to taking screenshots in Linux using gnome-screenshot 

# Current window
gnome-screenshot -w

# an area
gnome-screenshot -a

# delay
gnome-screenshot –delay=[SECONDS]

# interactive mode
gnome-screenshot -i

# directly save your screenshot
gnome-screenshot –file=[FILENAME]

# copy to the clipboard
gnome-screenshot -c

Shutter

# https://launchpad.net/~shutter/+archive/ubuntu/ppa
sudo add-apt-repository ppa:shutter/ppa
sudo apt-get update
sudo apt-get install shutter
  • After we launch Shutter, the tool sits at the (upper-right) Ubuntu system tray
  • The built-in editor (click Edit button on the rhs) is convenient but limited (GIMP is more powerful but complicated). Tools includes
    • Select item to move or resize it
    • Draw a freehand line
    • Highlighter
    • Draw a straight line
    • Draw an arrow
    • Draw a rectangle
    • Draw an ellipse
    • Add a text
    • Censor portions of the screenshot
    • Pixelize selected areas
    • Crop
  • screenshots
  • Each taken screenshots are placed in different tabs in Shutter
  • If we close Shutter, the unsaved screenshots are still there because by default it automatically save files in the ~/Pictures folder. We can modify the settings from Edit -> Preferences.
  • It also supports 'cropping'. Click 'Edit' button, Tool -> Crop, Select a region, Click 'Crop' button on the RHS.

GIMP

File -> Create -> Screenshot

import

command from ImageMagick

scrot

from scrot package. Note that by default the title bar will not be included.

It seems scrot is better for my need.

Note: there seems no way to copy the screenshot to the clipboard. 

scrot -c -d 5 screenshot.png     # -c is count down, -d is delay

scrot -b -c -d 5 -u screenshot.png  # -b is title bar, -u is the current foc'u'sed window

scrot -s screenshot.png          # select an area

# sudo apt-get install mirage    # 105 kB
mirage screenshot.png            # view the image

scrot -b -d 5 '%Y:%m:%d:%H:%M:%S.png' -e 'mv $f ~/Desktop/'

ScreenCloud

Hotshots

Only supports Ubuntu up to 14.04. 

sudo add-apt-repository ppa:ubuntuhandbook1/apps
sudo apt-get update
sudo apt-get install hotshots

Pinta, mtPaint, MyPaint

Paint.NET_like_program

Screencaster/Record desktop

kazam

Although Kazam can do screenshots, Shutter (can sit on the system tray) is more convenient for taking care of screenshots.

The default frame rate is only 15. If we want to increase it, go to File -> Preferences -> Screencast tab.

When recording, it will have 5 seconds (adjustable) to wait. After launching Kazam, it will show an icon (video recorder) on the top-right corner. Keyboard shortcuts are available. For example (Windows key=Super key),

  • Record=Ctrl + Windows + r,
  • Finish=Ctrl + Windows + f,
  • Pause=Ctrl + Windows + p.
sudo apt-get install kazam

You can choose fullscreen, window or a specific area. However, if you want to change the window or area once you have chosen one, you have to restart the program. One nice thing with Kazam is the output video is in .mp4 format (not ogv or ogg format). For a 14-seconds video with 15 frames per second (default), the video file size is 1.4MB.

A good introduction Create Screencast Videos With Ease Using Kazam

SimpleScreenRecorder (Qt based)

source code and this article 

sudo add-apt-repository ppa:maarten-baert/simplescreenrecorder
sudo apt-get update
sudo apt-get install simplescreenrecorder

VokaScreen

It is used in youtube videos of QML tutorials.

Istanbul

Saved files are in the ogg format. 

sudo apt-get install istanbul

RecordMyDesktop

http://www.youtube.com/watch?v=A0Tn3Z8OklQ.

    • The recorded video is in the ogv format.
    • It can be run from the command line.
    • We need to run ffmpeg to convert video to flv (Quality seems to be reduced) OR we can use online service (http://video.online-convert.com/convert-to-mp4) to convert ogv file to mp4 file (Same quality as I can tell).
sudo apt-get install recordmydesktop gtk-recordmydesktop

Create animated Gif of a screencast

Wallpaper

How to Use Bing’s Background of the Day as Your Ubuntu Wallpaper

Conky

Conky is a free, light-weight system monitor for X, that displays any information on your desktop.

Step 1. Install conky-all package

Step 2. create ~/.conkyrc file. This file can be downloaded from web.

Step 3. Run it: $ conky. If we want to run a specific configuration file, use conky -c CONKYRCFILE

Step 4. If you want to stop Conky: $ killall conky

Note that conky works automatically on Ubuntu's Unity.

For Lubuntu (tested on 14.04), the Conky's transparent function does not work at first. But This conkyrc works on Lubuntu desktop (mainly, tranparent function). To deal with the autostart, follow the suggestion from askubuntu.com. That is, go to ~/.config/autostart folder, create or copy+paste the file called conky.desktop with a content like 

[Desktop Entry]
Type=Application
Exec=sh "/home/brb/.conky/conky-startup.sh"
Hidden=false
NoDisplay=false
X-GNOME-Autostart-enabled=true
Name[en_IN]=Conky
Name=Conky
Comment[en_IN]=
Comment=

and 

brb@brb-VirtualBox:~$ cat .conky/conky-startup.sh 
conky &
exit 0
brb@brb-VirtualBox:~$ ls -l .conky/conky-startup.sh 
-rw-rw-r-- 1 brb brb 37 Aug 30 20:17 .conky/conky-startup.sh

Another way to configure conky is to install conky-manager. See this and project website page. But it seems it does not work well with desktop wallpaper.

What should I do when Ubuntu freezes?

Press Alt+Print and then type 'REISUB' (not work, it only does screenshot)

Customize the desktop

  • Install Cairo-Dock.

Remove overlay scroll bar

http://www.itworld.com/article/2698420/disable-overlay-scroll-bars-in-ubuntu-14-04.html

Change scroll bar color

See this post. Run sudo apt-get install gnome-color-chooser.

Go to Engines tab in gnome-color-chooser and choose clearlooks engine for scrollbars.

How to turn off/disable Compiz's “drag to maximize” behaviour?

http://askubuntu.com/questions/72452/how-to-turn-off-compizs-drag-to-maximize-behaviour

On Mint, go to Preferences -> Window Tiling -> Enable Window Tiling and snapping -> Off.

Grub2

To show the grub2 screen, run 'sudo nano /etc/default/grub' and comment out the line GRUB_HIDDEN_TIMEOUT=0 and change the line GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" to GRUB_CMDLINE_LINUX_DEFAULT="". Remember to run 'sudo update-grub' after any change to grub.

To add a splash image, follow the instruction at https://help.ubuntu.com/community/Grub2/Displays. Note that Grub2 will search the image based on some priority and there are also some minor requirements on the images. To test 

sudo apt-get install grub2-splashimages
sudo cp /usr/share/images/grub/Moraine_Lake_17092005.tga /boot/grub/
sudo update-grub

Network Manager

DNS problem and 127.0.1.1

(Ubuntu 16.04 on Odroid) For some reason, pinging my domain always goes to the wrong IP. If I go to System -> Administration -> Network to change DNS from 127.0.1.1 to 8.8.8.8, it fixes the problem. But if I reboot, the DNS entry goes back to 127.0.1.1 again.

When I follow this post nameserver 127.0.1.1 in resolv.conf won't go away, the DNS will be changed to local IP of my router and the problem is fixed (even I reboot the computer).

See also the next: How to flush the DNS cache

How to flush the DNS cache

sudo /etc/init.d/dns-clean

NM-applet

https://wiki.archlinux.org/index.php/NetworkManager

Turn on/off wifi adapter

The command-line equivalent of unchecking the nm-applet's 'Enable Wi-fi' should be 

nmcli nm wifi off
# OR
nmcli radio wifi off
# OR
sudo ifconfig wlan0 down

Proxy

Internet Shut Down

DHCP Server

How to Install a DHCP Server in Ubuntu and Debian

File Server

Network File System (NFS)

NFS allows a system to share directories and files with others over a network. By using NFS, users and programs can access files on remote systems almost as if they were local files.

Some of the most notable benefits that NFS can provide are:

  • Local workstations use less disk space.
  • There is no need for users to have separate home directories on every network machine. Home directories could be set up on the NFS server and made available throughout the network.

Server part: 

sudo apt-get install nfs-kernel-server

Configure the directories to be exported by adding them to the /etc/exports file. 

/home/USERNAME/SHAREFOLDER 192.168.1.0/24(rw,sync,no_subtree_check)
/ubuntu  *(ro,sync,no_root_squash)
/home    *(rw,sync,no_root_squash)
# replace * with one of the hostname formats.

where 'ro' means read only. See Linux Home Server Howto. The no_root_squash option will not prevent root on a client machine from writing files to the server as root; by default, NFS will map any requests from root on the client to the 'nobody' user on the server. See digitalocean.

To start the NFS server: 

sudo service nfs-kernel-server start

Client part:

PS.

  • There is no need to enter any password related to the NFS server.
  • See digitalocean for other ways to specify the arguments in </etc/fstab>.
sudo apt-get install nfs-common

sudo mount example.hostname.com:/ubuntu /mnt/ubuntu
# The mount point directory /mnt/ubuntu must exist.
# or modify the /etc/fstab file 
# example.hostname.com:/ubuntu /mnt/ubuntu nfs rsize=8192,wsize=8192,timeo=14,intr
# localip:/sharedfolder /mnt/ubuntu nfs defaults 0 0

brb@brb-P45T-A:~$ ps -ef | grep nfs
root       675     2  0 11:16 ?        00:00:00 [nfsiod]
root     14783     2  0 16:51 ?        00:00:00 [nfsv4.0-svc]
brb      14960 13491  0 16:56 pts/0    00:00:00 grep --color=auto nfs

To make the mounting permanently, run sudo nano /etc/fstab and include a line like 

1.2.3.4:/home    /mnt/nfs/home   nfs auto,noatime,nolock,bg,nfsvers=4,intr,tcp,actimeo=1800 0 0

How to configure NFS Server and Client Configuration on Ubuntu 16.10

Boot from an NFS server

With NFS booting, the core kernel and file systems are kept on a central server and then pushed out onto client systems to be booted on there. That means your files and desktop will always be available wherever you want to log in.

At the end, the client computer does not need any internal storage. Cool!

CIFS (Common Internet File System) and NTLMv2 (NT LAN Manager)

Note that this method is most suitable for transferring small files. Users transferring large amounts of data to and from Helix/Biowulf should continue to use scp or sftp. (nih.gov)

Samba - allows Linux to transfer files with Windows clients

$ ps -ef | grep smbd    # see if the Samba daemon (smbd)
$ ps -ef | grep nmbd    # see if the NetBIOS name server daemon (nmbd) is running

$ sudo service smbd stop  # does not stop nmbd 
$ sudo service nmbd stop

$ sudo service smbd start
$ sudo service nmbd start
sudo apt-get install samba samba-common
sudo apt-get install python-glade2
sudo apt-get install system-config-samba

Use Dash and search for 'samba'. It will ask for the user's password first. The samba password can also be set by 

sudo smbpasswd -a USERNAME

A non-gui way to configuration samba is adding the following to the end of /etc/samba/smb.conf file, sudo nano -w /etc/samba/smb.conf (-w means no-wrap). Any line beginning with a semicolon (“;”) or a hash (“#”) character is ignored. 

[brb]
        path = /home/brb
;       writeable = no
;       browseable = yes
        guest ok = yes

smb://192.168.1.3/

On Windows PC, go to start and open 'Run' then enter ip with double backslash. Like this (\\192.168.1.3).

Remove and re-install Samba

$ sudo apt-get remove --purge samba
$ sudo apt-get remove --purge smbclient libsmbclient

$ sudo apt-get install samba
$ sudo apt-get install smbclient libsmbclient

SambaCry vulnerability and check Samba version

http://pcworld.com/article/3199106/linux/the-sambacry-scare-gives-linux-users-a-taste-of-wannacry-petya-problems.html

To check your samba version 

$ smbd -V
Version 4.3.11-Ubuntu

Change the default session when using auto login

See this post. We need to edit the file /etc/lightdm/lightdm.conf. Note that on my Ubuntu 12.04, I have the following desktop options. 

$ ls -lt /usr/share/xsessions/
total 16
-rw-r--r-- 1 root root 205 Apr 19  2012 gnome.desktop
-rw-r--r-- 1 root root 188 Apr 19  2012 gnome-shell.desktop
-rw-r--r-- 1 root root 208 Apr 19  2012 ubuntu-2d.desktop
-rw-r--r-- 1 root root 185 Apr 19  2012 ubuntu.desktop

Debian root user from remote access

When you login by SSH, then use the username you have chosen when you installed Debian as the root user is disabled for remote logins. Then run the command "su" to become root user. See howtoforget.com.

Terminal

Directory color on ssh

When I ssh to biowulf from Gnome/Terminal, the folder color became dark blue which is hard to read when the background color is black. A solution is https://serverfault.com/questions/137649/changing-terminal-colors-in-ubuntu-server.

1. append this to your ~/.bashrc 

    if [ "$TERM" != "dumb" ]; then
      [ -e "$HOME/.dir_colors" ] && 
      DIR_COLORS="$HOME/.dir_colors" [ -e "$DIR_COLORS" ] ||
      DIR_COLORS="" 
      eval "`dircolors -b $DIR_COLORS`" 
      alias ls='ls --color=auto'
    fi

2. create/edit your ~/.dir_colors, for example with 

  $ dircolors --print-database > .dir_colors

3. Then force a read of your .bashrc file with: 

  $ source ~/.bashrc

Everything should be pretty.

Remember terminal tabs

The trick on this post works for me on my GNOME Terminal 3.6.2 (gnome-terminal --version). 

# To save configuration into /home/$USER/terminal.cfg:
gnome-terminal --save-config=/home/$USER/terminal.cfg

# To load it back:
gnome-terminal --load-config=/home/$USER/terminal.cfg

To recall the titles, follow this simple hack. That is, add an entry Title=xxxx to each tab section.

Terminal tab color

If we open several tabs in the (GNOME) Terminal, the default color of the active tab is not quite different from the other tabs. We need a solution to change the tab colors.

http://askubuntu.com/questions/355297/gnome-terminal-tabs-no-contrast-between-active-and-inactive-tabs

Solution: edit ~/.config/gtk-3.0/gtk.css (you might have to create it) and add: 

TerminalWindow,
TerminalWindow.background {
        background-color: #6e6e6e;
        color: #000000;
}

TerminalWindow .notebook tab {
        padding: 2;
        background-color: #6e6e6e;
}

TerminalWindow .notebook tab:active {
        background-color: #d1d1d1;
}

Then close ALL terminal windows start and test.

How to practically use your Linux terminal for everything

Install Microsoft Font in linux suite

PCWorld.com

Install language packs

sudo apt-get install language-pack-[cod] language-pack-gnome-[cod] language-pack-[cod]-base language-pack-gnome-[cod]-base

For example, [cod]=en or [cod]=zh.

Change locale language and character set

Display Chinese character (from vanilla Debian/Ubuntu system)

sudo apt-get install fonts-arphic-ukai fonts-arphic-uming

sudo apt-get install language-support-fonts-zh
# Or
sudo apt-get install ttf-arphic-uming ttf-wqy-zenhei

Chinese Input

Zhuyin-keyboard.gif

http://pinyinjoe.com/linux/ubuntu-12-chinese-setup.htm

Note: If ibus does not have chewing selection, we should install it first. 

sudo apt-get install ibus-chewing
  1. type 'language' in Dash search (Or click Power button on the top-right corner -> System Settings...) and click 'Language Support' (this may not be installed from (x)Ubuntu automatically. In such case, IBUS will be missing eg Chewing method). The 'Language Support' is under Settings menu in xUbuntu.
  2. choose 'IBUS' for keyboard input method. IBUS is under 'System' menu in xUbuntu.
  3. Click 'Install/Remove Languages'. Choose Chinese -> Chewing. Note that Chewing is not the same as Bopomofo. I cannot get used to Bopomofo method.
  4. Settings > Keyboard Input Method > Input method > Select an input method > Show only input methods for your region > Chinese 酷 Chewing. If we cannot find this dialog, we can launch it by ibus-setup command. If Chewing is not shown as one of Chinese input methods, we can log out and log in the desktop.
  5. Click 'Close' button.

Log out and Log in. Click Keyboard icon on top right corner. Modify its Preferences. Go to 'Input Method' tab. Add Chinese-Chewing and click 'Close' button. We can use Ctrl+Space to switch language input.

On Ubuntu 13.10, type 'language support' in the Dash. Click 'Install/Remove Languages' and check Chinese. In the 'Keyboard input method system', change from default to 'IBus'. We shall see a keyboard icon on the top right of menu. Click it and choose Text Entry Settings. Add Chinese Chewing input. Then change 'switch to next source using' method to 'Ctrl + Space' by pressing both keys on your keyboard. Bingo!

On Ubuntu 14.04, the super key is Windows.

Note that do not use 'Text Entry' application (shown as one options when we search 'language'). This application will conflict with ibus.

If for some reason, ibus-daemon was not started automatically, try the approach here by adding ibus-daemon to the startup application list.

/usr/lib/ibus/ibus-ui-gtk3 high memory usage

For some reason, the ubuntu 14.04 is not responsive. Htop shows ibus-ui-gtk3 is the culprit. The suggestion is to run 

killall ibus-daemon

After I run this command and wait a few seconds, the memory leak problem is solved.

Change time zone

http://wiki.debian.org/TimeZoneChanges 

$sudo dpkg-reconfigure tzdata

The command launches an ncurses based interface which allows arrow keys to be used to select the region specific time zone.

Make script run at boot time with init.d directory

http://www.debian-administration.org/articles/28

For example, see here from running a python script for raspberry pi.

  1. Create a script /etc/init.d/lcd
  2. Make the script executable
  3. Make the script known to the system by using the update-rc.d' command
sudo update-rc.d lcd defaults

wireless connection randomly drop off

My wireless adapter is TP-LINK, TL-WN722N.

http://askubuntu.com/questions/73607/wireless-connection-keeps-dropping-with-an-intel-3945abg-card 

sudo iwconfig wlan0 power off

5 Best Linux/Ubuntu compatible USB Wifi cards:

  • AirLink101 AWLL6075 Wireless N Mini USB Adapter
  • Medialink – Wireless N USB Adapter – 802.11n
  • ASUS (USB-N13) Wireless-N USB Adapter
  • Panda Mini Wifi (b/g/n) 150Mbps Card
  • TP-Link TL-WN722N 150Mbps High Gain Wireless USB Adapter

To show (USB) wireless adapter information 

sudo lshw -C network
sudo lsusb -v
iwconfig

To check wifi connection information (SSID, channel, address, frequency, qualiyt, signal level, ...) 

sudo iwlist wlan0 scan

My experience is quality should be at least 50/70 and signal level should be larger than -60dBm.

Create an iso file from a CD or HD

Method 1 (Better than Method 2). No need to umount the cdrom. 

mkisofs -o /tmp/DVD.iso /tmp/directory/

to make an ISO from files on your hard drive.

Method 2. Make sure the cdrom is NOT mounted. Type mount to confirm it. If cd was mouted automatically unmout it with umount command: like umount /dev/cdrom or umount /mnt/cdrom. Note that no extra forward slash after /dev/cdrom for the command below. 

dd if=/dev/cdrom of=~/DVD.iso

We can mount the iso file to a directory to check the iso file content is correct. 

mount -t iso9660 -o loop,ro DVD.iso /mnt

Have fun with /etc/hosts file

su -c "nano /etc/hosts" 

74.125.67.100   DNS_NAME1 DNS_NAME2

Block malware, adware

How to Use Your Computer’s Hosts File to Block Tons of Malware, Porn, and Other Types of Websites

DNS tricks

5 DNS Servers Guaranteed to Improve Your Online Safety

http://www.makeuseof.com/tag/best-dns-providers-security/

5 Nifty Ways to Use DNS to Your Advantage

http://www.makeuseof.com/tag/nifty-ways-use-dns-advantage/

Mount a remote file system over ssh

The trick is to use the sshfs tool.

On Ubuntu 

# Install the program
sudo apt-get install sshfs

# Mount the file system
sudo mkdir /mnt/droplet <--replace "droplet" whatever you prefer
sudo sshfs [email protected]:/ /mnt/droplet

# Unmount the file system
sudo umount /mnt/droplet

# Permanently Mounting the Remote File System
sudo nano /etc/fstab
sshfs#[email protected]:/ /mnt/droplet

Nautilus (File Manager)

Undo Ctrl+L

Press ESC.

Mount another Linux system in Nautilus

Very easy. Check out howtogeek.com

Create a desktop shortcut

Navigate to your application in Nautilus. Right-click, select "Make Link". Then drag shortcut to your desktop. Works in Ubuntu 12.04.

.desktop file format

Open a terminal

sudo apt-get install nautilus-open-terminal
nautilus -q

In Ubuntu 15.10, the functionality is already included in nautilus!

Mount iso file

$ sudo mkdir /mnt/iso
$ sudo mount -o loop /tmp/file.iso /mnt/iso
$ sudo umount /mnt/iso

Check ubuntu version from command line

http://www.howtogeek.com/206240/how-to-tell-what-distro-and-version-of-linux-you-are-running/ 

lsb_release -a
# OR
cat /etc/issue
# OR
cat /etc/*release
# check kernel version
uname -r
# check 32/64 bit kernel
uname -a

keyboard shortcuts

Go to keyboard app to change the settings. Note: Super key is also Windows key. Use 'Backspace' key to disable a shortcut.

  • List from ubuntu.com
  • Super: Open the Dash. Press and hold it to see a cheat sheet with a bunch of other nifty shortcuts.
  • Super + Number: Open the application that is at that position in the dock
  • Alt + Mouse: Move an application. Useful if an application's title bar is too high.
  • Alt + F2: Run an application by typing its name in the box which appears (same function as the Super key).
  • Alt + Space: Activate the window menu. Not useful so I disable it.
  • Ctrl + Super + D: To minimize all windows
  • Ctrl + Alt + Arrows: move to another workspace
  • Ctrl + Alt + Shift + Arrows: move current application to another workspace

On Xubuntu,

  • Alt + F1 (Ctrl + ESC): Application menu.
  • Alt + F2 (Super + R): Application Finder.

Add a new keyboard shortcut

Let's say I want to assign Ctrl+Alt+s to bring the shutdown dialog.

  1. System Settings -> Keyboard -> Shortcuts -> Windows -> Toggle shaded state -> Backspace to disable it. For some reason, I cannot assign a new keybinding using the method of Custom Shortcuts' method.
  2. Open Ubuntu Software Center and install Compiz Config Setting Manager program. Open the program by search Compiz and then create a new command gnome-session-quit --power-off --force with a name like 'Shutdown' and the keyboard binding we want. CCSM -> General -> Commands.

With this approach, it will give a dialog with 4 options (lock suspend restart shutdown) but no one is pre-selected. If we want the shutdown icon being preselected, use this script where the zenity command was used to create a dialog. The zenity program was pre-installed in Ubuntu. See How to Make Simple Graphical Shell Scripts with Zenity on Linux.

Firefox

  • Space: page down
  • Shift + space: page up

BioLinux

Bio-Linux 7.0 (2012/11/21) screenshot.

Biolinux can be installed in two ways.

  1. One is to download iso image file. http://nebc.nerc.ac.uk/downloads/
  2. The other way is to install Bio-linux software/package by using apt-get install method. See http://nebc.nerc.ac.uk/tools/bio-linux/other-bl-docs/package-repository

BioLinux.png

Check Biolinux version

cat /etc/bio-linux.version  # 8.0.5 as of June 2015

Installation

wget -qO- http://nebc.nerc.ac.uk/downloads/bl8_only/upgrade8.sh | sudo sh

This takes a long long time.

Software list

FAQ

Sample Data

Did not find them useful. 

brb@biolinux[brb] ls Desktop/Sample\ Data/                                                         [10:05AM]
act       cytoscape               glimmer3      mrbayes             peptide_seqs       splitstree
artemis   dendroscope             happy         mspcrunch           phylip             squint
blast     dotter                  hmmer         multiple_alignment  qiime              t-coffee
blast+    dust                    jalview       mummer              rasmol             tree-puzzle
blixem    fasta                   jprofilegrid  muscle              rdp_classifier     treeview
cap3      fastDNAml               mesquite      mview               readseq
catchall  forester-archaeopteryx  mira          njplot              samtools
clustal   gap4                    mothur        nucleotide_seqs     sanger_tracefiles
brb@biolinux[brb] ls Desktop/Sample\ Data/fasta/                                                   [10:05AM]
bovgh.seq        hahu.aa       mgstm1.e05           mgstm1.nt1r  myosin_bp.aa  oohu.raa
bovprl.seq       hsgstm1b.gcg  mgstm1.eeq           mgstm1.nts   n0.aa         prio_atepa.aa
egmsmg.aa        hsgstm1b.seq  mgstm1.esq           mgstm1.raa   n1.aa         prot_test.lib
grou_drome.pseg  humgstd.seq   mgstm1.gcg           mgstm1.rev   n2.aa         prot_test.lseg
gst.nlib         lcbo.aa       mgstm1_genclone.seq  mgstm1.seq   n2_fs.lib     qrhuld.aa
gst.seq          m1r.aa        mgstm1.lc            mgtt2_x.seq  n2s.aa        sql
gstt1_drome.aa   m2.aa         mgstm1.nt            ms1.aa       n2t.aa        titin_hum.aa
gstt1_pssm.asn1  mchu.aa       mgstm1.nt1           mu.lib       n_fs.lib      titin_hum.seq
gtm1_human.aa    mgstm1.3nt    mgstm1.nt12r         musplfm.aa   ngt.aa        xurt8c.aa
gtt1_drome.aa    mgstm1.aa     mgstm1.nt13          mwkw.aa      ngts.aa       xurt8c.lc
h10_human.aa     mgstm1.aaa    mgstm1.nt13r         mwrtc1.aa    oohu.aa       xurtg.aa

CloudBioLinux

Apache configuration

https://help.ubuntu.com/11.10/serverguide/httpd.html

Nginx

Run both Nginx and Apache at the same time

https://stackoverflow.com/questions/23024473/how-can-i-run-both-nginx-and-apache-together-on-ubuntu

Install Nginx as Reverse Proxy for Apache

(Excerpt from thegeekstuff) For example, let us say we have an enterprise application that is running on Apache and PHP on app.thegeekstuff.com, and we also have Nginx running on example.com.

In this example scenario, when someone goes to example.com, we can setup Nginx as a reverse proxy so that it will serve the enterprise apache/php application that is running on app.thegeekstuff.com.

But, for the end-user, they’ll only see example.com, they won’t even know anything about app.thegeekstuff.com. End-user will think the whole apache/php application is getting served directly from example.com.

Tomcat

Device manager

By default, ubuntu does not provide any graphical tool like device manager on Windows. A very close one is lshw (hardware lister). A GUI tool based on it is called lshw-gtk (seems not as informative as the command line one) and can be installed by sudo apt-get install lshw-gtk in Ubuntu/Debian or yum install lshw in Red Hat/Fedora/CentOS.

As you can see the line containing 'display' shows the motherboard (P45T-A), CPU (Intel Core 2 Duo E8400), graphical card (GeForce 9400 GT in this case), et al. 

brb@brb-P45T-A:~$ sudo lshw -short
H/W path         Device      Class       Description
====================================================
                             system      P45T-A (To Be Filled By O.E.M.)
/0                           bus         P45T-A
/0/0                         memory      64KiB BIOS
/0/4                         processor   Intel(R) Core(TM)2 Duo CPU     E8400  @
/0/4/5                       memory      64KiB L1 cache
/0/4/6                       memory      6MiB L2 cache
/0/f                         memory      8GiB System Memory
/0/f/0                       memory      2GiB DIMM DDR2 Synchronous 800 MHz (1.2
/0/f/1                       memory      2GiB DIMM DDR2 Synchronous 800 MHz (1.2
/0/f/2                       memory      2GiB DIMM DDR2 Synchronous 800 MHz (1.2
/0/f/3                       memory      2GiB DIMM DDR2 Synchronous 800 MHz (1.2
/0/100                       bridge      4 Series Chipset DRAM Controller
/0/100/1                     bridge      4 Series Chipset PCI Express Root Port
/0/100/1/0                   display     G96 [GeForce 9400 GT]
/0/100/1a                    bus         82801JI (ICH10 Family) USB UHCI Control
/0/100/1a.1                  bus         82801JI (ICH10 Family) USB UHCI Control
/0/100/1a.2                  bus         82801JI (ICH10 Family) USB UHCI Control
/0/100/1a.7                  bus         82801JI (ICH10 Family) USB2 EHCI Contro
/0/100/1b                    multimedia  82801JI (ICH10 Family) HD Audio Control
/0/100/1c                    bridge      82801JI (ICH10 Family) PCI Express Root
/0/100/1c/0      eth0        network     AR8121/AR8113/AR8114 Gigabit or Fast Et
/0/100/1c.3                  bridge      82801JI (ICH10 Family) PCI Express Root
/0/100/1c.3/0    wlan0       network     AR93xx Wireless Network Adapter
/0/100/1c.4                  bridge      82801JI (ICH10 Family) PCI Express Root
/0/100/1c.4/0                storage     JMB361 AHCI/IDE
/0/100/1c.4/0.1              storage     JMB361 AHCI/IDE
/0/100/1d                    bus         82801JI (ICH10 Family) USB UHCI Control
/0/100/1d.1                  bus         82801JI (ICH10 Family) USB UHCI Control
/0/100/1d.2                  bus         82801JI (ICH10 Family) USB UHCI Control
/0/100/1d.7                  bus         82801JI (ICH10 Family) USB2 EHCI Contro
/0/100/1e                    bridge      82801 PCI Bridge
/0/100/1f                    bridge      82801JIR (ICH10R) LPC Interface Control
/0/100/1f.2                  storage     82801JI (ICH10 Family) 4 port SATA IDE 
/0/100/1f.3                  bus         82801JI (ICH10 Family) SMBus Controller
/0/100/1f.5                  storage     82801JI (ICH10 Family) 2 port SATA IDE 
/0/1             scsi0       storage     
/0/1/0.0.0       /dev/sda    disk        250GB Samsung SSD 840
/0/1/0.0.0/1     /dev/sda1   volume      224GiB EXT4 volume
/0/1/0.0.0/2     /dev/sda2   volume      8190MiB Extended partition
/0/1/0.0.0/2/5   /dev/sda5   volume      8190MiB Linux swap / Solaris partition
/0/1/0.1.0       /dev/sdb    disk        2TB ST2000DM001-9YN1
/0/1/0.1.0/1     /dev/sdb1   volume      1863GiB EXT4 volume
/0/2             scsi2       storage     
/0/2/0.0.0       /dev/cdrom  disk        DVDRAM GH24NS90
/1                           power       Nikon Ultra Plus
/2                           power       To Be Filled By O.E.M.

For storage part, ubuntu provides a graphical tool. See "disk utility" on gnome based ubuntu or search for "disk" in launcher.

Set static IP - /etc/network/interfaces

root@debian:~# cat /etc/network/interfaces
auto lo eth0
iface lo inet loopback
iface eth0 inet dhcp

Now edit the file /etc/network/interfaces 

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.1.3
gateway 192.168.1.1
netmask 255.255.255.0
network 192.168.1.0                 # Optional
broadcast 192.168.1.255             # Optional
dns-nameservers 192.168.1.1 8.8.8.8 # Or skip 192.168.1.1

After it, restart the network by issuing 

/etc/init.d/networking restart

OR sudo reboot

Note: It does not work by editing /etc/resolv.conf since this file will be overwritten.

Change IP address from the command line

/sbin/ifconfig eth0 192.168.1.17 netmask 255.255.255.0 up
/sbin/ifconfig eth0

Windows OS.

ufw (uncomplicated firewall)

The default firewall configuration tool for Ubuntu is ufw. Developed to ease iptables firewall configuration, ufw provides a user friendly way to create an IPv4 or IPv6 host-based firewall. By default UFW is disabled.

Suppose I have a virtual machine running a web server at port 8888 (The vm may be initialized by the Vagrant command). I can access the webpage from my host machine using http://localhost:8888.

Now I want the web page to be accessible from other local machines. We can use the ufw command to enable the firewall wall and open ports for certain services. 

sudo ufw allow 8888/tcp
sudo ufw show added
sudo ufw enable
sudo ufw status

nmap localhost

Now I can go to another machine, open a browser at http://hostip:8888/. I should be able to get the same result as I got from the host machine.

In one instance sshing to a server failed (connection time) for some reason. The solution is to run the following command on the server 

sudo ufw allow 22/tcp

A graphical interface program is called Gufw Firewall.

Other things ufw can do:

  • Allow/Deny by ports and (optional) protocols
  • Allow/Deny by service name
  • Disable ping requests.
  • Allow by specific IP
  • Allow by subnet
  • Allow by specific port and IP address
  • Deny by certain IP address
  • Deny by certain IP address and certain port

DNS Server

Protecting Your Privacy With Firefox on Linux

What is my DNS server

/etc/resolv.conf

dig Command Examples

https://www.cyberciti.biz/faq/linux-unix-dig-command-examples-usage-syntax/

dig (domain information groper) is a DNS lookup utility.

DNStracer

http://www.ubuntugeek.com/dnstracer-trace-dns-queries-to-the-source.html

Dyndns and ddclient

See

nano /etc/ddclient.conf 

protocol=namecheap
ssl=yes
use=web, web=dynamicdns.park-your-domain.com/getip
server=dynamicdns.park-your-domain.com
login=yourdomain.com
password=a9438540ba8a449fb0ed09c3737b9e32
@

Note that the specification should depend on the domain name registrar (eg namecheap). For namecheap, the login/password is NOT your actual credential from your domain name registrar. The password should be obtained from the domain name registrar website. The last line is about the host. If I am setting it up for a subdomain, I should enter the subdomain name (and skip the domain name part). The ssl=yes is to ensure the connection is made over https instead of http.

And run sudo ddclient -daemon=0 -debug -verbose -noquiet to verify ddclient is working. You shall get a long return with the last line looks like 

SUCCESS:  updating YOURSUBDOMAIN: good: IP address set to XX.XXX.XXX.XXX

No matter which method we use, we can go to our DNS account (in namecheap, go to Dashboard -> MANAGE button -> Domains -> Advanced DNS) and temporarily change the global IP address to another one, run the update script and then check if the global IP address has been updated to the correct one.

namecheap

Mail

  • check the option of Mail Routing: I have mail server with another name and would like to add MX hostname...
  • In 'MX hostname' entering aspmx.l.google.com
  • In 'Primary' choose 'Yes, use it as my primary mail relay.'

no-ip

Similar to Dyndns. It has its own client program. Needs to build it yourself.

Also see the troubleshooting guide.

See http://ducky-pond.com/posts/12 for instruction of setting autostart on Debian system.

See http://www.coulterfamily.org.uk/pages/PCs/Linux/FAQ-LINUX-NO-IP-CLIENT.php for another approach.

Note: If noip2 cannot start automatically or noip2 does not update even it can be seen from ps -ef command, use sudo crontab -e command. For some reason, after I use sudo crontab, noip2 can update IP. So the only problem right now is it cannot update every 30 minutes even sudo noip2 -S says so. The problems may be 1. ps -ef shows the command runs from nobody user 2. sudo noip2 -S says it updates every 30 minutes via /dev/eth0 with NAT enabled.

Update: An alternative is to use ddclient. However, ddclient never updates the IP.

Mail

  • Allow only one MX record for each host for free no-ip account.
  • Click Host/Redirects > Manage Hosts > Modify.

webmin

See http://www.webmin.com/deb.html 

wget http://prdownloads.sourceforge.net/webadmin/webmin_1.600_all.deb
dpkg --install webmin_1.600_all.deb

The install will be done automatically to /usr/share/webmin, the administration username set to root and the password to your current root password. You should now be able to login to Webmin at the URL http://localhost:10000/. Or if accessing it remotely, replace localhost with your system's IP address.

Virtualize Linux

http://www.linuxuser.co.uk/features/how-to-virtualise-linux-part-1

CPU

$ grep name /proc/cpuinfo | sort -u
model name	: AMD Phenom(tm) II X6 1055T Processor      # home

model name	: Intel(R) Xeon(R) CPU  X7560  @ 2.27GHz    # helix

model name	: Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz  # office

Find out if the OS is running in a virtual environment

Linux adds the hypervisor flag to /proc/cpuinfo if the kernel detects running on some sort of a hypervisor. See here. 

cat /proc/cpuinfo | grep hypervisor

Virtualbox

See here.

VBoxClient: the VirtualBox kernel service is not running

Got the above message on the upper right corner of the screen for every booting. Note I don't install VirtualBox. A solution is run 

gksudo gedit /etc/X11/Xsession.d/98vboxadd-xclient

find the line 

notify-send "VBoxClient: the VirtualBox kernel service is not running.  Exiting."

Now change notify-send to echo.

Thin client

https://help.ubuntu.com/community/ThinClients

UbuntuLTSP

LTSP

Raspberry Pi

Running Raspberry Pi’s as Thin Clients with Ubuntu 14.04 LTS

[https://help.ubuntu.com/community/UbuntuLTSP/RaspberryPi UbuntuLTSP/RaspberryPi ]

Remote desktop

Remote desktop connection from Windows 7

xtightvncserver may not play well in Unity.

Install xrdp on Ubuntu. That's it. See 

sudo apt-get install xrdp
sudo /etc/init.d/xrdp start

On Windows 7, open its Remote Desktop client utility mstsc.exe.

Remote desktop connection to Windows 7 from xubuntu

sudo apt-get update
sudo apt-get install rdesktop
rdesktop xxx.xxx.x.x -f -u USERNAME -p PASSWORD
rdesktop 192.168.1.4 -g 1280x720 -u USERNAME -p PASSWORD

where -f option means full screen and -g means geometry.

Remote desktop connection to NCI

Note the information here is outdated now.

The version 1.7.1 of rdesktop program in Ubuntu 14 has a bug with mouse cursor (invisible). To fix the bug, download the version 1.8.3 and build it. See http://ubuntuforums.org/showthread.php?t=2266743 

sudo apt-get update
sudo apt-get install build-essential libx11-dev libssl-dev libgssglue-dev libpcsclite-dev
tar zxvf rdesktop-1.8.3.tar.gz 
cd rdesktop-1.8.3
./configure
make
sudo make install

After that the new rdesktop is located under /usr/local/bin folder. The old rdesktop installed through apt-get is not deleted.

ts.nci.nih.gov:1494

DOMAIN: NIH

resolution: 1024 x 768

Use remmina or rdesktop or freerdp (sudo apt-get install freerdp-x11) 

rdesktop ts.nci.nih.gov:1494 -d NIH -u XXXXX -g 1024x768

To share a folder from the local machine, use "-r" option 

rdesktop ts.nci.nih.gov:1494 -d NIH -u XXXXX -g 1280x1024 -r disk:remotedisk=/home/$USER/Downloads

The new secure connection relies on the SmartCard is using ncits-p111.nci.nih.gov.

RealVNC

The default vnc server included in Ubuntu is not quite compatible with other clients. For example I can connect to Ubuntu 14.04 from Ubuntu 16.04 but not from a Mac. The 3rd party software is better. See the following screenshot after I have installed RealVNC server.

Realvnc-server.png

Note that RealVNC Server is not free for commercial users but is free for home users up to 5 computers & 3 users; see https://manage.realvnc.com/. An email and a password can be used to sign in your account for activating the license in VNC Server.

To start VNC server, use (https://www.realvnc.com/docs/debian-install-remove.html) 

sudo /etc/init.d/vncserver-x11-serviced start

The RealVNC will have an icon sitting on the tray at the top-right corner.

Before using VNC viewer, we also need to create a VNC password in VNC Server.

On Mac, the viewer can be launched from Applications -> VNC Viewer (if we have drag and drop the app to the Applications folder).

On Chrome OS, there is an VNC Viewer for Google Chrome to use. The IP address is special. For example, 192.168.5.127:80 for port 5980. See https://www.realvnc.com/docs/faq/connect-fail.html. Unfortunately I cannot connect successfully:(

Remote desktop connection from Ubuntu to Ubuntu

  • https://help.ubuntu.com/16.04/ubuntu-help/sharing-desktop.html
  • How to Remote Access to Ubuntu 16.04 from Windows. The instruction is the same for older versions of Ubuntu.
    • On a Ubuntu server. Go to Dash, type 'desktop sharing' and select it. Check sharing. Close the dialog. Open a terminal and run ps -ef | grep vino to make sure the server is running.
    • Disable encryption. Run sudo apt install dconf-editor. Go to Dash and type 'dconf' and select 'dconf editor'. When it opens, navigate to org -> gnome -> desktop -> remote-access, and uncheck the value of 'require-encryption.' Another way is to run gsettings set org.gnome.Vino require-encryption false.
    • On a client machine (eg another Ubuntu), open a remote desktop client program (eg Remmina). Choose VNC as the protocol. Enter necessary information to connect to the server.
  • Install a VNC server on non-Unity desktop

Allow for remote desktop connection

  1. Go to System -> Preference -> Remote desktop. Allow other uses to view your desktop & uncheck you must confirm each access & require user to enter this password.
  2. Go to System -> Preference -> Monitor. Change monitor resolution to 1280 x 720.

Allow for remote desktop connection when vino failed

  • Teamviewer. It works fine. After launching it, the software automatically creates an ID and password. We can change the password so it is fixed. Then launch the software on the client. Use the partner's ID and password to connect to it.
  • NoMachine. I ran the service on my Ubuntu 12.04 server. I also tested the client on my Android 6.0 tablet and Odroid xu4 running Ubuntu 15.10. I am using the version 5.0.63. I need to use my server's user account info to connect.
odroid@odroid:~/Downloads$ ps -ef | grep nx
nx       12168     1  1 21:11 ?        00:00:20 /usr/NX/bin/nxserver.bin root 12151 --daemon
odroid   12199 12168  1 21:11 ?        00:00:12 /usr/NX/bin/nxnode.bin
nx       12225 12168  0 21:11 ?        00:00:00 /usr/NX/bin/nxd
odroid   12263 12199  1 21:11 ?        00:00:13 /usr/NX/bin/nxclient.bin --monitor --pid 1153
nx       15916 12225 24 21:28 ?        00:00:08 /usr/NX/bin/nxserver.bin -c /etc/NX/nxserver --login -H 5
odroid   15944 15916 33 21:29 ?        00:00:04 /usr/NX/bin/nxnode.bin -H 5
odroid   16130  8527  0 21:29 pts/1    00:00:00 grep --color=auto nx

odroid@odroid:~/Downloads$ ls /usr/NX/bin
drivers       nxd         nxkb        nxplayer      nxsh         nxusbd
nxagent       nxesd       nxkeygen    nxplayer.bin  nxspool
nxauth        nxexec      nxlocate    nxpost        nxssh
nxclient      nxfs        nxlpd       nxprint       nxssh-add
nxclient.bin  nxfsm       nxnode      nxserver      nxssh-agent
nxcodec.bin   nxfsserver  nxnode.bin  nxserver.bin  nxtunctl

If for some reason some nx programs disappeared, restart the service by 

sudo /usr/NX/bin/nxserver --restart

It is interesting that nomachine is faster than 'ssh -X' method when I tested running a Qt application launched from Qt Creator. For example, the progress bar is not moving when it is supposed to move forward and backward when the app is launched through 'ssh -X'.

NoMachine and Amazon cloud

https://www.nomachine.com/accessing-your-remote-linux-desktop-on-amazon-elastic-compute-cloud-via-NoMachine

It seems nomachine is using port 4000. 

odroid@odroid:~$ sudo nmap -sV localhost

Starting Nmap 7.01 ( https://nmap.org ) at 2016-11-16 20:10 EST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000051s latency).
Other addresses for localhost (not scanned): ::1
rDNS record for 127.0.0.1: odroid
Not shown: 997 closed ports
PORT     STATE SERVICE         VERSION
22/tcp   open  ssh             OpenSSH 7.2p2 Ubuntu 4ubuntu2.1 (Ubuntu Linux; protocol 2.0)
4000/tcp open  remoteanything?
7001/tcp open  X11             (access denied)
Service Info: OSs: Linux, Unix; CPE: cpe:/o:linux:linux_kernel

odroid@odroid:~$ sudo lsof -i :4000
COMMAND PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
nxd     735   nx    3u  IPv4  19208      0t0  TCP *:4000 (LISTEN)
nxd     735   nx    4u  IPv6  21234      0t0  TCP *:4000 (LISTEN)

X2Go

used in Linux Data Science Virtual Machine by Microsoft.

Remote desktop connection through vmware workstation

We shall be able to remote desktop connect to a Windows guest machine if the guest machine has configured to use bridge connection and a static IP. If there is a problem, it is likely caused by Window's firewall. See the two screenshots. We can just turn off the firewall of home network but keep the firewall on for the public network.

We don't need to use port forward for the remote desktop connection.

ViewActiveNetwork.png Firewall.png

Install sshd

apt-get update
apt-get install openssh-server

vsftpd and virtual users

ProFTPd

How to install ProFTPd with TLS support on Ubuntu 16.04

Install LAMP

See the page [1] 

apt-get install apache2
a2enmod rewrite
apt-get install mysql-server   [I choose branch name as MYSQL root password]
apt-get install php5 php-pear php5-suhosin
apt-get install php5-mysql
/etc/init.d/apache2 restart

Another instruction including how to set up user directories for Apache web server http://wiki.debian.org/LaMp 

apt-get install mysql-server mysql-client
mysql_secure_installation
apt-get install apache2 apache2-doc
apt-get install php5 php5-mysql libapache2-mod-php5
apt-get install python libapache2-mod-python

The apache configuration file is in /etc/apache2/apache2.conf.

For PHP, it is also useful to install php for command line. 

sudo aptitude install php5-cli

Restarting apache before testing on web browser 

/etc/init.d/apache2 restart

Debian 8

https://www.digitalocean.com/community/tutorials/how-to-install-linux-apache-mysql-php-lamp-stack-on-debian-8 

sudo aptitude install apache2 apache2-doc
sudo aptitude install mysql-server php5-mysql
# optional
sudo mysql_secure_installation
sudo aptitude install php5-common libapache2-mod-php5 php5-cli
sudo service apache2 restart

AMPPS for a local server

http://www.howtogeek.com/219983/how-to-use-ampps-to-install-joomla-locally/

XAMPP is a free, open source cross platform web server solution stack package for Windows, Mac, and Linux. AMPPS is a software stack from Softaculous enabling Apache, MySQL, MongoDB, PHP, Perl, Python, and Softaculous auto-installer on a desktop.

Content management systems

ProcessWire

How to Install ProcessWire CMS on CentOS 7

Mediawiki

Install mediawiki using tar ball

Install mediawiki using aptitude 

aptitude install mediawiki php5 apache2 mediawiki-extensions libapache2-mod-php5

This will install latex. After the end, we can use Synaptic package manager to see what were installed. Now following the instruction in https://help.ubuntu.com/community/MediaWiki, we remove the '#' from the third line so that it reads 'Alias /mediawiki /var/lib/mediawiki': 

sudo nano /etc/mediawiki/apache.conf
sudo /etc/init.d/apache2 restart

Now we can start mediawiki by opening a browser and pointing it to http://localhost/mediawiki.

We need to enter 

Site config
  admin username: WikiSysop
  password:

Database config
  Database name: wikidb
  DB username: 
  DB password: 
  Superuser name: root
  Superuser password: [depend on how it was chosen when installing MYSQL]

Press the button of "Installing mediawiki". We will be welcomed to the wiki page. Following the instruction,

For security reason, I remove new account creation and anonymous editing. I also remove edit counters.

Backup and Restore mediawiki: The process involves 3 parts: mediawiki system, mysql and images. See docs.google.com note.

Install moinmoin

UpnP server

minidlna

http://bbrks.me/rpi-minidlna-media-server/

It works even I use my phone to tether data (I don't need to turn on wifi on my phone). 

sudo apt-get update
sudo apt-get install minidlna
sudo nano /etc/minidlna.conf # the default location of media files is on /var/lib/minidlna

# rebuild the database. See the comments in <etc/minidlna.conf>
sudo service minidlna force-reload 
sudo service minidlna start
sudo update-rc.d minidlna defaults # ask minidlna to start up automatically upon boot.

Too bad is when I played certain videos the program crashed. The /var/log/syslog showed kernel: [96495.690373] minidlna[1627]: segfault at 0 ip 00007f4af2de9964 sp 00007fffa43014f8 error 4 in libc-2.15.so[7f4af2d54000+1b4000]. Also the minidlna process becomes 2 instead of 1 after the crash.

Kodi

Too bad the Kodi's upnp function is not stable. Kodi server disappeared so the client cannot find it.

PLEX

This seems to be the best from my test. 

sudo add-apt-repository ppa:stebbins/handbrake-releases
sudo apt-get update
sudo apt-get install handbrake-gtk
sudo apt-get install handbrake-cli

Storage server GlusterFS

https://www.howtoforge.com/tutorial/high-availability-storage-with-glusterfs-on-debian-8-with-two-nodes/

Security

https connection

HTTPOXY

https://www.howtoforge.com/tutorial/httpoxy-protect-your-server/

Fail2Ban, Tinyhoneypot and IPv4 security

https://www.howtoforge.com/tutorial/increase-ipv4-security-with-fail2ban-and-tinyhoneypot-on-debian-jessie/

MYSQL security

Just execute mysql_secure_installation from the command line.

  • You can set a password for root accounts.
  • You can remove root accounts that are accessible from outside the local host.
  • You can remove anonymous-user accounts.
  • You can remove the test database, which by default can be accessed by anonymous users.

See http://www.mysql-optimization.com/mysql-secure-installation-program.html

HTTPS connection issue

An https connection problem with certificate. Error message comes from Google-chrome and Safari browsers.

Your connection is not private:

HttpsNotPrivateChrome.png, HttpsNotPrivateSafari.png

If we use wget or curl on a terminal, we will get an error message 

$ wget https://ftp-trace.ncbi.nlm.nih.gov/sra/sdk/2.7.0/sratoolkit.2.7.0-ubuntu64.tar.gz
--2017-04-14 09:40:01--  https://ftp-trace.ncbi.nlm.nih.gov/sra/sdk/2.7.0/sratoolkit.2.7.0-ubuntu64.tar.gz
Resolving ftp-trace.ncbi.nlm.nih.gov (ftp-trace.ncbi.nlm.nih.gov)... 130.14.250.7, 2607:f220:41e:250::13
Connecting to ftp-trace.ncbi.nlm.nih.gov (ftp-trace.ncbi.nlm.nih.gov)|130.14.250.7|:443... connected.
ERROR: cannot verify ftp-trace.ncbi.nlm.nih.gov's certificate, issued by ‘CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US’:
  Unable to locally verify the issuer's authority.
To connect to ftp-trace.ncbi.nlm.nih.gov insecurely, use `--no-check-certificate'.

# curl -L https://ftp-trace.ncbi.nlm.nih.gov/sra/sdk/2.7.0/sratoolkit.2.7.0-ubuntu64.tar.gz -o sratoolkit.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (60) SSL certificate problem: Invalid certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

It is interesting after a few tries, curl works again and wget works after adding the option --no-check-certificate 

$ wget --no-check-certificate https://ftp-trace.ncbi.nlm.nih.gov/sra/sdk/2.7.0/sratoolkit.2.7.0-ubuntu64.tar.gz
--2017-04-14 09:51:32--  https://ftp-trace.ncbi.nlm.nih.gov/sra/sdk/2.7.0/sratoolkit.2.7.0-ubuntu64.tar.gz
Resolving ftp-trace.ncbi.nlm.nih.gov (ftp-trace.ncbi.nlm.nih.gov)... 130.14.250.11, 2607:f220:41e:250::13
Connecting to ftp-trace.ncbi.nlm.nih.gov (ftp-trace.ncbi.nlm.nih.gov)|130.14.250.11|:443... connected.
WARNING: cannot verify ftp-trace.ncbi.nlm.nih.gov's certificate, issued by ‘CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US’:
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: 63707890 (61M) [application/x-gzip]
Saving to: ‘sratoolkit.2.7.0-ubuntu64.tar.gz’

sratoolkit.2.7.0-ubuntu64.tar. 100%[==================================================>]  60.76M  6.50MB/s    in 9.2s    

2017-04-14 09:51:42 (6.59 MB/s) - ‘sratoolkit.2.7.0-ubuntu64.tar.gz’ saved [63707890/63707890]

$ curl -L -O https://ftp-trace.ncbi.nlm.nih.gov/sra/sdk/2.7.0/sratoolkit.2.7.0-ubuntu64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 60.7M  100 60.7M    0     0  6312k      0  0:00:09  0:00:09 --:--:-- 6820k

HTTPS at NCBI: Guidance for Users

https://www.ncbi.nlm.nih.gov/home/develop/https-guidance/

Compiling R

Use the following command to download required components before building any R packages. See also ubuntu package for R 

sudo apt-get build-dep r-base

AppImage file - new way of installing an application

What is an “AppImage”? How do I install it?

  • AppImages can be downloaded and run without installation or the need for root rights.
  • The key idea of the AppImage format is one app = one file. Every AppImage contains an app and all the files the app needs to run. In other words, each AppImage has no dependencies other than what is included in the targeted base operating system(s).

Some examples

PPA management

Add a ppa repository, 

# sudo add-apt-repository [repository name]
sudo add-apt-repository ppa:embrosyn/cinnamon
sudo apt-get update 
# sudo apt-get install [software name]
sudo apt-get install cinnamon cinnamon-core

Remove a ppa repository. 

# method 1: add-apt-repository
sudo add-apt-repository --remove ppa:embrosyn/cinnamon
# method 2: ppa-purge
sudo apt-get install ppa-purge
sudo ppa-purge ppa:embrosyn/cinnamon
# method 3: use GUI

Create .deb file

checkinstall command

dpkg-buildpackage command

Package maintenance

Package repository and /etc/apt/sources.list

echo "new line of text" | sudo tee -a /etc/apt/sources.list
# OR
# use 'add-apt-repository' command 
sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 886DDD89
sudo add-apt-repository "deb http://deb.torproject.org/torproject.org $(lsb_release -s -c) main"
sudo apt-get update
sudo sh -c 'echo "deb http://cran.rstudio.com/bin/linux/ubuntu trusty/" >> /etc/apt/sources.list'
# Or the following if the OS is Ubuntu 
# (other derived distributions like Linux Mint do not count)
# because $(lsb_release -s -c) returns the CodeName which diffs from each Linux distributions.
# sudo sh -c 'echo "deb http://cran.rstudio.com/bin/linux/ubuntu $(lsb_release -s -c)/" >> /etc/apt/sources.list' 

gpg --keyserver keyserver.ubuntu.com --recv-key E084DAB9
gpg -a --export E084DAB9 | sudo apt-key add -
sudo apt-get update
sudo apt-get -y install r-base

sources.list.d

This directory has more contents than the /etc/apt/sources.list file. For example, on my Mint 17.2 

$ cat /etc/apt/sources.list
#deb cdrom:[Linux Mint 17.2 _Rafaela_ - Release amd64 20150627]/ trusty contrib main non-free
deb http://download.virtualbox.org/virtualbox/debian trusty contrib
$ ls /etc/apt/sources.list.d
additional-repositories.list  google-chrome.list                   shutter-ppa-trusty.list
docker.list                   mc3man-mpv-tests-trusty.list         stebbins-handbrake-releases-trusty.list
ekozincew-ppa-trusty.list     official-package-repositories.list   webupd8team-java-trusty.list
getdeb.list                   openshot_developers-ppa-trusty.list  wine-wine-builds-trusty.list

See an example of how to use it: https://apt.syncthing.net/ 

# Add the release PGP keys:
curl -s https://syncthing.net/release-key.txt | sudo apt-key add -

# Add the "stable" channel to your APT sources:
echo "deb https://apt.syncthing.net/ syncthing stable" | sudo tee /etc/apt/sources.list.d/syncthing.list

# Update and install syncthing:
sudo apt-get update
sudo apt-get install syncthing

E: Could not get lock /var/lib/dpkg/lock

I get the above message when I run sudo apt-get update.

Solution 1:

Unable to lock the administration directory (/var/lib/dpkg/) is another process using it?.

I can reproduce the error from a clean boot. The most possible explanation is the answer from poolie:

the command-line apt overlaps with update-manager automatically polling. So if you try again in a few minutes that should fix it.

From my experience, after I wait about 10 minutes, ps -A | grep apt won't show anything.

Solution 2:

Fix “Unable to lock the administration directory (/var/lib/dpkg/)” in Ubuntu 

# Method 1: Find and Kill all apt-get or apt Processes
$ ps -A | grep apt
1760 ?    00:00:00 apt.systemd.dai
3489 ?    00:00:00 aptd
$ sudo kill -9 1760
$ sudo kill -9 3489

# Method 2: Delete the lock Files
sudo rm /var/lib/apt/lists/lock
# and 
sudo rm /var/lib/dpkg/lock

Difference between dpkg and aptitude/apt-get/gdebi

https://askubuntu.com/questions/309113/what-is-the-difference-between-dpkg-and-aptitude-apt-get

Install a deb file

# dpkg gives more clear about the error message
sudo dpkg -i application.deb
# Or
sudo apt install application.deb

However, installing from a .deb file through dpkg does not resolve dependency issues the same way installation through apt will. See here.

So you should run sudo apt-get install -f to fix and complete the installation if any errors were encountered by dpkg. 

sudo apt install -f dep1.deb dep2.deb dep3.deb

A better way to install a deb file is using the gdebi tool.

Remove deb packages

See the man page. 

dpkg -l | grep 'tcl'
sudo dpkg -r tcl8.4 # -r means remove/uninstall
sudo dpkg -P tcl8.4 # -P means purge

dpkg -l | grep 'tcl'   (optional)

gdebi: an improvement over dpkg

How does gdebi automatically gets missing dependancies ? Can I use gdebi for all other installations of .deb packages? 

sudo apt-get install gdebi-core

# wget newPackage.deb
# sudo gdebi -n newPackage.deb

deb files caches

See http://askubuntu.com/questions/444240/ubuntu-updates-blocked-apt-get 

sudo rm -f /etc/apt/sources.list.d/*

sudo dpkg -i /var/cache/apt/archives/*.deb

sudo dpkg --configure -a

List all available packages (from repositories)

apt-cache pkgnames # pkgnames is part of the command

Show package information

apt-cache show <package_name>

Check dependencies for specific packages

apt-cache showpkg <package_name>

Upgrade a package

apt-get install <package_name>

Install specific version of the package

Use the "=" with the package-name and append desired version. 

sudo apt-get install vsftpd=2.3.5-3ubuntu1

Remove a package

sudo apt-get remove <package_name>

To completely remove a package including their configuration files 

sudo apt-get purge <package_name>
sudo apt-get remove --purge <package_name>

Rollback an apt-get upgrade

http://www.cyberciti.biz/howto/debian-linux/ubuntu-linux-rollback-an-apt-get-upgrade/

Clean up disk space

The clean command is used to free up the disk space by cleaning retrieved .deb files from the local repository. 

sudo apt-get clean

Auto clean up apt-get cache

sudo apt-get autoclean

The 'autoclean' command deletes all .deb files from /var/cache/apt/archives to free up disk space.

Download only source code of package

sudo apt-get --download-only source <package_name>

To download and unpack source code of a package 

sudo apt-get source <package_name>

To download, unpack and compile a package 

sudo apt-get --compile source <package_name>

Download without installing

sudo apt-get download <package_name>

Check change log of package

Note that the change log may not be found. 

sudo apt-get changelog <package_name>

Check broken dependencies

sudo apt-get check

Search missing package's full name

Use the apt-file command. See this post

At first, install apt-file command and prepare it. 

$ sudo apt-get install apt-file
$ sudo apt-file update

To find zlib.h, 

$ apt-file search zlib.h

It reports too many result. Let’s narrow down. 

$ apt-file search /usr/include/zlib.h
zlib1g-dev: /usr/include/zlib.h

Now you know zlib.h is in zlib1g-dev package. 

$ sudo apt-get install zlib1g-dev

Find package information before installing it

Use "-s" option for simulation. No sudo is necessary. 

apt-get -s install PACKAGENAME

The output is too much (include other dependences)

A better way is to use aptitude (which is not installed by default in Ubuntu) 

aptitude search <package> -F "%c %p %d %V"

For example, 

debian@beaglebone:~/qt-4.8.5/bin$ aptitude search  qtcreator -F "%c %p %d %V"
p qtcreator                         lightweight integrated development environme 2.5.0-2       
p qtcreator:armel                   lightweight integrated development environme 2.5.0-2       
p qtcreator-dbg                     debugging symbols for Qt Creator IDE         2.5.0-2       
p qtcreator-dbg:armel               debugging symbols for Qt Creator IDE         2.5.0-2       
p qtcreator-doc                     documentation for Qt Creator IDE             2.5.0-2

If a package is already installed, we can use the following way to check version number. 

apt-show-versions <package>

List files in a package

dpkg -L <package_name>

List racing games package (kind of search packages by key words)

apt-cache search racing game
apt-cache search vsftpd

Search installed packages

dpkg -l libgtk* | grep -e '^i'
dpkg -l libpng* | grep -e '^i'
dpkg -l libjpeg* | grep -e '^i'

To search x-org related packages 

udooer@udoo:~$ dpkg -l | grep xserver-xorg | awk '{$1=$3=$4=""; print $0}'  # Skip columns 1,3,4
 imx-xserver-xorg-extension-viv-9t6-hdmi   Freescale Xorg server driver extension for HDMI performance
 imx-xserver-xorg-video-viv-9t6   Xorg server driver for imx6, vivante
 xserver-xorg   X.Org X server
 xserver-xorg-core   Xorg X server - core server
 xserver-xorg-dev   Xorg X server - development files
 xserver-xorg-input-all   X.Org X server -- input driver metapackage
 xserver-xorg-input-evdev   X.Org X server -- evdev input driver
 xserver-xorg-input-synaptics   Synaptics TouchPad driver for X.Org server
 xserver-xorg-input-wacom   X.Org X server -- Wacom input driver
 xserver-xorg-video-all   X.Org X server -- output driver metapackage
 xserver-xorg-video-fbdev   X.Org X server -- fbdev display driver
 xserver-xorg-video-modesetting   X.Org X server -- Generic modesetting driver
 xserver-xorg-video-omap   X.Org X server -- OMAP display driver
 xserver-xorg-video-vesa   X.Org X server -- VESA display driver

List of installed packages

dpkg --get-selections 
# Or using [https://wiki.debian.org/ListInstalledPackages dpkg-query] utility.
dpkg-query -l

Install a list of packages

How to get list of installed packages on Ubuntu / Debian Linux

Suppose we want to install all packages currently installed on server 1 onto server 2, we can do that by 

# server 1
dpkg --get-selections | grep -v deinstall > mylist.txt
# server 2
sudo dpkg --set-selections < mylist.txt

Show (sort) package size

dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n

Check if a library is installed or not

Use ldconfig -p | grep LIBNAME. For example, to check if libxml is installed or not, 

EXIST=`ldconfig -p | grep libxml | wc -l`
if [ $EXIST -ng 0 ]; then echo EXISTING; fi

How to uninstall software

http://www.howtogeek.com/229699/how-to-uninstall-software-using-the-command-line-in-linux/

Upgrade software packages

Upgrade all the currently installed software packages on the system 

sudo apt-get upgrade

If you want to upgrade, unconcerned of whether software packages will be added or removed to fulfill dependencies, use 

sudo apt-get dist-upgrade

List of available (uninstalled) packages

aptitude -F "%p" search "?not(?installed)"

See http://superuser.com/questions/408082/how-do-i-get-a-list-of-available-i-e-uninstalled-packages-in-debian

Check Change Log of Package

http://www.tecmint.com/useful-basic-commands-of-apt-get-and-apt-cache-for-package-management/ 

sudo apt-get changelog PKGNAME

Clean up/remove packages

ubuntugeek.com

unmet dependencies after adding a PPA

http://askubuntu.com/questions/140246/how-do-i-resolve-unmet-dependencies-after-adding-a-ppa 

$ sudo apt-get install libgl1-mesa.dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'libgl1-mesa-dev-lts-quantal' for regex 'libgl1-mesa.dev'
Note, selecting 'libgl1-mesa-dev' for regex 'libgl1-mesa.dev'
Note, selecting 'libgl1-mesa-dev-lts-saucy' for regex 'libgl1-mesa.dev'
Note, selecting 'libgl1-mesa-dev-lts-trusty' for regex 'libgl1-mesa.dev'
Note, selecting 'libgl1-mesa-dev-lts-utopic' for regex 'libgl1-mesa.dev'
Note, selecting 'libgl1-mesa-dev-lts-vivid' for regex 'libgl1-mesa.dev'
Note, selecting 'libgl1-mesa-dev-lts-raring' for regex 'libgl1-mesa.dev'
libgl1-mesa-dev is already the newest version.
libgl1-mesa-dev set to manually installed.
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libgl1-mesa-dev : Conflicts: libgl-dev
                   Conflicts: libgl1-mesa-dri-dev
 libgl1-mesa-dev-lts-utopic : Depends: mesa-common-dev-lts-utopic (= 10.3.2-0ubuntu1~trusty2) but it is not going to be installed
                              Depends: libgl1-mesa-glx-lts-utopic (= 10.3.2-0ubuntu1~trusty2) but it is not going to be installed
                              Conflicts: libgl-dev
                              Conflicts: libgl1-mesa-dev
                              Conflicts: libgl1-mesa-dri-dev
 libgl1-mesa-dev-lts-vivid : Depends: mesa-common-dev-lts-vivid (= 10.5.9-2ubuntu1~trusty2) but it is not going to be installed
                             Depends: libgl1-mesa-glx-lts-vivid (= 10.5.9-2ubuntu1~trusty2) but it is not going to be installed
                             Conflicts: libgl-dev
                             Conflicts: libgl1-mesa-dev
                             Conflicts: libgl1-mesa-dri-dev
                             Conflicts: xorg-renamed-package-lts-utopic
E: Unable to correct problems, you have held broken packages.

View logs

  • Apache /var/log/apache2/access.log

Terminal Assistant

Torrent

Top 6 Torrent Alternatives to The Pirate Bay (7/5/2017)

(old) Popular search sites

  • torrentz
  • thepiratebay
  • isohunt
  • kickass torrents

How to set up torrent scheduling on Linux

https://www.howtoforge.com/tutorial/how-to-set-up-torrent-scheduling-on-linux/. It covers 'Deluge', 'Transmission' and 'Rtorrent'.

This can be useful for people who want to take advantage of their computer while they are not using it, like during the nighttime for example. This way, large portions of huge files can be downloaded without delaying your work activities, or interrupting/undermining your media consumption.

Torrent client command line: aria2c

aria2 is a lightweight multi-protocol & multi-source command-line download utility. It supports HTTP/HTTPS, FTP, BitTorrent and Metalink. aria2 can be manipulated via built-in JSON-RPC and XML-RPC interfaces.

http://linuxconfig.org/aria2-all-in-one-command-line-download-tool 

sudo apt-get install aria2
aria2c magnet:?xt=urn:btih:1e99d95f1764644a86a8e99bfd80c ...

Torrent client: transmission-cli

https://wiki.archlinux.org/index.php/Transmission

Torrent client command line: rtorrent

We first need to create an .rtorrent.rc file under $HOME directory. Then run 

rtorrent XXX.torrent

In the simplest case, .rtorrent.rc looks like 

# On the terminal, mkdir ~/Downloads/rsession
download_rate = 0
upload_rate =50
directory = ~/Downloads
session = ~/Downloads/rsession
port_range = 55556-55560
scgi_port = 127.0.0.1:5000
use_udp_trackers = yes
encryption = allow_incoming,try_outgoing,enable_retry

As you can see here, I have created a sub-directory rsession under ~/Downloads/.

Keybinding:

  • ctrl + q = quit application
  • ctrl + d = stop an active download
  • ctrl + s = start downloading

glibc

Patch glibc 2.9

sudo apt-get update
sudo apt-get upgrade
sudo reboot

SSL

Install commercial SSL certificate

check openssl version

http://askubuntu.com/questions/504928/how-to-check-which-openssl-version-is-use-by-apache-on-ubuntu-12-04 

odroid@odroid:~$ apt-cache policy openssl
openssl:
  Installed: 1.0.2g-1ubuntu4.5
  Candidate: 1.0.2g-1ubuntu4.5
  Version table:
 *** 1.0.2g-1ubuntu4.5 500
        500 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf Packages
        500 http://ports.ubuntu.com/ubuntu-ports xenial-security/main armhf Packages
        100 /var/lib/dpkg/status
     1.0.2g-1ubuntu4 500
        500 http://ports.ubuntu.com/ubuntu-ports xenial/main armhf Packages

# Ubuntu 14.04
$ openssl version 
OpenSSL 1.0.1f 6 Jan 2014
# Macbook Pro 10.11
$ openssl version 
OpenSSL 0.9.8zh 14 Jan 2016

check openssl location

A more general way is to use openssl version -d 

# On Ubuntu 14.04
$ openssl version -d
OPENSSLDIR: "/usr/lib/ssl"
$ ls /usr/lib/ssl
certs  misc  openssl.cnf  private
$ ls -l /usr/lib/ssl
total 4
lrwxrwxrwx 1 root root   14 Oct  7 11:03 certs -> /etc/ssl/certs
drwxr-xr-x 2 root root 4096 Feb 11 09:12 misc
lrwxrwxrwx 1 root root   20 Jan 30 15:42 openssl.cnf -> /etc/ssl/openssl.cnf
lrwxrwxrwx 1 root root   16 Oct  7 11:03 private -> /etc/ssl/private
$ ls -l /etc/ssl/certs | head
total 912
lrwxrwxrwx 1 root root     26 Oct  7 11:03 00673b5b.0 -> thawte_Primary_Root_CA.pem
lrwxrwxrwx 1 root root     45 Oct  7 11:03 02265526.0 -> Entrust_Root_Certification_Authority_-_G2.pem
lrwxrwxrwx 1 root root     29 Oct  7 11:03 024dc131.0 -> Microsec_e-Szigno_Root_CA.pem
lrwxrwxrwx 1 root root     31 Oct  7 11:03 02b73561.0 -> Comodo_Secure_Services_root.pem
lrwxrwxrwx 1 root root     36 Oct  7 11:03 03179a64.0 -> Staat_der_Nederlanden_EV_Root_CA.pem
lrwxrwxrwx 1 root root     25 Oct  7 11:03 034868d6.0 -> Swisscom_Root_EV_CA_2.pem
lrwxrwxrwx 1 root root     16 Oct  7 11:03 03f2b8cf.0 -> WoSign_China.pem
lrwxrwxrwx 1 root root     41 Oct  7 11:03 04f60c28.0 -> USERTrust_ECC_Certification_Authority.pem
lrwxrwxrwx 1 root root     40 Oct  7 11:03 052e396b.0 -> AddTrust_Qualified_Certificates_Root.pem
$ ls -l /etc/ssl/certs | wc -l
533

# On Macbook Pro 10.11
$ openssl version -d
OPENSSLDIR: "/System/Library/OpenSSL"
$ ls -l /System/Library/OpenSSL
total 8
drwxr-xr-x  2 root  wheel    68 May 15  2016 certs
drwxr-xr-x  8 root  wheel   272 May 15  2016 misc
-rw-r--r--  1 root  wheel  9390 May 15  2016 openssl.cnf
drwxr-xr-x  2 root  wheel    68 May 15  2016 private
$ ls -l /System/Library/OpenSSL/certs/
$ # empty results

openssl & patch bug

brb@vm-1404:~$ sudo dpkg -l | grep  openssl
ii  libgnutls-openssl27:amd64   2.12.23-12ubuntu2.2   amd64        GNU TLS library - OpenSSL wrapper
ii  openssl                     1.0.1f-1ubuntu2.15    amd64        Secure Sockets Layer toolkit - cryptographic utility
ii  python-openssl              0.13-2ubuntu6         amd64        Python 2 wrapper around the OpenSSL library

brb@vm-1404:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.1 LTS
Release:	14.04
Codename:	trusty

brb@vm-1404:~$ openssl version -a
OpenSSL 1.0.1f 6 Jan 2014
built on: Fri Jun 20 18:54:02 UTC 2014
platform: debian-amd64
options:  bn(64,64) rc4(16x,int) des(idx,cisc,16,int) blowfish(idx) 
compiler: cc -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -m64 
OPENSSLDIR: "/usr/lib/ssl"

brb@vm-1404:~$ sudo apt-get changelog openssl | grep CVE-2014-0224
    - debian/patches/CVE-2014-0224-regression2.patch: accept CCS after
    - debian/patches/CVE-2014-0224.patch: set the CCS_OK flag when using
    - debian/patches/CVE-2014-0224-1.patch: only accept change cipher spec
    - debian/patches/CVE-2014-0224-2.patch: don't accept zero length master
    - debian/patches/CVE-2014-0224-3.patch: allow CCS after resumption in
    - CVE-2014-0224

Let's Encrypt SSL

It takes less than 5 minutes for the whole process! 

sudo apt-get update
sudo apt-get install software-properties-common
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install python-certbot-apache
sudo certbot --apache

Let’s Encrypt certificates are valid for 90 days, but it’s recommended that you renew the certificates every 60 days to allow a margin of error. The Let's Encrypt client has a renew command that automatically checks the currently installed certificates and tries to renew them if they are less than 30 days away from the expiration date.

So we can add the following line to the cron job that will execute the letsencrypt-auto renew command every Monday at 2:30 am. sudo crontab -e 

30 2 * * 1 certbot renew >> /var/log/le-renew.log

Note if you currently have already had a working SSL certificate (eg from a purchased one), then even you have removed the certificate from the relevant Apache files you still can't run through sudo certbot. But once your current working SSL certificate expires, sudo certbot will successfully go through even you don't manually remove the certificate from your system.

Install webmin

apt-get install perl libnet-ssleay-perl openssl libauthen-pam-perl libpam-runtime libio-pty-perl apt-show-versions python
wget http://prdownloads.sourceforge.net/webadmin/webmin_1.600_all.deb
dpkg --install webmin_1.600_all.deb

Webmin.png

File does not exist: /var/www/favicon.ico from /var/log/apache2/error.log

The reason? You have not created a favicon, also known as a website icon, for your website. It’s the icon that displays in the address bar of your web browser when you connect to a website. A web browser will request this icon file from every website.

If you choose to create one. Use a program such as Gimp and create a 16×16 pixel image and save it as a .ico filetype. Then upload that file to the DocumentRoot of your website. You will need one for each VirtualHost. If you don’t have Gimp, there are online resources such as favicon.cc where you can create a .ico file and download it for your own use.

As you know by now, not having a favicon.ico file, won’t stop web browsers from requesting it each time. But you can tell Apache not to log the event as an error message. You will still see the request in the access.log, but at least you will have a cleaner error.log file.

Add the following block of code to each VirtualHost, or at least the ones which don’t have a favicon file. 

Redirect 404 /favicon.ico
<Location /favicon.ico>
 ErrorDocument 404 "No favicon"
</Location>

Don’t forget to restart apache after making the change. If you want make a “global” change, which would apply to any and all VirtualHosts, you can create a file in Apache’s conf.d folder with a name such as nofavicon.conf and then add that block of code to the file. That would disable favicon across the board and save you from having to edit each VirtualHost.

Or, you create an empty file with the name “favicon.ico” in the directory root of Apache (for exemple /var/www/).

Show weather on the taskbar

http://www.noobslab.com/2012/10/important-thingstweaks-to-do-after.html

WebCam

https://help.ubuntu.com/community/Webcam

Install cheese or guvcview. An example of using computer vision on Raspberry Pi.

Watch TV

Freetuxtv

Running concrete5 On Nginx (LEMP)

http://www.howtoforge.com/running-concrete5-on-nginx-lemp-on-debian-squeeze-ubuntu-12.10

Open mms stream in google chrom in linux

http://www.thermetics.net/2011/12/07/how-to-open-mms-links-from-chrome-under-ubuntu/

Download mms stream (suitable if mms is an extension; for example, studioclassroom)

  • Use mimms (will NOT output to speaker at the same time; so is best in terms of performance)
sudo apt-get install mimms
mimms -t 60 mms://example.com/video

where -t option specifies number of minutes. See http://linuxers.org/howto/how-download-mms-streaming-videos-ubuntu The output will have the same extension as the input. But it is not always clear. For example 

$ mimms -t 3 mms://bcr.media.hinet.net/RA000073                                                                                                    
mms://bcr.media.hinet.net/RA000073 => RA000073.wmv
1.48 MB / ∞ B (8.79 kB/s, ∞ s remaining)           
Download stopped after user-specified timeout.


  • Use mplayer (will output to speaker at the same time, so suffer from performance, anyway do not get output)
mplayer mms:/link/something.xxx -dumpstream -dumpfile file.xxx

mms:/link/something.xxx - link to the stream you wish to download file.xxx - file to which you wish to download the stream, be careful to write the same extension xxx

Wait for the file to download and that's it. See http://ubuntuhowtos.com/howtos/download_mms_stream

Keep a linux process running after log out

# nohup command-with-options &

Open Firefox in cron job

The trick is to use export DISPLAY=:0 

export DISPLAY=:0
firefox http://www.google.com &

Close firefox gracefully

Use wmctrl command. 

sudo apt-get install wmctrl
wmctrl -c firefox

Record audio out from command line

http://www.pantz.org/software/alsa/recording_sound_from_your_web_browser_using_linux.html

Step 1: Install required programs 

sudo apt-get install gnome-media pavucontrol lame

Step 2: Create a script file <recordfm.sh> 

#!/bin/bash
# Get pulseaudio monitor sink monitor device then pipe it to 
# sox to record wav, lame to encode to mp3, or flac to encode flac
FILENAME="$1"
STOPTIME="$2"
# Encoding options for lame and flac.
LAMEOPTIONS="--preset cbr 192 -s 44.1" 

if [ -z "$FILENAME" ]; then
    echo -e "
    Usage: $0 /path/to/output.mp3
    Usage: $0 /path/to/output.mp3 stopinseconds" >&2
    exit 1
fi

# Get sink monitor:
MONITOR=$(pactl list | egrep -A2 '^(\*\*\* )?Source #' | \
    grep 'Name: .*\.monitor$' | awk '{print $NF}' | tail -n1)
echo "set-source-mute ${MONITOR} false" | pacmd >/dev/null

# Record it raw, and pipe to lame for an mp3
echo "Recording to $FILENAME ..."

if [[ $FILENAME =~ .mp3$ ]]; then
  if [ -z $STOPTIME ]; then
    parec -d $MONITOR | lame $LAMEOPTIONS -r - $FILENAME 
  else
    echo -e "\nStopping in $STOPTIME seconds"
    parec -d $MONITOR | lame $LAMEOPTIONS -r - $FILENAME 2>&1 &
    SPID=$!
    sleep $STOPTIME
    kill -9 $SPID
  fi
fi

Step 3: play the music or launch a browser with a desired url.

Step 4: run the bash script 

chmod +x recordfm.sh
./recordfm.sh test.mp3 10

where <test.mp3> is the output filename and 10 is recording length (seconds). It works.

Note the script teaches us how to find out the ID for a process we just launched (cleaner than using ps -ef | grep commands). The command is 

SPID=$!
echo $SPID

Advanced audio control

https://www.howtoforge.com/tutorial/advanced-audio-control-on-linux/

  • Alsamixer
  • Pulse Audio Volume Control
  • Pulse Audio Equalizer

VLC

How to play DVDs

https://help.ubuntu.com/community/RestrictedFormats/PlayingDVDs

Use VLC to listen tunein radio

Download link http://addons.videolan.org/content/show.php?content=152788. Right click the links and save them (3 lua files)

Create two new subfolders and place the lua files

  • move the “tunein.lua” file to the VLC folder so it can be found and run by VLC.
    • In Linux, move the “tunein.lua” file to ~/.local/share/vlc/lua/sd/
    • In Windows, move the “tunein.lua” file to %AppData%/VLC/VideoLAN/lua/sd.
    • In Mac OS X, move the “tunein.lua” file to VLC.app/Contents/MacOS/share/lua/sd
  • move the “radiotimes.lua” and “streamtheworld.lua” files from the Playlist folder to “/lua/playlist” folder in the VLC directory.

Open VLC. Go to “View -> Playlist”. At the sidebar, under the Internet section, you should now see “TuneIn Radio” in the list. Click on it and you will be able to access your Favorites.

Taiwan radio

  • rtsp://live2.rti.org.tw/rti4.sdp (discovered by open http://music.rti.org.tw/music in Linux Chrome browser)
  • rtmp://news98-t24.griffinmas.com/live/news98

Use VLC to record internet radio (suitable if the stream is continuous)

http://www.ubuntugeek.com/how-to-record-internet-radio-using-vlc.html 

sudo apt-get install vlc browser-plugin-vlc

sudo apt-get install ubuntu-restricted-extras

sudo apt-get install lame  libmp3lame0
sudo apt-get install ffmpeg
sudo apt-get install libavcodec-extra-53 libavdevice-extra-53 libavfilter-extra-2 libavformat-extra-53 \
                     libavutil-extra-51 libpostproc-extra-52 libswscale-extra-2

A successful run will have an output like 

VLC media player 2.0.3 Twoflower (revision 2.0.2-93-g77aa89e)
[0x2329ca8] dummy interface: using the dummy interface module...
[0x7fac2c007428] mux_dummy mux: Open
[0x7fac2c003598] access_mms access: selecting stream[0x1] audio (39 Kib/s)
[0x7fac2c003598] access_mms access: connection successful
[0x7fac2c003598] access_mms access error: failed to send command
[0x7fac2c005fe8] idummy demux: command `quit'

Play audio using a command line mode

See also my Beaglebone page for a comparison of different possibilities. For VLC, there are 3 interface modes. The following example is to run vlc in a text mode with the ncurses library. 

sudo apt-get install vlc-nox
vlc -I ncurses XXX.mp3
vlc --help

Play youtube using VLC from a command line

See this post 

vlc -I http https://www.youtube.com/watch?v=UlW77conmAc

Keyboard shortcuts

cheatography.com and howtogeek

  • Ctrl + arrow: forward/backward 1 minute
  • Alt + arrow: forward/backward 10 seconds
  • [ or -: decrease speed
  • ]: increase speed

SMPlayer

MPV - terminal media player

CVS

CVS server

# Original data dir: /home/mli/Downloads/hmv_arc
# CVS dir on server (CVSROOT): /home/mli/cvsrep
# CVS dir on local:  /home/mli/Downloads/localcvs
# Project name: mycvs
# cvs user name: mli

sudo apt-get install cvs

mkdir ~/cvsrep
export CVSROOT=/home/mli/cvsrep
cvs init
sudo groupadd mycvsgrp
sudo useradd -G mycvsgrp mli # Assume mli is a new user
sudo usermod -a -G mycvsgrp mli  # assume mli is an existing user  
groups mli  # view groups a user is in use
sudo chown -R :mycvsgrp /home/mli/cvsrep # change the group ownership of cvsrep directory to mycvsgrp.
cd ~/Downloads/hmv_arc
cvs import -m "initial" mycvs mli START # import files to CVS repository
                                        # the new subfolder mycvs has owner mli.mli

Note 1. It is OK to use the same CVSROOT for multiple modules/projects since each module/project will be saved under a separate subfolder.

Note 2. The cvs version that I have installed in my ubuntu server is 1.12.13. 

$ cvs -v

Concurrent Versions System (CVS) 1.12.13-MirDebian-6 (client/server)


Quick test to checkout project to the same machine 

cd ~/Downloads
mkdir localcvs
cd localcvs
cvs checkout mycvs

This will create a new subfolder 'mycvs' under ~/Downloads/localcvs.

CVS repository on server has a structure 

cvsrep/CVSROOT
cvsrep/mycvs

CVS sandbox on local machine has a structure 

mycvs/CVS
mycvs/[files1]
mycvs/[files2]

CVS client (ubuntu)

Check out 

cvs -d :ext:[email protected]:/home/mli/cvsrep checkout mycvs
# OR 2 steps
export CVSROOT=:ext:[email protected]:/home/mli/cvsrep
cvs checkout mycvs

Commit a new file 

cvs add mynewfile
cvs commit -m "my log message" mynewfile

Update repository 

cvs update -P -d
# OR
cvs update filename

where -P "prunes" directories that are empty, and -d tells cvs to include any new directories that aren't in your local workspace

Get a list of all tags and branches 

# Lists all tags and braches for each and any file together with the revision it belongs to.
cvs status -v

# http://stackoverflow.com/questions/566093/how-do-i-identify-what-branches-exist-in-cvs
cvs log -h | awk -F"[.:]" '/^\t/&&$(NF-1)==0{print $1}' | sort -u

Encoding of a file 

$ cvs -d :ext:[email protected]:/home/mli/cvsrep checkout mycvs
$ file -bi mycvs/src/hmvUnicode.rc
text/x-c; charset=utf-16le

CVS client (windows)

I use WinCVS for a demonstration

  • Remote -> checkout module
  • Module name: mycvs
  • CVSROOT:
    • protocol: ssh
    • repository path: /home/mli/cvsrep
    • user name:
    • host name: taichi.selfip.net

If I use TortoiseCVS (1.12.5 from 1/24/2011), I need to choose ext as protocol instead ssh. Still the checked out file 'hmv_.rc' still contains unreadable Chinese characters. The cvsnt is the latest free version (2.5.05). If I want to use WinCVS + cvsnt from TortoiseCVS, the options in the CVSROOT dialog looks weird and cannot create a connection.

For the unicode encoding. If I commit the file at first from ubuntu os, but check out in Windows. The checked out file has right encoding (using Notepad ++, or from VS2010). However, the file does not have right line ending and it shows Chineses character when I open it in either Notepad++ or VS2010.

To see hidden characters in Linux, try either one of the following 2 methods:

  • Open the file in EMACS and do a M-X hexl-mode
  • geany editor.

The solution I have found to overcome accessing unicode (utf-16) file on Windows OS is using Cygwin.

  • Download setup.exe from http://cygwin.com/install.html
  • Root directory = c:\cygwin
  • Local package directory = C:\Users\brb\Downloads
  • Direct connection
  • Download site: ftp://cygwin.mirrors.pair.com (Some mirrors are not updated & contain old version of packages! For example, make sure the cvs version is 1.12.13.)
  • Search: cvs. Click plus sign next to "Devel". Click 'Skip' corresponding to cvs package.
  • Search: ssh. Click plus sign next to "Net". Click 'skip' correspond to openssh package.
  • Click 'Next' button.
  • Click 'Finish' button.
  • Now open 'Cygwin Terminal' icon on Windows Desktop.
export CVSROOT=:ext:[email protected]:/home/mli/cvsrep
cvs checkout mycvs

The 'mycvs' directory should be under C:\cygwin\home\brb (a.k.a. /home/brb in cygwin) directory. We can open 'hmv_.rc' file in Notepad++ to double check if the file looks normal OR use md5sum to check.

Difference between CRLF (Windows), LF (Linux, Mac) and CR

This is a good summary I found: http://stackoverflow.com/questions/1552749/difference-between-cr-lf-lf-and-cr-line-break-types

  • The Carriage Return (CR) character (0x0D, \r) moves the cursor to the beginning of the line without advancing to the next line. This character is used as a new line character in Commodore and Early Macintosh operating systems (OS-9 and earlier).
  • The Line Feed (LF) character (0x0A, \n) moves the cursor down to the next line without returning to the beginning of the line. This character is used as a new line character in UNIX based systems (Linux, Mac OSX, etc)
  • The End of Line (EOL) character (0x0D0A, \r\n) is actually two ASCII characters and is a combination of the CR and LF characters. It moves the cursor both down to the next line and to the beginning of that line. This character is used as a new line character in most other non-Unix operating systems including Microsoft Windows, Symbian OS and others.

Common CVS commands

cvs checkout MODULE
cvs checkout DIR
cvs checkout DIR/SUBDIR
cvs co DIR/SUBDIR/FILENAME
# check out a specific tag and put it in a specified directory. 
# the specified directory name will replace the module name in output.
mkdir localcvs
cvs checkout -r v4_3 -d localcvs MODULE

cvs add myfile.c
cvs add -kb myfile.bin
# If you accidentally add a file, simply skip the commit for that file.

cvs update –dA DIR/SUBDIR 
# -d: Create  any  directories  that  exist in the repository if they're missing from the working directory.
# -A: Reset any sticky tags, dates, or -k options. Needed after you use "cvs update -D" or "cvs update -r".
cvs update –A DIR/SUBDIR/FILENAME    

cvs commit
cvs commit –m "add test suite" DIR/SUBDIR/FILENAME

mkdir ~/original
touch ~/original/newfile
cvs import ~/original VENDORTAG RELEASETAG

mkdir ~/localcvs
cd ~/localcvs
cvs checkout common/too

cvs diff -r1.23 -r1.24 SUBDIR/FILENAME     # Difference between specified versions 1.23 & 1.24.
cvs diff -D "1 hour ago" MODULE
cd LOCALCVS; cvs diff 
cvs checkout -D "1 hour ago" MODULE
cvs checkout -D "2013-02-27 01:30" MODULE

rm file(s); cvs remove file(s); cvs commit -m "Comment text" file(s)
# You must rm the file before issuing the cvs remove command. The remove is not final until the commit has been issued.

# cvs does not let you remove directories. However it does let you ignore any directories that are empty.
cvs co -P Offline
cvs update -PdA
# P: Prune empty directories. d: create new directories. A: Reset sticky tags

cvs history -c -a -D "1 day ago"  -z "-0500"
# find all changes submitted to the repository by anyone in the past day. 
# -z is used to adjust the time zone.

cvs history -c -a -D "1 day ago" -f Mu2eG4/src
# find all changes submitted to Mu2eG4/src (or any other subdirectory) by anyone in the past day

cvs history -c -u USER "1 day ago" -f Mu2eG4/src 
# find all changes submitted by USER to Mu2eG4/src in the past day

cvs log FILENAME

Bypass SSH password login (convenient for CVS, git etc)

http://www.howtogeek.com/tips/bypass-ssh-logins-by-adding-your-key-to-a-remote-server-in-a-single-command/

  1. ssh-keygen -t rsa
  2. (make sure the remote server has .ssh directory)
  3. cat ~/.ssh/id_rsa.pub | ssh user@hostname 'cat >> .ssh/authorized_keys'
  4. ssh user@hostname

It helps with CVS log in too when the CVS works by using ssh protocol. Note that step 3 allows to run a shell command at a remote machine.

See https://help.github.com/articles/generating-ssh-keys also for similar instruction when work on github.

The ssh key can be copied to another a machine (pay attention to mode). Or let the new machine to create its own key pair and use ssh-copy-id to append the identity file to remote machine's ~/.ssh/authorized_keys file. See http://superuser.com/questions/332510/how-to-transfer-my-linux-ssh-keys-to-another-machine.

We can even have multiple ssh key on local machine by using <.ssh/config> file. See http://www.karan.org/blog/index.php/2009/08/25/multiple-ssh-private-keys.

Install Ubuntu to a USB flash drive

The following approach assumes the boot loader will not be put on internal hard disk if you are careful enough. See also http://fernhilllinuxproject.com/guidesandhowtos/installubuntutousbdrive.html

  1. Boot from live CD in ubuntu.
  2. Insert the USB flash drive in USB Port.
  3. On desktop double click on icon 'install ubuntu 12.04'
  4. Click on continue and it will ask if you want to unmount the flash drive click on yes
  5. Choose some thing else when ask to where to install ubuntu.
  6. Next you will see your hard disk partitions and flash drive. Click on flash drive partition and then click on change. Change the partition type to ext4 and mount as to '/'. Click on ok to close the dialogue box.
  7. In last you will see a drop down menu on where to install the the boot loader. Initially it will be showing sdc,sdd but on drop down you must select sdc1 or sdd1. Failing this step you may render your computer unbootable.
  8. Then click on install and linux will be installed on your USB Flash drive.

Install a new hard drive

See also https://help.ubuntu.com/community/InstallingANewHardDrive and Mount drive

  1. Use sudo fdisk command to create partition table. Then 'n', 'p', '1', 'w' and several returns.
  2. Use sudo mkfs -t ext4 /dev/sdb1 to create a new partition
  3. Use sudo mkdir /mnt/ssd to create a new mount point
  4. Use sudo nano -Bw /etc/fstab to do auto mount on boot
  5. sudo chown -R USERNAME:USERNAME /mnt/ssd to give the ownership to the USERNAME.

How to install a Ceph Storage Cluster on Ubuntu

https://www.howtoforge.com/tutorial/how-to-install-a-ceph-cluster-on-ubuntu-16-04/

Building a simple Beowulf Like Cluster with Ubuntu

See also the Raspberry Pi page where a simple cluster was built.

MPICH2

Resource:

  1. http://byobu.info/article/Building_a_simple_Beowulf_cluster_with_Ubuntu/
  2. https://help.ubuntu.com/community/MpichCluster
  3. https://help.ubuntu.com/community/SettingUpNFSHowTo (NFS configure)

Here is my record for creating a cluster environment based on ubuntu 13.04. The master node is running on ubuntu 13.04 desktop with virtualBox 4.2. The virtualBox has added a host-only adapter (vboxnet0) with ip 192.168.56.1. This adapter will be added to the master node so I can use this ip to identify the master node in the host-only network. Creating cluster using VirtualBox is just for the education purpose, not for real practice.

  • (virtualBox) Create two virtual machines running on ubuntu 13.04 server. The ssh server was checked during installation. The host name for each of them is ubuntuNode1 and ubuntuNode2 respectively. The network adapter is left by default (NAT) during installation. But after the installation is done, I shutdown the system and add a host-only adapter (vboxnet0) to each of them. Then after the system is up again, I change the IP so it is static. Do sudo nano /etc/network/interfaces and append the following before running sudo /etc/init.d/networking restart to take the change in effect.
auto eth1
iface eth1 inet static
address 192.168.56.101 # use 192.168.56.102 for ubuntuNode2
netmask 255.255.255.0
network 192.168.56.0
broadcast 192.168.56.255

Note that it is better not to add host-only network before installation, or during installation it will ask what is the primary network (confusing). So at the end each new nodes should have both eth0 and eth1 adapters and they should not reside in same subset. ifconfig may not show all adapters so we should use ifconfig -a instead. Although host-only network is used for communication between guest and guest OR guest and host, since each guest node has NAT adapter by default so the guest machine can still access the internet.

  • (master node) Edit /etc/hosts so it like like
127.0.0.1 localhost
192.168.56.1 ubuntu1304
192.168.56.101 ubuntuNode1
192.168.56.102 ubuntuNode2

Note that the master node will be used to start jobs on the cluster although it is OK to let the master node as one of nodes to execute the jobs.

  • (all nodes) Run
sudo adduser mpiuser --uid 999

It is recommeneded to use the same password for the user. This will create a new directory /home/mpiuser. This is the home directory for user mpiuser and we will use it to execute jobs on the cluster.

  • (master node) Run
sudo apt-get install nfs-kernel-server
  • (other nodes) Run
sudo apt-get install nfs-client
  • (master node) Add the following to the file /etc/exports
/home/mpiuser *(rw,sync,no_subtree_check)

or something like 

/home/mpiuser 192.168.56.0/24(rw,sync,no_subtree_check)

Some people create a shared folder under /srv directory.

Now run 

sudo service nfs-kernel-server restart
  • (master node)
sudo ufw allow from 192.168.56.0/24
  • (other nodes)
sudo mount ubuntu1304:/home/mpiuser /home/mpiuser

And if we want to mount the NFS shared directory when the compute nodes are booted, edit /etc/fstab by adding 

ubuntu1304:/home/mpiuser /home/mpiuser nfs
  • (master node)
sudo apt-get install ssh
su mpiuser
ssh-keygen
ssh-copy-id localhost

We can test if the ssh works without passwords 

ssh ubuntuNode1
echo $HOSTNAME
  • (all nodes)
sudo apt-get install mpich2
which mpirun
which mpiexec
  • (master node, mpiuser)

Go to the home directory of mpiuser and create a new file hosts. Include host names for computing nodes (it is OK to include master node, ubuntu1304, as one of computing nodes) 

ubuntuNode1
ubuntuNode2
  • (all nodes)
sudo apt-get install build-essential
  • (master node, mpiuser). Creating a test file <mpi_hello.c>
#include <stdio.h>
#include <mpi.h>

int main(int argc, char** argv) {
    int myrank, nprocs;

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
    MPI_Comm_rank(MPI_COMM_WORLD, &myrank);

    printf("Hello from processor %d of %d\n", myrank, nprocs);

    MPI_Finalize();
    return 0;
}

And compile and run it by 

mpicc mpi_hello.c -o mpi_hello
mpiexec -n 3 -f hosts ./mpi_hello

We should see something like 

Hello from processor 0 of 2
Hello from processor 1 of 2

Successful! That's it.

OpenMPI

With R

File sharing in a local network - woof

Sharing internet

Simple gui approach

When I tested the method with my rtl8187 wifi adapter by Rosewill (output by lsusb) on Odroid Xu4 running Ubuntu 14.04, I got an error on Step 3 (Create New Wi-Fi Network) ??? 

Failed to activate connection
(32) Access Point (AP) mode is not supported by this device

When I tried the wifi adapter RT2870/RT3070 802.11n by MediaLink (MWN-USB150N), there is no problem to create it. My android device can connect to it. However, there is no internet access:( ...... After some tweaks on command line (iptables; see Sharing internet via wifi: iptables below), it works!!!

And the output of nm-tool command also confirms the wifi device is connected. 

odroid@odroid:~/Downloads$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth2
10.42.0.0       0.0.0.0         255.255.255.0   U         0 0          0 wlan3
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth2

The ap hotspot from the wifi adapter has an IP 10.42.0.1 (ifconfig).

Sharing internet via wifi: ap-hotspot

I cannot make it to work to share the internet via wifi on my Xubuntu 13.04. However the solution in http://askubuntu.com/questions/287251/creating-an-infrastructure-hotspot-using-ubuntu-12-10 works for me. 

$ # Under rare situation, we need to run the following line to get 'add-apt-repository'
$ sudo apt-get install apt-get install software-properties-common
$ sudo su -
# add-apt-repository ppa:nilarimogard/webupd8
# aptitude update
# aptitude install ap-hotspot
# ap-hotspot configure
# ap-hotspot start

In the step of 'ap-hotspot configure' it will ask for Access Point name and WPA passphrase.

When I run the final line 'ap-hotspot start', it will ask me to disconnect my current wifi first.

At the end, I found my original configuration in ubuntu does not work even its setting is there. apt-hotspot create an infrastracture AP instead of adhoc.

Sharing internet via wifi: iptables

1. Assume the primary wired network connection, eth0 is connected to Internet.

2. Using your distro's network managment tool, create a new ad hoc wireless connection with the following settings: 

IP address: 10.99.66.55
Subnet mask: 255.255.0.0 (16)

3. Use the following shell script to share the internet connection 

#!/bin/bash
#filename: netsharing.sh
#Replace 10.99.0.0/16 by your network/netmask
#Usage: ./netsharing.sh input_interface output_interface
#Example: ./netsharing.sh eth0 wlan0

echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -A FORWARD -i $1 -o $2 -s 10.99.0.0/16 -m conntrack --ctstate NEW -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A POSTROUTING -t nat -j MASQUERADE

4. Run the script as follows: 

sudo ./netsharing.sh eth0 wlan0

5. Connect your devices to the wireless network you just created with the following settings (Question: how about the SSID here?) 

IP address: 10.99.66.56 (and so on)
Subnet mask: 255.255.0.0

To make this more convenient, you might want to install a DHCP and DNS server on your machine, so it's not necessary to configure IPs on devices manually. A handy tool for this is dnsmasq which you can use for performing both DHCP and DNS operations.

Credit: Linux Shell Scripting Cookbook

A similar approach discussed on the Odroid forum.

iptables

~/.xsession-errors file is filling the hard disk

It is related to vino-server. See

Short solution is to use "kill -9 xxx" to kill the process and rm to remove ~/.xsession-errors file. The long time solution is to uninstall vino.

JRE and JDK

Install openjdk or Sun jdk. See http://www.maketecheasier.com/install-java-runtime-in-ubuntu/ (Ubuntu 12.04)

If we have multiple versions of JRE/JDK, we can use the following command to set the default version 

sudo update-alternatives --config java

This approach seems to be working in the case JAVA_HOME cannot be honored.

OpenJDK

On my Ubuntu 12.04, I have two versions. 

$ whereis java
java: /usr/bin/java /usr/bin/X11/java /usr/share/java /usr/share/man/man1/java.1.gz
$ java -version
java version "1.6.0_38"

$ sudo apt-get -y install openjdk-7-jdk
$ ls -l /usr/lib/jvm
total 12
lrwxrwxrwx 1 root root   20 Nov 16  2013 java-1.6.0-openjdk-amd64 -> java-6-openjdk-amd64
lrwxrwxrwx 1 root root   20 Mar 24 06:20 java-1.7.0-openjdk-amd64 -> java-7-openjdk-amd64
drwxr-xr-x 7 root root 4096 Feb 20 08:43 java-6-openjdk-amd64
drwxr-xr-x 3 root root 4096 Jan 31  2014 java-6-openjdk-common
drwxr-xr-x 7 root root 4096 Mar 30 18:16 java-7-openjdk-amd64

$ ls -l /usr/bin/java
lrwxrwxrwx 1 root root 22 Jan 31  2014 /usr/bin/java -> /etc/alternatives/java

Question: how to switch to 1.6 or 1.7 version of java? (For example, snpEff requires java 1.7) 

$ update-java-alternatives -l
java-1.6.0-openjdk-amd64 1061 /usr/lib/jvm/java-1.6.0-openjdk-amd64
java-1.7.0-openjdk-amd64 1051 /usr/lib/jvm/java-1.7.0-openjdk-amd64
$ sudo apt-get install icedtea-7-plugin
$ sudo update-java-alternatives -s java-1.7.0-openjdk-amd64
update-java-alternatives: plugin alternative does not exist: /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/IcedTeaPlugin.so
$ update-java-alternatives -l
java-1.6.0-openjdk-amd64 1061 /usr/lib/jvm/java-1.6.0-openjdk-amd64
java-1.7.0-openjdk-amd64 1051 /usr/lib/jvm/java-1.7.0-openjdk-amd64
$ java -version
java version "1.7.0_95"

Question: How to install OpenJDK 8 on 14.04 LTS? (for example, Picard 2 requires Java 1.8) 

$ sudo apt-get -y install openjdk-8-jdk   # works for Ubuntu 14.10 and later
# Unable to locate package openjdk-8-jdk on my Ubuntu 14.04

So the solution is to install Sun jdk.

Oracle JAVA

sudo apt-add-repository -y ppa:webupd8team/java
sudo apt-get update
echo debconf shared/accepted-oracle-license-v1-1 select true |  sudo debconf-set-selections
echo debconf shared/accepted-oracle-license-v1-1 seen true |   sudo debconf-set-selections
sudo apt-get install oracle-java8-installer

java -version

CPU/system load

I-Nex

A look at I-Nex – a CPU-Z like Application for GNU/Linux

CPU-G

See this instruction to install CPU-G program for viewing hardware information (process, mb, graphic, memory, system).

CoreFreq

http://www.tecmint.com/corefreq-linux-cpu-monitoring-tool/

system load indicator

System Load Indicator: it is used to view system information (cpu, memory, network) in real-time. 

sudo apt-get install indicator-multiload

sudo apt-get install indicator-cpufreq
indicator-cpufreq

Hard drive specification

http://www.cyberciti.biz/faq/find-hard-disk-hardware-specs-on-linux/ 

# hdparm command
sudo hdparm -I /dev/sda

# OR using lshw command
sudo apt-get install lshw
lshw -class disk -class storage

# Find Out Disks Name Only
lshw -short -C disk

Hard disk speed (3.0 or 6.0 Gb/s)

http://www.cyberciti.biz/faq/freebsd-command-to-find-sata-link-speed/ 

$ dmesg | grep -i SATA
[    0.311173] pci 0000:00:11.0: set SATA to AHCI mode
[    1.510881] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 4 ports 6 Gbps 0xf impl SATA mode
[    1.512669] ata1: SATA max UDMA/133 abar m1024@0xfe02f000 port 0xfe02f100 irq 19
[    1.512672] ata2: SATA max UDMA/133 abar m1024@0xfe02f000 port 0xfe02f180 irq 19
[    1.512675] ata3: SATA max UDMA/133 abar m1024@0xfe02f000 port 0xfe02f200 irq 19
[    1.512677] ata4: SATA max UDMA/133 abar m1024@0xfe02f000 port 0xfe02f280 irq 19
[    1.832349] ata4: SATA link down (SStatus 0 SControl 300)
[    1.832418] ata2: SATA link down (SStatus 0 SControl 300)
[    2.004290] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    2.004313] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

To test the disk performance, follow cyberciti.biz. Note that the parameter oflag=dsync makes a difference.

Here I have a 7200rpm Seagate (ST2000DM001-1CH164) and a 5400rpm WD-blue (WD30EZRZ-00WN9B0) 

# Hard disk info https://en.wikipedia.org/wiki/Hdparm
sudo hdparm -I /dev/sda
sudo hdparm -I /dev/sdb

# Writing speed, where /tmp is a directory from the disk 
dd if=/dev/zero of=/tmp/test1.img bs=2G count=1; rm /tmp/test1.img
# 1.4 GB/s from either 5400 or 7200 rpm

dd if=/dev/zero of=/tmp/test1.img bs=2G count=1 oflag=dsync; rm /tmp/test1.img
# 115 MB/s from 5400 rpm
# 166 MB/s from 7200 rpm

# Latency
dd if=/dev/zero of=/tmp/test2.img bs=512 count=1000 oflag=dsync; rm /tmp/test2.img
# 27.7 kB/s from 5400 rpm  <==== This is interesting
# 12.5 kB/s from 7200 rpm

# Read speed
dd if=/dev/zero of=/tmp/test3.img bs=1G count=1 oflag=direct; rm /tmp/test3.img
# 122 MB/s from 5400 rpm
# 180 MB/s from 7200 rpm

Hard disk directory size

See Display directory size with sorting and human readable by using the ncdu utility.

Monitor harddisk health by command line

Using smartctl or the GUI version 

sudo apt-get install gsmartcontrol
sudo gsmartcontrol

How to check Hard Drive Disk Health in GNU/Linux

Hard disk temperature

http://www.cyberciti.biz/tips/howto-monitor-hard-drive-temperature.html

It seems the hddtemp command does not work on SSDs. 

sudo apt-get install hddtemp
sudo hddtemp /dev/sdb

/dev/sdb: ST2000DM001-9YN164: 40°C

For SSD, 

sudo apt-get install smartmontools
sudo smartctl -a /dev/sda

Hard disk power on time/hours

sudo smartctl --all /dev/sda | grep Power_On_Hours

Sample output: 

$ sudo smartctl -A /dev/sda | grep -i power
  9 Power_On_Hours          0x0032   034   034   000    Old_age   Always       -       58541
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       164
$ sudo smartctl -A /dev/sdb | grep -i power
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       585
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       43
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       32

system's hardware temperatures and voltages

https://www.howtoforge.com/tutorial/ubuntu-performance-monitoring/ 

sudo apt-get install lm-sensors
sensors

brb@brb-P45T-A:~$ sensors
acpitz-virtual-0
Adapter: Virtual device
temp1:        +30.0°C  (crit = +110.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Core 0:       +41.0°C  (high = +78.0°C, crit = +100.0°C)
Core 1:       +36.0°C  (high = +78.0°C, crit = +100.0°C)

nouveau-pci-0100
Adapter: PCI adapter
temp1:        +68.0°C  (high = +95.0°C, hyst =  +3.0°C)
                       (crit = +125.0°C, hyst =  +3.0°C)
                       (emerg = +135.0°C, hyst = +10.0°C)

where nouveau is an open-source driver set for Nvidia cards. It is not clear about acpitz-virtual-0. Some suggested to use inxi which will produce human readable system info.

And on a Dell T3600 machine (running the samtools mpileup & bcftools programs), 

brb@T3600 ~ $ sensors
nouveau-pci-0300
Adapter: PCI adapter
fan1:        3510 RPM
temp1:        +52.0°C  (high = +95.0°C, hyst =  +3.0°C)
                       (crit = +105.0°C, hyst =  +5.0°C)
                       (emerg = +135.0°C, hyst =  +5.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +70.0°C  (high = +81.0°C, crit = +91.0°C)
Core 0:         +60.0°C  (high = +81.0°C, crit = +91.0°C)
Core 1:         +60.0°C  (high = +81.0°C, crit = +91.0°C)
Core 2:         +56.0°C  (high = +81.0°C, crit = +91.0°C)
Core 3:         +60.0°C  (high = +81.0°C, crit = +91.0°C)
Core 4:         +70.0°C  (high = +81.0°C, crit = +91.0°C)
Core 5:         +60.0°C  (high = +81.0°C, crit = +91.0°C)

When all cores are 100% used (htop), the fan is getting noisy 

brb@T3600 ~ $ sensors
nouveau-pci-0300
Adapter: PCI adapter
fan1:        4560 RPM
temp1:        +61.0°C  (high = +95.0°C, hyst =  +3.0°C)
                       (crit = +105.0°C, hyst =  +5.0°C)
                       (emerg = +135.0°C, hyst =  +5.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +82.0°C  (high = +81.0°C, crit = +91.0°C)
Core 0:         +78.0°C  (high = +81.0°C, crit = +91.0°C)
Core 1:         +81.0°C  (high = +81.0°C, crit = +91.0°C)
Core 2:         +78.0°C  (high = +81.0°C, crit = +91.0°C)
Core 3:         +80.0°C  (high = +81.0°C, crit = +91.0°C)
Core 4:         +81.0°C  (high = +81.0°C, crit = +91.0°C)
Core 5:         +77.0°C  (high = +81.0°C, crit = +91.0°C)

Unlock keyring

I got the prompt of unlocking keyring every time I open google chrome browser. 

google-chrome --password-store=basic

It is also helpful to remove ubuntu one from Ubuntu 12.04. See http://hex.ro/wp/blog/removing-ubuntuone-from-ubuntu-12-04/

netcat (nc) - arbitrary TCP and UDP connections and listens

Netcat or nc is a networking utility for debugging and investigating the network.

The nc (or netcat) utility is used for just about anything under the sun involving TCP, UDP, or UNIX-domain sockets. It can open TCP connections, send UDP packets, listen on arbitrary TCP and UDP ports, do port scanning, and deal with both IPv4 and IPv6.

For example, we can create simple network sockets and use them for text communication. We need two sockets: one listens for connections and the other connects to this one. 

# create a listening socket on the local machine
nc -l 1234

# connect to the socket from a 2nd computer 
nc IP_LocalMachine 1234

# send messages. 
# Type something and press Enter on the terminal from the 2nd computer.
# The message will appear on the terminal of the local machine.

To transfer files over the network (no any password is needed!!) 

# receiver machine
nc -l 1234 > destination_file

# sender machine
nc IP_Receiver 1234 < source_filename

List of all services/daemons

Run service --status-all to get a list off all the Upstart services and their status. See

  • man service
  • man initctl
service --status-all   # output format is clean

sudo initctl list      # show the process number too

where in the output "+" means started, "-" stopped, and "?" unknown.

Google Drive or other cloud services

Video rip/convert/transcoder

sudo add-apt-repository ppa:stebbins/handbrake-releases
sudo apt-get update
sudo apt-get install handbrake-gtk

sudo apt-get install handbrake-cli
  • libav-tools

To convert youtube flv file to mp4. 

sudo apt-get install libav-tools
avconv -i INPUT.flv -codec copy OUTPUT.mp4

To merge audio and video 

# naive: use the one with longer duration as the total length
avconv -i music.m4a -i input.mp4 -acodec aac -strict experimental output.mp4
# improved: specify the start time (-ss) and duration (-t)
# unfortunately the music at the specified end time may not be the end
# so a better way is to use a video editor (eg OpenShot) and specify fade out on the end of the audio!
avconv -i music.m4a -i input.mp4 -acodec aac -strict experimental -ss 00:00:00 -t 00:01:01 -codec copy output.mp4

To extract audio only: 

avconv -i INPUT.flv -codec copy -vn OUTPUT.mp4

Video editing in Linux

Pitivi

Blender

Looks very professional too. Windows/Linux/OSX (binary files are provided). Worth to try.

Kdenlive

More complicated than OpenShot. Worth to try. Ubuntu 16.04.1 is needed.

Shotcut

I tested inserting a text in a video. Compared to Youtube video editor

  • The text is really a text. No pop-up shape to select
  • Not sure how to control the text so it only appears at a certain time interval

Not as intuitive to use.

Youtube Video Editor

Good

  • Annotation and Title (Video Manager -> Videos -> Edit -> End screens and annotations -> Annotations -> Add annotations (Speech bubble, Note, Title, Spotlight, Label). However, the annotations do not show up on mobile. See this post for a discussion.
  • Add photos

Bad

  • Music cannot have fade in/out

Notes

7 Things You Need to Build a Low-Cost YouTube Studio

http://www.makeuseof.com/tag/build-low-cost-youtube-studio/

Free or Open source Subtitle editor

Comparison of subtitle editors

For TV captions, use white color font with black color for borders and transparent background.

OpenShot-qt

sudo apt-get install openshot

When I needed to export the video (choose 'youtube' profile, 'youtube-HD' target, 'HD 270, 29 .97 bps', and 'high' quality), I found I need to install libx264 code. On Ubuntu, I open software center and seach 'libavformat'. I choose 'libavformat-extra-53'.

An introduction to video editing in Openshot 2.0 from howtoforge.

Don't use the version (1.4.3 date 2009) because it crashed too often.

The new version 2.0.7 (date 2016) looks a little different (theme is black. Cool!). Its icon and command (openshot-qt) are different too. The project saved from 1.4.3 cannot be opened in 2.0.7. The tools icons are different: Add track, Snapping tool, Add Marker, Previous Marker, Next Marker, Zoom in/out.

This version of OpenShot + (old) Core2Duo Ubuntu = Dynamic Heatmap Viewer video.

Audio library

You can download free music from Youtube Audio Library. If you use a copy righted music and upload your video to youtube, the video will show Ad eventually.

Take a snapshot

The keyboard shortcut Ctrl+d does not work.

One suggestion is to use VLC. VLC -> Video -> Take Snapshot. The snapshot will be automatically created in ~/Pictures/ directory (*.png format).

Procedure
  1. Put audio and video files in one folder
  2. Use openshot to create a new video. Also
    • Use the +/- sign for zoom in and zoom out
    • Right click video file and select Volume -> Entire clip -> level 0%
    • Right click audio and select Volume -> End of clip -> fade out (slow)
  3. openshot -> Save (arrow/download-like button)
  4. openshot -> Export (red circle button)
    • Modify the file name so it won't overwrite the original (openshot won't check it)
    • Select Profile = Web, Target=Youtube-HD, Video Profile=HD 1080p 25 fps, Quality=High.
  5. Check the exported video (play it first by VLC).
    • On one instance the audio is fuzzy until the middle of the video. So I have to change the audio
    • On another instance the video length is longer than I expected because the final annotation slide lasts too long. A solution is to change the setting (Profile=All Formats, Target=MP4 (h.264), Video Profile=HD 1080p 24 or 23.98 fps). If I use 25 fps, the file will be wrong.
  6. Upload to Youtube. Use Youtube video editor to include annotation.

VideoLAN Movie Creator

ffdiaporama

Create videos from images, movie clips and music.

Flowblade Movie Editor

It is written in Python. Only Linux version is available (no Windows nor OS X). Good for beginners. 

sudo apt-get install flowblade

Lightworks

Free and Pro versions are available. Windows/Linux/OSX.

Youtube command line tools

Play audio only

http://unix.stackexchange.com/questions/229787/audio-only-youtube-player/229790#229790 

# play in background (prompt will return, a new vlc process will be launched)
cvlc --vout none https://www.youtube.com/watch?v=1O0W7jSd940
# or play in foreground (prompt will not return)
/usr/bin/vlc -I dummy --vout none https://www.youtube.com/watch?v=1O0W7jSd940

Using this approach to play audio only will save CPU power. Tested on Odroid.

However, it seems this approach does not work on a playlist, for example, https://www.youtube.com/playlist?list=PL6h94tLpXv3LabUa7B0tCz7K0pI5ZzZEi. See mpsyt for a solution!

mpsyt: mps-youtube (mp3 + stream + youtube)

By default, mps-youtube is basically a YouTube audio player (and downloader), but you can enable external video playback (via mpv or MPlayer) from its options. Check out

  • An article on webupd8.
  • More keyboard controls can be found on the webpage of mps.

Installation on Linux (works on Odroid with low CPU usage for audio stream from youtube but Odroid gives dirty noise when I played the music). 

sudo apt-get install python3-pip
sudo pip3 install mps-youtube
sudo pip3 install youtube_dl

# On Ubuntu/Mint. Do not use mplayer. Use mpv instead.
# sudo add-apt-repository ppa:mc3man/mpv-tests
# sudo apt-get update && sudo apt-get install mpv

mpsyt   # launch
set player mpv
pl https://www.youtube.com/playlist?list=PL6h94tLpXv3LabUa7B0tCz7K0pI5ZzZEi   # a playlist
h       # help
Space   # pause
p       # play
q       # quit mpsyt

h search
url https://www.youtube.com/watch?v=hgIfZz8STLk   # retrieve specific youtube video by url or id

Installation on Windows

Some highlight

  • Search
  • Local playlist
  • support YouTube Playlists
  • Download
  • Music Album Matching

If something is wrong with using 'set' command, just run rm -rf ~/.config/mps-youtube/ and restart everything.

By default, it only streams audio. To watch the video, use set show_video true.

Below is what I got from the set command (as you can see the default player is mpv) 

  Key                 Value                                                  
  order             : relevance
  user_order        : 
  max_res           : 2160p
  player            : mpv
  playerargs        : 
  encoder           : 0 [None]
  notifier          : 
  checkupdate       : True
  show_mplayer_keys : True
  fullscreen        : False
  show_status       : True
  columns           : 
  ddir              : /home/odroid/Downloads/mps
  overwrite         : True
  show_video        : False
  search_music      : True
  window_pos        : 
  window_size       : 
  download_command  : 
  audio_format      : auto
  api_key           : AIzaSyCIM4EzNqi1in22f4Z3Ru3iYvLaY8tc3bo

The screenshot shows mpsyt contains basic playing keyboard shortcuts.

Mps-youtube.png

[Update 4-7-2017]: to fix an error Signature extraction failed: Traceback (most recent call last): 

sudo apt-get remove -y youtube-dl  # Removing youtube-dl (2014.02.17-1) ...
sudo pip3 install -U youtube-dl

$ youtube-dl --version
2017.04.11

Download youtube video using command line

sudo apt-get install youtube-dl
  • Fix the error 'WARNING: Your copy of avconv is outdated and unable to properly mux separate video and audio files, youtube-dl will download single file media. Update avconv to version 10-0 or newer to fix this.':

(works) http://askubuntu.com/questions/563245/avconv-warning-while-downloading-youtube-video 

$ sudo add-apt-repository ppa:heyarje/libav-11 && sudo apt-get update
$ sudo apt-get install libav-tools

(not tested) Compile your own. The source code is available at https://libav.org/.

  • For videos with 1080p resolution, youtube-dl will download 720p by default. To download 1080p, see this post
# show the available resolutions
youtube-dl -F XXX
# download the desired resolution
youtube-dl -f 137+141 XXXX 
# Note the two numbers 137, 141 is case dependent. They could be 137 & 140.
youtube-dl -f 22 XXX
# if the format 22 is mp4. If I do not specify the format, the mkv file will be downloaded
# But playing mkv file tends to be stuttering with VLC.

This assumes Ubuntu has installed avconv; otherwise video(mp4) and audio(m4a) files will be downloaded separately. To install avconv, run 

sudo apt-get install libav-tools
  • To convert the video to mp3 format, use soundconverter
sudo apt-get install soundconverter
mps-youtube
  • 4k videos samples (vlc v2.1.4 cannot play)

Unity

Unity LauncherSwitcher

Ubuntugeek.com

Create Unity Launcher

Take RStudio for example,

  • Desktop icons are saved under /usr/share/icons/hicolor/16x16/apps/rstudio.png, where other than 16x16 there are also icons from 24x24, 32x32, 48x48 sizes (directories). Another place is ~/.local/share/icons/hicolor/16x16/ .
  • Unity launcher is created at /usr/share/applications/rstudio.desktop or ~/.local/share/applications/XXXX.desktop.
  • Desktop shortcut can be created by copy XXXX.desktop to ~/Desktop directory.

The icon size in the launcher can be adjusted by opening System Settings -> Apperance -> Launcher Icon Size (at the bottom).

k2pdfopt has an example how to do it for creating a desktop shortcut and be in the right click menu.

How to Convert the Unity Launcher into a Dock-Style Launcher

See an article from Howtogeek.com.

How to Get Unity’s Global App Menu in Linux Mint Cinnamon

http://www.makeuseof.com/tag/get-unitys-global-app-menu-linux-mint-cinnamon/

Remove floppy icon from Launcher

http://askubuntu.com/questions/457970/how-to-completely-disable-floppy-in-ubuntu-14-04 

echo "blacklist floppy" | sudo tee /etc/modprobe.d/blacklist-floppy.conf
sudo rmmod floppy
sudo update-initramfs -u

Application Launcher

Cerebro

Cerebro is an Open Source OS X Spotlight Equivalent for Linux

Ulauncher

Ulauncher is a Lightweight App Launcher for Linux Desktops

Synapse

Synapse or Albert — What’s Your Favourite App Launcher for Linux?

Gnome Do

Release June 2009

What date was the system installed

ls -l /var/log/installer

What is the last log in time for users

lastlog  # all users
last     # current user

What is the reboot time

last reboot

Crop an image

sudo apt-get install gthumb

Open an image. Click color palette icon on top-right corner (or use keyboard shortcut 'e'). Click 'crop'. There will be a rectangle on image where you can resize the rectangle.

Pinta and mtPaint can also take a screenshot of the desktop and crop the image.

Mind-mapping

KeePass

Search Ubuntu Software Center to install KeePassX (more reviews) or KeePass2. There is no need to use ppa.launchpad.net to install from. Note that the kdbx file used in KeePass2 cannot be opened in KeePassX. To use the command line to install KeePass2, 

sudo apt-get install keepass2

Note:

Security:

KeePass with KeeAgent

http://code.mendhak.com/keepass-and-keeagent-setup/

Update Firefox

See https://help.ubuntu.com/community/FirefoxNewVersion. The following instruction is used to get security-testing packages. 

sudo add-apt-repository ppa:ubuntu-mozilla-security/ppa
sudo apt-get update
sudo apt-get install firefox

Bluetooth

* https://zach-adams.com/2014/07/bluetooth-audio-sink-stream-setup-failed/ Linux Mint 
sudo apt-get install bluetooth
sudo apt-get install bluetooth bluez-utils blueman

Then run lsusb | grep Bluetooth command which will shows the name of your bluetooth device. 

sudo apt-get install bluez

On my bluetooth adapter, the lsusb shows, 

$ lsusb
Cambridge Silicon Radio, Ltd Bluetooth Dongle (HCI mode)
...
$ sudo hcitool dev
Devices:
	hci0	00:1A:7D:DA:71:09

I can use Ubuntu's bluetooth setting dialog to connect my bluetooth keyboard without any problem (It will ask me to enter some code on pairing).

Note that the MAC address of my bluetooth keyboard shown above is the from the controller/adapter. It is NOT the same as my bluetooth keyboard one (90:7F:61:8F:D0:38) as shown from the bluetooth setting (GUI) dialog.

Bluetooth Audio Receiver

Remember to change the Mode from Telephony Duplex (HSP/HFP) to High Fidelity Playback (A2DP) from Sound Settings dialog (launched from Taskbar). See also Windows OS.

BluetoothAudioLinux.png

Default applications and mime-types

See /etc/gnome/defaults.list.

How to easily open a PDF (or any) file from command line in Ubuntu 

sudo apt-get install libgnome2-bin
gnome-open [name-of-PDF-file]

gnome-open [dir-name-or-path] # To open a directory in Nautilus file manager

gnome-open https://www.howtoforge.com/ # To open a website in your system's default web browser

Change default audio player

Right-click an MP3 file, and from the menu select “Properties”. In the window that appears, go to the “Open With” tab and select VLC or whatever. Click the “Set As Default” button to set VLC as the default player.

You might also want to set the default player in the sound menu.

Unity display timeout

Go to Dash -> power setting -> Brightness Settings

Screensaver showing current time

xscreensaver + gltext

http://askubuntu.com/questions/64086/how-can-i-change-or-install-screensavers 

sudo apt-get install xscreensaver 
# sudo apt-get remove gnome-screensaver

Click Dash > xscreensaver or run the command line xscreensaver-demo. Mode = Only One Screen Saver & pick GLText. Click Setting & choose 'Display date and time' ,'Don't rotate'.

gluqlo

[Note that gluqlo uses too much cpu] I like gluqlo (flip clock) screensaver. However, it is not easy to make it to replace the gnome-screensaver. 

sudo apt-add-repository ppa:alexanderk23/ppa
sudo apt-get update
sudo apt-get install gluqlo
  • Follow the instruction to install it on Ubuntu machine.
  • Follow the instruction there to remove gnome-screensaver and install xscreensaver.
  • Configure xscreensaver to use only 1 screensaver. Edit ~/.xscreensaver file and add a line like
gluqlo -root                                \n\
  • Still follow the instruction to allow xscreensaver to start when the machine starts up. Don't try to edit ~/.xinitrc file as other sites suggested; start Dash and type 'startup' and follow the screen dialog to add xscreensaver -nosplash.
  • Still follow the instruction to add lock screen keyboard shortcut.

Note that the above steps work for Ubuntu 12.04 & 13.10 but not 14.04 (Ubuntu 14.04 changed to use LockScreen instead LightDM program to lock the screen). A solution on Ubuntu 14.04 is to disable screen lock.

  1. Still follow the above instruction to remove gnome-screensaver and install xscreensaver. Set gluqlo as the only one screensaver.
  2. Go to Brightness & Lock panel from the Unity Launcher. And set Turn screen off when inactive: to Never.
  3. Install "Unity Tweak Tool" with sudo apt-get install unity-tweak-tool. Run it from the Launcher and select System > Security > Enhance system security by disabling Desktop lock.
  4. To enable locking desktop (required a password to unlock the desktop), check 'Lock Screen After' option. I pick 1 minute.

For some reason, gluqlo suddenly uses all my cpu (6 cores) resource (Ubuntu 14.04). The computer thus makes some noise. I have to use 'kill' command to kill them.

Flash for browser

On Ubuntu 12.04, there is no way to directly install the pepperflashplugin-nonfree plugin. So we have to use ppa from other people. See this post 

sudo apt-get update
sudo apt-get install chromium-browser
sudo add-apt-repository ppa:skunk/pepper-flash
sudo apt-get update
sudo apt-get install pepflashplugin-installer
sudo update-pepperflashplugin-nonfree --install

The last step gives me an error: sudo: update-pepperflashplugin-nonfree: command not found

Gedit

  • To split a screen, Do "Documents -> New Tab Group. No extra plugin is needed to download. I am using version 3.10.4 from Ubuntu 14.04.
  • Restore tabs plugin. It works on my gedit 3.4 (ubuntu 12.04). Follow the instruction there exactly.
  • Source code browser plugin. This makes gedit a good IDE for developing C++/Java code since the left panel can show symbols. Click F9 to show the side panel.
  • Darkmate theme.
cd /usr/share/gtksourceview-3.0/styles
sudo gedit darkermate.xml

gedit > Edit > preferences > font and colors > color scheme.

  • Gedit has no built-in options to show special characters except through gedit-plugins (sudo apt-get install gedit-plugins). See Draw Spaces.

Text file line ending in DOS and Unix

A DOS/Windows text file can be converted to Unix format by simply removing all ASCII CR characters with 

$ tr -d '\r' < inputfile > outputfile  # inputfile and outputfile cannot be the same

or, if the text has only CR newlines, by converting all CR newlines to LF with 

$ tr '\r' '\n' < inputfile > outputfile

Geany

Geany can be used to run a bash script file line by line. See Debugging_Scripts.

Display special characters

Geany has a way to show special characters (Tabs/LF/CR). Edit > Preferences > Display > Tick, Show whitespace (tabs) & Show Line endings(CR/LF).

For DOS text file, the line ending is CR+LF.

For Unix text file, the line ending is LF.

Font size

Users can use either one of the following methods

  1. Edit -> Preferences -> Interface -> Fonts to adjust the font size.
  2. Keyboard bindings: Ctrl + Shift + '+' to increase the font size or Ctrl + '-' to decrease the font size. This does not affect the font size in Preferences.

Printing

The font size in Preferences affects the printing. The font size changed by using the keyboard bindings does not affect printing.

Remove vertical line

Edit -> Preferences -> Editor -> Display -> Uncheck Long line marker.

SQL

MySQL Workbench

http://www.mysql.com/products/workbench/

sqliteman

https://sourceforge.net/projects/sqliteman/

User Interface Designer

Glade - RAD tool to enable quick & easy development of user interfaces for the GTK+ toolkit and the GNOME desktop environment

Devhelp - API documentation browser for GTK+ and GNOME

HTML editor

  • Atom
  • Bluefish
  • Brackets features
    • Inline Editors
    • Live Preview
    • Preprocessor Support
  • MonoDevelop
  • Kompozer and the installation instruction for Ubuntu.
  • BlueGriffon: a new WYSIWYG content editor. The interesting thing is the software BlueGriffon EPUB Edition: a cross-platform Wysiwyg editor able to natively create and edit EPUB2 and EPUB3 ebooks!

npm and Javascript

See npm package manager in Javascript.

chm reader

sudo apt-get install xchm

SCR3310 smart card

  • The usb device should be recognized by Ubuntu/Mint. Thus, the smart card can be used by Windows virtual machine (tested on Windows 10 VM).
brb@T3600 ~ $ lsusb
Bus 002 Device 003: ID 413c:2107 Dell Computer Corp.
Bus 002 Device 033: ID 09c3:0013 ActivCard, Inc.
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 003: ID 3938:1031
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0b95:7720 ASIX Electronics Corp. AX88772
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
sudo apt-get install libpcsclite1 pcscd pcsc-tools
lsusb 
# Bus 006 Device 002: ID 04e6:5116 SCM Microsystems, Inc. SCR331-LC1 / SCR3310 SmartCard Reader
dmesg | grep SCR3310
# [ 2005.300052] usb 6-1: Product: SCR3310 v2.0 USB SC Reader
  • Search "SCR3310 driver linux" on google.com.
# Download pcsc-lite-1.8.13.tar.bz2 from 
# https://alioth.debian.org/frs/?group_id=30105
sudo apt-get install libudev-dev
cd pcsc-lite-1.8.13
./configure
make
sudo make install

# Download libusb
http://libusb.info/
cd libusb-1.0.19
./configure
make
sudo make install

# Download scmccid_5.0.27_linux
# http://www.identive-infrastructure.com/index.php/products-solutions/smart-card-readers-a-terminals/smart-card-readers/scr3310

cd scmccid_5.0.27_l32r
sudo ./install.sh

Chroot

Note that we have to change the conf file a little bit. The 'location' word needs to be changed to 'directory'. Also at the last step when we are ready to test a 32-bit GUI app, we need to issue DISPLAY in a separate line; such as 

export DISPLAY=:0.0
su brb # brb is my root user in the host system that can invoke the schroot program
       # firefox does not allow to use root to start it
firefox

For a recap: 

1. Install the packages
sudo apt-get install debootstrap schroot -y
2. Create a schroot configuration file
sudo nano /etc/schroot/chroot.d/precise_i386.conf
3. Install 32-bit ubuntu with debootstrap
sudo mkdir -p /srv/chroot/precise_i386
sudo debootstrap --variant=buildd --arch=i386 precise /srv/chroot/precise_i386 http://archive.ubuntu.com/ubuntu/
4. Test the chroot environment
schroot -l
schroot -c precise_i386 -u root
uname -a
cat /etc/issue
5. Additional configuration
apt-get install ubuntu-minimal
# That's all.

The article also mentioned the home directories (Documents, Downloads, ...) of the users within the chroot are shared with the host. How to access them from the host?

Check/Diagnostic SD card

http://askubuntu.com/questions/69932/is-there-an-sd-card-diagnostic-utility

Look for Disk Utility on you dash (Alt+F2 and type 'disk')

Ubuntu Snappy Core

How Snappy packages are different from Deb

An article from PCWorld

  • Applications are no longer installed system-wide. The base Ubuntu operating system is kept securely isolated from applications you install later. Both the base system and Snappy packages are kept as read-only images.
  • Snappy packages can include all the libraries and files they need, so they don’t depend on other packages.
  • An update can never fail, as a package installation could potentially fail and become incomplete with typical Linux packages.
  • Snappy also supports “delta” updates, which means only the changed bits of the package need to be downloaded and installed.
  • Snappy-based Ubuntu systems might be standard.

Snap commands

6 Essential Ubuntu Snap Commands You Should Know

Docker

I haven't found any tutorial yet!

Ova image

Sorry, I don't get the command line back. Booting stuck in the middle.

Beaglebone

http://beagleboard.org/snappy or http://www.ubuntu.com/things#try-beaglebone

Raspberry Pi 2

https://darrenjw2.wordpress.com/2015/02/07/getting-started-with-snappy-ubuntu-core-on-the-raspberry-pi-2/

unable to open mtp device ubuntu 14.04

Use this tip. 

sudo apt-get install mtpfs

Install/upgrade google chrome browser

wget -N https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo dpkg -i google-chrome-stable_current_amd64.deb

Note that '-N' option.

(Mar 7, 2016). We may experience an error "Failed to fetch http://dl.google.com/linux/chrome/deb/dists/stable/Release" when we run sudo apt-get update. It is because the 32-bit chrome has been discontinued. The solution is to modify the file </etc/apt/sources.list.d/google-chrome.list>. See reddit. 

$ sudo sed -i -e 's/deb http/deb [arch=amd64] http/' "/etc/apt/sources.list.d/google-chrome.list"
$ cat "/etc/apt/sources.list.d/google-chrome.list"
### THIS FILE IS AUTOMATICALLY CONFIGURED ###
# You may comment out this entry, but any other modifications may be lost.
deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main

Another suggestion to modify </opt/google/chrome/cron/google-chrome> (though the file exists) does not work .

How to Install Chrome on Linux and Easily Migrate Your Browsing From Windows 

sudo apt-get install libxss1 libappindicator1 libindicator7wget \
    https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb

sudo dpkg -i google-chrome*.deb

Message of the day /etc/motd

.Trash-1000 folder

See this post. Ubuntu will create such folders when a file is deleted from a USB drive. Presumably this would allow a file to be restored if you accidentally deleted it.

Try to empty the paperbin or delete the folder with the terminal-command as root: sudo rm -rf /path/to/folder/.Trash-1000

Xbox wireless Gamepad

https://www.howtoforge.com/tutorial/how-to-configure-your-gamepad-on-ubuntu/

Twitter client

Corebird

Birdie

Birdie vs Corebird

Elasticsearch & Kibana

Elasticsearch and Kibana : installation and basic usage on Ubuntu 16.04

TexLive

Recover files from Windows drives

sudo fdisk –l
sudo apt-get install ntfs-3g

sudo mount -t ntfs-3g /dev/sdaX /PARTITION/POINT

RHEL/CentOS

Download urls

Installation screenshots

30 Things to Do After Minimal RHEL/CentOS 7 Installation

http://www.tecmint.com/things-to-do-after-minimal-rhel-centos-7-installation/

Change hostname

  • Change the ^HOSTNAME line in /etc/sysconfig/network
  • Change the hostname in /etc/hosts
  • Run /bin/hostname new_hostname for the hostname change to take effect immediately.
  • Run /sbin/service syslog restart for syslog to log using the new hostname.

Note that using the command line 'hostname' to change the machine's hostname works only for the current session.

Check CentOS version

$ cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

switch to root

su   # Press 'Enter'. It will ask for root's password.

sudoer

Some distributions do not come with sudo command.

As root type:

visudo

and add a line 

MyUserName ALL = ALL

sudo: wheel group

When I install the OS (Red hat 7.4) I have a chance to create a new user with administrator right. This user will be added to the wheel group.

See also How To Create a Sudo User on CentOS

openssh-server

Add an existing user to have sudo privilege

sudo adduser USERNAME sudo

See help.ubuntu.com.

What is my IP address

ifconfig eth0

What services get started at boot time

chkconfig --list

Is xxx service running

xxx status

What services are currently running

ps -e

and 

lsof -i

will show you services that are listening to TCP or UDP endpoints.

Choosing a web hosting service for your website

Install Apache

# Step 1: Install Apache
sudo yum -y update
sudo yum -y install httpd

# Step 2: Allow Apache Through the Firewall
sudo firewall-cmd --permanent --add-port=80/tcp
sudo firewall-cmd --reload
netstat -ant | grep :80

# Step 3: Configure Apache to Start on Boot
sudo systemctl start httpd
sudo systemctl enable httpd
sudo systemctl status httpd

Open ports in a firewall

On CentOS/RHEL 7. 

# Open port 80
sudo firewall-cmd --zone=public --add-port=80/tcp --permanent
sudo firewall-cmd --reload 
# Check the updated rules with:
firewall-cmd --list-all

On CentOS/RHEL 6 

# Open port 80
sudo iptables -I INPUT -p tcp -m tcp --dport 80 -j ACCEPT
sudo service iptables save

To check 

$ netstat -tulpn | grep 8787
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:8787            0.0.0.0:*               LISTEN      -                   
$ netstat -tulpn | grep 80
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::80                   :::*                    LISTEN      -

What network ports are open: lsof or netstat

lsof stands for List Open Files. 

sudo yum install -y lsof
sudo lsof -i :22   # check port 22
sudo lsof -i :3838 # check port 3838
sudo lsof -i  | grep -i "listen"
              # for example *.ssh (LISTEN) is shown on the last column
# or
netstat -aut  # List only UDP or TCP connections
              # for example 0.0.0.0:ssh
# or
netstat -ant # Disable reverse dns lookup for faster output
             # for example 0.0.0.0:22 is shown on the 4th column
             # for shiny application, netstat -ant can grep 3838, but netstat -aut can't.

You can use lsof command for a list of things (15 Linux lsof Command Examples)

  • List processes which opened a specific file
  • List opened files under a directory
  • List opened files based on process names starting with
  • List processes using a mount point
  • List files opened by a specific user
  • List all open files by a specific process
  • Kill all process that belongs to a particular user
  • List all network connections
  • List processes which are listening on a particular port
  • List all TCP or UDP connections
  • List all Network File System ( NFS ) files

and the netstat command: 10 basic examples of linux netstat command 

# https://cyruslab.net/2014/07/11/installing-netstat-on-centos-7-minimal-installation/
sudo yum install net-tools

What firewall rules do I in place

iptables -L

See this article: 20 Iptables Examples For New SysAdmins from cyberciti.biz.

Routing table

How to read the routing table?

Ubuntu wireless adapter: 

brb@brb-P45T-A:~$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 wlan0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 wlan0

Ubuntu virtual machine: 

brb@vm-1404:~$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG        0 0          0 eth0
10.0.2.0        0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth1
brb@vm-1404:~$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 08:00:27:ee:7d:45  
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:feee:7d45/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:831 errors:0 dropped:0 overruns:0 frame:0
          TX packets:558 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:578902 (578.9 KB)  TX bytes:55508 (55.5 KB)

brb@vm-1404:~$ ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 08:00:27:cb:96:6c  
          inet addr:192.168.1.244  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fecb:966c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:84 errors:0 dropped:0 overruns:0 frame:0
          TX packets:54 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:8287 (8.2 KB)  TX bytes:8966 (8.9 KB)

A default gateway is set as follows: 

route add default gw IP_ADDRESS INTERFACE_NAME

route add default gw 192.168.0.1 wlan0

What packages do I have installed

rpm -qa | less
# or
rpm -qa | grep xxx

Install/uninstall new packages

rpm -Uvh foo-1.0-1.i386.rpm

# To install the package anyway and the same version you are trying to install is already installed
# -i: install
# -U: upgrade
# -h: print 50 hask marks as the package archive is unpacked
rpm -ivh --replacepkgs foo-1.0-1.i386.rpm

To uninstall a package 

rpm -e foo  # not the name of the original package file foo-1.0-1.i386.rpm

What version of package xxx do I have installed

rpm -qi xxx

List of available (uninstalled) packages

yum list available

List All Configured Repositories

yum -v repolist
yum -v repolist | less
yum repolist

To list only enabled repositores 

yum repolist enabled

To list only disabled repositories 

yum repolist disabled

To list available packages under a repo called ksplice-uptrack, enter: 

yum --disablerepo="*" --enablerepo="ksplice-uptrack" list available

yum

Searching Packages

yum search vim gvim emacs

List packages

yum list all  # list all available & installed
yum list installed
yum list available

yum grouplist
yum repolist  # Not reposlist

Display package information

yum info package_name…

Listing Files Contained in a Package

repoquery --list package_name

install/remove

yum install -y packagename1 packagename2
yum remove packagename1 packagename2

yum groupremove group

yum equivalent of apt-get update

yum check-update

sendmail

su                             # type your root password to switch the account
yum install m4 telnet mailx
yum install sendmail sendmail-cf
nano /etc/mail/sendmail.mc
m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf
service sendmail restart
netstat -an | grep :25 | grep tcp
ps -ef | grep -v grep | grep -i sendmail
nano /etc/mail/local-host-names
service sendmail restart
chkconfig sendmail on
useradd testuser1
useradd testuser2   
passwd testuser2 
mail -s "Test mail from testuser1" testuser2
tail /var/log/maillog
su testuser2 # run 'mail' command to see if the mail has been received.

nano /etc/mail/local-host-names # create a line, says, xyz.com
nano /etc/mail/sendmail.cf      # After the line of "Smart" relay host (may be null), edit as the following
                                # DSmailfwd.nih.gov
nano /etc/postfix/main.cf #  change inet_protocols from all to ipv4.
nano /etc/sysconfig/sendmail    # make sure DAEMON=yes
nano /etc/mail/relay-domains    # this is a new file with 1 line 128.231.90.107
service sendmail restart
mail -s "Test mail from testuser1" [email protected]
tail /var/log/maillog           # Should not see any ERR.
netstat -nutlap | grep 25

Power Manager for GNOME

The configuration defaults for GNOME power manager have not installed correctly. Cannot login

This error will results in a log-in problem except root account. The symptom is 50GB in root (/) is used up.

The problem was caused by a bug in yum where /var/cache/yum/x86_64/6Workstation takes about 42GB space. The 'yum' does not remove old generated .sqlite files.

See https://bugzilla.redhat.com/show_bug.cgi?id=632391

I use 'du -k' command to find out which directory took space. I use 'rm' command to delete the contents.

Even I delete the content, the directory still grows up daily.

Upgrade Python from 2.6.x to 2.7.x

This instruction tells how to install Python 2.7 from source. 

yum -y update
yum groupinstall -y 'development tools'
yum install -y zlib-devel bzip2-devel openssl-devel xz-libs wget
wget http://www.python.org/ftp/python/2.7.8/Python-2.7.8.tar.xz
xz -d Python-2.7.8.tar.xz
tar -xvf Python-2.7.8.tar

# Enter the directory:
cd Python-2.7.8

# Run the configure:
./configure --prefix=/usr/local

# compile and install it:
make
make altinstall

# Checking Python version:
[root@nicetry ~]# python2.7 -V
Python 2.7.8

wget --no-check-certificate https://pypi.python.org/packages/source/s/setuptools/setuptools-1.4.2.tar.gz

# Extract the files:
tar -xvf setuptools-1.4.2.tar.gz
cd setuptools-1.4.2

# Install setuptools using the Python 2.7.8:
python2.7 setup.py install
curl https://raw.githubusercontent.com/pypa/pip/master/contrib/get-pip.py | python2.7 -
pip2.7 install virtualenv

Install Meld

Have not found a solution yet. We need to install it from source. However, the source depends on

  • Python 2.7 (see above for the instruction)
  • GTK+ 3.6
  • GLib 2.34
  • PyGObject 3.8
  • GtkSourceView 3.6

(Update) A binary version of meld is already available in the git. See this post. 

$ cd ~/Downloads/
$ git clone https://git.gnome.org/browse/meld
$ cd meld
$ sudo ln -s /home/$USER/Downloads/meld/bin/meld /usr/bin/meld

Install the EPEL repository

https://support.rackspace.com/how-to/install-epel-and-additional-repositories-on-centos-and-red-hat/ 

sudo yum install epel-release

If that command doesn’t work,

  • CentOS and Red Hat Enterprise Linux 6.x
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
sudo rpm -Uvh epel-release-6*.rpm
  • CentOS and Red Hat Enterprise Linux 7.x
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo rpm -Uvh epel-release-latest-7*.rpm

Adding, Enabling, and Disabling a Yum Repository

VirtualBox guest addition

(Works on CentOS 7 & VB 5.0.40) https://wiki.centos.org/HowTos/Virtualization/VirtualBox/CentOSguest 

yum install dkms
yum groupinstall "Development Tools"

After running VBoxLinuxAdditions.run & reboot, GA does not work. However, after I run VBoxLinuxAdditions.run & reboot again, GA works.

(CentOS 6) Check out this post. 

su
# click VirtualBox -> Devices -> Install guest addition
mkdir /media/VirtualBoxGuestAdditions
mount -r /dev/cdrom /media/VirtualBoxGuestAdditions
rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
yum install gcc kernel-devel kernel-headers dkms make bzip2 perl
KERN_DIR=/usr/src/kernels/`uname -r`
export KERN_DIR
cd /media/VirtualBoxGuestAdditions
./VBoxLinuxAdditions.run

(Update for 64-bit CentOS 6.5 + VirtualBox 4.3.18) The installation still failed and it showed a missing package which can be installed with 

yum install kernel-devel-2.6.32-431.el6.x86_64

Then I re-run ./VBoxLinuxAdditions.run to finish the installation of guest addition. Reboot and GA works.

Rockstor Linux

Build and manage your own Linux & BTRFS powered advanced NAS and Cloud storage with ease

  1. Personal Cloud Server
  2. SMB Cloud Server
  3. Traditional NAS server

Kernel

System Call

Other Flavors

Archlinux

Pacman

pacman -Sy openssh

Alpine Linux

Online Tools

Non-boring presentation

Prezi. Also funny comics can be found from xkcd.com.

Flow chart/Mind-mapping

MindMup, LucidChart

Image editor

Pixlr Editor (vs Pinta or Shotwell in Ubuntu)

Create gifs on any platform. An online tool is called Giphy. The source file can be a video file (local < 100MB or youtube). If the source file is a series of png file, click slideshow option. The resulting file can be embedded in html files or downloaded to your local machines.

Video editor

WeVideo (vs OpenShot in Ubuntu)

Virus

Virus Total

Finance

Mint

Retrieved from "https://wiki.taichimd.us/index.php?title=Linux&oldid=14109"