Linux
Some books
- UNIX in a nutshell
- sed & awk
- lex & yacc
- bash
- Linux Pocket Guide
- bash cookbook: pdf is online
- Classic Shell Scripting
- GNU EMACS
- Learning the vi and Vim editors 7th
- UNIX POWER TOOLS
- Bash Pocket Reference
- System Administration 3rd
- TCP/IP 3rd: pdf is online
- DNS and BIND 5th
- Network Troubleshooting Tools
- 25 Free Books To Learn Linux For Free
- 17 books for Linux and open source fans
Beautiful desktop
- http://lifehacker.com/the-aincrad-desktop-1732684767
- http://lifehacker.com/the-distant-pyramid-desktop-1654404411
- http://lifehacker.com/the-flat-n-fuzzy-desktop-1693121677
- http://lifehacker.com/the-midsummer-nights-desktop-1704207155
.desktop file
This is not related to beautiful desktop. It is used to launch applications in Linux. Without the .desktop file, your application won’t show up in the Applications menu and you can’t launch it with third-party launchers such as Synapse and Albert Launcher.
- https://wiki.archlinux.org/index.php/Desktop_entries
- https://www.maketecheasier.com/create-desktop-file-linux/
The .desktop files are commonly saved in
- ~/local/share/applications
- /usr/share/applications
List of installed desktop environment
ls -l /usr/share/xsessions/
Themes
5 of the Best Linux Dark Themes that Are Easy on the Eyes
Virtual consoles/virtual terminals
Linux allows virtual consoles (aka virtual terminals) to be opened while an X Window System is executing.
Use Ctrl + Alt + FX to open a virtual console-- there are six virtual text-based consoles (F1 to F6). Use Alt + F7 (or possibly other keybinds) to return to the X Window System.
Managing devices in Linux -> Fun with device files.
Change console fonts
https://www.linux.com/learn/intro-to-linux/2018/1/how-change-your-linux-console-fonts
Desktops/Workspaces
Ctrl + Alt + -> or Ctrl + Alt + <- to switch workspaces.
Ctrl + Alt + down can list the open applications on the current workspace.
Ctrl + Alt + up can show all workspaces and the open applications. We can use mouse to move an app to any workspace.
SuperKey + left tile a window to left. SuperKey + right tile a window to right.
Complete List of Linux Mint 18 Keyboard Shortcuts for Cinnamon for more examples.
Virtual memory
vmstat – A Standard Nifty Tool to Report Virtual Memory Statistics
hcache
A tool fork from pcstat, with a feature that showing top X biggest cache files globally
How much resource is used by a process
Find the process ID first by ps -ef | grep APPLICATIONAME. Then
ps -p <pid> -o %cpu,%mem,cmd
For example,
$ ps -ef | grep akregator brb 15013 1942 1 10:41 ? 00:00:05 akregator --icon akregator -caption Akregator brb 15186 24045 0 10:50 pts/11 00:00:00 grep --color=auto akregator $ ps -p 15013 -o %cpu,%mem,cmd %CPU %MEM CMD 1.0 0.8 akregator --icon akregator -caption Akregator
All You Need To Know About Processes in Linux
http://www.tecmint.com/linux-process-management/
Things to do after a fresh install of GNU/Linux
- Run upgrade such as apt-get update; apt-get upgrade. It helps to resolve the unmet dependencies issue too.
- Increase audio quality
- Make sure firewall is enabled.
- Disable any unnecessary services
- Install Timeshift
- Install ClamAV / Clamtk antivirus
Query whether the OS is 64-bit or 32-bit
SYSTEM_ARCH=getconf LONG_BIT echo $SYSTEM_ARCH
Switch user in command line
use
su newusername
to switch to another user.
Directory permission / attribute
See http://unix.stackexchange.com/questions/21251/how-do-directory-permissions-in-linux-work
When applying permissions to directories on Linux, the permission bits have different meanings than on regular files.
- The write bit allows the affected user to create, rename, or delete files within the directory, and modify the directory's attributes
- The read bit allows the affected user to list the files within the directory
- The execute bit allows the affected user to enter the directory, and access files and directories inside
When we create a new directory, the attribute is 775. Some pre-created directories (Desktop, Documents, Music, Pictures, Public) have an attribute 755.
Special permissions
- Understanding Linux File Permissions
- How to use special permissions: the setuid, setgid and sticky bits
s bit - setuid, getuid
IP address fundamental
http://www.howtogeek.com/133943/geek-school-learning-windows-7-ip-addressing-fundamentals/
Shell
Change to root shell
The following command will switch to an environment similar to what the user would expect had the user logged in directly.
sudo su - # OR sudo su # OR sudo -s
This can be useful when running 'su' or 'su -' failed because of an authentication failure error (note Ubuntu locked the root account).
See also
- wiki.archlinux.org.
- Difference of 'su', 'sudo -s' and 'sudo -i' from askubuntu.com. 'sudo -s' and 'sudo su'?
For sudo to work, my account ('debian' in this case) has to be included in the config file /etc/sudoers.
debian ALL=NOPASSWD: ALL
When sudo is invoked, it asks for the password of the user who started it.
Switch to another user and run a command
runuser -l command
runuser -l userNameHere -c 'command'
su - command (keep the dash sign after su)
- su means 'substitute user'.
- http://unix.stackexchange.com/questions/156343/pass-arguments-to-a-command-run-by-another-user
- http://unix.stackexchange.com/questions/87860/how-does-this-su-c-command-seem-to-pass-two-commands-instead-of-one
su - username -c 'command' sudo su - # switch to root account whoami sudo su - -c "R -q -e \"install.packages('mypackage', repos='http://cran.rstudio.com/')\"" # OR sudo su -c "COMMAND_REQUIRE_ROOT_ACCESS" # OR sudo "COMMAND_REQUIRE_ROOT_ACCESS" man su
What’s the Difference Between Bash, Zsh, and Other Linux Shells
https://www.howtogeek.com/68563/htg-explains-what-are-the-differences-between-linux-shells/
Bash shell programming
http://bash.cyberciti.biz/guide/Main_Page
Fish shell
Oh My Fish! Make Your Shell Beautiful
Redirect standard error
http://bash.cyberciti.biz/guide/Standard_error. Use 2> operator.
command 2> errors.txt
Quotes and asterisk
Combining these two will not work. For example
brb@T3600 ~ $ ls -l ~/GSE48215/*.fastq -rw-r--r-- 1 brb brb 16226673016 Jun 14 14:13 /home/brb/GSE48215/SRR925751_1.fastq -rw-r--r-- 1 brb brb 16226673016 Jun 14 14:13 /home/brb/GSE48215/SRR925751_2.fastq brb@T3600 ~ $ ls -l '~/GSE48215/*.fastq' ls: cannot access ~/GSE48215/*.fastq: No such file or directory brb@T3600 ~ $ ls -l "~/GSE48215/*.fastq" ls: cannot access ~/GSE48215/*.fastq: No such file or directory
ls command
To use UID/GID instead of the user name and group name in ls -l, use the -n option.
ls -n
To make a pretty output by showing selected columns (col 9 is the file name and col 5 is the file size)
$ ls -nt bad | grep -v ^total | awk '{ printf "%-20s %15i\n", $9, $5}' recal.bai 8069704 recal.bam 12275091222 recal_data.table 1012453 realigned_reads.bai 8065496
cp command
Linux cp command tutorial for beginners (8 examples)
copy a directory
cp -avr Dir1 Dir2
where -a will preserve the attributes of files/directories, -v means verbally and -r means copy the directory recursively.
Copy a file with progress bar with pv (plus how to eject the USB drive)
http://www.tecmint.com/monitor-copy-backup-tar-progress-in-linux-using-pv-command/
sudo apt-get install pv pv file1 > file2 # don't forget the ">" operator and the destination is a file, not a directory
After that, instead of clicking the reject icon from the file manager to eject it, it is better to use a command line to do that because there is no expect time for users to know when it will take for finish writing the data to a USB drive.
sudo apt-get install udisks sudo udisks --unmount /dev/sdb1 # /dev/sdb1 is the partition sudo udisks --detach /dev/sdb # /dev/sdb is the device
My testing shows this procedure works (tested by running md5sum after eject/plug-in) when I need to copy a 9GB file.
Reliable way: Split the large file and copy smaller chunks
# Use 'sudo iotop -o' to monitor the I/O split -b 4G inputFile # create xaa, xab, ... files cat x* > outputFile # merge them. md5sum check succeeds type x* > outputFile # Windows OS. http://stackoverflow.com/questions/60244/is-there-replacement-for-cat-on-windows
It is interesting copying smaller files (eg 4GB) to USB drives is quite stable (just use the cp command). Even for a not-too large file (6.7GB), pv step looks OK but the unmount/detach step failed.
For a 6.7GB file, it will split it into a 4GB and 2.7GB files. Merge takes longer time if it is done on the USB drive. That is, it is best to do merge in the final destination (internal disk/storage).
- split in the internal hdd: 1min 38sec
- merge in the internal hdd: 37sec
- merge in the USB 3.0 drive: 2min 17sec
Remember: Use a reliable USB drives.
The operation could not be completed because the volume is dirty
On a USB 2.0 drive, I can copy files to there but the drive cannot be rejected (Ubuntu has a pop-up showing it is still writing data to it).
When I forcibly rejects the drive and plug it in a Windows PC, Windows shows the message The operation could not be completed because the volume is dirty. This gives a way to run chkdsk (check and repair a file system).
- Open a Windows File Manager
- Right click the USB drive
- Properties
- Tools -> Check now... Start
Done. Now I can use the drive again.
The Linux equivalent to chkdsk is fsck. fsck is a front end that calls the appropriate tool (fsck.ex2, fsck.ex3, e2fsck, ...) for the filesystem in question.
umount /dev/sdb1 # thumb drive sudo fsck /dev/sdb1 sudo fsck -a /dev/sdb1 # auto repair
For the root disk, you have to use a live CD. Otherwise, you will see a message like
$ fsck /dev/sdb1 fsck from util-linux 2.20.1 e2fsck 1.42.9 (4-Feb-2014) /dev/sdb1 is mounted. WARNING!!! The filesystem is mounted. If you continue you ***WILL*** cause ***SEVERE*** filesystem damage. Do you really want to continue<n>? no
Multiple files, new directory
rm -r ~/Documents/htg/{done,ideas,notes}
mkdircd MyNewDirectory
alias
https://www.cyberciti.biz/faq/how-to-turn-on-or-off-colors-in-bash/
$ alias | grep ls $ unalias ls $ alias ls='ls --color=auto' # save it in ~/.bash_profile or ~/.bashrc
To avoid using the alias, use one of the following ways (eg use the command's full path)
$ \ls $ /bin/ls $ command ls $ 'ls'
ls
Follow the symbolic link
Use -H option
ls -lH myDir
ls | more without lose color
$ ls --color=auto $ ls --color | more
Most likely your ls is aliased to ls --color=auto. If you do ls --color (which is morally equivalent to ls --color=always), that will force it to turn on colors.
ls output with color background
In my case, after I apply chmod 755 -R XXXX, the weird green background color goes away.
ls on BSD/macOS
Use the -G option to get a color output
$ ls -G
Meld and Diffuse
To make meld to be in the right click menu, follow
- http://askubuntu.com/questions/112164/how-can-i-diff-two-files-with-nautilus
- http://superuser.com/questions/307927/right-click-files-to-meld
Another method of comparing two files without using the 'browse' button will be to use the command line.
The 'nautilus-compare' program does not work from my testing on Ubuntu 14.04.
Refresh does not work
On Ubuntu 14, Meld version is 1.8.4. The current version is 1.16.2 (Jul 30 2016). The current version requires GTK+ 3.14 or higher.
brb@brb-P45T-A:$ ~/binary/meld-3.16.2/bin/meld Meld requires GTK+ 3.14 or higher.
- http://unix.stackexchange.com/questions/149377/how-to-install-meld-3-11-in-ubuntu-14-04
- http://askubuntu.com/questions/638443/how-to-upgrade-gtk-3-10-to-gtk-3-14-on-ubuntu-14-04
Final though
- I install kdiff3 (<2 MB to download) and the 'File' -> 'Reload' (F5) function there works though it shows an extra space on the place I modified.
- Beyond Compare (commercial $60, trial version can be downloaded)
- diffuse. When I modified a file, diffuse can detect a change and ask me to reload the file. I am using the apt-get to install the software and the version number is 0.4.7 (2014). To copy lines from left panel to right panel, use 'Ctrl + Shift + >' or the Copy Selection Right icon. One drawback is it cannot save the history from the GUI though we can use the command line to include the file names in the arguments.
- Alternatively we can use WinMerge on Linux. To do that, install Wine on Ubuntu. Download Winmerge (I am using 2.14.0). Then on a terminal, run the following command. At the end, WinMerge will be launched. WinMerge can also be launched from Mint Menu -> Wine -> WinMerge. One problem is I cannot increase the font size (though acceptable) from View -> Select Font.
wine WinMerge-2.14.0-Setup.exe
diff
Run diff with large files
Meld freezes When I tested it with two large files (800k & 936k lines coming from human gtf files). Actually the whole linux system became unresponsive.
Actually Meld is sluggish when it is used in small files in Odroid XU4 running Ubuntu 16.04 MATE. I have used Meld 3.14.2 and the latest 3.16.2.
Directory
diff -qr dir1 dir2
where -q means to report only when files differ and -r is to recursively compare any subdirectories found.
diff & colordiff-color in terminal
PS. For a GUI version of diff, Meld works fine. Need to install first. apt-get install colordiff. http://www.cyberciti.biz/programming/color-terminal-highlighter-for-diff-files/
sudo apt-get install colordiff diff -y file1 file2 | colordiff # Ignore same rows (two ways): # diff -C0 file1 file2 | colordiff # diff -U0 file1 file2 | colordiff # On systems that I have no root right, I need to install it from the source (just need to run the 'make') $ diff file1 file2 | ~/bin/colordiff-1.0.18/colordiff.pl
where -y option means to show the output in two columns.
Interpretation of the diff output:
The first line of the diff output will contain:
- line numbers corresponding to the first file,
- a letter (a for add, c for change, or d for delete), and
- line numbers corresponding to the second file.
In our output above, 2,4c2,4 means: "Lines 2 through 4 in the first file need to be changed in order to match lines 2 through 4 in the second file." It then tells us what those lines are in each file:
- Lines preceded by a < are lines from the first file;
- lines preceded by > are lines from the second file.
- The three dashes ("---") merely separate the lines of file 1 and file 2.
2,4c2,4 < I need to run the laundry. < I need to wash the dog. < I need to get the car detailed. --- > I need to do the laundry. > I need to wash the car. > I need to get the dog detailed.
colordiff -ur path1 path2 # If you change -ur to -urN then that will also show the contents of files that are only present in one of the paths.
The meaning of colors can be found in /etc/colordiffrc (man colordiff)
- plain=off
- newtext=darkgreen
- oldtext=darkred
- diffstuff=darkcyan
- cvsstuff=cyan
gnome-terminal
Remember the session
- gnome-terminal --help-all --tab-with-profile
- Can no longer set terminal title in Ubuntu 16 (gnome-terminal)
- Opening multiple tabs with gnome-terminal: use --tab and profile options
- Save multiple gnome-terminal layout?: --load-config and --save-config options. NOTE gnome 3.18 option "--save-config" is no longer supported. But strangely enough, "--load-config" is still there.
- How to remember multiple tabs' session in terminal? (Alike FireFox session manager): --profile= and --save-config options. --working-directory and --tab options.
The following is proved working on Ubuntu 18.04
gnome-terminal --tab --working-directory=$HOME/Downloads --tab --working-directory=$HOME/Documents
Fun: piano
Let Us Play Piano In Terminal Using Our PC Keyboard
Terminals in grids
Terminator
- 20 Useful Terminal Emulators for Linux
- https://wiki.archlinux.org/index.php/Terminator (include some keyboard shortcuts)
- Ctrl + Shift+ O Split terminals horizontally
- Ctrl + Shift+ E Split terminals vertically
- mouse can be used to resize split screens and switch to each screen
- Change the font size
- Keyboard shortcuts
- Shift + Ctrl + p/n: switch to the previous/next view
- Ctrl -: decrease font
- Shift Ctrl +: increase font
- You can take a screenshot to record the directories for all split screens.
Byobu
Byobu is a GPLv3 open source text-based window manager and terminal multiplexer. It appeared at Think inside the box.
How To Install and Use Byobu for Terminal Management on Ubuntu 16.04
- https://help.ubuntu.com/community/Byobu
- F1 to go to help, e.g. key bindings. ESC to go back.
- F9 - change configuration
- Mouse is not useful.
- Cheat sheet
- Shift + Alt + arrow keys: resize split screen
- Note that even you closed a byobu window, it is still in the background. F3/F4 will let you move to the previous/next window
- split screen : https://askubuntu.com/questions/493082/how-to-split-byobu-screen. Horizontal: Shift + F2. Vertical: Ctrl + F2. Note that you cannot use mouse to move the focus:(
- Legend of color-coded values at the bottom of byobu
tmux
GNU screen
- https://www.rackaid.com/blog/linux-screen-tutorial-and-how-to/
- http://unix.stackexchange.com/questions/7453/how-to-split-the-terminal-into-more-than-one-view
How to do it...
- Run screen command first (run sudo apt-get install screen if necessary). You are now inside of a window within screen. This functions just like a normal shell except for a few special characters.
- Create screen windows: Ctrl + a, then c. To close a screen window: exit. Once you close all screen windows, you shall see a message [screen is terminating] on the terminal.
- View a list of open windows: Ctrl + a, then ".
- Switch between windows: Ctrl + a and n for the next window and Ctrl +a and p for the previous window.
- Attaching to and detaching screens: To detach (save) from the current screen session, Ctrl +a, and d (these keyboard shortcuts won't affect current execution). This will drop you into your shell. This is useful when you need to run a time-consuming job or your connection is dropped. To attach to an existing screen, use:
screen -r -d
- Split screen:
- To split the screen horizontally, Ctrl +a and S (capital).
- To unsplit the screen, Ctrl +a and Q (capital).
- To switch from one to the other: Ctrl +a and TAB.
- Note: After splitting, you need to go into the new region and start a new session via Ctrl + a then c before you can use that area.
Guake / Yakuake / Tilda
Drop down terminals for the GNOME / KDE / GTK Environments. Great for quick access to a terminal!
System date/time, ntpd
- Install and configure Network Time Protocol (NTP) Server,Clients on Ubuntu 16.10 Server
- How to bind ntpd to specific IP addresses on Linux/Unix
Change the date/timestamp of a file - touch
Modify the file relative to its existing modification time
filename=MyFileName touch -d "$(date -R -r $filename) - 2 hours" $filename # 2 hours before touch -d "$(date -R -r $filename) + 2 hours" $filename # 2 hours later
See How can I change the date modified/created of a file?
Find binary file location
- which - Display the full path of shell commands. See examples from cyberciti.biz.
$ which ls /bin/ls
- whereis - locate the binary, source, and manual page files for a command. See examples from cyberciti.biz.
$ whereis ls ls: /bin/ls /usr/share/man/man1p/ls.1p.gz /usr/share/man/man1/ls.1.gz
- type -a
$ type -a ls ls is aliased to `ls --color=tty' ls is /bin/ls
Use locate command mindfully. It is used to find the location of files and directories. Note that locate does not search the files on disk rather it searches for file paths in a database. For example, the following command will search .png files over the system (not only the personal directory).
locate "*.png"
find: Find a file based on file name
$ find . -iname '*.txt' # -iname or -name is necessary
It also works for searching files on subdirectories.
$ find . -name transcripts.gtf ./RH_bio/transcripts.gtf ./dT_ori/transcripts.gtf ./dT_tech/transcripts.gtf ./dT_bio/transcripts.gtf ./RH_ori/transcripts.gtf ./RH_tech/transcripts.gtf
Find files and execute something (google: find --exec)
$ find ./ -name "*.tar.gz" -exec tar zxvf {} \;
Find files modified in one day.
$ find . -mtime -1
Find files modified in one day and contain string 'est'
$ find . -mtime -1 -exec grep --with-filename est {} \;
If the search directory is not the current directory, we need to add a forward slash to the directory name.
$ find ~/Desktop -iname '*.txt' # Not working $ find ~/Desktop/ -iname '*.txt' # Working
The following example shows we can list multiple search criteria. The “‑r” option in tar appends files to an archive. xargs is a handy utility that converts a stream of input (in this case the output of find) into command line arguments for the supplied command (in this case tar, used to create a backup archive).
find / -type f -mtime -7 | xargs tar -rf weekly_incremental.tar gzip weekly_incremental.tar
xargs
-exec COMMAND {} +
- Find the total file size of a list of files.
- Why does 'find -exec cmd {} +' need to end in '{} +'?
- How to run find -exec?
The following will find out the total file size of the 'accepted_hits.bam' file under all sub-directories.
find ./ -iname "accepted_hits*" -exec du -ch {} + | grep total$
where '-c' produces a grand total, and will substitute {} with the filename(s) found in -exec.
grep: Find a file by searching contents
grep -r -i "Entering" ~/Downloads/R-3.0.0/
where -r means recursively searching the directory and -i means case insensitive.
Sometimes using -R is more effective because of the symbolic links issue.
$ grep -r -i phpmyadmin /etc/apache2/ # nothing returned $ grep -R -i phpmyadmin /etc/apache2/ /etc/apache2/conf-enabled/phpmyadmin.conf:# phpMyAdmin default Apache configuration ... /etc/apache2/conf-available/phpmyadmin.conf:# phpMyAdmin default Apache configuration ...
We can also display the row numbers for matches by using the -n parameter in grep.
# What variants appear in dbsnp grep -n 'rs[0-9]' XXX.vcf
To exclude lines with a pattern, using the -v parameter.
# How many variant were called grep -v "^#" XXX.vcf | head
To show only matched filenames, using the -l parameter.
grep -l "iterator" *.cpp # if we add '-n', the '-n' option won't work.
To search with certain file extensions, use --include argument; see this post.
grep -r -i --include \*.h --include \*.cpp KEYWORD ~/path[12345] # escape with \ just in case you have a directory with asterisks in the filenames
If the pattern is saved in a file, use the -f parameter
grep -f PATTERNFILE INPUTFILE
If there are two keywords, use the following
$ grep "begin\|completed" --color swarm_58606147_0.o # needs an escape begin 2018-01-12 14:46:05 alignment is completed 2018-01-12 16:45:24 marking duplication is completed 2018-01-12 17:52:01 assign read group is completed 2018-01-12 18:22:49 indel re-alignment is completed 2018-01-12 19:29:32 BQSR is completed 2018-01-12 22:26:22 GATK is completed 2018-01-12 23:43:3 $ egrep "begin|completed" --color swarm_58606147_0.o # no need an escape if we use extended regular expressions
We can use R to compute the time spent in each step; see Dealing with dates.
Check https://www.howtoforge.com/tutorial/linux-grep-command/ for more examples
- Using grep to search only for words ("-w" option)
- Using grep to search two different words (egrep -w 'word1|word2' /path/to/file)
- Count line for matched words ("-c" option)
- Grep invert match ("-v" option)
- How to list only the names of matching files ("-l" option)
GUI
A GUI version of a tool to search files is searchmonkey (open source, Linux, Windows). On Ubuntu, we install it by
sudo apt-get install searchmonkey
It is also useful to change the settings so we can click a filename and open it in the desired text editor. To do that, go to Settings -> Preferences -> System Call -> Text Editor. I enter 'geany' since I want to use geany to open my C programs. Note. the v2.0 source code needs to be built using i386 gcc library and Qt 4.8.x. Still, I cannot get rid of some errors coming from the source code.
Summary of find and grep commands
Command | Examples |
---|---|
find | find [DIRECTORY] -iname '*.txt'
find [DIRECTORY] -maxdepth 2 -iname *.php find -name '*.php' -o -name '*.txt' # OR operator |
grep | grep -r -i "check_samtools" DIRECTORY/
dpkg -l libgtk* | grep '^i' |
Count number of columns: awk
The following command shows the number of columns for the first few rows of a text file.
head MYFILE | awk '{ print NF}' head MYFILE | awk -F '\t' '{ print NF}'
Count number of rows in a file: wc
wc -l MYFILE
The source code of wc (or any Linux command) can be found by using this method
brb@brb-T3500:~/Downloads$ which wc /usr/bin/wc brb@brb-T3500:~/Downloads$ dpkg -S /usr/bin/wc coreutils: /usr/bin/wc brb@brb-T3500:~/Downloads$ sudo apt-get source coreutils [sudo] password for brb: Reading package lists... Done Building dependency tree Reading state information... Done Need to get 12.3 MB of source archives. Get:1 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main coreutils 8.21-1ubuntu5.1 (dsc) [1,635 B] Get:2 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main coreutils 8.21-1ubuntu5.1 (tar) [12.3 MB] Get:3 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main coreutils 8.21-1ubuntu5.1 (diff) [31.6 kB] Fetched 12.3 MB in 22s (559 kB/s) gpgv: Signature made Tue 13 Jan 2015 10:33:04 PM EST using RSA key ID 9D8D2E97 gpgv: Cannot check signature: public key not found dpkg-source: warning: failed to verify signature on ./coreutils_8.21-1ubuntu5.1.dsc dpkg-source: info: extracting coreutils in coreutils-8.21 dpkg-source: info: unpacking coreutils_8.21.orig.tar.gz dpkg-source: info: applying coreutils_8.21-1ubuntu5.1.diff.gz
As we can see from the coreutils-8.21/src directory, there are over 100 C programs including <cat.c>, <chmod.c>, <cp.c>, ...<wc.c>.
Print certain rows/lines of a text file
The following example will print out lines 10 to 60 of FILENAME.
sed -n '10,60p' FILENAME
Or to print out line 60,
sed -n '60p' FILENAME
It seems this method is not as fast as I expected. For example, the tail command will immediately print out the result without waiting!
output colored console to html
Use ansi2html.sh. It only requires gawk.
- Use wget to download it
- sudo apt-get install gawk
- chmod +x ansi2html.sh
- colordiff file1 file2 | ./ansi2html.sh > diff.html
using a the result of a diff in a if statement
ls -lR $dir > a ls -lR $dir > b DIFF=$(diff a b) if [ "$DIFF" != "" ] then echo "The directory was modified" fi
Another example
if [ "$(diff file1.html file2.html)" == "" ]; then echo Same; else echo Different; fi
Prompt
Colored prompt
- http://www.tldp.org/HOWTO/Bash-Prompt-HOWTO/x329.html
- https://www.cyberciti.biz/faq/bash-shell-change-the-color-of-my-shell-prompt-under-linux-or-unix/
For example, the following code will change the prompt to a light blue color. NOTE that we need \[ and \] in order to avoid a problem of miscalculating the cursor's starting position.
# blue export PS1='\[\e[1;34m\]\u@\h:\w\$ \[\e[0m\]' # bright blue (good) export PS1='\[\e[0;34m\]\u@\h:\w\$ \[\e[0m\]' # darker blue # yellow export PS1='\[\e[1;33m\]\u@\h:\w\$ \[\e[0m\]' # bright yello export PS1='\[\e[0;33m\]\u@\h:\w\$ \[\e[0m\]' # dark yellow (good) # red export PS1='\[\e[1;31m\]\u@\h:\w\$ \[\e[0m\]' # bright red export PS1='\[\e[0;31m\]\u@\h:\w\$ \[\e[0m\]' # dark red (good) # green export PS1='\[\e[1;32m\]\u@\h:\w\$ \[\e[0m\]' # bright green export PS1='\[\e[0;32m\]\u@\h:\w\$ \[\e[0m\]' # dark green # cyan export PS1='\[\e[1;36m\]\u@\h:\w\$ \[\e[0m\]' # bright cyan export PS1='\[\e[0;36m\]\u@\h:\w\$ \[\e[0m\]' # dark cyan (good) # purple export PS1='\[\e[1;35m\]\u@\h:\w\$ \[\e[0m\]' # bright purple (good) export PS1='\[\e[0;35m\]\u@\h:\w\$ \[\e[0m\]' # dark purple
To make a permanent change, we can add the line to ~/.bashrc file and (is it necessary) un-comment the following line
force_color_prompt=yes
Add a timestamp to your Bash prompt
- See man 3 strftime for the date, time format. In Linux Mint, the date applets uses the %A %B %e, %H:%M which gives a format like 'Friday July 15, 10:23'.
- For a Good Strftime - Online date/time formatting tool
- http://bneijt.nl/blog/post/add-a-timestamp-to-your-bash-prompt/. Set
PS1 | Prompt |
---|---|
default | brb@p45t:~/Downloads$ |
PS1='[\D{%F %T}] \u@\h \W\$ ' | [2016-07-08 16:56:48] brb@brb-P45T-A ~/Downloads$ |
PS1="\[\033[1;34m\]\$(date +%H:%M%p) \w$\[\033[0m\] " | 10:54AM ~/Downloads$ |
From here, we can skip %F (not showing the date), \W (not showing the current directory) and change %T to %H:%M (not showing seconds).
- http://askubuntu.com/questions/193416/adding-timestamps-to-terminal-prompts. Add this line to the ~/.bashrc file:
export PROMPT_COMMAND="echo -n \[\$(date +%H:%M%p)\]\ "
and the output will be something like:
[07:03AM] user@hostname:~$
To the right hand side/Aligned to right and zsh
- See an example from Biolinux. echo $SHELL shows Bio-Linux is using zsh.
- http://zsh.sourceforge.net/Intro/intro_14.html
- https://www.howtoforge.com/tutorial/how-to-setup-zsh-and-oh-my-zsh-on-linux/. Note that there is a green arrow for zsh. This is quite special. I need to reboot to see a switch to zsh.
- https://superuser.com/questions/362372/how-to-change-the-login-shell-on-mac-os-x-from-bash-to-zsh chsh -s /bin/zsh
- https://superuser.com/questions/776759/switch-from-zsh-to-default-os-x. chsh -s /bin/bash
Listen to pandora in Europe: install squid proxy
http://www.cyberciti.biz/faq/access-pandora-radio-using-proxy-server-outside-usa/
Interestingly, the firefox connection settings should choose HTTP Proxy instead of 'SOCKS host'.
scp
file path with spaces
Use double quotes around the full path and a backslash to escape any space.
scp [email protected]:"web/tmp/Master\ File\ 18\ 10\ 13.xls" .
Recursive copying
Use -r parameter.
Preserve permissions and modes
Use -p parameter.
scp files through one intermediate host
http://stackoverflow.com/questions/9139417/how-to-scp-with-a-second-remote-host
The following command is tested.
scp -o 'ProxyCommand ssh user@remote1 nc %h %p' user@remote2:path/to/file .
A second method which is useful for ssh and scp commands
$ ssh -L 9999:host2:22 user1@host1 # leave the terminal # Open a new terminal $ scp -P 9999 fileName user2@localhost:/path/to/dest/fileName # transfer from local to remote. Note: Upper P. $ scp -P 9999 user2@localhost:/path/to/source/fileName fileName # transfer from remote to local. Note: Upper P. # If we only want to use ssh $ ssh -p 9999 user2@localhost # Note: lower p.
scp with non-standard port: -P (capital)
Use -P argument.
scp -P 23 myfile user@remoteip:
scp or ssh without password
- http://www.thegeekstuff.com/2008/06/perform-ssh-and-scp-without-entering-password-on-openssh/
- https://toic.org/blog/2008/ssh-basics/
Steps:
- Verify that local-host and remote-host is running openSSH (ssh -V)
- Generate key-pair on the local-host using ssh-keygen (Enter a passphrase here, do not leave it empty. A passphrase should be at least several words long, something you can easily remember. It's a bad idea to use a single word as a passphrase.)
- Install public key on the remote-host
- Give appropriate permission to the .ssh directory on the remote-host (chmod 755 ~/.ssh; chmod 644 ~/.ssh/authorized_keys)
- Login from the local-host to remote-host using the SSH key authentication to verify whether it works properly
- Start the SSH Agent on local-host to perform ssh and scp without having to enter the passphrase several times (ssh-agent $SHELL)
- Load the private key to the SSH agent on the local-host (ssh-add, need to enter the passphrase 1 time only)
- Perform SSH or SCP to remote-home from local-host without entering the password. It works for all remote machines containing the key from local-local.
Another option is to use ssh -i IDENTITY_FILE. See superuser.com.
ssh with password on the command line
Install sshpass utility. See https://serverfault.com/questions/241588/how-to-automate-ssh-login-with-password
SSH
Best security practices
Top 20 OpenSSH Server Best Security Practices
- Use SSH public key based login
- Disable root user login
- Disable password based login
- Limit Users’ ssh access
- Disable Empty Passwords
- Use strong passwords and passphrase for ssh users/keys
- Firewall SSH TCP port # 22
- Change SSH Port and limit IP binding
- Use TCP wrappers (optional)
- Thwart SSH crackers/brute force attacks such as using fail2ban and DenyHosts software
- Rate-limit incoming traffic at TCP port # 22 (optional)
- Use port knocking (optional)
- Configure idle log out timeout interval
- Enable a warning banner for ssh users
- Disable .rhosts files (verification)
- Disable host-based authentication (verification)
- Patch OpenSSH and operating systems
- Chroot OpenSSH (Lock down users to their home directories)
- Disable OpenSSH server on client computer
- Bonus tips from Mozilla
Install OpenSSL
How to Install the latest OpenSSL version from Source on Linux
Way to avoid ssh connection timeout
- https://superuser.com/questions/98562/way-to-avoid-ssh-connection-timeout-freezing-of-gnome-terminal
- man ssh_config
Put the following in your ~/.ssh/config.
Host remotehost HostName remotehost.com ServerAliveInterval 240
To enable it for all hosts use:
Host * ServerAliveInterval 240
Also make sure to run chmod 600 ~/.ssh/config
Change to a different port
$ sudo nano /etc/ssh/sshd_config # looking for the line containing port $ sudo service ssh restart # tested on Ubuntu 14.04
Remember to change the Router settings.
On the client PC, use ssh USERNAME@HOSTNAME -p NEWPORT for a connection.
For security reason, use the port < 1024 (privileged ports and can only be opened by root)
- Why putting SSH on port 2222 is a bad idea
- Here is a list of TCP and UDP port numbers.
ssh alias
With this trick, ssh and scp (scp alias_name:Downloads/myfile .) work perfectly.
Modify ~/.ssh/config
Host * ServiceAliveInterval 120 ServiceAliveCountMax 30 Host your-alias_name User username HostName remote.sshserver.com Port 50001 IdentifyFile ~/.ssh/id_file ServiceAliveInterval 120 Host work User abcde HostName work.workserver.com ServiceAliveCountMax 5 StrictHostKeyChecking yes
Running commands on a remote host
ssh user@host 'COMMANDS' ssh user@host "command1; command2; command3" COMMANDS="command1; command2; command3" ssh user@host "$COMMANDS"
A practical example
#!/bin/bash IP_LIST="192.168.0.1 192.168.0.5 192.168.0.9" USER="test" for IP in $IP_LIST; do utime=$(ssh ${USER}@${IP} uptime | awk '{ print $3 }' ) echo $IP uptime: $utime done
Disable root log in
Modify /etc/ssh/sshd_config. Change this line:
#PermitRootLogin yes
to
PermitRootLogin no
and run /etc/init.d/sshd restart.
However, that line in my Ubuntu is
PermitRootLogin without-password
According to this post, “without-password” means password authentication is disabled for root.
ssh log files
- /var/log/syslog
- /var/log/auth.log (see who is trying to connect; check out http://ip-lookup.net/index.php to see their geolocation)
It is also helpful to check /etc/hosts.allow and /etc/hosts.deny for any possible wrong configuration.
Note that auth.log can show ssh security attacks.
$ grep sshd /var/log/auth.log Feb 19 11:04:12 phenom sshd[16922]: Failed password for root from 92.62.131.23 port 49383 ssh2 Feb 19 11:04:12 phenom sshd[16922]: Received disconnect from 92.62.131.23: 11: Bye Bye [preauth] Feb 19 11:04:14 phenom sshd[16924]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=92.62.131.23 user=root Feb 19 11:04:36 phenom sshd[16998]: Invalid user enea from 113.160.227.93 Feb 19 11:04:36 phenom sshd[16998]: input_userauth_request: invalid user enea [preauth] Feb 19 11:04:37 phenom sshd[16998]: pam_unix(sshd:auth): check pass; user unknown Feb 19 11:04:37 phenom sshd[16998]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=113.160.227.93 Feb 19 11:04:39 phenom sshd[16998]: Failed password for invalid user enea from 113.160.227.93 port 36090 ssh2 Feb 19 11:04:39 phenom sshd[16998]: Connection closed by 113.160.227.93 [preauth] Feb 19 11:05:11 phenom sshd[17060]: refused connect from 58.218.198.170 (58.218.198.170) Feb 19 11:05:55 phenom sshd[17353]: refused connect from 58.218.198.170 (58.218.198.170) Feb 19 11:06:38 phenom sshd[17732]: refused connect from 58.218.198.170 (58.218.198.170) Feb 19 11:07:20 phenom sshd[17850]: refused connect from 58.218.198.170 (58.218.198.170) Feb 19 11:07:40 phenom sshd[17874]: refused connect from 221.194.47.221 (221.194.47.221) Feb 19 11:08:01 phenom sshd[17955]: refused connect from 58.218.198.170 (58.218.198.170) Feb 19 11:08:41 phenom sshd[18118]: refused connect from 58.218.198.170 (58.218.198.170) Feb 19 11:09:22 phenom sshd[18280]: refused connect from 58.218.198.170 (58.218.198.170) Feb 19 11:10:02 phenom sshd[18353]: Invalid user support from 103.89.89.223 Feb 19 11:10:02 phenom sshd[18353]: input_userauth_request: invalid user support [preauth] Feb 19 11:10:02 phenom sshd[18353]: pam_unix(sshd:auth): check pass; user unknown Feb 19 11:10:02 phenom sshd[18353]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.89.89.223 Feb 19 11:10:03 phenom sshd[18424]: refused connect from 58.218.198.170 (58.218.198.170) Feb 19 11:10:04 phenom sshd[18353]: Failed password for invalid user support from 103.89.89.223 port 54218 ssh2 Feb 19 11:10:05 phenom sshd[18353]: fatal: Read from socket failed: Connection reset by peer [preauth] Feb 19 11:10:07 phenom sshd[18425]: Did not receive identification string from 103.89.89.223 Feb 19 11:10:17 phenom sshd[18443]: Address 113.160.227.93 maps to static.vnpt.vn, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
DenyHosts
Note that denyhosts package is no longer available in Ubuntu 14.04, 16.04 now. We can install install from its source DenyHosts-2.6.tar.gz.
- How To Install DenyHosts on Ubuntu 16.04 LTS and https://www.cyberciti.biz/faq/how-to-install-denyhosts-intrusion-prevention-security-for-ssh-on-ubuntu/
- https://www.digitalocean.com/community/tutorials/how-to-install-denyhosts-on-ubuntu-12-04
- tecmint.com or howtoforge (installed from source)
- /etc/hosts.deny will records the IPs that are blocked. If the normal ssh connection failed (e.g. get a message ssh_exchange_identification: read: Connection reset by peer), check /etc/hosts.deny file to see if your IP is in it. One method is to add your IP to /var/lib/denyhosts/allowed-host file so your IP won't be blocked.
- Visualising SSH attacks with R
- A few minutes of run of denyhosts accumulates hundreds of IP in /etc/hosts.deny file. But I remove the service since I did not spend enough time to understand it.
Procedures: follow the README.txt file.
Log in history: last command
The following command also shows how long a user has been logged in.
last <username> | less
w/who can show who (and when) are currently logging in.
Generate a strong password
Put in your ~/.bashrc. See Top 20 OpenSSH Server Best Security Practices.
$ genpasswd() { local l=$1 [ "$l" == "" ] && l=20 tr -dc A-Za-z0-9_ < /dev/urandom | head -c ${l} | xargs } $ genpasswd 16
login banners/messages
https://kerneltalks.com/tips-tricks/how-to-configure-login-banners-in-linux/
There are two types of banners you can configure.
Banner message to display before user log in (configure in file of your choice eg. /etc/login.warn) Banner message to display after user successfully logged in (configure in /etc/motd)
ssh key
SSH key is useful if you want a password-less login to a remote system. Some useful resources:
- https://help.ubuntu.com/community/SSH/OpenSSH/Keys
- https://help.github.com/articles/generating-ssh-keys
Also there are different kinds of keys (see for example <~/.ssh/known_hosts file>): RSA, DSA and ECDSA (newer). They're keys generated using different encryption algorithms. See SSH key-type, rsa, dsa, ecdsa, are there easy answers for which to choose when?
The steps are
- Check if there is an existing key
ls -al ~/.ssh
- Create a new RSA key pair:
ssh-keygen -t rsa ssh-keygen -f ~/.ssh/personalid -C "bitbucket"
where the comment 'bitbucket' will appear at the end of <~/.ssh/personalid> file.
- Copy the public key to a remote host ([email protected]) over ssh. The current user (eg brb) and the remote user (eg git)have not any relationship (they most likely have different user names):
ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected] # this will 'append' the key to the remote-host’s .ssh/authorized_key.
Or (may not work:()
cat ~/.ssh/id_rsa.pub | ssh [email protected] "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
- Delete the authorized key. Open the text file '.ssh/authorized_keys' and remove the offending lines.
- Test if this is working by trying 'ssh [email protected]'.
- To disable the password for root login. Type sudo nano /etc/ssh/sshd_config
PermitRootLogin without-password
Then run the following to put the changes into effect:
reload ssh # Or service ssh restart
If we like to ask all users to use key-based to log in, we can modify the line
PasswordAuthentication no
in sshd_config.
Multiple ssh keys
<Method 1> If we want to use a specific key in ssh, use
ssh -i ~/.ssh/xxx_id_rsa [email protected]
<Method 2> Another way is to use ssh-add & ssh-agent to manager your keys. ssh-agent keeps your key in its memory and pulls it up whenever it is asked for it.
$ ssh-keygen -f ~/.ssh/personalid -C "bitbucket" $ eval $(ssh-agent -s) # Ensure ssh-agent is enabled: $ ssh-add ~/.ssh/personalid # ssh-add program will ask you for your passphrase $ ssh-add -l
<Method 3> <~/.ssh/config> file.
- http://askubuntu.com/questions/269140/how-to-use-multiple-ssh-keys-with-different-accounts-and-hosts or http://nerderati.com/2011/03/17/simplify-your-life-with-an-ssh-config-file/
- Configure multiple SSH identities for bitbucket accounts
- Multiple SSH Keys settings for different github account
ssh key management
- Using privacyIDEA (howtoforge.com).
Copy ssh keys to another computer
http://askubuntu.com/questions/134975/copy-ssh-private-keys-to-another-computer
$ chown brb:brb ~/.ssh/id_rsa* $ chmod 600 ~/.ssh/id_rsa $ chmod 644 ~/.ssh/id_rsa.pub
If we do not change the permission correctly in <id_rsa>, we will get a warning: Unprotected private key file. Permissions 0664 for '/home/USERNAME/.ssh/id_rsa' are too open.
Preserve ssh keys when upgrading computers
- An article from bsdnewsletter.com.
- https://askubuntu.com/questions/17097/how-to-backup-restore-the-host-key-in-ssh-server
ls -l /etc/ssh/*key* > ~/key_list # optional mkdir ~/serverkeys && sudo cp -p /etc/ssh/*key* ~/serverkeys/ # back up, -p will preserve mode, ownership and timestamps sudo cp -p ~/serverkeys/*key* /etc/ssh # copy back ls -l /etc/ssh/*key* | diff - ~/key_list # optional
If diff produces no output, you're finished.
Pay attention to the permissions. All the /etc/ssh/* files should be owned by root:root, with 644 permissions except for those that end in *key, which should be 600.
udooer@udoo:~$ ls -l /etc/ssh/*key* total 32 -rw------- 1 root root 668 Dec 8 14:43 ssh_host_dsa_key -rw-r--r-- 1 root root 599 Dec 8 14:43 ssh_host_dsa_key.pub -rw------- 1 root root 227 Dec 8 14:43 ssh_host_ecdsa_key -rw-r--r-- 1 root root 171 Dec 8 14:43 ssh_host_ecdsa_key.pub -rw------- 1 root root 399 Dec 8 14:43 ssh_host_ed25519_key -rw-r--r-- 1 root root 91 Dec 8 14:43 ssh_host_ed25519_key.pub -rw------- 1 root root 1679 Dec 8 14:43 ssh_host_rsa_key -rw-r--r-- 1 root root 391 Dec 8 14:43 ssh_host_rsa_key.pub udooer@udoo:~$ cd /etc/ssh; sudo tar -czvf ~/Downloads/sshkeys.tar.gz *key* -rw------- root/root 668 2017-12-08 14:43 ssh_host_dsa_key -rw-r--r-- root/root 599 2017-12-08 14:43 ssh_host_dsa_key.pub -rw------- root/root 227 2017-12-08 14:43 ssh_host_ecdsa_key -rw-r--r-- root/root 171 2017-12-08 14:43 ssh_host_ecdsa_key.pub -rw------- root/root 399 2017-12-08 14:43 ssh_host_ed25519_key -rw-r--r-- root/root 91 2017-12-08 14:43 ssh_host_ed25519_key.pub -rw------- root/root 1679 2017-12-08 14:43 ssh_host_rsa_key -rw-r--r-- root/root 391 2017-12-08 14:43 ssh_host_rsa_key.pub udooer@udoo:~/$ cd /etc/ssh; sudo tar -xzvf ~/Downloads/sshkeys.tar.gz
Disable SSH host key checking
ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no [email protected]
To disable the checking for all hosts, in your ~/.ssh/config (if this file doesn't exist, just create it):
Host * StrictHostKeyChecking no
Handling the ssh key change when connecting to a remote machine
An article from cybercitz.biz.
- Method 1. Remove the key using ssh-keygen -R command.
$ ssh-keygen -R {server.name.com} $ ssh-keygen -R {ssh.server.ip.address} $ ssh-keygen -R server.example.com
- Method 2. Add correct host key in /home/user/.ssh/known_hosts
- Method 3. Just delete the known_hosts file If you have only used one ssh server
SSH Port forwarding
- Chapter 9 Port forward. SSH Mastery OpenSSH, PuTTY, Tunnels and Keys by Michael W. Lucas
Verizon Quantum Gateway Router
User guide p98. Click 'Advanced' button first.
- Source port: Any (this is the key)
- Destination Ports: the port you want to use (connect from outside)
- Forward to Port: Same as incoming port or the port used in the local computer
What is tunnel
https://www.howtogeek.com/299845/why-is-a-network-tunnel-called-a-tunnel/. A tunnel provides a direct path that avoids some type of complexity you would otherwise have to deal with.
Local port forwarding
- https://toic.org/blog/2010/ssh-port-forwarding/
- https://www.howtogeek.com/168145/how-to-use-ssh-tunneling/
This port forwarding involves three computers (local, remote and hostname) as you can see from the SSH syntax.
For example, we like to access home's router (192.168.1.1) information from an outsider computer. Suppose the host 'hostname' is one computer in the home network and it can be accessed from outside world.
# ssh -L localhost:localport:remoteIP:remoteport hostname # ssh -L localport:remoteIP:remoteport hostname ssh -L 8080:192.168.1.1:80 username@hostname
The -L option specifies local port forwarding. In this case, port 8080 on the local machine was forwarded to port 80 on the remote machine. For the duration of the SSH session, pointing your browser at http://localhost:8080/ would send you to http://192.168.1.1/ as if you are in the same local network of 'hostname'.
The reason it works is because the 'ssh' trick. In addition to being able to make yourself in the home network environment, the traffic on http://localhost:8080 is encrypted too.
Note that this forwarding uses port 8080 on the client rather than port 80. Binding to port 80 would require using root privileges every time we SSH.
To stop the ssh session, use ps -ef to find the process id and kill it.
Remote port forwarding (Reverse port forwarding)
- https://www.howtoforge.com/reverse-ssh-tunneling (use ssh option -f to detach ssh process from the tty, -N to not execute any command over ssh and option -i for key authentication)
- http://man.openbsd.org/ssh
- https://toic.org/blog/2009/reverse-ssh-port-forwarding/
- https://www.howtogeek.com/168145/how-to-use-ssh-tunneling/
This is most useful in situations where you have a machine which isn't publicly accessible from the internet, but you want others to be able to access a service on this machine. In this case, if you have SSH access to a remote machine which is publicly accessible on the internet, you can set up a reverse port forward on that remote machine to the local machine which is running the service.
ssh -R 8000:localhost:80 user@REMOTE_MACHINE
This will forward port 8000 on the remote machine to port 80 on the local machine. Using this method, if you browse to http://localhost on the remote machine, you will actually connected to a web server running on port 8000 of the local machine.
Example 2: Suppose you have two machine
- machine A (userA): under firewall. cannot be directly accessed (like corporate machines)
- machine B (userB): local machine (like home machines)
Our goal is to access machine A directly from machine B.
We can run the following on the machine A
# ssh -R remoteIP:remoteport:localIP:localport hostname # ssh -R remoteport:localIP:localport hostname ssh -R 2222:localhost:22 userB@machineB_IP ssh -i /path/to/priv/key/id_rsa -f -N -R 2222:localhost:22 userB@machineB_IP
Then we can access machine A from machine B by
ssh -p 2222 userA@localhost
If you want remote port forwarding configured every time you connect to a host, use the RemoteForward option in ssh_config .
LocalForward server-IP:server-port client-IP:client-port
Dynamic port forwarding, SOCKS proxy, bypass blocked websites from work computer
- http://www.panix.com/~ruari/censorship.html
- http://www.cyberciti.biz/faq/set-up-ssh-tunneling-on-a-linux-unix-bsd-server-to-bypass-nat/
- https://www.howtogeek.com/168145/how-to-use-ssh-tunneling/
ssh -D 4096 user@remoteip ssh -D 4096 -p 23 user@remoteip
This will require you to enter the password and leave you in the remote machine. If a nonstandard port is required, we can use -p option.
Now in the firefox, we need to go to Edit -> Preferences -> Advanced -> Network tab -> Settings... Check 'Manual proxy configuration' (The default is 'Use system proxy settings') and enter 'localhost' for SOCKS (SOCKS5 by default) Host and '4096' for the Port. Don't enter 'localhost' in the HTTP Proxy.
Note that in addition to the Firefox, we can use SeaMonkey (seems better than Firefox since the form works better on 1024x600 resolution). The network setting option in my 32-bit maxthon browser does not work (cannot show options). For the Opera browser, it cannot connect to Internet after I made a change to the network setting.
On Windows, we can use Putty. In short, in the left-hand panel, navigate through Connection > SSH > Tunnels. Enter 4096 in the Source Port box and select the Dynamic radio button. Click Add and “D4096″ will appear in the Forwarded Ports list. The setting in the firefox end is the same. See also my Windows wiki page.
Linux journal also put a video on youtube. We can use http://www.ipligence.com/geolocation to check the current location. The port number is 1080 in the example. The example actually also use '-N' option which means no interaction; i.e. ssh -N -D 1080 user@remoteip. So we won't see anything after we type our password. Once we want to stop SOCK proxy, we just need to hit Ctr+C on terminal.
Backgrounding OpenSSH Forwarding
Use the -N flag to tell ssh to not run anything, including a terminal, on the remote server, and the -f flag to tell ssh to go into the background on the client.
ssh -fNL 2222:localhost:22 user@remotehost &
By backgrounding this command, you get your original terminal back.
ssh through an intermediate server
- http://www.cyberciti.biz/faq/linux-unix-ssh-proxycommand-passing-through-one-host-gateway-server/
- https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts#Port_Forwarding_via_an_Intermediate_Host (more examples)
Simple method is
$ ssh -tt vivek@Jumphost ssh -tt vivek@FooServer
Another method is to use ssh ProxyCommand to tunnel connections.
A third method is to
$ ssh -L 9999:host2:22 user1@host1 # leave this terminal # open a new terminal tab $ ssh -p 9999 user2@localhost
Graphical way to display disk usage
For example, to use xdiskusage, we run apt-get install xdiskusage and launch it by xdiskusage ~/.
- Ubuntu has a built-in program called "Disk Usage Analyzer". Just search it from Dash. Looks useful!
Display files sorted by modified date in a directory recursively
stat --printf="%y %n\n" $(ls -tr $(find DIRNAME -type f))
find -type f -printf '%T+\t%p\n' | sort -n
Both of methods give the same output. Note the latest changed file is shown at the bottom of the output.
Sort files by their size
use the '-S' option.
ls -lS
df: Display disk space
df -h df -h -T # show the 't'ype of the file system df -h -t ext4 # show file systems of given type (ext4 in this example) df -a # show all file system (include ones that have a size of zero blocks)
Note for the NTFS type, it will be reported as fuseblk by mount or df command.
rm command and trash can
Make “rm” Command To Move The Files To “Trash Can” Instead Of Removing Them Completely
du and ncdu : Display directory size with sorting and human readable
Use ncdu program (more interactive). Although it is a command line program, we can use the mouse to move through each directory to see its sub-directories.
ncdu can show the hidden directory size. This is useful. For example, ~/.local/share/Trash and ~/.singularity/docker can take a lot of space.
sudo apt-get install ncdu ncdu
And the du method.
du -sh ~/* # won't include hidden directories, Fast du -h ~/ --max-depth=1 # include hidden directories, SLOW du -h ~/ --max-depth=1 --exclude ".*" | sort -nr | cut -f2 | xargs -d '\n' du -sh du -a -h ~/ # kilobytes will be used, '-a' is to see all files, not just directories. du -a ~/ | sort -nr | head -n 10 # sort from the largest file size first
The --exclude is to hide hidden directories, '-n' is to compare according to string numerical value, and '-r' is to reverse the result.
Note that the 'du' commands may be cheating. See the following screenshot.
The discrepancy is explained by 'sector'. See http://askubuntu.com/questions/122091/difference-between-filesize-and-size-on-disk.
$ sudo dumpe2fs /dev/sda1 | grep -i "block size" dumpe2fs 1.41.14 (22-Dec-2010) Block size: 4096
To show a file size in terms of blocks, we can use
ls -s
So for example, if a file takes 150 blocks, and if a block takes 4096 bytes, then the file takes 150*4096/1024 KB on disk.
Apache benchmark (ab) testing
ab -n 100 -c 10 http://taichimd.us/
Monitor progress of running a command
How to monitor progress of Linux commands using PV and Progress utilities
# Method 1: rsync rsync --progress -a sourceDirectory destinationDirectory rsync --info=progress2 source dest # Method 2: pv sudo apt-get install pv ## copy a single file pv inputfile > outputfile ## multiple files or directories tar c sourceDirectory | pv | tar x -C destinationDirectory
rsync
Copy large file
If we need to copy large file (say > 4GB), we shall
- format USB drive to NTFS (exFat seems not work)
- Run rsync --progress source dest
- Run sync
The last step (rsync) is important. We can use sudo iotop to check if rsync is finished or not.
speed comparison of cp vs rsync
BigData basic: copy & delete folder containing large number of files
rsync with exclude files/directories
See http://www.thegeekstuff.com/2011/01/rsync-exclude-files-and-folders/. The key is excluded files are relative to the current directory even we specify the absolute path. For example /path1/path2/file does not mean the file is located under /path1/path2; it means the file is located under ./path1/path2.
rsync -avz --exclude '/path1/path2/file' source/ destination/
We add add multiple --exclude to exclude more files/directories.
--exclude=".*" # exclude both hidden files and directories --exclude ".*" # same as above --exclude ".*/" # exclude hidden directories ONLY --exclude ".git" # exclude .git directory ONLY; relative to the directory to be synchronized.
rsync with -a option
The -a flag in there stands for “archive,” and it’s important to include. It makes sure that the sync command is recursive (meaning any sub-folders and files inside of old_movies are copied too) and it’s important for preserving all of those modification dates, symbolic links, permissions, and other goodies we talked about earlier.
rsync with non-standard port
Use -e option
rsync -avz -e "ssh -p 23" mydir user@remoteip:
rsync with progress bar
Use --progress option.
rsync -avz --progress file1 file2
The 'rsync' command works on transferring files local to local too.
Or it is better to use -P option which is the same as --partial --progress. When it is used you’ll get a progress dialog at the command line that shows you which file is currently transferring, what percentage of that transfer is complete, and how many more files are left to check. As each file completes, you’ll see an ever-growing list of completed file transfers, which is great for making sure everything transfers successfully. It also allows you to easily resume suspended or interrupted transfers. Combined, you can see how it’ll show you which file was the last one to go, where it failed, and if it failed, give you the option to resume. It’s a pretty powerful combination.
rsync on Windows
Download and install command line rsync from http://www.rsync.net/resources/howto/windows_rsync.html. The website also provides a documentation. Some people are concern about the license issue. The website here provides a link to the free, old but usable version 4.0.5 which is newer than I tested v3.1.0.
Below are my note by using cwrsync v3.1.0 installer got from http://www.rsync.net.
cd C:\Program Files (x86)\cwRsync\bin ssh-keygen -t rsa -N '' rsync -av "/cygdrive/c/Users/brb/.ssh/id_rsa.pub" [email protected]:.ssh/authorized_keys rsync -av "/cygdrive/c/Users/brb/Downloads/cytokineMC.txt" [email protected]:Downloads/
sudo
How to Keep ‘sudo’ Password Timeout Session Longer in Linux
http://www.tecmint.com/set-sudo-password-timeout-session-longer-linux/
How to run multiple commands in sudo
https://www.cyberciti.biz/faq/how-to-run-multiple-commands-in-sudo-under-linux-or-unix/
How do I run specific sudo commands without a password?
https://askubuntu.com/questions/159007/how-do-i-run-specific-sudo-commands-without-a-password
Text browser
Links
- http://pcworld.com/article/3196428/linux/why-installing-a-text-mode-web-browser-is-a-good-idea.html
- http://links.twibright.com/user_en.html
- https://en.wikipedia.org/wiki/Links_%28web_browser%29
Alternative browsers
Chrome or Chromium
Install the latest version of chromium.
git clone https://github.com/scheib/chromium-latest-linux.git cd chromium-latest-linux ./update-and-run.sh # next time ./run.sh
chromium --proxy-server="10.130.5.180:3128"
Vivaldi
A sidebar provides a place to quickly access your bookmarks, downloads, and history. Less standard is the built-in ability to write and save notes, also available in the sidebar.
While cool, Vivaldi is also a proprietary browser.
The proxy setting is not found in the latest version. To use it,
$ vivaldi --proxy-server="127.0.0.1:8118"
To uninstall it, open the package manager to remove it.
Opera
Like Chrome, Opera is closed source.
GNOME Web
There are browsers made specifically for Linux, and GNOME Web is the most mature of the bunch.
It looks and feels like a program intended to run on Linux.
Web lacks the kind of extensions you see on Chrome and Firefox (though ad-block does come built-in).
Eolie
Eolie is another browser built specifically for GNOME.
The URL bar shows a site’s title rather than the web address.
Brave
https://en.wikipedia.org/wiki/Brave_(web_browser)
The browser uses a fork of Electron, called Muon, designed with a focus on browser features. For example, it has support for Chrome extensions, and a higher level of security.
Midori
Qupzilla
QupZilla is a new and very fast QtWebEngine browser. It aims to be a lightweight web browser available through all major platforms.
Filezilla
Keyboard shortcut. Especially, Alt+Down=Transfers the currently selected item to an item of the same name in the other pane.
The device is busy
brb@brb-P45T-A:~$ sudo umount /media/brb/TOSHIBA [sudo] password for brb: umount: /media/brb/TOSHIBA: device is busy. (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1)) brb@brb-P45T-A:~$ sudo umount /dev/sdc1 umount: /media/brb/TOSHIBA: device is busy. (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1)) brb@brb-P45T-A:~$ lsof /media/brb/TOSHIBA/ COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME VBoxSVC 5600 brb 18w REG 8,33 4294967295 3 /media/brb/TOSHIBA/Windows 10.ova (deleted) brb@brb-P45T-A:~$ kill -9 5600 brb@brb-P45T-A:~$ lsof /media/brb/TOSHIBA/ brb@brb-P45T-A:~$ sudo umount /dev/sdc1 brb@brb-P45T-A:~$
# fuser -m /dev/sdc1 /dev/sdc1: 538 # ps auxw|grep 538 donncha 538 0.4 2.7 219212 56792 ? SLl Feb11 11:25 rhythmbox
Another handy one is:
umount -l /dev/sdwhatever
Kill a process and the pstree command
# find the PID pgrep ProgramName # Kill the ProgramName process kill -9 PID
Another command is killall. For example, if Firefox is acting up (as Firefox will do from time to time) simply type killall firefox and it should kill the application completely.
In the rare circumstances that this doesn’t work you can always type xkill and then click on the window that won’t close; this will completely close a given window immediately. See this.
How To kill An Inactive OR Idle SSH Sessions. The pstree -p command can show a tree diagram of all the processes.
Create an ext3 file system on a USB flash drive
umount /dev/sdb1 (depending on the device of course) sudo mkfs.ext3 /dev/sdb1 sudo e2label /dev/sdb1 usbdrive (change the label)
We can create MS-DOS file system by
sudo mkfs.vfat /dev/sdb1
Add a new user with home directory
adduser xxx
adduser is better than useradd since useradd does not create home directory and it does not even ask the password for new user. adduser will interactively ask user information.
To delete the user and home directory, use
deluser --remove-home xxx
gzip with multi cores
Use pigz utility. It makes a lot of difference. For example for a 21GB file, gzip can't finish the job after 30 minutes. But pigz only took 7 minutes on a 12-core machine.
sudo apt-get install pigz pigz -9 FILENAME # compress & convert the file to FILENAME.gz tar cf - paths-to-archive | pigz -9 -p 12 > archive.tar.gz
There is no need to use pigz to un-compress the file. gunzip is fast enough and only takes 4 minutes to decompress.
The '-9' (best compression) option does not make difference (6.6G vs 6.5G).
Compress a folder without full path name
Suppose we want to compress the folder ~/Documents and its subfolders. We want to include Documents folder name but not /home/brb/Documents name.
# Method 1. Include 'Documents' as the top folder name cd ~/ tar -czvf tmp.tar.gz Documents # Method 2. Mind the last dot. Not include 'Documents' as the top folder. tar -czvf tmp.tar.gz -C /home/brb/Documents . # Double check the tarball tar -tzvf tmp.tar.gz
If we want to strip the upper directories when we uncompress a tar file, use --strip-components. For example, we can use --strip-components=1 to remove the Documents folder.
squashfs
squashfs is a heavy-compression based read-only filesystem that is capable of compressing 2 to 3 GB of data onto a 700MB. Linux liveCD are built using squashfs. These CDs make use of a read-only compressed filesystem which keeps the root filesystem on a compressed file. It can be loopback mounted and loads a complete Linux env. Thus when some file are required by processes, they are decompressed and loaded onto the RAM and used.
- https://en.wikipedia.org/wiki/SquashFS
- http://squashfs.sourceforge.net/
- http://elinux.org/Squash_FS_Howto
# create a squashfs file sudo mksquashfs /etc test.squashfs # mount the squashfs file mkdir /mnt/squash mount -o loop compressedfs.squashfs /mnt/squash # you can acess the contents at /mnt/squashfs # exclude files sudo mksquashfs /etc test.squashfs -e /etc/passwd /etc/shadow # or specify a list of exclude files given in a file cat excludelist # /etc/passwd sudo mksquashfs /etc test.squashfs -ef excludelist
List contents of tar.gz or tar.bz2
tar -tzvf myfile.tar.gz tar -tjvf myfile.tar.bz2 # replace z with j
Extract files
Extract tar.gz or zip to a specified directory
tar xzvf XXXX.tar.gz -C DIRECTORY # single or double quotes will give an error # # tar xzvf ~/Downloads/inSilicoDb_2.7.0.tar.gz -C "~/Downloads" # tar: ~/Downloads: Cannot open: No such file or directory # tar: Error is not recoverable: exiting now # $ tar xzvf ~/Downloads/inSilicoDb_2.7.0.tar.gz -C '~/Downloads' # tar: ~/Downloads: Cannot open: No such file or directory # tar: Error is not recoverable: exiting now unzip XXX.zip -d DIRECTORY
Extract gz file but keep the original gz file
gunzip -c x.txt.gz > x.txt
gunzip -c which simply writes the output stream to stdout
Extract .xz file
xz -d archive.xz
Extract tar.xz file
The bottomline is we don't need the 'z' parameter (used for gz ONLY but does not work for xz file) in the tar command for tar.xz files. And the method also works for tar.gz files. The argument '-f' means the archive file. Recall that the tar command can be used to store and extract files, so no default parameters.
tar xf archive.tar.xz tar xf archive.tar.gz
Extract tar.bz2 file
tar -xjvf archive.tar.bz2 # replace z with j as we compare it to tar.gz file
How To Extract and Decompress a .bz2/.tbz2 File
See this article from cyberciti.biz.
bzip2 -d your-filename-here.bz2 # OR bzip2 -d -v your-filename-here.bz2 # OR bzip2 -d -k your-filename-here.bz2 # OR bunzip2 filename.bz2
10 Basic Encryption Terms Everyone Should Know and Understand
https://www.makeuseof.com/tag/encryption-terms/
How to Encrypt and Decrypt Files and Directories Using Tar and OpenSSL
http://www.tecmint.com/encrypt-decrypt-files-tar-openssl-linux/
How to install and use 7zip file archiver
https://www.howtoforge.com/tutorial/how-to-install-and-use-7zip-file-archiver-on-ubuntu-linux/
Compare zip, tar.xz, tar.gz, 7z
The compression rate comparison is (from best to worst) 7z > tar.xz > tar.gz > zip.
For example, consider qt-everywhere-opensource-src-5.5.0 from http://download.qt.io/official_releases/qt/5.5/5.5.0/single/
- zip 540M
- tar.xz 305M
- tar.gz 436M
- 7z 297M
Extract one files from tar.gz
Extract a file called etc/default/sysstat from config.tar.gz tarball:
$ tar -zxvf config.tar.gz etc/default/sysstat
Noe that a new directory etc/default will be created under the current directory if it does not exist.
You can also extract those files that match a specific globbing pattern (wildcards). For example, to extract from cbz.tar all files that begin with pic, no matter their directory prefix, you could type:
$ tar -xf cbz.tar --wildcards --no-anchored 'pic*'
To extract all php files, enter:
$ tar -xf cbz.tar --wildcards --no-anchored '*.php'
remove leading directory components on extraction with tar
--strip-components option
AVFS and Archivemount
If we want to extract certain files from a tarballj/archive, it is more efficient to use a virtual filesystem like AVFS. PS. for a large archive file, even extracting only a single file at the top directory it is terribly slow if we use the tar command directly.
Before we install the utility, let's look at the package dependecies of AVFS and Archivemount.
$ apt-cache showpkg archivemount Package: archivemount Versions: 0.8.1-1 (/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages) Description Language: File: /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages MD5: d6302be9f06a91afa32326ab175e2086 Description Language: en File: /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_i18n_Translation-en MD5: d6302be9f06a91afa32326ab175e2086 Reverse Depends: archivemount:i386,archivemount Dependencies: 0.8.1-1 - libarchive13 (0 (null)) libc6 (2 2.4) libfuse2 (2 2.8.1) fuse (2 2.8.5-2) archivemount:i386 (0 (null)) Provides: 0.8.1-1 - Reverse Provides: brb@T3600 ~ $ apt-cache showpkg avfs Package: avfs Versions: 1.0.1-2 (/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages) (/var/lib/dpkg/status) Description Language: File: /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages MD5: bce08fbc36fd7b8e3c454f36f0daf699 Description Language: en File: /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_i18n_Translation-en MD5: bce08fbc36fd7b8e3c454f36f0daf699 Reverse Depends: avfs:i386,avfs worker,avfs Dependencies: 1.0.1-2 - libc6 (2 2.14) libfuse2 (2 2.8.1) fuse (0 (null)) unzip (0 (null)) zip (0 (null)) arj (0 (null)) lha (0 (null)) zoo (0 (null)) rpm (0 (null)) p7zip (16 (null)) p7zip-full (0 (null)) cdparanoia (0 (null)) wget (0 (null)) avfs:i386 (0 (null)) Provides: 1.0.1-2 - Reverse Provides:
Install it now.
sudo apt-get install avfs mountavfs # Assume MyFile.tar.gz exists in the current directory ls ~/.avfs/$PWD/MyFile.tar.gz# # Alternatively, browse the content in Nautilus, but you need to add a trailing # character by hand to the path # (Ctrl-L to access the address bar). ... cat ~/.avfs/$PWD/MyFile.tar.gz#/README # another tarball ls ~/.avfs/$PWD/MyFile2.tar.gz# umountavfs
- Filesystem in Userspace (FUSE)
- Develop your own filesystem with FUSE
- Google: ubuntu mount tar.gz file which gives many choices like archivemount.
- HOWTO: setup avfs / fuse on debian
For some reason, avfs sometimes does not work:( In this case, Ubuntu's Archive Manager does work. Maybe the file is too large.
brb@T3600 ~/Downloads $ time ls ~/.avfs/$PWD/Homo_sapiens_UCSC_hg19.tar.gz#/ ls: cannot access /home/brb/.avfs//home/brb/Downloads/Homo_sapiens_UCSC_hg19.tar.gz#/nown exact 1 SingleClassTriAllelic,InconsistentAlleles 2 1000GENOMES,SSMP, 2 A,T, 22.000000,2274.000: Input/output error ls: cannot access /home/brb/.avfs//home/brb/Downloads/Homo_sapiens_UCSC_hg19.tar.gz#/chr12 25482890 rs544684287 G A 0 . molType=genomic;class=single chr12 25482914 rs558575390 T G 0 . m: Input/output error 000,?0.999500,0.000500,??797?chr3?27877637?27877638?rs1478557?0?+?G?G?A 4?rs555100828?0?+?T?T?C 76?chr2?103777623?103777624?rs181283085?0?+?A?A?A chr12?25482890?rs544684287?G?A?0?.?molType=genomic;class=single?chr12?25482914?rs558575390?T?G?0?.?m G?A Homo_sapiens nown?exact?1?SingleClassTriAllelic,InconsistentAlleles?2?1000GENOMES,SSMP,?2?A,T,?22.000000,2274.000 README.txt T?C real 25m51.340s user 0m0.000s sys 0m0.003s brb@T3600 ~/Downloads $ ls ~/.avfs/$PWD/annovar.latest.tar.gz#/ annovar
For archivemount, see Cool User File Systems: ArchiveMount
archivemount files.tgz mntDir umount mntDir
Show folder size for one level only
du --max-depth=1 -h
The graphical tool is called Disk Usage Analyze which is already available on Ubuntu.
Soft link
ln -s /full/path/of/original/file /full/path/of/soft/link/file
Self-hosted servers
- https://github.com/Kickball/awesome-selfhosted This is a list of Free Software network services and web applications which can be hosted locally. Selfhosting is the process of locally hosting and managing applications instead of renting from SaaS providers.
- Sovereign: A set of Ansible playbooks to build and maintain your own private cloud: email, calendar, contacts, file sync, IRC bouncer, VPN, and more.
DNS server
setup
- https://support.rackspace.com/how-to/changing-dns-settings-on-linux/
- https://helix.nih.gov/user_guides/kerb5_config.html
DNSmasq (DNS + DHCP server)
- Man page and Setup
- Dnsmasq For Easy LAN Name Services
- https://wiki.archlinux.org/index.php/dnsmasq
- https://wiki.debian.org/HowTo/dnsmasq
- http://www.linuxjournal.com/content/dnsmasq-pint-sized-super-d%C3%A6mon
- https://blogging.dragon.org.uk/howto-setup-dnsmasq-as-dns-dhcp/
Local forwarding server
dnsmasq program is running on my Ubuntu and Linux/Mint machines.
See nameserver 127.0.1.1 in resolv.conf won't go away!
$ ps -ef | grep -i dnsmasq $ sudo netstat -anp | grep -i dnsmasq
Change DNS setting
- Why Changing DNS Settings Increases Your Internet Speed, 5 DNS Servers Guaranteed to Improve Your Online Safety
- https://1.1.1.1/, 隱私優先、速度最快,公共DNS服務1.1.1.1上線了
- Google: 8.8.8.8 and 8.8.4.4
- OpenDNS: 208.67.220.220 and 208.67.222.222
- DNS Watch: 84.200.69.80 and 84.200.70.40
- OpenNIC: 206.125.173.29 and 45.32.230.225
- UncensoredDNS: 91.239.100.100 and 89.233.43.71
- Change DNS Settings on Windows / Mac / Android / IOS / Linux
Query DNS server
To list the current DNS servers used by my system,
Method 1:
# Ubuntu >= 15 $ nmcli device show <interfacename> | grep IP4.DNS # Ubuntu <= 14 $ nmcli dev list iface <interfacename> | grep IP4
Method 2:
$ cat /etc/resolv.conf
Another way is to use the R packages: gdns and dnsflare. More Options For Querying DNS From R with 1.1.1.1.
3 Ways to Check DNS Propagation Status
https://www.makeuseof.com/tag/check-dns-propagation-status/
Email server
POP, IMAP and Exchange
- https://support.office.com/en-US/article/What-are-IMAP-and-POP-ca2c5799-49f9-4079-aefe-ddca85d5b1c9?ui=en-US&rs=en-US&ad=US&fromAR=1
- https://www.howtogeek.com/99423/email-whats-the-difference-in-pop3-imap-and-exchange/
POP works by contacting your email service and downloading all of your new messages from it. Once they are downloaded onto your PC or Mac, they are deleted from the email service.
IMAP allows you to access your email wherever you are, from any device. When you read an email message using IMAP, you aren't actually downloading or storing it on your computer; instead, you're reading it from the email service. As a result, you can check your email from different devices, anywhere in the world: your phone, a computer, a friend's computer.
Exchange offers the same syncing capabilities as IMAP, plus much more. Exchange is a Microsoft product, giving you the ability to still use Outlook as your email service and benefit from Exchange functionality.
Configure Postfix to use Gmail as a Mail Relay
https://www.howtoforge.com/tutorial/configure-postfix-to-use-gmail-as-a-mail-relay/
How to Build an Email Server on Ubuntu Linux
- https://www.linux.com/learn/how-build-email-server-ubuntu-linux, Part 2 & Part 3
- https://help.ubuntu.com/community/Postfix
- https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-postfix-as-a-send-only-smtp-server-on-ubuntu-14-04
- https://www.digitalocean.com/community/tutorials/how-to-install-and-setup-postfix-on-ubuntu-14-04
Webmail
Install WebMail Lite PHP based Webmail Client on Debian 9.1
sendemail package
- http://bhfsteve.blogspot.com/2013/03/monitoring-web-page-for-changes-using.html
- sudo apt-get install sendemail
- Usage
sendEmail -f $USERNAME -s smtp.gmail.com:587 \ -xu $USERNAME -xp $PASSWORD -t $USERNAME \ -o tls=yes \ -u "Web page changed" \ -m "Visit it at $URL" sendEmail -f [email protected] -t [email protected] \ -s smtp.gmail.com:587 -xu [email protected] -xp YOURPASSWORD \ -o tls=yes \ -u "Hello from sendEmail" \ -m "How are you? I'm testing sendEmail from the command line."
Kolab
Install and Configure Kolab Groupware on Ubuntu 16.04 LTS
Backup
- rdiff-backup. See
- Use a Dropbox folder.
If we don't want to install dropbox software, we can install bash dropbox uploader: http://www.andreafabrizi.it/?dropbox_uploader OR https://github.com/andreafabrizi/Dropbox-Uploader
It allows to upload/download/delete/list files and show info of user. The version I am using is v0.9.7. It works on linux, Windows/Cygwin, Raspberry Pi, etc.
I install it under ~/Downloads/andreafabrizi-Dropbox-Uploader-cdc2466 directory
Instruction with screenshots: http://www.jobnix.in/dropbox-command-line-interface-cli-client/
Sample usages:
./dropbox_uploader.sh list / ./dropbox_uploader.sh upload ~/Desktop/ConfigurateNote.txt
back up DVDs
$ df $ sudo dd if=/dev/sr0 of=filename.iso status=progress # Don't add the 'bs' parameter or you'll get an error reading '/dev/sr0': Input/output error
back up a remote drive using SSH and save the resulting archive to your local machine
# ssh [email protected] "dd if=/dev/sda | gzip -1 -" | dd of=backup.gz
where "-1" (one) indicates the fastest compression method and the dash means ‘standard input’.
Backups vs. Archives
How to Archive Your Data (Virtually) Forever
Timeshift
CloudBerry
CloudBerry Backup Protects Files on Windows, Mac, and Linux
Github, Bitbucket, Gitlab
We can use these git services to get real-time data (eg temperature, IP, etc).
Running a cron job as a user
Some examples
- http://www.thegeekstuff.com/2009/06/15-practical-crontab-examples/
- https://crontab.guru/examples.html and tips
- A specific time
MIN HOUR DOM MON DOW CMD 30 08 10 06 * /home/ramesh/full-backup # 30 – 30th Minute # 08 – 08 AM # 10 – 10th Day # 06 – 6th Month (June) # * – Every day of the week
- Twice a day
00 11,16 * * * /home/ramesh/bin/incremental-backup # 00 – 0th Minute (Top of the hour) # 11,16 – 11 AM and 4 PM # * – Every day # * – Every month # * – Every day of the week
- Every 10 minutes
*/10 * * * * /home/ramesh/check-disk-space
crontab
- crontab cron-file-winter; crontab -l
Make sure the .sh file gives a complete path. For example,
#!/bin/sh R --vanilla < arraytoolsip.R
does not work in cron job although it works perfect when we manually run it from the right path. The sh file should be
#!/bin/sh R --vanilla < $HOME/Dropbox/scripts/arraytoolsip.R
To disable everything on crontab -l, run crontab -e then comment out each line you don't want to run with #. OR run crontab -r to empty the current crontab.
PATH and Shell
Cron knows nothing about your shell; it is started by the system, so it has a minimal environment. If you want anything, you need to have that brought in yourself. For example, to use 'ifconfig' command, I need to give it a complete path in my script file.
$ cat syncIP /sbin/ifconfig eth0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}'
and the cron job
06 15 * * * /home/MYUSERNAME/Ubuntu\ One/syncIP > $HOME/Ubuntu\ One/ip.txt 2>&1
See here on how to add environment variable into cron environment.
Disable mail alert
If something went wrong with executing a cron job, cron will output a message "You have new mail in /var/mail/$USER". You can open this file using a text editor. To disable this alert, run 'crontab -e (see this post
0 1 5 10 * /path/to/script.sh >/dev/null 2>&1 # OR 0 1 5 10 * /path/to/script.sh > /dev/null
Running crontab as root
Use sudo crontab -e to edit. After saving it, no need to initialize it. Use sudo crontab -l to list the cron job.
md5sum
Linux md5sum Command Explained For Beginners (5 Examples)
How to verify files?
md5sum file1.txt file2.txt file3.txt > hashes md5sum --check hashes
Mount drive
/etc/fstab and blkid
- https://help.ubuntu.com/community/Fstab
- https://help.ubuntu.com/community/UsingUUID
- Mount /tmp securely
- http://www.thegeekstuff.com/2013/01/mount-umount-examples/
- Graphical method using Disks
- http://www.instructables.com/id/Using-a-USB-external-hard-drive-with-your-Raspberr/?ALLSTEPS Use UUID instead of /dev/sdXY to specify the partition in /etc/fstab to avoid any changes with /dev/sdXY. The UUID can be obtained using
sudo blkid
and the result should be compared with
sudo fdisk -l
- Run mount -a to remount /etc/fstab without reboot, except the partitions with noauto option.
The following example shows a problem (as found from the output of df command) with </etc/fstab> where we use /dev/sdXY instead of UUID for specifying hard disks.
$ sudo blkid /dev/sda1: LABEL="WD640" UUID="d3a0a512-bf96-4199-9674-f410f22f0a92" TYPE="ext4" /dev/sdb1: UUID="afaa4bde-1172-4c54-8b0a-a324ad855355" TYPE="ext4" /dev/sdb5: UUID="fb2a4ada-d80a-4e23-b4a2-67376b8b7e72" TYPE="swap" $ sudo fdisk -l Disk /dev/sda: 640.1 GB, 640135028736 bytes ... Device Boot Start End Blocks Id System /dev/sda1 2048 1250263039 625130496 83 Linux Disk /dev/sdb: 640.1 GB, 640135028736 bytes ... Device Boot Start End Blocks Id System /dev/sdb1 * 2048 1217761279 608879616 83 Linux /dev/sdb2 1217763326 1250263039 16249857 5 Extended /dev/sdb5 1217763328 1250263039 16249856 82 Linux swap / Solaris $ cat /etc/fstab proc /proc proc nodev,noexec,nosuid 0 0 UUID=afaa4bde-1172-4c54-8b0a-a324ad855355 / ext4 errors=remount-ro 0 1 UUID=fb2a4ada-d80a-4e23-b4a2-67376b8b7e72 none swap sw 0 0 /dev/sdb1 /mnt/WD640 ext4 rw,nosuid,nodev 0 2 $ df -h Filesystem Size Used Avail Use% Mounted on /dev/sdb1 572G 413G 130G 77% / ... /dev/sdb1 572G 413G 130G 77% /mnt/WD640
To fix the error here, modify the line starting /dev/sdb1 in /etc/fstab and replace it with the UUID. Then run sudo umount /mnt/WD640 and sudo mount -a. Done!
~$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sdb1 572G 413G 130G 77% / ... /dev/sda1 587G 283G 275G 51% /mnt/WD640
usb drive
Run the following to confirm the USB device is detected.
sudo fdisk -l # OR dmesg | grep -i "SCSI"
Now suppose the usb device is found in dev/sdb1.
sudo mkdir /mnt/usb sudo mount -t vfat -o rw,users /dev/sdb1 /mnt/usb
The above mount command assumes the usb drive has Windows vfat partition and users give non-root users the ability to unmount the drive. If the USB drive is partitioned linux ext2/3, we can merely run mount command as
sudo mount /dev/sdb1 /mnt/usb
At the end, run umount command like
sudo umount /mnt/usb
To make the mounting automatically, edit the file /etc/fstab.
/dev/sdb1 /mnt/usb vfat defaults 0 0 /dev/sdb1 /mnt/usb ext3 defaults 0 0
and run
sudo mount -a
Mount an iso file
sudo mkdir -p /mnt/mount_point # create a mount point sudo mount -o loop /home/user/disk.iso /mnt/mount_point mount # verify
Simple way of Sharing files between Ubuntu 16.04 and Windows 10 by using open-source NitroShare which is based on Qt framework.
exFat - cross platform partition format
- Mac
- Gparted cannot create exFAT partition (it is greyed out)
- This Trick Makes a USB Drive Work Perfectly With Windows, Mac, Linux, and Anything Else
- How to Mount and Use an exFAT Drive on Linux or How to get a drive formatted with exfat working?
sudo apt-get install exfat-utils exfat-fuse # Still need to create a partition (ex. FAT32) first using gparted in order to get it mounted sudo fdisk -l sudo mkfs.exfat -n LABEL /dev/sd** # LABEL with whatever you want to label your drive
This should delivery a working exfat file system (read and write support, but not formatting the drives with exfat via Gnome Disks and GParted).
NTFS usb drive in xubuntu
http://xflinux.blogspot.com/2011/01/mount-ntfs-volumes-automatically-in.html
sudo apt-get install ntfs-config
Now go to Applications>> System>> Ntfs Configuration Tool
Expand the "Advanced Configuration" and select all those partitions you want to be auto mounted and writable( The tool will detect all partitions at its startup).
Make sure the " Enable write support for internal devices" option is selected. Now click Close.
Many drives, one folder
- mhddfs program.
Partition tables
Partition Tables and the Dangers of Editing Them
parted command
How to partition a disk in Linux
Recommended partition schemes
How do I send an already-running process into the background
http://stackoverflow.com/questions/625409/how-do-i-put-an-already-running-process-under-nohup
- 'Ctrl+Z' to stop (pause) the program and get back to the shell.
- bg to run it in the background.
- disown -h [job-spec] where [job-spec] is the job number (like %1 for the first running job; find about your number with the jobs command) so that the job isn't killed when the terminal closes.
run commands in a background and allow log off
nohup /path/to/script >output 2>&1 &
Or to disable output and be more safe. It also explains the concept of file descriptor/fd in Unix.
nohup command </dev/null >/dev/null 2>&1 &
See also Anonymous named pipe.
Notepadqq - Notepad++-like editor
Notepadqq. It is written using Qt. It does not have printing function:(
Note apps that can sync
Top 8 Notepad Apps for Linux That You Can Sync. Some are compatible with Evernote.
Evernote
Evernote alternative
How to Install Turtl Server - Evernote Alternative - on Ubuntu 16.04
Backup/restore Evernote
https://www.makeuseof.com/tag/backup-restore-evernote/
Markdown
Preview markdown/view markdown offline
- Atom text editor has a built-in function to preview HTML or markdown files. Menu -> Packages -> Markdown Preview -> Toggle Preview.
- http://stackoverflow.com/questions/9843609/view-markdown-files-offline. Grip works fine.
sudo pip install grip grip readme.md
- For image, see http://stackoverflow.com/questions/13051428/how-to-display-images-in-markdown-files-of-github. The trick is adding ?raw=true after the image name.
# title 1 ![screenshot](myfile.png?raw=true)
- Chrome markdown preview plus extension does not show images from github.
Markdown editor
7 Best Note-Taking Tools for Programmers
http://www.makeuseof.com/tag/best-note-taking-tools-programmers/
RStudio
We can create different levels of sections.
On ODroid (ARM works too!) Ubuntu 16.04
# Note: OpenJDK 8 will not work # We have to install Oracle Java sudo add-apt-repository ppa:webupd8team/java sudo apt-get update sudo apt-get install oracle-java8-installer sudo apt-get install netbeans # version 8.1 in my case
See screenshots
Note:
- Netbeans has a built-in support for HTML/XML files. XML has an advantage over HTML since HTML cannot have any tags you want.
- We need to download a plugin for markdown file support. Go to Tools -> Plugins. In the 'Settings' tab make sure the 3 items are checked. Go to 'Available plugins' tab
search 'markdown'. Install 'Markdown support'. It works on Netbeans 8.0 on x64 Ubuntu 14.04 and Netbeans 8.1 on my ARM Ubuntu 16.04.
- For Markdown or XML, the comment syntax can be found here.
- For some reason, the order of headlines on the navigator pane is not the same as they appeared on the file. So it is better to use XML file format.
- My hack on Netbeans options (change to use a dark color on background).
- Profile: NetBeans
- Syntax. Default: Foreground=White, Background=Dark Gray. Comment: Foreground=cyan. Keyword: Foreground=Orange.
- Highlighting. Highlight Caret Row: Foreground=Dark Gray. Background=Pink.
- For choosing colors, go to Google: rgb to hex
- For some reason, it makes my graphical Mint desktop unstable. I have to use Ctrl + Alt + F1 and Ctrl + Alt + F8 to fix it temporarily. Deal breaker!
IntelliJ IDEA
It requires JDK. The community version is free. Download the tarball. Extract it and run bin/idea.sh. It even identifies a mismatch in my XML documentation that netbeans does not find.
- Viewing Structure of a Source File or Alt + 7
- Open a terminal at the bottom; Alt + F12
- SOLARIZED color. Copy icls file to ~/.IdeaIC2016.2/config/colors directory. Restart Intellij IDEA. Go to File -> Settings -> Editor -> Colors & Fonts -> Font.
- To deactivate spelling checking: Ctrl + Alt + s -> Editor -> Inspections -> Spelling -> Typo. Uncheck it.
- We need to create a project for IntelliJ IDEA to automatically open the file we are working on. IDEA will create a hidden folder call .idea. For git repository, we shall create the .gitignore file contains
.idea/workspace.xml .idea/misc.xml
Zim
- Available in Linux, Windows & Mac.
- The side panel contains a hierarchical view of the pages.
- Right click on the page tab, we can create a new page or sub page.
- On Ubuntu, the title bar is on the top of the desktop.
- Auto save. Auto re-open the last session.
- Handles several types of markup, like headings, bullet lists and of course bold, italic and highlighted. This markup is saved as wiki text so you can easily edit it with other editors.
- Toggle notebook editable.
- Insert image (cannot resize)
- Plugins, e.g. Equation editor, R plot,
- The default folder for storing the notes is ~/Notebook. Each page has its own file in storage.
~/Notebooks/Notes/ ~/Notebooks/Notes/notebook.zim ~/Notebooks/Notes/Home.txt
If we create a 2nd page called 'Home2' with sub pages 'Subpage1' and 'subpage2' we will have
~/Notebooks/Notes/Home2.txt ~/Notebooks/Notes/Home2 ~/Notebooks/Notes/Home2/subpage1.txt ~/Notebooks/Notes/Home2/subpage2.txt
Cherrytree - a hierarchical note taking application
featuring rich text and syntax highlighting, storing data in a single xml or sqlite file.
Vim
with the Tagbar plugin.
The instruction works for cpp file.
Unfortunately xml files are not supported from my testing. See its wiki for supported filetypes.
Lime
Maybe
nano editor
Keyboard shortcuts
Actually there is no need to memorize them because the common shortcuts are always displayed at the bottom of the screen (ctrl+g to get more).
- Ctrl+c: cur pos
- Ctrl+y: prev screen
- Ctrl+v: next screen
- Ctrl+k: cut
- Ctrl+u: paste
- Ctrl+w: search
- Alt+w: search next (macOS does not work)
- Ctrl+r: insert another file at cur
- Alt+r: search and replace
Permission denied and sudoedit command
When I run 'nano tmp', I got a message: Error reading /home/odroid/.nano/search_history: Permission denied. Press Enter to continue.
odroid@odroid:~$ ls -ld /home/odroid/.nano drwxr-xr-x 2 root root 4096 Feb 12 08:01 /home/odroid/.nano odroid@odroid:~$ ls -l /home/odroid/.nano total 4 -rw------- 1 root root 15 Feb 12 08:01 search_history
A simple solution is
sudo chown -R odroid:odroid /home/odroid/.nano # note '-R' has to be capital
This seems to be a bug in nano after we use 'sudo nano [file]' (eg 'sudo nano /etc/chromium-browser/default') when the 'nano' program has not been run before.
- superuser.com. A better habit to get into is to use sudoedit or sudo -e instead of 'sudo nano'.
- Why use 'sudoedit' or 'sudo -e' - security reason
- http://superuser.com/questions/785187/sudoedit-why-use-it-over-sudo-vi
- gksudo is also useful if we want to run a GUI program under root. For example gedit or gparted.
- visudo is used to edit /etc/sudoers file only
The bottom line is use something like below for editing system files
EDITOR=nano sudoedit /etc/chromium-browser/default
show line number/cursor position
Use the -c option for cursor position.
nano -c FILENAME
Enable soft line wrapping
nano -$ FILENAME nano --softwrap FILENAME
Not that these options are not available on the NIH/Biowulf. I need to use emacs/vi/joe. The keyboard shortcuts for joe can be found here.
syntax highlight
Add a syntax highlight support for some languages
$ ls /usr/share/nano/ asm.nanorc html.nanorc mutt.nanorc perl.nanorc ruby.nanorc c.nanorc java.nanorc nanorc.nanorc pov.nanorc sh.nanorc groff.nanorc man.nanorc patch.nanorc python.nanorc tex.nanorc $ cat ~/.nanorc include /usr/share/nano/sh.nanorc include /usr/share/nano/c.nanorc include ~/r.nanorc
R syntax highlight file r.nanorc. Note that I have to comment out line 29 starting with 'header'. A personal copy is saved in github.
To disable syntax highlight (useful if we use a terminal app on an Android ebook reader such as Boox), add -Ynone parameter.
vi editor
Keys
- Page down: ctrl +f. Page up: Ctrl +b
- ^: beginning of a line. $: end of a line.
- command mode : this is the default when you run vi. Hit Esc key to return to the command mode. Command mode is right for moving about a file, copying or deleting a line, saving a file, etc.
- Insert/edit mode : hit "i" (insert text before cursor position) or "a" (add text after cursor position) to enter the edit mode. The screen will show the text -- INSERT -- on the last line of the vi editor.
- Command mode/leave edit mode: "ESC". In this mode, you can search, navigate or enter an insert model.
- Last line mode: Press ':', vi editor will show ':' on the last line. If we continue to type 'q[Enter]' we will quit vi. If we continue to type 'wq', it will write the file and quits.
- Moving around
- line beginning: "0"
- line end: "$"
- last row: "G"
- delete entire line: "dd"
- undo: "u"
- search forward: "/pattern" (case sensitive). Hit "n" to repeat search.
- Highlight search ":set hlsearch". To disable highlight ":set nohlsearch"
- search backward: "?pattern" (case sensitive). Hit "n" to repeat search.
- save: ":w"
- quit: ":q"
- quit without saving: ":q!"
- save and quit: ":x" or ":wq" (note that ":qw" won't work; you want to write and then quit!)
- Run external command ":! command"
- Display line numbers ":set nu". Add "set number" to your .vimrc file in your home directory.
- Ignore cases when searching ":set ic"
Some helps
- http://mathlab.cit.cornell.edu/local_help/vi.html
- http://www.lagmonster.org/docs/vi.html and more complete one.
- Difference between vi and vim. Especially Vim allows the screen to be split for editing multiple files. Use ":split" to split a screen using the same file, ":split filename" to split the screen using a new file and "Ctrl-w + Ctrl-w" to switch screens/viewports. More keyboard controls can be found at linux.com.
- Advanced concepts
color schemes
On my Mint 18.2, the color syntax is off. It does not work if I try to enable it.
The solution is install vim (sudo apt-get install vim). After that, the syntax highlight works automatically; no need to turn it on manually.
To change the color scheme on-the-fly, type :colorscheme then Space followed by TAB. The 'darkblue' looks cool.
The list can be found at /usr/share/vim/vimNN/colors.
On Raspbian OS , we should use the method described here. That is, sudo nano /etc/vim/vimrc and uncomment out the line containing syntax on.
plugin
On Mac, I need to run mkdir ~/.vim/plugin'. Then I can put the downloaded .vim file (e.g. R syntax highlight) there.
Also I may need to modify ~/.vimrc file by adding some options,
syntax on filetype plugin on
Cloud
http://slidedecks.wilmoore.com/2012-confoo/diy-private-cloud-using-virtualBox-and-chef/#66
http://www.datacentermap.com/blog/cloud-software-389.html
- OpenStack:Open source software for building private and public clouds. Great for large infrastructures. Cf: Amazon Elastic Compute Cloud (EC2).
- CloudStack
- Eucalyptus API compatible with Amazon EC2
- ownCloud. Cf: Dropbox. Install owncloud on Debian 8, install owncloud client on Ubuntu 14.04.
Commercial cloud services
- DigitalOcean - simple pricing. One droplet = one server.
- Amazon EC2
- Google cloud
- Microsoft Azure
Manage all your cloud storage
http://www.makeuseof.com/tag/chrome-extensions-you-need-manage-cloud-storage/
Comparison of Linux VPS providers
https://www.ghacks.net/2017/06/15/a-comparison-of-linux-vps-providers-for-beginner-hosting/
Boot
U-boot
http://www.denx.de/wiki/U-Boot
Pandora linux client
COW (copy on write) file system
The cow filesystem was found on xubuntu live CD. See http://en.wikipedia.org/wiki/Copy-on-write
How To Protect Your Server Against the Dirty COW Linux Vulnerability (10/21/2016)
Apache redirection
http://cran.r-project.org/mirror-howto.html
Redirect a Website URL from One Server to Different Server in Apache
Important linux directories
- /bin - executables used by the base system
- /boot
- /dev
- /etc - configuration files
- /media
- /mnt
- /opt - optional application packages
- /proc - process information only
- /sbin - critical executables for running the system, but should be used by superuser
- /usr - non-critical files. Inside is /usr/bin, which contains most of the libraries used by apps.
- /var - variable data such as databases, mails spools and system logs.
Difference of /bin, /sbin, /usr/local/bin, ...
- /bin : For essential binaries; e.g. bash, cat, ls.
- /sbin : is similar to /bin but for scripts with superuser (root) privileges required; e.g. shutdown command is located here. Local users have to use sudo to run binaries here.
- /usr/bin : Same as first, but for general system-wide & non-essential binaries; e.g. grep, zip, docker, etc.
- /usr/sbin : Same as above, but for scripts with superuser (root) privileges required.
- /usr/local/bin or /usr/local/sbin for system-wide available (personal) scripts.
If you want to create your own scripts and make them available to all users, you’re pretty safe adding them to /usr/local/bin. Or to add my scripts to my local bin (~/bin) and then I create a symbolic link in /usr/local/bin to the commands I want to make public. As a result, I can manage all my scripts from the same directory but still make some of them publicly available since /usr/local/bin is added to $PATH. See this post.
DHCP lease time
On Windows, it is 8 days.
Get a New IP Address
dhclient -r # release your IP Address dhclient # get your DHCP to issue you a new IP Address based on how it’s been configured.
Open a file/URL using the default application from the command line
- gnome-open (works on Mint)
- kde-open (KDE users)
- xdg-open (window-manager independent). XDG stands for X Desktop Group; see https://en.wikipedia.org/wiki/Freedesktop.org
See also
- How does Linux choose which application to open a file?
- How to Change Your Default Applications on Ubuntu: 4 Ways
Check a file's encoding
file -bi myfile
For example,
file -bi Downloads/hmv_.rc # text/x-c++; charset=utf-16le
Know you system using the command line
System monitor tools (TUI)
conky
- Ubuntu > Conky (internal link)
- Raspberry Pi case (internal link)
htop command
glances command
- https://nicolargo.github.io/glances/ and its Documentation
- Glances includes disk i/o, network too. Its official website at http://glances.readthedocs.io/en/latest/.
- https://www.tecmint.com/glances-an-advanced-real-time-system-monitoring-tool-for-linux/
Glances is similar to htop but it provides network stats too. Install it by sudo apt-get install glances.
scout_realtimep
This is used by Dataplicity
gtop command
https://www.cyberciti.biz/howto/gtop-awesome-system-monitoring-dashboard-for-terminal/.
Pros:
- CPU history graph in time
- Memory history graph in time (not useful)
- Network bandwidth usage is real-time. It is accurate as what nload gives.
- Percentage usage of memory, swap, disk usage
- Top processes
$ sudo apt install npm nodejs $ npm install gtop -g $ gtop
- Press p to sort by process ID (PID).
- Press c to sort by CPU usage.
- Press m to sort by memory usage.
It can be installed on Linux Mint 18.2 but not in Ubuntu 14.04 or raspbian (9 stretch).
$ npm install gtop -g npm http GET https://registry.npmjs.org/gtop npm http GET https://registry.npmjs.org/gtop npm http GET https://registry.npmjs.org/gtop npm ERR! Error: CERT_UNTRUSTED npm ERR! at SecurePair.<anonymous> (tls.js:1370:32) npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17) npm ERR! at SecurePair.maybeInitFinished (tls.js:982:10) npm ERR! at CleartextStream.read [as _read] (tls.js:469:13) npm ERR! at CleartextStream.Readable.read (_stream_readable.js:320:10) npm ERR! at EncryptedStream.write [as _write] (tls.js:366:25) npm ERR! at doWrite (_stream_writable.js:223:10) npm ERR! at writeOrBuffer (_stream_writable.js:213:5) npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:180:11) npm ERR! at write (_stream_readable.js:583:24) npm ERR! If you need help, you may report this log at: npm ERR! <http://github.com/isaacs/npm/issues> npm ERR! or email it to: npm ERR! <[email protected]> npm ERR! System Linux 4.4.0-119-generic npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "gtop" "-g" npm ERR! node -v v0.10.25 npm ERR! npm -v 1.3.10
gotop
A terminal based graphical activity monitor inspired by gtop and vtop. It is quite beautiful.
Gotop – Yet Another TUI Graphical Activity Monitor, Written In Go
Compared to gtop, it has a temperature monitor. However, it can only show the average CPU usage (one line) on my Xeon computer.
git clone --depth 1 https://github.com/cjbassi/gotop /tmp/gotop /tmp/gotop/scripts/download.sh sudo cp gotop /usr/local/bin; rm gotop gotop
Note the temperatures do not show up in Raspbian (raspberry pi 3 b+).
S-tui command
Monitor Linux CPU temperature, frequency, power in a graphical way
System monitor tools (GUI)
Comparisons:
- https://www.tecmint.com/linux-performance-monitoring-tools/
- https://linoxide.com/monitoring-2/linux-performance-monitoring-tools/
- http://www.linuxscrew.com/2012/03/22/linux-monitoring-tools/
- https://www.infoworld.com/article/2683857/network-monitoring/article.html#slide2
- http://www.thegeekstuff.com/2011/12/linux-performance-monitoring-tools
Some lists:
- Nagios, Install Nagios core 4.1.1 on Ubuntu 16.04 (Xenial Xerus) Server
- Icinga (Nagios fork)
- Server Monitoring with Munin and Monit on Ubuntu 16.04 LTS
- Cacti
- Install Ganglia on Ubuntu 16.04 Server (Xenial Xerus)
- Linux Dash Web based monitoring tool. Source code is on github.
- Monitorix and on Ubuntu16.04
- sysstat & the sar command
- https://www.maketecheasier.com/monitor-linux-performance-with-sysstat/
- https://www.tecmint.com/install-sysstat-in-linux/
- http://www.thegeekstuff.com/2011/03/sar-examples/
- https://www.blackmoreops.com/2014/06/18/sysstat-sar-examples-usage/
- Visualize sar data with kSar.
- export LC_ALL=C. This will convert date/time. For example, 12:00:01 AM will become 00:00:01 and 12/09/2017 will become 12/09/17.
- sar -A -f /var/log/sysstat/saXX > ~/Downloads/sardata.txt.
- Click on Data -> Load from text file. Select ~/Downloads/sardata.txt file. Note that nothing will happen in the kSar GUI.
- Click 'kSar' to show the tree.
- kSar (depends on JDK) for graphics (instead of usinsg the sadf command).
- Download and unzip it to ~/bin.
- Execute bash ~/bin/kSar-5.0.6/run.sh.
- On the GUI, click Data -> Run local command.. -> sar 2 10, for example. This will start to record the cpu usage 10 times with a 2 seconds interval.
- You can view the real-time plot (shown on the right panel) by clicking kSar -> CPU -> CPU all (left panel).
# CPU sar 2 10 # every two seconds, 10 times # Memory sar -r # look at the kbcommit and commit columns sar -r -f /var/log/sysstat/sa02
Git and Github
Check out the Github page.
Bitbucket (free for 5 users)
- Unlimited private repos
- Code reviews
- JIRA integration
- REST API
- Custom domains
See this post to know how to fix the problem of unknown author. In short, when I uncheck "Use global user setting" from Repository-> Repository Settings -> Advanced does the commit author change as expected.
See here for a list of Android apps related to bitbucket.
Image
English original, ImageMagick 入门:使用命令行来编辑图片
Convert a color image to black and white
- https://linux.cn/article-8851-1.html
- https://www.imagemagick.org/script/command-line-options.php#canny
$ convert filename.jpg -canny 0x1 -negate canny.jpg convert.im6: unrecognized option `-canny' @ error/convert.c/ConvertImageCommand/1107. $ convert --version Version: ImageMagick 6.7.7-10 2017-07-31 Q16 http://www.imagemagick.org Copyright: Copyright (C) 1999-2012 ImageMagick Studio LLC Features: OpenMP
The current imagemagick version is 7.0.7-15.
Install/build the latest imagemagick
- https://www.imagemagick.org/script/install-source.php
- https://gist.github.com/makenova/78bb63aaa1050e2ad8019ee1e7e7b433
- https://www.tutorialspoint.com/articles/how-to-install-imagemagick-on-ubuntu
# remove version installed with apt-get sudo apt-get remove imagemagick && sudo apt-get autoremove # install dependencies sudo apt-get install build-essential sudo apt-get build-dep imagemagick -y # download ImageMagick source wget http://www.imagemagick.org/download/ImageMagick.tar.gz tar xzvf ImageMagick.tar.gz # build source cd ImageMagick-* ./configure make # install and verify sudo make install sudo checkinstall ********************************************************************** Done. The new package has been installed and saved to /home/XXX/Downloads/ImageMagick-7.0.7-15/imagemagick-7.0.7_15-1_amd64.deb You can remove it from your system anytime using: dpkg -r imagemagick-7.0.7 ********************************************************************** $ convert --version # bash: /usr/bin/convert: No such file or directory $ which convert /usr/local/bin/convert $ whereis convert convert: /usr/local/bin/convert $ /usr/local/bin/convert -version Version: ImageMagick 7.0.7-15 Q16 x86_64 2017-12-20 http://www.imagemagick.org Copyright: © 1999-2018 ImageMagick Studio LLC License: http://www.imagemagick.org/script/license.php Features: Cipher DPC HDRI OpenMP Delegates (built-in): bzlib fontconfig freetype jbig jng jpeg lzma pangocairo png tiff x xml zlib
Convert an image to sketch (online tool)
http://www.snapstouch.com/sketch.aspx
Convert an image file to a different format (eg icon)
Using the imagemagic program.
convert winamp-ncrow.png -resize 32x32 winamp-ncrow.ico
4 Ways to Batch Convert Your PNG to JPG and Vice-Versa
Rotate an image
http://www.imagemagick.org/script/command-line-options.php#rotate
convert winamp-ncrow.png -rotate 45 winamp-ncrow2.png # 45 degrees
One problem with this simple approach is the picture size (not image file) changed (become smaller) if the degree is not one of 90,180 or 270.
Create an animated gif file
Use the script here. See the last example on here. The rotation speed looks good too! Just change the source image file in the script.
#!/bin/sh # # Create a rotating figure using Distort SRT transformations # command='convert -delay 10 koala.gif -virtual-pixel white' for i in `seq 5 5 360`; do command="$command \\( -clone 0 -distort SRT $i \\)" done command="$command -delete 0 -loop 0 animate_distort_rot.gif" eval $command chmod 644 animate_distort_rot.gif
Tool to convert a sequence of numbered PNG files to an animated GIF?. Convert command line option. The option '-loop 0' means repeats infinitely and '-delay 200' means 2 seconds delay between each frame.
convert -delay 200 -loop 0 file_1.png file_2.png file_3.png animated.gif
Edit gif file
gifsicle package
Replace transparency in PNG images with white background
http://stackoverflow.com/questions/2322750/replace-transparency-in-png-images-with-white-background
convert image.png -background white -alpha remove white.png
Remove GPS metadata from jpg files - exiftool
sudo apt-get install libimage-exiftool-perl exiftool -gps:all= -xmp:geotag= image.jpg
The image file will be updated. To check the current metadata, use
exiftool image.jpg
Note that the above command only remove gps information. The other information like date/time of creation, camera model are not changed.
Exitftool can also be used to edit the metadata on PDF files.
exiftool -Title="This is the Title" -Author="Happy Man" -Subject="PDF Metadata" drawing.pdf
Edit Svg image
- Inkscape
- Inkscape from Fedora magazine
Animated gif
- How to generate a animated GIF or movie out of images on Linux (GIMP or PhotoFilmStrip)
- convert (.gif to .png) this image to get the original view? (ImageMagick)
Display images in the terminal
GIMP
login shell (.bash_profile) vs interactive shell (.bashrc)
- login shell - non desktop environment. ~/.bash_profile is sourced for the bash shell.
- interactive shell - Ctrl+Alt+t to open a terminal from a graphical mode (desktop environment). ~/.bashrc is source. We usually edit ~/.bashrc to set up the environment to include fancy prompt, set aliases, set history options, or define custom shell functions.
export environment variables
- Both a login shell and an interactive one. SSH (Putty) to connect to a remote machine.
- When a shell runs a script or a command passed on its command line, it's a non-interactive, non-login shell.
History of commands
history command with date and time
Running the following code once and history will give date and time the next time you issue the history command.
echo 'export HISTTIMEFORMAT="%d/%m/%y %T "' >> ~/.bashrc
Note that the original post asks to write the line to ~/.bash_profile but this is not working in the desktop environment.
Bang bang - Run a command/Fetch parameters from previous history
- http://unixhelp.ed.ac.uk/shell/tcsh_hist3.html
- http://codytaylor.org/2009/09/linux-bang-commands.html
- http://craig-russell.co.uk/2011/09/28/bang-bang-command-recall-in-linux.html#.VHXnq3Wx3UY
- http://requiremind.com/linux-command-line-tips-become-a-master/
- ^P: Move up through the command history list one command at a time.
- ^N: Move down through the command history list one command at a time.
- !!: Run the previous command.
- !n: Run command number n
- !string: Run most recent command starting with characters in string
- !?string: Run most recent command containing characters that match string
- !!*: Fetch parameters from last command
For example,
!-1 !4 !tail
Increase history limit
http://unix.stackexchange.com/questions/17574/is-there-a-maximum-size-to-the-bash-history-file
Not to add to bash history
Add a space after the command.
How to Clear Bash History on Linux
$ cat /dev/null > ~/.bash_history && history -c && exit
Listen to HiChannel internet radio
Use Radio Tray
- http://endroid.blogspot.com/2012/02/listen-hichannel-radio-online-by.html
- http://abcde9990101.blogspot.com/2011/05/ubunturadio-tray.html
I use it to listen m3u file (VLC also supports it too).
Web Analytics Reporting Tools
Painting software
- Krita - professional painting program made by artists that want to see affordable art tools for everyone. Krita Is the Free GIMP Alternative You Should Be Using.
- Pinta. It can be install by apt-get command. It works just line Window's paint. Ctr + v to paste an image and save to a file. To crop an image, click the selection tool (1st one) on the left hand side, then select a rectangle. Now click 'Image' > 'Crop to Selection' to finish.
- mtPaint. It is included in Odroid - xu4 - Lubuntu 14.04. To crop an image, just select an area and click Image > Crop. It can be used to take a screenshot from the desktop by using either the application or through the command line (mtpaint -s). It will then display the screenshot in the application if you use the command line.
- MyPaint
Take a screenshot (and edit them)
See Take screenshots.
Calibre - Read ebook in epub format
$ sudo -v && wget -nv -O- https://github.com/kovidgoyal/calibre/raw/master/setup/linux-installer.py | \ sudo python -c "import sys; main=lambda:sys.stderr.write('Download failed\n'); exec(sys.stdin.read()); main()" 2014-03-19 15:54:28 URL:https://raw.github.com/kovidgoyal/calibre/master/setup/linux-installer.py [25423/25423] -> "-" [1] Installing to /opt/calibre Downloading tarball signature securely... Will download and install calibre-1.28.0-x86_64.tar.bz2 Downloading calibre-1.28.0-x86_64.tar.bz2 100% [===============================================================================================] Downloaded 63255897 bytes Checking downloaded file integrity... Extracting files to /opt/calibre ... Extracting application files... Creating symlinks... Symlinking /opt/calibre/fetch-ebook-metadata to /usr/bin/fetch-ebook-metadata Symlinking /opt/calibre/lrf2lrs to /usr/bin/lrf2lrs Symlinking /opt/calibre/ebook-convert to /usr/bin/ebook-convert Symlinking /opt/calibre/ebook-meta to /usr/bin/ebook-meta Symlinking /opt/calibre/ebook-edit to /usr/bin/ebook-edit Symlinking /opt/calibre/lrfviewer to /usr/bin/lrfviewer Symlinking /opt/calibre/calibre to /usr/bin/calibre Symlinking /opt/calibre/markdown-calibre to /usr/bin/markdown-calibre Symlinking /opt/calibre/calibre-debug to /usr/bin/calibre-debug Symlinking /opt/calibre/calibre-parallel to /usr/bin/calibre-parallel Symlinking /opt/calibre/web2disk to /usr/bin/web2disk Symlinking /opt/calibre/calibre-server to /usr/bin/calibre-server Symlinking /opt/calibre/calibre-customize to /usr/bin/calibre-customize Symlinking /opt/calibre/ebook-polish to /usr/bin/ebook-polish Symlinking /opt/calibre/ebook-viewer to /usr/bin/ebook-viewer Symlinking /opt/calibre/calibre-smtp to /usr/bin/calibre-smtp Symlinking /opt/calibre/lrs2lrf to /usr/bin/lrs2lrf Symlinking /opt/calibre/ebook-device to /usr/bin/ebook-device Symlinking /opt/calibre/calibredb to /usr/bin/calibredb Setting up command-line completion... Installing bash completion to /etc/bash_completion.d/calibre Setting up desktop integration... Creating un-installer: /usr/bin/calibre-uninstall Run "calibre" to start calibre
By default, your books are saved in $HOME/Calibre Library folder.
Create ebooks
https://itsfoss.com/create-ebook-calibre-linux
Fetch News and recipes
- How to Get Free Magazines on Your Kindle with Calibre 2/19/2018.
- The news can be scheduled to download once per day (e.g. 6AM).
- The advantage is you can read articles offline once the news has been automatically sent to Kindle. Also you can search the definition or the translation of any word in articles. The disadvantage is you can't download the latest minutes news after the scheduled time.
- Because of this feature Calibre can replace the Reabble service. The free plan can only download 7 news per day.
- http://manual.calibre-ebook.com/news.html
- https://michaeltalbotuk.wordpress.com/2013/03/23/how-to-use-calibre-to-read-newspapers-magazines-for-free/
- To customize a builtin recipe: Add custom news source -> Customize builtin recipe.
Some RSS feeds
- Top Chinese News RSS Feeds
- https://www.makeuseof.com/tag/top-10-rss-feeds-medical-news-alerts/
- 自由電子報 (38MB so better selectively eg 國際 副刊) , Readers Digest (38MB), USA Today (6MB), Associated Press (8MB), National Geography (11MB), Newsweek (5MB, include a cover), Smithsonian Magazine (16MB) , TIME Magazine (username/password is required), xkcd (2MB), BBC News (4MB), Popular Science (1.6MB), The Washington Post (8MB), ScienceDaily (1.2MB)
Builtin feeds that do not work
- US and World Report news
- Wired Magazine (monthly ed)
- Discover Magazine
The recipes files (*.recipe) and the <index.json> file are saved in $HOME/.config/calibre/custom_recipes. They can be loaded through 'Add custom news source' dialog.
Tips
- Calibre will show the file size for each title/recipe. If the title is too large, send to kindle may not work (>40MB not work but 27MB works)
- Each recipe has two parameters: oldest article (how many days) and max # of articles per feed (default is 100). Since all recipes are scheduled to download every day, I can set 1 day for the oldest article.
A commercial service Keendly does it for a fee.
Remove a schedule download
Check the checkbox 'Schedule for download' at the top of source.
Get books
How to turn a Raspberry Pi into an eBook server
Remove DRM from Ebooks
https://www.makeuseof.com/tag/network-boot-raspberry-pi-without-microsd/
Create RSS feeds
- How to Find or Create an RSS Feed for Any Website. The trick to create an rss feed does not work by using the fivefilters service on https://kknews.cc/health/ (attribute name: loop-title)
- http://fetchrss.com/ works on https://kknews.cc/health/. Be skillful when using the mouse to select a new item, headline and summary. Free plan only got 5 RSS feeds and 5 news per feed.
- https://www.wikihow.com/Create-an-RSS-Feed
- https://www.makeuseof.com/tag/how-to-create-an-rss-feed-for-your-site-from-scratch/
RSS reader
Some references:
- 5 Best Feed Reader Apps for Linux
- 14 Best RSS Feed Readers for Linux in 2018. It contains nice screenshots.
Some examples:
- Akregator. KDE based. This is preinstalled in CentOS-KDE under the Internet category. It is also called 'Feed Reader'.
- QuiteRSS. It works on Linux, Windows and MacOS.
- Liferea. GTK based. It is considered one of the best RSS feed readers on Ubuntu Linux. It can synchronize with several online feed managers such as InoReader among others.
- FeedReader. Looks nice. Works with several online feed managers.
- Newsbeuter: RSS feed in terminal
- Newsboat: terminal
- RSSOwl. Depends on Java. Cross platform.
- Firefox and Thunderbird have built-in support for RSS.
Display/screen
Turn off/on your monitor via command line
- https://systembash.com/how-to-turn-off-your-monitor-via-command-line-in-ubuntu/
- http://askubuntu.com/questions/62858/turn-off-monitor-using-command-line
xset dpms force off # Press any key to turn it on xset dpms force on xset -q # check the status of the X server settings
If we want to turn off/on the screen via ssh, add
export DISPLAY=:0.0
first before calling the xset command, or use '-display' argument
xset -display :0.0 dpms force off xset -display :0.0 dpms force on
autoxrandr
Plug your laptop into different monitor setups. https://www.donarmstrong.com/posts/autorandr/
Add new screen/display resolutions
- http://ubuntuforums.org/showthread.php?t=1112186 (tested on UDOObuntu 2 beta 2running Ubuntu 14.04 + SainSmart 9" LCD display. The commands (not persistent) are
xrandr | grep maximum gtf 800 480 59.9 # give some output used in the following line xrandr --newmode "800x480_59.90" 29.53 800 816 896 992 480 481 484 497 -HSync +Vsync xrandr --addmode "DISP3 BG" 800x480_59.90 xrandr --output "DISP3 BG" --mode 800x480_59.90
I cannot find the file /etc/X11/xorg.conf in my UDOObuntu 2 beta 2. It seems this file does not exist anymore. See this post about how to re-create it.
Wayland
- Linux on the GPD Pocket 2 (Ubuntu, Debian, and Fedora)
- “xrandr -o right” command can be used to rotate the screen
- But it does not work in Fedora because Fedora uses the Wayland display server rather than xserver.
export DISPLAY
If we want to run a GUI app on a remote computer (such as Raspberry Pi/Beaglebone Black) and show the GUI app on the remote computer's screen using ssh, we can issue the following command before running the app.
export DISPLAY=:0.0
LVM Demystified
http://www.linuxjournal.com/content/lvm-demystified
See which groups you belong to, id & group commands
groups groups <username>
groupadd, chgrp, usermod, ACL (access control lists)
- Create a Shared Directory for All Users in Linux
- Assign Read/Write Access to a User on Specific Directory in Linux
- Linux Chgrp Command for Beginners (5 Examples)
sudo mkdir -p /var/www/reports/ sudo groupadd project sudo usermod -a -G project tecmint sudo chgrp -R project /var/www/reports/ sudo chmod -R 2775 /var/www/reports/
create more system users and add them to the directory group as follows:
sudo useradd -m -c "Aaron" -s/bin/bash -G project aaron sudo useradd -m -c "John" -s/bin/bash -G project john sudo useradd -m -c "Ravi" -s/bin/bash -G project ravi sudo mkdir -p /var/www/reports/aaron_reports sudo mkdir -p /var/www/reports/john_reports sudo mkdir -p /var/www/reports/ravi_reports
http://www.cyberciti.biz/tips/linux-shared-library-management.html
- ldconfig : Updates the necessary links for the run time link bindings.
- ldd : Tells what libraries a given program needs to run.
- ltrace : A library call tracer.
- ld.so/ld-linux.so: Dynamic linker/loader.
Install binary software using sudo
One example (Calibre) is like
sudo -v && wget -nv -O- https://raw.githubusercontent.com/kovidgoyal/calibre/master/setup/linux-installer.py | \ sudo python -c "import sys; main=lambda:sys.stderr.write('Download failed\n'); exec(sys.stdin.read()); main()"
Note that in wget the option "-O-" means writing to standard output (so the file from the URL is NOT written to the disk) and "-nv" means no verbose.
If the option "-O-" is not used, we'd better to use "-N" option in wget to overwrite an existing file.
See the Logging and Download options in wget's manual.
-O file --output-document=file The documents will not be written to the appropriate files, but all will be concatenated together and written to file. If - is used as file, documents will be printed to standard output, disabling link conversion. (Use ./- to print to a file literally named -.)
Log files
$ ls -lt /var/log
ssh log files: /var/log/auth.log
- /var/log/syslog
- /var/log/auth.log: it includes ssh log in information and lots of CRON sessions opened and closed every minutes.
And
- Why do I see a CRON session opening and closing every hour in /var/log/auth.log?
- remove cron from /var/log/auth.log
- What the %$#@ is pam_unix (cron:session) doing every ten minutes? (/var/log/auth.log)
Apache log
- /var/log/apache2/error.log (small 83K). Useful to troubleshoot errors/crashes of Apache.
grep "May 08" /var/log/apache2/error.log
- /var/log/apache2/access.log (large 10M)
/var/log/maillog
uptime command
uptime watch -n 60 uptime
Linux command similar to top to show hard disk activity
Use iotop. On ubuntu, we can use sudo apt-get install to install it. Use sudo iotop to launch it.
sudo apt-get install iotop sudo iotop -o -u $USER
Another program is iostat and the -d (disk) option. The -x option will display extension I/O status.
sudo apt-get install sysstat iostat -dx 5 # every 5 seconds
24 iostat, vmstat and mpstat Examples for Linux Performance Monitoring
Install Apache HBase
Follow the Quick Start to downloaded hbase tar ball. Suppose we save the tar ball under ~/Downloads folder and extract it in the same directory. We shall edit conf/hbase-site.xml file according to their instruction. The following is my case.
$ tar xzvf hbase-0.98.3-hadoop2-bin.tar.gz $ cd hbase-0.98.3-hadoop2/ $ cat conf/hbase-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>hbase.rootdir</name> <value>file:///home/brb/Downloads/hbase-0.98.3-hadoop2/hbase</value> </property> <property> <name>hbase.zookeeper.property.dataDir</name> <value>/home/brb/Downloads/hbase-0.98.3-hadoop2/zookeeper</value> </property> </configuration>
Before we follow the getting started guide to launch HBase, we shall make sure JAVA_HOME environment variable is created.
$ ls /usr/lib/java $ export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-amd64
Note that the last line may be replaced by
export JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:bin/javac::")
Then we can launch HBase,
$ ./bin/start-hbase.sh starting master, logging to /home/brb/Downloads/hbase-0.98.3-hadoop2/bin/../logs/hbase-brb-master-brb-P45T-A.out brb@brb-P45T-A:~/Downloads/hbase-0.98.3-hadoop2$ ./bin/hbase shell 2014-07-06 09:51:34,621 INFO [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available HBase Shell; enter 'help<RETURN>' for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version 0.98.3-hadoop2, rd5e65a9144e315bb0a964e7730871af32f5018d5, Sat May 31 19:56:09 PDT 2014 hbase(main):001:0> create 'test', 'cf' 2014-07-06 09:51:49,510 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 0 row(s) in 2.0770 seconds => Hbase::Table - test hbase(main):002:0> list 'test' TABLE test 1 row(s) in 0.0530 seconds => ["test"] hbase(main):003:0> exit brb@brb-P45T-A:~/Downloads/hbase-0.98.3-hadoop2$ ./bin/hbase shell2014-07-06 09:53:37,480 INFO [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available HBase Shell; enter 'help<RETURN>' for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version 0.98.3-hadoop2, rd5e65a9144e315bb0a964e7730871af32f5018d5, Sat May 31 19:56:09 PDT 2014 hbase(main):001:0> list 'test' TABLE 2014-07-06 09:53:44,373 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable test 1 row(s) in 1.4800 seconds => ["test"] hbase(main):002:0> put 'test', 'row1', 'cf:a', 'value1' 0 row(s) in 0.4460 seconds hbase(main):003:0> put 'test', 'row2', 'cf:b', 'value2' 0 row(s) in 0.0140 seconds hbase(main):004:0> put 'test', 'row3', 'cf:c', 'value3' 0 row(s) in 0.0050 seconds hbase(main):005:0> scan 'test' ROW COLUMN+CELL row1 column=cf:a, timestamp=1404654837532, value=value1 row2 column=cf:b, timestamp=1404654856976, value=value2 row3 column=cf:c, timestamp=1404654866298, value=value3 3 row(s) in 0.0560 seconds hbase(main):006:0> get 'test', 'row1' COLUMN CELL cf:a timestamp=1404654837532, value=value1 1 row(s) in 0.0280 seconds hbase(main):007:0> disable 'test' 0 row(s) in 1.6050 seconds hbase(main):008:0> drop 'test' 0 row(s) in 0.2290 seconds hbase(main):009:0> exit brb@brb-P45T-A:~/Downloads/hbase-0.98.3-hadoop2$
curl vs wget
sudo apt-get install curl
For example, the Download link at the National Geographic Travel Photo Contest 2014 works for curl but not wget. I can use curl with -o option but wget with -o will not work in this case. Note with curl, we can also use the -O (capital O) option which will write output to a local file named like the remote file.
curl \ http://travel.nationalgeographic.com/u/TvyamNb-BivtNwcoxtkc5xGBuGkIMh_nj4UJHQKuoXEsSpOVjL0t9P0vY7CvlbxSYeJUAZrEdZUAnSJk2-sJd-XIwQ_nYA/ \ -o owl.jpg
Should I Use Curl Or Wget? and curl vs Wget
- The main benefit of using the wget command is that it can be used to recursively download files.
- The curl command lets you use wildcards to specify the URLs you wish to retrieve. And curl supports more protocols than wget (HTTP, HTTPS, FTP) does.
The wget command can recover when a download fails whereas the curl command cannot.
Actually curl supports continuous downloading too. But not all FTP connection supports continuous downloading. The following examples show it is possible to use the continuous downloading option in wget/curl for downloading file from ncbi FTP but not from illumina FTP.
$ wget -c ftp://igenome:[email protected]/Drosophila_melanogaster/Ensembl/BDGP6/Drosophila_melanogaster_Ensembl_BDGP6.tar.gz --2017-04-13 10:46:16-- ftp://igenome:*password*@ussd-ftp.illumina.com/Drosophila_melanogaster/Ensembl/BDGP6/Drosophila_melanogaster_Ensembl_BDGP6.tar.gz => ‘Drosophila_melanogaster_Ensembl_BDGP6.tar.gz’ Resolving ussd-ftp.illumina.com (ussd-ftp.illumina.com)... 66.192.10.36 Connecting to ussd-ftp.illumina.com (ussd-ftp.illumina.com)|66.192.10.36|:21... connected. Logging in as igenome ... Logged in! ==> SYST ... done. ==> PWD ... done. ==> TYPE I ... done. ==> CWD (1) /Drosophila_melanogaster/Ensembl/BDGP6 ... done. ==> SIZE Drosophila_melanogaster_Ensembl_BDGP6.tar.gz ... 762893718 ==> PASV ... done. ==> REST 1706053 ... REST failed, starting from scratch. ==> RETR Drosophila_melanogaster_Ensembl_BDGP6.tar.gz ... done. Length: 762893718 (728M), 761187665 (726M) remaining (unauthoritative) 0% [ ] 374,832 79.7KB/s eta 2h 35m ^C $ curl -L -O -C - ftp://igenome:[email protected]/Drosophila_melanogaster/Ensembl/BDGP6/Drosophila_melanogaster_Ensembl_BDGP6.tar.gz ** Resuming transfer from byte position 1706053 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 727M 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 curl: (31) Couldn't use REST $ wget -c ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b147_GRCh37p13/VCF/common_all_20160601.vcf.gz --2017-04-13 10:52:02-- ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b147_GRCh37p13/VCF/common_all_20160601.vcf.gz => ‘common_all_20160601.vcf.gz’ Resolving ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)... 2607:f220:41e:250::7, 130.14.250.10 Connecting to ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)|2607:f220:41e:250::7|:21... connected. Logging in as anonymous ... Logged in! ==> SYST ... done. ==> PWD ... done. ==> TYPE I ... done. ==> CWD (1) /snp/organisms/human_9606_b147_GRCh37p13/VCF ... done. ==> SIZE common_all_20160601.vcf.gz ... 1023469198 ==> EPSV ... done. ==> RETR common_all_20160601.vcf.gz ... done. Length: 1023469198 (976M) (unauthoritative) 24% [===========================> ] 255,800,120 55.2MB/s eta 15s ^C $ wget -c ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b147_GRCh37p13/VCF/common_all_20160601.vcf.gz --2017-04-13 10:52:11-- ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b147_GRCh37p13/VCF/common_all_20160601.vcf.gz => ‘common_all_20160601.vcf.gz’ Resolving ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)... 2607:f220:41e:250::7, 130.14.250.10 Connecting to ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)|2607:f220:41e:250::7|:21... connected. Logging in as anonymous ... Logged in! ==> SYST ... done. ==> PWD ... done. ==> TYPE I ... done. ==> CWD (1) /snp/organisms/human_9606_b147_GRCh37p13/VCF ... done. ==> SIZE common_all_20160601.vcf.gz ... 1023469198 ==> EPSV ... done. ==> REST 267759996 ... done. ==> RETR common_all_20160601.vcf.gz ... done. Length: 1023469198 (976M), 755709202 (721M) remaining (unauthoritative) 47% [++++++++++++++++++++++++++++++========================> ] 491,152,032 50.6MB/s eta 12s ^C $ curl -L -O -C - ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b147_GRCh37p13/VCF/common_all_20160601.vcf.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 65 976M 65 639M 0 0 83.7M 0 0:00:11 0:00:07 0:00:04 90.4M^C
curl man page, supported protocols
https://curl.haxx.se/docs/manpage.html
wget and username/password
http://www.cyberciti.biz/faq/wget-command-with-username-password/
Download and Un-tar(Extract) in One Step
If we don't want to avoid saving a temporary file, we can use one piped statement.
curl http://download.osgeo.org/geos/geos-3.5.0.tar.bz2 | tar xvz # OR wget http://download.osgeo.org/geos/geos-3.5.0.tar.bz2 -O - | tar jx
See shellhacks.com. Note that the magic part of the wget option "-O -"; it will output the document to the standard output instead of a file.
Download and execute the script in one step
See Execute bash script from URL.
curl -s https://server/path/script.sh | sudo sh curl -s http://server/path/script.sh | sudo bash /dev/stdin arg1 arg2
curl and POST request
- http://superuser.com/questions/149329/what-is-the-curl-command-line-syntax-to-do-a-post-request
- https://learn.adafruit.com/raspberry-pi-physical-dashboard?view=all (the original post I saw)
- http://conqueringthecommandline.com/book/curl
curl and proxy
How to use curl command with proxy username/password on Linux/ Unix
Website performance
httpstat – A Curl Statistics Tool to Check Website Performance
wget to download a folder
wget -A pdf,jpg,PDF,JPG -m -p -E -k -K -np http://site/path/
wget to download a website
- http://linux.about.com/od/commands/a/Example-Uses-Of-The-Command-Wget.htm
- https://www.gnu.org/software/wget/manual/wget.html
To download a copy of a complete web site, use the recursive option ('-r') By default it will go up to five levels deep. You can change the default level by using the '-l' option.
All files linked to in the documents are are downloaded to enable complete offline viewing ('-p' and '--convert-links' options). Instead of having the progress messages displayed on the standard output, you can save it to a log file with the -o option.
wget -p --convert-links -r -l2 linux.about.com -o logfile wget -p --convert-links -r -l1 https://csgillespie.github.io/efficientR # create csgillespie/efficientR
aria2 - command line downloader supports torrents and multi-connection
The -x argument helps a little bit.
# Download a file 112MB; see https://www.archlinux.org/download/ $ time aria2c http://mirror.jmu.edu/pub/archlinux/iso/2016.11.01/archlinux-bootstrap-2016.11.01-i686.tar.gz # 16 seconds $ time aria2c -x10 http://mirror.jmu.edu/pub/archlinux/iso/2016.11.01/archlinux-bootstrap-2016.11.01-i686.tar.gz # 11 seconds
Axel
It can create an unlimited number of worker threads to download any kind of data. See https://www.beginnersheap.com/top-5-command-line-download-accelerators-linux/
lftp
- It supports FXP (site-to-site transfers) and dropping to background
- How to use lftp to accelerate ftp/https download speed on Linux/UNIX. It can launch several commands in parallel in the background.
Apply a patch to source code
- http://www.cyberciti.biz/faq/appy-patch-file-using-patch-command/.
- http://www.thegeekstuff.com/2014/12/patch-command-examples/
For example Tophat 2.0.12 compatibility with Samtools 1.0,
brb@brb-VirtualBox:~/Downloads$ ls support_for_tophat_1.patch tophat-2.0.12 tophat-2.0.12.tar.gz brb@brb-VirtualBox:~/Downloads$ grep -r -i "check_samtools" tophat-2.0.12/ tophat-2.0.12/src/tophat.py:def check_samtools(): tophat-2.0.12/src/tophat.py: check_samtools() brb@brb-VirtualBox:~/Downloads$ cp support_for_tophat_1.patch tophat-2.0.12/src/ brb@brb-VirtualBox:~/Downloads$ cd tophat-2.0.12/src/ brb@brb-VirtualBox:~/Downloads/tophat-2.0.12/src$ patch tophat.py < support_for_tophat_1.patch patching file tophat.py Hunk #1 succeeded at 1540 (offset 3 lines). Hunk #2 succeeded at 1563 (offset 3 lines). brb@brb-VirtualBox:~/Downloads/tophat-2.0.12/src$ ls
Get internal IP address
$ hostname -I
Get external IP address
How to find your IP address in Linux
https://github.com/jakewmeyer/Geo (one shell script)
It seems there is no way to get the external IP address without not using external services.
# http://unix.stackexchange.com/questions/22615/how-can-i-get-my-external-ip-address-in-a-shell-script sudo apt-get install dnsutils dig +short myip.opendns.com @resolver1.opendns.com # OR curl http://ipecho.net/plain; echo # OR curl ipv4.ipogre.com
The above only gives the IP. The following method gives geo information too.
curl ipinfo.io # OR give a specific IP (domain name does not work) curl ipinfo.io/216.58.194.46 { "ip": "216.58.194.46", "hostname": "dfw25s12-in-f14.1e100.net", "city": "Mountain View", "region": "California", "country": "US", "loc": "37.4192,-122.0574", "org": "AS15169 Google Inc.", "postal": "94043" }
IP geolocation
Test url: ubuntu.mirrors.pair.com
- https://www.ipligence.com/geolocation
- http://www.infosniper.net/
- http://www.ipfingerprints.com/
- http://ip-api.com/ (it shows your IP, internal IP, OS, browser/user-agent, DNS server from outside?). The final query URL is simple; for example http://ip-api.com/#ubuntu.mirrors.pair.com
- https://www.iplocation.net/ (not work)
- http://geobytes.com/iplocator/ (not work)
- ipstack. How to Use the IPStack API for IP Geolocation Lookups. 10,000 searches per month for free.
Domain
- WHOIS LOOKUP. For example, consider "r-pkg.org" domain,
- The 'Sponsoring Registrar' shows who is the sponsoring registrar (eg GoDaddy.com).
- The 'Registrant Name' shows who registered this domain.
- Command Line Interface.
sudo apt-get install whois whois r-pkg.org
- IP-Lookup. For example, consider "r-pkg.org" domain.
- The linux command line tool 'ping' shows the ip address.
- When we use the ip address to search in the IP-Lookup, the IP owner info > 'Organization' or 'OrgName' field shows the owner of this IP (eg Digital Ocean, Inc).
- The Domain owner info there gives the same (or less) information as WHOIS LOOKUP.
- WhoIsHostingThis or webhostinghero (the returned result will be like Amazon, GoDaddy, CloudFlare, Github, Verizon, etc).
Subnet
- Class A: 255.0.0.0 or /8;
- Class B, 255.255.0.0 or /16;
- Class C, 255.255.255.0 or /24.
For example, in the subnet 192.168.5.0/255.255.255.0 (192.168.5.0/24) the identifier 192.168.5.0 commonly is used to refer to the entire subnet.
In the /16 subnet 192.168.0.0/255.255.0.0, which is equivalent to the address range 192.168.0.0–192.168.255.255
ping command
ping uses the ICMP Echo Message to force a remote host to echo a packet back to the local host. If packets can travel to and from a remote host, it indicates that the two hosts can successfully communicate.
Build a home network
- Virtual router using vSphere. (Good ! It includes an introduction to vSphere installation)
- http://rbgeek.wordpress.com/2012/05/14/ubuntu-as-a-firewallgateway-router/
IP Subnet Calculator
https://www.dan.me.uk/ipsubnets?ip=10.0.0.0
- CIDR block IP range (network - broadcast) Subnet Mask IP Quantity
- 10.0.0.0/24 10.0.0.0 - 10.0.0.255 255.255.255.0 256
- 10.0.0.0/16 10.0.0.0 - 10.0.255.255 255.255.0.0 65536=256^2
- 10.0.0.0/8 10.0.0.0 - 10.255.255.255 255.0.0.0 16777216=256^3
How Use Your Router and ISP’s Modem/Router Combo In Tandem
http://www.howtogeek.com/255206/how-use-your-router-and-isps-modemrouter-combo-in-tandem/
Troubleshoot and repair network problems
http://www.linuxuser.co.uk/features/troubleshot-repair-linux-networks
Computer 1 can ping Computer 2 but not reverse
Use the route command to solve. See also this post.
Simple Network Management Protocol (snmp)
- https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol
- Default port number is 161
- What Is SNMP? How To Install & Configure SNMP in Linux
- Install and configure SNMP on Ubuntu
- Change port number on Dell iDrac 8
- DDoS attack
Monitor network by Cacti (GUI)
- http://www.ubuntugeek.com/install-cacti-monitoring-tool-on-ubuntu-15-10-server.html
- http://www.cacti.net/
Monitor network by command line
3 Simple, Excellent Linux Network Monitors: iftop, nethogs and vnstat.
iftop
Use the interface top iftop command. On ubuntu, we need to use sudo apt-get install iftop and then run it by sudo iftop -i eth0. After that, we can press some keys to toggle options.
- p: port
- s: source
- d: destination
See thegeekstuff.
It is strange that the output shows other devices names in my network.
$ dig A pandora.com $ ipcalc -b 208.85.40.20 $ sudo iftop -F 208.85.40.20/24 -i wlan0
nethogs
$ sudo nethogs wlan0
nload
nload -m
nload – Monitor Linux Network Bandwidth Usage in Real Time
The result is the same as gtop (gtop is cooler) gives.
bmon
https://www.tecmint.com/bmon-network-bandwidth-monitoring-debugging-linux/
vnstat
http://www.thegeekstuff.com/2011/11/vnstat-network-traffic-monitor/
# 1. Install vnStat sudo apt-get install vnstat # 2. Pick a Interface to Monitor using vnStat vnstat -u -i eth0 vnstat --iflist vnstatd -d # start the daemon ps -ef | grep vnst # 3. vnStat Basic Usage vnstat # 4. vnStat hours, days, months, weeks Network Data vnstat -d vnstat -m # 5. Export the data to Excel or other DB vnstat --dumpdb # 6. Display Live Network Statistics vnstat -l # 7. Change the default vnstat output format vnstat -s (--short) vnstat --style 0 # 8. Display Top 10 Traffic Days vnstat --top10
ifconfig - spoof the hardware address at the software level
To change the MAC address temporarily on a NIC (network interface controller),
sudo ifconfig eth0 down sudo ifconfig eth0 hw ether 00:11:22:33:44:55 sudo ifconfig eth0 up
And it seems there is no need to modify /etc/network/interfaces.
For wlan
sudo ifconfig wlan0 down sudo ifconfig wlan0 hw ether 00:11:22:33:44:55 sudo ifconfig wlan0 up
See
- 7 Examples To Configure Network Interface
- 15 Useful “ifconfig” Commands to Configure Network Interface in Linux
- What does ifconfig promisc mode do, or promiscuous mode in general?
ip command
It is said ip is replacing the old ifconfig command on modern Linux distributions.
http://www.makeuseof.com/tag/networking-commands-linux-terminal/
ip a ip addr ip address show ip link set DEVICE down # eg ip link set eth0 down ip link set DEVICE up
iptables
- See the ufw command which provides an easy way to configure iptables.
- How to disable iptables firewall temporarily
route
7 Linux Route Command Examples
- Display Existing Routes (route -n)
- Adding a Default Gateway (route add default gw 192.168.1.1)
- List Kernel’s Routing Cache Information (route -Cn)
- Reject Routing to a Particular Host or Network (route add -host 192.168.1.51 reject)
- Make 192.168.3.* Accessible from 192.168.1.* (route add -net 192.168.3.0 netmask 255.255.255.0 gw 192.168.3.10)
- Make 192.168.1.* Accessible from 192.168.3.* (route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.1.10)
- Allow Internet Access/External World (route add default gw 125.250.60.59)
On Ubuntu 16.04, it shows
$ route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default FIOS_Quantum_Ga 0.0.0.0 UG 600 0 0 wlp3s0 link-local * 255.255.0.0 U 1000 0 0 wlp3s0 192.168.1.0 * 255.255.255.0 U 600 0 0 wlp3s0 $ route -n # showing numerical IP address instead of host name. Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.1.1 0.0.0.0 UG 600 0 0 wlp3s0 169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 wlp3s0 192.168.1.0 0.0.0.0 255.255.255.0 U 600 0 0 wlp3s0
Flag value 'U' means up and 'G' means gateway'.
Connect two networks
The trick is explained in this post or the above route command.
For example, my network structure is
- Modem/router: LAN IP 192.168.1.*/24
- PC1: connect to Modem/router
- Second router (ASUS) connect to Modem/router: its WAN IP is 192.168.1.ASUS. It's LAN IP 192.168.2.*/24
- PC2 (raspberry pi): connect to the second router (ASUS): its IP is 192.168.1.212
By default, PC2 can ssh to PC1 but PC1 cannot access PC2.
The following command will solve the problem that PC1 cannot access PC2:
# From PC1 $ sudo route add -net 192.168.2.0 netmask 255.255.255.0 gw 192.168.1.ASUS $ ssh [email protected] # 192.168.2.212 is the IP address for the Raspberry Pi $ netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 192.168.2.0 192.168.1.ASUS 255.255.255.0 UG 0 0 0 eth0
One article from linux.com using the ip command.
traceroute
sudo apt-get install traceroute traceroute 8.8.8.8
On Windows, we can use the tracert command. For example, tracert www.microsoft.com.
netstat
How to use netstat in GNU/Linux
- -l or --listening shows only the sockets currently listening for incoming connection.
- -a or --all shows all sockets currently in use.
- -e --show extended/additional information
- -t or --tcp shows the tcp sockets.
- -u or --udp shows the udp sockets.
- -n or --numeric shows the hosts and ports as numbers, instead of resolving in dns and looking in /etc/services.
- -s --Print network stats
- -r --Print the network routing information
- -p --Print PID and name of the program to which each socket belongs
netstat -l # only listening ports netstat -rn # displays the system's routing table netstat -at netstat -ant # For tcp sudo netstat -pant # show ports and programs (pant = 喘氣). Best of the best!!! sudo netstat -peanut # (output is too wide) netstat -anp | grep 3306 | wc -l # print the number of connections for the port mysql port i.e. 3306.
nmap - port scanning & IPs in local network
nmap - Network exploration tool and security / port scanner
- https://nmap.org/book/nmap-os-db.html. Local OS database is located at /usr/share/nmap/nmap-os-db. The 2nd line will show the revision number.
- Modifying the nmap-os-db Database Yourself
- Download the latest from https://svn.nmap.org/nmap/nmap-os-db. Note that the current revision number has to be found from the website. You can edit the file and insert the revision number on the 2nd line of your local copy.
- Even I update the database, it cannot detect my Ubuntu 14.04 OS (it only shows OS details: Linux 3.8 - 4.9). For the Raspberry Pi, it can show information from the network adapter; e.g. MAC Address: AA:BB:CC:DD:EE:FF (Raspberry Pi Foundation) but not the OS name (OS details: Linux 3.2 - 4.8).
sudo mv /usr/share/nmap/nmap-os-db /usr/share/nmap/nmap-os-db-old cd /usr/share/nmap sudo wget https://svn.nmap.org/nmap/nmap-os-db
- http://www.cyberciti.biz/networking/nmap-command-examples-tutorials/
- http://bencane.com/2013/02/25/10-nmap-commands-every-sysadmin-should-know/
- http://www.tecmint.com/nmap-command-examples/
sudo apt-get install nmap nmap 192.168.1.99 # does not require root privileges # used to check open ports nmap 192.168.1.* # show IPs and ports in LAN sudo nmap -sP 192.168.1.1/24 # show connected IPs (no hostnames?) and MAC addresses # If you don't use 'sudo' only partial devices can be found # The output may contains the hostname. For example, # Nmap scan report for brb-P45T-A.fios-router.home (192.168.1.xxx) nmap -sV 192.168.1.1 # show Daemon name (in VERSION column) together with port number nmap -T4 -F 192.168.1.99-255 # show connected IPs and open ports # -F means fast nmap -F taichimd.us nmap -v taichimd.us nmap -A 192.168.1.1 # Aggressive scan (more output) nmap -p http,ssh,mysql taichimd.us # scan ports/services # note that mysql will be shown as closed nmap --open taichimd.us # scan open ports sudo nmap -traceroute nih.gov sudo nmap -sS -O 192.168.1.99 # -O shows operating system # eth0 MAC $ nmap localhost Starting Nmap 7.01 ( https://nmap.org ) at 2017-10-09 15:01 EDT Nmap scan report for localhost (127.0.0.1) Host is up (0.00016s latency). Not shown: 996 closed ports PORT STATE SERVICE 22/tcp open ssh 25/tcp open smtp 80/tcp open http 631/tcp open ipp
A gui version of nmap is called Zenmap.
nslookup and host
$ host google.com google.com has address 172.217.5.238 google.com has IPv6 address 2607:f8b0:4004:802::200e google.com mail is handled by 30 alt2.aspmx.l.google.com. google.com mail is handled by 10 aspmx.l.google.com. google.com mail is handled by 40 alt3.aspmx.l.google.com. google.com mail is handled by 50 alt4.aspmx.l.google.com. google.com mail is handled by 20 alt1.aspmx.l.google.com. $ nslookup google.com Server: 127.0.1.1 Address: 127.0.1.1#53 Non-authoritative answer: Name: google.com Address: 172.217.7.238
dig
$ dig world.std.com ; <<>> DiG 9.9.5-3ubuntu0.16-Ubuntu <<>> google.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49227 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;google.com. IN A ;; ANSWER SECTION: google.com. 130 IN A 172.217.5.238 ;; Query time: 11 msec ;; SERVER: 127.0.1.1#53(127.0.1.1) ;; WHEN: Fri Dec 01 17:32:37 EST 2017 ;; MSG SIZE rcvd: 55
arp (Address Resolution Protocol)
The arp command can be used to show the MAC addresss of all hosts in LAN
arp -a
nmcli
Copy text to a clipboard to be used in other apps
Install the xclip program. See here or here.
sudo apt-get install xclip # Examples sort -n -k 3, -k 2 file.txt | xclip -selection clipboard cat ~/.ssh/id_rsa.pub | xclip -sel clip
Works.
Start Emacs without X
Add -nw (no window) option.
emacs -nw
Audio
mp3 codecs
https://help.ubuntu.com/community/RestrictedFormats
sudo apt-get install ubuntu-restricted-extras
Concatenate mp3 files
sudo apt-get install mp3wrap mp3wrap output.mp3 *.mp3
Reduce the size of an mp3 file
Specify a new lower bitrate using the -b option in lame. For example if your starting mp3 has a quality of 256kbs you can lower its bitrate to 128kbps (or even lower like 64kbps) by:
lame --mp3input -b 128 input.mp3 output.mp3
Convert ogg to mp3
ffmpeg is not included in Ubuntu repository. Use the avconv command. http://superuser.com/questions/15327/how-to-convert-ogg-to-mp3
sudo apt-get install libav-tools avconv -i input.ogg -c:a libmp3lame -q:a 2 output.mp3
Convert m4a to mp3
avconv -i input.m4a output.mp3
Remove the vocals from any song using Audacity
https://www.makeuseof.com/tag/remove-vocals-song-audacity/
Normalize the volume of an audio file
- Can You Losslessly Increase the Volume of MP3 Files?
- Use Audacity. To raise (Amplify) volume: 1. Edit > Select All. 2. Effect > Amplify. Increase db from 0 to 15, for example. Check clip3. Export > MP3 or just start to listen.
- Command line tool: avconv (replace ffmpeg program). See this post.
avconv -ss 00:00:10 -i OLD.mp3 -vol 2560 NEW.mp3
The anconv/ffmpeg -vol parameter amplifies the sound. The default value is 256 (no amplification), and you can adjust the number accordingly. Here it’s 2560, as it’s 10 times louder. Note that these are not decibel values or anything that sophisticated, but just an integer value. 512 equals to twice the volume, 768 three times, 1024 four times, etc. The -ss parameter specifies the start time offset. Here it will skip the first 10 seconds.
- Command line tool: sox.
- http://askubuntu.com/questions/246242/how-to-normalize-sound-in-mp3-files
- http://www.linuxandlife.com/2013/03/how-to-use-sox-audio-editing.html
- http://digitalcardboard.com/blog/2009/08/25/the-sox-of-silence/ deal with several kinds of silence.
- http://www.thegeekstuff.com/2009/05/sound-exchange-sox-15-examples-to-manipulate-audio-files/
I found the converted file by sox has about one half file size compared to anconv/ffmpeg program (source file=47MB, anconv converted=135MB, sox converted file=54MB).
sudo apt-get install sox libsox-fmt-all sox --norm OLD.mp3 NEW.mp3 trim 10 sox --norm OLD.mp3 NEW.mp3 silence 1 0.1 1% sox -v 4.0 OLD.mp3 NEW.mp3 # increase volume
where '--norm' will normalize the audio and the 'trim' option set to skip the first 10 seconds. The silence parameter allows to trim silence at the beginning without a need to specify the number of seconds.
cut, delete or trim an audio
- Open the audio file in audacity.
- select a region in the waveform area. Do not select in the time interval area (above the waveform).
- To precisely select a range from one position to the end. Click Zoom in several times. Click one position in the waveform and click Edit -> Select -> Cursor to the track end to select
- Similarly, if we want to precisely select a range from the start to some position, we can click one position in the waveform and then click Edit -> Select -> Track start to cursor.
- To move around the track, use the scrollbar (below the waveform and above the bottom toolbar, not quite clear in Ubuntu/Unity)
- Click Edit -> Remove Audio or labels -> Cut/Delete/Trim Audio
- play the new audio by clicking the green triangle.
- File -> Export -> mp3 format.
Helpful resource for Audacity.
Fade out at the end of an audio
- Select a region.
- Effect -> Fade out
Wireshark
- http://www.howtogeek.com/204458/why-you-shouldn%E2%80%99t-use-mac-address-filtering-on-your-wi-fi-router/
- http://www.howtogeek.com/191482/how-an-attacker-could-crack-your-wireless-network-security/
- http://www.howtogeek.com/104278/how-to-use-wireshark-to-capture-filter-and-inspect-packets/
sudo apt-get install wireshark sudo chmod 4711 `which dumpcap`
Track the Time a Command Takes
time command
Use time command.
time COMMAND time (COMMAND1; COMMAND2) time (COMMAND1 && COMMAND2) help time
When I run a set of 7 jobs using parallel, time command gives an output
real 15m53.788s user 95m20.238s sys 9m1.320s
Here we see the real time is about 16m and the user time is about 6-7 times the real time. Indicating the parallel executing works.
/usr/bin/time command
/usr/bin/time provides more information then time command.
man time
Magazines
Latex
Editors
- 10 Best LaTeX Editors For Linux.
- Texmaker. R's installr package has a function to install Texmaker. Cross platforms.
- TEXworks. Cross platforms.
Online editing
- Latex Base. You can start to try it without registration. Free accounts cannot publish but still can download.
- Overleaf. Free account for 1GB space.
- ShareLatex
Missing cls
- texlive-latex-extra packages and texlive-publishers packages.
- https://tex.stackexchange.com/questions/179214/elsarticle-cls-not-found-when-using-texmaker-but-texlive-latex-extra-is-install/179250
$ apt-cache search IEEEtran texlive-publishers - TeX Live: Publisher styles, theses, etc.
sudo apt-get install texlive-publishers
Missing sty
$ apt-cache search pseudocode gpt - G-Portugol is a portuguese structured programming language libgportugol-dev - Development files for the G-Portugol library libgportugol0 - G-Portugol library texlive-science - TeX Live: Natural and computer sciences $ sudo apt-get install texlive-science
PDF reader
The default one Evince seems slow when I try to view odroid magazine. I installed and tried MuPDF (github source code). It seems faster and I don't see blank pages when I view one odroid magazine. In terms of speed, mupdf >> xpdf >> okular >> Evince.
sudo apt-get install mupdf
Keyboard shortcuts for mupdf (man mupdf) or http://mupdf.com/docs/manual:
W - fit to width H - fit to height L - rotate page left (clockwise) R - rotate page right (counter-clockwise) 12g - go to page 12 >,< - go to the next or previous page +,- - zoom in or out / - search for text n,N - Find the next or previous search result. h,j,k,l - Scroll page left, down, up, or right.
To copy a text, use right mouse button to select a text. Then use Ctrl+c to copy it.
Other pdf viewer choices are
- acroread
- Allow to have custom colors for page background and document text.
- The custom colors works well on Macbook Pro (2880 x 1440). Background color #494949 and text color #494949.
- xpdf. old-fashioned. slow.
- evince. slow.
- okular (KDE/Qt application)
- Allow to change its background color. Though it works, the result using 'invert colors' option is not good on Dell U2312HM. We can try other option like 'dark & light colors' where we can change the individual colors for the background (say #494949) and text.
- Not as fast as mupdf. It can open a variety of ebook formats.
- MacOS should work but it needs to install KDE.
- Able to show file properties eg Page Size (eg 50x36 in), Creator (eg PowerPoint), Producer (eg Mac OS X Quartz PDFContext), PDF version (eg 1.3)
- kpdf
- gv
- qpdfview. slow. Used by Raspbian june 2018.
- Foxit or PDF-XChange Viewer(needs wine)
PDF crop
pdfcrop (briss is better)
https://askubuntu.com/questions/124692/command-line-tool-to-crop-pdf-files
sudo apt-get install texlive-extra-utils pdfcrop input.pdf output.pdf # no margins, works but seems too tight pdfcrop --margins 5 input.pdf output.pdf # crop pdf but keep 5 bp from each side of page pdfcrop --margins '5 10 20 30' input.pdf output.pdf # left, top, right and bottom margins of 5, 10, 20, and 30 pt # To actually crop something away, use negative values in the argument for crop. # For example, to crops 50 pts from the left, top, right, bottom (in this order). pdfcrop --margins '-50 -50 -50 -50' input.pdf output.pdf
One problem I found is (for newer PDFs with meta data) --margins initially removes the entire margin before implementing the adjustment. This will cause some pages being chopped out.
This java program gives me a better control on cropping
- Download the file briss-0.9.tar.gz (8.7 MB) and extract it
- Run java -jar briss-0.9.jar
- Load the pdf file. It will ask what pages to be excluded from merging (This function does not work). Click 'Cancel' to continue.
- It will automatically create two rectangle areas; one for odd (left) pages and the other for even (right)pages
- Now we work on the left page first. Enlarge the selection to suit our need. Then right click & choose 'Select/Deselect rectangle' (a dash line will be added to the edges of the rectangle) and then 'Copy rectangles'.
- Work on the right page. Right click and choose 'Delete rectangle'. Then 'Paste rectangles'.
- Now we can click 'Action -> Preview' to preview the result. If we are satisfied with the result, we can click 'Action -> Crop PDF'. Done.
Remove certain pages
https://www.linux.com/learn/manipulating-pdfs-pdf-toolkit
sudo apt install pdftk # remove pages 10 to 25 from a PDF file pdftk myDocument.pdf cat 1-9 26-end output removedPages.pdf # remove the last page pdftk infile.pdf cat 1-r2 output outfile.pdf # remove the last 2 pages pdftk infile.pdf cat 1-r3 output outfile.pdf
PDF highlight and annotation
Install Okular by
sudo apt-get install okular
To highlight a line, click F6 (Tools -> Review) to turn on the annotation tool bar (it will be shown on the left hand side of the documentation). You can then click
- the 4th icon to highlight a line (it may not be able to select the right texts we want. But when it works the result is nice)
- the last icon to draw an ellipse or a rectangle (to change from an ellipse to a rectange you can click Settings -> configure Okular... -> annotation)
Another method is to use a windows program and run it using Wine. See the discussion here.
Merge multiple pdf files into one pdf file
https://stackoverflow.com/questions/2507766/merge-convert-multiple-pdf-files-into-one-pdf
pdfunite in-1.pdf in-2.pdf in-n.pdf out.pdf
Print multiple pages per sheet: pdfnup
The program is similar to psnup.
sudo apt install texlive-extra-utils
Flow chart
- LibreOffice Draw OR MS_PowerPoint (insert > shape). Check youtube.
- yEd
- Dia & wikipedia
- (online) www.draw.io
Clock
xclock
oclock -geometry 500x500+100+0 &
oclock
oclock -bg blue -geometry 500x500+100+0 -bd purple -transparent & oclock -bg blue -geometry 500x500+100+0 -bd purple -jewel green &
See oclock, X - a portable, network-transparent window system which includes an example of specifying the geometry parameter.
dclock
Digital clock for the X Window System with flexible display.
sudo apt-get install dclock dclock -h dclock -d dclock -date "Today is %A %B %Y" -geometry 577x194+119+139
Lubuntu digital clock format
http://netgator.blogspot.com/2012/09/change-edit-panel-digital-clock-format.html. My format is
%a, %x, %r # Tue, 05/17/2016, 09:42:27 PM %a %m-%d-%y, %I:%M %p # Mon 05-30-16, 08:31 AM
Take a break
Stretchly. It's open-source and cross-platform. Nodejs is required.
Workrave is another choice. The source code is available too.
wine and winetricks
Running Linux in the AWS/Amazon Web Services
Forum software
- Simple Machines® Forum (SMF). For example http://pibot.org/forum/
RAID
Timer
- http://zeegaree.com/. Require 3 libraries that we need to install them using apt-get install. See the github page.
How to track you laptop using Prey
https://www.howtoforge.com/tutorial/how-to-track-your-linux-laptop/
last command
Linux last Command Tutorial for Beginners (8 Examples)
Display a list of system shutdown/reboot date/time
Linux Find Out Last System Reboot Time and Date Command
# Works on Linux and Mac last shutdown last reboot
Automatic reboot after power failure
It seems there is no reliable way to find out when the power failed.
The linux command 'last' can show some information about system reboot.
Another way is to modify the BIOS to select the option like 'Power off and Reboot'. This won't automatically boot your computer when it is shutdown normally.
Wake up and Shut Down Linux Automatically
- https://help.ubuntu.com/community/WakeOnLan
- https://www.cyberciti.biz/faq/ubuntu-linux-wake-on-lan-client-command-installation-examples/
- Wake up and Shut Down Linux Automatically
Two best options
- Bios: BIOS may have an easy-to-use wakeup scheduler
- wakeonlan:
- Eanble it: Check if it is enabled by default. If not, we can 1) enable it through a command (ethtool -s eth0 wol g) or 2) using the Network Manager
- Send a wake up command: (from a second linux) /usr/bin/wakeonlan D0:50:99:82:E7:2B where D0:50:99:82:E7:2B is the IP on the machine you want to wake it up
How to update Lenovo BIOS from Linux without using Windows
https://www.cyberciti.biz/faq/update-lenovo-bios-from-linux-usb-stick-pen/
Internet speed test
= Web
- https://fast.com/ (automatically run)
- https://www.bing.com/search?q=internet+speed+test
- http://www.speedtest.net/
Speedtest-cli
sudo apt-get intall python-pip sudo pip install speedtest-cli # A slightly modified code that will create a one-line space/semi-colon # delimited result is git clone https://github.com/HenrikBengtsson/speedtest-cli-extras.git speedtest-cli-extras/bin/speedtest-csv
works. But if I want to put it in cron, cron will issue an error speedtest-cli cannot be found. So I need to modify line 52 of the code in <speedtest-cli-extras/bin/speedtest-csv> to explicitly specify the location of speedtest-cli.
/usr/local/bin/speedtest-cli --share > $log
NOTE: 1. the results differ from the network connection. For example, the speed is good when I test it on the machine directly connected to the router. 2. It is helpful to modify the last line of the bash script to output what I need. 3. The separator is ";" in the output.
uname - Print system information
https://www.lifewire.com/display-system-information-uname-command-3964321
uname -a will give you
- OS (uname = uname -s if you are under a Linux environment)
- OS (uname -s) eg Linux
- node name (uname -n=hostname)
- kernel release (uname -r) eg 3.16.0-38-generic
- kernel version (uname -v)
- machine architecture (uname -m) eg x86_64
- processor (uname -p)
- hardware platform (uname -i)
- operating system (uname -o)
How to check if running in Cygwin, Mac or Linux?
python--hwinfo
Linux Logo and the current system information
odroid@odroid:~$ sudo apt-get install screenfetch odroid@odroid:~$ screenfetch ./+o+- odroid@odroid yyyyy- -yyyyyy+ OS: Ubuntu 15.10 wily ://+//////-yyyyyyo Kernel: armv7l Linux 3.10.96-77 .++ .:/++++++/-.+sss/` Uptime: 4d 23h 8m .:++o: /++++++++/:--:/- Packages: 2000 o:+o+:++.`..```.-/oo+++++/ Shell: 2263 .:+o:+o/. `+sssoo+/ Resolution: 1920x1080 .++/+:+oo+o:` /sssooo. DE: MATE 1.10.2 /+++//+:`oo+o /::--:. WM: Metacity (Marco) \+/+o+++`o++o ++////. GTK Theme: 'Ambiant-MATE' [GTK2/3] .++.o+++oo+:` /dddhhh. Icon Theme: Ambiant-MATE .+.o+oo:. `oddhhhh+ Font: Ubuntu 10 \+.++o+o``-````.:ohdhhhhh+ CPU: ARMv7 rev 3 (v7l) @ 1.4GHz `:o+++ `ohhhhhhhhyo++os: GPU: Gallium 0.4 on llvmpipe (LLVM 3.6, 128 bits) .o:`.syhhhhhhh/.oo++o` RAM: 537MiB / 1990MiB /osyyyyyyo++ooo+++/ ````` +oo+++o\: `oo++. odroid@odroid:~$ screenfetch -s # take a screenshot and auto save it to ~/ directory. odroid@odroid:~$ sudo apt-get install linuxlogo odroid@odroid:~$ linuxlogo _,met$$$$$gg. ,g$$$$$$$$$$$$$$$P. ,g$$P"" """Y$$.". ,$$P' `$$$. ',$$P ,ggs. `$$b: `d$$' ,$P"' . $$$ ,#. $$P d$' , $$P ##: :## :###: $$: $$. - ,d$$' ##' `## `#' $$; Y$b._ _,d$P' __ ## __ ## __ _ __ _ Y$$. `.`"Y$$$$P"' ,####:## ,######. ##.#####. :### ,######. ###.####: `$$b "-.__ ,##' `### ##: :## ###' `### ##' #: `## `###' `##: `Y$$b ## `## ## ## ##' `## ## ___,## ##: `## `Y$$. ## ## #######: ## ## ## .####### ##' ## `$$b. ## ## ##' ## ## ## ##' `## ## ## `Y$$b. ##. ,## ## ## ,## ## ## ## ## ## `"Y$b._ :#:._,### ##:__,## ##:__,##' ,##. ##.__:##. ## ## `"""" `:#### ### ######' `######' #### `#####"## ## ## Linux Version 3.10.96-77, Compiled #1 SMP PREEMPT Fri Feb 5 04:47:32 BRST 2016 Eight ARM Processors, 2GB RAM, 456.00 Bogomips Total odroid odroid@odroid:~$ linuxlogo -f -L list odroid@odroid:~$ linuxlogo -f -L ubuntu .-. .-'``(|||) ,`\ \ `-`. 88 88 / \ '``-. ` 88 88 .-. , `___: 88 88 88,888, 88 88 ,88888, 88888 88 88 (:::) : ___ 88 88 88 88 88 88 88 88 88 88 88 `-` ` , : 88 88 88 88 88 88 88 88 88 88 88 \ / ,..-` , 88 88 88 88 88 88 88 88 88 88 88 `./ / .-.` '88888' '88888' '88888' 88 88 '8888 '88888' `-..-( ) `-` Linux Version 3.10.96-77, Compiled #1 SMP PREEMPT Fri Feb 5 04:47:32 BRST 2016 Eight ARM Processors, 2GB RAM, 192.00 Bogomips Total odroid odroid@odroid:~$ screenfetch -h odroid@odroid:~$ linuxlogo -h
Dictionary - Artha
- Lifehacker. Once it is launched, it is sitting on the task bar. Press Ctrl+Alt+W after selecting a word to look it up in Artha (a balloon tip will pop up on the screen top-right). It also supports using regular expressions to search words.
sudo apt-get install artha
Translation
- Translate Shell. No installation is needed. It is just a bash script (4990 lines) so it works on ODroid SOC. See also A Tool To Use Google Translate From Command Line In Linux
odroid@odroid:~/binary$ ./trans :zh-TW word word /wərd/ 字 (Zì) Definitions of word [ English -> 正體中文 ] noun 字 word, character, letter, calligraphy, symbol, style of writing 詞 word, term, speech, statement 單詞 word, individual word 話 words, word, dialect, saying, talk, speech 言 word, speech, character 言辭 words, word, what one says 筆墨 pen and ink, words, word, writings 約言 pledge, promise, word verb 為 ... 措辭 word odroid@odroid:~/binary$ time ./trans -brief :zh-TW word 字 real 0m4.249s user 0m2.670s sys 0m1.330s
ASCII art
____ ____ ____ ____ _____ _ | __ )| _ \| __ ) / ___| ___ __ |_ _|__ ___ | |___ | _ \| |_) | _ \ ____\___ \ / _ \/ _` || |/ _ \ / _ \| / __| | |_) | _ <| |_) |_____|__) | __/ (_| || | (_) | (_) | \__ \ |____/|_| \_\____/ |____/ \___|\__, ||_|\___/ \___/|_|___/ |_|
____ _____ ____ _____ _______ _ | _ \| __ \| _ \ / ____| |__ __| | | | |_) | |__) | |_) |____| (___ ___ __ _| | ___ ___ | |___ | _ <| _ /| _ <______\___ \ / _ \/ _` | |/ _ \ / _ \| / __| | |_) | | \ \| |_) | ____) | __/ (_| | | (_) | (_) | \__ \ |____/|_| \_\____/ |_____/ \___|\__, |_|\___/ \___/|_|___/ | | |_|
___ ___ ___ ___ _____ _ | _ ) _ \ _ )___/ __| ___ __ |_ _|__ ___| |___ | _ \ / _ \___\__ \/ -_) _` || |/ _ \/ _ \ (_-< |___/_|_\___/ |___/\___\__, ||_|\___/\___/_/__/ |_|
Software that scan Malware and rootkits
Text to speech
- http://www.eguidedog.net/ekho.php. Compilation/build works on x86 Ubuntu 14 and Odroid Ubuntu 15.10. On Odroid I have to follow their instruction to use 'make CXXFLAGS=-DNO_SSE' instead of 'make'. However, sound feels shaky on Odroid xu4.
- http://audiobookmaker.com/
- http://project-modelino.com/online-resources-category.php?site_language=english&learn_language=chinese&category=tts
VPN
- The Biggest Misconceptions About VPNs
- Why Is Everyone Talking About VPNs?
- The Laziest, Cheapest Way to Circumvent Your Snooping ISP
- Your Pick For the Best VPN Service Is Private Internet Access
- How to Set Up Your Own Completely Free VPN In the Cloud
- How—and why—you should use a VPN any time you hop on the internet
- remoteaccessvpn.nih.gov for NIH. Download and unzip the profile and place the profile (.xml)in “/opt/cisco/anyconnect/profile/” directory
OpenVPN
- Tutorial from nordvpn (free 3-day trial)
- How To Set Up an OpenVPN Server on Ubuntu 14.04
- How to Build An OpenVPN Access Point by Hak5 in Youtube.
- Secure you server administration with multiplatform VPN connection by howtoforge.
List of free and fast VPNs
- 5 Great Free VPN Services Compared: Which Is Fastest?
- How to Choose the Best (and Fastest) Alternative DNS Server
- Windscribe, mentioned by Sick of NBC's vapid Olympics coverage? Use a VPN and you can watch the BBC's coverage instead
- Some result from one slickdeals comment:
- my connection speed is 150 down, when i do a speed test i get 170 down.
- with nordvpn trial i get 165 down
- with windscribe free i get 95 down
- with vpnsecure trial i get 30 down
How to Set Up a VPN on Your Router
https://www.makeuseof.com/tag/setup-vpn-router/
Mono Project
Mono is a software platform designed to allow developers to easily create cross platform applications part of the .NET Foundation
Mono is required for Repetier-Host software for 3D printing.
NAS server
OpenMediaVault
4 easy Linux projects for newbies and intermediate users. OpenMediaVault is a linux-based system.
Docker container for OpenMediaVault. OpenMediaVault插件之Docker教程
FreeNAS
ZFS system (FreeBSD-based).
10 Reasons Why You Should Store Your Data on a FreeNAS Box. Note With the current version of FreeNAS (FreeNAS 11) comes a hypervisor. See
- https://doc.freenas.org/11/vms.html
- https://forums.freenas.org/index.php?threads/freenas-11-0-released.55327/
- Virtualize FreeNAS
- bhyve, the BSD Hypervisor
- FreeNAS as hypervisor host
Change detection
http://bhfsteve.blogspot.com/2013/03/monitoring-web-page-for-changes-using.html
3 command-line tools for feigning productivity
https://opensource.com/article/18/2/command-line-tools-productivity: Blessed-contrib (javascript), Genact, Hollywood.
Mind mapping
Diagram
- Calligra Flow. Microsoft Visio alternative.
Open source surveillance
Systemctl, systemd
- Linux 系统开机启动项清理 中文 & English
- Chkservice – An Easy Way to Manage Systemd Units in Terminal
Bitcoin
Check weather
- Display Weather Forecast In Your Terminal With Wttr.in
$ curl wttr.in $ curl wttr.in/washington $ curl wttr.in/olney $ curl wttr.in/~olney $ curl wttr.in/taipei
- 10 Ways to Check the Weather From Your Linux Desktop
Best Linux Adobe Alternatives You Need to Know
- https://linux.cn/article-8928-1.html and https://www.maketecheasier.com/adobe-alternatives-for-linux/
Linux distributions
The Top 10 Open Source Distros You Haven’t Heard About
Popular Linux distributions
2017年度最佳Linux版本出爐, https://www.linuxquestions.org/questions/linux-news-59/2017-linuxquestions-org-members-choice-award-winners-4175623289/
Small/lightweight Linux distributions
- Install Linux to Save Space! These Tiny Linux Distros Are Super Small 10/10/2017
- 13 Lightweight Linux Distributions to Give Your Old PC a New Lease of Life 5/5/2017. Some distributions that can be run in RAM: Macpup, Porteus.
10 Best And Most Secure Linux Distributions
https://fossbytes.com/secure-linux-distros-privacy-anonymity/