Linux: Difference between revisions
(→diff) |
|||
| (70 intermediate revisions by the same user not shown) | |||
| Line 30: | Line 30: | ||
By default, the man program normally uses a terminal '''pager''' program such as '''less''' to format its output. | By default, the man program normally uses a terminal '''pager''' program such as '''less''' to format its output. | ||
[https://www.tecmint.com/view-colored-man-pages-in-linux/ Add the following to "~/.bashrc" file] | [https://www.tecmint.com/view-colored-man-pages-in-linux/ Add the following to "~/.bashrc" file]. For example, '''LESS_TERMCAP_mb''' customizes the appearance of blink text in the '''less''' pager (or '''man''' command) and '''LESS_TERMCAP_us''' customizes the appearance of underlined text in the '''less''' pager. | ||
<pre> | <pre> | ||
export LESS_TERMCAP_mb=$'\e[1;32m' | # Customize less colors | ||
export LESS_TERMCAP_md=$'\e[1; | export LESS_TERMCAP_mb=$'\e[1;32m' # Blinking text: bold green | ||
export LESS_TERMCAP_me=$'\e[0m' | export LESS_TERMCAP_md=$'\e[1;34m' # Bold text: bold blue | ||
export LESS_TERMCAP_me=$'\e[0m' # End mode | |||
export LESS_TERMCAP_so=$'\e[01; | export LESS_TERMCAP_so=$'\e[01;47;34m' # Standout: bold white on blue | ||
export | export LESS_TERMCAP_se=$'\e[0m' # End standout mode | ||
export LESS_TERMCAP_us=$'\e[1;4;31m' | export LESS_TERMCAP_us=$'\e[1;4;31m' # Underlined text: bold underlined red | ||
export LESS_TERMCAP_ue=$'\e[0m' # End underline | |||
</pre> | </pre> | ||
| Line 248: | Line 249: | ||
== Free up memory == | == Free up memory == | ||
[https://www.makeuseof.com/improve-performance-free-up-ram-on-linux/ How to Free Up Memory and Improve RAM Performance on Linux] | * [https://www.makeuseof.com/improve-performance-free-up-ram-on-linux/ How to Free Up Memory and Improve RAM Performance on Linux] | ||
* [https://linuxtldr.com/clear-buffer-cache-memory-linux/ How to Clear Buffer and Cache Memory in Linux] | |||
== Monitor Memory Utilization And Send an Email == | == Monitor Memory Utilization And Send an Email == | ||
| Line 324: | Line 326: | ||
On raspbian commenting out some lines does help. Interestingly, the same lines does not make any difference on x86 server. | On raspbian commenting out some lines does help. Interestingly, the same lines does not make any difference on x86 server. | ||
== the source command == | |||
The . is a shorthand for the '''source''' command in bash. <syntaxhighlight lang="sh" inline>source ~/.bashrc</syntaxhighlight> and <syntaxhighlight lang="sh" inline>. ~/.bashrc</syntaxhighlight> are equivalent. | |||
== login shell (.bash_profile) vs interactive shell (.bashrc) == | == login shell (.bash_profile) vs interactive shell (.bashrc) == | ||
* http://stackoverflow.com/questions/18186929/differences-between-login-shell-and-interactive-shell | * http://stackoverflow.com/questions/18186929/differences-between-login-shell-and-interactive-shell | ||
* [https://linuxtldr.com/interactive-login-and-non-login-shell/ What is Interactive Login and Non-Login Shell] | |||
* http://serverfault.com/questions/8882/what-is-the-difference-between-a-login-and-an-interactive-bash-shell | * http://serverfault.com/questions/8882/what-is-the-difference-between-a-login-and-an-interactive-bash-shell | ||
| Line 375: | Line 381: | ||
[https://www.howtogeek.com/427004/how-to-use-the-pinky-command-on-linux/ How to Use the pinky Command on Linux] | [https://www.howtogeek.com/427004/how-to-use-the-pinky-command-on-linux/ How to Use the pinky Command on Linux] | ||
== Switch user | == su: Switch to another user and run a command == | ||
<ul> | |||
<li>su means 'substitute user'. | |||
<li>Use | |||
<pre> | |||
su newusername # keeps the environment variables of the original user | |||
su - newusername # clears all the environment variables | |||
</pre> | |||
<li>[https://www.cyberciti.biz/open-source/command-line-hacks/linux-run-command-as-different-user/ Linux Run Command As Another User] | |||
'''runuser -l''' command | '''runuser -l''' command | ||
{{Pre}} | {{Pre}} | ||
runuser -l userNameHere -c 'command' | runuser -l userNameHere -c 'command' | ||
</pre> | </pre> | ||
</ul> | |||
== Keyboard shortcut to move cursor in the terminal == | == Keyboard shortcut to move cursor in the terminal == | ||
| Line 554: | Line 467: | ||
== Recording your terminal: asciinema == | == Recording your terminal: asciinema == | ||
https://asciinema.org/ | https://asciinema.org/ | ||
== Dictionary/translation == | |||
<ul> | |||
<li>Dictionary. [https://www.linuxtechi.com/linux-curl-command-examples-part-1/#Example_6_Lookup_word-meaning_using_DICT_protocol_with_curl_command Lookup word-meaning using DICT protocol with curl command] | |||
<pre> | |||
$ curl dict://dict.org/d:word | |||
</pre> | |||
The output of the above method is too complicated. The 'trans' utility provides a better output (even still no colors) | |||
<pre> | |||
$ trans -d word | |||
</pre> | |||
<li>[https://github.com/soimort/translate-shell Translation shell] | |||
<pre> | |||
$ wget git.io/trans | |||
$ chmod +x ./trans | |||
$ sudo apt update; sudo apt install gawk | |||
$ sudo mv trans /usr/local/bin/ | |||
$ trans :zh-TW "interrogate" | |||
interrogate | |||
/inˈterəˌɡāt/ | |||
審問 | |||
(Shěnwèn)p | |||
</pre> | |||
</ul> | |||
== Clear screen == | == Clear screen == | ||
| Line 619: | Line 557: | ||
== List only directories == | == List only directories == | ||
<pre> | <pre> | ||
ls -d */ | ls -d */ # current directory | ||
ls -ld ~/Downloads/*/ # ~/Downloads | |||
ls -l -d */ | ls -l -d */ | ||
</pre> | </pre> | ||
| Line 641: | Line 580: | ||
Opened MS-Office documents' filenames start with "~$". <u>The dollar sign character has to be escaped</u>; eg ''' ls -l ~\$* ''' to list these kind of files or '''rm ~\$*''' to delete these files. | Opened MS-Office documents' filenames start with "~$". <u>The dollar sign character has to be escaped</u>; eg ''' ls -l ~\$* ''' to list these kind of files or '''rm ~\$*''' to delete these files. | ||
== Check non-English characters == | |||
[https://www.baeldung.com/linux/find-non-ascii-chars How to Find Non-ASCII Characters in Text Files in Linux] | |||
<pre> | |||
perl -ne 'print if /[^[:ascii:]]/' sample.txt | |||
</pre> | |||
== ls | more without lose color == | == ls | more without lose color == | ||
| Line 651: | Line 596: | ||
== ls directories color == | == ls directories color == | ||
[https://www.howtogeek.com/307899/how-to-change-the-colors-of-directories-and-files-in-the-ls-command/ How to Change the Colors of Directories and Files in the ls Command]. No need to use the export command when we want to add it to .bashrc file. | [https://www.howtogeek.com/307899/how-to-change-the-colors-of-directories-and-files-in-the-ls-command/ How to Change the Colors of Directories and Files in the ls Command]. No need to use the export command when we want to add it to .bashrc file. | ||
<pre> | |||
LS_COLORS= | # orange color | ||
export LS_COLORS=$LS_COLORS:'di=0;33:' | |||
# OR yellow color if your terminal supports 256 colors | |||
export LS_COLORS=$LS_COLORS:'di=38;5;226:' | |||
</pre> | </pre> | ||
| Line 705: | Line 653: | ||
= tree command = | = tree command = | ||
'''tree -d''': show directories only | '''tree -d''': show directories only | ||
== tre command == | |||
* [https://github.com/dduan/tre github]. Binary files for x86 and arm linux are available. To use it, run "tre DIRECTORYNAME" or "tre" to list files recursively from the current directory. | |||
= cp command = | = cp command = | ||
| Line 835: | Line 786: | ||
# delete all hidden files | # delete all hidden files | ||
find . -name ".*" -type f -delete | find . -name ".*" -type f -delete | ||
find . -name ".DS_Store" -delete | |||
</pre> | </pre> | ||
== Create a new directory and cd to it == | == Create a new directory and cd to it == | ||
[http://www.howtogeek.com/273304/how-to-make-a-new-directory-and-change-to-it-with-a-single-command-in-linux/ How to Make a New Directory and Change to It with a Single Command in Linux] | [http://www.howtogeek.com/273304/how-to-make-a-new-directory-and-change-to-it-with-a-single-command-in-linux/ How to Make a New Directory and Change to It with a Single Command in Linux] | ||
== Set permissions for new files or directories == | |||
* umask: user file-creation mode mask | |||
* With '''umask 007''', ''newly created files and directories will have all permissions (read, write, and execute) for the user and the group, but no permissions for others''. This is a good practice when you want to share data with other users in the same group but want to completely exclude users who are not group members. | |||
* Set up default umask on Linux: adding the command ''umask 007'' to the '''.bashrc''' file sets the default umask value for your shell. This means that every time you start a new shell session, the umask will be set to 007. | |||
* How it works: | |||
** The default permissions for files are 666 (read and write for owner, group, and others) and for directories are 777 (read, write, and execute for owner, group, and others). | |||
** The default permissions for files are 666 and for directories are 777. In binary, these are 110 110 110 and 111 111 111 respectively. The umask value 007 is 000 000 111 in binary. | |||
** The AND operation is performed between the binary representations of the default permissions and the bitwise '''NOT of the umask value'''. That is, the mask is '''negated''' (its bitwise compliment is taken) and this value is then applied to the default permissions using a logical AND operation. | |||
** The result of the AND operation is 110 110 000 for files and 111 111 000 for directories. In decimal, these are 660 and 770 respectively. | |||
* See [https://www.cyberciti.biz/tips/understanding-linux-unix-umask-value-usage.html What is Umask and How To Setup Default umask Under Linux?], [https://www.computerhope.com/unix/uumask.htm Linux umask command]. | |||
== chown and chmod recursively (-R) == | == chown and chmod recursively (-R) == | ||
| Line 910: | Line 874: | ||
$ rm -- -foo.txt | $ rm -- -foo.txt | ||
</pre> | </pre> | ||
== Inodes == | |||
* https://en.wikipedia.org/wiki/Inode | |||
* [https://www.stackscale.com/blog/inodes-linux/ Inodes in Linux: limit, usage and helpful commands] | |||
* [https://linuxhandbook.com/inode-linux/ Everything You Need to Know About inodes in Linux] | |||
== Recover Deleted Files == | == Recover Deleted Files == | ||
| Line 929: | Line 898: | ||
$ alias checkport='sudo lsof -i -P -n | grep LISTEN' | $ alias checkport='sudo lsof -i -P -n | grep LISTEN' | ||
</pre> | </pre> | ||
'''NOTE''' the last column of the output from "checkport" will show '''who can connection to the port'''. For example, if it shows "127.0.0.1:10999 (LISTEN)", it means only the server can connect to port 10999. If it shows "*:22 (LISTEN)", it means any machines can connect to the server through port 22. | |||
To avoid using the alias, use one of the following ways (eg use the command's full path) | To avoid using the alias, use one of the following ways (eg use the command's full path) | ||
| Line 981: | Line 951: | ||
= diff = | = diff = | ||
[https://www.linuxlinks.com/best-free-console-based-diff-tools/ 15 Best Free and Open Source Terminal-Based Diff Tools] | |||
== Run diff with large files == | == Run diff with large files == | ||
diff (or even better the cmp command) works fine with 8G fastq files. Note cmp compare files byte by byte so it probably won't run out of memory. [https://superuser.com/a/690732 How to diff large files on Linux]. | diff (or even better the cmp command) works fine with 8G fastq files. Note cmp compare files byte by byte so it probably won't run out of memory. [https://superuser.com/a/690732 How to diff large files on Linux]. | ||
| Line 1,207: | Line 1,179: | ||
See [[Linux_Programming#xargs|Linux Programming]] | See [[Linux_Programming#xargs|Linux Programming]] | ||
== -exec COMMAND {} | == -exec COMMAND {} \; == | ||
* [https://unix.stackexchange.com/a/12904 How to run find -exec?], [https://stackoverflow.com/a/6085237 Using semicolon (;) vs plus (+) with exec in find] | <ul> | ||
<li>Search and list files with details | |||
<syntaxhighlight lang='sh'> | |||
find . -type f -name "*street*" -exec ls -lh {} \; | |||
</syntaxhighlight> | |||
'''-exec ls -lh {} \;''' executes the '''ls -lh''' command on each found file. The '''{}''' is a placeholder for the current file, and '''\;''' indicates the end of the -exec command. | |||
<li>[https://unix.stackexchange.com/a/12904 How to run find -exec?], [https://stackoverflow.com/a/6085237 Using semicolon (;) vs plus (+) with exec in find] | |||
{{Pre}} | {{Pre}} | ||
# This command will run grep chrome on each file individually. | |||
find . -exec grep chrome {} \; | find . -exec grep chrome {} \; | ||
# | |||
# This command will run grep chrome on batches of files, reducing the number of times the command is executed. | |||
# Using + can significantly improve performance | |||
find . -exec grep chrome {} + | find . -exec grep chrome {} + | ||
</pre> | </pre> | ||
* Find files and execute something (google: find --exec) | Note: | ||
* The backslash before ; is to '''escape''' ; so linux won't interpret it directly. | |||
* Command Compatibility: Not all commands support being executed with multiple arguments at once. For example, grep can handle multiple files, but some other commands might not. In other words, while '''-exec COMMAND {} +''' can be more efficient, it’s important to ensure the command you’re using supports this usage and behaves as expected. | |||
<li>Find files and execute something (google: find --exec) | |||
{{Pre}} | {{Pre}} | ||
$ find ./ -name "*.tar.gz" -exec tar zxvf {} \; | $ find ./ -name "*.tar.gz" -exec tar zxvf {} \; | ||
</pre> | </pre> | ||
<li>Find and move files to a new directory | |||
{{Pre}} | {{Pre}} | ||
find OLDDIR -type f -exec mv -t NEWDIR {} + | find OLDDIR -type f -exec mv -t NEWDIR {} + | ||
</pre> | </pre> | ||
<li>[http://unix.stackexchange.com/questions/41550/find-the-total-size-of-certain-files-within-a-directory-branch Find the total file size of a list of files]. | |||
<li>[https://unix.stackexchange.com/a/196917 What is meaning of {} + in find's -exec command?] | |||
<li>[http://unix.stackexchange.com/questions/179346/why-does-find-exec-cmd-need-to-end-in?rq=1 Why does 'find -exec cmd {} +' need to end in '{} +'?] | |||
<li>[http://unix.stackexchange.com/questions/12902/how-to-run-find-exec How to run find -exec?] | |||
The following will find out the total file size of the 'accepted_hits.bam' file under all sub-directories. | The following will find out the total file size of the 'accepted_hits.bam' file under all sub-directories. | ||
{{Pre}} | {{Pre}} | ||
| Line 1,232: | Line 1,222: | ||
</pre> | </pre> | ||
where '-c' produces a grand total, and will substitute {} with the filename(s) found in '''-exec'''. | where '-c' produces a grand total, and will substitute {} with the filename(s) found in '''-exec'''. | ||
</ul> | |||
== How to find and delete directory recursively == | == How to find and delete directory recursively == | ||
| Line 1,352: | Line 1,343: | ||
Use grep '''--color=always''' .For example, grep --color=always KEYWORD Myfile | more. | Use grep '''--color=always''' .For example, grep --color=always KEYWORD Myfile | more. | ||
== Include/follow soft/symbolic link files == | |||
grep '''grep -Rl''' instead of "grep -rl". Recall "-l" is to list filename only. See "man grep". | |||
== Compressed files == | == Compressed files == | ||
| Line 1,963: | Line 1,957: | ||
* [https://linuxhandbook.com/at-command/ Schedule Jobs in Linux With 'at' Command] | * [https://linuxhandbook.com/at-command/ Schedule Jobs in Linux With 'at' Command] | ||
* [https://opensource.com/article/21/7/alternatives-cron-linux?utm_source=pocket_mylist 4 alternatives to cron in Linux] | * [https://opensource.com/article/21/7/alternatives-cron-linux?utm_source=pocket_mylist 4 alternatives to cron in Linux] | ||
* On Debian, I need to run '''sudo apt install at'''. | |||
* [https://www.networkworld.com/article/972036/using-the-at-command-to-schedule-tasks-on-linux.html Using the at command to schedule tasks on Linux] | |||
<pre> | <pre> | ||
$ echo "rsync -av /home/tux/ me@myserver:/home/tux/" | at 1:30 AM | $ echo "rsync -av /home/tux/ me@myserver:/home/tux/" | at 1:30 AM | ||
$ echo "command_to_be_run" | at 09:00 | |||
$ atq # list of jobs | |||
$ atrm 6 # delete the 6th job | |||
</pre> | </pre> | ||
= Cron job by root = | |||
Note that there is a "user" field for cron jobs defined in /etc/crontab or /etc/cron.d. | |||
<ul> | |||
<li>Place one: '''/etc/crontab''' and '''/etc/cron.daily, /etc/cron.hourly, /etc/cron.monthly, /etc/cron.weekly'''. | |||
<pre> | |||
cat /etc/crontab | |||
# How to List Daily Cron Jobs | |||
ls -la /etc/cron.daily | |||
</pre> | |||
<li>Place two: '''/etc/cron.d'''. It contains "anacron" and "e2scrub_all" on my Debian 11. See [https://superuser.com/a/170870 How to run a cron job as a specific user?] | |||
</ul> | |||
= Running a cron job as a user = | = Running a cron job as a user = | ||
| Line 1,971: | Line 1,986: | ||
* http://www.thegeekstuff.com/2009/06/15-practical-crontab-examples/ | * http://www.thegeekstuff.com/2009/06/15-practical-crontab-examples/ | ||
* https://crontab.guru/examples.html and [https://crontab.guru/tips.html tips] | * https://crontab.guru/examples.html and [https://crontab.guru/tips.html tips] | ||
* Can I break a line cron command into multiple lines? No. Creating another script file in this case. | |||
* A specific time | * A specific time | ||
{{Pre}} | |||
MIN HOUR DOM MON DOW CMD | MIN HOUR DOM MON DOW CMD | ||
30 08 10 06 * | 30 08 10 06 * $HOME/full-backup >> $HOME/myscript.log 2>&1; echo "Executed at $(date)\n----------" >> $HOME | ||
/myscript.log | |||
# 30 – 30th Minute | # 30 – 30th Minute | ||
# 08 – 08 AM | # 08 – 08 AM | ||
| Line 2,106: | Line 2,123: | ||
* [https://www.tecmint.com/fsck-repair-file-system-errors-in-linux/ How to Use ‘fsck’ to Repair File System Errors in Linux] | * [https://www.tecmint.com/fsck-repair-file-system-errors-in-linux/ How to Use ‘fsck’ to Repair File System Errors in Linux] | ||
* [https://www.maketecheasier.com/repair-mac-hard-disk-with-fsck/ How to Repair Hard Disks with fsck on macOS] | * [https://www.maketecheasier.com/repair-mac-hard-disk-with-fsck/ How to Repair Hard Disks with fsck on macOS] | ||
* [https://ostechnix.com/fsck-command-examples/ How To Use Fsck Command To Check And Repair Linux File Systems] (2024 Update) | |||
== Fsck error on boot == | == Fsck error on boot == | ||
| Line 2,131: | Line 2,149: | ||
= Swap partition = | = Swap partition = | ||
<ul> | |||
<li>[https://wiki.archlinux.org/title/Swap Swap partition] and swap file | |||
<syntaxhighlight lang='sh'> | |||
# mkswap /dev/sdxy | |||
# swapon /dev/sdxy | |||
# nano /etc/fstab | |||
</syntaxhighlight> | |||
'''UUID=device_UUID none swap defaults 0 0 ''' | |||
<li>[https://www.howtogeek.com/455981/how-to-create-a-swap-file-on-linux/ How to Create a Swap File on Linux] | |||
<li>[https://www.tecmint.com/disable-swap-partition-in-centos-ubuntu/ How to Permanently Disable Swap in Linux] | |||
</ul> | |||
== Swap file vs swap partition == | == Swap file vs swap partition == | ||
[https://www.jetsonhacks.com/2019/04/25/jetson-nano-run-on-usb-drive/ Jetson Nano – Run on USB Drive] | * [https://www.jetsonhacks.com/2019/04/25/jetson-nano-run-on-usb-drive/ Jetson Nano – Run on USB Drive] | ||
* If "htop" or "free -h" shows swap but "Disks" does not show a swap partition, it is likely the system is using a swap file instead of a dedicated swap partition. Use '''sudo swapon --show''' to check for the presence of a swap file or check '''/etc/fstab''' to see the a line referencing a file rather than a partition. | |||
= Mount drive, add a new hard drive = | = Mount drive, add a new hard drive = | ||
| Line 2,208: | Line 2,236: | ||
== Sharing files with Windows by using NitroShare == | == Sharing files with Windows by using NitroShare == | ||
[http://www.ubuntugeek.com/simple-way-of-sharing-files-between-ubuntu-16-04-and-windows-10.html Simple way of Sharing files between Ubuntu 16.04 and Windows 10] by using open-source [https://nitroshare.net/ NitroShare] which is based on Qt framework. | [http://www.ubuntugeek.com/simple-way-of-sharing-files-between-ubuntu-16-04-and-windows-10.html Simple way of Sharing files between Ubuntu 16.04 and Windows 10] by using open-source [https://nitroshare.net/ NitroShare] which is based on Qt framework. | ||
== NTFS usb drive in xubuntu == | == NTFS usb drive in xubuntu == | ||
| Line 2,280: | Line 2,307: | ||
ps -p 3403 # Listing Processes by Process ID | ps -p 3403 # Listing Processes by Process ID | ||
ps -Flww -p THE_PID # Listing a process detail like "htop" gives | |||
ps -u mary # Listing Processes Owned by a User | ps -u mary # Listing Processes Owned by a User | ||
| Line 2,315: | Line 2,344: | ||
* [http://www.2daygeek.com/kill-inactive-idle-ssh-sessions/ How To kill An Inactive OR Idle SSH Sessions]. The '''pstree -p''' command can show a tree diagram of all the processes. | * [http://www.2daygeek.com/kill-inactive-idle-ssh-sessions/ How To kill An Inactive OR Idle SSH Sessions]. The '''pstree -p''' command can show a tree diagram of all the processes. | ||
* [https://www.howtoforge.com/linux-pkill-command/ pkill] command. For example, ''pkill gedit''. | * [https://www.howtoforge.com/linux-pkill-command/ pkill] command. For example, ''pkill gedit''. | ||
== Kill a process running on a specific port == | |||
[https://linuxtldr.com/kill-a-process-running-on-a-specific-port-in-linux/ Kill a Process Running on a Specific Port in Linux (via 4 Methods)] | |||
== How to Kill Zombie Processes on Linux == | == How to Kill Zombie Processes on Linux == | ||
| Line 2,366: | Line 2,398: | ||
$ kill $(pidof firefox) | $ kill $(pidof firefox) | ||
</pre> | </pre> | ||
== How to keep resource usage low == | |||
* [https://www.howtogeek.com/habits-to-keep-linux-resource-usage-low/ Habits to Keep Linux Resource Usage Low] | |||
== Avoid concurrency == | |||
[https://www.howtogeek.com/limit-bash-scripts-to-run-once-at-a-time/ Avoid These Problems By Limiting Bash Scripts to Run Once At A Time]. '''pgrep, lsof''' and '''flock''' commands. | |||
== All You Need To Know About Processes in Linux == | == All You Need To Know About Processes in Linux == | ||
http://www.tecmint.com/linux-process-management/ | http://www.tecmint.com/linux-process-management/ | ||
== wait command and background jobs == | |||
The '''wait''' command in Linux is a shell built-in command that pauses the execution of a shell script until all background jobs or specified JobID/PIDs terminate and return their exit status. | |||
<pre> | |||
# Example 1: Wait for all background processes to finish | |||
command1 & | |||
command2 & | |||
wait | |||
echo "All background processes have finished." | |||
# Example 2: Wait for a specific process to finish | |||
command1 & | |||
PID=$! | |||
command2 & | |||
wait $PID | |||
echo "Command1 has finished." | |||
</pre> | |||
== run commands in a background and allow log off == | == run commands in a background and allow log off == | ||
| Line 2,516: | Line 2,571: | ||
* /boot | * /boot | ||
* /dev | * /dev | ||
* /etc - configuration files | * '''/etc''' - configuration files | ||
* /media | * /media | ||
* /mnt | * /mnt | ||
| Line 2,522: | Line 2,577: | ||
* /proc - process information only. [https://levelup.gitconnected.com/access-kubernetes-objects-data-from-proc-directory-8d2ec6a0faba Access Kubernetes Objects Data From /Proc Directory] | * /proc - process information only. [https://levelup.gitconnected.com/access-kubernetes-objects-data-from-proc-directory-8d2ec6a0faba Access Kubernetes Objects Data From /Proc Directory] | ||
* /sbin - critical executables for running the system, but should be used by superuser | * /sbin - critical executables for running the system, but should be used by superuser | ||
* /usr - | * '''/usr''' - user programs and data. For example /usr/bin contains most of the libraries used by apps. [https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s11.html /usr/share/] contains Architecture-independent data (eg some pretty images are located in /usr/share/backgrounds) | ||
* /var - variable data such as databases, mails spools and system logs. | * '''/var''' - variable data such as '''databases''', mails spools and system '''logs'''. | ||
= Difference of /bin, /sbin, /usr/local/bin, ... = | = Difference of /bin, /sbin, /usr/local/bin, ... = | ||
| Line 2,614: | Line 2,669: | ||
== nmon == | == nmon == | ||
* https://en.wikipedia.org/wiki/Nmon. The stats are saved to a comma-separated values (CSV) data file for later graphing and analysis | * https://en.wikipedia.org/wiki/Nmon. The stats are saved to a comma-separated values (CSV) data file for later graphing and analysis | ||
* [https://nmon.sourceforge.io/pmwiki.php HomePage] | |||
* [https://www.admin-magazine.com/HPC/Articles/Nmon-All-Purpose-Admin-Tool Monitoring with Nmon] | * [https://www.admin-magazine.com/HPC/Articles/Nmon-All-Purpose-Admin-Tool Monitoring with Nmon] | ||
== CPU frequency == | == CPU frequency == | ||
[https://ostechnix.com/optimize-performance-and-battery-life-with-auto-cpufreq/ How To Optimize Performance And Battery Life With Auto-cpufreq In Linux] | |||
[https://unix.stackexchange.com/a/265611 What is the correct way to view your CPU speed on Linux?] (x86) | [https://unix.stackexchange.com/a/265611 What is the correct way to view your CPU speed on Linux?] (x86) | ||
<pre> | <pre> | ||
| Line 2,706: | Line 2,763: | ||
=== Bashtop and btop === | === Bashtop and btop === | ||
* [https://www.linuxuprising.com/2020/04/bashtop-is-cool-linux-resource-monitor.html Bashtop Is A Cool Linux Resource Monitor Written In Bash] | * [https://www.linuxuprising.com/2020/04/bashtop-is-cool-linux-resource-monitor.html Bashtop Is A Cool Linux Resource Monitor Written In Bash] | ||
* [https://github.com/aristocratos/btop *btop]. C++ version and continuation of bashtop and bpytop. This is very similar to bashtop but it also shows the IP. Mac, Linux x86, armv7, aarch64, arm64, armv5. [https://lindevs.com/install-btop-on-raspberry-pi/ Install btop++ on Raspberry Pi], [https://lindevs.com/install-btop-on-ubuntu/ Ubuntu 22.04]. | * [https://github.com/aristocratos/btop *btop] 20k star. C++ version and continuation of bashtop and bpytop. This is very similar to bashtop but it also shows the IP. Mac, Linux x86, armv7, aarch64, arm64, armv5. [https://lindevs.com/install-btop-on-raspberry-pi/ Install btop++ on Raspberry Pi], [https://lindevs.com/install-btop-on-ubuntu/ Ubuntu 22.04]. | ||
=== bottom === | |||
* https://github.com/ClementTsang/bottom 10k star | |||
== S-tui command == | == S-tui command == | ||
| Line 2,740: | Line 2,800: | ||
== [https://www.zabbix.com/ Zabbix] == | == [https://www.zabbix.com/ Zabbix] == | ||
[https://www.tecmint.com/install-zabbix-on-debian-10/ How to Install Zabbix on Debian 10] | * [https://www.tecmint.com/install-zabbix-on-debian-10/ How to Install Zabbix on Debian 10] | ||
* [https://forum.proxmox.com/threads/smart-error-health-detected-on-host.109580/ SMART error (Health) detected on host] | |||
* [https://www.zabbix.com/integrations/smart Zabbix + S.M.A.R.T.] | |||
== Munin and Monit == | == Munin and Monit == | ||
| Line 3,022: | Line 3,084: | ||
== export DISPLAY == | == export DISPLAY == | ||
If we want to run a GUI app on a remote computer (such as Raspberry Pi/Beaglebone Black) and show the GUI app on the remote computer's screen using ssh, we can issue the following command before running the app. | <ul> | ||
<li>[https://askubuntu.com/a/432257 What is the $DISPLAY environment variable?] | |||
The value of the display environment variable is: | |||
<pre> | |||
hostname:displaynumber.screennumber | |||
</pre> | |||
<li>If we want to run a GUI app on a remote computer (such as Raspberry Pi/Beaglebone Black) and show the GUI app on the remote computer's screen using ssh, we can issue the following command before running the app. | |||
{{Pre}} | {{Pre}} | ||
export DISPLAY=:0.0 | export DISPLAY=:0.0 | ||
</pre> | </pre> | ||
</ul> | |||
= See which groups you belong to, id & group commands = | = See which groups you belong to, id & group commands = | ||
<pre> | <pre> | ||
id <username> | id <username> | ||
| Line 3,037: | Line 3,104: | ||
== Main Types of User Accounts on Linux == | == Main Types of User Accounts on Linux == | ||
[https://www.makeuseof.com/types-of-user-accounts-on-linux/ The 4 Main Types of User Accounts on Linux] | [https://www.makeuseof.com/types-of-user-accounts-on-linux/ The 4 Main Types of User Accounts on Linux]: | ||
* Superuser Account | |||
* Regular Accounts | |||
* System Accounts (Core OS functionality, eg root, bin, sys, daemon) '''awk -F: '$3 <= 100 {print $1, $3, $7}' /etc/passwd''' Note that the third field is the user ID (UID). System accounts generally have UIDs below 100 (though this threshold may vary slightly depending on your distribution). | |||
* Service Accounts (eg SQL, apache). | |||
== List all user groups == | |||
[https://www.makeuseof.com/how-to-list-all-user-groups-on-linux/ How to List All User Groups on Linux] | |||
To see a list of groups | |||
<pre> | |||
cat /etc/group | |||
# OR | |||
getent group | |||
</pre> | |||
Note that "getenv group" Lists groups from all configured sources, including both local and networked sources, depending on your system’s NSS (Name Service Switch) configuration (in /etc/nsswitch.conf). | |||
== finger: show user information == | == finger: show user information == | ||
finger USERNAME | finger USERNAME | ||
= groupadd, chgrp, usermod, ACL (access control lists) = | = groupadd, chgrp, usermod, ACL (access control lists) = | ||
| Line 3,123: | Line 3,201: | ||
== Logrotate == | == Logrotate == | ||
[https://www.cloudsavvyit.com/176/how-to-setup-logrotate-on-linux-to-keep-your-server-from-running-out-of-space/ How to Setup Logrotate on Linux (to Keep Your Server from Running Out of Space)] | [https://www.cloudsavvyit.com/176/how-to-setup-logrotate-on-linux-to-keep-your-server-from-running-out-of-space/ How to Setup Logrotate on Linux (to Keep Your Server from Running Out of Space)] | ||
= uprecords command = | |||
* [https://linuxtldr.com/uprecords-command/ Check the Linux Uptime History Using the Uprecords] | |||
* [https://linuxhandbook.com/uptime-command/ How to Check Uptime of Your Linux Server] | |||
= '''uptime''' command = | = '''uptime''' command = | ||
| Line 3,227: | Line 3,309: | ||
* https://www.raspberrypi.org/documentation/remote-access/ip-address.md | * https://www.raspberrypi.org/documentation/remote-access/ip-address.md | ||
== Private network == | == Private/internal/local IP/network == | ||
https://en.wikipedia.org/wiki/IPv4#Private_networks | https://en.wikipedia.org/wiki/IPv4#Private_networks | ||
| Line 3,233: | Line 3,315: | ||
* 172.16.0.0 – 172.31.255.255 | * 172.16.0.0 – 172.31.255.255 | ||
* 192.168.0.0 – 192.168.255.255 | * 192.168.0.0 – 192.168.255.255 | ||
There are other special-use addresses for private networks | |||
* 100.64.0.0 to 100.127.255.255 (similar to private IP addresses like 192.168.x.x or 10.x.x.x) | |||
= Get external IP address = | = Get external IP address = | ||
* http://ifconfig.me | |||
* Simple IPv4 | |||
** http://checkip.amazonaws.com, | |||
** https://ipecho.net/plain, | |||
[https://askubuntu.com/a/145017 Command for determining my public IP?] '''wget -qO- https://ipecho.net/plain ; echo ''' | ** https://ipinfo.io/ip, | ||
** https://api.ipify.org/ | |||
[https://www.2daygeek.com/linux-command-find-check-domain-ip-address/ 5 Commands to Find the IP Address of a Domain in the Linux Terminal] | * [https://askubuntu.com/a/145017 Command for determining my public IP?] '''wget -qO- https://ipecho.net/plain ; echo ''' | ||
* [https://opensource.com/article/18/5/how-find-ip-address-linux How to find your IP address in Linux] | |||
* [https://www.2daygeek.com/linux-command-find-check-domain-ip-address/ 5 Commands to Find the IP Address of a Domain in the Linux Terminal] | |||
* [https://www.makeuseof.com/get-public-ip-address-in-linux/ How to Find the Public IP Address on a Linux System] | |||
[https://www.makeuseof.com/get-public-ip-address-in-linux/ How to Find the Public IP Address on a Linux System] | |||
It seems there is no way to get the external IP address without not using external services. | It seems there is no way to get the external IP address without not using external services. | ||
| Line 3,490: | Line 3,574: | ||
* [https://opensource.com/article/20/7/nmcli Manage network connections from the Linux command line with nmcli] | * [https://opensource.com/article/20/7/nmcli Manage network connections from the Linux command line with nmcli] | ||
* Wifi | * Wifi | ||
<ul> | |||
<li> [https://kifarunix.com/connect-to-wifi-in-linux-using-nmcli-command/ Connect to WiFi in Linux Using NMCLI command] | |||
<pre> | |||
nmcli radio wifi on | |||
nmcli dev status | |||
nmcli dev wifi list # Show signal/bar/rate | |||
sudo nmcli dev wifi rescan | |||
sudo nmcli --ask dev wifi connect MySSID | |||
</pre> | |||
<li>[https://www.tecmint.com/nmcli-connect-wi-fi-from-linux-terminal/ How to Connect Wi-Fi from Linux Terminal Using Nmcli Command] | |||
{{Pre}} | {{Pre}} | ||
# Find the up network adaptors | # Find the up network adaptors | ||
| Line 3,503: | Line 3,596: | ||
$ nmcli dev show | grep 'IP4.DNS' | $ nmcli dev show | grep 'IP4.DNS' | ||
</pre> | </pre> | ||
</ul> | |||
== iwconfig == | == iwconfig == | ||
| Line 3,714: | Line 3,808: | ||
|- | |- | ||
| ss | | ss | ||
| | | sudo ss -tulpn | grep LISTEN | ||
sudo ss -tulpn | grep LISTEN | | contain '''pid''', <br /> | ||
| contain pid, <br /> | add "sudo" can show program/command names on the last column, <br /> | ||
add "sudo" can show program/command names on the last column | ''' ''Even DietPi includes the ss command'' ''' | ||
|- | |- | ||
| lsof | | lsof | ||
| sudo lsof -i -P -n | egrep "PID|LISTEN" | | sudo lsof -i -P -n | egrep "PID|LISTEN" | ||
| '''sudo''' is required. Contain pid. Using '''egrep''' to include header in output. It includes the command/program names on the first column. | | '''sudo''' is required. Contain pid. Using '''egrep''' to include header in output. It includes the command/program names on the first column. <br /> | ||
lsof may not be available by default in many OS. | |||
|- | |- | ||
| '''nmap''' (network mapper) | | '''nmap''' (network mapper) | ||
| Line 3,736: | Line 3,831: | ||
| need to specify a port number. More accurate than '''nmap'''. "-w3" means time out after 3 seconds. | | need to specify a port number. More accurate than '''nmap'''. "-w3" means time out after 3 seconds. | ||
|} | |} | ||
== What are network interface ports == | |||
[https://www.howtoforge.com/understanding-network-interface-ports-in-linux Understanding Network Interface Ports in Linux]. In networking, the term "port" can refer to different concepts depending on the context: | |||
* Hardware Ports: | |||
* Software Ports: These are '''virtual endpoints''' in an operating system used by '''network protocols''' to manage '''connections'''. Software ports are identified by a number, ranging from 0 to 65535. For example, port 80 is used for HTTP, and port 443 is used for HTTPS. | |||
== Socket == | == Socket == | ||
| Line 3,853: | Line 3,953: | ||
<pre> | <pre> | ||
sudo apt install lsof # dietpi | |||
# Basic lsof Output | # Basic lsof Output | ||
sudo su | sudo su | ||
| Line 4,239: | Line 4,341: | ||
= Hardware information = | = Hardware information = | ||
[https://www.2daygeek.com/how-to-check-system-hardware-manufacturer-model-and-serial-number-in-linux/ How To Check System Hardware Manufacturer, Model And Serial Number In Linux] | * [https://www.2daygeek.com/how-to-check-system-hardware-manufacturer-model-and-serial-number-in-linux/ How To Check System Hardware Manufacturer, Model And Serial Number In Linux] | ||
* [https://www.howtogeek.com/426199/how-to-list-your-computers-devices-from-the-linux-terminal/ How to List Your Computer's Devices From the Linux Terminal]: mount, lsblk, df, fdisk, /proc, lspci, lsusb, lsdev, lshw, lsscsi, dmidecode, hwinfo. | |||
== Command Line == | == Command Line == | ||
| Line 4,372: | Line 4,475: | ||
|_| | |_| | ||
</pre> | </pre> | ||
= Install And Manage Software From Source = | |||
[https://ostechnix.com/install-and-manage-software-from-source-using-gnu-stow/ How To Install And Manage Software From Source Using GNU Stow In Linux] | |||
= Software that scan Malware and rootkits = | = Software that scan Malware and rootkits = | ||
| Line 4,449: | Line 4,555: | ||
== How to Run a Linux Program at Startup with systemd == | == How to Run a Linux Program at Startup with systemd == | ||
[https://www.howtogeek.com/687970/how-to-run-a-linux-program-at-startup-with-systemd/ How to Run a Linux Program at Startup with systemd], [https://www.makeuseof.com/create-systemd-service-on-linux/ How to Create a New systemd Service on Linux] | <ul> | ||
<li>[https://www.howtogeek.com/687970/how-to-run-a-linux-program-at-startup-with-systemd/ How to Run a Linux Program at Startup with systemd] | |||
* Creating the Service Program for systemd to Start '''sudo nano /usr/local/bin/htg.sh''', '''sudo chmod +x /usr/local/bin/htg.sh''' | |||
* Creating the Service Unit File '''sudo nano /etc/systemd/system/htg.service''', '''sudo chmod 640 /etc/systemd/system/htg.service''' | |||
* Starting the Service Automatically with the systemd Command '''sudo systemctl daemon-reload''', '''sudo systemctl enable htg''', '''sudo systemctl start htg''' | |||
* Verifying the Service '''sudo systemctl status htg.service''' | |||
* Stopping and Disabling the Service - '''sudo systemctl stop htg.service''' | |||
<li>[https://www.makeuseof.com/create-systemd-service-on-linux/ How to Create a New systemd Service on Linux] | |||
<li>[https://dmpop.xyz/article.php?id=2024-10-29_systemd-mount-dev Automatically mount a storage device with a systemd service] | |||
</ul> | |||
== How Long Does it Take To Boot Your Linux System == | == How Long Does it Take To Boot Your Linux System == | ||
| Line 4,458: | Line 4,573: | ||
$ systemd-analyze blame # breakdown the boot time into each unit | $ systemd-analyze blame # breakdown the boot time into each unit | ||
</pre> | </pre> | ||
== Check if Your Linux System Uses systemd == | |||
[https://itsfoss.com/check-if-systemd/ How to Check if Your Linux System Uses systemd] | |||
== chkservice == | == chkservice == | ||
| Line 4,466: | Line 4,584: | ||
* [https://www.ostechnix.com/different-ways-to-update-linux-kernel-for-ubuntu/ Different Ways To Update Linux Kernel For Ubuntu] | * [https://www.ostechnix.com/different-ways-to-update-linux-kernel-for-ubuntu/ Different Ways To Update Linux Kernel For Ubuntu] | ||
= | = Firmware update = | ||
[https://9to5linux.com/fwupd-1-9-9-released-with-support-for-lenovo-x1-yoga-gen7-530e-2-in-1-laptops Fwupd 1.9.9 Released with Support for Lenovo X1 Yoga Gen7 530E 2-in-1 Laptops] | |||
= Game = | = Game = | ||
Latest revision as of 17:02, 19 December 2024
man
- How to Use Linux’s man Command: Hidden Secrets and Basics
- How to Easily Read a Linux Man Page
- Underlined or Italicized Text: It means you need to replace it with an appropriate argument.
- Ellipses: It means that argument or expression is repeatable.
- Enter – Move down one line
- Space – Move down one page
- g – Move to the top of the page
- G – Move to the bottom of the page
- q – Quit
Search within a man page
Use / and type your search pattern.
Use 'n' for forward search and 'N' for reverse search.
The matched line will be moved to the top of the screen.
By default, the search is case insensitive or we can use man -i COMMAND.
Regular expression is supported. For example to find all of the long arguments with: /(--)[a-Z]
Colored man pages
This is a cool tip!
By default, the man program normally uses a terminal pager program such as less to format its output.
Add the following to "~/.bashrc" file. For example, LESS_TERMCAP_mb customizes the appearance of blink text in the less pager (or man command) and LESS_TERMCAP_us customizes the appearance of underlined text in the less pager.
# Customize less colors export LESS_TERMCAP_mb=$'\e[1;32m' # Blinking text: bold green export LESS_TERMCAP_md=$'\e[1;34m' # Bold text: bold blue export LESS_TERMCAP_me=$'\e[0m' # End mode export LESS_TERMCAP_so=$'\e[01;47;34m' # Standout: bold white on blue export LESS_TERMCAP_se=$'\e[0m' # End standout mode export LESS_TERMCAP_us=$'\e[1;4;31m' # Underlined text: bold underlined red export LESS_TERMCAP_ue=$'\e[0m' # End underline
(New way, The most Pager) How To Display Color Man Pages in Linux and Unix or How to Display man Pages in Color on Linux.
See here.
When inside the man page, press ! followed by a valid shell command.
For example : !man cat
View a specific "Section"
$ whatis printf printf (1) - format and print data printf (3) - formatted output conversion Printf (3o) - Formatted output functions. $ man 3 printf
Read man pages in vi without using temporary files
What is a way to read man pages in vim without using temporary files
man find | vi -
Search man page referenced by
man -f KEYWORD
This command is equivalent to whatis -r KEYWORD
Search from all man pages
man -k KEYWORD will give you a list of all man pages which relate to 'KEYWORD'.
TLDR pages/cheat sheet: alternative to Man
- https://tldr.sh/
- TLDR pages: Simplified Alternative To Linux Man Pages. Be sure to install the latest versions of nodejs and npm. Tested on Ubuntu 16.04, 20.04. Example:
$ npm install -g tldr $ tldr tar
- Display Linux Commands Cheatsheets With Tealdeer In Terminal A full list of commands is on TLDR Pages.
Cheat.sh (better than TLDR)
Cheat.sh Shows Cheat Sheets On The Command Line Or In Your Code Editor. There are different ways to use it. One way does not require to install anything as long as we have the curl command.
curl cheat.sh/tar curl cht.sh/python/random+list # Python programming language cheat sheet for random list
My test shows cheat.sh can find more commands and it gives colored output.
Some books
- UNIX in a nutshell
- sed & awk
- lex & yacc
- bash
- Linux Pocket Guide
- bash cookbook: pdf is online
- Classic Shell Scripting
- GNU EMACS
- Learning the vi and Vim editors 7th
- UNIX POWER TOOLS
- Bash Pocket Reference
- System Administration 3rd
- TCP/IP 3rd: pdf is online
- DNS and BIND 5th
- Network Troubleshooting Tools
- 25 Free Books To Learn Linux For Free
- 17 books for Linux and open source fans
Beautiful desktop
- http://lifehacker.com/the-aincrad-desktop-1732684767
- http://lifehacker.com/the-distant-pyramid-desktop-1654404411
- http://lifehacker.com/the-flat-n-fuzzy-desktop-1693121677
- http://lifehacker.com/the-midsummer-nights-desktop-1704207155
.desktop file
This is not related to beautiful desktop. It is used to launch applications in Linux. Without the .desktop file, your application won’t show up in the Applications menu and you can’t launch it with third-party launchers such as Synapse and Albert Launcher.
- https://wiki.archlinux.org/index.php/Desktop_entries
- https://www.maketecheasier.com/create-desktop-file-linux/
The .desktop files are commonly saved in
- ~/local/share/applications
- /usr/share/applications
List of installed desktop environment
ls -l /usr/share/xsessions/
Themes
5 of the Best Linux Dark Themes that Are Easy on the Eyes
Virtual consoles/virtual terminals
Linux allows virtual consoles (aka virtual terminals) to be opened while an X Window System is executing.
Use Ctrl + Alt + FX to open a virtual console-- there are six virtual text-based consoles (F1 to F6). Use Alt + F7 (or possibly other keybinds) to return to the X Window System.
Managing devices in Linux -> Fun with device files.
Change/increase console fonts
- https://www.linux.com/learn/intro-to-linux/2018/1/how-change-your-linux-console-fonts
- https://youtu.be/LOg4xfDQafc
Desktops/Workspaces
Ctrl + Alt + -> or Ctrl + Alt + <- to switch workspaces.
Ctrl + Alt + down can list the open applications on the current workspace.
Ctrl + Alt + up can show all workspaces and the open applications. We can use mouse to move an app to any workspace.
SuperKey + left tile a window to left. SuperKey + right tile a window to right.
Complete List of Linux Mint 18 Keyboard Shortcuts for Cinnamon for more examples.
Mouse
How To Bind Mouse Buttons To Keyboard Keys Or Commands (Linux Using X11)
Virtual memory
vmstat
- vmstat – A Standard Nifty Tool to Report Virtual Memory Statistics
- How to Use the vmstat Command on Linux
hcache
A tool fork from pcstat, with a feature that showing top X biggest cache files globally
Memory: free command
- Is Linux Eating Your RAM? How to Manage Your Memory
- How to Use the free Command on Linux (*detailed and comprehensive)
- How to Check Memory Usage From the Linux Terminal
Three types of memory reported by the free command.
- Used: RAM that is currently in use by an application.
- Available: RAM that may be in use for disk caching but can be freed up for applications. What is 'available' vs 'free' memory in free command?. Available = Estimation of how much memory is available for starting new applications, without swapping.
- Shared: Amount of memory used by the tmpfs file systems.
- Free: RAM that is not in use by an application or disk caching.
- Total = Used + Free + Buffers/Cache
How to Clear RAM Memory Cache and Buffer
# To clear pagecache, enter the following command: echo 1 > /proc/sys/vm/drop_caches # To clear dentries and inodes, change the number to 2: echo 2 > /proc/sys/vm/drop_caches # To clear pagecache, dentries, and inodes all together, change the number to 3: echo 3 > /proc/sys/vm/drop_caches
How do I determine the number of RAM slots in use
sudo dmidecode -t memory
Logging memory
- Linux system monitor tools where Sysstat package can be used.
- How Much Memory Is Installed and Being Used on Your Linux Systems?
- How to log the memory consumption on Linux?
- 8 Commands to Check Memory Usage on Linux
- 5 commands to check memory usage on Linux
- free -m, free -h, watch -n 10 free -m (free -s 10 gives an error 'seconds argument `10' failed)
- head -3 /proc/meminfo
- vmstat -s
- top
- htop
Shows Per-Program Memory Usage On Linux
- atop command. atop -m and press p (per program). Look at the 'RSIZE' column.
- How to view summaric memory usage of groups of commands (instead of processes). ps -C firefox --no-headers -o pmem. The result is one column. It shows the percentage not the actual memory. And it cannot catch some programs like "Web Content" or "WebExtensions" associated with Firefox.
- ps_mem: https://www.linuxuprising.com/2019/04/psmem-shows-per-program-memory-usage-on.html.
$ pip install ps_mem $ ps_mem -p $(pgrep -d, -u $USER) Private + Shared = RAM used Program ... 249.9 MiB + 43.9 MiB = 293.9 MiB firefox 549.7 MiB + 82.2 MiB = 631.9 MiB Web Content (6) 1.0 GiB + 149.6 MiB = 1.2 GiB chrome (16)
3.2 GiB
Check RAM information
- How to Use the pmap Command on Linux
- How to find the number of RAM Slots in Linux 2021
- 5 commands to check memory usage on Linux
sudo dmidecode -t memory sudo dmidecode -t 17
Free up memory
- How to Free Up Memory and Improve RAM Performance on Linux
- How to Clear Buffer and Cache Memory in Linux
Monitor Memory Utilization And Send an Email
- Bash Script to Monitor Memory Usage on Linux
- A Shell Script to Send Email Alert When Memory Gets Low. This uses $(free -mt | grep Total | awk '{print $4}') to get the available memory.
- Getting Started with Monit for Server Monitoring
zram
Things to do after a fresh install of GNU/Linux
- Run upgrade such as apt-get update; apt-get upgrade. It helps to resolve the unmet dependencies issue too.
- Increase audio quality
- Make sure firewall is enabled.
- Disable any unnecessary services
- Install Timeshift
- Install ClamAV / Clamtk antivirus
Query whether the OS is 64-bit or 32-bit
SYSTEM_ARCH=getconf LONG_BIT echo $SYSTEM_ARCH
Command line improved
https://remysharp.com/2018/08/23/cli-improved
Directory permission / attribute
See http://unix.stackexchange.com/questions/21251/how-do-directory-permissions-in-linux-work
When applying permissions to directories on Linux, the permission bits have different meanings than on regular files.
- The write bit allows the affected user to create, rename, or delete files within the directory, and modify the directory's attributes
- The read bit allows the affected user to list the files within the directory
- The execute bit allows the affected user to enter the directory, and access files and directories inside
When we create a new directory, the attribute is 775. Some pre-created directories (Desktop, Documents, Music, Pictures, Public) have an attribute 755.
Making a new temporary directory
https://www.howtoforge.com/linux-mktemp-command/
mktemp # temp directory is under /tmp mktemp -d tempdirXXX # temp directory is under the current directory mktemp tempfileXXX # temp file under the current directory
Shell
Login shell and non-login shell
Login Shell
- /etc/profile
- /etc/profile.d/*.sh
- ~/.bash_profile or ~/.profile (for example, environment variable like PATH)
- ~/.bashrc
- /etc/bashrc
- ~/.bashrc
Non-Login shell
- ~/.bashrc
- /etc/bash.bashrc or /etc/bashrc
- ~/.bashrc (bash-related settings, for example, prompt string, aliases)
Note: Bash only reads the first of the files in ~/ that it finds (and ignore the rest). rc means run commands for example, .nanorc.
Aliases and Functions for Individual Users
- /etc/profile (systemwide environment and shell variables)
- /etc/profile.d/*.sh (systemwide environment and shell variables)
- ~/.bash_profile (user environment and shell variables)
- ~/.bashrc (executes /etc/bashrc)
- /etc/bashrc (systemwide aliases and shell functions)
- ~/.bashrc (user aliases and shell functions)
Why does it take tens of seconds to get a shell prompt?
https://serverfault.com/a/722496 If your profile or bashrc have expensive things, consider trimming them back.
On raspbian commenting out some lines does help. Interestingly, the same lines does not make any difference on x86 server.
the source command
The . is a shorthand for the source command in bash. source ~/.bashrc and . ~/.bashrc are equivalent.
login shell (.bash_profile) vs interactive shell (.bashrc)
- http://stackoverflow.com/questions/18186929/differences-between-login-shell-and-interactive-shell
- What is Interactive Login and Non-Login Shell
- http://serverfault.com/questions/8882/what-is-the-difference-between-a-login-and-an-interactive-bash-shell
- login shell - non desktop environment. ~/.bash_profile is sourced for the bash shell. It is the shell you get when logging in or opening a new terminal session.
- You do not usually have .bash_profile on Ubuntu, nor should you usually create that file.
- On new Ubuntu, there is no .bash_profile. It has .profile.
- you should not put aliases in .profile at all, nor is .bash_profile a good place for them because you will want your aliases to work in interactive shells whether or not they are login shells.
- Login shells are interactive shells.
- interactive shell - Ctrl+Alt+t to open a terminal from a graphical mode (desktop environment) and also the ssh connection. ~/.bashrc is source. We usually edit ~/.bashrc to set up the environment to include fancy prompt, set aliases, set history options, or define custom shell functions. Bash Check If Shell Is Interactive or Not Under Linux / Unix Oses
- Non-interactive shell - instances of the shell you can't use interactively. Shells that are started to run a command or script.
To determine the shell type: echo $-
export environment variables
- Both a login shell and an interactive one. SSH (Putty) to connect to a remote machine.
- When a shell runs a script or a command passed on its command line, it's a non-interactive, non-login shell.
/root/.bashrc
If we use "sudo SOME_COMMAND", ~/.bashrc won't work. In this case, we have to
- run "sudo su"
- Edit /root/.bashrc
Login banner
How to configure login banners in Linux (RedHat, Ubuntu, CentOS, Fedora). /etc/motd and /etc/login.warn
Change to root shell
The following command will switch to an environment similar to what the user would expect had the user logged in directly.
sudo su - # OR sudo su # OR sudo -s
This can be useful when running 'su' or 'su -' failed because of an authentication failure error (note Ubuntu locked the root account).
See also
- wiki.archlinux.org.
- Difference of 'su', 'sudo -s' and 'sudo -i' from askubuntu.com. 'sudo -s' and 'sudo su'?
For sudo to work, my account ('debian' in this case) has to be included in the config file /etc/sudoers.
debian ALL=NOPASSWD: ALL
When sudo is invoked, it asks for the password of the user who started it.
pinky: find out about the people logged on to your Linux computer
How to Use the pinky Command on Linux
su: Switch to another user and run a command
- su means 'substitute user'.
- Use
su newusername # keeps the environment variables of the original user su - newusername # clears all the environment variables
- Linux Run Command As Another User
runuser -l command
runuser -l userNameHere -c 'command'
Keyboard shortcut to move cursor in the terminal
- Alt + b: go left (back) one word
- Alt + f: go forward on word
Example: cd ~/bitbucket/gbmpdx/annovar_biowulf (Now press Alt+b to see the cursor moves)
Record terminal session to a text file
script history_log.txt # recording begins exit # stop recording
This will include everything showing on your screen.
Tools To Record Your Terminal And Generate Animated Gif or SVG Images
Asciinema & agg
- It works on Ubuntu 23.04
- Asciinema
- 3 Ways to Solve Pip Install Error on Ubuntu 23.04
- https://github.com/asciinema/agg
$ # sudo pip3 install asciinema # Error $ # sudo apt install python3-asciinema # Error $ sudo apt install pipx $ pipx ensurepath Success! Added /home/brb/.local/bin to the PATH environment variable. Consider adding shell completions for pipx. Run 'pipx completions' for instructions. You will need to open a new terminal or re-login for the PATH changes to take effect. Otherwise pipx is ready to go!
Open another tab
pipx install asciinema asciinema rec 2g-test asciinema play 2g-test chmod +x Downloads/agg-x86_64-unknown-linux-gnu Downloads/agg-x86_64-unknown-linux-gnu 2g-test 2g-test.gif open 2g-test.gif
terminalizer
sudo apt update sudo apt install nodejs npm sudo npm install -g npm sudo npm install -g terminalizer terminalizer record demo terminalizer play demo terminalizer render demo # https://github.com/faressoft/terminalizer/issues/29 # https://github.com/faressoft/terminalizer/issues/211
Termtosvg
(archived) Termtosvg – Record Your Terminal Sessions As SVG Animations In Linux
Record and Replay Linux Terminal Sessions Activity: script
Learn how to Record and Replay Linux Terminal Sessions Activity
Recording your terminal: asciinema
Dictionary/translation
- Dictionary. Lookup word-meaning using DICT protocol with curl command
$ curl dict://dict.org/d:word
The output of the above method is too complicated. The 'trans' utility provides a better output (even still no colors)
$ trans -d word
- Translation shell
$ wget git.io/trans $ chmod +x ./trans $ sudo apt update; sudo apt install gawk $ sudo mv trans /usr/local/bin/ $ trans :zh-TW "interrogate" interrogate /inˈterəˌɡāt/ 審問 (Shěnwèn)p
Clear screen
ctrl + l
Clear text
ctrl + u: "deletes" all the entered text to the left of the cursor. Does not work in macOS shell but R console from RStudio in macOS still works.
Redirect standard error
http://bash.cyberciti.biz/guide/Standard_error. Use 2> operator.
command 2> errors.txt
Redirect standard output
This can be used in the cron job or displaying a clock on the desktop.
$ cat ~/bin/clock dclock -date "Today is %A %B %Y" -led_off black -bg black -fg yellow -geometry 577x194+119+139 &>/dev/null &
Quotes and asterisk
Combining these two will not work. For example
brb@T3600 ~ $ ls -l ~/GSE48215/*.fastq -rw-r--r-- 1 brb brb 16226673016 Jun 14 14:13 /home/brb/GSE48215/SRR925751_1.fastq -rw-r--r-- 1 brb brb 16226673016 Jun 14 14:13 /home/brb/GSE48215/SRR925751_2.fastq brb@T3600 ~ $ ls -l '~/GSE48215/*.fastq' ls: cannot access ~/GSE48215/*.fastq: No such file or directory brb@T3600 ~ $ ls -l "~/GSE48215/*.fastq" ls: cannot access ~/GSE48215/*.fastq: No such file or directory
cat command alternatives
- bat
- Bat – Linux cat command with syntax highlighting and Git integration
- Direct installing bat using 'apt install' does not work:( Grab the binary from github works.
$ curl -s https://api.github.com/repos/sharkdp/bat/releases/latest |grep browser_download_url | cut -d '"' -f 4 | grep 'amd64.deb' | grep -v musl | wget -i - $ sudo dpkg -i bat_*_amd64.deb
ls command
List Files With Detailed Information from How to Use the ls Command in Linux
To use UID/GID instead of the user name and group name in ls -l, use the -n option.
ls -n
To make a pretty output by showing selected columns (col 9 is the file name and col 5 is the file size)
$ ls -nt bad | grep -v ^total | awk '{ printf "%-20s %15i\n", $9, $5}'
recal.bai 8069704
recal.bam 12275091222
recal_data.table 1012453
realigned_reads.bai 8065496
Follow the symbolic link
Use -H option
ls -lH myDir
List only directories
ls -d */ # current directory ls -ld ~/Downloads/*/ # ~/Downloads ls -l -d */
List only files
ls -l | egrep -v '^d'
Find and Delete Broken Symbolic Links
find /path/to/directory -xtype l -delete
Special characters, escape
List of characters which needs to be escaped in a linux shell command
| & ; < > ( ) $ ` \ " ' <space> <tab> <newline>
Opened MS-Office documents' filenames start with "~$". The dollar sign character has to be escaped; eg ls -l ~\$* to list these kind of files or rm ~\$* to delete these files.
Check non-English characters
How to Find Non-ASCII Characters in Text Files in Linux
perl -ne 'print if /[^[:ascii:]]/' sample.txt
ls | more without lose color
$ ls --color=auto $ ls --color | more
Most likely your ls is aliased to ls --color=auto. If you do ls --color (which is morally equivalent to ls --color=always), that will force it to turn on colors.
ls directories color
How to Change the Colors of Directories and Files in the ls Command. No need to use the export command when we want to add it to .bashrc file.
# orange color export LS_COLORS=$LS_COLORS:'di=0;33:' # OR yellow color if your terminal supports 256 colors export LS_COLORS=$LS_COLORS:'di=38;5;226:'
ls output with color background
In my case, after I apply chmod 755 -R XXXX, the weird green background color goes away.
ls output without user/group columns
https://unix.stackexchange.com/a/205165
ls -lhog
To further skip the first 2 columns (the permissions and link count) we can use
ls -lhog | sed 's/^[^ ][^ ]* *[^ ][^ ]* //' # total 95G # 51G May 7 11:37 SRR10156301_3.fastq --> file # 23G May 7 11:35 SRR10156301_2.fastq --> file # 18G May 7 11:35 SRR10156301_1.fastq --> file # 4.0K May 7 11:24 SRR10156301 --> directory
ls output selected columns
The following example will sort the output by file size and output only the file size and file name columns.
ls -lS . | awk '{print $5 "\t" $9}'
ls -lhS . | awk '{print $5 "\t" $9}'
# Mac
ls -lS *.Rmd |head | awk '{print $6 "\t" $10}'
# a range of columns:
ls -lS *.Rmd |head | awk -v f=6 -v t=10 \
'{for(i=f;i<=t;i++) printf("%s%s",$i,(i==t)?"\n":OFS)}'
ls suddenly wrapping items with spaces in single quotes
Use ls -N to remove single quotes for files containing spaces. See Why is 'ls' suddenly wrapping items with spaces in single quotes?
Better to add export QUOTING_STYLE=literal to .bashrc
ls on BSD/macOS
Use the -G option to get a color output
$ ls -G
realpath
realpath FILENAME to get the full path of a file.
tree command
tree -d: show directories only
tre command
- github. Binary files for x86 and arm linux are available. To use it, run "tre DIRECTORYNAME" or "tre" to list files recursively from the current directory.
cp command
Linux cp command tutorial for beginners (8 examples)
copy a directory
cp -avr Dir1 Dir2
where -a will preserve the attributes of files/directories, -v means verbally and -r means copy the directory recursively.
Copy a file with progress bar with pv (plus how to eject the USB drive)
http://www.tecmint.com/monitor-copy-backup-tar-progress-in-linux-using-pv-command/
sudo apt-get install pv pv file1 > file2 # don't forget the ">" operator and the destination is a file, not a directory
After that, instead of clicking the reject icon from the file manager to eject it, it is better to use a command line to do that because there is no expect time for users to know when it will take for finish writing the data to a USB drive.
sudo apt-get install udisks sudo udisks --unmount /dev/sdb1 # /dev/sdb1 is the partition sudo udisks --detach /dev/sdb # /dev/sdb is the device
My testing shows this procedure works (tested by running md5sum after eject/plug-in) when I need to copy a 9GB file.
Reliable way: Split the large file and copy smaller chunks
# Use 'sudo iotop -o' to monitor the I/O split -b 4G inputFile # create xaa, xab, ... files cat x* > outputFile # merge them. md5sum check succeeds type x* > outputFile # Windows OS. # https://stackoverflow.com/a/60254 # Use Prefix, and use numeric suffixes starting from 0 split -b 4M -d inputFile inputFile.part md5sum inputFile cat inputFile.part* > inputFile2 ech "LONG_MD5_SUM_From_inputFile inputFile2" | md5sum -c
It is interesting copying smaller files (eg 4GB) to USB drives is quite stable (just use the cp command). Even for a not-too large file (6.7GB), pv step looks OK but the unmount/detach step failed.
For a 6.7GB file, it will split it into a 4GB and 2.7GB files. Merge takes longer time if it is done on the USB drive. That is, it is best to do merge in the final destination (internal disk/storage).
- split in the internal hdd: 1min 38sec
- merge in the internal hdd: 37sec
- merge in the USB 3.0 drive: 2min 17sec
Remember: Use a reliable USB drives.
The operation could not be completed because the volume is dirty
On a USB 2.0 drive, I can copy files to there but the drive cannot be rejected (Ubuntu has a pop-up showing it is still writing data to it).
When I forcibly rejects the drive and plug it in a Windows PC, Windows shows the message The operation could not be completed because the volume is dirty. This gives a way to run chkdsk (check and repair a file system).
- Open a Windows File Manager
- Right click the USB drive
- Properties
- Tools -> Check now... Start
Done. Now I can use the drive again.
The Linux equivalent to chkdsk is fsck. fsck is a front end that calls the appropriate tool (fsck.ex2, fsck.ex3, e2fsck, ...) for the filesystem in question.
umount /dev/sdb1 # thumb drive sudo fsck /dev/sdb1 sudo fsck -a /dev/sdb1 # auto repair
For the root disk, you have to use a live CD. Otherwise, you will see a message like
$ fsck /dev/sdb1 fsck from util-linux 2.20.1 e2fsck 1.42.9 (4-Feb-2014) /dev/sdb1 is mounted. WARNING!!! The filesystem is mounted. If you continue you ***WILL*** cause ***SEVERE*** filesystem damage. Do you really want to continue<n>? no
Files
List files by sorting according to the file size
Use the -S option in ls.
List files using wildcard without showing files under subdirectories
Use the -d option in ls. For example, the following command will not showing files under R-3.4.4 and R-3.5.3
$ ls -d R* drwxr-xr-x 15 brb brb 4096 Mar 14 09:48 R-3.4.4 -rw-rw-r-- 1 brb brb 30474612 Mar 15 2018 R-3.4.4.tar.gz drwxr-xr-x 15 brb brb 4096 Mar 14 09:31 R-3.5.3 -rw-rw-r-- 1 brb brb 30205979 Mar 11 04:04 R-3.5.3.tar.gz
Delete multiple files
How to Remove Multiple Subdirectories with One Linux Command
rm -r ~/Documents/htg/{done,ideas,notes}
Delete a certain type of files recursively under a directory
For example to delete *.o files under the current directory,
find . -type f -name '*.o' -delete
Remove all files/directories except for one file/some file type
https://unix.stackexchange.com/a/153863
# recursively, including hidden ones
find . ! -name 'file.txt' -type f -exec rm -f {} +
# Non-recursively
find . -maxdepth 1 ! -name 'file.txt' -type f -exec rm -f {} +
find: warning: you have specified the -maxdepth option after a non-option argument !, but options are not positional (-maxdepth affects tests specified before it as well as those specified after it). Please specify options before other arguments.
Bash Find And Delete All Hidden Files Directories
rm -rf .??* # tested, http://labtestproject.com/linuxcmd/rm.html # list all hidden files/directories find . -name ".*" -print # delete all hidden files find . -name ".*" -type f -delete find . -name ".DS_Store" -delete
Create a new directory and cd to it
How to Make a New Directory and Change to It with a Single Command in Linux
Set permissions for new files or directories
- umask: user file-creation mode mask
- With umask 007, newly created files and directories will have all permissions (read, write, and execute) for the user and the group, but no permissions for others. This is a good practice when you want to share data with other users in the same group but want to completely exclude users who are not group members.
- Set up default umask on Linux: adding the command umask 007 to the .bashrc file sets the default umask value for your shell. This means that every time you start a new shell session, the umask will be set to 007.
- How it works:
- The default permissions for files are 666 (read and write for owner, group, and others) and for directories are 777 (read, write, and execute for owner, group, and others).
- The default permissions for files are 666 and for directories are 777. In binary, these are 110 110 110 and 111 111 111 respectively. The umask value 007 is 000 000 111 in binary.
- The AND operation is performed between the binary representations of the default permissions and the bitwise NOT of the umask value. That is, the mask is negated (its bitwise compliment is taken) and this value is then applied to the default permissions using a logical AND operation.
- The result of the AND operation is 110 110 000 for files and 111 111 000 for directories. In decimal, these are 660 and 770 respectively.
- See What is Umask and How To Setup Default umask Under Linux?, Linux umask command.
chown and chmod recursively (-R)
Use -R (capital R).
chmod -R u=rwx,go=rx /var/www/html
Note it is better not to remove 'x' on folders. Otherwise we will lose the permission to change file attributes for any file under the folder.
Note "chmod -R ugo+rwx XXX" is the same as "chmod -R 777 XXX" b/c "u" stands for user, "g" stands for group and "o" stands for others.
Get the chmod numerical value for a file/directory
https://unix.stackexchange.com/a/46921
# BSD/OS X: stat -f "%OLp" <file> # Linux: stat --format '%a' <file> stat -c %a <file>
After that we can use ls -lhog to verify.
Files under a directory have question mark attribute
Use sudo chmod -R a+x /some/directory to fix. This happened when I unzip a zip file compressed in a Windows OS.
Files have an integer owner in attributes
What does the “number” in the owner field of files signify in linux?
You probably did a copy that preserved the original group and owner of these files. Within linux internally the owner and group is basically just an id.
You can change the ower and group to an existing owner and group with the commands chown and chgrp respectively.
Uppercase S in permissions of a folder and setGID
- Understanding Linux File Permissions
- How to use special permissions: the setuid, setgid and sticky bits
- Uppercase S in permissions of a folder
- How to Create a Shared Directory for All Users in Linux
- SetUID, SetGID, and Sticky Bits in Linux File Permissions, What are Set UID, Get UID, and Sticky Bits in Linux File Permissions?
I happen to create this case by chmod -R 760 ShareFolder. To make 'S' to become 's', I just need to use chmod -R 2770 ShareFolder.
immutable files
Manage file and directory attributes using chattr and lsattr command
Display files sorted by modified date in a directory recursively
stat --printf="%y %n\n" $(ls -tr $(find DIRNAME -type f))
find -type f -printf '%T+\t%p\n' | sort -n
Both of methods give the same output. Note the latest changed file is shown at the bottom of the output.
Sort files by their size
use the '-S' option.
ls -lS
Files starting with a dash (meta-characters)
Move File Starting With A Dash
$ > '-foo.txt' $ rm "-foo.txt" rm: invalid option -- 'o' Try 'rm ./-foo.txt' to remove the file '-foo.txt'. Try 'rm --help' for more information. $ rm -- -foo.txt
Inodes
- https://en.wikipedia.org/wiki/Inode
- Inodes in Linux: limit, usage and helpful commands
- Everything You Need to Know About inodes in Linux
Recover Deleted Files
How to Recover Deleted Files on Linux
alias
https://www.cyberciti.biz/faq/how-to-turn-on-or-off-colors-in-bash/
$ alias # list all aliases $ alias | grep ls $ unalias ls $ alias ls='ls --color=auto' # save it in ~/.bash_profile or ~/.bashrc $ alias server_name="ssh -v -l john 192.168.1.11" # or modify /etc/hosts $ alias open='xdg-open' $ alias sshnocheck='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' $ alias scpnocheck='scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' $ alias sshserver='ssh [email protected]' $ alias checkport='sudo lsof -i -P -n | grep LISTEN'
NOTE the last column of the output from "checkport" will show who can connection to the port. For example, if it shows "127.0.0.1:10999 (LISTEN)", it means only the server can connect to port 10999. If it shows "*:22 (LISTEN)", it means any machines can connect to the server through port 22.
To avoid using the alias, use one of the following ways (eg use the command's full path)
$ \ls $ /bin/ls $ command ls $ 'ls'
Useful aliases (Added to ~/.bashrc)
alias nano="nano -c --softwrap"
We can use the backslash to escape the double quote (or others like dollar sign).
If a command needs the sudo right, include the command inside alias. In the following example, if we skip "sudo" then running the command "sudo lsof2" will result in an error: sudo: lsof2: command not found.
alias lsof2="sudo lsof -i -P | egrep \"PID|LISTEN\""
Use unalias [alias name] to remove an alias.
Find file defining an alias
how to find file defining an alias. It works on macOS. Many of them are defined in ~/.oh-my-zsh/lib/directories.zsh. Another way (without saving) is How to find out where alias (in the bash sense) is defined when running Terminal in Mac OS X.
lolcat - bring color to text
sudo apt-get install ruby sudo gem install lolcat lolcat -h lolcat --version fortune | lolcat ps | lolcat man ls | lolcat lolcat test.R sudo apt install figlet figlet Merry Christmas | lolcat alias lolls="ls -l | lolcat" lolls
File manager
Cloud commander
- https://cloudcmd.io/
- https://www.ostechnix.com/cloud-commander-a-web-file-manager-with-console-and-editor/. You can upload files from the Cloud services like Google drive, Dropbox, Amazon cloud drive, Facebook, Twitter, Gmail, GtiHub, Picasa, Instagram and many.
diff
15 Best Free and Open Source Terminal-Based Diff Tools
Run diff with large files
diff (or even better the cmp command) works fine with 8G fastq files. Note cmp compare files byte by byte so it probably won't run out of memory. How to diff large files on Linux.
Another strategy is to split a large file into small pieces. For example,
split -b 500MB FILEname # into 500MB files each split -l 200 FILEname # into smaller files with 200 lines each
Meld freezes When I tested it with two large files (800k & 936k lines coming from human gtf files). Actually the whole linux system became unresponsive.
Actually Meld is sluggish when it is used in small files in Odroid XU4 running Ubuntu 16.04 MATE. I have used Meld 3.14.2 and the latest 3.16.2.
Kompare
https://apps.kde.org/en/kompare
Meld cannot change theme. So if my desktop has a dark theme, meld is hard to read.
Kompare still has a light them.
Directory
diff -qr dir1 dir2
where -q means to report only when files differ and -r is to recursively compare any subdirectories found.
diff & colordiff-color on terminal: compare side by side
PS. For a GUI version of diff, Meld works fine. Need to install first. apt-get install colordiff. http://www.cyberciti.biz/programming/color-terminal-highlighter-for-diff-files/
sudo apt-get install colordiff diff -y file1 file2 | colordiff # Ignore same rows (two ways): # diff -C0 file1 file2 | colordiff # diff -U0 file1 file2 | colordiff # On systems that I have no root right, I need to install it from the source # (just need to run 'make') $ diff file1 file2 | ~/bin/colordiff-1.0.18/colordiff.pl
where -y option means to show the output in two columns.
Interpretation of the diff output:
The first line of the diff output will contain:
- line numbers corresponding to the first file,
- a letter (a for add, c for change, or d for delete), and
- line numbers corresponding to the second file.
In our output above, 2,4c2,4 means: "Lines 2 through 4 in the first file need to be changed in order to match lines 2 through 4 in the second file." It then tells us what those lines are in each file:
- Lines preceded by a < are lines from the first file (color in red);
- lines preceded by > are lines from the second file (color in green).
- The three dashes ("---") merely separate the lines of file 1 and file 2.
2,4c2,4 < I need to run the laundry. < I need to wash the dog. < I need to get the car detailed. --- > I need to do the laundry. > I need to wash the car. > I need to get the dog detailed.
colordiff -ur path1 path2
If you change -ur to -urN then that will also show the contents of files that are only present in one of the paths.
The meaning of colors can be found in /etc/colordiffrc (man colordiff or colordiff web site)
- plain=off
- newtext=darkgreen
- oldtext=darkred
- diffstuff=darkcyan
- cvsstuff=cyan
git diff --no-index -- file.a file.b
Any visual diff in Linux console?
gnome-terminal
Remember the session
- gnome-terminal --help-all --tab-with-profile
- Can no longer set terminal title in Ubuntu 16 (gnome-terminal)
- Opening multiple tabs with gnome-terminal: use --tab and profile options
- Save multiple gnome-terminal layout?: --load-config and --save-config options. NOTE gnome 3.18 option "--save-config" is no longer supported. But strangely enough, "--load-config" is still there.
- How to remember multiple tabs' session in terminal? (Alike FireFox session manager): --profile= and --save-config options. --working-directory and --tab options.
The following is proved working on Ubuntu 18.04
gnome-terminal --tab --working-directory=$HOME/Downloads \
--tab --working-directory=$HOME/Documents
Fun: piano
Let Us Play Piano In Terminal Using Our PC Keyboard
Terminals in grids
See Terminal_multiplexer.
Tilix
Terminator
GNU screen
tmux*
Byobu
Guake / Yakuake / Tilda
Drop down terminals for the GNOME / KDE / GTK Environments. Great for quick access to a terminal!
System date/time, ntpd
- Install and configure Network Time Protocol (NTP) Server,Clients on Ubuntu 16.10 Server
- How to bind ntpd to specific IP addresses on Linux/Unix
- How To Verify NTP Setup (Sync) is Working or Not In Linux?
$ timedatectl
Local time: Mon 2019-06-10 08:37:09 EDT
Universal time: Mon 2019-06-10 12:37:09 UTC
RTC time: Mon 2019-06-10 12:37:09
Time zone: America/New_York (EDT, -0400)
Network time on: yes
NTP synchronized: yes
RTC in local TZ: no
Linux file timestamps
Linux File Timestamps Explained: atime, mtime, and ctime
Change the date/timestamp of a file - touch
Modify the file relative to its existing modification time
filename=MyFileName touch -d "$(date -R -r $filename) - 2 hours" $filename # 2 hours before touch -d "$(date -R -r $filename) + 2 hours" $filename # 2 hours later
See How can I change the date modified/created of a file?
Find binary file location, type
- which - Display the full path of shell commands. See examples from cyberciti.biz.
$ which ls /bin/ls
- whereis - locate the binary, source, and manual page files for a command. See examples from cyberciti.biz.
$ whereis ls ls: /bin/ls /usr/share/man/man1p/ls.1p.gz /usr/share/man/man1/ls.1.gz
$ type ls ls is aliased to `ls --color=tty' ls is /bin/ls
- locate. Use locate command mindfully. It is used to find the location of files and directories. Note that locate does not search the files on disk rather it searches for file paths in a database. For example, the following command will search .png files over the system (not only the personal directory).
locate "*.png"
find: Find a file
find operates recursively into sub folders.
-iname
Search by a file name
$ find . -type f -name "abc*" # find a file starting with abc $ find . -iname '*.txt' # -iname or -name is necessary
It also works for searching files on subdirectories.
$ find . -name transcripts.gtf ./RH_bio/transcripts.gtf ./dT_ori/transcripts.gtf ./dT_tech/transcripts.gtf ./dT_bio/transcripts.gtf ./RH_ori/transcripts.gtf ./RH_tech/transcripts.gtf
-mtime, -ctime, -atime argument
Find files modified in 10 days.
$ find . -ctime -10 # include subdirectories . ./mediawiki-1.32.0.tar.gz ./d ./d/deepSurv.pdf
Find files modified in one day and contain string 'est'
$ find . -mtime -1 -exec grep --with-filename est {} \;
If the search directory is not the current directory, we need to add a forward slash to the directory name.
$ find ~/Desktop -iname '*.txt' # Not working $ find ~/Desktop/ -iname '*.txt' # Working
The following example shows we can list multiple search criteria. The “‑r” option in tar appends files to an archive. xargs is a handy utility that converts a stream of input (in this case the output of find) into command line arguments for the supplied command (in this case tar, used to create a backup archive).
find / -type f -mtime -7 | xargs tar -rf weekly_incremental.tar gzip weekly_incremental.tar
What is the difference between mtime, atime and ctime?
- mtime (modification time) indicates the time the contents of the file has been changed. Mind you, only the contents. Not the attributes.
- ctime (change time) is the timestamp of a file that indicates the time that it was changed. Now, the modification can be in terms of its content or in terms of its attributes.
- atime (access time) is the timestamp that indicates the time that a file has been accessed.
xargs
-exec COMMAND {} \;
- Search and list files with details
find . -type f -name "*street*" -exec ls -lh {} \;-exec ls -lh {} \; executes the ls -lh command on each found file. The {} is a placeholder for the current file, and \; indicates the end of the -exec command.
- How to run find -exec?, Using semicolon (;) vs plus (+) with exec in find
# This command will run grep chrome on each file individually. find . -exec grep chrome {} \; # This command will run grep chrome on batches of files, reducing the number of times the command is executed. # Using + can significantly improve performance find . -exec grep chrome {} +Note:
- The backslash before ; is to escape ; so linux won't interpret it directly.
- Command Compatibility: Not all commands support being executed with multiple arguments at once. For example, grep can handle multiple files, but some other commands might not. In other words, while -exec COMMAND {} + can be more efficient, it’s important to ensure the command you’re using supports this usage and behaves as expected.
- Find files and execute something (google: find --exec)
$ find ./ -name "*.tar.gz" -exec tar zxvf {} \; - Find and move files to a new directory
find OLDDIR -type f -exec mv -t NEWDIR {} + - Find the total file size of a list of files.
- What is meaning of {} + in find's -exec command?
- Why does 'find -exec cmd {} +' need to end in '{} +'?
- How to run find -exec?
The following will find out the total file size of the 'accepted_hits.bam' file under all sub-directories.
find ./ -iname "accepted_hits*" -exec du -ch {} + | grep total$where '-c' produces a grand total, and will substitute {} with the filename(s) found in -exec.
How to find and delete directory recursively
How to find and delete directory recursively on Linux or Unix-like system. Application: recursively remove backups older than 30 days.
Find all soft link files
find /tmp -type l
Recursive statistics on file types in directory?
You could use find and uniq for this. This is fast!
$ find . -type f | sed 's/.*\.//' | sort | uniq -c
Exclude or Ignore Files
Find command Exclude or Ignore Files (e.g. Ignore All Hidden .dot Files )
Avoid Permission Denied Messages
How to fix find command permission denied messages
- Redirecting ALL standard error (not only permission denied error): 2>/dev/null.
find . -iname "data*.txt" -print 2>/dev/null
- Focus on the 'permission denied' message: grep -v "Permission denied"
find / -name foo 2>&1 | grep -v "Permission denied"
Find Files That Have Been Modified Recently in Linux
Find Files That Have Been Modified Recently in Linux
fd: The Find Command Alternative
Fd: The Find Command Alternative For Mastering File Search In Linux
grep: Find a file by searching contents
grep -r -i "Entering" ~/Downloads/R-3.0.0/
where -r means recursively searching the directory and -i means case insensitive.
Sometimes using -R is more effective because of the symbolic links issue.
$ grep -r -i phpmyadmin /etc/apache2/ # nothing returned $ grep -R -i phpmyadmin /etc/apache2/
We can also display the row numbers for matches by using the -n parameter in grep.
# What variants appear in dbsnp grep -n 'rs[0-9]' XXX.vcf
To exclude lines with a pattern, using the -v parameter.
# How many variant were called grep -v "^#" XXX.vcf | head
To exclude binary files, use -I parameter.
To show only matched filenames, using the -l parameter.
grep -l "iterator" *.cpp # if we add '-n', the '-n' option won't work.
To search with certain file extensions, use --include argument; see this post.
grep -r -i --include \*.h --include \*.cpp KEYWORD ~/path[12345] # escape with \ just in case you have a directory with asterisks in the filenames
How To Find All Files Containing Specific Text On Linux From The Command Line
grep -Rni --exclude-dir={Private,Personal} --include={*.txt,*.js} 'text' ~/Documents
# Exclude hidden directories
grep -R --exclude-dir=".*" 'text' ~/Documents
The only issue with using the -f argument is that grep is going to attempt to interpret the keywords as if they are patterns, which can slow it down when parsing against an extremely large file. So you can also specify the -F parameter, which tells grep to only do exact matches against the strings.
grep -f searchstringsFile filetosearch > output.txt # -F, --fixed-strings grep -F searchstring filetosearch > output.txt
If the pattern is saved in a file, use the -f parameter
grep -f PATTERNFILE INPUTFILE
If there are two keywords, use the following
$ grep "begin\|completed" --color swarm_58606147_0.o # needs an escape begin 2018-01-12 14:46:05 alignment is completed 2018-01-12 16:45:24 marking duplication is completed 2018-01-12 17:52:01 assign read group is completed 2018-01-12 18:22:49 indel re-alignment is completed 2018-01-12 19:29:32 BQSR is completed 2018-01-12 22:26:22 GATK is completed 2018-01-12 23:43:3 $ egrep "begin|completed" --color swarm_58606147_0.o # no need an escape if we use extended regular expressions
We can use R to compute the time spent in each step; see Dealing with dates.
Check https://www.howtoforge.com/tutorial/linux-grep-command/ for more examples
- Using grep to search only for words / exact match ("-w" option)
- Using grep to search two different words (egrep -w 'word1|word2' /path/to/file)
- Count line for matched words ("-c" option)
- Grep invert match ("-v" option)
- How to list only the names of matching files ("-l" option)
Preserve colouring after piping grep to grep
https://stackoverflow.com/a/2327216
Use grep --color=always .For example, grep --color=always KEYWORD Myfile | more.
Include/follow soft/symbolic link files
grep grep -Rl instead of "grep -rl". Recall "-l" is to list filename only. See "man grep".
Compressed files
zgrep or zipgrep command
GUI
A GUI version of a tool to search files is searchmonkey (open source, Linux, Windows). On Ubuntu, we install it by
sudo apt-get install searchmonkey
It is also useful to change the settings so we can click a filename and open it in the desired text editor. To do that, go to Settings -> Preferences -> System Call -> Text Editor. I enter 'geany' since I want to use geany to open my C programs. Note. the v2.0 source code needs to be built using i386 gcc library and Qt 4.8.x. Still, I cannot get rid of some errors coming from the source code.
-- option
How To Use grep To Search The --help Output To Find Out What CLI Arguments That Begin With A Dash Do
rm --help | grep -w -- -r rm --help | grep -w -- '-[rf]' # multiple one letter
Summary of find and grep commands
| Command | Examples |
|---|---|
| find | find [DIRECTORY] -iname '*.txt'
find [DIRECTORY] -maxdepth 2 -iname *.php find -name '*.php' -o -name '*.txt' # OR operator |
| grep | grep -r -i "check_samtools" DIRECTORY/
dpkg -l libgtk* | grep '^i' |
Format the output: column
This command will make the output of some command easy to read; see 18 Commands That Will Change The Way You Use Linux Forever.
For example: mount | column -t
- column command in Linux with examples
- Different Ways to Use Column Command in Linux
- Display Specific Columns From a File in Linux
Count number of columns: awk
The following command shows the number of columns for the first few rows of a text file.
head MYFILE | awk '{ print NF}'
head MYFILE | awk -F '\t' '{ print NF}'
Count number of rows in a file: wc
wc -l MYFILE
The source code of wc (or any Linux command) can be found by using this method
brb@brb-T3500:~/Downloads$ which wc /usr/bin/wc brb@brb-T3500:~/Downloads$ dpkg -S /usr/bin/wc coreutils: /usr/bin/wc brb@brb-T3500:~/Downloads$ sudo apt-get source coreutils
As we can see from the coreutils-8.21/src directory, there are over 100 C programs including <cat.c>, <chmod.c>, <cp.c>, ...<wc.c>.
Print certain rows/lines of a text file
The following example will print out lines 10 to 60 of FILENAME.
sed -n '10,60p' FILENAME
Or to print out line 60,
sed -n '60p' FILENAME
It seems this method is not as fast as I expected. For example, the tail command will immediately print out the result without waiting!
Print a text file with line number: less
How to Use the less Command on Linux
less -N myfile
output colored console to html
Use ansi2html.sh. It only requires gawk.
- Use wget to download it
- sudo apt-get install gawk
- chmod +x ansi2html.sh
- colordiff file1 file2 | ./ansi2html.sh > diff.html
using a the result of a diff in a if statement
ls -lR $dir > a
ls -lR $dir > b
DIFF=$(diff a b)
if [ "$DIFF" != "" ]
then
echo "The directory was modified"
fi
Another example
if [ "$(diff file1.html file2.html)" == "" ]; then echo Same; else echo Different; fi
Prompt
Colored prompt
- http://www.tldp.org/HOWTO/Bash-Prompt-HOWTO/x329.html
- https://www.cyberciti.biz/faq/bash-shell-change-the-color-of-my-shell-prompt-under-linux-or-unix/
For example, the following code will change the prompt to a light blue color. NOTE that we need ∖[ and ∖] in order to avoid a problem of miscalculating the cursor's starting position.
# blue export PS1='\[\e[1;34m\]\u@\h:\w\$ \[\e[0m\]' # bright blue (good) export PS1='\[\e[0;34m\]\u@\h:\w\$ \[\e[0m\]' # darker blue # yellow export PS1='\[\e[1;33m\]\u@\h:\w\$ \[\e[0m\]' # bright yellow export PS1='\[\e[0;33m\]\u@\h:\w\$ \[\e[0m\]' # dark yellow (good) # red export PS1='\[\e[1;31m\]\u@\h:\w\$ \[\e[0m\]' # bright red export PS1='\[\e[0;31m\]\u@\h:\w\$ \[\e[0m\]' # dark red (good) # green export PS1='\[\e[1;32m\]\u@\h:\w\$ \[\e[0m\]' # bright green export PS1='\[\e[0;32m\]\u@\h:\w\$ \[\e[0m\]' # dark green # cyan export PS1='\[\e[1;36m\]\u@\h:\w\$ \[\e[0m\]' # bright cyan export PS1='\[\e[0;36m\]\u@\h:\w\$ \[\e[0m\]' # dark cyan (good) # purple export PS1='\[\e[1;35m\]\u@\h:\w\$ \[\e[0m\]' # bright purple (good) export PS1='\[\e[0;35m\]\u@\h:\w\$ \[\e[0m\]' # dark purple
To make a permanent change, we can add the line to ~/.bashrc file and (is it necessary) un-comment the following line
force_color_prompt=yes
Some of my settings in .bashrc file
# Office Linux Mint, bright blue, display current time # Example: 12:45PM ~/Downloads$ PS1="\[\033[1;34m\]\$(date +%H:%M%p) \w$\[\033[0m\] " # Biowulf and Helix, dark yellow # Example: biowulf:~/R$ PS1='\[\e[0;33m\]\h:\w\$ \[\e[0m\]' alias nano="nano -c --softwrap" # Mac, light green export PS1="\[\e[0;32m\]mac$\[\e[0m\] "
Shorten prompt
How can I shorten my command line (bash) prompt? It is useful especially in VM.
Add a timestamp to your Bash prompt
- See man 3 strftime for the date, time format. In Linux Mint, the date applets uses the %A %B %e, %H:%M which gives a format like 'Friday July 15, 10:23'.
- For a Good Strftime - Online date/time formatting tool
- http://bneijt.nl/blog/post/add-a-timestamp-to-your-bash-prompt/. Set
From here, we can skip %F (not showing the date), \W (not showing the current directory) and change %T to %H:%M (not showing seconds).PS1 Prompt default brb@p45t:~/Downloads$ PS1='[\D{%F %T}] \u@\h \W\$ ' [2016-07-08 16:56:48] brb@brb-P45T-A ~/Downloads$ PS1="\[\033[1;34m\]\$(date +%H:%M%p) \w$\[\033[0m\] " 10:54AM ~/Downloads$
- Another solution: using PROMPT_COMMAND variable. http://askubuntu.com/questions/193416/adding-timestamps-to-terminal-prompts. Add this line to the ~/.bashrc file:
export PROMPT_COMMAND="echo -n \[\$(date +%H:%M%p)\]\ "
and the output will be something like:
[07:03AM] user@hostname:~$
zsh: display time to the right hand side
add a line RPROMPT='%*' to ~/.zshrc
Proxy
The 15 Best Web Proxies for Geo-Blocked Content and Online Privacy
Listen to pandora in Europe: install squid proxy
http://www.cyberciti.biz/faq/access-pandora-radio-using-proxy-server-outside-usa/
Interestingly, the firefox connection settings should choose HTTP Proxy instead of 'SOCKS host'.
SSH, scp
See ssh.
Graphical way to display disk usage
For example, to use xdiskusage, we run apt-get install xdiskusage and launch it by xdiskusage ~/.
- Ubuntu has a built-in program called "Disk Usage Analyzer". Just search it from Dash. Looks useful!
df : Display disk space
df -h df -h -T # show the 't'ype of the file system like tmpfs, ext4, squashfs (snap), vfat df -h -t ext4 # show file systems of given type (ext4 in this example) df -a # show all file system (include ones that have a size of zero blocks) df -h | grep -v snap # ignore snap partitions df -h | grep -v loop
Note for the NTFS type, it will be reported as fuseblk by mount or df command.
duf
Terminal Tip ‘duf’ is Prettier Alternative to the ‘df’
Disk encryption
Encrypt files
croc Is A Tool For Resumable, Encrypted File And Folder Transfers Between Computers (Command Line)
rm command and trash can
Make “rm” Command To Move The Files To “Trash Can” Instead Of Removing Them Completely
du/ncdu and block size: Display directory size with sorting and human readable
Use ncdu program (more interactive). Although it is a command line program, we can use the mouse to move through each directory to see its sub-directories.
ncdu can show the hidden directory size. This is useful. For example, ~/.local/share/Trash and ~/.singularity/docker can take a lot of space.
sudo apt-get install ncdu ncdu
And the du method.
du -csh *.jpg # total is at the bottom
du -sh ~/* # won't include hidden directories, Fast
du -h ~/ --max-depth=1 # include hidden directories, SLOW
du -h ~/ --max-depth=1 --exclude ".*" | sort -nr | cut -f2 | xargs -d '\n' du -sh
du -a -h ~/ # kilobytes will be used, '-a' is to see all files, not just directories.
du -a ~/ | sort -nr | head -n 10 # sort from the largest file size first (in bytes)
# this includes directories and any files under any directories
du -sh * | sort -hr | head -n 10 # this does not go to subdir; only show files and top directories
The --exclude is to hide hidden directories, '-n' is to compare according to string numerical value, and '-r' is to reverse the result.
Note that the 'du' commands may be cheating. See the following screenshot.
The discrepancy is explained by 'sector'. See http://askubuntu.com/questions/122091/difference-between-filesize-and-size-on-disk. Note: it seems 4096 is what I see from all devices.
$ sudo dumpe2fs /dev/sda1 | grep -i "block size" dumpe2fs 1.41.14 (22-Dec-2010) Block size: 4096 $ sudo tune2fs -l /dev/sda1 | grep -i "block size" $ sudo fdisk -l | tail $ sudo tune2fs -l /dev/mmcblk0p2 | grep -i "block size"
To show a file size in terms of blocks, we can use
ls -s
So for example, if a file takes 150 blocks, and if a block takes 4096 bytes, then the file takes 150*4096/1024 KB on disk.
gdu
gdu Is A Fast Console Disk Usage Analyzer (Alternative To ncdu, du, Etc.)
Find the total size of certain files within a directory
Find the total size of certain files within a directory branch
du -ch ./photos/*.jpg | grep total
find ./photos -type f -name '*.jpg' -exec du -ch {} + | grep total$
Apache benchmark (ab) testing
ab -n 100 -c 10 http://taichimd.us/
Monitor progress of copying/transferring files: pv
How to monitor progress of Linux commands using PV and Progress utilities
# Method 1: rsync rsync --progress -a sourceDirectory destinationDirectory rsync --info=progress2 source dest # Method 2: pv sudo apt-get install pv ## copy a single file pv inputfile > outputfile ## multiple files or directories tar c sourceDirectory | pv | tar x -C destinationDirectory ## https://stackoverflow.com/a/26226261 docker save <image> | bzip2 | pv | \ ssh user@host 'bunzip2 | docker load'
rsync
See Backup.
Wireless File Transfer Apps on Linux
The 7 Best Wireless File Transfer Apps on Linux
sudo
How to Control sudo Access on Linux
https://www.howtogeek.com/447906/how-to-control-sudo-access-on-linux/
sudo adduser NEWUSER sudo usermod -a -G sudo NEWUSER
sudo vs su
The Difference Between sudo and su Explained: password and shell.
How to Keep ‘sudo’ Password Timeout Session Longer in Linux
http://www.tecmint.com/set-sudo-password-timeout-session-longer-linux/
How to run multiple commands in sudo
https://www.cyberciti.biz/faq/how-to-run-multiple-commands-in-sudo-under-linux-or-unix/
Run sudo commands without a password?
- How to Use sudo Commands Without Password in Linux
- How do I run specific sudo commands without a password?
Alternative browsers, text browsers
See Browser.
Filezilla
Keyboard shortcut. Especially, Alt+Down=Transfers the currently selected item to an item of the same name in the other pane.
The device is busy
brb@brb-P45T-A:~$ sudo umount /media/brb/TOSHIBA
[sudo] password for brb:
umount: /media/brb/TOSHIBA: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
brb@brb-P45T-A:~$ sudo umount /dev/sdc1
umount: /media/brb/TOSHIBA: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
brb@brb-P45T-A:~$ lsof /media/brb/TOSHIBA/
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
VBoxSVC 5600 brb 18w REG 8,33 4294967295 3 /media/brb/TOSHIBA/Windows 10.ova (deleted)
brb@brb-P45T-A:~$ kill -9 5600
brb@brb-P45T-A:~$ lsof /media/brb/TOSHIBA/
brb@brb-P45T-A:~$ sudo umount /dev/sdc1
brb@brb-P45T-A:~$
# fuser -m /dev/sdc1 /dev/sdc1: 538 # ps auxw|grep 538 donncha 538 0.4 2.7 219212 56792 ? SLl Feb11 11:25 rhythmbox
Another handy one is:
umount -l /dev/sdwhatever
mkfs command
dd and mkfs
How to Use the mkfs Command on Linux, How to Format Storage Drives Using the Linux Terminal
dd if=/dev/zero of=~/howtogeek.img bs=1M count=250 mkfs.ext2 ~/howtogeek.img sudo mkdir /mnt/geek sudo mount ~/howtogeek.img /mnt/geek sudo chown dave:users /mnt/geek/ cd /mnt/geek cp ~/Documents/Code/*.? . sudo umount /mnt/geek cd /mnt sudo rmdir geek
Format a USB drive: exfat
Easily Format A USB Flash Drive On Ubuntu 18.04 Using USB Stick Formatter (mintStick deb & source)
This is a GUI application. After the installation, search "USB Stick Formatter".
For some reason, it doesn't have the 'exFAT' option. My system has installed exFAT drivers. This post said installing exFAT related drivers only helps reading/writing but not formatting.
If I want exFAT format, I need to use the USB Stick Formatter to format the drive first (for example fat32), plug it and then using the following command to format it to exFAT.
sudo apt install exfat-fuse exfat-utils # ubuntu 20.04 and lower sudo apt install exfat-fuse exfatprogs # ubuntu 22.04 and higher sudo fdisk /dev/sdc # g,p,n,p,1,ENTER,ENTER,w sudo fdisk -l /dev/sdc # the new partition has Id 83 and Type Linux. sudo mkfs.exfat -n Staples /dev/sdc1 sudo fdisk -l /dev/sdc # still Linux? but Gparted shows exfat sudo mount /dev/sdc1 /mnt # to use
Note that fdisk or sfdisk cannot differentiate NTFS/exFAT. But cfdisk or GParted can.
sudo cfdisk /dev/sdX
Create an ext3/ext4 file system on a USB flash drive
- https://en.wikipedia.org/wiki/Smartmontools. My ubuntu 20.04 has smartctl 7.1 while the latest version is 7.3 (Fedora has it)
- How to Format Disk Partitions in Linux
- GUI: using Ubuntu's Disks utility.
umount /dev/sdb1 (depending on the device of course) lsblk # check the drive's partition name sudo mkfs -t ext4 /dev/sdb1 lsblk -f # Verify sudo mkfs.ext3 /dev/sdb1 sudo e2label /dev/sdb1 usbdrive (change the label) sudo mkfs.vfat /dev/sdb1 # MS-DOS
exFat - cross platform partition format
- Mac
- Gparted cannot create exFAT partition (it is greyed out)
- This Trick Makes a USB Drive Work Perfectly With Windows, Mac, Linux, and Anything Else
- How to Mount and Use an exFAT Drive on Linux or How to get a drive formatted with exfat working?
- How to format an SD Card to ExFat in Ubuntu
sudo apt-get install exfat-utils exfat-fuse # Still need to create a partition (ex. FAT32) first using gparted in order to get it mounted sudo fdisk -l sudo mkfs.exfat -n LABEL /dev/sd** # LABEL with whatever you want to label your drive lsblk -f # verify the partitions, no sudo is needed
This should delivery a working exfat file system (read and write support, but not formatting the drives with exfat via Gnome Disks and GParted).
Add a new user with home directory; list all users
adduser xxx
adduser is better than useradd since useradd does not create home directory and it does not even ask the password for new user. adduser will interactively ask user information.
To delete the user and home directory, use
deluser --remove-home xxx
To view the user information, type id USERNAME or cat /etc/passwd.
How to List Users in Linux cat /etc/passwd | wc -l
gzip with multi cores: pigz
Use pigz utility. It makes a lot of difference. For example for a 21GB file, gzip can't finish the job after 30 minutes. But pigz only took 7 minutes on a 12-core machine.
According to 'pigz --help', the default threads is the number of online processors, or 8 if unknown.
sudo apt-get install pigz pigz -9 FILENAME # best compression method & convert the file to FILENAME.gz tar cf - paths-to-archive | pigz -9 -p 12 > archive.tar.gz
There is no need to use pigz to un-compress the file. gunzip is fast enough and only takes 4 minutes to decompress.
The '-9' (best compression) option does not make difference (6.6G vs 6.5G).
Note that we have to be careful when we use md5sum to compare compressed files.
Compress a folder without full path name
Suppose we want to compress the folder ~/Documents and its subfolders. We want to include Documents folder name but not /home/brb/Documents name.
# Method 1. Include 'Documents' as the top folder name cd ~/ tar -czvf tmp.tar.gz Documents # Method 2. Mind the last dot. Not include 'Documents' as the top folder. tar -czvf tmp.tar.gz -C /home/brb/Documents . # Double check the tarball tar -tzvf tmp.tar.gz
If we want to strip the upper directories when we uncompress a tar file, use --strip-components. For example, we can use --strip-components=1 to remove the Documents folder.
Fix mess created by accidentally untarred files in the current dir
Suppose I accidentally untar a tarball in /var/www/html/ directory instead of /home/projects/www/current. It created mess in /var/www/html/. The easiest way to fix this mess:
cd /var/www/html/ /bin/rm -f "$(tar ztf /path/to/file.tar.gz)" ## or better ## tar ztf /path/to/file.tar.gz | xargs -d'\n' rm -v
lzma
- https://en.wikipedia.org/wiki/Lempel–Ziv–Markov_chain_algorithm
- nomadBSD. Since NomadBSD is designed to be a persistent system, we do not provide ISO files, as ISO-9660 is a read-only filesystem.
squashfs
List contents of tar.gz or tar.bz2
tar -tzvf myfile.tar.gz tar -tjvf myfile.tar.bz2 # replace z with j
gzip: stdin: not in gzip format
I got the following message when I try to run tar -xzvf or tar -tzvf command.
$ tar -tzvf filename.tar.gz gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error is not recoverable: exiting now $ file filename.tar.gz filename.tar.gz: POSIX tar archive
The answer is How to solve: stdin: not in gzip format.
Solution: Since it was not a gzipped file, a simple tar is able to extract the file: tar xvf MyFile.tar.gz
Extract files, AVFS
See Extract files.
Show folder size for one level only
du --max-depth=1 -h
The graphical tool is called Disk Usage Analyze which is already available on Ubuntu.
Soft link and hard link
Soft link
ln -s /full/path/of/original/file /full/path/of/soft/link/file
Understanding Linux Links Part 1 & Part 2
The order of original and linked above in ln -s is similar to the mount command where we put the original volume first and the system's directory second. See an example here.
But when we issue "ls -l" we see it list the the original file at the end; e.g. /full/path/of/soft/link/file -> /full/path/of/original/file.
Hard link
What's the difference between a hard links and copied files?
- Why are hard links not allowed for directories?
- Hard linked file looks the same as the source file when viewed by "ls"
- Change in either of hard linked file and the source file will affect the other
- Hard linked file and source file can be deleted separately without affecting the other
echo "abcd" > foo ln foo foo2 ls -l foo* echo "efgh" >> foo2 cat foo rm foo cat foo2
Self-hosted servers
- https://github.com/Kickball/awesome-selfhosted This is a list of Free Software network services and web applications which can be hosted locally. Selfhosting is the process of locally hosting and managing applications instead of renting from SaaS providers.
- Sovereign: A set of Ansible playbooks to build and maintain your own private cloud: email, calendar, contacts, file sync, IRC bouncer, VPN, and more.
Cockpit: manage and monitor my servers using just a web browser
- https://cockpit-project.org/
- https://localhost:9090 is the default interface
- Cockpit: an overview
- Cockpit and the evolution of the Web User Interface
- How I use Cockpit for my home's Linux server management
- How to add public SSH keys for users in Cockpit
- How to Manage Linux Servers with the Cockpit Web Interface including Monitoring Multiple Computers.
DNS
Port number is 53. An example is Pi-hole.
Email server
See Mail_server.
Backup
See Backup.
at command: Schedule a task
- Schedule a task with the Linux at command
- Schedule Jobs in Linux With 'at' Command
- 4 alternatives to cron in Linux
- On Debian, I need to run sudo apt install at.
- Using the at command to schedule tasks on Linux
$ echo "rsync -av /home/tux/ me@myserver:/home/tux/" | at 1:30 AM $ echo "command_to_be_run" | at 09:00 $ atq # list of jobs $ atrm 6 # delete the 6th job
Cron job by root
Note that there is a "user" field for cron jobs defined in /etc/crontab or /etc/cron.d.
- Place one: /etc/crontab and /etc/cron.daily, /etc/cron.hourly, /etc/cron.monthly, /etc/cron.weekly.
cat /etc/crontab # How to List Daily Cron Jobs ls -la /etc/cron.daily
- Place two: /etc/cron.d. It contains "anacron" and "e2scrub_all" on my Debian 11. See How to run a cron job as a specific user?
Running a cron job as a user
Some examples
- http://www.thegeekstuff.com/2009/06/15-practical-crontab-examples/
- https://crontab.guru/examples.html and tips
- Can I break a line cron command into multiple lines? No. Creating another script file in this case.
- A specific time
MIN HOUR DOM MON DOW CMD 30 08 10 06 * $HOME/full-backup >> $HOME/myscript.log 2>&1; echo "Executed at $(date)\n----------" >> $HOME /myscript.log # 30 – 30th Minute # 08 – 08 AM # 10 – 10th Day # 06 – 6th Month (June) # * – Every day of the week
- Twice a day
00 11,16 * * * /home/ramesh/bin/incremental-backup # 00 – 0th Minute (Top of the hour) # 11,16 – 11 AM and 4 PM # * – Every day # * – Every month # * – Every day of the week
- Every 10 minutes
*/10 * * * * /home/ramesh/check-disk-space
# Will only run on odd days: 0 0 1-31/2 * * command # Will only run on even days: 0 0 2-30/2 * * command
crontab
- How to Use Crontab to Automate Repetitive Tasks in Linux. Some examples: play sound, backup files, check if sites are online.
- 50 Amazing Linux Crontab Commands For The SysAdmins
crontab SOME-CRON-FILE; crontab -l
Make sure the .sh file gives a complete path. For example,
#!/bin/sh R --vanilla < arraytoolsip.R
does not work in cron job although it works perfect when we manually run it from the right path. The sh file should be
#!/bin/sh R --vanilla < $HOME/Dropbox/scripts/arraytoolsip.R
To disable everything on crontab -l, use one of the following methods:
- run crontab -e then comment out each line you don't want to run with #. OR
- run crontab -r to empty the current crontab. OR
- run crontab with no arguments, and then type Ctrl+D. It will create an empty crontab, overwriting your previous crontab.
GUI
Schedule Commands And Scripts In Linux With Zeit (GUI For Cron And At)
PATH and Shell
Cron knows nothing about your shell; it is started by the system, so it has a minimal environment. If you want anything, you need to have that brought in yourself. For example, to use 'ifconfig' command, I need to give it a complete path in my script file.
$ cat syncIP
/sbin/ifconfig eth0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}'
and the cron job
06 15 * * * /home/MYUSERNAME/Ubuntu\ One/syncIP > $HOME/Ubuntu\ One/ip.txt 2>&1
See here on how to add environment variable into cron environment.
Disable mail alert
If something went wrong with executing a cron job, cron will output a message "You have new mail in /var/mail/$USER". You can open this file using a text editor. To disable this alert, run 'crontab -e (see this post)
0 1 5 10 * /path/to/script.sh >/dev/null 2>&1 # OR 0 1 5 10 * /path/to/script.sh > /dev/null
Another way is to add MAILTO="" at the top of the crontab file.
Run a command at boot
- How to Execute a Command or Script at Reboot or Startup
- Run Script or Command at Reboot or Startup Using /etc/rc.d/rc.local File
- Execute a Command or Script at Reboot or Startup Using the crontab
- Run a Command or Script at Reboot or Startup Using the systemd service unit
- Sending an email on boot
@reboot sleep 300 && python /home/pi/startup_mailer.py
- pausing for 10 seconds before running the python script
@reboot /bin/sleep 10 ; /usr/bin/python /home/pi/PumpkinPi/src/pumpkinpi.py &
rc.local
How to enable rc.local shell script on systemd while booting Linux system
Running crontab as root
Use sudo crontab -e to edit. After saving it, no need to initialize it. Use sudo crontab -l to list the cron job.
Display and back up cron jobs
Linux List / Display and view all cron jobs
Check log
sudo grep CRON /var/log/syslog --color
Anacron
Anacron keeps track of the last time a task was run, and if it was missed, it runs it.
Anacron typically runs daily, while cron runs every minute.
cat /etc/anacrontab
GUI cron
md5sum
Linux md5sum Command Explained For Beginners (5 Examples)
How to verify files?
md5sum file1.txt file2.txt file3.txt > hashes md5sum --check hashes
fsck
- Repairing Linux ext2 or ext3 or ext4 File System (fsck)
- 10 Linux Fsck Command Examples to Check and Repair Filesystem
- How to Use ‘fsck’ to Repair File System Errors in Linux
- How to Repair Hard Disks with fsck on macOS
- How To Use Fsck Command To Check And Repair Linux File Systems (2024 Update)
Fsck error on boot
fsck error on boot: /dev/sda6: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY
This happened when I resize an Ubuntu partition.
fsck -fy /dev/sda1
Force fsck on the Next Reboot or Boot Sequence
https://www.linuxuprising.com/2019/05/how-to-force-fsck-filesystem.html
https://www.cyberciti.biz/faq/linux-force-fsck-on-the-next-reboot-or-boot-sequence/ Create a blank file /forcefsck and fsck will check your drive next time your reboot.
sudo touch /forcefsck
The fsck was used to fix a journal checksum error on a USB drive which has been formatted as Ext4 was used on a security camera application; see motionEyeOS.
Can I run fsck or e2fsck when Linux file system is mounted?
https://www.cyberciti.biz/faq/can-i-run-fsck-or-e2fsck-when-linux-file-system-is-mounted/
What is the difference between fsck and e2fsck?
Swap partition
- Swap partition and swap file
# mkswap /dev/sdxy # swapon /dev/sdxy # nano /etc/fstab
UUID=device_UUID none swap defaults 0 0
- How to Create a Swap File on Linux
- How to Permanently Disable Swap in Linux
Swap file vs swap partition
- Jetson Nano – Run on USB Drive
- If "htop" or "free -h" shows swap but "Disks" does not show a swap partition, it is likely the system is using a swap file instead of a dedicated swap partition. Use sudo swapon --show to check for the presence of a swap file or check /etc/fstab to see the a line referencing a file rather than a partition.
Mount drive, add a new hard drive
/etc/fstab and blkid
See Fstab.
autofs, /etc/auto.master
Building a network attached storage device with a Raspberry Pi
Fix a malfunctioning USB device or port
5 Ways to Fix a Malfunctioning USB Device or Port on Linux
Check the physical health of a USB stick
Linux check the physical health of a USB stick
USB drive
- Rename USB drive partition label. It seems if a device does not have a label, Ubuntu will use its 32-digit UUID as the mount point (eg. /media/$USER/$Long_UUID). This is very cumbersome. To fix that, we can open the Disks utility and select the partition of the device. Click the two-gear icon and pick Edit Filesystem... where we can change the filesystem label. After that, we can reject the USB and re-plug it to see the new mount directory based on the new label we specified (/media/$USER/$Label). We can also use the command lsblk (no sudo needed) to check.
- If I use "GParted" utility to check the "partition name ", it is not the same as the name I just specified through the "Disks" utility. But the "Information" window give a complete data. It is a little confusing that the partition label becomes the filsystem label and the Partition name shown on GParted was different & seems not to be used.
Run the following to confirm the USB device is detected.
sudo fdisk -l # OR dmesg | grep -i "SCSI"
Now suppose the usb device is found in dev/sdb1.
sudo mkdir /mnt/usb sudo mount -t vfat -o rw,users /dev/sdb1 /mnt/usb
The above mount command assumes the usb drive has Windows vfat partition and users give non-root users the ability to unmount the drive. If the USB drive is partitioned linux ext2/3, we can merely run mount command as
sudo mount /dev/sdb1 /mnt/usb
At the end, run umount command like
sudo umount /mnt/usb
To make the mounting automatically, edit the file /etc/fstab.
/dev/sdb1 /mnt/usb vfat defaults 0 0 /dev/sdb2 /mnt/usb2 ext3 defaults 0 0 UUID=XXXXXXXXXX /mnt/usb3 ntfs-3g rw 0 0
and run
sudo mount -a
Mount an iso file
sudo mkdir -p /mnt/mount_point # create a mount point sudo mount -o loop /home/user/disk.iso /mnt/mount_point mount # verify
Simple way of Sharing files between Ubuntu 16.04 and Windows 10 by using open-source NitroShare which is based on Qt framework.
NTFS usb drive in xubuntu
http://xflinux.blogspot.com/2011/01/mount-ntfs-volumes-automatically-in.html
sudo apt-get install ntfs-config
Now go to Applications>> System>> Ntfs Configuration Tool
Expand the "Advanced Configuration" and select all those partitions you want to be auto mounted and writable( The tool will detect all partitions at its startup).
Make sure the " Enable write support for internal devices" option is selected. Now click Close.
Many drives, one folder
- mhddfs program.
Partition tables
Partition Tables and the Dangers of Editing Them
parted command
How to partition a disk in Linux
Recommended partition schemes
HOME /home directory
How to Move Your Linux home Directory to Another Drive
/var directory filled up
How to move /var directory to another partition
blkid | grep sdc1 # get UUID mkdir /mnt/newvar mount /dev/sdc1 /mnt/newvar df -h /mnt/newvar rsync -aqxP /var/* /mnt/newvar # q=quiet,x=one-file-system, P=partial,progress umount /mnt/newvar/ /mnt/var/ nano /etc/fstab # UUID=XXXX /var ext4 defaults 0 2
Reboot
Why put things other than /home to a separate partition?
Why put things other than /home to a separate partition?
The /var partition is used by Docker and Apache.
/home, /boot and /var partitions can be separated.
Process/job
ps and top commands
How to Use the ps Command to Monitor Linux Processes, Linux how long a process has been running?
ps -C shutter # Listing only a Process by Command
# Adding the 'watch' command to show the process in real-time
ps -C dd --format pid,cmd,%cpu # Show PID, CMD and %CPU
sudo ps -p {PID} -o pid,cmd,lstart,etimes,etime
ps -e | less # Listing Process for All Users
ps -eH --forest | less # hierarchy
ps -e | grep firefox # Listing Processes by Name
ps -p 3403 # Listing Processes by Process ID
ps -Flww -p THE_PID # Listing a process detail like "htop" gives
ps -u mary # Listing Processes Owned by a User
sudo pkill top # Killing Processes by Name
sudo killall top # Killing Multiple Processes by Name
Kill a process and the pstree command
- https://en.m.wikipedia.org/wiki/Pstree
- On Ubuntu docker container, we can need to run apt install psmisc to get the pstree command.
- Killing a process and all of its descendants. This covers a PPID, PID and more importantly PGID, SID. Also ps j -A command can show these IDs for the running processes.
$ tail -f /var/log/syslog | grep "CRON" & $ ps j $ kill -SIGTERM -- -($Some_PGID)
- pgrep & kill
# find the PID pgrep ProgramName # Kill the ProgramName process kill -9 PID
- killall. For example, if Firefox is acting up (as Firefox will do from time to time) simply type killall firefox and it should kill the application completely.
In the rare circumstances that this doesn’t work you can always type xkill and then click on the window that won’t close; this will completely close a given window immediately. See this. To kill a privileges process, use for example sudo killall crond.
sudo killall -u USERNAME
- How To kill An Inactive OR Idle SSH Sessions. The pstree -p command can show a tree diagram of all the processes.
- pkill command. For example, pkill gedit.
Kill a process running on a specific port
Kill a Process Running on a Specific Port in Linux (via 4 Methods)
How to Kill Zombie Processes on Linux
How to Kill Zombie Processes on Linux
Simulate/produce high cpu load
How can I produce high CPU load on a Linux server?
# method 1: sudo apt install stress stress --cpu 3 # method 2: for i in 1 2 3 ; do while : ; do : ; done & done jobs # list background jobs for i in 1 2 3 4; do kill %$i; done # kill "job" (not "PID") 1,2,3,4
ps, pgrep and pidof: How much resource is used by a process
Find the process ID first by ps -ef | grep APPLICATIONAME where "-e" is to show the running processes and "-f" is for a full listing. Then
ps -p <pid> -o %cpu,%mem,cmd
For example,
$ ps -ef | grep akregator brb 15013 1942 1 10:41 ? 00:00:05 akregator --icon akregator -caption Akregator brb 15186 24045 0 10:50 pts/11 00:00:00 grep --color=auto akregator $ ps -p 15013 -o %cpu,%mem,cmd %CPU %MEM CMD 1.0 0.8 akregator --icon akregator -caption Akregator
pgrep
08:49AM ~$ ps -ef | grep firefox brb 7798 7778 0 08:49 pts/2 00:00:00 grep --color=auto firefox brb 25486 24869 0 Sep10 ? 00:42:48 /usr/lib/firefox/firefox brb 25612 25486 0 Sep10 ? 00:19:49 /usr/lib/firefox/firefox ..... 08:49AM ~$ pgrep firefox 25486
pidof
08:49AM ~$ pidof firefox 27951 25961 25612 25486 08:51AM ~$ pidof /usr/lib/firefox/firefox 27951 25961 25612 25486 $ kill $(pidof firefox)
How to keep resource usage low
Avoid concurrency
Avoid These Problems By Limiting Bash Scripts to Run Once At A Time. pgrep, lsof and flock commands.
All You Need To Know About Processes in Linux
http://www.tecmint.com/linux-process-management/
wait command and background jobs
The wait command in Linux is a shell built-in command that pauses the execution of a shell script until all background jobs or specified JobID/PIDs terminate and return their exit status.
# Example 1: Wait for all background processes to finish command1 & command2 & wait echo "All background processes have finished." # Example 2: Wait for a specific process to finish command1 & PID=$! command2 & wait $PID echo "Command1 has finished."
run commands in a background and allow log off
nohup /path/to/script >output 2>&1 &
Or to disable output and be more safe. It also explains the concept of file descriptor/fd in Unix.
nohup command </dev/null >/dev/null 2>&1 &
See also Anonymous named pipe.
See also How to Setup OpenVPN on Fedora 24+ where we use nohup openvpn ... & to start the connection in the background and also make it not terminate on exiting the terminal.
job: How do I send an already-running process into the background
- How to Run Linux Commands in the Background
- http://stackoverflow.com/questions/625409/how-do-i-put-an-already-running-process-under-nohup
- bg Command Examples
- How to Run and Control Background Processes on Linux
- Running Linux Commands in Background and Foreground
Steps:
- 'Ctrl+Z' to stop (pause) the program and get back to the shell. It sends SIGTSTP to a foreground application.
- bg to run it in the background.
- jobs -l to get the jobID and process ID
- disown -h [job-spec] where [job-spec] is the job number (like %1 for the first running job; find about your number with the jobs command) so that the job isn't killed when the terminal closes.
Stopped job
A stopped job is one that has been temporarily put into the background and is no longer running, but is still using resources (i.e. system memory). Because that job is not attached to the current terminal, it cannot produce output and is not receiving input from the user.
- jobs -s showing stopped jobs
- jobs -l showing the job PID
Send kill to a stopped job, it will do nothing but queue than bring it in in foreground, it will terminate. So don't repeatly sending a 'kill' command.
- fg %1 move the stopped job ID #1 to the foreground (works)
- kill %1 # kill job ID #1
- kill 12345 # kill job PID 12345
- kill -9 `jobs -ps` may not work
- kill -9 $(jobs -p)
nice
How to Set Process Priorities With nice and renice on Linux
watch command
Watching activity on Linux with watch and tail commands. Both the watch -n and tail -f commands can provide auto-updating views of information/
We can use the watch command to monitor a specific process such as the progress of the dd command.
Terminal 1
watch -n 10 who watch ps -C dd --format pid,cmd,%cpu
Terminal 2
dd if=/dev/zero of=/dev/null
Use --color for color output.
while + sleep
How To Repeat A Command Every X Seconds On Linux: watch, while + sleep,
ulimit
- How to find ulimit for user on Linux
- Setup a Completely Unlimited limits.conf Configuration for Testing Servers
Notepadqq - Notepad++-like editor
Notepadqq. It is written using Qt. It does not have printing function:(
Note apps that can sync
Top 8 Notepad Apps for Linux That You Can Sync. Some are compatible with Evernote.
Evernote
Evernote alternative
How to Install Turtl Server - Evernote Alternative - on Ubuntu 16.04
Backup/restore Evernote
https://www.makeuseof.com/tag/backup-restore-evernote/
Markdown
Preview markdown/view markdown offline
- Atom text editor has a built-in function to preview HTML or markdown files. Menu -> Packages -> Markdown Preview -> Toggle Preview.
- http://stackoverflow.com/questions/9843609/view-markdown-files-offline. Grip works fine.
sudo pip install grip grip readme.md
- For image, see http://stackoverflow.com/questions/13051428/how-to-display-images-in-markdown-files-of-github. The trick is adding ?raw=true after the image name.
# title 1 
- Chrome markdown preview plus extension does not show images from github.
Markdown editor
nano/pico editor
The nano editor is also called pico in R. See ?edit in R.
vi editor
Cloud
Boot
U-boot
http://www.denx.de/wiki/U-Boot
Pandora linux client
COW (copy on write) file system
tmpfs and /dev/shm
- Storing Files/Directories In Memory With tmpfs
- What Is /dev/shm And Its Practical Usage
- Use a RAM Disk to Improve Disk Access Times
- How to create RAM disk in Linux
- How to Enlarge tmpfs Space in Linux, RAM Drive vs tmpfs instance
Apache redirection
http://cran.r-project.org/mirror-howto.html
Redirect a Website URL from One Server to Different Server in Apache
Important linux directories
Filesystem Hierarchy Standard, Wikipedia
- /bin - executables used by the base system
- /boot
- /dev
- /etc - configuration files
- /media
- /mnt
- /opt - optional application packages
- /proc - process information only. Access Kubernetes Objects Data From /Proc Directory
- /sbin - critical executables for running the system, but should be used by superuser
- /usr - user programs and data. For example /usr/bin contains most of the libraries used by apps. /usr/share/ contains Architecture-independent data (eg some pretty images are located in /usr/share/backgrounds)
- /var - variable data such as databases, mails spools and system logs.
Difference of /bin, /sbin, /usr/local/bin, ...
- /bin : For essential binaries; e.g. bash, cat, ls.
- /sbin : is similar to /bin but for scripts with superuser (root) privileges required; e.g. shutdown command is located here. Local users have to use sudo to run binaries here.
- /usr/bin : Same as first, but for general system-wide & non-essential binaries; e.g. grep, zip, docker, etc.
- /usr/sbin : Same as above, but for scripts with superuser (root) privileges required.
- /usr/local/bin or /usr/local/sbin for system-wide available (personal) scripts. For example, install docker-compose is merely to download the binary and place it under /usr/local/bin/ directory.
If you want to create your own scripts and make them available to all users, you’re pretty safe adding them to /usr/local/bin. Or to add my scripts to my local bin (~/bin) and then I create a symbolic link in /usr/local/bin to the commands I want to make public. As a result, I can manage all my scripts from the same directory but still make some of them publicly available since /usr/local/bin is added to $PATH. See this post.
DHCP
DHCP server IP
Linux find DHCP server IP address using CLI
Get a New IP Address
dhclient -r # release your IP Address dhclient # get your DHCP to issue you a new IP Address based on how it’s been configured.
Open a file/URL using the default application from the command line
- gnome-open (works on Mint)
- kde-open (KDE users)
- xdg-open (window-manager independent). XDG stands for X Desktop Group; see https://en.wikipedia.org/wiki/Freedesktop.org
See also
- How does Linux choose which application to open a file?
- How to Change Your Default Applications on Ubuntu: 4 Ways
Check a file's encoding
file -bi myfile
For example,
file -bi Downloads/hmv_.rc # text/x-c++; charset=utf-16le
Know you system using the command line
Alerting and visualization tools
5 alerting and visualization tools for sysadmins
System monitor tools (TUI)
glances command: more than htop
- https://nicolargo.github.io/glances/ and its Documentation. It seems to be more stable to install via apt command instead of the bash script. There is no need to use 'sudo' to run the command.
- Glances includes Disk I/O, Network I/O, internal and external IPs, current time, et al. Its official website at http://glances.readthedocs.io/en/latest/.
- Glances uses online services to grab the external IP address. https://github.com/nicolargo/glances/issues/961
- To hide the internal and external IPs, type "I" (capital).
- To disable the IP, type "--disable-ip" (not available in v3.x.x ?)
- It shows a warning or critical alert (e.g. memory high usage) at the bottom of the screen.
- My command: glances -t 5 # update every 5 seconds
- https://www.tecmint.com/glances-an-advanced-real-time-system-monitoring-tool-for-linux/
- Monitoring & Troubleshooting Basics with Glances (video)
Glances is similar to htop but it provides network stats and disk usage too. It also supports web UI. Install it by sudo apt-get install glances.
conky and autostart
- Ubuntu > Conky (internal link)
- Raspberry Pi case (internal link)
For auto start on Lubuntu, see How can I add new autostart programs in Lubuntu?
On Lubuntu 18.04, add the path to the application to ~/.config/lxsession/Lubuntu/autostart
top and htop command
- 15 simple TOP command examples on Linux to monitor processes
- M (capital): sorted by memory
- P: sorted by CPU
- T: sorted by running time
- c: full command path
- u: filter by user
- V: forest/tree mode; parent child hierarchy. This will disable 'M', 'P' or 'T'.
- f: show/hide columns
- top -d 5 change the refresh interval from 3 seconds to 5 seconds
- Why are some processes highlighted in top?
- Htop command has a screen help. htop command in Linux
- Tree view. Press 't'. Press '+' or '-' to expand or collapse.
- ">" to change to sort view.
- Shortcut character keys and function keys
- Understand Linux CPU Load. The maximum number should be the number of cores.
- In Armbian, htop shows CPU temperature and frequencies; see a screenshot.
nmon
- https://en.wikipedia.org/wiki/Nmon. The stats are saved to a comma-separated values (CSV) data file for later graphing and analysis
- HomePage
- Monitoring with Nmon
CPU frequency
How To Optimize Performance And Battery Life With Auto-cpufreq In Linux
What is the correct way to view your CPU speed on Linux? (x86)
watch -n.1 "cat /proc/cpuinfo | grep \"^[c]pu MHz\""
How can I get the current CPU frequency of an ARM processor on Ubuntu?
ls /sys/devices/system/cpu/cpu0/cpufreq/
scout_realtimep
This is used by Dataplicity
gtop command (100% Javascript)
https://www.cyberciti.biz/howto/gtop-awesome-system-monitoring-dashboard-for-terminal/.
Pros:
- CPU history graph in time
- Memory history graph in time (not useful)
- Network bandwidth usage is real-time. It is accurate as what nload gives.
- Percentage usage of memory, swap, disk usage
- Top processes
$ docker run --rm -it \
--name gtop \
--net="host" \
--pid="host" \
aksakalli/gtop
$ sudo apt install npm nodejs
$ npm install gtop -g
$ gtop
- Press p to sort by process ID (PID).
- Press c to sort by CPU usage.
- Press m to sort by memory usage.
It can be installed on Linux Mint 18.2 but not in Ubuntu 14.04 or raspbian (9 stretch).
$ npm install gtop -g npm http GET https://registry.npmjs.org/gtop npm http GET https://registry.npmjs.org/gtop npm http GET https://registry.npmjs.org/gtop npm ERR! Error: CERT_UNTRUSTED npm ERR! at SecurePair.<anonymous> (tls.js:1370:32) npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17) npm ERR! at SecurePair.maybeInitFinished (tls.js:982:10) npm ERR! at CleartextStream.read [as _read] (tls.js:469:13) npm ERR! at CleartextStream.Readable.read (_stream_readable.js:320:10) npm ERR! at EncryptedStream.write [as _write] (tls.js:366:25) npm ERR! at doWrite (_stream_writable.js:223:10) npm ERR! at writeOrBuffer (_stream_writable.js:213:5) npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:180:11) npm ERR! at write (_stream_readable.js:583:24) npm ERR! If you need help, you may report this log at: npm ERR! <http://github.com/isaacs/npm/issues> npm ERR! or email it to: npm ERR! <[email protected]> npm ERR! System Linux 4.4.0-119-generic npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "gtop" "-g" npm ERR! node -v v0.10.25 npm ERR! npm -v 1.3.10
gotop
A terminal based graphical activity monitor inspired by gtop and vtop. It is quite beautiful.
Gotop – Yet Another TUI Graphical Activity Monitor, Written In Go
Compared to gtop, it has a temperature monitor. However, it can only show the average CPU usage (one line) on my Xeon computer.
git clone --depth 1 https://github.com/cjbassi/gotop /tmp/gotop /tmp/gotop/scripts/download.sh sudo cp gotop /usr/local/bin; rm gotop gotop
Note the temperatures do not show up in Raspbian (raspberry pi 3 b+).
termui: Golang terminal dashboard
https://github.com/gizak/termui
Bashtop and btop
- Bashtop Is A Cool Linux Resource Monitor Written In Bash
- *btop 20k star. C++ version and continuation of bashtop and bpytop. This is very similar to bashtop but it also shows the IP. Mac, Linux x86, armv7, aarch64, arm64, armv5. Install btop++ on Raspberry Pi, Ubuntu 22.04.
bottom
S-tui command
Monitor Linux CPU temperature, frequency, power in a graphical way
below
below: a time traveling resource monitor
System monitor tools (GUI)
Comparisons:
- https://www.tecmint.com/category/monitoring-tools/
- https://www.tecmint.com/linux-performance-monitoring-tools/
- https://linoxide.com/monitoring-2/linux-performance-monitoring-tools/
- http://www.linuxscrew.com/2012/03/22/linux-monitoring-tools/
- https://www.infoworld.com/article/2683857/network-monitoring/article.html#slide2
- http://www.thegeekstuff.com/2011/12/linux-performance-monitoring-tools
Some lists:
- Install Ganglia on Ubuntu 16.04 Server (Xenial Xerus)
- Linux Dash Web based monitoring tool. Source code is on github.
- Monitorix and on Ubuntu16.04
Linux-Dash
https://github.com/afaqurk/linux-dash. Not working when I tested on RPi and Ubuntu.
Nagios
Zabbix
Munin and Monit
Server Monitoring with Munin and Monit on Ubuntu 16.04 LTS
Cacti
- http://terraltech.com/monitoring-with-cacti/
- https://www.howtoforge.com/tutorial/install-cacti-on-debian-9/
sysstat, sar
# CPU sar 2 10 # every two seconds, 10 times # Memory sar -r # look at the kbcommit and commit columns sar -r -f /var/log/sysstat/sa02
- https://www.maketecheasier.com/monitor-linux-performance-with-sysstat/
- https://www.tecmint.com/install-sysstat-in-linux/
- http://www.thegeekstuff.com/2011/03/sar-examples/
- https://www.blackmoreops.com/2014/06/18/sysstat-sar-examples-usage/
- Visualize sar data with kSar.
- export LC_ALL=C. This will convert date/time. For example, 12:00:01 AM will become 00:00:01 and 12/09/2017 will become 12/09/17.
- sar -A -f /var/log/sysstat/saXX > ~/Downloads/sardata.txt.
- Click on Data -> Load from text file. Select ~/Downloads/sardata.txt file. Note that nothing will happen in the kSar GUI.
- Click 'kSar' to show the tree.
- kSar (depends on JDK) for graphics (instead of usinsg the sadf command).
- Download and unzip it to ~/bin.
- Execute bash ~/bin/kSar-5.0.6/run.sh.
- On the GUI, click Data -> Run local command.. -> sar 2 10, for example. This will start to record the cpu usage 10 times with a 2 seconds interval.
- You can view the real-time plot (shown on the right panel) by clicking kSar -> CPU -> CPU all (left panel).
Stacer
Linux System Optimizer and Monitoring
Prometheus
- https://prometheus.io/docs/introduction/overview/
- How to Install Prometheus System Monitoring Tool on Debian 11
Curses and ncurses: TUI library
Position text on your screen in Linux with ncurses
Calcurse
Keep up with your calendar and to-do list with Calcurse
Bitbucket (free for 5 users)
- Unlimited private repos
- Code reviews
- JIRA integration
- REST API
- Custom domains
See this post to know how to fix the problem of unknown author. In short, when I uncheck "Use global user setting" from Repository-> Repository Settings -> Advanced does the commit author change as expected.
See here for a list of Android apps related to bitbucket.
Image
See Images.
GIMP
Reload/Refresh .profile file
. ~/.profile
. is a bash builtin and a synonym for source, see man bash.
History of commands
history command with date and time
Running the following code once and history will give date and time the next time you issue the history command.
echo 'export HISTTIMEFORMAT="%Y-%m-%d %T "' >> ~/.bashrc
Note that the original post asks to write the line to ~/.bash_profile but this is not working in the desktop environment.
Note that on zsh, the above method is not supported. We can use history -i instead. Type man zshoptions or man zshbuiltins for more information.
Bang bang - Run a command/Fetch parameters from previous history
- http://unixhelp.ed.ac.uk/shell/tcsh_hist3.html
- http://codytaylor.org/2009/09/linux-bang-commands.html
- ^P: Move up through the command history list one command at a time.
- ^N: Move down through the command history list one command at a time.
- !!: Run the previous command. For example, we can run sudo !! in order to run the previous command with sudo.
- !n: Run command number n (useful)
- !string: Run most recent command starting with characters in string (useful). For example, !ls
- !?string: Run most recent command containing characters that match string
- !*: Fetch parameters from last command (useful). For example, if we run "ls /var" first. Then when we run stat !*, it would run stat /var. Or we can run cd !* and it will cd to /var directory.
- !_: Fetch the last parameter from last command. For example, if we run "ls /var/ /etc" first. Then when we run stat $_, it would run stat /etc. (zsh shell only)
- Ctrl + r and type a keyword (most useful). Press ctrl + r to scroll the match. This is called reverse i search.
For example,
!-1 !4 !tail
Recall commands with reverse-i-search
Ctrl + r. See Bash bang commands: A must-know trick for the Linux command line
To continue with the search, just hit Ctrl + r keys again.
To run a forward search, hit Ctrl + s. How to cycle through reverse-i-search in BASH?
Increase history limit
http://unix.stackexchange.com/questions/17574/is-there-a-maximum-size-to-the-bash-history-file Unlimited Bash History
$ echo $HISTSIZE # $HISTSIZE variable controls how much history is displayed 500 $ export HISTSIZE=1000 $ echo “HISTSIZE=1000” >> ~/.bashrc # $HISTFILESIZE variable controls how many commands are retained in your .bash_history file. $ echo $HISTFILESIZE 2000 $ wc -l .bash_history 2000 .bash_history
Not to add to bash history
Add a space after the command.
It is useful if there is a password in the command.
For example,
$ ls ~/ # this won't be recorded in history $ ls ~/Downloads/ # this will be recorded in history $ history
Delete a single command from history
- How to delete a single command from history on a Linux/Unix Bash shell
- Make Bash history more useful with these tips
To stop adding history entries, you can place a space before the command, as long as you have ignorespace in your HISTCONTROL environment variable.
You can force Bash to exclude commands starting with empty space by placing this in your .bashrc file:
export HISTCONTROL=$HISTCONTROL:ignorespace
How to Clear Bash History on Linux
$ cat /dev/null > ~/.bash_history && history -c && exit
multiple terminals
Use ‘’’history -a’’’ to write the history to the file if we do not plan to close the terminal. See How to use the history command on Linux.
Listen to HiChannel internet radio
Use Radio Tray
- http://endroid.blogspot.com/2012/02/listen-hichannel-radio-online-by.html
- http://abcde9990101.blogspot.com/2011/05/ubunturadio-tray.html
I use it to listen m3u file (VLC also supports it too).
Web Analytics Reporting Tools
- Piwik (open source). How to Install Piwik with Nginx on Ubuntu 15.10
- How to Install Matomo (formerly Piwik) Web Analytics on CentOS 7
- AWStats
- Google Analytics
Painting software
- Krita - professional painting program made by artists that want to see affordable art tools for everyone. Krita Is the Free GIMP Alternative You Should Be Using.
- Pinta. It can be install by apt-get command. It works just line Window's paint. Ctr + v to paste an image and save to a file. To crop an image, click the selection tool on the most left hand side (it's a black color on v1.6 but a gray color on v1.7), then select a rectangle. Now click 'Image' > 'Crop to Selection' to finish. Pinta New Release After 5 Years. Here’s How to Get it!
sudo add-apt-repository ppa:pinta-maintainers/pinta-stable sudo apt update sudo apt install pinta
- mtPaint. It is included in Odroid - xu4 - Lubuntu 14.04. To crop an image, just select an area and click Image > Crop. It can be used to take a screenshot from the desktop by using either the application or through the command line (mtpaint -s). It will then display the screenshot in the application if you use the command line.
- MyPaint
Take a screenshot (and edit them)
See Take screenshots.
Cozy - audiobook player
ebook readers
7 Best eBook Readers for Linux: Calibre, FBReader, Okular, Lucidor, Bookworm, Easy Ebook Viewer and Buka.
Calibre - Read ebook in epub format
See Calibre
RSS reader
Some references:
- 5 Best Feed Reader Apps for Linux
- 14 Best RSS Feed Readers for Linux in 2018. It contains nice screenshots.
Some examples:
- Fluent Reader. Open source. Linux, Windows, macOS.
- Akregator. KDE based. This is preinstalled in CentOS-KDE under the Internet category. It is also called 'Feed Reader'.
- QuiteRSS. It works on Linux, Windows and MacOS.
- Liferea. GTK based. It is considered one of the best RSS feed readers on Ubuntu Linux. It can synchronize with several online feed managers such as InoReader among others.
- FeedReader. Looks nice. Works with several online feed managers.
- Newsbeuter: RSS feed in terminal
- Newsboat: terminal. Newsboat: The Best Terminal-Based RSS Feed Reader for Linux. Not for general use since it assumes the articles are all text-based.
- RSSOwl. Depends on Java. Cross platform.
- Firefox and Thunderbird have built-in support for RSS.
Clear gibberish all over the screen
Just type “reset”. See BASH Fix Display and Console Garbage and Gibberish on a Linux / Unix / macOS. It is useful, for example, accidentally I run cat command over binary file.
Display/screen
Turn off/on your monitor via command line
- https://systembash.com/how-to-turn-off-your-monitor-via-command-line-in-ubuntu/
- http://askubuntu.com/questions/62858/turn-off-monitor-using-command-line
xset dpms force off # Press any key to turn it on xset dpms force on xset -q # check the status of the X server settings
If we want to turn off/on the screen via ssh, add
export DISPLAY=:0.0
first before calling the xset command, or use '-display' argument
xset -display :0.0 dpms force off xset -display :0.0 dpms force on
autoxrandr
Plug your laptop into different monitor setups. https://www.donarmstrong.com/posts/autorandr/
Move a window without clicking the titlebar
Hold down the Alt key and then click in the window anywhere, and move your mouse.
Add new screen/display resolutions
- http://ubuntuforums.org/showthread.php?t=1112186 (tested on UDOObuntu 2 beta 2running Ubuntu 14.04 + SainSmart 9" LCD display. The commands (not persistent) are
xrandr | grep maximum gtf 800 480 59.9 # give some output used in the following line xrandr --newmode "800x480_59.90" 29.53 800 816 896 992 480 481 484 497 -HSync +Vsync xrandr --addmode "DISP3 BG" 800x480_59.90 xrandr --output "DISP3 BG" --mode 800x480_59.90
I cannot find the file /etc/X11/xorg.conf in my UDOObuntu 2 beta 2. It seems this file does not exist anymore. See this post about how to re-create it.
Wayland
- echo $XDG_SESSION_TYPE to check whether X11 or Wayland is in use
- To change from wayland to X11:
- Method 1: log out, click the username, choose Xorg and type the password
- Method 2:
sudo nano /etc/gdm3/custom.conf
Change the line WaylandEnable=true to WaylandEnable=false And restart the system1.
- Ubuntu 22.04 LTS with GNOME 42 and Wayland as the default
- Linux on the GPD Pocket 2 (Ubuntu, Debian, and Fedora)
- “xrandr -o right” command can be used to rotate the screen
- But it does not work in Fedora because Fedora uses the Wayland display server rather than xserver.
- How To Automate Repetitive Typing (Text Expansion) With AutoKey On Linux. AutoKey is an X11 application, it won't work properly if you use Wayland (e.g. Fedora uses Wayland by default; if you want to use Xorg you must logout, and select "GNOME on Xorg" from the cog icon next to the Sign In button).
- How to Configure Xorg as Default GNOME Session in Fedora
- How to know whether Wayland or X11 is being used
loginctl show-session $(awk '/tty/ {print $1}' <(loginctl)) -p Type | awk -F= '{print $2}'
export DISPLAY
- What is the $DISPLAY environment variable?
The value of the display environment variable is:
hostname:displaynumber.screennumber
- If we want to run a GUI app on a remote computer (such as Raspberry Pi/Beaglebone Black) and show the GUI app on the remote computer's screen using ssh, we can issue the following command before running the app.
export DISPLAY=:0.0
See which groups you belong to, id & group commands
id <username> groups groups <username>
Main Types of User Accounts on Linux
The 4 Main Types of User Accounts on Linux:
- Superuser Account
- Regular Accounts
- System Accounts (Core OS functionality, eg root, bin, sys, daemon) awk -F: '$3 <= 100 {print $1, $3, $7}' /etc/passwd Note that the third field is the user ID (UID). System accounts generally have UIDs below 100 (though this threshold may vary slightly depending on your distribution).
- Service Accounts (eg SQL, apache).
List all user groups
How to List All User Groups on Linux To see a list of groups
cat /etc/group # OR getent group
Note that "getenv group" Lists groups from all configured sources, including both local and networked sources, depending on your system’s NSS (Name Service Switch) configuration (in /etc/nsswitch.conf).
finger: show user information
finger USERNAME
groupadd, chgrp, usermod, ACL (access control lists)
- Linux Add User To Group Using Command-Line: useradd
- Create a Shared Directory for All Users in Linux
- Assign Read/Write Access to a User on Specific Directory in Linux
- Linux Chgrp Command for Beginners (5 Examples)
sudo mkdir -p /var/www/reports/ sudo groupadd project sudo usermod -a -G project tecmint sudo chgrp -R project /var/www/reports/ sudo chmod -R 2775 /var/www/reports/
create more system users and add them to the directory group as follows:
sudo useradd -m -c "Aaron" -s/bin/bash -G project aaron sudo useradd -m -c "John" -s/bin/bash -G project john sudo useradd -m -c "Ravi" -s/bin/bash -G project ravi sudo mkdir -p /var/www/reports/aaron_reports sudo mkdir -p /var/www/reports/john_reports sudo mkdir -p /var/www/reports/ravi_reports
See who's in a group
grep '^group_name_here:' /etc/group finger USERNAME # See more detail about a user
Add a standard user to sudo group
This is useful on Debian distribution where a new user does not have the sudo power. First log in as root,
# usermod -aG sudo username
http://www.cyberciti.biz/tips/linux-shared-library-management.html
- ldconfig : Updates the necessary links for the run time link bindings.
- ldd : Tells what libraries a given program needs to run.
- ltrace : A library call tracer.
- ld.so/ld-linux.so: Dynamic linker/loader.
Log files
$ ls -lt /var/log
ssh log files: /var/log/auth.log
- /var/log/syslog: it is useful to use tail -f /var/log/syslog to show the log in real time
- /var/log/auth.log: it includes ssh log in information and lots of CRON sessions opened and closed every minutes.
And
- Why do I see a CRON session opening and closing every hour in /var/log/auth.log?
- remove cron from /var/log/auth.log
- What the %$#@ is pam_unix (cron:session) doing every ten minutes? (/var/log/auth.log)
How to Monitor Log Files in Real Time in Linux (Desktop and Server) . sudo apt install lnav; sudo lnav
Apache log
- /var/log/apache2/error.log (small 83K). Useful to troubleshoot errors/crashes of Apache.
grep "May 08" /var/log/apache2/error.log
- /var/log/apache2/access.log (large 10M)
/var/log/maillog
Logrotate
How to Setup Logrotate on Linux (to Keep Your Server from Running Out of Space)
uprecords command
uptime command
uptime watch -n 60 uptime
Windows
- How to check PC uptime using Command Prompt
- 4 Ways to Check the System Uptime in Windows 10
- How to find the Windows system uptime using the command line
Find out from the logs what caused system shutdown?
How to find out from the logs what caused system shutdown?
Get notified when a system is rebooted
Get Notified When Your Raspberry Pi is Booted with Pushbullet. This makes use of /etc/rc.local file. If it does not work, we can use cron to run a command at startup; see sending an email on boot.
timeout command
- https://www.howtoforge.com/linux-timeout-command/
- https://www.howtogeek.com/423286/how-to-use-the-timeout-command-on-linux/
Linux command similar to top to show hard disk activity
Use iotop. On ubuntu, we can use sudo apt-get install to install it. Use sudo iotop to launch it. Use -o to show processes that are actually doing IO.
sudo apt-get install iotop sudo iotop -o -u $USER
Another program is iostat and the -d (disk) option. The -x option will display extension I/O status.
sudo apt-get install sysstat iostat -dx 5 # every 5 seconds
24 iostat, vmstat and mpstat Examples for Linux Performance Monitoring
curl and wget
See Curl.
Torrent
Torrench
Torrench: How To Search And Download Torrent Files Using Terminal (Linux, Mac, Windows)
aria2 - command line downloader supports torrents and multi-connection
The -x argument helps a little bit. Download a file 112MB; see https://www.archlinux.org/download/
$ time aria2c http://mirror.jmu.edu/pub/archlinux/iso/2016.11.01/archlinux-bootstrap-2016.11.01-i686.tar.gz # 16 seconds $ time aria2c -x10 http://mirror.jmu.edu/pub/archlinux/iso/2016.11.01/archlinux-bootstrap-2016.11.01-i686.tar.gz # 11 seconds
Axel
It can create an unlimited number of worker threads to download any kind of data. See https://www.beginnersheap.com/top-5-command-line-download-accelerators-linux/
lftp
- It supports FXP (site-to-site transfers) and dropping to background
- How to use lftp to accelerate ftp/https download speed on Linux/UNIX. It can launch several commands in parallel in the background.
Apply a patch to source code
- http://www.cyberciti.biz/faq/appy-patch-file-using-patch-command/.
- http://www.thegeekstuff.com/2014/12/patch-command-examples/
- How to Apply a Patch to a File (and Create Patches) in Linux
For example Tophat 2.0.12 compatibility with Samtools 1.0,
$ ls support_for_tophat_1.patch tophat-2.0.12 tophat-2.0.12.tar.gz $ grep -r -i "check_samtools" tophat-2.0.12/ tophat-2.0.12/src/tophat.py:def check_samtools(): tophat-2.0.12/src/tophat.py: check_samtools() $ cp support_for_tophat_1.patch tophat-2.0.12/src/ $ cd tophat-2.0.12/src/ $ patch tophat.py < support_for_tophat_1.patch patching file tophat.py Hunk #1 succeeded at 1540 (offset 3 lines). Hunk #2 succeeded at 1563 (offset 3 lines).
IP address fundamental
http://www.howtogeek.com/133943/geek-school-learning-windows-7-ip-addressing-fundamentals/.
There are three classes for private IP ranges.
- 1-126.0.0.0 from Class A
- 128-191.0.0.0 from Class B
- 192-223.0.0.0 from Class C
Gateway IP address
How To Find Default Gateway IP Address In Linux And Unix From Commandline
Get internal IP address
$ hostname -I $ ifconfig # also works on Android through Termux
Private/internal/local IP/network
https://en.wikipedia.org/wiki/IPv4#Private_networks
- 10.0.0.0 – 10.255.255.255
- 172.16.0.0 – 172.31.255.255
- 192.168.0.0 – 192.168.255.255
There are other special-use addresses for private networks
- 100.64.0.0 to 100.127.255.255 (similar to private IP addresses like 192.168.x.x or 10.x.x.x)
Get external IP address
- http://ifconfig.me
- Simple IPv4
- Command for determining my public IP? wget -qO- https://ipecho.net/plain ; echo
- How to find your IP address in Linux
- 5 Commands to Find the IP Address of a Domain in the Linux Terminal
- How to Find the Public IP Address on a Linux System
It seems there is no way to get the external IP address without not using external services.
host myip.opendns.com resolver1.opendns.com # Look for the bottom line output myip.opendns.com has address XXX.XX.XX.XXX # https://unix.stackexchange.com/a/81699 sudo apt-get install dnsutils dig +short myip.opendns.com @resolver1.opendns.com # OR curl ifconfig.me # by google # also works on Android through Termux # curl http://ipecho.net/plain; echo # OR curl ipv4.ipogre.com
To store my IP in a shell variable
myip="$(dig +short myip.opendns.com @resolver1.opendns.com)"
echo "My WAN/Public IP address: ${myip}"
The above only gives the IP. The following method gives geo information too.
curl ipinfo.io # ifconfig.me, icanhazip.com, ipecho.net/plain, ifconfig.co
# OR give a specific IP (domain name does not work)
curl ipinfo.io/216.58.194.46
{
"ip": "216.58.194.46",
"hostname": "dfw25s12-in-f14.1e100.net",
"city": "Mountain View",
"region": "California",
"country": "US",
"loc": "37.4192,-122.0574",
"org": "AS15169 Google Inc.",
"postal": "94043"
}
IP geolocation
Test url: ubuntu.mirrors.pair.com
- https://iplocation.net
- command line - input is an IP address
curl https://ipapi.co/8.8.8.8/json/ curl https://ipinfo.io/8.8.8.8
- command line - input is a web address
IP_ADDRESS=$(dig +short example.com) curl https://ipapi.co/$IP_ADDRESS/json/
- https://www.ipligence.com/geolocation
- http://www.ipfingerprints.com/
- http://ip-api.com/ (it shows your IP, internal IP, OS, browser/user-agent, DNS server from outside?). The final query URL is simple; for example http://ip-api.com/#ubuntu.mirrors.pair.com
- ipstack. How to Use the IPStack API for IP Geolocation Lookups. 10,000 searches per month for free.
- The Free IP Geolocation API Is a Valuable Webmaster Tool
Domain, WHOIS
- WHOIS LOOKUP. For example, consider "r-pkg.org" domain,
- The 'Sponsoring Registrar' shows who is the sponsoring registrar (eg GoDaddy.com).
- The 'Registrant Name' shows who registered this domain.
- Command Line Interface.
sudo apt-get install whois whois r-pkg.org
- The 9 Best Websites to Look Up WHOIS Information for Free
- IP-Lookup. For example, consider "r-pkg.org" domain.
- The linux command line tool 'ping' shows the ip address.
- When we use the ip address to search in the IP-Lookup, the IP owner info > 'Organization' or 'OrgName' field shows the owner of this IP (eg Digital Ocean, Inc).
- The Domain owner info there gives the same (or less) information as WHOIS LOOKUP.
- WhoIsHostingThis or webhostinghero (the returned result will be like Amazon, GoDaddy, CloudFlare, Github, Verizon, etc).
Subnet
- Class A: 255.0.0.0 or /8;
- Class B, 255.255.0.0 or /16;
- Class C, 255.255.255.0 or /24.
For example, in the subnet 192.168.5.0/255.255.255.0 (192.168.5.0/24) the identifier 192.168.5.0 commonly is used to refer to the entire subnet.
In the /16 subnet 192.168.0.0/255.255.0.0, which is equivalent to the address range 192.168.0.0–192.168.255.255
IPv6
- IPv6
- Testing IPv6 Networking in KVM: Part 1
- Should You Care About IPv6 Connectivity For Your Web Server?
ping command
ping uses the ICMP Echo Message to force a remote host to echo a packet back to the local host. If packets can travel to and from a remote host, it indicates that the two hosts can successfully communicate.
Check if a server is up/down if pining to that machine is not allowed
Hint: use nmap to scan open ports
nmap -PS22 -p22 <hostname> # port 22 only # OR nc -z -v <hostname> 22
Shell script to check whether a server is reachable?. I found if a server is up and I am trying a closed port, it will take about 2 minutes before it returns.
if nc -z $server 22 2>/dev/null; then
echo "$server ✓"
else
echo "$server ✗"
fi
6 Methods to Quickly Check if a Website is up or down from the Linux Terminal
Check if Sites are Online using a PHP script
How to Use Crontab to Automate Repetitive Tasks in Linux
Build a home network
- Virtual router using vSphere. (Good ! It includes an introduction to vSphere installation)
- http://rbgeek.wordpress.com/2012/05/14/ubuntu-as-a-firewallgateway-router/
IP Subnet Calculator
https://www.dan.me.uk/ipsubnets?ip=10.0.0.0
- CIDR block IP range (network - broadcast) Subnet Mask IP Quantity
- 10.0.0.0/24 10.0.0.0 - 10.0.0.255 255.255.255.0 256
- 10.0.0.0/16 10.0.0.0 - 10.0.255.255 255.255.0.0 65536=256^2
- 10.0.0.0/8 10.0.0.0 - 10.255.255.255 255.0.0.0 16777216=256^3
How Use Your Router and ISP’s Modem/Router Combo In Tandem
http://www.howtogeek.com/255206/how-use-your-router-and-isps-modemrouter-combo-in-tandem/
Troubleshoot and repair network problems
http://www.linuxuser.co.uk/features/troubleshot-repair-linux-networks
Computer 1 can ping Computer 2 but not reverse
Use the route command to solve. See also this post.
Simple Network Management Protocol (snmp)
- https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol
- Default port number is 161
- What Is SNMP? How To Install & Configure SNMP in Linux
- Install and configure SNMP on Ubuntu
- Change port number on Dell iDrac 8
- DDoS attack
Monitor network by Cacti (GUI)
- http://www.ubuntugeek.com/install-cacti-monitoring-tool-on-ubuntu-15-10-server.html
- http://www.cacti.net/
Monitor network by command line
3 Simple, Excellent Linux Network Monitors: iftop, nethogs and vnstat.
bandwidth
bandwhich Shows What`s Taking Up Your Network Bandwidth On Linux And macOS
iftop
Use the interface top iftop command. On ubuntu, we need to use sudo apt-get install iftop and then run it by sudo iftop -i eth0. After that, we can press some keys to toggle options.
- p: port
- s: source
- d: destination
See thegeekstuff.
It is strange that the output shows other devices names in my network.
$ dig A pandora.com $ ipcalc -b 208.85.40.20 $ sudo iftop -F 208.85.40.20/24 -i wlan0
nethogs
$ sudo nethogs wlan0
Why does the snapd service use so much data?
nload
nload -m
nload – Monitor Linux Network Bandwidth Usage in Real Time
The result is the same as gtop (gtop is cooler) gives.
bmon
https://www.tecmint.com/bmon-network-bandwidth-monitoring-debugging-linux/
vnstat for network traffic monitor
- Ubuntu install vnstat console network traffic monitor
- How to Monitor and Log Network Traffic on Linux Using vnStat
- How to Monitor Monthly Bandwidth Usage in Linux
# 1. Install vnStat sudo apt-get install vnstat # 2. Pick a Interface to Monitor using vnStat vnstat -u -i eth0 vnstat --iflist vnstatd -d # start the daemon ps -ef | grep vnst # 3. vnStat Basic Usage vnstat # 4. vnStat hours, days, months, weeks Network Data vnstat -d vnstat -m # 5. Export the data to Excel or other DB vnstat --dumpdb # 6. Display Live Network Statistics vnstat -l # 7. Change the default vnstat output format vnstat -s (--short) vnstat --style 0 # 8. Display Top 10 Traffic Days vnstat --top10
- package includes net-tools programs arp, hostname, ifconfig, ipmaddr, iptunnel, mii-tool, nameif, netstat, plipconfig, rarp, route and slattach. Many of them are obsoleted by the ip program.
- The 7 Best Networking Tools That Replace the Deprecated net-tools on Linux 2022
nmcli and nmtui (useful for getting DNS IPs)
- How to add network bridge with nmcli (NetworkManager) on Linux
- How to Configure and Manage Network Connections Using ‘nmcli’ Tool
- How To Enable (UP) And Disable (DOWN) A Network Interface Port (NIC) In Linux?
- Manage network connections from the Linux command line with nmcli
- Wifi
- Connect to WiFi in Linux Using NMCLI command
nmcli radio wifi on nmcli dev status nmcli dev wifi list # Show signal/bar/rate sudo nmcli dev wifi rescan sudo nmcli --ask dev wifi connect MySSID
- How to Connect Wi-Fi from Linux Terminal Using Nmcli Command
# Find the up network adaptors $ nmcli dev status DEVICE TYPE STATE CONNECTION docker0 bridge connected docker0 eno1 ethernet connected Wired connection 1 # Find the DNS $ nmcli dev show | grep 'IP4.DNS'
iwconfig
8 Linux Commands: To Find Out Wireless Network Speed, Signal Strength And Other Information
Getting information (Link Quality, Signal level) about the wireless.
sudo iwconfig
See link quality continuously on screen
watch -n 1 cat /proc/net/wireless
(GUI) NetworkManager
ps -ef | grep NetworkManager
ifconfig - spoof the hardware address at the software level
To change the MAC address temporarily on a NIC (network interface controller),
sudo ifconfig eth0 down sudo ifconfig eth0 hw ether 00:11:22:33:44:55 sudo ifconfig eth0 up
And it seems there is no need to modify /etc/network/interfaces.
For wlan
sudo ifconfig wlan0 down sudo ifconfig wlan0 hw ether 00:11:22:33:44:55 sudo ifconfig wlan0 up
See
- 7 Examples To Configure Network Interface
- 15 Useful “ifconfig” Commands to Configure Network Interface in Linux
- What does ifconfig promisc mode do, or promiscuous mode in general?
ip command
It is said ip is replacing the old ifconfig command on modern Linux distributions.
http://www.makeuseof.com/tag/networking-commands-linux-terminal/
ip a ip addr ip address show ip link set DEVICE down # eg ip link set eth0 down ip link set DEVICE up
iptables
- See the ufw command which provides an easy way to configure iptables.
- How to disable iptables firewall temporarily
route
7 Linux Route Command Examples
- Display Existing Routes (route -n)
- Adding a Default Gateway (route add default gw 192.168.1.1)
- List Kernel’s Routing Cache Information (route -Cn)
- Reject Routing to a Particular Host or Network (route add -host 192.168.1.51 reject)
- Make 192.168.3.* Accessible from 192.168.1.* (route add -net 192.168.3.0 netmask 255.255.255.0 gw 192.168.3.10)
- Make 192.168.1.* Accessible from 192.168.3.* (route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.1.10)
- Allow Internet Access/External World (route add default gw 125.250.60.59)
On Ubuntu 16.04, it shows
$ route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default FIOS_Quantum_Ga 0.0.0.0 UG 600 0 0 wlp3s0 link-local * 255.255.0.0 U 1000 0 0 wlp3s0 192.168.1.0 * 255.255.255.0 U 600 0 0 wlp3s0 $ route -n # showing numerical IP address instead of host name. Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.1.1 0.0.0.0 UG 600 0 0 wlp3s0 169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 wlp3s0 192.168.1.0 0.0.0.0 255.255.255.0 U 600 0 0 wlp3s0
Flag value 'U' means up and 'G' means gateway'.
Connect two networks
The trick is explained in this post or the above route command.
For example, my network structure is
- Modem/router: LAN IP 192.168.1.*/24
- PC1: connect to Modem/router
- Second router (ASUS) connect to Modem/router: its WAN IP is 192.168.1.ASUS. It's LAN IP 192.168.2.*/24
- PC2 (raspberry pi): connect to the second router (ASUS): its IP is 192.168.1.212
By default, PC2 can ssh to PC1 but PC1 cannot access PC2.
The following command will solve the problem that PC1 cannot access PC2:
# From PC1 $ sudo route add -net 192.168.2.0 netmask 255.255.255.0 gw 192.168.1.ASUS $ ssh [email protected] # 192.168.2.212 is the IP address for the Raspberry Pi $ netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 192.168.2.0 192.168.1.ASUS 255.255.255.0 UG 0 0 0 eth0
One article from linux.com using the ip command.
traceroute
sudo apt-get install traceroute traceroute 8.8.8.8
On Windows, we can use the tracert command. For example, tracert www.microsoft.com.
nslookup (convert between hostname and ip) and host
- https://en.wikipedia.org/wiki/Nslookup
- https://en.wikipedia.org/wiki/Host_(Unix)
- How to get hostname from IP (Linux)?
$ host google.com google.com has address 172.217.5.238 google.com has IPv6 address 2607:f8b0:4004:802::200e google.com mail is handled by 30 alt2.aspmx.l.google.com. google.com mail is handled by 10 aspmx.l.google.com. google.com mail is handled by 40 alt3.aspmx.l.google.com. google.com mail is handled by 50 alt4.aspmx.l.google.com. google.com mail is handled by 20 alt1.aspmx.l.google.com. $ nslookup google.com Server: 127.0.1.1 Address: 127.0.1.1#53 Non-authoritative answer: Name: google.com Address: 172.217.7.238
mac$ nslookup cran.r-project.org Server: 156.40.70.10 Address: 156.40.70.10#53 Non-authoritative answer: cran.r-project.org canonical name = cran.wu-wien.ac.at. Name: cran.wu-wien.ac.at Address: 137.208.57.37 mac$ nslookup 137.208.57.37 Server: 156.40.70.10 Address: 156.40.70.10#53 Non-authoritative answer: 37.57.208.137.in-addr.arpa name = cran.wu-wien.ac.at.
The first two lines show the IP address of my DNS. If we run the command inside a Docker container where DNS was specified manually, the IP address we specified will be shown here.
dig
Dig provided by Google G Suite Toolbox. If the office internet is interrupted, we can use a cell phone to search for the IP address of a website for trouble shooting.
$ sudo apt-get install dnsutils $ dig world.std.com ; <<>> DiG 9.9.5-3ubuntu0.16-Ubuntu <<>> google.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49227 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;google.com. IN A ;; ANSWER SECTION: google.com. 130 IN A 172.217.5.238 ;; Query time: 11 msec ;; SERVER: 127.0.1.1#53(127.0.1.1) ;; WHEN: Fri Dec 01 17:32:37 EST 2017 ;; MSG SIZE rcvd: 55
arp (Address Resolution Protocol)
The arp command can be used to show the MAC addresss of all hosts in LAN
arp -a
Check connectivity
Using bash tcp built-ins to test connectivity when nothing else is available:
echo >/dev/tcp/google.com/80; if [ $? -eq 0 ]; then echo "Connection Successful"; fi
Find open ports
| command | example | comment |
|---|---|---|
| ss | sudo ss -tulpn | grep LISTEN | contain pid, add "sudo" can show program/command names on the last column, |
| lsof | sudo lsof -i -P -n | egrep "PID|LISTEN" | sudo is required. Contain pid. Using egrep to include header in output. It includes the command/program names on the first column. lsof may not be available by default in many OS. |
| nmap (network mapper) | nmap localhost (OR IP) | shortest output and input. Need to install. No 'sudo' needed. Works on non local. |
| netstat (old 'net-tools' ) |
netstat -tlpn | grep LISTEN | awk '{print $4 "\t" $7}' | show pid and program name. Need to install netstat. Windows already has it. |
| nc | nc -zvw3 <hostname> <port> | need to specify a port number. More accurate than nmap. "-w3" means time out after 3 seconds. |
What are network interface ports
Understanding Network Interface Ports in Linux. In networking, the term "port" can refer to different concepts depending on the context:
- Hardware Ports:
- Software Ports: These are virtual endpoints in an operating system used by network protocols to manage connections. Software ports are identified by a number, ranging from 0 to 65535. For example, port 80 is used for HTTP, and port 443 is used for HTTPS.
Socket
What Are Unix Sockets and How Do They Work
nmap/network mapper - port scanning & IPs in local network
nmap - Network exploration tool and security / port scanner
- nmap does not show all open ports By default, Nmap scans the most common 1,000 ports for each protocol.
- https://nmap.org/book/nmap-os-db.html. Local OS database is located at /usr/share/nmap/nmap-os-db. The 2nd line will show the revision number.
- Modifying the nmap-os-db Database Yourself
- Download the latest from https://svn.nmap.org/nmap/nmap-os-db. Note that the current revision number has to be found from the website. You can edit the file and insert the revision number on the 2nd line of your local copy.
- Even I update the database, it cannot detect my Ubuntu 14.04 OS (it only shows OS details: Linux 3.8 - 4.9). For the Raspberry Pi, it can show information from the network adapter; e.g. MAC Address: AA:BB:CC:DD:EE:FF (Raspberry Pi Foundation) but not the OS name (OS details: Linux 3.2 - 4.8).
sudo mv /usr/share/nmap/nmap-os-db /usr/share/nmap/nmap-os-db-old cd /usr/share/nmap sudo wget https://svn.nmap.org/nmap/nmap-os-db
- http://www.cyberciti.biz/networking/nmap-command-examples-tutorials/
- http://bencane.com/2013/02/25/10-nmap-commands-every-sysadmin-should-know/
- http://www.tecmint.com/nmap-command-examples/
sudo apt-get install nmap nmap 192.168.1.100 # does not require root privileges # used to check open ports nmap 192.168.1.* # show IPs and ports in LAN sudo nmap -sP 192.168.1.1/24 # show connected IPs (no hostnames?) and MAC addresses # If you don't use 'sudo' only partial devices can be found # The output may contains the hostname. For example, # Nmap scan report for brb-P45T-A.fios-router.home (192.168.1.xxx) nmap -sV 192.168.1.1 # show Daemon name (in VERSION column) together with port number nmap -T4 -F 192.168.1.99-255 # show connected IPs and open ports # -F means fast nmap -F taichimd.us # Note that domain name != server nmap -v taichimd.us nmap -A 192.168.1.1 # Aggressive scan (more output) nmap -p http,ssh,mysql taichimd.us # scan ports/services # note that mysql will be shown as closed # ssh port is not correct. # Not sure how to get the correct ssh port using nmap nmap --open taichimd.us # scan open ports sudo nmap -traceroute nih.gov sudo nmap -sS -O 192.168.1.99 # -O shows operating system # eth0 MAC $ nmap localhost # showing the true ports from the server Starting Nmap 7.01 ( https://nmap.org ) at 2017-10-09 15:01 EDT Nmap scan report for localhost (127.0.0.1) Host is up (0.00016s latency). Not shown: 996 closed ports PORT STATE SERVICE 22/tcp open ssh 25/tcp open smtp 80/tcp open http 631/tcp open ipp $ nmap localhost -p 3838-4000 # Shiny - A gui version of nmap is called Zenmap. How to install Zenmap Nmap GUI on Ubuntu 20.04 LTS
netstat: get a list of all open ports
How to use netstat in GNU/Linux
- -l or --listening shows only the sockets currently listening for incoming connection.
- -a or --all shows all sockets currently in use.
- -e --show extended/additional information
- -t or --tcp shows the tcp sockets.
- -u or --udp shows the udp sockets.
- -n or --numeric shows the hosts and ports as numbers, instead of resolving in dns and looking in /etc/services.
- -s --Print network stats
- -r --Print the network routing information
- -p --Print PID and name of the program to which each socket belongs
netstat -l # only listening ports netstat -rn # displays the system's routing table netstat -at netstat -ant # For tcp sudo netstat -pant # show ports and programs (pant = 喘氣). Best of the best!!! sudo netstat -peanut # (output is too wide) netstat -anp | grep 3306 | wc -l # print # of connections for the port mysql port
Other commands
sudo lsof -i -P -n | grep LISTEN sudo ss -tulpn sudo netstat -tulpn | grep LISTEN sudo netstat -tulpn # include dhcpcd & avahi-daemon
ss command
ss is one of 4 commands to check open ports for Linux. The 4 commands are netstat, nmap, ss and lsof.
How to Use the ss Command on Linux
lsof command
How to Use lsof in Linux (With a Practical Example. How to discover what files are currently open and in use on your system?
sudo apt install lsof # dietpi # Basic lsof Output sudo su lsof | head -n10 lsof -i # this gives a more list than nmap command
Common ports
- List of TCP and UDP port numbers
- Well-known ports: 0 to 1023
- Registered ports: 1024 to 49151
- Dynamic, private or ephemeral ports: 49152–65535
- A List of Common Ports
- Replace the default port (such as 22 for ssh) with anyone from 1024-65535 because ports numbers up to 1023 are “well known” ports & should be avoided
- Some services:
- 81: Nginx Proxy Manager
- 1194: OpenVPN
- 3306: MYSQL
- 3838: Shiny
- 8000: portainer
- 9000: portainer (URL)
- 9090: cockpit
- 17500, 17600, 17603: Dropbox
Copy text to a clipboard to be used in other apps
Install the xclip program. See here or here.
sudo apt-get install xclip # Examples sort -n -k 3, -k 2 file.txt | xclip -selection clipboard cat ~/.ssh/id_rsa.pub | xclip -sel clip
Works.
Start Emacs without X
Add -nw (no window) option.
emacs -nw
Audio
Record audio from mic
How to Save Important Voicemails for Both iPhone and Android. Hint: use Audacity.
mp3 codecs
https://help.ubuntu.com/community/RestrictedFormats
sudo apt-get install ubuntu-restricted-extras
Concatenate mp3 files
sudo apt-get install mp3wrap mp3wrap output.mp3 *.mp3
Reduce the size of an mp3 file
Specify a new lower bitrate using the -b option in lame. For example if your starting mp3 has a quality of 256kbs you can lower its bitrate to 128kbps (or even lower like 64kbps) by:
lame --mp3input -b 128 input.mp3 output.mp3
Convert ogg to mp3
ffmpeg is not included in Ubuntu repository. Use the avconv command. http://superuser.com/questions/15327/how-to-convert-ogg-to-mp3
sudo apt-get install libav-tools avconv -i input.ogg -c:a libmp3lame -q:a 2 output.mp3
Convert m4a/webm to mp3
avconv -i input.m4a output.mp3
Remove the vocals from any song using Audacity
https://www.makeuseof.com/tag/remove-vocals-song-audacity/
How to Remove Ambient Noise From Audio Files Using Audacity
How to Remove Ambient Noise From Audio Files Using Audacity
Normalize the volume of an audio file
- Can You Losslessly Increase the Volume of MP3 Files?
- Use Audacity. To raise (Amplify) volume:
- Edit > Select All.
- Effect > Amplify. Increase db. Adjust the sound until the highest peaks and lowest valleys of the waveform reach the top and bottom of the window.
- Check clip3. Export > MP3 or just start to listen.
- Command line tool: avconv (replace ffmpeg program). See this post.
avconv -ss 00:00:10 -i OLD.mp3 -vol 2560 NEW.mp3
The anconv/ffmpeg -vol parameter amplifies the sound. The default value is 256 (no amplification), and you can adjust the number accordingly. Here it’s 2560, as it’s 10 times louder. Note that these are not decibel values or anything that sophisticated, but just an integer value. 512 equals to twice the volume, 768 three times, 1024 four times, etc. The -ss parameter specifies the start time offset. Here it will skip the first 10 seconds.
- Command line tool: sox.
- http://askubuntu.com/questions/246242/how-to-normalize-sound-in-mp3-files
- http://www.linuxandlife.com/2013/03/how-to-use-sox-audio-editing.html
- http://digitalcardboard.com/blog/2009/08/25/the-sox-of-silence/ deal with several kinds of silence.
- http://www.thegeekstuff.com/2009/05/sound-exchange-sox-15-examples-to-manipulate-audio-files/
I found the converted file by sox has about one half file size compared to anconv/ffmpeg program (source file=47MB, anconv converted=135MB, sox converted file=54MB).
sudo apt-get install sox libsox-fmt-all sox --norm OLD.mp3 NEW.mp3 trim 10 sox --norm OLD.mp3 NEW.mp3 silence 1 0.1 1% sox -v 4.0 OLD.mp3 NEW.mp3 # increase volume
where '--norm' will normalize the audio and the 'trim' option set to skip the first 10 seconds. The silence parameter allows to trim silence at the beginning without a need to specify the number of seconds.
cut, delete or trim an audio
- Open the audio file in audacity.
- select a region in the waveform area. Do not select in the time interval area (above the waveform).
- To precisely select a range from one position to the end. Click Zoom in several times. Click one position in the waveform and click Edit -> Select -> Cursor to the track end to select
- Similarly, if we want to precisely select a range from the start to some position, we can click one position in the waveform and then click Edit -> Select -> Track start to cursor.
- To move around the track, use the scrollbar (below the waveform and above the bottom toolbar, not quite clear in Ubuntu/Unity)
- Click Edit -> Remove Audio or labels -> Cut/Delete/Trim Audio
- play the new audio by clicking the green triangle.
- File -> Export -> mp3 format.
Helpful resource for Audacity.
Fade out at the end of an audio
- Select a region.
- Effect -> Fade out
Wireshark
- http://www.howtogeek.com/204458/why-you-shouldn%E2%80%99t-use-mac-address-filtering-on-your-wi-fi-router/
- http://www.howtogeek.com/191482/how-an-attacker-could-crack-your-wireless-network-security/
- http://www.howtogeek.com/104278/how-to-use-wireshark-to-capture-filter-and-inspect-packets/
sudo apt-get install wireshark sudo chmod 4711 `which dumpcap`
Track the Time a Command Takes
How To Find The Execution Time Of A Command Or Process In Linux
time command
- What do 'real', 'user' and 'sys' mean in the output of time(1)?
- Linux time Command Tutorial for Beginners (with Examples)
time COMMAND time (COMMAND1; COMMAND2) time (COMMAND1 && COMMAND2) help time
When I run a set of 7 jobs using parallel, time command gives an output
real 15m53.788s # the wall clock time the command took from execution till termination user 95m20.238s # the time taken by the user space sys 9m1.320s # the time taken by kernel space
Here we see the real time is about 16m and the user time is about 6-7 times the real time. Indicating the parallel executing works.
/usr/bin/time command
/usr/bin/time provides more information then time command.
man time
Magazines
Latex
Editors
- 10 Best LaTeX Editors For Linux.
- Texmaker. R's installr package has a function to install Texmaker. Cross platforms.
- TEXworks. Cross platforms.
- TexStudio. See a dark theme example.
Online editing
- Latex Base. You can start to try it without registration. Free accounts cannot publish but still can download.
- Overleaf. Free account for 1GB space.
- ShareLatex
Missing cls
- texlive-latex-extra packages and texlive-publishers packages.
- https://tex.stackexchange.com/questions/179214/elsarticle-cls-not-found-when-using-texmaker-but-texlive-latex-extra-is-install/179250
$ apt-cache search IEEEtran texlive-publishers - TeX Live: Publisher styles, theses, etc.
sudo apt-get install texlive-publishers
Missing sty
$ apt-cache search pseudocode gpt - G-Portugol is a portuguese structured programming language libgportugol-dev - Development files for the G-Portugol library libgportugol0 - G-Portugol library texlive-science - TeX Live: Natural and computer sciences $ sudo apt-get install texlive-science
See PDF.
Flow chart
- LibreOffice Draw OR MS_PowerPoint (insert > shape). Check youtube.
- yEd
- Dia & wikipedia
- (online) www.draw.io
Clock
xclock (analog)
oclock -geometry 500x500+100+0 &
oclock (analog)
oclock -bg blue -geometry 500x500+100+0 -bd purple -transparent & oclock -bg blue -geometry 500x500+100+0 -bd purple -jewel green &
See oclock, X - a portable, network-transparent window system which includes an example of specifying the geometry parameter.
dclock (digital)
Digital clock for the X Window System with flexible display.
- http://manpages.org/dclock
- Note that we can send the command to the background using Ctrl +z and run bg. See run a command from the terminal without blocking it?
sudo apt-get install dclock dclock -h dclock -d dclock -date "Today is %A %B %Y" -led_off black -bg black -fg yellow -geometry 577x194+119+139 # 'q' to quit dclock -date "Today is %m/%d/%y" -led_off black -bg black -fg yellow -geometry 400x150+0+0 # width x hight + X + Y
In practice, I create a shell script file <bin/clock> with the following content. The first ampersand sign is to hide warnings messages and the 2nd ampersand sign is to put the process in the background.
dclock -date "Today is %A %B %Y" -led_off black -bg black -fg yellow -geometry 577x194+119+139 &>/dev/null &
Lubuntu digital clock format
http://netgator.blogspot.com/2012/09/change-edit-panel-digital-clock-format.html. My format is
%a, %x, %r # Tue, 05/17/2016, 09:42:27 PM %a %m-%d-%y, %I:%M %p # Mon 05-30-16, 08:31 AM
xdaliclock (digital)
https://www.mankier.com/1/xdaliclock
Scaling is not good. Colors is changing with time.
xdaliclock -noseconds -cycle # 'q' to quit
date command
sudo apt install toilet
sudo apt install figlet
watch -n 60 "date +'%m/%d/%y %H:%M:%S' | toilet"
ls /usr/share/figlet # list of fonts
# looking for *.tlf
date +'%m/%d/%y%H:%M:%S' | toilet -f mono12
date +'%m/%d/%y%H:%M:%S' | toilet -f bigmono12 # good on 1024x600
date +'%m/%d/%y%H:%M:%S' | toilet -f bigascii12
date +'%m/%d/%y%H:%M:%S' | toilet -f ascii12
watch --color "date +'%m/%d/%y%H:%M:%S' | toilet -f bigmono12 -F metal"
while true; do echo "$(date '+%H:%M:%S' | toilet -f bigmono12 -F border --metal)"; sleep 1; done
Reminder take a break, relieve eye strain
Stretchly. It's open-source and cross-platform. Nodejs is required.
Workrave is another choice. The source code is available too.
Prevent Eye Strain While Working On Your Linux Desktop With Safe Eyes, Github source code
wine
Running Linux in the AWS/Amazon Web Services
Forum software
- Simple Machines® Forum (SMF). For example http://pibot.org/forum/
RAID
Timer
- http://zeegaree.com/. Require 3 libraries that we need to install them using apt-get install. See the github page.
How to track you laptop using Prey
https://www.howtoforge.com/tutorial/how-to-track-your-linux-laptop/
last command
Linux last Command Tutorial for Beginners (8 Examples)
Display a list of system shutdown/reboot date/time
Linux Find Out Last System Reboot Time and Date Command
# Works on Linux and Mac last shutdown last reboot
Automatic reboot after power failure
It seems there is no reliable way to find out when the power failed.
The linux command 'last' can show some information about system reboot.
Another way is to modify the BIOS to select the option like 'Power off and Reboot'. This won't automatically boot your computer when it is shutdown normally.
How to restart/shutdown server safely
How to restart CentOS or RHEL server safely
Wake up and Shut Down Linux Automatically
- https://help.ubuntu.com/community/WakeOnLan
- https://www.cyberciti.biz/faq/ubuntu-linux-wake-on-lan-client-command-installation-examples/
- Wake up and Shut Down Linux Automatically
Two best options
- Bios: BIOS may have an easy-to-use wakeup scheduler
- wakeonlan:
- Eanble it: Check if it is enabled by default. If not, we can 1) enable it through a command (ethtool -s eth0 wol g) or 2) using the Network Manager
- Send a wake up command: (from a second linux) /usr/bin/wakeonlan D0:50:99:82:E7:2B where D0:50:99:82:E7:2B is the IP on the machine you want to wake it up
BIOS
Find out BIOS version
Linux Find Out BIOS Version Using a Command Line Option
How to update Lenovo BIOS from Linux without using Windows
https://www.cyberciti.biz/faq/update-lenovo-bios-from-linux-usb-stick-pen/
Internet speed test
Web
- https://fast.com/ (automatically run)
- https://www.bing.com/search?q=internet+speed+test
- http://www.speedtest.net/
Speedtest-cli
sudo apt-get intall python-pip sudo pip install speedtest-cli # A slightly modified code that will create a one-line space/semi-colon # delimited result is git clone https://github.com/HenrikBengtsson/speedtest-cli-extras.git speedtest-cli-extras/bin/speedtest-csv
works. But if I want to put it in cron, cron will issue an error speedtest-cli cannot be found. So I need to modify line 52 of the code in <speedtest-cli-extras/bin/speedtest-csv> to explicitly specify the location of speedtest-cli.
/usr/local/bin/speedtest-cli --share > $log
NOTE: 1. the results differ from the network connection. For example, the speed is good when I test it on the machine directly connected to the router. 2. It is helpful to modify the last line of the bash script to output what I need. 3. The separator is ";" in the output.
curl and wget
How to Run Speed Test from the Command Line to Check Internet Connection Speed
uname - Print system information
- https://www.lifewire.com/display-system-information-uname-command-3964321
- How to decipher Linux release info
uname -a will give you
- OS (uname = uname -s if you are under a Linux environment)
- OS (uname -s) eg Linux
- node name (uname -n=hostname)
- kernel release (uname -r) eg 3.16.0-38-generic
- kernel version (uname -v)
- machine architecture (uname -m) eg x86_64
- processor (uname -p)
- hardware platform (uname -i)
- operating system (uname -o)
How to check if running in Cygwin, Mac or Linux?
Hardware information
- How To Check System Hardware Manufacturer, Model And Serial Number In Linux
- How to List Your Computer's Devices From the Linux Terminal: mount, lsblk, df, fdisk, /proc, lspci, lsusb, lsdev, lshw, lsscsi, dmidecode, hwinfo.
Command Line
- Linux commands to display your hardware information
- inxi: 3 ways to use the Linux inxi command
- inxi -b
- inxi -W taipei,taiwan (check the weather)
- How To Find Linux System Details Using inxi
hwinfo
On Ubuntu, use sudo apt install -y hwinfo to install hwinfo. Install hwinfo on Ubuntu 20.04
dmesg command
How to Use the dmesg Command on Linux
sudo dmesg -L -T sudo dmesg -L -T --follow # Watching Live Events sudo dmesg -L -T | grep -i usb # Search for a specific term sudo dmesg | grep -E "memory|tty|dma" # Search for multiple terms
where -L to is force color output and -T is to make timestamp human-readable.
Linux Logo and the current system information
odroid@odroid:~$ sudo apt-get install screenfetch
odroid@odroid:~$ screenfetch
./+o+- odroid@odroid
yyyyy- -yyyyyy+ OS: Ubuntu 15.10 wily
://+//////-yyyyyyo Kernel: armv7l Linux 3.10.96-77
.++ .:/++++++/-.+sss/` Uptime: 4d 23h 8m
.:++o: /++++++++/:--:/- Packages: 2000
o:+o+:++.`..```.-/oo+++++/ Shell: 2263
.:+o:+o/. `+sssoo+/ Resolution: 1920x1080
.++/+:+oo+o:` /sssooo. DE: MATE 1.10.2
/+++//+:`oo+o /::--:. WM: Metacity (Marco)
\+/+o+++`o++o ++////. GTK Theme: 'Ambiant-MATE' [GTK2/3]
.++.o+++oo+:` /dddhhh. Icon Theme: Ambiant-MATE
.+.o+oo:. `oddhhhh+ Font: Ubuntu 10
\+.++o+o``-````.:ohdhhhhh+ CPU: ARMv7 rev 3 (v7l) @ 1.4GHz
`:o+++ `ohhhhhhhhyo++os: GPU: Gallium 0.4 on llvmpipe (LLVM 3.6, 128 bits)
.o:`.syhhhhhhh/.oo++o` RAM: 537MiB / 1990MiB
/osyyyyyyo++ooo+++/
````` +oo+++o\:
`oo++.
odroid@odroid:~$ screenfetch -h
Dictionary - Artha
- Lifehacker. Once it is launched, it is sitting on the task bar. Press Ctrl+Alt+W after selecting a word to look it up in Artha (a balloon tip will pop up on the screen top-right). It also supports using regular expressions to search words.
sudo apt-get install artha
Translation
- Translate Shell. No installation is needed. It is just a bash script (4990 lines) so it works on ODroid SOC. See also A Tool To Use Google Translate From Command Line In Linux
odroid@odroid:~/binary$ ./trans :zh-TW word
word
/wərd/
字
(Zì)
Definitions of word
[ English -> 正體中文 ]
noun
字
word, character, letter, calligraphy, symbol, style of writing
詞
word, term, speech, statement
單詞
word, individual word
話
words, word, dialect, saying, talk, speech
言
word, speech, character
言辭
words, word, what one says
筆墨
pen and ink, words, word, writings
約言
pledge, promise, word
verb
為 ... 措辭
word
odroid@odroid:~/binary$ time ./trans -brief :zh-TW word
字
real 0m4.249s
user 0m2.670s
sys 0m1.330s
ASCII art/ word art
- figlet. How to Create ASCII Text Banners in Terminal, lolcat
- toilet command. date | toilet . Color output. man toilet. TOIlet prints text using large characters made of smaller characters. It is similar in many ways to FIGlet with additional features such as Unicode handling, colour fonts, filters and various export formats.
- http://patorjk.com/software/taag/#p=display&f=Ivrit&t=BRR-SeqTools
____ ____ ____ ____ _____ _
| __ )| _ \| __ ) / ___| ___ __ |_ _|__ ___ | |___
| _ \| |_) | _ \ ____\___ \ / _ \/ _` || |/ _ \ / _ \| / __|
| |_) | _ <| |_) |_____|__) | __/ (_| || | (_) | (_) | \__ \
|____/|_| \_\____/ |____/ \___|\__, ||_|\___/ \___/|_|___/
|_|
____ _____ ____ _____ _______ _
| _ \| __ \| _ \ / ____| |__ __| | |
| |_) | |__) | |_) |____| (___ ___ __ _| | ___ ___ | |___
| _ <| _ /| _ <______\___ \ / _ \/ _` | |/ _ \ / _ \| / __|
| |_) | | \ \| |_) | ____) | __/ (_| | | (_) | (_) | \__ \
|____/|_| \_\____/ |_____/ \___|\__, |_|\___/ \___/|_|___/
| |
|_|
___ ___ ___ ___ _____ _
| _ ) _ \ _ )___/ __| ___ __ |_ _|__ ___| |___
| _ \ / _ \___\__ \/ -_) _` || |/ _ \/ _ \ (_-<
|___/_|_\___/ |___/\___\__, ||_|\___/\___/_/__/
|_|
Install And Manage Software From Source
How To Install And Manage Software From Source Using GNU Stow In Linux
Software that scan Malware and rootkits
Text to speech
- http://www.eguidedog.net/ekho.php. Compilation/build works on x86 Ubuntu 14 and Odroid Ubuntu 15.10. On Odroid I have to follow their instruction to use 'make CXXFLAGS=-DNO_SSE' instead of 'make'. However, sound feels shaky on Odroid xu4.
- http://audiobookmaker.com/
- http://project-modelino.com/online-resources-category.php?site_language=english&learn_language=chinese&category=tts
VPN
Mono Project
Mono is a software platform designed to allow developers to easily create cross platform applications part of the .NET Foundation
Mono is required for Repetier-Host software for 3D printing.
NAS server
OpenMediaVault
FreeNAS
ZFS system (FreeBSD-based).
10 Reasons Why You Should Store Your Data on a FreeNAS Box. Note With the current version of FreeNAS (FreeNAS 11) comes a hypervisor. See
- https://doc.freenas.org/11/vms.html
- https://forums.freenas.org/index.php?threads/freenas-11-0-released.55327/
- Virtualize FreeNAS
- bhyve, the BSD Hypervisor
- FreeNAS as hypervisor host
Change detection
http://bhfsteve.blogspot.com/2013/03/monitoring-web-page-for-changes-using.html
3 command-line tools for feigning productivity
https://opensource.com/article/18/2/command-line-tools-productivity: Blessed-contrib (javascript), Genact, Hollywood.
Mind mapping
Diagram
- Calligra Flow. Microsoft Visio alternative.
Open source surveillance
Systemctl, systemd
- Linux 系统开机启动项清理 中文 & English
- Chkservice – An Easy Way to Manage Systemd Units in Terminal
- Control an application to run when Raspberry pi boots up
- Why Linux’s systemd Is Still Divisive After All These Years
- How to Manage Systemd Services with Systemctl on Linux
- Manage Services with Systemd
- Manage Systemd Unit Files
- Manage Sockets with Systemctl
- Systemctl Additional Commands
Systemd vs SysVinit
- https://kernelmastery.com/systemd-vs-sysvinit/
- https://linoxide.com/linux-command/systemd-vs-sysvinit-cheatsheet/
- https://www.2daygeek.com/sysvinit-vs-systemd-cheatsheet-systemctl-command-usage/
Systemctl vs service commands
What is the difference between service and systemctl? service is an "high-level" command used for starting and stopping services in different unixes and linuxes. Depending on the "lower-level" service manager, service redirects on different binaries. For example, on CentOS 7 it redirects to systemctl.
$ service nginx start # VS $ systemctl start nginx $ systemctl # list all services $ cat /lib/systemd/system/rsyslog.service $ systemctl status rsyslog $ cat /lib/systemd/system/ufw.service
How to Run a Linux Program at Startup with systemd
- How to Run a Linux Program at Startup with systemd
- Creating the Service Program for systemd to Start sudo nano /usr/local/bin/htg.sh, sudo chmod +x /usr/local/bin/htg.sh
- Creating the Service Unit File sudo nano /etc/systemd/system/htg.service, sudo chmod 640 /etc/systemd/system/htg.service
- Starting the Service Automatically with the systemd Command sudo systemctl daemon-reload, sudo systemctl enable htg, sudo systemctl start htg
- Verifying the Service sudo systemctl status htg.service
- Stopping and Disabling the Service - sudo systemctl stop htg.service
- How to Create a New systemd Service on Linux
- Automatically mount a storage device with a systemd service
How Long Does it Take To Boot Your Linux System
Find Out How Long Does it Take To Boot Your Linux System
$ systemd-analyze # total boot time along with the time taken by
# firmware, boot loader, kernel and the userspace
$ systemd-analyze blame # breakdown the boot time into each unit
Check if Your Linux System Uses systemd
How to Check if Your Linux System Uses systemd
chkservice
chkservice Is A systemd Unit Manager With A Terminal User Interface
Kernel
Firmware update
Fwupd 1.9.9 Released with Support for Lenovo X1 Yoga Gen7 530E 2-in-1 Laptops
Game
See Game.
Best Linux Adobe Alternatives You Need to Know
- https://linux.cn/article-8928-1.html and https://www.maketecheasier.com/adobe-alternatives-for-linux/