Linux
Handy Linux tips
ls | more without lose color
ls --color | more
Most likely your ls is aliased to ls --color=auto. If you do ls --color (which is morally equivalent to ls --color=always), that will force it to turn on colors.
colordiff-color in terminal diff
Need to install first. apt-get install colordiff. http://www.cyberciti.biz/programming/color-terminal-highlighter-for-diff-files/
Listen to pandora in Europe: install squid proxy
http://www.cyberciti.biz/faq/access-pandora-radio-using-proxy-server-outside-usa/
Interestingly, the firefox connection settings should choose HTTP Proxy instead of 'SOCKS host'.
Bypass blocked website from work computer
http://www.panix.com/~ruari/censorship.html
ssh -D 4096 user@remoteip ssh -D 4096 -p 23 user@remoteip
This will require you to enter the password and leave you in the remote machine. If a nonstandard port is required, we can use -p option.
Now in the firefox, we need to go to Edit -> Preferences -> Advanced -> Network tab -> Settings... Check 'Manual proxy configuration' (The default is 'Use system proxy settings') and enter 'localhost' for SOCKS Host and '4096' for the Port. Don't enter 'localhost' in the HTTP Proxy.
On Windows, we can use Putty. In short, in the left-hand panel, navigate through Connection > SSH > Tunnels. Enter 4096 in the Source Port box and select the Dynamic radio button. Click Add and “D4096″ will appear in the Forwarded Ports list. The setting in the firefox end is the same.
Graphical way to display disk usage
For example, to use xdiskusage, we run apt-get install xdiskusage and launch it by xdiskusage ~/.
Display directory size with sorting and human readable
du -k ~/ --max-depth=1 --exclude ".*" | sort -nr | cut -f2 | xargs -d '\n' du -sh
The --exclude is to hide hidden directories.
Note that the 'du' commands may be cheating. See the following screenshot.
The discrepancy is explained by 'sector'. See http://askubuntu.com/questions/122091/difference-between-filesize-and-size-on-disk.
$ sudo dumpe2fs /dev/sda1 | grep -i "block size" dumpe2fs 1.41.14 (22-Dec-2010) Block size: 4096
To show a file size in terms of blocks, we can use
ls -s
So for example, if a file takes 150 blocks, and if a block takes 4096 bytes, then the file takes 150*4096/1024 KB on disk.
Apache benchmark (ab) testing
ab -n 100 -c 10 http://taichi.selfip.net:81/ ab -n 100 -c 10 http://taichi.selfip.org/
rsync with non-standard port
Use -e option
rsync -avz -e "ssh -p 23" mydir user@remoteip:
rsync with progress bar
Use --progress option.
rsync -avz --progress file1 file2
The 'rsync' command works on transferring files local to local too.
scp with non-standard port
Use -P argument.
ssh -P 23 myfile user@remoteip:
The device is busy
See http://ocaoimh.ie/2008/02/13/how-to-umount-when-the-device-is-busy/
# fuser -m /dev/sdc1 /dev/sdc1: 538 # ps auxw|grep 538 donncha 538 0.4 2.7 219212 56792 ? SLl Feb11 11:25 rhythmbox
Another handy one is:
umount -l /dev/sdwhatever
Kill a process
kill -9 PID
Create an ext3 file system on a USB flash drive
umount /dev/sdb1 (depending on the device of course) sudo mkfs.ext3 /dev/sdb1 sudo e2label /dev/sdb1 usbdrive (change the label)
We can create MS-DOS file system by
sudo mkfs.vfat /dev/sdb1
Port forwarding
For example, we like to use source port number 8080, destination port 80 (the http port), and destination server www.ubuntuforums.org.
ssh -L 8080:www.ubuntuforums.org:80 <host>
Where <host> should be replaced by the name of your laptop. The -L option specifies local port forwarding. For the duration of the SSH session, pointing your browser at http://localhost:8080/ would send you to http://www.ubuntuforums.org/.
To stop the ssh session, use ps -ef to find the process id and kill it.
Add a new user with home directory
adduser xxx
adduser is better than useradd since useradd does not create home directory and it does not even ask the password for new user. adduser will interactively ask user information.
To delete the user, use
deluser xxx
Compress a folder without full path name
Suppose we want to compress the folder ~/Documents and its subfolders. We want to include Documents folder name but not /home/brb/Documents name.
tar -czvf tmp.tar.gz -C ~/ Documents tar -tzvf tmp.tar.gz
Show folder size for one level only
du --max-depth=1 -h
Soft link
ln -s /full/path/of/original/file /full/path/of/soft/link/file
Backup
- Use dropbox folder.
If we don't want to install dropbox software, we can install bash dropbox uploader: http://www.andreafabrizi.it/?dropbox_uploader
I install it under ~/Downloads/andreafabrizi-Dropbox-Uploader-cdc2466 directory
Sample usages:
./dropbox_uploader.sh list / ./dropbox_uploader.sh upload ~/Desktop/ConfigurateNote.txt
Running a cron job as a user
- crontab cron-file-winter; crontab -l
Make sure the .sh file gives a complete path. For example,
#!/bin/sh R --vanilla < arraytoolsip.R
does not work in cron job although it works perfect when we manually run it from the right path. The sh file should be
#!/bin/sh R --vanilla < $HOME/Dropbox/scripts/arraytoolsip.R
Running crontab as root
Use sudo crontab -e to edit. After saving it, no need to initialize it. Use sudo crontab -l to list the cron job.
Mount usb drive
Run
fdisk -l
OR
dmesg | grep -i "SCSI"
to confirm the USB device is detected.
Now suppose the usb device is found in dev/sdb1.
mkdir /mnt/usb mount -t vfat -o rw,users /dev/sdb1 /mnt/usb
The above mount command assumes the usb drive has Windows vfat partition and users give non-root users the ability to unmount the drive. If the USB drive is partitioned linux ext2/3, we can merely run mount command as
mount /dev/sdb1 /mnt/usb
At the end, run umount command like
umount /mnt/usb
To make the mounting automatically, edit the file /etc/fstab.
/dev/sdb1 /mnt/usb vfat defaults 0 0 /dev/sdb1 /mnt/usb ext3 defaults 0 0
and run
mount -a
run commands in a background and allow log off
nohup /path/to/script >output 2>&1 &
vi keys
- editmode / insert: "i"
- editmode / append: "a"
- leave edit mode: "ESC"
- save: ":w"
- quit: ":q"
- save and quit: ":wq"
- quit without saving: ":q!"
Some helps
Cloud
http://slidedecks.wilmoore.com/2012-confoo/diy-private-cloud-using-virtualBox-and-chef/#66
http://www.datacentermap.com/blog/cloud-software-389.html
- OpenStack:Open source software for building private and public clouds. Great for large infrastructures. Cf: Amazon Elastic Compute Cloud (EC2).
- CloudStack
- Eucalyptus API compatible with Amazon EC2
- ownCloud. Cf: Dropbox.
U-boot
http://www.denx.de/wiki/U-Boot
Pandora linux client
http://kevinmehall.net/p/pithos/
COW (copy on write) file system
The cow filesystem was found on xubuntu live CD. See http://en.wikipedia.org/wiki/Copy-on-write
Apache redirection
http://cran.r-project.org/mirror-howto.html
Important linux directories
- /bin - executables used by the base system
- /boot
- /dev
- /etc - configuration files
- /media
- /mnt
- /opt - optional application packages
- /proc - process information only
- /sbin - critical executables for running the system, but should be used by superuser
- /usr - non-critical files. Inside is /usr/bin, which contains most of the libraries used by apps.
- /var - variable data such as databases, mails spools and system logs.
List of available (uninstalled) packages
aptitude -F "%p" search "?not(?installed)"
List of installed packages
dpkg --get-selections
Debian/Ubuntu/Bio-Linux
BioLinux
Bio-Linux 7.0 (2012/11/21) screenshot.
Apache configuration
https://help.ubuntu.com/11.10/serverguide/httpd.html
Device manager
By default, ubuntu does not provide any graphical tool like device manager on Windows. A very close one is lshw (hardware lister). A GUI tool based on it is called lshw-gtk and can be installed by sudo apt-get install lshw-gtk in Ubuntu/Debian or yum install lshw in Red Hat/Fedora/CentOS.
$ sudo lshw -short
H/W path Device Class Description
===================================================
system To Be Filled By O.E.M. (To Be Filled By O.E.M.)
/0 bus 970 Extreme4
/0/4 processor AMD Sempron(tm) 145 Processor
/0/4/5 memory 128KiB L1 cache
/0/4/6 memory 1MiB L2 cache
/0/10 memory 8GiB System Memory
/0/10/0 memory 2GiB DIMM DDR3 Synchronous 667 MHz (1.5 ns)
/0/10/1 memory 2GiB DIMM DDR3 Synchronous 667 MHz (1.5 ns)
/0/10/2 memory 2GiB DIMM DDR3 Synchronous 667 MHz (1.5 ns)
/0/10/3 memory 2GiB DIMM DDR3 Synchronous 667 MHz (1.5 ns)
/0/0 memory 64KiB BIOS
/0/100 bridge RD890 PCI to PCI bridge (external gfx0 port B)
/0/100/2 bridge RD890 PCI to PCI bridge (PCI express gpp port B)
/0/100/2/0 display RV770 [Radeon HD 4850]
/0/100/2/0.1 multimedia RV770 HDMI Audio [Radeon HD 4850/4870]
/0/100/9 bridge RD890 PCI to PCI bridge (PCI express gpp port H)
/0/100/9/0 bus ASM1042 SuperSpeed USB Host Controller
/0/100/a bridge RD890 PCI to PCI bridge (external gfx1 port A)
/0/100/a/0 bus ASM1042 SuperSpeed USB Host Controller
/0/100/11 storage SB7x0/SB8x0/SB9x0 SATA Controller [IDE mode]
/0/100/12 bus SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
/0/100/12.2 bus SB7x0/SB8x0/SB9x0 USB EHCI Controller
/0/100/13 bus SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
/0/100/13.2 bus SB7x0/SB8x0/SB9x0 USB EHCI Controller
/0/100/14 bus SBx00 SMBus Controller
/0/100/14.1 storage SB7x0/SB8x0/SB9x0 IDE Controller
/0/100/14.2 multimedia SBx00 Azalia (Intel HDA)
/0/100/14.3 bridge SB7x0/SB8x0/SB9x0 LPC host controller
/0/100/14.4 bridge SBx00 PCI to PCI Bridge
/0/100/14.5 bus SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
/0/100/15 bridge SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0)
/0/100/15.2 bridge SB900 PCI to PCI bridge (PCIE port 2)
/0/100/15.2/0 bus VT6315 Series Firewire Controller
/0/100/15.3 bridge SB900 PCI to PCI bridge (PCIE port 3)
/0/100/15.3/0 eth0 network RTL8111/8168B PCI Express Gigabit Ethernet controller
/0/100/16 bus SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
/0/100/16.2 bus SB7x0/SB8x0/SB9x0 USB EHCI Controller
/0/101 bridge Family 10h Processor HyperTransport Configuration
/0/102 bridge Family 10h Processor Address Map
/0/103 bridge Family 10h Processor DRAM Controller
/0/104 bridge Family 10h Processor Miscellaneous Control
/0/105 bridge Family 10h Processor Link Control
/0/1 scsi2 storage
/0/1/0.0.0 /dev/cdrom disk CDDVDW SH-S223Q
/0/2 scsi4 storage
/0/2/0.0.0 /dev/sda disk 80GB MAXTOR 4K080H4
/0/2/0.0.0/1 /dev/sda1 volume 66GiB EXT4 volume
/0/2/0.0.0/2 /dev/sda2 volume 8186MiB Extended partition
/0/2/0.0.0/2/5 /dev/sda5 volume 8186MiB Linux swap / Solaris partition
/0/3 scsi6 storage
/0/3/0.0.0 /dev/sdb disk SCSI Disk
/0/3/0.0.1 /dev/sdc disk SCSI Disk
/0/3/0.0.2 /dev/sdd disk SCSI Disk
/0/3/0.0.3 /dev/sde disk SCSI Disk
/1 wlan0 network Wireless interface
For storage part, ubuntu provides a graphical tool. See "disk utility" on gnome based ubuntu or search for "disk" in launcher.
Set static IP
root@debian:~# cat /etc/network/interfaces auto lo eth0 iface lo inet loopback iface eth0 inet dhcp
Now edit the file /etc/network/interfaces
# The primary network interface auto eth0 iface eth0 inet static address 192.168.1.3 gateway 192.168.1.1 netmask 255.255.255.0 network 192.168.1.0 broadcast 192.168.1.255 dns-nameservers 192.168.1.1 8.8.8.8
After it, restart network
/etc/init.d/networking restart
OR sudo reboot
Note: It does not work by editing /etc/resolv.conf since this file will be overwritten.
Change IP address from the command line
/sbin/ifconfig eth0 192.168.1.17 netmask 255.255.255.0 up /sbin/ifconfig eth0
Dyndns
See https://help.ubuntu.com/community/DynamicDNS
nano /etc/ddclient.conf
daemon=600 protocol=dyndns2 use=web, web=checkip.dyndns.com, web-skip='IP Address' server=members.dyndns.org login=YOURNAME password='YOURPASSWORD' taichi.selfip.net
And run ddclient -daemon=0 -debug -verbose -noquiet to confirm it is working.
webmin
See http://www.webmin.com/deb.html
wget http://prdownloads.sourceforge.net/webadmin/webmin_1.600_all.deb dpkg --install webmin_1.600_all.deb
The install will be done automatically to /usr/share/webmin, the administration username set to root and the password to your current root password. You should now be able to login to Webmin at the URL http://localhost:10000/. Or if accessing it remotely, replace localhost with your system's IP address.
Virtual box
Install guest addition for virtualbox on Debian See the link [1]
apt-get update apt-get install build-essential module-assistant m-a prepare cd Downloads tar xzvf /media/cdrom0/VMwareTools-9.2.0-799703.tar.gz cd vmware-tools-distrib ./vmware-install.pl [Accept all default] shutdown -r now
virtual machine software benchmark
Emulation vs virtualization
Virtualization allows users to configure virtual machines outfitted with similar hardware and processors as the host system, whereas emulation simulates different hardware and processor configurations in software.
Emulators are what allow you to run old Commodore 64 or NES games on a PC, for example.
Remote desktop connection
- Go to System -> Preference -> Remote desktop. Allow other uses to view your desktop & uncheck you must confirm each access & require user to enter this password.
- Go to System -> Preference -> Monitor. Change monitor resolution to 1280 x 720.
Remote desktop connection when vino failed
Try teamviewer. It works fine. After launching it, the software automatically creates an ID and password. We can change the password so it is fixed. Then launch the software on the client. Use the partner's ID and password to connect to it.
Remote desktop connection through vmware workstation
We shall be able to remote desktop connect to a Windows guest machine if the guest machine has configured to use bridge connection and a static IP. If there is a problem, it is likely caused by Window's firewall. See the two screenshots. We can just turn off the firewall of home network but keep the firewall on for the public network.
We don't need to use port forward for the remote desktop connection.
Instal sshd
apt-get update apt-get install openssh-server
Install LAMP
See the page [2]
apt-get install apache2 a2enmod rewrite apt-get install mysql-server [I choose branch name as MYSQL root password] apt-get install php5 php-pear php5-suhosin apt-get install php5-mysql /etc/init.d/apache2 restart
Another instruction including how to set up user directories for Apache web server http://wiki.debian.org/LaMp
apt-get install mysql-server mysql-client mysql_secure_installation apt-get install apache2 apache2-doc apt-get install php5 php5-mysql libapache2-mod-php5 apt-get install python libapache2-mod-python
The apache configuration file is in /etc/apache2/apache2.conf.
For PHP, it is also useful to install php for command line.
sudo aptitude install php5-cli
Restarting apache before testing on web browser
/etc/init.d/apache2 restart
Install mediawiki
And also the link http://www.mediawiki.org/wiki/Manual:Running_MediaWiki_on_Debian_GNU/Linux
aptitude install mediawiki php5 apache2 mediawiki-extensions libapache2-mod-php5
This will install latex. After the end, we can use Synaptic package manager to see what were installed. Now following the instruction in https://help.ubuntu.com/community/MediaWiki, we remove the '#' from the third line so that it reads 'Alias /mediawiki /var/lib/mediawiki':
sudo nano /etc/mediawiki/apache.conf sudo /etc/init.d/apache2 restart
Now we can start mediawiki by opening a browser and pointing it to http://localhost/mediawiki.
We need to enter
Site config admin username: WikiSysop password: Database config Database name: wikidb DB username: DB password: Superuser name: root Superuser password: [depend on how it was chosen when installing MYSQL]
Press the button of "Installing mediawiki". We will be welcomed to the wiki page. Following the instruction,
For security reason, I remove new account creation and anonymous editing. I also remove edit counters.
Backup and Restore mediawiki
See docs.google.com note. The process involves 3 parts: mediawiki system, mysql and images.
Install moinmoin
- Comparison of mediawiki vs moinmoin http://www.wikimatrix.org/compare/MediaWiki+MoinMoin
- Comparison of wiki software http://en.wikipedia.org/wiki/Comparison_of_wiki_software
- Moinmoin website http://moinmo.in/
- moinmoin was used by Bioconductor, ubuntu, etc.
MYSQL security
Just execute mysql_secure_installation from the command line.
- You can set a password for root accounts.
- You can remove root accounts that are accessible from outside the local host.
- You can remove anonymous-user accounts.
- You can remove the test database, which by default can be accessed by anonymous users.
See http://www.mysql-optimization.com/mysql-secure-installation-program.html
Compiling R
Use the following command to download required components before building any R packages. See also ubuntu package for R
sudo apt-get build-dep r-base
Package maintenance
https://help.ubuntu.com/community/AptGet/Howto#Maintenance_commands To show package information
apt-cache show <package_name>
To list files in a package.
dpkg -L <package_name>
To list racing games package
apt-cache search racing game
The following command list all packages installed in the system
apt-cache pkgnames
View logs
- Apache /var/log/apache2/access.log
Torrent
Popular search sites
- torrentz
- thepiratebay
- isohunt
- kickass torrents
Torrent client: transmission-cli
Torrent client: rtorrent
Install webmin
apt-get install perl libnet-ssleay-perl openssl libauthen-pam-perl libpam-runtime libio-pty-perl apt-show-versions python wget http://prdownloads.sourceforge.net/webadmin/webmin_1.600_all.deb dpkg --install webmin_1.600_all.deb
File does not exist: /var/www/favicon.ico from /var/log/apache2/error.log
The reason? You have not created a favicon, also known as a website icon, for your website. It’s the icon that displays in the address bar of your web browser when you connect to a website. A web browser will request this icon file from every website.
If you choose to create one. Use a program such as Gimp and create a 16×16 pixel image and save it as a .ico filetype. Then upload that file to the DocumentRoot of your website. You will need one for each VirtualHost. If you don’t have Gimp, there are online resources such as favicon.cc where you can create a .ico file and download it for your own use.
As you know by now, not having a favicon.ico file, won’t stop web browsers from requesting it each time. But you can tell Apache not to log the event as an error message. You will still see the request in the access.log, but at least you will have a cleaner error.log file.
Add the following block of code to each VirtualHost, or at least the ones which don’t have a favicon file.
Redirect 404 /favicon.ico <Location /favicon.ico> ErrorDocument 404 "No favicon" </Location>
Don’t forget to restart apache after making the change. If you want make a “global” change, which would apply to any and all VirtualHosts, you can create a file in Apache’s conf.d folder with a name such as nofavicon.conf and then add that block of code to the file. That would disable favicon across the board and save you from having to edit each VirtualHost.
Or, you create an empty file with the name “favicon.ico” in the directory root of Apache (for exemple /var/www/).
Show weather on the taskbar
http://www.noobslab.com/2012/10/important-thingstweaks-to-do-after.html
Screencaster
Watch TV
Running concrete5 On Nginx (LEMP)
http://www.howtoforge.com/running-concrete5-on-nginx-lemp-on-debian-squeeze-ubuntu-12.10
Download mms stream (suitable if mms is an extension; for example, studioclassroom)
- Use mimms (will NOT output to speaker at the same time; so is best in terms of performance)
sudo apt-get install mimms mimms -t 60 mms://example.com/video
where -t option specifies number of minutes. See http://linuxers.org/howto/how-download-mms-streaming-videos-ubuntu The output will have the same extension as the input. But it is not always clear. For example
$ mimms -t 3 mms://bcr.media.hinet.net/RA000073 mms://bcr.media.hinet.net/RA000073 => RA000073.wmv 1.48 MB / ∞ B (8.79 kB/s, ∞ s remaining) Download stopped after user-specified timeout.
- Use mplayer (will output to speaker at the same time, so suffer from performance, anyway do not get output)
mplayer mms:/link/something.xxx -dumpstream -dumpfile file.xxx
mms:/link/something.xxx - link to the stream you wish to download file.xxx - file to which you wish to download the stream, be careful to write the same extension xxx
Wait for the file to download and that's it. See http://ubuntuhowtos.com/howtos/download_mms_stream
Use VLC to record internet radio (suitable if the stream is continuous)
sudo apt-get install ubuntu-restricted-extras
sudo apt-get install lame libmp3lame0 sudo apt-get install ffmpeg sudo apt-get install libavcodec-extra-53 libavdevice-extra-53 libavfilter-extra-2 libavformat-extra-53 libavutil-extra-51 libpostproc-extra-52 libswscale-extra-2
A successful run will have an output like
VLC media player 2.0.3 Twoflower (revision 2.0.2-93-g77aa89e) [0x2329ca8] dummy interface: using the dummy interface module... [0x7fac2c007428] mux_dummy mux: Open [0x7fac2c003598] access_mms access: selecting stream[0x1] audio (39 Kib/s) [0x7fac2c003598] access_mms access: connection successful [0x7fac2c003598] access_mms access error: failed to send command [0x7fac2c005fe8] idummy demux: command `quit'
CVS server
# Original data dir: /home/mli/Downloads/hmv_arc # CVS dir on server (CVSROOT): /home/mli/cvsrep # CVS dir on local: /home/mli/Downloads/localcvs sudo apt-get install cvs mkdir ~/cvsrep export CVSROOT=/home/mli/cvsrep cvs init sudo groupadd mycvsgrp sudo useradd -G mycvsgrp mli # Assume mli is a new user sudo usermod -a -G mycvsgrp mli # assume mli is an existing user groups mli # view groups a user is in use sudo chown -R :mycvsgrp /home/mli/cvsrep cd hmv_arc cvs import -m "initial" mycvs mli START
Quick test on the same machine
cd ~/Downloads
mkdir localcvs
cd localcvs
cvs checkout mycvs
# CVS dir on local after: /home/mli/Downloads/localcvs/mycvs/
/home/mli/Downloads/localcvs/mycvs/CVS
export CVS_RSH=ssh
brb@ubuntu:~/Downloads$ cvs -d :ext:[email protected]:/home/mli/cvsrep checkout mycvs
CVS client (ubuntu)
cvs -d :ext:[email protected]:/home/mli/cvsrep checkout mycvs
CVS client (windows)
I use WinCVS for a demonstration
- Remote -> checkout module
- Module name: mycvs
- CVSROOT:
- protocol: ssh
- repository path: /home/mli/cvsrep
- user name:
- host name: taichi.selfip.net
RHEL/CentOS
What is my IP address
ifconfig eth0
What is my DNS server
/etc/resolv.conf
What services get started at boot time
chkconfig --list
Is xxx service running
xxx status
What services are currently running
ps -e
and
lsof -i
will show you services that are listening to TCP or UDP endpoints.
What network ports are open
lsof -i
or
netstat -aut
What firewall rules do I in place
iptables -L
What packages do I have installed
rpm -qa | less
or
rpm -qa | grep xxx
What version of package xxx do I have installed
rpm -qi xxx
List of available (uninstalled) packages
yum list available