Linux Programming

Shell Programming

Redirect

Redirecting output. File descriptor number 1 (2) means standard output (error).

./myProgram > stdout.txt        # redirect std out to <stdout.txt>
./myProgram 2> stderr.txt       # redirect std err to <stderr.txt> by using the 2> operator
./myProgram > stdout.txt 2> stderr.txt # combination of above two
./myProgram > stdout.txt 2>&1   # redirect std err to std out <stdout.txt>
./myProgram >& /dev/null        # prevent writing std out and std err to the screen
ps >> outptu.txt                # append

Redirecting input

./myProgram < input.txt

To redirect to both a file and the screen the same time, use tee command. See http://www.cyberciti.biz/faq/linux-redirect-error-output-to-file/

Pipe

The operator is |.

ps > psout.txt
sort psout.txt > pssort.out

can be simplified to

ps | sort > pssort.out

For example,

$ head /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
sync:x:4:65534:sync:/bin:/bin/sync

$cat /etc/passwd | cut -d: -f7 | sort | uniq -c | sort -nr
     18 /bin/sh
     13 /bin/false
      2 /bin/bash
      1 /bin/sync

where cut command will extract the 7th field separated by the : character and write to the output stream. sort command will sort alphabetically sorts the line it reads from its input and returns the new sort to its output. The uniq command will remove and count duplicated lines. The final sort command will sort its input numerically in reverse order.

Variables

food=Banana
echo $food
food="Apple"
echo $food

Environment variables

$HOME
$PATH
$0 -- name of the shell script
$# -- number of parameters passed
$$ process ID of the shell script, often used inside a script for generating unique temp filenames
$? -- the exit value of the last run command

For example,

#!/bin/sh
/usr/local/bin/my-command
if [ "$?" -ne "0" ]; then
  echo "Sorry, we had a problem there!"
fi

Parameter variables

$1, $2, .... -- parameters given to the script
$* -- list of all the parameters, in a single variable
$@ -- subtle variation on $*. 
$! -- the process id of the last command run in the background.

For example,

$ touch /tmp/tmpfile_$$

$ set foo bar bam
$ echo $#
3
$ echo $@
foo bar bam
$ set foo bar bam &
[1] 28212
$ echo $!
28212
[1]+  Done                    set foo bar bam

We can also use parentheses around the variable name.

QT_ARCH=x86_64
QT_SDK_BINARY=QtSDK-4.8.0-${QT_ARCH}.tar.gz
QT_SD_URL=https://xxx.com/$QT_SDK_BINARY

Conditions

We can use the test command to check if a file exists. The command is test -f <filename>.

[] is just the same as writing test, and would always leave a space after the test word.

if test -f fred.c; then ...; fi

if [ -f fred.c ]
then
...
fi

if [ -f fred.c ]; then
...
fi

Arithmetic comparison

expr1 -eq expr2  ==> check equal
expr1 -ne expr2  ==> check not equal
expr1 -gt expr2  ==> expr1 > expr2
expr1 -ge expr2  ==> expr1 >= expr2
expr1 -lt expr2  ==> expr1 < expr2
expr1 -le expr2  ==> expr1 <= expr2
! expr  ==> opposite of expr

File conditionals

-d file  ==> True if the file is a directory
-e file  ==> True if the file exists
-f file  ==> True if the file is a regular file
-r file  ==> True if the file is readable
-s file  ==> True if the file has non-zero size
-w file  ==> True if the file is writable
-x file  ==> True if the file is executable

Example: Suppose we want to know if the first argument (if given) match a specific string. We can use (note the space before and after '==')

#!/bin/bash
if [ $1 == "console" ]; then
  echo 'Console'
else
  echo 'Non-console'
fi

Control Structures

if

if condition
then
  statements
elif [ condition ]; then
  statements
else 
  statements
fi

For example, we can run a cp command if two files are different.

if ! cmp -s "$filesrc" "$filecur"
then
     cp $filesrc $filecur
fi

for

for variable in values
do 
  statements
done

For example, to convert pdfs to tifs using ImageMagick (for looping over files, check cyberciti.biz)

outdir="../plosone"
indir="../fig"

if [[ ! -d  $outdir ]];
then
   mkdir $outdir
fi

in=(file1.pdf file2.pdf file3.pdf)

for (( i=0; i<${#in[@]} ; i++ ))
do
  convert -strip -units PixelsPerInch -density 300 -resample 300 \
          -alpha off -colorspace RGB -depth 8 -trim -bordercolor white \
          -border 1% -resize '2049x2758>' -resize '980x980<' +repage \
          -compress lzw $indir/${in[$i]} $outdir/Figure$[$i+1].tiff
done

'''while'''
<pre>
while condition do
  statements
done

until

until condition
do 
  statements
done

AND list

statement1 && statement2 && statement3 && ...

OR list

statement1 || statement2 || statement3 || ...

Commands

break  ==> escaping from an enclosing for, while or until loop
:      ==> null command
continue ==> make the enclosing for, while or until loo continue at the next iteration
.      ==> executes the command in the current shell
eval   ==> evaluate arguments
exec   ==> replacing the current shell with a different program
export ==> make the variable named as its parameter available in subshells
expr   ==> evaluate its arguments as an expression
printf ==> similar to echo
set    ==> sets the parameter variables for the shell. Useful for using fields in commands that output spaced-separated values
shift  ==> moves all the parameter variables down by one.
trap   ==> specify the actions to take on receipt of signals.
unset  ==> remove variables or functions from the environment.

set -e: Exit immediately if a command exits with a non-zero status. Type help set in command line. See stackoverflow.com

trap

• http://linuxcommand.org/wss0160.php
• http://www.tutorialspoint.com/unix/unix-signals-traps.htm
• http://www.ibm.com/developerworks/aix/library/au-usingtraps/
• http://bash.cyberciti.biz/guide/Trap_statement
• http://steve-parker.org/sh/trap.shtml (trap with a user-defined function)
• http://www.turnkeylinux.org/blog/shell-error-handling (set -e)
• http://unix.stackexchange.com/questions/17314/what-is-signal-0-in-a-trap-command (do something on EXIT)
• http://unix.stackexchange.com/questions/79648/how-to-trigger-error-using-trap-command

The syntax to use trap command is

trap command signal

For example,

$ cat traptest.sh
#!/bin/sh

trap 'rm -f /tmp/tmp_file_$$' INT
echo creating file /tmp/tmp_file_$$
date > /tmp/tmp_file_$$

echo 'press interrupt to interrupt ...'
while [ -f /tmp/tmp_file_$$ ]; do
  echo file exists
  sleep 1
done
echo the file no longer exists

trap - INT
echo creaing file /tmp/tmp_file_$$
date > /tmp/tmp_file_$$
echo 'press interrupt to interrupt ...'
while [ -f /tmp/tmp_file_$$ ]; do
  echo file exists
  sleep 1
done
echo we never get here
exit 0

will get an output like

$ ./traptest.sh
creating file /tmp/tmp_file_21389
press interrupt to interrupt ...
file exists
file exists
^Cthe file no longer exists
creaing file /tmp/tmp_file_21389
press interrupt to interrupt ...
file exists
file exists
^C

The first when we use trap, it will delete the file when we hit Ctrl+C. The second time when we use trap, we do not specify any command to be exected when an INT signal occurs. So the default behavior occurs. That is, the final echo and exit statements are never executed.

Note that the following two are different.

trap - INT
trap '' INT

The second command will IGNORE signals (Ctrl+C in this case) so if we apply this statement above, we will not be able to use Ctrl+C to kill the execution.

Command Execution

$(command)
`command`
# Example
sudo apt-get install linux-headers-$(uname -r)

Note all new scripts should use the $(...) form, which was introduced to avoid some rather complex rules.

Example

#!/bin/sh
echo The current directory is $PWD
echo The current users are $(who)
exit 0

The concept of putting the result of a command into a script variable is very powerful, as it makes it easy to use existing commands in scripts and capture their output.

Arithmetic Expansion

$((...))

is a better alternative to the expr command.

Parameter Expansion

${parameter}

pause by read -p command

http://www.cyberciti.biz/tips/linux-unix-pause-command.html

read -p "Press [Enter] key to start backup..."

If we want to ask users about a yes/no question, we can use this method

while true; do
    read -p "Do you wish to install this program? " yn
    case $yn in
        [Yy]* ) make install; break;;
        [Nn]* ) exit;;
        * ) echo "Please answer yes or no.";;
    esac
done

OR

echo "Do you wish to install this program?"
select yn in "Yes" "No"; do
    case $yn in
        Yes ) make install; break;;
        No ) exit;;
    esac
done

Working with Files

Low-level File Access

• file descriptors: 0 means standard input, 1 means standard output, 2 means standard error.
• size_t write(int fildes, const void *buf, size_t nbytes);

#include <unistd.h>
#include <stdlib.h>
int main()
{
  if ((write(1, "Here is some data\n", 18)) != 17)
    write(2, "A write error has occurred on file descriptor\n", 46);
  exit(0);
}

• size_t read(int fildes, void *buf, size_t nbytes); returns the number of data bytes actually read. If a read call returns 0, it had nothing to read; it reached the end of the file. An error on the call will cause it to return -1.
• To create a new file descriptor we use the open system call. int open(const char *path, int oflags, mode_t mode);

• The next program will do file copy.

#include <unistd.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
int main()
{
  char c;
  int in, out;
  in = open("file.in", O_RDONLY);
  out = open("file.out", O_WRONLY|O_CREAT, S_IRUSER|S_IWUSR);
  while(read(in,&c,1) == 1)
    write(out,&c,1)
  exit(0);
}

The Standard I/O Library

• fopen, fclose
• fread, fwrite
• fflush
• fseek
• fgetc, getc, getchar
• fputc, putc, putchar
• fgets, gets
• printf, fprintf and sprintf
• scanf, fscanf and sscanf

Formatted Input and Output

• prinf, fprintf and sprintf
• scanf, fscanf and sscanf

Scanning Directories

• opendir, closedir
• readdir
• telldir
• seekdir

Curses

A technique between command line and full GUI.

Example: vi.