Docker: Difference between revisions

From 太極
Jump to navigation Jump to search
Line 1,219: Line 1,219:
sudo docker push ouruser/sinatra
sudo docker push ouruser/sinatra
sudo docker rmi training/sinatra
sudo docker rmi training/sinatra
</syntaxhighlight>
* The above exercise works on my home computer (64-bit Ubuntu 12.04). However when I want to remove an image, I need to do something extra.
<syntaxhighlight lang='bash'>
$ sudo docker rmi training/sinatra
Error response from daemon: Conflict, cannot delete f0f4ab557f95 because the container bc5175998df3 is using it, use -f to force
FATA[0000] Error: failed to remove one or more images
$ sudo docker ps
CONTAINER ID        IMAGE              COMMAND            CREATED            STATUS              PORTS              NAMES
$
$ sudo docker rm $(sudo docker ps -aq)
bc5175998df3
b97cb467529c
$ sudo docker images
REPOSITORY          TAG                IMAGE ID            CREATED            VIRTUAL SIZE
ouruser/sinatra    v2                  bd6395724621        9 minutes ago      316.3 MB
ubuntu              14.04              ed5a78b7b42b        4 days ago          188.3 MB
ubuntu              latest              ed5a78b7b42b        4 days ago          188.3 MB
training/sinatra    latest              f0f4ab557f95        6 months ago        447 MB
$ sudo docker rmi training/sinatra
Untagged: training/sinatra:latest
Deleted: f0f4ab557f954f3e04177663a3af90e88641bcdcce1f02ac900dbd9768ef4945
...
Deleted: 3e76c0a80540a0d36493ae7110796fc92f559a191454e3ac19c1d4c650bdd9e0
$ sudo docker images
REPOSITORY          TAG                IMAGE ID            CREATED            VIRTUAL SIZE
ouruser/sinatra    v2                  bd6395724621        10 minutes ago      316.3 MB
ubuntu              latest              ed5a78b7b42b        4 days ago          188.3 MB
ubuntu              14.04              ed5a78b7b42b        4 days ago          188.3 MB
$ sudo docker rmi ouruser/sinatra
Error response from daemon: No such image: ouruser/sinatra:latest
FATA[0000] Error: failed to remove one or more images 
$ sudo docker rmi ouruser/sinatra:v2
Untagged: ouruser/sinatra:v2
Deleted: bd6395724621a89384ec58c116ce113ae8371f31f20f0adc540bbc76c6049d12
...
Deleted: 41ce107b0a0bb6a70834477e3b550386fc453de363d8a20f0579df055e9ece50
</syntaxhighlight>
</syntaxhighlight>


Line 1,262: Line 1,225:
$ sudo docker run -t -i training/sinatra /bin/bash
$ sudo docker run -t -i training/sinatra /bin/bash
2014/12/31 02:43:26 exec format error
2014/12/31 02:43:26 exec format error
</pre>
* I get lots of errors when I run docker build command on my 64-bit Ubuntu.
<pre>
$ sudo docker build -t="ouruser/sinatra:v2" .
Sending build context to Docker daemon  2.56 kB
Sending build context to Docker daemon
Step 0 : FROM ubuntu:14.04
Pulling repository ubuntu
ed5a78b7b42b: Download complete
511136ea3c5a: Download complete
fe95bf7d5f50: Download complete
9a4594fe74ea: Download complete
8c4b1edcceea: Download complete
---> ed5a78b7b42b
Step 1 : MAINTAINER Kate Smith <[email protected]>
---> Running in 63614919cafd
---> 5bac5869eb36
Removing intermediate container 63614919cafd
Step 2 : RUN apt-get update && apt-get install -y ruby ruby-dev
---> Running in 68e8ccfa5f7c
Err http://archive.ubuntu.com trusty InRelease
 
Err http://archive.ubuntu.com trusty-updates InRelease
 
Err http://archive.ubuntu.com trusty-security InRelease
 
Err http://archive.ubuntu.com trusty Release.gpg
  Could not resolve 'archive.ubuntu.com'
Err http://archive.ubuntu.com trusty-updates Release.gpg
  Could not resolve 'archive.ubuntu.com'
Err http://archive.ubuntu.com trusty-security Release.gpg
  Could not resolve 'archive.ubuntu.com'
Reading package lists...
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/InRelease 
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/InRelease 
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-security/InRelease 
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/Release.gpg  Could not resolve 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/Release.gpg  Could not resolve 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-security/Release.gpg  Could not resolve 'archive.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
ruby-dev : Depends: ruby1.9.1-dev (>= 1.9.3.194-1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
2014/12/30 16:03:21 The command [/bin/sh -c apt-get update && apt-get install -y ruby ruby-dev] returned a non-zero code: 100
</pre>
</pre>


Revision as of 09:44, 28 October 2019

Official web page http://docker.io.

Docker is both a client and a server: the server is a daemon that runs on Linux. The normal approach was that you used docker on the same server the daemon was running on - however it was possible to connect the docker client to a remote docker daemon.

Installation

Which OS to install?

Containers vs virtual machines

OS containers vs application containers

Differences:

  • OS containers: LXC, OpenVZ, Linux VServer, BSD Jails and Solaris zones. The container acts as VPS.
  • App containers: Docker, Rocket. The container acts as an application.

Current release version

Ubuntu x86 and Mint

# Uninstall old versions
$ sudo apt-get remove docker docker-engine docker.io

# Install using the repository
$ sudo apt-get update
$ sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
# Note that for Linux Mint we need to change $(lsb_release -cs) to a proper name
# For example, xenial for Mint 18.x and bionic for Mint 19.x
$ sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
$ sudo apt-get update
$ sudo apt-get install docker-ce

# Verify
$ sudo docker run hello-world

Without sudo, Post-installation

To use docker without sudo, follow the instruction on the official guide. 

# Add the docker group if it doesn't already exist.
# sudo groupadd docker

# Add your user to the docker group.
sudo usermod -aG docker $USER

# Log out and log in

CentOS

https://docs.docker.com/engine/installation/linux/docker-ce/centos/

Boot2Docker

For Windows and OS operation systems, we have to use Boot2Docker. Boot2Docker is a local virtual machine with its own network interface and IP address. To find the Boot2Docker IP address you can check the value of the DOCKER_HOST environment variable. You're be prompted to set this variable when you start or install Boot2Docker the first time. You can find the IP address by running boot2docker ip command.

Note that since Windows and OS X don't share a file system as Linux, the command 'docker run' with '-v' flag to mount a local directory into a Docker container will not work with Boot2Docker release prior to 1.3. The support for volumes is now available for OS X but is not yet present for Windows with the release of Boot2Docker 1.3.

Windows

Docker can be run on Windows 10 Pro as a native application; see

The information below is based on running Docker on Windows 7.1 and 8. Your processor needs to support hardware virtualization.

  • Windows Installer includes msys-git, Virtualbox, Boot2Docker-cli management tool and Boot2Docker ISO.
  • Installation instruction for Windows OS. It will install Boot2Docker management tool with the boot2docker iso (based on Tiny Core Linux), Virtualbox and MYSYS-git UNIX tools.
  • Docker needs Admin right to be installed. However, Virtualbox can be installed by user's account.
  • If the installer detects a version of VirtualBox installed, the VirtualBox checkbox will not be checked by default (Windows OS). The VirtualBox cannot be used anymore after updating my VB from 4.3.18 to 4.3.20. The error may be related to Windows update according to Virtualbox forum.
Error in supR3HardenedWinReSpawn
  • Note that boot2docker cannot be installed/run inside a Windows guest machine. See this post and my Virtualbox wiki here. If we try to launch boot2docker-vm from Virtualbox, we will see a message "This kernel requires an x86-64 CPU, but only detected an i686 CPU."
  • After I switch back to an old version of virtualbox, every thing works again. I can even install Docker successfully.
    • Boot2Docker Start icon cannot be run directly because Notepad++ will automatically open it. A possible solution is to go to control panel and change default program for .sh file from Notepad++ to C:\Program Files (x86)\Git\bin\bash.exe.
    • The above step does not work well since a terminal appears and disappears quickly.
    • A working approach is to open Git Bash from Start menu. And run /c/Program Files/Boot2DockerforWindows/start.sh (or boot2docker start or boot2docer init)
    • A new VM called 'boot2docker-vm' will be created (we can open VirtualBox Manager to check). But I got an error error in run: Failed to start machine "boot2docker-vm" (run again with -v for details). The VM has an error on Network>Adapter2>VirtualBox Host-Only Ethernet Adapter #2. So I open the setting of <boot2docker-vm>, go to Network > Adapter 2 and change the dropdown list of Name from VirtualBox Host-Only Ethernet Adapter #2 to VirtualBox Host-Only Ethernet Adapter.
    • Now it works either I directly click boot2docker-vm VM from VB Manager or use the command start.sh from Git Bash.

Boot2docker-vm.png 

$ # boot2docker is in the PATH variable, so there is not need to cd to the folder.
$ boot2docker start
initializing...
Virtual machine boot2docker-vm already exists

starting...
Waiting for VM and Docker daemon to start...
........o
Started.
Writing c:\Users\brb\.boot2docker\certs\boot2docker-vm\ca.pem
Writing c:\Users\brb\.boot2docker\certs\boot2docker-vm\cert.pem
Writing c:\Users\brb\.boot2docker\certs\boot2docker-vm\key.pem
Docker client does not run on Windows for now. Please use
    "c:\Program files\Boot2Docker for Windows\boot2docker.exe" ssh
to SSH into the VM instead.


192.168.56.101
connecting...
                        ##        .
                  ## ## ##       ==
               ## ## ## ##      ===
           /""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
           \______ o          __/
             \    \        __/
              \____\______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.4.1, build master : 86f7ec8 - Tue Dec 16 23:11:29 UTC 2014

Docker version 1.4.1, build 5bc2ff8
docker@boot2docker:~$ docker
Usage: docker [OPTIONS] COMMAND [arg...]

A self-sufficient runtime for linux containers.

Options:
  --api-enable-cors=false                      Enable CORS headers in the remote
 API
  -b, --bridge=""                              Attach containers to a pre-existi
ng network bridge
...
Run 'docker COMMAND --help' for more information on a command.
docker@boot2docker:~$ docker run hello-world
Unable to find image 'hello-world:latest' locally
hello-world:latest: The image you are pulling has been verified
511136ea3c5a: Pull complete
31cbccb51277: Pull complete
e45a5af57b00: Pull complete
Status: Downloaded newer image for hello-world:latest
Hello from Docker.
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (Assuming it was not already locally available.)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

For more examples and ideas, visit:
 http://docs.docker.com/userguide/

docker@boot2docker:~$ ls
boot2docker, please format-me
docker@boot2docker:~$ pwd
/home/docker
docker@boot2docker:~$ ls /
bin/     dev/     home/    lib/     mnt/     proc/    run/     sys/     usr/
c/       etc/     init     linuxrc  opt/     root/    sbin/    tmp      var/

docker@boot2docker:~$ docker run hello-world
Hello from Docker.
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (Assuming it was not already locally available.)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

For more examples and ideas, visit:
 http://docs.docker.com/userguide/
docker@boot2docker:~$
docker@boot2docker:~$
docker@boot2docker:~$
docker@boot2docker:~$ docker run -it ubuntu bash
Unable to find image 'ubuntu:latest' locally
ubuntu:latest: The image you are pulling has been verified
53f858aaaf03: Pull complete
837339b91538: Pull complete
615c102e2290: Pull complete
b39b81afc8ca: Pull complete
511136ea3c5a: Already exists
Status: Downloaded newer image for ubuntu:latest


root@ea7e3289a01a:/# pwd
/
root@ea7e3289a01a:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs           19G  269M   17G   2% /
none             19G  269M   17G   2% /
tmpfs          1005M     0 1005M   0% /dev
shm              64M     0   64M   0% /dev/shm
/dev/sda1        19G  269M   17G   2% /etc/hosts
tmpfs          1005M     0 1005M   0% /proc/kcore
root@ea7e3289a01a:/# ls
bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
boot  etc  lib   media  opt  root  sbin  sys  usr
root@ea7e3289a01a:/# exit
exit


docker@boot2docker:~$ pwd
/home/docker
docker@boot2docker:~$ ls
boot2docker, please format-me
docker@boot2docker:~$ exit
[Press any key to exit]

brb@NCI-01825357 /c/Program files/Boot2Docker for Windows
$ boot2docker down

brb@NCI-01825357 /c/Program files/Boot2Docker for Windows
$
$ boot2docker --help
Usage: c:\Program files\Boot2Docker for Windows\boot2docker.exe [<options>] <command> [<args>]

Boot2Docker management utility.

Commands:
   init                Create a new Boot2Docker VM.
   up|start|boot       Start VM from any states.
   ssh [ssh-command]   Login to VM via SSH.
   save|suspend        Suspend VM and save state to disk.
   down|stop|halt      Gracefully shutdown the VM.
   restart             Gracefully reboot the VM.
   poweroff            Forcefully power off the VM (may corrupt disk image).
   reset               Forcefully power cycle the VM (may corrupt disk image).
   delete|destroy      Delete Boot2Docker VM and its disk image.
   config|cfg          Show selected profile file settings.
   info                Display detailed information of VM.
   ip                  Display the IP address of the VM's Host-only network.
   shellinit           Display the shell commands to set up the Docker client.
   status              Display current state of VM.
   download            Download Boot2Docker ISO image.
   upgrade             Upgrade the Boot2Docker ISO image (restart if running).
   version             Display version information.

Options:
      --basevmdk="": Path to VMDK to use as base for persistent partition
      --clobber=false: overwrite Docker client binary on boot2docker upgrade
      --dhcp=true: enable VirtualBox host-only network DHCP.
      --dhcpip=192.168.59.99: VirtualBox host-only network DHCP server address.
....
  -v, --verbose=false: display verbose command invocations.
      --vm="boot2docker-vm": virtual machine name.
      --waittime=300: Time in milliseconds to wait between port knocking retries during 'start'
error in run: config error: pflag: help requested

brb@NCI-01825357 /c/Program files/Boot2Docker for Windows

The big picture 


                           start.sh                      docker run -it ubuntu bash
Git Bash Git Bash         ---------->  boot2docker-vm       ------------->   ubuntu
                                   docker@boot2docker:
   <-------               <----------                       <------------- 
   boot2docker down           exit                                 exit
   (shutdown boot2docker) (boot2docker-vm is still on)
    |
    |
    |  boot2docker up (start boot2docker)
    |
    |  boot2docker ssh (log into docker acct)
    |
    v
   boot2docker-vm
   docker@boot2docker

Increase boot2docker vmdk space

https://docs.docker.com/articles/b2d_volume_resize/

Install utilities in Boot2docker VM

http://blog.tutum.co/2014/11/05/how-to-use-docker-on-windows/

For example, to install cifs-utils, 

wget http://distro.ibiblio.org/tinycorelinux/5.x/x86/tcz/cifs-utils.tcz
tce-load -i cifs-utils.tcz

Mac

If you have Mac, you don't have to use boot2docker (iso & its management tool). You can use other Linux which comes with docker pre-installed. See this post.

Odroid

  • See Odroid magazine 2015 January and 2015 February. Note that the current versions of Docker and Docker Hub are not aware of the architecture for which the image has been built. All standard images are intended for the x86 architecture, and the autobuild feature offered by the Docker registry is only available for x86.

GUI/TUI interface manager

Dry

Dry – An Interactive CLI Manager For Docker Containers. The TUI is built on top of termui; a cross-platform, easy-to-compile, and fully-customizable terminal dashboard. It is inspired by blessed-contrib, but purely in Go.

LazyDocker (TUI)

LazyDocker: New Docker And Docker Compose Terminal UI, Github

Dockly (TUI)

Dockly – Manage Docker Containers From Terminal

Portainer* (nice)

Go to http://192.168.1.98:9000. But I still don't see the other containers that are running now:(
A username/password is required to access the page unless we use the --no-auth option; see Configuration.

DockerUI (Deprecated, Development continues at Portainer)

https://github.com/kevana/ui-for-docker. A quick start:

  1. Run: 
    docker run -d -p 9000:9000 --privileged \
    -v /var/run/docker.sock:/var/run/docker.sock uifd/ui-for-docker
    where -v means to bind mount a volume.
  2. Open your browser to http://<dockerd host ip>:9000

Note: Anyone in the local network can access the website without any authentication.

Rancher

$ sudo apt-get install ufw
$ sudo ufw allow 4500/udp
$ sudo ufw allow 500/udp
  • discoposse.com
    • Part 1 Installing Rancher and Setting Access Control
    • Part 2 Adding a Docker Host to Rancher
    • Part 3 Adding the DockerHub to our Rancher Registry
    • Part 4 Using the Catalog Example with GlusterFS

Seagull

https://youtu.be/TuT5gb8oRw8 

docker run -d -p 127.0.0.1:10086:10086 -v /var/run/docker.sock:/var/run/docker.sock tobegit3hub/seagull

The only issue is there is no username/password to protect other people to access the web GUI. The solution of binding to localhost to restrict the access does not work for remote administration.

That is, the tool is suitable for home use.

Kitematic (Mac and Windows only)

Available for Mac OS X 10.8+ and Windows 7+ (64-bit) and Ubuntu. https://github.com/docker/kitematic/releases/

Run containers through a simple, yet powerful graphical user interface.

A Share your Shiny Apps with Docker and Kitematic!

Shipyard (retired)

Usage

Basics, docs, cheatsheet

Note that we need sudo is needed unless it is on a Mac OS.

If docker cannot find an image, it will try to pull it from its repository. 

$ sudo docker run -it ubuntu /bin/bash
Unable to find image 'ubuntu' locally
Pulling repository ubuntu
04c5d3b7b065: Download complete 
511136ea3c5a: Download complete 
c7b7c6419568: Download complete 
70c8faa62a44: Download complete 
d735006ad9c1: Download complete 
root@ec83b3ac878d:/#
purpose command
run a container docker container run -d -p 80:80 httpd
list running cotainer docker container ls
view logs of Docker container docker container logs cranky_cori
identify Docker container process? docker container top cranky_cori
stop Docker container? docker container stop cranky_cori
list stopped or not running Docker containers docker container ls -a
start Docker container docker container start c46f2e9e4690
remove Docker container docker container rm cranky_cori
list Docker images docker images
remove Docker image docker rmi iman/touch

Restart docker daemon

When I try the Chap5 > Continuous integration (Jenkins) of the Docker Book, I found I cannot stop/kill the container. See others' report here. The solution is to restart the docker daemon. 

sudo service docker start

After that, I can stop and rm the container. 

sudo docker stop jenkins
sudo docker rm jenkins
sudo docker ps -a

images vs containers

$ sudo docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
iman                           latest              6e0f5644b2fd        About a minute ago   460.4 MB
iman/touch                     latest              77b9ac5951c2        4 minutes ago        460.4 MB
<none>                         <none>              aaa75e64ddf0        5 weeks ago          188.3 MB
ouruser/sinatra                v2                  ea8c9f407a8d        5 weeks ago          447 MB
ubuntu                         14.04               ed5a78b7b42b        5 weeks ago          188.3 MB
ubuntu                         latest              ed5a78b7b42b        5 weeks ago          188.3 MB
eddelbuettel/docker-ubuntu-r   add-r-devel-san     3c19d078c5d9        3 months ago         460.4 MB
hello-world                    latest              ef872312fe1b        4 months ago         910 B
training/sinatra               latest              f0f4ab557f95        8 months ago         447 MB

$ sudo docker ps -a
CONTAINER ID IMAGE                                          COMMAND              CREATED        STATUS                   PORTS NAMES
8fbdbcdb5126 iman/touch:latest                              "/bin/bash"          2 minutes ago  Exited (0) 2 minutes ago       thirsty_engelbart   
dc9e82f2c00a eddelbuettel/docker-ubuntu-r:add-r-devel-san   "/bin/bash"          9 minutes ago  Exited (0) 3 minutes ago       kickass_bardeen     
532a90f36aa8 eddelbuettel/docker-ubuntu-r:add-r-devel-san   "/bin/bash"          18 hours ago   Exited (0) 18 hours ago        happy_lalande       
7634024ee0bf eddelbuettel/docker-ubuntu-r:add-r-devel-san   "/bin/bash"          18 hours ago   Exited (0) 18 hours ago        insane_mclean       
14034a9720cb eddelbuettel/docker-ubuntu-r:add-r-devel-san   "/bin/bash"          18 hours ago   Exited (0) 18 hours ago        naughty_lumiere     
ca90954628db eddelbuettel/docker-ubuntu-r:add-r-devel-san   "/bin/bash"          19 hours ago   Exited (130) 18 hours ago      sick_hawking        
8bbdcb7c339f eddelbuettel/docker-ubuntu-r:add-r-devel-san   "/bin/bash"          19 hours ago   Exited (0) 19 hours ago        modest_davinci      
e8e24f80f0dd aaa75e64ddf0                                   "/bin/sh -c 'apt-get 5 weeks ago    Exited (100) 5 weeks ago       berserk_hodgkin     
d41959e0eb55 aaa75e64ddf0                                   "/bin/sh -c 'apt-get 5 weeks ago    Exited (100) 5 weeks ago       jovial_curie        
b408c0e2805b aaa75e64ddf0                                   "/bin/sh -c 'apt-get 5 weeks ago    Exited (100) 5 weeks ago       lonely_tesla        
72a551e4b492 ouruser/sinatra:v2                             "/bin/bash"          5 weeks ago    Exited (0) 5 weeks ago         jolly_meitner       
75fd6cc4658b training/sinatra:latest                        "/bin/bash"          5 weeks ago    Exited (0) 5 weeks ago         evil_yalow          
cc8886f5a02e training/sinatra:latest                        "/bin/bash"          5 weeks ago    Exited (130) 5 weeks ago       elegant_curie       
0585e4f5fecd eddelbuettel/docker-ubuntu-r:add-r-devel-san   "/bin/bash"          5 weeks ago    Exited (0) 5 weeks ago         elated_euclid       
brb@brbweb4:~/Downloads$

When we want to delete a container, we use the container's CONTAINER ID or NAME (last column output from docker ps -a). But when we want to delete an image, we use the image's REPOSITORY or IMAGE ID (2nd column output from docker images) 

$ sudo docker rm thirsty_engelbart  # iman/touch
$ sudo docker rm dc9e82f2c00a       # eddelbuettel/docker-ubuntu-r:add-r-devel-san
$ sudo docker ps -a   # check to see the container is gone now

$ sudo docker rmi 6e0f5644b2fd
$ sudo docker rmi iman/touch
$ sudo docker images  # check to see the images are gone now

Docker command line / CLI

$ docker

Usage:	docker COMMAND

A self-sufficient runtime for containers

Options:
      --config string      Location of client config files (default "/home/brb/.docker")
  -D, --debug              Enable debug mode
  -H, --host list          Daemon socket(s) to connect to
  -l, --log-level string   Set the logging level ("debug"|"info"|"warn"|"error"|"fatal") (default "info")
      --tls                Use TLS; implied by --tlsverify
      --tlscacert string   Trust certs signed only by this CA (default "/home/brb/.docker/ca.pem")
      --tlscert string     Path to TLS certificate file (default "/home/brb/.docker/cert.pem")
      --tlskey string      Path to TLS key file (default "/home/brb/.docker/key.pem")
      --tlsverify          Use TLS and verify the remote
  -v, --version            Print version information and quit

Management Commands:
  config      Manage Docker configs
  container   Manage containers
  image       Manage images
  network     Manage networks
  node        Manage Swarm nodes
  plugin      Manage plugins
  secret      Manage Docker secrets
  service     Manage services
  swarm       Manage Swarm
  system      Manage Docker
  trust       Manage trust on Docker images
  volume      Manage volumes

Commands:
  attach      Attach local standard input, output, and error streams to a running container
  build       Build an image from a Dockerfile
  commit      Create a new image from a container's changes
  cp          Copy files/folders between a container and the local filesystem
  create      Create a new container
  diff        Inspect changes to files or directories on a container's filesystem
  events      Get real time events from the server
  exec        Run a command in a running container
  export      Export a container's filesystem as a tar archive
  history     Show the history of an image
  images      List images
  import      Import the contents from a tarball to create a filesystem image
  info        Display system-wide information
  inspect     Return low-level information on Docker objects
  kill        Kill one or more running containers
  load        Load an image from a tar archive or STDIN
  login       Log in to a Docker registry
  logout      Log out from a Docker registry
  logs        Fetch the logs of a container
  pause       Pause all processes within one or more containers
  port        List port mappings or a specific mapping for the container
  ps          List containers
  pull        Pull an image or a repository from a registry
  push        Push an image or a repository to a registry
  rename      Rename a container
  restart     Restart one or more containers
  rm          Remove one or more containers
  rmi         Remove one or more images
  run         Run a command in a new container
  save        Save one or more images to a tar archive (streamed to STDOUT by default)
  search      Search the Docker Hub for images
  start       Start one or more stopped containers
  stats       Display a live stream of container(s) resource usage statistics
  stop        Stop one or more running containers
  tag         Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
  top         Display the running processes of a container
  unpause     Unpause all processes within one or more containers
  update      Update configuration of one or more containers
  version     Show the Docker version information
  wait        Block until one or more containers stop, then print their exit codes

Run 'docker COMMAND --help' for more information on a command.

Version, system information

Docker version 

$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:51 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:15 2018
  OS/Arch:          linux/amd64
  Experimental:     false

System information.

  • what mode the Docker engine is operating in (swarm mode or not)
  • what storage drive is used for the union filesystem
  • what version of the Linux kernel we have on our host
  • et al
$ docker system info
Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 10
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-33-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.674GiB
Name: t420s
ID: VLWB:6BN3:U7KB:L4T4:GQIB:54F3:YZKJ:PAIR:HEUM:UQIC:XLZU:3IFJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

List resource consumption

$ docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              10                  2                   2.58GB              1.519GB (58%)
Containers          2                   0                   304B                304B (100%)
Local Volumes       2                   0                   314.7MB             314.7MB (100%)
Build Cache         0                   0                   0B                  0B

$ docker system df -v  # more detailed information
# We can use the information to clean up our system

A brief intro to docker virtualization

docker search --help
docker search redis
docker search -s 100 redis
docker pull --help
docker pull ubuntu # download all versions of ubuntu
docker images    # available local container images
docker pull centos:latest
docker run --help
cat /etc/issue   # look at the current distr name before running docker
docker run -it centos:latest /bin/bash
                 # create a container & execute as a sudo

cat /etc/redhat-release
yum
cd /home
touch temp.txt
ls
exit

docker ps   # current running processes
docker ps -a # show all processes including closed
docker restart c85850ed0e13
docker ps   # container c85850ed0e13 is running
docker attach c85850ed0e13 # log into the system

ls /home
exit

docker ps -a
docker rm c85850ed0e13 # delete the container

Note: Following the discussion, using attach can only launch one instance of shell. If we use exec, we can launch multiple instances. 

sudo docker exec -i -t c85850ed0e13 bash #by ID
or
$ sudo docker exec -i -t loving_heisenberg bash #by Name

docker pull

https://docs.docker.com/engine/reference/commandline/pull/ 

$ docker pull ubuntu:zesty
$ docker run -ti --rm ubuntu:zesty /bin/bash 
# lsb_release -a         
bash: lsb_release: command not found
# cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=17.04
DISTRIB_CODENAME=zesty
DISTRIB_DESCRIPTION="Ubuntu 17.04"
NAME="Ubuntu"
VERSION="17.04 (Zesty Zapus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 17.04"
VERSION_ID="17.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=zesty
UBUNTU_CODENAME=zesty

Update images

I have an image called 'rocker/tidyverse:latest' that was pulled sometime ago. I ran docker pull rocker/tidyverse:latest' again. It will pull the latest image. The old image was not deleted and it has a new tag <none>.

How to remove a signed image with a <none> tag

<none>:<none> images

Exit/detach from a container without stopping it

$ docker container run -it ubuntu:latest /bin/bash
# Ctrl+p, Ctrl+q to exit the container without terminating it
$ docker ps -a # showing the container 70c5aceb5512 is running in the background

# You can reattach your terminal to it with the "docker container exec" command
$ docker container exec -it 70c5aceb5512 bash

Dockerizing Applications/Detached mode

$ sudo docker run -d --name insane_babbage ubuntu:14.04 /bin/sh -c "while true; do echo hello world; sleep 1; done"
$ sudo docker ps -l
$ sudo docker logs insane_babbage
$ sudo docker stop insane_babbage
$ sudo docker ps

The -d flag tells Docker to run the container and put it in the background, to daemonize it.

According to https://docs.docker.com/engine/reference/run/#detached-vs-foreground, containers started in detached mode exit when the root process used to run the container exits, unless you also specify the --rm option. If you use -d with --rm, the container is removed when it exits or when the daemon exits, whichever happens first.

Automatically restart after reboot

https://stackoverflow.com/questions/18786054/how-to-auto-restart-a-docker-container-after-a-reboot-in-coreos

Add a --restart=always parameter. It will always restart a stopped container unless it has been explicitly stopped, such as via a "docker container stop" command. See the following 

$ docker run -d --restart always myCustomeDocker

$ docker container run --name neverdie -it --restart always ubuntu /bin/bash
# exit
$ docker ps -a  # the container is still ther
$ docker stop neverdie
$ docker ps -a

Working with Containers

$ sudo docker run -i -t ubuntu /bin/bash
$ sudo docker version
$ sudo docker
$ sudo docker attach --help

Alpine image

apk add htop

Running a Web Application

$ sudo docker run -d -P training/webapp python app.py

Alpine linux is 6MB. It is a good OS to run a web application. See the demo here.

Viewing our Web Application Container

$ sudo docker ps -l
$ sudo docker run -d -p 5000:5000 training/webapp python app.py

Container networking

$ docker network create MyNewNetworkName
$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
abae0010bf2c        MyNewNetworkName    bridge              local
bf4e73473028        bridge              bridge              local
7dac0804bc33        host                host                local
dbe7f0daef6d        none                null                local
$ docker network remove MyNewNetworkName
# OR docker network rm MyNewNetworkName

Host network

If you use the host network driver for a container, that container’s network stack is not isolated from the Docker host. For instance, if you run a container which binds to port 80 and you use host networking, the container’s application will be available on port 80 on the host’s IP address.

ping, ifconfig and ip commands not found in Ubuntu container

apt update
apt install iputils-ping  # ping 
apt install net-tools     # ifconfig
apt install iproute2      # ip

Network Port Shortcut

$ sudo docker port nostalgic_morse 5000

Viewing the Web Application's Logs

$ sudo docker logs -f nostalgic_morse

Looking at our Web Application Container's processes

$ sudo docker top nostalgic_morse

Inspecting our Web Application Container

$ sudo docker inspect nostalgic_morse

Obtain the container's IP address, log into a running server

$ docker inspect <container id> | grep "IPAddress"

Then to log into a running server, use 

$ docker exec -it <contianer id> bash

docker exec

Run a command in a running container

  • Usage: 
    docker exec [OPTIONS] CONTAINER COMMAND [ARG...]
  • Examples: 
    $ docker exec -d ubuntu_bash touch /tmp/execWorks # do st in the background

$ docker exec -it ubuntu_bash bash

$ docker exec -it -e VAR=1 ubuntu_bash bash # set an environment variable

$ docker exec -it ubuntu_bash pwd
$ docker exec -it -w /root ubuntu_bash pwd # change the working directory

Restart a Container

$ sudo docker start nostalgic_morse
OR
$ sudo docker restart nostalgic_morse

Inspect container images and their metadata

MicroBadger

Know the container size

docker ps -s

Meaning of two sizes

  • The "size" information shows the amount of data (on disk) that is used for the writable layer of each container
  • The "virtual size" is the amount of disk-space used for the read-only image data used by the container.

Removing our Web Application Container

$ sudo docker stop nostalgic_morse
$ sudo docker rm nostalgic_morse

Note: Always remember that deleting a container is final!

Dockerize an SSH service

https://docs.docker.com/engine/examples/running_ssh_service/#environment-variables

Remove old docker containers

This post on stackoverflow.com. 

$ sudo docker ps -a | grep 'weeks ago' | awk '{print $1}' | xargs --no-run-if-empty sudo docker rm

Similarly to remove all exited containers 

$ sudo docker ps -a | grep Exit | awk '{print $1}' | xargs sudo docker rm

To kill/stop (not delete) all running containers 

$ sudo docker kill $(sudo docker ps -q)

To delete all stopped containers 

$ sudo docker rm $(sudo docker ps -a -q)
OR
$ sudo docker rm `sudo docker ps -a -q`

It is also helpful to create bash aliases for these commands by editing ~/.bash_aliases file.

docker create vs docker run

https://stackoverflow.com/questions/37744961/docker-run-vs-create

docker create is similar to docker run -d except the container is never started.

Volume

Inspect the 'Mountpoint' of a volume

$ docker volume create crv
$ docker volume ls

$ docker run -d \
     --name mycloud \
     -p 81:80 \
     -v apps:/var/www/html/custom_apps \
     nextcloud

# docker inspect is not quite useful. It does not show how the volume was created
# But we can examine (ls, du, ...) the directory contents
$ docker inspect apps   
[
    {
        "CreatedAt": "2018-10-23T09:41:52-04:00",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/apps/_data",
        "Name": "apps",
        "Options": null,
        "Scope": "local"
    }
]

Remove an an unnamed volume

If you created an unnamed volume, it can be deleted at the same time as the container with the -v flag. Note that this only works with unnamed volumes. 

docker rm -v container_name

If the volume is named, it stays present. To remove a named volume, use docker volume rm volume_name .

Start a container with a volume

--mount -v
docker run -d \

--name devtest \
--mount source=myvol2,target=/app \
nginx:latest

docker run -d \

--name devtest \
-v myvol2:/app \
nginx:latext

Note

  • target in "--mount" can be replaced by destination or dst.
  • To use a read-only volume, add the ,readonly option in "--mount" or the :ro option in "-v".

A simple example

From the book "Learn Docerk -Fundamentals of Docker 18.x". Chap 5. Data Volumes and System Management > Creating and mounting data volumes. 

# Create a volume
docker volume create my-data
docker volume inspect my-data
# The host folder can be found in the output under 'Mountpoint'
# In my case,
#        "Mountpoint": "/var/lib/docker/volumes/my-data/_data",

# Mount a volume into a container
docker run --name test -it -v my-data:/data alpine /bin/sh
# cd /data
# echo 'some data' > data.txt
# echo 'more data' > data2.txt
# exit
docker inspect my-data
sudo ls /var/lib/docker/volumes/my-data/_data
# We can even try to output the content of say, the second file:
sudo cat /var/lib/docker/volumes/my-data/_data/data2.txt
# We can create a new file in this folder from the host and then use the volume with another container
echo "the file is created on host" > sudo tee /var/lib/docker/volumes/my-data/_data/host-data
# Let's delete the test container and run another one
docker rm test

# This time we are mounting our volume to a different container folder
docker run --name test2 -it -v my-data:/app/data centos:7 /bin/bash
# We are able to see three files:
# ls /app/data

# Remove volumes
docker volume rm my-data # Or 
docker volume rm $(docker volume ls -q)

# Remove all running containers to clean up the system,
docker rm -f $(docker ls -aq)

Sharing data between containers

docker run -it --name writer -v shared-data:/data alpine /bin/sh
# create a file inside it
# echo 'my sample file' > /data/sample.txt
# exit
docker run -it --name reader -v shared-data:/app/data:ro ubuntu:17.04 /bin/bash
# ls -l /app/data

Using host volumes

Use volumes that mount a specific host folder

  • It may be possible for the "docker volume" command to mount a local directory to a volume. See examples in the "docker volume create" documentation.
  • Specifying a directory name instead of giving a volume name in the "docker run" 's -v option
  • Since we are specifying a directory name instead of letting docker to create a new volume, "docker volume ls" will not getting a new volume
docker run -it --name test -v $(pwd)/src:/app/src alpine /bin/sh

# Make a sample to demonstrate how that works
mkdir ~/my-web; cd ~/my-web
echo "<h1>My website</h1>" > index.html

# Create 'Dockerfile'
echo -e 'FROM nginx:alpine
COPY . /usr/share/nginx/html' > Dockerfile

docker image build -t my-website:1.0 .
docker run -d -p 8080:80 --name my-site my-website:1.0

# Open http://localhost:8080. It looks good
# Now modify index.html and refresh the website. It does not refresh
# Let's stop and rm the container and rebuild using a volume
docker rm -f my-site
docker run -d -v $(pwd):/usr/share/nginx/html \
   -p 8080:80 --name my-site my-website:1.0
# Now any changes on index.html will refresh on the website

Define volumes in images

A few samples of volume definition 

VOLUME /app/data
VOLUME /app/data, /app/profiles, /app/config
VOLUME {"/app/data", "/app/profiles", "/app/config"]

The first line defines a single volume to be mounted at /app/data.

We can use the docker image inspect command to get information about the volumes defined in the Dockerfile. 

docker image pull mongo:3.7
docker image inspect --format='{{json .ContainerConfig.Volumes}}' \
       mongo:3.7 | jq
# {
#   "/data/configdb": {},
#   "/data/db": {}
# }

# now run an instance of MongoDB and inspect the volume information
docker run --name my-mongo -d mongo:3.7
docker inspect --format '{{json .Mounts}}' my-mongo | jq
# [
#  {
#    "Type": "volume",
#    "Name": "535e0138b9a32e89f71380e9e73bb0de64ce0d1cad78fcda0ec1d49e11d76d7a",
#    "Source": "/var/lib/docker/volumes/535e0138b9a32e89f71380e9e73bb0de64ce0d1.../_data",
#    "Destination": "/data/configdb",
#    "Driver": "local",
#    "Mode": "",
#    "RW": true,
#    "Propagation": ""
#  },
#  {
#    "Type": "volume",
# SKIP

Differences between VOLUME and '-v|--volume'

https://stackoverflow.com/a/25312719

Backup and restore container

Container Memory Limits, Setting Available CPUs, Allocating memory and CPU

docker run \
    -rm \ ## Automatically remove the container when it exits
    --memory=6g \ ## memory limit
    --cpus=1.5 \ ## number of CPUs
    -v /shared/data-store:/home/rstudio/data \
    -v /shared/library-store:/usr/local/lib/R/host-site-library \
    -e PASSWORD=bioc \
    -p 8787:8787 \
         bioconductor/bioconductor_full:devel

Work with container images

Create an image interactively using commit - Example 1

The example is from the book 'Learn Docker - Fundamentals of Docker 18.x'. 

docker container run -it --name sample alpine /bin/sh
# apk update && apk add iputils
# ping 127.0.0.1
# exit
docker container ls -a | grep sample
docker container diff sample

We can now use the docker container commit command to persist our modifications and create a new image from them 

docker container commit sample my-alpine
docker images ls

If we want to see how our custom image has been built, we can use the history command as follows: 

docker image history my-alpine
# IMAGE               CREATED              CREATED BY                                      SIZE    COMMENT
# 0f105057899b        About a minute ago   /bin/sh                                         1.55MB              
# 196d12cf6ab1        4 weeks ago          /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B                  
# <missing>           4 weeks ago          /bin/sh -c #(nop) ADD file:25c10b1d1b41d46a1…   4.41MB

The first layer in the preceding list is the one we just created by adding the iputils package.

Create an image interactively using commit - Example 2

Note that it is better/necessary to put the Dockerfile in an empty directory to avoid the problem of taking a long time to build the image (sending build context to Docker daemon ...GB ) since it will grab files from the current directory. 

sudo docker search sinatra
sudo docker pull training/sinatra
sudo docker run -t -i training/sinatra /bin/bash
sudo docker commit -m="Added json gem" -a="Kate Smith" 0b2616b0e5a8 ouruser/sinatra:v2
sudo docker images

mkdir sinatra
cd sinatra
touch Dockerfile
sudo docker build -t="ouruser/sinatra:v2" .
sudo docker push ouruser/sinatra
sudo docker rmi training/sinatra
  • I get an error when I try to launch sinatra on my 32-bit ubuntu (Docker can only be installed through apt-get on 32-bit)
$ sudo docker run -t -i training/sinatra /bin/bash
2014/12/31 02:43:26 exec format error

How to copy Docker images from one host to another without using a repository

https://stackoverflow.com/questions/23935141/how-to-copy-docker-images-from-one-host-to-another-without-using-a-repository 

docker save -o out.tar <image name>
# Or better to compress the file
docker save <docker image name> | gzip > out.tar.gz

And restore 

docker load -i out.tar
# Or decompress the file
docker load < out.tar.gz

Resources allocated to a container using docker?

https://stackoverflow.com/questions/16084741/how-do-i-set-resources-allocated-to-a-container-using-docker

hub.docker.com

docker tag local-image:tagname new-repo:tagname
docker push new-repo:tagname
docker pull phusion/baseimage
docker run -ti phusion/baseimage /bin/bash
  • https://dockerfile.github.io/ which includes dockerfiles for different purposes. The ubuntu-desktop one also works well (client needs a vnc viewer in order to see the desktop).

ARM architeture from hub.docker.com

Note that if I follow the Debian instruction from Docker on Raspberry Pi 3, I'll get an error. 

pi@pi3:~ $ sudo add-apt-repository \
>    "deb [arch=armhf] https://download.docker.com/linux/debian \
>    $(lsb_release -cs) \
>    stable"
Traceback (most recent call last):
  File "/usr/bin/add-apt-repository", line 95, in <module>
    sp = SoftwareProperties(options=options)
  File "/usr/lib/python3/dist-packages/softwareproperties/SoftwareProperties.py", line 109, in __init__
    self.reload_sourceslist()
  File "/usr/lib/python3/dist-packages/softwareproperties/SoftwareProperties.py", line 599, in reload_sourceslist
    self.distro.get_sources(self.sourceslist)    
  File "/usr/lib/python3/dist-packages/aptsources/distro.py", line 89, in get_sources
    (self.id, self.codename))
aptsources.distro.NoDistroTemplateException: Error: could not find a distribution template for Raspbian/stretch

Now try this: How to Install Docker on Raspbian OS by Shahriar Shovon.

Set up a private Docker registry

$ curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X GET http://localhost:5000/v2/_catalog
$ OR
$ curl -H "Accept: application/xml" -H "Content-Type: application/json" -X GET http://localhost:5000/v2/_catalog

Google cloud registry

Using google cloud registry for private docker images

Dockerfile

  • Dockerfile Reference
  • Using Dockerfiles to Automate Building of Images from digitalocean.com.
  • Remember to put the Dockerfile in an empty directory.
  • Keywords
    • FROM. If we want to start from scratch, we can use FROM scratch.
    • RUN. The argument for RUN is any valid Linux command.
    • USER. This is useful if we want to create new files with a non-root owner privilege. For example, new files created under a binding directory with a non-root user ownership will belong to the current user in the host system.
    • COPY & ADD.
      • "COPY . /app" will copy all files and folders from the current directory recursively to the /app folder. We can use "ADD" too but "ADD" will automatically unpack tarballs. See What is the difference between the `COPY` and `ADD` commands in a Dockerfile?
      • "ADD sample.tar /app/bin" will unpack the sample.tar' file into the target folder
      • "ADD http://example.com/sample.txt /data/" will copy the remote file sample.txt into the target file
    • WORKDIR. Define the working directory or context that is used when a container is run from the image.
    • CMD & ENTRYPOINT. These two are actually definitions of what will happen when a container is started from the image.
      • Use CMD without ENTRYPOINT: "CMD command param1 param2". This form is called the shell form.
      • If we use ENTRYPOINT + CMD, ENTRYPOINT defines the command and CMD defines parameters. The example above will run ping 8.8.8.8 -c 3. This form is called the exec form.
  • The Docker Book
  • rocker (R and RStudio)
  • Bioconductor

Examples of Dockerfile

  • A sample. Six layers (lines). The first line is the Base Layer.
FROM python:2.7
RUN mkdir -p /app
WORKDIR /app
COPY ./requirements.txt /app/
RUN pip install -r requirements.txt
CMD ["python", "main.py"]
  • Another example
FROM alpine:latest
ENTRYPOINT ["ping"]
CMD ["8.8.8.8", "-c", "3"]

How to use Dockerfile

https://docs.docker.com/engine/reference/commandline/build/ 

docker build -t [myname] .  
# Multiple tags
docker build -t arraytools/biospear:latest -t arraytools/biospear:3.6.0 .

In the above example, we can create the image by 

docker image build -t pinger .

We can run a container from the pinger image 

docker container run --rm -it pinger

The advantage of using ENTRYPOINT + CMD (exec form) instead of using CMD alone (shell form) is we can override the CMD part that I have defined in the Dockerfile. 

docker container run --rm -it pinger -w 5 127.0.0.1
# ping the loopback for 5 seconds

If we want to overwrite what's defined in the ENTRYPOINT in the Dockerfile, we need to use the --entrypoint parameter. 

docker container run --rm -it --entrypoint /bin/sh pinger
# we'll be inside the container. Type exit to leave the container

When we use the shell form, the ENTRYPOINT is have the default value of /bin/sh -c and whatever is the value of CMD will be passed as a string to the shell command.

Temporary failure resolving 'deb.debian.org' when running "docker build"

Add "--net=host" to the docker build command. See Docker build “Could not resolve 'archive.ubuntu.com'” apt-get fails to install anything

Best practices for writing Dockerfiles

https://docs.docker.com/develop/develop-images/dockerfile_best-practices/

Use multi-stage builds

https://docs.docker.com/develop/develop-images/multistage-build/

With multi-stage builds, we have a single Dockerfile containing multiple FROM instructions. Each FROM instruction is a new build stage that can easily COPY artifacts from previous stages.

An example from the "Docker Deep Dive" book.

tag after image was built

$ docker tag <imageID> <newName>/<repoName>:<tagName>

About storage drivers

https://docs.docker.com/storage/storagedriver/#sharing-promotes-smaller-images

Dockerfile in One Line

FROM ubuntu

Using This simple Dockerfile and the docker command sudo docker build -t scooby_snacks . will result in 

$ sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
ubuntu              15.04               2427658c75a1        42 hours ago        117.5 MB
ubuntu              vivid               2427658c75a1        42 hours ago        117.5 MB
ubuntu              vivid-20150218      2427658c75a1        42 hours ago        117.5 MB
ubuntu              utopic-20150211     78949b1e1cfd        42 hours ago        194.4 MB
ubuntu              utopic              78949b1e1cfd        42 hours ago        194.4 MB
ubuntu              14.10               78949b1e1cfd        42 hours ago        194.4 MB
ubuntu              14.04               2d24f826cb16        42 hours ago        188.3 MB
ubuntu              14.04.2             2d24f826cb16        42 hours ago        188.3 MB
ubuntu              trusty              2d24f826cb16        42 hours ago        188.3 MB
ubuntu              trusty-20150218.1   2d24f826cb16        42 hours ago        188.3 MB
ubuntu              latest              2d24f826cb16        42 hours ago        188.3 MB
scooby_snacks       latest              2d24f826cb16        42 hours ago        188.3 MB
ubuntu              precise             1f80e9ca2ac3        42 hours ago        131.5 MB
ubuntu              precise-20150212    1f80e9ca2ac3        42 hours ago        131.5 MB
ubuntu              12.04.5             1f80e9ca2ac3        42 hours ago        131.5 MB
ubuntu              12.04               1f80e9ca2ac3        42 hours ago        131.5 MB
ubuntu              14.04.1             5ba9dab47459        3 weeks ago         188.3 MB
ubuntu              12.10               c5881f11ded9        8 months ago        172.2 MB
ubuntu              quantal             c5881f11ded9        8 months ago        172.2 MB
ubuntu              13.04               463ff6be4238        8 months ago        169.4 MB
ubuntu              raring              463ff6be4238        8 months ago        169.4 MB
ubuntu              13.10               195eb90b5349        8 months ago        184.7 MB
ubuntu              saucy               195eb90b5349        8 months ago        184.7 MB
ubuntu              10.04               3db9c44f4520        10 months ago       183 MB
ubuntu              lucid               3db9c44f4520        10 months ago       183 MB

Run a shell script on host

$ docker run -v /path/to/sample_script.sh:/sample_script.sh \
  --rm ubuntu bash sample_script.sh

# GATK container example
# First we log in interactive and see where is the default location (/usr in this case)
$ docker run --rm -i -t broadinstitute/gatk3:3.8-0 bash
$ cat > tmp.sh << EOF
> pwd
> ls
> java -jar GenomeAnalysisTK.jar --version
> EOF
$ docker run --rm -v $(pwd):/usr/my broadinstitute/gatk3:3.8-0 bash my/tmp.sh
# ALTERNATIVELY, WE CAN PUT OUR SCRIPT IN THE TOP DIRECTORY (Hopefully the name is not duplicated)
$ docker run --rm -v $(pwd)/tmp.sh:/tmp.sh broadinstitute/gatk3:3.8-0 bash /tmp.sh

docker run -d -v$(pwd):/my SOMEIMAGE bash 
docker exec -d Test bash /my/script.sh

Best practices

Intro Guide to Dockerfile Best Practices

Link containers together

Manage data in containers

Assign a static IP to a container

Docker DNS/internet problem

I got an error on resolving the debian server when I was creating an image from a Dockerfile that needs to run apt update and apt install commands. See RStudio in Docker – now share your R code effortlessly!. The problem happened on my Linux Mint Desktop but not on a VirtualBox VM (Ubuntu 18.04).

Fix Docker's networking DNS config

A temporary solution is to add the --dns option to docker run command. This works well when I use the IP from any one of my 2 DNS servers. It does not work however if I use the IP from google DNS or OpenDNS.

A permanent solution is to create a new file /etc/docker/daemon.json and include the working DNS server IPs (these are obtained through the nmcli command or the NetworkManager GUI; see Query DNS server). 

{
    "dns": ["XXX.XX.XX.XX", "YYY.YY.YY.YY"]
}

Then restart the docker service: sudo service docker restart

A quick test on the DNS problem is 

docker run --rm busybox nslookup google.com

Working with Docker hub

https://docs.docker.com/userguide/dockerrepos/

Enabling HTTPS/Let's encrypt

Nginx reverse proxy

Setting up a Reverse-Proxy with Nginx and docker-compose

Running multiple web applications on a Docker host

GUI apps

Pruning unused resources

  • Prune containers
docker container prune # remove all containers that are not in ''running'' status
                       # Docker will ask for confirmation before deleting the containers

docker container prune -f
docker container rm -f $(docker container ls -aq) # remove even the running containers
  • Prune images
docker images prune # unused image layers
  • Prune volumes
docker volume prune # unused volumes by at least one container

docker volume prune --filter 'label=demo'
docker volume prune --filter 'label=demo' --filter 'label=test'
  • Prune networks
docker network prune
  • Prune everything
docker system prune

Misc

LXC (raw Linux containers)

LXC vs Docker

Vagrant vs Docker

Date/Time zone

docker run --rm -t -i -v /etc/localtime:/etc/localtime:ro ubuntu date

Access the internet from the container

Run the container with the '--net=host' option 

sudo docker run --net=host -it ubuntu /bin/bash

How to transfer/copy an image to another host

How to copy Docker images from one host to another without using a repository 

# Step 1: save the Docker image as a tar file:
docker save -o <path for generated tar file> <image name>

# Step 2: copy your image to a new system with regular file transfer tools such as cp or scp. 

# Step 3: After that you will have to load the image into Docker:
docker load -i <path to image tar file>

The tar file size is the same as what we get from 'docker image'. If we use the 'gzip' utility, it can reduce the file size (e.g 2.7GB to 1.1GB).

Or https://stackoverflow.com/a/39716019 

# Step 1:
docker save docker-image-name | gzip > my-image.tar.gz
# Step 3:
docker load < my-image.tar.gz

Where are Docker images stored on the host: /var/lib/docker

The default is /var/lib/docker. The location can be changed by modifying the file /etc/default/docker. Three options if we are tight on the disk space.

1. Create a softlinks for the Docker data directory (/var/lib/docker) and for /var/lib/docker/tmp as described at miscellaneous-options. See this. See for how to stop docker daemon on different OS. 

sudo service docker stop   # or sudo systemctl stop docker
sudo mv /var/lib/docker /a/new/location
sudo ln -s /a/new/location /var/lib/docker # Create a symbolic link
sudo service docker start  # or sudo systemctl start docker

2. Change the default location to another place. For example, 

sudo nano /etc/default/docker
# Add a line DOCKER_OPTS="-g /home/brb/Docker"

Then after running sudo service docker.io restart and then a simple pull sudo docker pull rocker/r-base or sudo docker run --rm -ti rocker/r-base (the Dockerfile of r-base is available on github.com, --rm option means Automatically remove the container when it exits), we will see something like this: 

$ docker run --rm -ti rocker/r-base
$ docker images
$ docker -v
Docker version 1.0.1, build 990021a

$ docker -D info | grep Root
 Root Dir: /home/brb/Docker/aufs

Consuming Docker system events

# Open a new terminal
docker system events
# This command is a blocking command. 
# Thus, when you execute it in your terminal session the according session is blocked.

# Open another terminal
docker container run --rm alpine echo "Hello World"

Monitor tools

Docker Machine

Docker Machine is a tool that lets you

  • Install Docker Engine on virtual hosts. You can use Machine (a unified way) to create Docker hosts on your local Mac or Windows box, on your company network, in your data center, or on cloud providers like Azure, AWS, or Digital Ocean. See the comment on here.
  • Provision and manage multiple remote Docker hosts
  • Provision Swarm clusters

Docker machine is not installed in Linux when you install Docker. See the instruction on here to install it.

My feeling is if we just want to play Docker on a local Linux machine, we don't really need to use Docker Machine (it just make life more complicated). But if we are working on Mac/Windows or we want to work on clouds or test on VirtualBox, we shall use Docker machines.

Use Docker-machine to Create Docker Servers. Compare the Docker images on the local machine (server 1) & a new host (server 2) created by docker-machine. Question: 1. how to tell we are in the host/machine environment? 2. how to exit the host environment after we use eval $() command? docker-machine stop MachineName. 

$ docker-machine help
$ docker-machine create --driver=virtualbox test
# Follow its hint on the output, issue the following command
$ docker-machine env test
# Follow its hint on the output, issue the following command
$ eval$(docker-machine env test) # will configure the docker CLI to connect to this docker machine 'test'
                                 # This is equivalent to running 4 export commands on the command line
$ docker-machine ls  # Very useful
$ docker-machine stop test
$ docker-machine ip test
$ docker-machine start test
$ docker-machine rm test

Play Docker Machine on Mac with Virtualbox. Docker can be used to create a virtual machine just like Vagrant. 

$ docker-machine create -d virtualbox demo
$ docker-machine ls

# first way to access a Docker host
$ docker-machine ssh demo
docker@demo:~$ docker images # empty for now

# second way to access 
$ docker-machine env demo
$ eval $(docker-machine env demo)
$ docker version

RancherOS demo video used the docker-machine command to pull and run the RancherOS. 

docker-machine create -d virtualbox --virtualbox-boot2docker-url https://releases.rancher.com/os/latest/rancheros.iso demo
docker-machine ssh demo
ps
docker ps
sudo system-docker ps

sudo ros help
sudo ros console list
sudo ros console switch ubunu
apt-get help

Docker Compose <docker-compose.yaml>

Docker Compose can help us out as it allows us to specify a single file in which we can define our entire environment structure and run it with a single command (much like a Vagrantfile works).

An example from 'Fundamentals of Docker'

git clone https://github.com/fundamentalsofdocker/labs.git
cd labs/ch08
docker-compose up
# Open http://localhost:3000/pet

The images do not show up:( The terminal shows what has happened under the hood. So the problem is the http links for images do not exist.

We can also run the application in the background 

docker-compose up -d

To stop and clean up the application, 

docker-compose down

If we also want to remove the volume for the database 

docker volume rm ch08_pets-data

An example from "Docker Deep Dive"

The example is the same as the one on Get started with Docker Compose (some files are a little different. This example works!) 

$ git clone https://github.com/nigelpoulton/counter-app.git
$ cd counter-app
$ ls
app.py  docker-compose.yml  Dockerfile  README.md  requirements.txt

$ cat requirements.txt 
flask

$ cat Dockerfile
FROM python:3.4-alpine
ADD . /code
WORKDIR /code
RUN pip install -r requirements.txt
CMD ["python", "app.py"]

$ cat docker-compose.yml 
version: "3.5"
services:
  web-fe:
    build: .
    command: python app.py
    ports:
      - target: 5000
        published: 5000
    networks:
      - counter-net
    volumes:
      - type: volume
        source: counter-vol
        target: /code
  redis:
    image: "redis:alpine"
    networks:
      counter-net:

networks:
  counter-net:

volumes:
  counter-vol:

$ docker-compose up &

$ docker container ls

$ docker network ls
NETWORK ID          NAME                     DRIVER              SCOPE
2acef6dabde6        bridge                   bridge              local
a2d42bc482ff        counterapp_counter-net   bridge              local
e1e093b64282        host                     host                local
7ecd0a6a9ebd        none                     null                local

# Open the browser http://localhost:5000
$ docker-compose ps
       Name                      Command               State           Ports         
-------------------------------------------------------------------------------------
counterapp_redis_1    docker-entrypoint.sh redis ...   Up      6379/tcp              
counterapp_web-fe_1   python app.py                    Up      0.0.0.0:5000->5000/tcp

$ docker-compose stop
$ docker-compose ps
# We can see stopping a Compose app does not delete the application

$ docker container ls -a
$ docker-compose rm     # delete a stopped Compose app
                        # images, volumes and source code remain
$ docker-compose restart
                        # If you made changes to your Compose app since stopping,
                        # these changes will not appear in the restarted app.
                        # You need to re-deploy the app to get the changes.
$ docker-compose ps
$ docker-compose down   # stop and delete the app
                        # images, volumes and source code remain
$ docker-compose down --volumes # remove the data volume used by the Redis container
$ docker-compose up -d 
$ docker volume ls
$ docker-compose 

# We can make changes to files in the volume, from the host side,
# and have them reflected immediately in the app.
$ nano app.py   # do some changes
$ docker volume inspect counterapp_counter-vol | grep Mount
$ sudo cp app.py \
  /var/lib/docker/volumes/counterapp_counter-vol/_data/app.py
# Our changes should be reflected 

$ docker-compose --help

Update the application Note that on Get started with Docker Compose it mounts the current directory to /code inside the container. So after we modify app.py, we don't need to copy it to the container.

Docker-Compose persistent data MySQL

https://stackoverflow.com/questions/39175194/docker-compose-persistent-data-mysql

Connect to Docker daemon over ssh using docker-compose

#DockerTips: Connect to Docker daemon over ssh using docker-compose

Docker Swarm

Moby Project

What is Docker's Moby Project?

Applications

Some popular Linux applications such as: owncloud, lamp, openstack, node.js, roundcube, DLNA/Server, madsonic, webproxy/webfilter, DHCP/DNS-Server like dnsmasq, cloudprint, ...

Dockerizing Compiled Software

https://ram.tianon.xyz/post/2017/12/26/dockerize-compiled-software.html

Data Science

Data Science

Illumina BaseSpace

Bioinformatics analyses

Cloud provider

Docker can be deployed on any cloud provider like Digital Ocean, Amazon Web Services (AWS), Microsoft Azure, Google Cloud Computing.

ownCloud

After I use docker run --rm --name myowncloud -p 81:80 owncloud:8.1, I find I can only access it through http://localhost:81. If I try to access it through another computer by http://xxx.xxx.xx.1:81, the URL is changed back to http://localhost:81 so it failed to load.

nextcloud

I feel nextcloud is a little modern than owncloud though their interface are very similar. I got the same problem (URL is changed from IP to localhost) when I try to access its web from another computer. Using the IP instead of localhost to finish the initial setup can fix the error.

Note:

  • The first we open the webpage, we need to create the admin username/password.
  • Nextcloud uses owncloud mobile application.
  • Nextcloud includes a small mp4 file. I can use browser to play mp4. Photos files can be opened in browsers too (including slideshow).

Nextcloud Apps

For mp3, no apps are shipped with nextcloud.

  • Audio Player. When I play mp3 files, the web interface does not show a progress bar:( 
    $ docker run --rm --name mynextcloud -v ~/Downloads/nextcloudapps:/var/www/html/custom_apps -p 81:80 nextcloud
# above will create an empty folder ~/Downloads/nextcloudapps (www-data:root)
$ tar xzvf Downloads/audioplayer-2.3.1.tar.gz -C Downloads/
$ docker cp Downloads/audioplayer mynextcloud:/var/www/html/custom_apps
# Go to nextcloud and '+Apps', a new app 'Audio Player' should have shown up.
  • Music. GOOD. The app is from the owncloud team though for some reason the app is not included in owncloud/nextcloud. Yes, a progress bar is shown when I played a song downloaded from Youtube:) 
    # No need to create a volume
$ docker run -d --name mycloud -p 81:80 nextcloud

# Open another terminal
# Don't download the zip file from github. Get the zip file from the app store
$ tar xzvf ~/Downloads/music_0.7.0_nc-signed.tar.gz -C ~/Downloads/
$ docker cp Downloads/music mycloud:/var/www/html/custom_apps
$ docker exec mycloud chmod -R 0750 /var/www/html/custom_apps
$ docker exec mycloud chown -R www-data:www-data /var/www/html/custom_apps
    Now open the website. Click the gear icon (Settings) and click "Apps". Find out the 'music' app from the bottom of the list and click the 'Enable' button. A new Music icon will be created next to the Gallery icon at the top. Note: if we want to play music repeatedly, I should not play the music directly from Files. I should click the Music icon at the top and let it start to scan musics. Then once I can play a music from there, it will show two new icons; one is shuffle and the other is loop. To have a persistent data, we can 'run' using 
    $ mkdir -p ~/nextcloud/{html,apps,config,data}
$ docker run -d \
    --name mycloud \
    -p 81:80 \
    -v ~/nextcloud/html:/var/www/html \
    -v ~/nextcloud/apps:/var/www/html/custom_apps \
    -v ~/nextcloud/config:/var/www/html/config \
    -v ~/nextcloud/data:/var/www/html/data \
    nextcloud
    If we like to delete the contain and its unamed volume, run 
    $ docker stop mycloud; docker rm -v mycloud
    If we run the container again mounting all volumes, all apps,files,configuration,admin's username & password are kept.
  • PDF viewer. PDF files will be downloaded instead of opening on the browser. This can be easily fixed. Click '+Apps' from the personal settings (the response is slow when I tested using Docker). Search for PDF viewer. Click the 'Enable' button (it will be changed to 'Disable' after that)
  • draw.io. After enable it, we can click the "+" sign and choose "Diagram" to create a new diagram (*.xml).
  • Radio. After enable it, a new "Radio" icon will be shown at the top. Only 20 stations are available from the 'Top'. No response after I clicked "Categories".

Use with R (r-base) & RStudio IDE

  • r-base (Official image, R version is tagged)
    • Github repository of the Official Docker Images.
    • An Introduction to Docker for R Users: how to write your own <Dockerfile>, install packages, run a script and get results.
    • Extensions from r-base. For example, r-spatial-base. It also mentions ropensci container is built upon rocker/rstudio.
    • The r-base image does not have pdflatex, git. Need to manually install them.
    • Not sure if the Docker Official Image is the same as the one provided by Rocker Project.
    • NOTE: Plotting just did not work insider a container. See Plotting R objects within docker container. Search "Share your X11 display to the docker container" or the Wiki from rocker.
docker pull r-base:3.5.3
docker run --net=host -it --rm r-base:3.5.3               # default is root "/"
docker run --net=host -v ~/Downloads:/src -it --rm r-base # /src does not exist
docker run --net=host -v ~/Downloads:/home/docker -it --rm r-base # /home/docker exists and is empty

Dockerfile

Create a new directory and a new file 'Dockerfile' with the content. 

FROM debian:testing
MAINTAINER Dirk Eddelbuettel [email protected]
## Remain current
RUN apt-get update -qq
RUN apt-get dist-upgrade -y
RUN apt-get install -y --no-install-recommends r-base r-base-dev r-recommended littler
RUN ln -s /usr/share/doc/littler/examples/install.r /usr/local/bin/install.r

NOTE

  1. I ran into errors when I use the above (short) Dockerfile. But the Dockerfile from rocker (leave out the last line of launching R) works well. The R packages built in the image include 'docopt', 'magrittr', 'stringi', and 'stringr'.
  2. Install R package is possible when we launch a container. But we are not able to save the packages?? The rocker wiki also mentions something about installing packages.
  3. See also How to save data in wiki.

docker run

Note that if we are using the Dockerfile above to create an image, we will be dropped to the Linux shell. If we are pulling the rocker/r-base image from Docker Hub, we will be in R console directly. See the last line of Dockerfile on github website for rocker.

Then run the following to do some exercise (We could possibly replace 21b6a9e8b9e8 with your image ID or use rocker/r-base). For simplicity, we can try the colortools package first which does not depend on other packages and there is no need to compile the package. 

sudo docker build -t debian:testing-add-r . # create an image based on the above Dockerfile
wget http://cran.r-project.org/src/contrib/sanitizers_0.1.0.tar.gz
sudo docker run -v `pwd`:/mytmp -t 21b6a9e8b9e8 \
     R CMD check --no-manual --no-build-vignettes /mytmp/sanitizers_0.1.0.tar.gz
sudo docker run -v `pwd`:/mytmp -t 21b6a9e8b9e8 \
     Rdevel CMD check --no-manual --no-build-vignettes /mytmp/sanitizers_0.1.0.tar.gz

sudo docker search eddelbuettel
sudo docker pull eddelbuettel/docker-ubuntu-r   # default tag is 'latest'; actually older than the other tags
sudo docker images eddelbuettel/docker-ubuntu-r # see the tag column
sudo docker pull eddelbuettel/docker-ubuntu-r:add-r # the tag name can only be obtained from hub.docker.com
sudo docker images eddelbuettel/docker-ubuntu-r # see the tag column
sudo docker pull eddelbuettel/docker-ubuntu-r:add-r-devel
sudo docker images eddelbuettel/docker-ubuntu-r # see the tag column
sudo docker run -v `pwd`:/mytmp -t 54d865dbd2c9 R CMD check --no-manual --no-build-vignettes /mytmp/sanitizers_0.1.0.tar.gz

sudo docker run -t -i eddelbuettel/docker-ubuntu-r /bin/bash
$ sudo docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
debian                         testing-add-r       21b6a9e8b9e8        28 minutes ago      572.2 MB
ubuntu                         14.04               ed5a78b7b42b        4 days ago          188.3 MB
ubuntu                         latest              ed5a78b7b42b        4 days ago          188.3 MB
debian                         testing             88ba2870bfbe        7 weeks ago         154.7 MB
eddelbuettel/docker-ubuntu-r   add-r-devel         c998a74a1fb4        11 weeks ago        460.4 MB
eddelbuettel/docker-ubuntu-r   add-r               54d865dbd2c9        11 weeks ago        460.4 MB
eddelbuettel/docker-ubuntu-r   latest              a7cd5ddeb98e        5 months ago        515.4 MB

sudo docker logs xxxxx                # view the log
sudo docker restart xxxxx
sudo docker exec -it xxxx /bin/bash   # view any changes in R library
sudo docker stop xxxxx
sudo docker docker rm xxxxx

This is another example of using 'docker run' accompanying MotifBreakR package.

Testing R packages

Docker for R Package Development

http://www.jimhester.com/2017/10/13/docker/

A DevOps Perspective

Reproducible

I am not successful running the example in vignette. 

> render_docker(input)
Sending build context to Docker daemon  6.144kB
Step 1/8 : FROM rocker/r-base:latest
 ---> 4e232e9b4097
Step 2/8 : MAINTAINER Nan Xiao <[email protected]>
 ---> Running in f75404db590a
Removing intermediate container f75404db590a
 ---> ceae0a54648c
Step 3/8 : RUN  rm -f /var/lib/dpkg/available   && rm -rf  /var/cache/apt/*   && apt-get update -qq   && apt-get install -y 
   --no-install-recommends     ca-certificates     libssl-dev     libcurl4-openssl-dev     libxml2-dev     git
 ---> Running in 8a4324cc7c53
W: Failed to fetch http://deb.debian.org/debian/dists/testing/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch http://http.debian.net/debian/dists/sid/InRelease  Temporary failure resolving 'http.debian.net'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
E: The value 'testing' is invalid for APT::Default-Release as such a release is not available in the sources

Debugging R memory problem

Docker image for debugging R memory problems (valgrind)

More examples

Building a Repository of Alpine-based Docker Images for R

RStudio in Docker – now share your R code effortlessly!

https://harini.blog/2019/05/25/rstudio-and-rshiny-in-docker/

It is interesting the Dockerfile uses littler's install2.r to install R packages. See http://dirk.eddelbuettel.com/code/littler.examples.html.

Note that the tag name should be lower case; awesomer instead awesomeR.

Also got an error when trying to build the image: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease Temporary failure resolving 'deb.debian.org' .

Try it again at home. The apt update part is OK but I still got a new error: Error: installation of package ‘gifski’ had non-zero exit status

Bioconductor

The Dockerfiler for different images can be found on github.com/Bioconductor/bioc_docker.

Today is 2/5/2015. 

$ sudo docker pull bioconductor/release_base
$ sudo docker images 
REPOSITORY                     TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
bioconductor/release_base      latest              7eefbf783d9b        8 days ago          2.602 GB
bioconductor/release_base      20150130            7eefbf783d9b        8 days ago          2.602 GB
bioconductor/release_base      20150129            7eefbf783d9b        8 days ago          2.602 GB
bioconductor/release_base      20150201            7eefbf783d9b        8 days ago          2.602 GB
bioconductor/release_base      20150202            7eefbf783d9b        8 days ago          2.602 GB
bioconductor/release_base      20150203            7eefbf783d9b        8 days ago          2.602 GB
bioconductor/release_base      20150204            7eefbf783d9b        8 days ago          2.602 GB
bioconductor/release_base      20150205            7eefbf783d9b        8 days ago          2.602 GB
bioconductor/release_base      20150128            7eefbf783d9b        8 days ago          2.602 GB
bioconductor/release_base      20150131            7eefbf783d9b        8 days ago          2.602 GB

Note that even we specify the tag 'latest' we still get the same list of images as above 

$ sudo docker pull bioconductor/release_base:latest

How I use Bioconductor with Docker, Part 2: More memory, faster Bioconductor with Docker

(2019-10-15)

BiocImageBuilder

Reproducible Bioconductor workflows using browser-based interactive notebooks and containers

Bioc Conference

  • Bioc2019 conference. Workshopt material. Before the meeting (6/20/2019) there are 13 downloads and it bumps to 32 downloads after the meeting for the docker image.
    • Download a tarball containing R packages (binary). 605 packages are included. It bundles 605 packages based on R 3.6.0 and Bioconductor 3.10 (BiocManager 1.30.4).
    • Run RStudio container. All R packages downloaded in the last step are mounted. (user=rstudio, ps=bioc). That is, built-in is at /usr/local/lib/R/library, custom is at /usr/local/lib/R/site-library. These two locations are what .libPaths() gives.

Docker for Shiny Server

# specify --name so we don't need to use container id to access it later
$ docker run -d --name rshiny -p 81:3838 rocker/shiny 

$ docker ps -l

# Run in container's bash
# https://stackoverflow.com/questions/30172605/how-to-get-into-a-docker-container
# non-interactive
$ docker exec -it rshiny ip addr  
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
22: eth0@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 scope global eth0
       valid_lft forever preferred_lft forever

# interactive shell
$ docker exec -it rshiny bash 
root@5b78988d5619:/# who
root@5b78988d5619:/# which R
/usr/bin/R
root@5b78988d5619:/# ls /home
docker	shiny
root@5b78988d5619:/# ls /srv/shiny-server
01_hello  02_text  03_reactivity  04_mpg  05_sliders  06_tabsets  07_widgets  08_html  09_upload  10_download  11_timer  index.html  sample-apps
root@5b78988d5619:/# R -e 'package.Description("glmnet")'
root@5b78988d5619:/# exit

$ docker stop rshiny
$ docker rm rshiny   # remove the container

It is also useful to use the -v option in docker run so the local directory (eg /srv/shinyapps/) can be mapped to the docker volume (/srv/shiny-server). 

# Put your shiny app in host's /srv/shinyapps/appdir,
$ docker run -d -p 81:3838 \
    -v /srv/shinyapps/:/srv/shiny-server/ \
    -v /srv/shinylog/:/var/log/shiny-server/ \
    rocker/shiny

And check running status 

$ docker ps -a

Check opened ports

See What network ports are open: lsof or netstat. 

$ netstat -ant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN     
...
tcp6       0      0 :::80                   :::*                    LISTEN     
tcp6       0      0 :::81                   :::*                    LISTEN     
tcp6       0      0 :::22                   :::*                    LISTEN     
tcp6       0      0 ::1:25                  :::*                    LISTEN

Nginx

docker run --name docker-nginx -p 80:80 nginx

httpd/apache

https://hub.docker.com/_/httpd/

Tomcat

https://hub.docker.com/_/tomcat/

MySQL

  • https://hub.docker.com/r/mysql/mysql-server/
  • Deploying MySQL on Linux with Docker
  • Note it seems impossible to use the 'grant' command interactively within the MySQL docker container.
  • https://dzone.com/articles/using-docker-to-create-a-mysql-server 
    docker run --detach --name test-mysql --env="MYSQL_ROOT_PASSWORD=mypassword" mysql
docker inspect test-mysql | grep IPAddress  # 172.17.0.2

# https://dev.mysql.com/doc/index-other.html
wget -O - http://downloads.mysql.com/docs/world.sql.gz | gunzip -c > world.sql

mysql -uroot -pmypassword -h 172.17.0.2 -e 'CREATE DATABASE world'
cat world.sql | mysql -uroot -pmypassword -h 172.17.0.2 world
# zcat world.sql.gz | mysql -uroot -pmypassword -h 172.17.0.2 world
mysql -uroot -pmypassword -h 172.17.0.2 -e 'USE world; SHOW TABLES'

mysql -uroot -pmypassword -h 172.17.0.2 -e 'DROP DATABASE world'
    # Let's see if we can access the database in R
system("sudo apt install libmysqlclient-dev")
library(RMySQL)
con <- dbConnect(MySQL(), "world", user="root", password="mypassword", host="172.17.0.2")
dbListTables(con)
head(dbReadTable(con, "city"))
head(dbReadTable(con, "country"))
head(dbReadTable(con, "countrylanguage"))

dbWriteTable(con, 'iris', iris)
# Error in .local(conn, statement, ...) :
#   could not run statement: The used command is not allowed with this MySQL version
dbDisconnect(con)
  • How To Create And Test MySQL Database Server in Docker On Ubuntu Linux (video). Use Dockerfile.
  • MySQL Docker Containers: Understanding the basics, https://youtu.be/qJQPckRYNis?t=16m29s 
    # Fire up
docker run --detach --name=test-mysql --env="MYSQL_ROOT_PASSWORD=mypassword" mysql
docker ps -a

# Connecting to the Container
docker inspect test-mysql | grep IPAddress
# sudo apt install mysql-client
# There are two methods to connect to the container
# method 1. https://hub.docker.com/r/mysql/mysql-server/
# docker exec -it test-mysql mysql -uroot -p
# method 2
mysql -uroot -pmypassword -h 172.17.0.20 -P 3306
docker run --detach --name test-wordpress --link test-mysql:mysql wordpress
docker exec -it test-wordpress bash
$ cat /etc/hosts

# Expose the MySQL container to the outside world
# Access the MySQL container directly from the machine’s port 6603
docker run --detach --name=test-mysql --env="MYSQL_ROOT_PASSWORD=mypassword" --publish 6603:3306 mysql
mysql -u root -pmypassword -h 127.0.0.1 -P 6603

# Configuration management
mkdir -p /root/container/test-mysql/conf.d
nano /root/container/test-mysql/conf.d/my-custom.cnf
docker run \
    --detach \
    --name=test-mysql \
    --env="MYSQL_ROOT_PASSWORD=mypassword" \
    --publish 6603:3306 \
    --volume=/root/docker/test-mysql/conf.d:/etc/mysql/conf.d \
    mysql
mysql -uroot -pmypassword -h127.0.0.1 -P6603 -e 'show global variables like "max_connections"';
docker run \
    --detach \
    --name=test-mysql \
    --env="MYSQL_ROOT_PASSWORD=mypassword" \
    --publish 6603:3306 \
    mysql \
    --max-connections=200 \
    --character-set-server=utf8mb4 \
    --collation-server=utf8mb4_unicode_ci

# Data Storage
mkdir -p /storage/docker/mysql-datadir
docker run \
    --detach \
    --name=test-mysql \
    --env="MYSQL_ROOT_PASSWORD=mypassword" \
    --publish 6603:3306 \
    --volume=/root/docker/test-mysql/conf.d:/etc/mysql/conf.d \
    --volume=/storage/docker/mysql-datadir:/var/lib/mysql \
    mysql
docker inspect test-mysql
ls -al /storage/docker/mysql-datadir/

# If a MySQL container runs on top of an existing MySQL datadir, 
# the $MYSQL_ROOT_PASSWORD variable should be omitted from the “run” command line;
docker stop test-mysql
docker rm -f test-mysql
docker run -d --name=another-new-mysql -p 6605:3306 -v /storage/docker/mysql-datadir:/var/lib/mysql mysql
docker logs another-new-mysql
  • Use Docker to run a typical MySQL server from Code Review Videos 
    docker volume create crv_mysql

docker run \
    -e MYSQL_ROOT_PASSWORD=my-secret-pw \
    -e MYSQL_DATABASE=devdb \
    -e MYSQL_USER=dbuser \
    -e MYSQL_PASSWORD=dbpassword \
    --mount type=volume,src=crv_mysql,dst=/var/lib/mysql \
    -p 3306:3306 \
    -d \
    mysql:latest

# Alternatively we can put username/password in a text file for security
cat > .env <<EOF
MYSQL_ROOT_PASSWORD=my-secret-pw
MYSQL_DATABASE=devdb
MYSQL_USER=dbuser
MYSQL_PASSWORD=dbpassword
EOF
docker run \
    --env-file .env \
    --mount type=volume,src=crv_mysql,dst=/var/lib/mysql \
    -p 3306:3306 \
    -d \
    mysql:latest

Backup and restore

https://gist.github.com/spalladino/6d981f7b33f6e0afe6bb 

# Backup
docker exec CONTAINER /usr/bin/mysqldump -u root --password=root DATABASE > backup.sql

# Restore
cat backup.sql | docker exec -i CONTAINER /usr/bin/mysql -u root --password=root DATABASE

Blog

Deploying Ghost Blog with MySQL and Traefik with Docker

Laravel

Dockerizing Laravel with Nginx MySQL and Docker Compose on Ubuntu 18.04 LTS

phpMyAdmin

docker run --name mysqlphp -e MYSQL_ROOT_PASSWORD=0000 -d mysql
docker run --name myadmin -d --link mysqlphp:db -p 8080:80 phpmyadmin/phpmyadmin

Now go to your browser , tap http://localhost:8080 and login with root/0000.

ERROR mysqli_real_connect(): The server requested authentication method unknown to the client [caching_sha2_password]

The solution can be found here. After the change, I can log in phpMyAdmin. 

docker exec -it mysqlphp bash

mysql -u root -p0000

ALTER USER root IDENTIFIED WITH mysql_native_password BY '0000';

exit
exit

Mail server

Setup Your Own Email Server with Docker

Mediawiki

$ ls -l ~/wiki_root
total 24
drwxr-xr-x 3 root     root     4096 Jul 10 10:09 data
drwxrwxr-x 3 brb      brb      4096 Jul 10 09:23 db
drwxrwxr-x 2 www-data www-data 4096 Jul 10 09:23 extensions
drwxrwxr-x 2 www-data www-data 4096 Jul 10 09:23 images
-rw-rw-r-- 1 www-data www-data 4152 Jul 10 11:12 LocalSettings.php

brb@galliumos:~$ tree mysql
mysql
├── conf.d
│   └── mysql.cnf
├── data
│   ├── auto.cnf
│   ├── ca-key.pem
│   ├── ca.pem
│   ├── client-cert.pem
│   ├── client-key.pem
│   ├── ib_buffer_pool
│   ├── ibdata1
│   ├── ib_logfile0
│   ├── ib_logfile1
│   ├── ibtmp1
│   ├── mediawiki [error opening dir]
│   ├── mysql [error opening dir]
│   ├── performance_schema [error opening dir]
│   ├── private_key.pem
│   ├── public_key.pem
│   ├── server-cert.pem
│   ├── server-key.pem
│   ├── sys [error opening dir]
│   └── wikidb [error opening dir]
├── dumps
└── initdb.d

9 directories, 15 files

brb@galliumos:~$ tree -L 2 mediawiki/
mediawiki/
├── config
│   └── LocalSettings.php
├── extensions
│   ├── Cite
│   ├── ConfirmEdit
│   ├── Gadgets
│   ├── ImageMap
│   ├── InputBox
│   ├── Interwiki
│   ├── LocalisationUpdate
│   ├── Nuke
│   ├── ParserFunctions
│   ├── PdfHandler
│   ├── Poem
│   ├── README
│   ├── Renameuser
│   ├── SpamBlacklist
│   ├── SyntaxHighlight_GeSHi
│   ├── TitleBlacklist
│   └── WikiEditor
└── images
    └── README

19 directories, 3 files

BookStack

BookStack is a simple, self-hosted, easy-to-use platform for organising and storing information.

https://github.com/solidnerd/docker-bookstack

Media server

Airsonic media server

https://airsonic.github.io/download/

My comment:

  • Added mp3/mp4 files are not recognized.
  • Cannot play by web browsers
  • Special clients are needed:(

Plex

Official Docker container for Plex Media Server

My comment:

  • After starting, web browser cannot connect to it

DockSTARTer: get started with home server apps running in Docker

DNS server

Private DNS server container

How to Deploy a Dynamic DNS Server with Docker on Debian 10

Pi-hole

KeeWeb: password manager compatible with KeePass

https://github.com/keeweb/keeweb

ONLYOFFICE

Gitlab

NAS & Docker

http://www.ithome.com.tw/tech/96500

Some examples

  • GetLab專案管理工具
  • Jenkins開發版本管理工具

Draw.io for diagrams and more

https://github.com/jgraph/drawio/tree/master/etc/docker

Run a benchmark program on ARM devices

Set up a Mastodon Instance on Linux

Your Own Social Network: How to Set Up a Mastodon Instance on Linux

Bioinformatics tools (Tophat, BWA, et al)

https://github.com/dceoy/docker-bio

GIVE - Genomic Interactive Visualization Engine

https://github.com/Zhong-Lab-UCSD/Genomic-Interactive-Visualization-Engine/blob/master/tutorials/GIVE-Docker.md

Elastic Search

Orchestrator

Kubernete

Kubernete vs Docker swarm

Other containers

Singularity and HPC systems

  • Singularity enables users to have full control of their environment; Singularity containers let users run applications in a Linux environment of their choosing. No 'sudo' is needed in general unless you want to build a container from a recipe.
  • Containers are more like an executable file for you to use
  • Containers are stored under the current location. It does not have a centre location (like /etc/default/docker if we use docker) to store images.
  • Can convert Docker containers to Singularity and run containers directly from Docker Hub
  • These bind points cannot be created unless the path already exists within the container. To ensure access to these storage spaces and remedy bind point errors, create these directories in the %post section of your Bootstrap file.
  • Singularity Hub

Ref:

Comparison of docker and singularity commands:

docker singularity
$ docker pull ubuntu:latest
$ docker pull broadinstitute/gatk3:3.8-0		 $ singularity pull docker://ubuntu:latest
$ singularity pull docker://broadinstitute/gatk3:3.8-0
$ docker shell (not exist) $ singularity shell docker://broadinstitute/gatk3-3.8-0
$ singularity shell gatk3-3.8-0.img
> ls # the default location depends on the host system

> ls /usr # this is from the container

$ singularity shell --bind ~/Downloads:/mnt XXX.img
$ singularity shell docker://ubuntu:latest
# container is ephemeral

$ docker run --name test -it ubuntu date

# The next example is similar to 'singularity exec'
$ docker run --rm -i -t \
-v $(pwd):/usr/my_data \
broadinstitute/gatk3:3.8-0 \
bash /usr/my_data/myscript.sh		 $ singularity run gatk3-3.8-0.img date
$ docker run --name ubuntu_bash --rm -i -t ubuntu bash
$ docker exec -d ubuntu_bash touch /tmp/execWorks		 # Most useful
$ singularity exec gatk3-3.8-0.img java -version
$ singularity exec xxx.img cat /etc/*release
$ singularity exec docker://rocker/tidyverse:latest R
$ singularity exec docker://rocker/tidyverse:latest Rscript myScript.R

Cache

When we run singularity exec docker://rocker/tidyverse:latest R, it will save something in the cache in our system.

It seems to be OK after I manually delete the directory $HOME/.singularity (tested in Biowulf).

RStudio

$ singularity exec docker://rocker/tidyverse:latest R
$ singularity exec docker://rocker/tidyverse:latest Rscript myScript.R

Shifter

Conda

Bioconda

Using docker to install conda (https://conda.io/docs/user-guide/tutorials/index.html) 

$ docker run -t -i --name test --net=host ubuntu bash
# apt-get update
# apt-get install -y wget bzip2 python
# wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
# wget https://repo.continuum.io/archive/Anaconda2-5.1.0-Linux-x86_64.sh
# bash Miniconda2-latest-Linux-x86_64.sh
# bash Anaconda2-5.1.0-Linux-x86_64.sh
# exit

$ docker start test 
$ docker exec -i -t test bash
# conda list  # WORKS!
# conda config --add channels r
# conda config --add channels defaults
# conda config --add channels conda-forge
# conda config --add channels bioconda
# conda install bwa  (Segmentation fault. Core dumped)
# which bwa
/root/anaconda2/bin/bwa
# conda install r   (Only get 3.4.2 but the latest is 3.4.3.)
# conda install bowtie
# bowtie --version
# conda install gatk (https://bioconda.github.io/recipes/gatk/README.html)
   (Due to license restrictions, this recipe cannot distribute and install GATK directly)
   (R is downgraded to 3.2.2:( )
   (Segmentation fault. Core dumped)
# exit
# docker stop test
# docker rm test

Get miniconda image instead of using a Ubuntu image 

$ docker pull continuumio/miniconda
$ docker run -i -t continuumio/miniconda /bin/bash
# conda install r   (get 3.4.2)
# conda config --add channels bioconda
# conda install bwa  (OK, no error)
# conda install gatk  (R was downgraded to 3.2.2, install openjdk 8.0.121)
# which gatk
/opt/conda/bin/gatk
# gatk -h
GATK jar file not found. Have you run "gatk-register"?

Issues:

  • R version is not up to date
  • So the problem is installing GATK requires an installation of R and the current R was affected.

CoreOS

Redhat has acquired CoreOS 1/31/2018.

Installation

We first boot a liveCD from any OS (CentOS works but Ubuntu 16.04 gave errors). In Virtualbox, we choose 'Red Hat' if we use CentOS.

Once the VM is created. We go to the settings. Create a bridged network or host-only network first (even we can get files from the host without creating a host-only network). Storage: choose CentOS-7.

  1. Get the install script from Github and create <coreos_install.sh> and chmod +x
  2. create <cloud-config.yaml> file which will include ssh_authorized_keys generated from another machine. It should also contain a new token for the cluster from https://discovery.etcd.io/new.
  3. ls -l /dev/sd*
  4. run sudo ./coreos_install.sh -d /dev/sda -C stable -c cloud-config.yaml. It will download the latest stable CoreOS, install to the HD
  5. Don't leave the VM or it will freeze. Issue sudo shutdown -h now once we see the word 'Success' at the last line of the output.
  6. Remove CentOS from the VM storage. Boot the coreOS VM.

The new screen shows corebm1 login with an IP. Go back to another machine and type ssh -i /tmp/CoreOSBM_rsa [email protected]. Inside CoreOS, we can type docker images.

The 'cloud-config.yaml file has to follow the format in https://coreos.com/os/docs/latest/cloud-config.html. Use the online validator https://coreos.com/validate/ to correct. At first I use the file from the youtube video. There is no error coming out when I ran the installation script. But I cannot connect to coreOS. The cloud-config.yaml file I use is (pay attention to '-', double quotes and indent characters) 

#cloud-config
#
# set hostname
hostname: CoreBM1

# Set ssh key
ssh_authorized_keys:
  - "ssh-rsa AAAAB3 ..... brb@T3600"

coreos:
  etcd:
    discovery: "https://discovery.etcd.io/d3e95 .... "
# sudo ./installos -d /dev/sda -C stable -c cloud-config.yaml

CoreOS exploration

brb@T3600 /tmp $ ssh -i /tmp/id_rsa [email protected]
Enter passphrase for key '/tmp/id_rsa':
CoreOS stable (1010.6.0)
core@CoreBM1 ~ $
core@CoreBM1 ~ $ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
core@CoreBM1 ~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.4G     0  1.4G   0% /dev
tmpfs           1.4G     0  1.4G   0% /dev/shm
tmpfs           1.4G  340K  1.4G   1% /run
tmpfs           1.4G     0  1.4G   0% /sys/fs/cgroup
/dev/sda9        18G   23M   17G   1% /
/dev/sda3       985M  589M  345M  64% /usr
tmpfs           1.4G     0  1.4G   0% /media
/dev/sda1       128M   37M   92M  29% /boot
tmpfs           1.4G     0  1.4G   0% /tmp
/dev/sda6       108M   52K   99M   1% /usr/share/oem
core@CoreBM1 ~ $ free -m
             total       used       free     shared    buffers     cached
Mem:          2713        187       2525          0          9        109
-/+ buffers/cache:         68       2644
Swap:            0          0          0
core@CoreBM1 ~ $ lsb_release -a
-bash: lsb_release: command not found
core@CoreBM1 ~ $ docker pull ubuntu
Using default tag: latest
latest: Pulling from library/ubuntu
f069f1d21059: Pull complete
ecbeec5633cf: Pull complete
ea6f18256d63: Pull complete
54bde7b02897: Pull complete
Digest: sha256:bbfd93a02a8487edb60f20316ebc966ddc7aa123c2e609185450b96971020097
Status: Downloaded newer image for ubuntu:latest
core@CoreBM1 ~ $ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu              latest              0f192147631d        7 days ago          132.7 MB
core@CoreBM1 ~ $

CoreOS cluster discovery

https://coreos.com/os/docs/latest/cluster-discovery.html

etcd

fleet

Resource

Internet

Reproducible Research through Containerisation: Docker and Singularity from rainsworth.

Books

  • The Docker Book: Containerization is the new virtualization
  • Orchestrating Docker
  • Docker Hands on: Deploy, Administer Docker Platform
  • Learn Docker - Fundamentals of Docker 18.x by Gabriel N. Schenker. Release Date: April 2018. https://github.com/fundamentalsofdocker/labs
  • Docker Deep Dive by by Nigel Poulton. 2017
  • Master Docker (2nd, 2017) by Russ McKendrick, Scott Gallagher. Source code on github.

Blogs

Tips/trouble shooting

Play with Docker (PWD)

Retrieved from "https://wiki.taichimd.us/index.php?title=Docker&oldid=22602"