Docker: Difference between revisions

From 太極
Jump to navigation Jump to search
(283 intermediate revisions by the same user not shown)
Line 17: Line 17:
* Differences: https://youtu.be/BFKNwr_wllc
* Differences: https://youtu.be/BFKNwr_wllc
* Work together: https://youtu.be/BFKNwr_wllc
* Work together: https://youtu.be/BFKNwr_wllc
=== KubeVirt ===
* [https://kubevirt.io/ KubeVirt] brings VMs into containerized workflows by converting the VMs into containers, where they run with the benefits of containerized applications. [https://opensource.com/article/20/12/containers-101 6 container concepts you need to understand]


== OS containers vs application containers ==
== OS containers vs application containers ==
Line 32: Line 35:


== Ubuntu x86 and Mint ==
== Ubuntu x86 and Mint ==
* https://docs.docker.com/install/linux/docker-ce/ubuntu/
<ul>
* [https://www.howtoforge.com/tutorial/how-to-install-kubernetes-on-ubuntu/ How to Install and Configure Kubernetes and Docker on Ubuntu 18.04 LTS]
<li>https://docs.docker.com/install/linux/docker-ce/ubuntu/
 
{{Pre}}
<syntaxhighlight lang='bash'>
# Uninstall old versions
# Uninstall old versions
$ sudo apt-get remove docker docker-engine docker.io
$ sudo apt-get remove docker docker-engine docker.io
Line 42: Line 44:
$ sudo apt-get update
$ sudo apt-get update
$ sudo apt-get install \
$ sudo apt-get install \
    apt-transport-https \
     ca-certificates \
     ca-certificates \
     curl \
     curl \
     software-properties-common
     gnupg \
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
    lsb-release
# Note that for Linux Mint we need to change $(lsb_release -cs) to a proper name
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
# For example, xenial for Mint 18.x and bionic for Mint 19.x
 
$ sudo add-apt-repository \
$ echo \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  stable"
 
$ sudo apt-get update
$ sudo apt-get update
$ sudo apt-get install docker-ce
$ sudo apt-get install docker-ce docker-ce-cli containerd.io


# Verify
# Verify
$ sudo docker run hello-world  
$ sudo docker run hello-world  
</syntaxhighlight>
</pre>
Install docker-compose https://docs.docker.com/compose/install/
<pre>
$ sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ docker-compose --version
</pre>
</li>
<li>[https://www.howtoforge.com/tutorial/how-to-install-kubernetes-on-ubuntu/ How to Install and Configure Kubernetes and Docker on Ubuntu 18.04 LTS] </li>
</ul>
 
=== One-line script ===
https://github.com/docker/docker-install, https://docs.docker.com/engine/install/ubuntu/, https://twitter.com/portainerio/status/1650171336864550912
 
Note that 1) the one-liner is a huge security issue. 2) but how will you add the current user to docker group and then logout and log back in.
{{Pre}}
$ curl -fsSL https://get.docker.com | bash
# Executing docker install script, commit: e5543d473431b782227f8908005543bb4389b8de
+ sudo -E sh -c 'apt-get update -qq >/dev/null'
[sudo] password for brb:
+ sudo -E sh -c 'DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null'
+ sudo -E sh -c 'install -m 0755 -d /etc/apt/keyrings'
+ sudo -E sh -c 'curl -fsSL "https://download.docker.com/linux/debian/gpg" | gpg --dearmor --yes -o /etc/apt/keyrings/docker.gpg'
gpg: WARNING: unsafe ownership on homedir '/home/brb/.gnupg'
+ sudo -E sh -c 'chmod a+r /etc/apt/keyrings/docker.gpg'
+ sudo -E sh -c 'echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian bullseye stable" > /etc/apt/sources.list.d/docker.list'
+ sudo -E sh -c 'apt-get update -qq >/dev/null'
+ sudo -E sh -c 'DEBIAN_FRONTEND=noninteractive apt-get install -y -qq docker-ce docker-ce-cli containerd.io docker-compose-plugin docker-ce-rootless-extras docker-buildx-plugin >/dev/null'
+ sudo -E sh -c 'docker version'
Client: Docker Engine - Community
Version:          24.0.7
API version:      1.43
Go version:        go1.20.10
Git commit:        afdd53b
Built:            Thu Oct 26 09:08:17 2023
OS/Arch:          linux/amd64
Context:          default
 
Server: Docker Engine - Community
Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:      go1.20.10
  Git commit:      311b9ff
  Built:            Thu Oct 26 09:08:17 2023
  OS/Arch:          linux/amd64
  Experimental:    false
containerd:
  Version:          1.6.26
  GitCommit:        3dd1e886e55dd695541fdcd67420c2888645a495
runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 
---------------
 
To run Docker as a non-privileged user, consider setting up the
Docker daemon in rootless mode for your user:
 
    dockerd-rootless-setuptool.sh install
 
Visit https://docs.docker.com/go/rootless/ to learn about rootless mode.
 
 
To run the Docker daemon as a fully privileged service, but granting non-root
users access, refer to https://docs.docker.com/go/daemon-access/
 
WARNING: Access to the remote API on a privileged Docker daemon is equivalent
        to root access on the host. Refer to the 'Docker daemon attack surface'
        documentation for details: https://docs.docker.com/go/attack-surface/
 
--------------
$ sudo groupadd docker
$ sudo usermod -aG docker $USER
$ newgrp docker
 
$ docker run hello-world
</pre>
This can install docker but you still need "sudo" to run it. See [https://docs.docker.com/engine/install/linux-postinstall/ Linux post-installation steps for Docker Engine]: 1) Manage Docker as a non-root user, and 2) Configure Docker to start on boot with systemd.
 
=== Docker Desktop ===
* https://docs.docker.com/desktop/linux/install/
* [https://www.docker.com/blog/the-magic-of-docker-desktop-is-now-available-on-linux/ The Magic of Docker Desktop is Now Available on Linux]
* [https://ostechnix.com/docker-desktop-for-linux/ A Beginners Manual To Docker Desktop For Linux]


== Without sudo, Post-installation ==
== Without sudo, Post-installation ==
Line 73: Line 160:


# Log out and log in
# Log out and log in
</pre>
After running this command, you need to log out and log back in for the changes to take effect. This is because group membership is determined at login time. When you log in, the system reads the group membership information and assigns the appropriate permissions to your user account.
== Upgrade Docker Desktop ==
It seems it does not affect running containers (e.g. RStudio on Mac).
== Is it fine to upgrade Docker-ce while a container is running? ==
[https://www.reddit.com/r/docker/comments/q21spb/comment/hfi97s4/ Doesn't matter. Your system will stop the container if you update docker.]
== Is there a way to hibernate a docker container ==
* https://stackoverflow.com/a/45170070
* Reference for [https://docs.docker.com/engine/reference/commandline/checkpoint/ docker checkpoint] (This command is experimental)
== Live restore ==
* [https://stackoverflow.com/a/45352682 Can I save my running containers when upgrading from `docker` to `docker-ce`?]
* [https://www.sobyte.net/post/2022-05/docker-live-restore/ Docker's Live Restore feature] Docker provides a feature that keeps containers running when Daemon is unavailable, thus reducing container downtime when Daemon is upgraded or has problems. That feature is called '''Live Restore'''.
== Rate limits for GitHub Apps ==
[https://docs.github.com/en/developers/apps/rate-limits-for-github-apps Rate limits for GitHub Apps]
When I tried several times of ''docker build'', I finally got a message
{{Pre}}
Downloading GitHub repo XXX/XXXXX@HEAD
Error: Failed to install 'unknown package' from GitHub:
  HTTP error 403.
  API rate limit exceeded for XXX.XX.XXX.X. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)
  Rate limit remaining: 0/60
  Rate limit reset at: 2021-04-12 20:32:28 UTC
  To increase your GitHub API rate limit
  - Use `usethis::browse_github_pat()` to create a Personal Access Token.
  - Use `usethis::edit_r_environ()` and add the token as `GITHUB_PAT`.
Execution halted
</pre>
</pre>


Line 84: Line 205:


== Windows ==
== Windows ==
Note many of the information here have not been updated.
Docker can be run on Windows 10 Pro as a native application; see  
Docker can be run on Windows 10 Pro as a native application; see  
* https://docs.docker.com/engine/installation/windows/.
* https://docs.docker.com/engine/installation/windows/.
Line 320: Line 443:


=== WSL2 ===
=== WSL2 ===
[https://poweruser.blog/docker-on-windows-10-without-hyper-v-a529897ed1cc Docker on Windows without Hyper-V]
* [https://poweruser.blog/docker-on-windows-10-without-hyper-v-a529897ed1cc Docker on Windows without Hyper-V]
* [https://docs.microsoft.com/en-us/windows/wsl/install-manual Manual installation steps for older versions of WSL]. I got this page when I installed Docker Desktop v4.9.0 (2022/6/14) on Windows 10.


== Mac ==
== Mac ==
If you have Mac, you don't have to use boot2docker (iso & its management tool). You can use other Linux which comes with docker pre-installed. See [https://serversforhackers.com/getting-started-with-docker/ this post].
* https://docs.docker.com/desktop/mac/
* Alternatives to Docker Desktop for Mac? Rancher is recommended. 2022-06-08
* Vagrant method. If you have Mac, you don't have to use boot2docker (iso & its management tool). You can use other Linux which comes with docker pre-installed. See [https://serversforhackers.com/getting-started-with-docker/ this post].
 
== Raspberry Pi ==
* [https://docs.balena.io/reference/hardware/devices/ Resin.io] (now called [https://www.balena.io/ balenaCloud]). They are supporting Docker on very low performance devices like Raspberry Pi 1 Zero or Beaglebone Black (512MB RAM, 1GHz ARM® Cortex-A8).
* Another option is to use QEMU which allows Docker to create a container based on ARM. [https://ricardodeazambuja.com/rpi/2020/12/29/rpi2docker/ Creating a docker container from a Raspberry Pi Zero image... and the other way around]
* [https://phoenixnap.com/kb/docker-on-raspberry-pi Docker on Raspberry Pi - Installation/uninstall Guide]
* [https://raspberrytips.com/docker-on-raspberry-pi/ Getting Started With Docker On Raspberry Pi (Full Guide)]


== ARM architeture from hub.docker.com ==
== ARM architeture from hub.docker.com ==
Line 346: Line 478:
* [https://matchboxdorry.gitbooks.io/matchboxblog/content/blogs/build_and_run_arm_images.html How to Build and Run ARM Docker Images on x86 Hosts]
* [https://matchboxdorry.gitbooks.io/matchboxblog/content/blogs/build_and_run_arm_images.html How to Build and Run ARM Docker Images on x86 Hosts]


=== docker run ===
=== docker-compose ===
Some images (like [https://hub.docker.com/_/r-base r-base]) provide arm64 but not not arm.
<ul>
<li>https://docs.docker.com/compose/install/. The binary version is only available for x86. So for raspi, see the Alternative Install Options where it suggests to use [https://docs.python-guide.org/dev/virtualenvs/ virtualenv]. See also [[Python#virtualenv|Python &rarr; virtualenv]].</li>
<li>[https://withblue.ink/2020/06/24/docker-and-docker-compose-on-raspberry-pi-os.html Docker and Docker Compose on Raspberry Pi OS]
<pre>
<pre>
# 64-bit OS
sudo apt update
docker pull r-base
sudo apt install -y python3-pip libffi-dev
docker run -it --rm r-base  # enter R directly
# Install Docker Compose from pip (using Python3)
# This might take a while
sudo pip3 install docker-compose
</pre>
</pre>
</li>
</ul>


I build a 32-bit armv7 image for r-base v4.0.2. This image works on either 32-bit or 64-bit arm OS (tested on 32-/64-bit Raspberry Pi and other 32-bit SBC devices).
=== Some examples* ===
<pre>
Not I use the arm64 image on my Pi3b+.
docker pull arraytools/r402armv7
docker run -it --rm arraytools/r402armv7 R
docker pull r-base
# Using default tag: latest
# latest: Pulling from library/r-base
# no matching manifest for linux/arm/v7 in the manifest list entries
</pre>


== GUI/TUI interface manager ==
[https://fleet.linuxserver.io/ Images] from https://www.linuxserver.io/. Some indices include number of pulls and stars.
=== [https://github.com/moncho/dry Dry] ===
[https://www.2daygeek.com/dry-an-interactive-cli-manager-for-docker-containers/ Dry – An Interactive CLI Manager For Docker Containers]. The TUI is built on top of [https://github.com/gizak/termui termui]; a cross-platform, easy-to-compile, and fully-customizable terminal dashboard. It is inspired by [https://github.com/yaronn/blessed-contrib blessed-contrib], but purely in Go.


=== LazyDocker (TUI) ===
[https://en.wikipedia.org/wiki/List_of_tz_database_time_zones List of tz database time zones]
[https://www.linuxuprising.com/2019/07/lazydocker-new-docker-and-docker.html?m=1 LazyDocker: New Docker And Docker Compose Terminal UI], [https://github.com/jesseduffield/lazydocker Github]


It works on ARM devices too.
'''[https://documentation.portainer.io/v2.0/deploy/linux/ Portainer]'''. The port number is 9000. Note '''the stack will be deployed using the equivalent of docker-compose. Only Compose file format version 2 is supported at the moment.'''


=== Dockly (TUI) ===
'''[https://hub.docker.com/r/trnape/rpi-samba Samba]'''. Tested on iOS, Ubuntu & Windows 10.
[https://www.ostechnix.com/dockly-manage-docker-containers-from-terminal/ Dockly – Manage Docker Containers From Terminal]
<pre>
mkdir -p /mnt/usb/share/{data,backups}
mkdir /mnt/usb/share/data/{alice,bob,documents}
touch /mnt/usb/share/backups/backupsfile
touch /mnt/usb/share/data/bob/bobfile
touch /mnt/usb/share/data/documents/documentfile


=== DockStation ===
docker run -d -p 445:445 \
It is not open source. It works with remote Docker containers.
  -v /mnt/usb/share/data:/share/data \
  -v /mnt/usb/share/backups:/share/backups \
  --name rpi-samba trnape/rpi-samba \
  -u "alice:abc123" \
  -u "bob:secret" \
  -u "guest:guest" \
  -s "Backup directory:/share/backups:rw:alice,bob" \
  -s "Bob (private):/share/data/bob:rw:bob" \
  -s "Documents (readonly):/share/data/documents:ro:guest,alice,bob"
</pre>
On '''Windows''', 1) right click on 'This PC' and choose 'Add a network location'. 2) type \\192.168.1.249\ and the dropdown list will populate all available folders. 3) choose the one (e.g. Bob) and then enter the credential. Done. On '''Ubuntu''', just type smb://192.168.1.249/. It will then populate the available folders.


* [https://github.com/DockStation/dockstation DockStation]
'''[https://www.digitalocean.com/community/tutorials/how-to-run-nginx-in-a-docker-container-on-ubuntu-14-04 Nginx]'''
* [https://www.upnxtblog.com/index.php/2018/01/17/top-6-gui-tools-for-managing-docker-environments/ TOP 6 GUI tools for managing Docker environments] 2018
<pre>
* [https://www.how2shout.com/tools/4-best-docker-gui-tools-to-manage-containers-graphically.html 4 Best Docker GUI tools to manage containers graphically] 2019
mkdir -p /mnt/usb/docker-nginx/html
echo "hello world" >> /mnt/usb/docker-nginx/html/index.html
nano /mnt/usb/docker-nginx/html/sharefile
docker run --name rpi-nginx -p 8086:80 \
  --restart always \
  -v /mnt/usb/docker-nginx/html:/usr/share/nginx/html \
  -d nginx:stable-alpine


=== [https://portainer.io/ Portainer]* (nice) ===
# Or a stack file
<ul>
version: '2'
<li>Portainer itself is a docker application. </li>
services:
<li>https://portainer.readthedocs.io/en/stable/deployment.html#manage-a-new-docker-environment. It seems it will portainer 1.24.1 instead of 2.0. So this instruction is outdated!!
    nginx:
{{Pre}}
        container_name: rpi-nginx
$ docker volume create portainer_data
        ports:
$ docker run -d -p 9000:9000 -p 8000:8000 --name portainer --restart always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer
            - '8086:80'
        restart: always
        volumes:
            - '/mnt/usb/docker-nginx/html:/usr/share/nginx/html'
        image: nginx:stable-alpine
</pre>
</pre>
</li>
Note consider to use a samba share folder (see above) as a nginx document root.
<li>[https://dbtechreviews.com/2020/08/update-portainer-to-version-2-0-super-easy/ Update Portainer to Version 2.0 – Super Easy!]. When the first time accessing http://localhost:9000, you need to create a password (at least 8 characters long) for the default user "admin".
<pre>
{{Pre}}
cp /mnt/usb/docker-nginx/html/* /mnt/usb/share/data/bob/
$ docker volume create portainer_data
rm -rf /mnt/usb/docker-nginx/html
$ docker run -d -p 9000:9000 -p 8000:8000 --name portainer --restart always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce
ln -s /mnt/usb/share/data/bob/ /mnt/usb/docker-nginx/html
</pre>
</pre>
* [https://dbtechreviews.com/2020/09/using-application-templates-in-portainer-2-0/ Using Application Templates in Portainer 2.0]
* [https://youtu.be/BwgKd6LohQo Install 80+ Docker Images In Seconds Using This Portainer App Template] (video)
* [https://youtu.be/2oi4IQF7VnE Raspberry Pi Home Server Episode 5: Remote Access with NGINX Proxy Manager]
* [https://youtu.be/ji0kgXhXR6g Raspberry Pi Home Server Episode 8: BitWardenRS - Password Manager with Remote Access] (video)
* [https://dbtechreviews.com/2020/07/install-nextcloud-on-a-raspberry-pi-4-with-openmediavault-and-docker/ Install NextCloud on a Raspberry Pi 4 with OpenMediaVault and Docker]
</li>
</ul>
* [https://github.com/portainer/portainer-docs/issues/91 Why port 8000 is needed to be published?], [https://forums.portainer.io/t/what-port-8000-used-for-do-i-need-to-expose-it/210 What port 8000 used for? Do I need to expose it?]
* [https://dzone.com/articles/a-comparison-of-docker-guis A Comparison of Docker GUIs]. A username/password is required to access the page unless we use the '''--no-auth''' option; see [https://portainer.readthedocs.io/en/stable/configuration.html Configuration].
* [https://www.howtoforge.com/tutorial/ubuntu-docker-portainer/ How to Manage Docker Containers using Portainer on Ubuntu 16.04]. Go to Local. It will show the status of all local stacks, containers, volumes and images. Very Cool!
* [https://www.linuxtechi.com/monitor-manage-docker-containers-portainer-part1/ Monitor and Manage Docker Containers with Portainer.io (GUI tool) – Part-1] 2019
* [https://youtu.be/BwgKd6LohQo Install 80+ Docker Images In Seconds Using This Portainer App Template] (video, 2020)
** https://raw.githubusercontent.com/SelfhostedPro/selfhosted_templates/portainer-2.0/Template/template.json (working), [https://github.com/SelfhostedPro/selfhosted_templates Github]
** https://raw.githubusercontent.com/nashosted/self-hosted-template/master/template.json (working)
* [https://www.cloudsavvyit.com/8911/how-to-get-started-with-portainer-a-web-ui-for-docker/ How to Get Started With Portainer, a Web UI for Docker]
* [https://www.linuxtechi.com/monitor-manage-docker-containers-portainer-io-part-2/ Monitor and Manage Docker Containers with Portainer.io (GUI tool) – Part-2]


=== Yacht ===
'''[https://github.com/michaelmiklis/docker-rpi-monitor Rpi-monitor]'''. I need to change '''/dev/vcsm''' to '''/dev/vcsm-cma'''. But the temperature part is not working. I am using 64-bit Raspberry Pi OS and it does not show attached USB disks. The port number is 8888.
[https://youtu.be/KstIS5DstTE Yacht Installed on Docker - Portainer Alternative] (video)


=== [https://github.com/crosbymichael/dockerui DockerUI] (Deprecated, Development continues at Portainer)===
'''[https://registry.hub.docker.com/r/linuxserver/code-server/ code-server]'''
https://github.com/kevana/ui-for-docker. A quick start:
<pre>
# Run:  <syntaxhighlight lang='bash'>
---
docker run -d -p 9000:9000 --privileged \
version: "2.1"
     -v /var/run/docker.sock:/var/run/docker.sock uifd/ui-for-docker
services:
</syntaxhighlight> where '''-v''' means to bind mount a volume.
  code-server:
# Open your browser to http://<dockerd host ip>:9000
    image: ghcr.io/linuxserver/code-server
    container_name: code-server
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/New_York
      - PASSWORD=password #optional
      - SUDO_PASSWORD=password #optional
     volumes:
      - /mnt/usb/code-server:/config
    ports:
      - 8443:8443
    restart: unless-stopped
</pre>


Note: Anyone in the local network can access the website without any authentication.
'''[https://docs.linuxserver.io/images/docker-mstream mstream]''' Music streaming. Works great.
 
=== Rancher ===
* https://rancher.com/quick-start/. The doc above is not good enough to follow. Search Google for tutorial or get started guide including screenshots.
* [https://www.howtoforge.com/tutorial/ubuntu-rancher-docker-container-manager/ How to Manage Docker Containers with Rancher on Ubuntu 16.04]
* [https://hackernoon.com/creating-a-scalable-application-using-docker-rancher-rancheros-cattle-5feb83a3e743 Creating A Scalable Application Using Docker + Rancher + RancherOs + Cattle]
* Initial setup video https://youtu.be/1EMW1y7529Y
* Training from Rancher Labs https://youtu.be/8K14A_CZFdI. It seems Rancher is complicated but powerful. The whole operation is like Amazon Cloud.
* [https://gist.github.com/lmmendes/fbed32a452cf02d2a1095658795cb3d2 Running Rancher locally (with two hosts) using MacOS Docker and VirtualBox]
* To allow UDP port
<syntaxhighlight lang='bash'>
$ sudo apt-get install ufw
$ sudo ufw allow 4500/udp
$ sudo ufw allow 500/udp
</syntaxhighlight>
* discoposse.com
** [http://discoposse.com/2015/11/08/rancher-part-1-installing-rancher-and-setting-access-control/ Part 1] Installing Rancher and Setting Access Control
** [http://discoposse.com/2015/11/09/rancher-part-2-adding-a-docker-host-to-rancher/ Part 2] Adding a Docker Host to Rancher
** [http://discoposse.com/2015/11/12/rancher-part-3-adding-the-dockerhub-to-our-rancher-registry/ Part 3] Adding the DockerHub to our Rancher Registry
** [http://discoposse.com/2015/11/13/rancher-part-4-using-the-catalog-example-with-glusterfs/ Part 4] Using the Catalog Example with GlusterFS
 
=== [https://github.com/tobegit3hub/seagull Seagull] ===
https://youtu.be/TuT5gb8oRw8


'''[https://registry.hub.docker.com/r/emby/embyserver/#installation emby]''' does not work on arm64. It works on x86 though. Even I copy a mp4 file to movies directory the movie does not show up:(
<pre>
<pre>
docker run -d -p 127.0.0.1:10086:10086 -v /var/run/docker.sock:/var/run/docker.sock tobegit3hub/seagull
version: '2.1'
services:
    embyserver:
        container_name: emby
        network_mode: bridge
        restart: always
        environment:
            - VERSION=latest
            - UID=1000
            - GID=1000
            - TZ=America/Denver
        volumes:
            - /media/crucial/emby/config:/config
            - /media/crucial/emby/tv:/mnt/tv
            - /media/crucial/emby/movies:/mnt/movies
        ports:
            - 8096:8096           
        image: 'emby/embyserver:latest'
</pre>
</pre>


The only issue is there is no username/password to protect other people to access the web GUI. The solution of binding to localhost to restrict the access does not work for remote administration.
'''[https://github.com/linuxserver/docker-jellyfin jellyfin]''' Jellyfin is descended from Emby's 3.5.2 release and ported to the .NET Core framework to enable full cross-platform support. [https://youtu.be/c28kUNKJcqQ How to Install Jellyfin on Docker with Portainer]


That is, the tool is suitable for home use.
'''[https://github.com/greensheep/plex-server-docker-rpi plex]''' We can access the plex server via http://IP:32400/web. Note that in the first server setup, we need to add '''Library'''' by choosing the new library name (eg Other Videos) shown on plex & the data source (eg /data) so our own media can be found. After we added new media files we can rescan by clicking the vertical 3 dots icon and selecting scan library files. Pi3b+ is still a little weak since I can see all threads are busy when I played a mp4 file.
{{Pre}}
mkdir -p /mnt/usb/plex/{config,data}
cp FILENAme.mp4 /mnt/usb/plex/data
docker run \
  -d \
  --name plex \
  --net host \
  -p 32400:32400 \
  --restart always \
  --volume /mnt/usb/plex/config:/config \
  --volume /mnt/usb/plex/data:/data \
  greensheep/plex-server-docker-rpi:latest


=== [https://kitematic.com/ Kitematic] (Mac, Windows and Ubuntu) ===
WARNING: The requested image's platform (linux/arm) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Owned by Docker. Available for Mac OS X 10.8+ and Windows 7+ (64-bit) and Ubuntu. https://github.com/docker/kitematic/releases/
</pre>
 
'''[https://youtu.be/7EoEll0lVXc Nextcloud]'''.
<pre>
sudo mkdir -p /srv/dev-disk-by-label-Files/Databases/NextCloud
sudo mkdir -p /srv/dev-disk-by-label-Files/Config/Nextcloud
</pre>
After that, copy and paste the [https://gist.github.com/dnburgess/fdcaf27280e031fe88dd2fb9728649f6 stack] into portainer. Wait for a few minutes on RPi3. The port number is 8080. Now we can create the admin username/password such as nextcloud/nextcloud. Click the little triangle next to "Storage and Database". Change to MySQL. In the next part we enter nextcloud/nextcloud/nextcloud/db (note the "db" replaces localhost b/c we use "db" as the service name). Again, wait for a few minutes.
 
'''[https://youtu.be/I3fOq6zx1Zk?t=305 Heimdall]''' (Dashboard for web apps). I keep the PUID (1000) and PGID (1000). The instruction says it is from the '''admin''' user account but I don't find ''admin'' account? Change the volume to /srv/dev-disk-by-label-Files/Config/Heimdall (use '''sudo mkdir''' to create the directory on terminal). Change the port to 83 & remove port 443. Define the endpoint from Portainer -> Endpoints -> local -> Public IP as ''raspberrypi.local'' (depending on your hostname). We need to wait a little bit. Now go to the container and find heimdall and click the port in order to open the website correctly (instead 0.0.0.0). I can add apps like nextcloud, portainer, pi-hole, other servers, etc. The '''Application Type''' entry has a good list of popular apps and it will pre-populate the button icon and the background color for our app.


Run containers through a simple, yet powerful graphical user interface.
'''[https://hub.docker.com/r/linuxserver/taisun taisun]''' The default port is 3000


It can not connect to remote docker machines.
'''[https://yacht.sh/Installation/yacht/ yacht]'''. The default login is '''[email protected]''' and '''pass'''. The name shown on portainer is '''pedantic_hermann'''
{{Pre}}
docker volume create yacht
docker run -d -p 8001:8000 -v /var/run/docker.sock:/var/run/docker.sock -v yacht:/config selfhostedpro/yacht
</pre>


A [https://www.r-bloggers.com/share-your-shiny-apps-with-docker-and-kitematic/ Share your Shiny Apps with Docker and Kitematic!]
'''[https://youtu.be/Nf7m3h11y-s CloudFlare DDNS]''' - Update CloudFlare with Your Dynamic IP Address


=== [https://github.com/shipyard/shipyard Shipyard] (retired) ===
'''[https://youtu.be/nQMYfpCY310 WatchTower]'''
* [https://www.sumologic.com/blog/code/managing-containers-docker-shipyard/ Managing Containers with Docker Shipyard]
* Three alternatives:
** [http://rancher.com/ Rancher]
** [https://github.com/kevana/ui-for-docker Docker UI]
** [https://portainer.io/ Portainer]


=== VS Code ===
'''[https://youtu.be/ji0kgXhXR6g bitwardenrs]'''. Use the terminal to create a volume first. The port number is 8100. This is straightforward.
* [https://youtu.be/i7ABlHngi1Q Docker For Beginners: From Docker Desktop to Deployment]
* [https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers Visual Studio Code Remote - Containers]


= Usage =
'''[https://youtu.be/-NyzdAYMarw Duplicati]''' for backup.
== Basics, docs, cheatsheet ==
* https://docs.docker.com/articles/basics/
* https://www.docker.com/sites/default/files/d8/2019-09/docker-cheat-sheet.pdf
* https://coderwall.com/p/2es5jw/docker-cheat-sheet-with-examples
* https://github.com/wsargent/docker-cheat-sheet
* [https://www.howtoforge.com/tutorial/how-to-use-docker-introduction/ How to use Docker in a practical way]
* https://hackernoon.com/introduction-to-docker-a7d9e1f6c0b3#.tezit0hkq
* [http://www.cnblogs.com/wanliwang01/p/docker01.html Docker快速入门]
* [http://blog.myplanet.com/docker-the-fun-and-easy-way Docker: The Fun and Easy Way]


Note that we need '''sudo''' is needed unless it is on a Mac OS.
'''[https://fleet.linuxserver.io/image?name=linuxserver/photoshow photoshow]'''. It works. It has a slideshow button. PhotoShow only displays videos in WebM.  


If docker cannot find an image, it will try to pull it from its repository.
'''R'''. [https://hub.docker.com/_/r-base r-base] provide arm64 image but not not 32-bit arm architecture.
<pre>
<pre>
$ sudo docker run -it ubuntu /bin/bash
# 64-bit OS
Unable to find image 'ubuntu' locally
docker pull r-base
docker run -it --rm r-base  # enter R directly
</pre>
 
'''rocker/rstudio''' DOES NOT work on arm64 even I can pull. ''WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested''
 
I build a 32-bit armv7 image for r-base v4.0.2. This image works on either 32-bit or 64-bit arm OS (tested on 32-/64-bit Raspberry Pi and other 32-bit SBC devices).
<pre>
docker pull arraytools/r402armv7
docker run -it --rm arraytools/r402armv7 R
docker pull r-base
# Using default tag: latest
# latest: Pulling from library/r-base
# no matching manifest for linux/arm/v7 in the manifest list entries
</pre>
 
== How and Why to Use A Remote Docker Host ==
* [https://www.cloudsavvyit.com/11185/how-and-why-to-use-a-remote-docker-host/ How and Why to Use A Remote Docker Host]
* [https://www.cloudsavvyit.com/12514/how-docker-contexts-let-you-work-with-multiple-hosts/ How Docker Contexts Let You Work with Multiple Hosts]
 
== Backup ==
* [https://docs.docker.com/storage/volumes/#backup-restore-or-migrate-data-volumes Backup, restore, or migrate data volumes]
* [https://stackoverflow.com/a/28145912 Backup a running Docker container?]
* [https://www.maketecheasier.com/copy-move-docker-container-to-another-host/ How to Copy/Move a Docker Container to Another Host]
* [https://www.reddit.com/r/docker/comments/vi0o4k/comment/idbndut/?utm_source=share&utm_medium=web2x&context=3 Is backing up `/var/lib/docker/volumes` a good way to back up your Docker containers? If not, what is?]
 
= Usage =
== Basics, docs, cheatsheet, introduction ==
* https://docs.docker.com/articles/basics/
* [https://www.fosstechnix.com/docker-command-cheat-sheet/ 81 Docker Command Cheat Sheet with Description]
* https://coderwall.com/p/2es5jw/docker-cheat-sheet-with-examples
* https://github.com/wsargent/docker-cheat-sheet
* [https://www.howtoforge.com/tutorial/how-to-use-docker-introduction/ How to use Docker in a practical way]
* https://hackernoon.com/introduction-to-docker-a7d9e1f6c0b3#.tezit0hkq
* [http://www.cnblogs.com/wanliwang01/p/docker01.html Docker快速入门]
* [http://blog.myplanet.com/docker-the-fun-and-easy-way Docker: The Fun and Easy Way]
* [https://www.r-bloggers.com/2023/06/a-gentle-introduction-to-docker/ A Gentle Introduction to Docker]. docker build & renv.
 
Note that we need '''sudo''' is needed unless it is on a Mac OS.
 
If docker cannot find an image, it will try to pull it from its repository.
<pre>
$ sudo docker run -it ubuntu /bin/bash
Unable to find image 'ubuntu' locally
Pulling repository ubuntu
Pulling repository ubuntu
04c5d3b7b065: Download complete  
04c5d3b7b065: Download complete  
Line 805: Line 1,002:
$ sudo docker exec -i -t loving_heisenberg bash #by Name
$ sudo docker exec -i -t loving_heisenberg bash #by Name
</pre>
</pre>
== Rootless mode ==
* [https://docs.docker.com/engine/security/rootless/ Run the Docker daemon as a non-root user (Rootless mode)]
* [https://serverfault.com/a/1128797 What's the difference between rootless Docker, running a container as a non-root user, and Podman?]
* [https://itnext.io/docker-running-in-rootless-mode-bdbcfc728b3a Docker Running In Rootless Mode]
* [https://mohitgoyal.co/2021/04/14/going-rootless-with-docker-and-containers/ Going rootless with Docker and Containers]
* [https://www.liquidweb.com/kb/how-to-docker-rootless-containers/ How to Run Rootless Docker Containers]


== docker pull ==
== docker pull ==
Line 839: Line 1,043:
* [https://bytefreaks.net/applications/docker/how-to-update-all-pulled-docker-images-that-are-tagged-as-latest How to update all pulled Docker images that are tagged as latest]
* [https://bytefreaks.net/applications/docker/how-to-update-all-pulled-docker-images-that-are-tagged-as-latest How to update all pulled Docker images that are tagged as latest]
* [https://vsupalov.com/docker-latest-tag/ What's Wrong With The Docker :latest Tag?] '''Do not run any container with the latest tag.'''
* [https://vsupalov.com/docker-latest-tag/ What's Wrong With The Docker :latest Tag?] '''Do not run any container with the latest tag.'''
* [https://www.reddit.com/r/docker/comments/vjx9ct/how_to_upgrade_container_properly/ How to upgrade container properly?]  "docker-compose pull" to update all your service and "docker-compose up -d" to start them all. docker swarm is even better because you can achieve zero downtme rolling upgrades.
* The following two are equivalent
:<syntaxhighlight lang='bash'>
docker compose pull && docker compose up -d
docker compose up --pull always -d
</syntaxhighlight>


=== <none>:<none> images ===
=== <none>:<none> images ===
Line 853: Line 1,063:
$ docker container exec -it 70c5aceb5512 bash
$ docker container exec -it 70c5aceb5512 bash
</syntaxhighlight>
</syntaxhighlight>
== How to start a stopped Docker container with a different command ==
[https://stackoverflow.com/a/39329138 How to start a stopped Docker container with a different command?]
== Clean shutdown DOCKER containers before reboot ==
* [https://forum.openmediavault.org/index.php?thread/22012-clean-shutdown-docker-containers-before-reboot-or-shutdown-omv/ Clean shutdown DOCKER containers before reboot or shutdown OMV] '''systemctl stop docker'''
* [https://linuxhandbook.com/docker-stop-container/ How to Stop Docker Containers] '''docker ps -q | xargs docker stop'''


== Dockerizing Applications/Detached mode ==  
== Dockerizing Applications/Detached mode ==  
Line 896: Line 1,113:
* [https://docs.docker.com/compose/environment-variables/ Environment variables in Compose]
* [https://docs.docker.com/compose/environment-variables/ Environment variables in Compose]
* An example: [https://github.com/tcardonne/docker-github-runner Docker github runner]
* An example: [https://github.com/tcardonne/docker-github-runner Docker github runner]
=== Docker container ID ===
* The full container ID is a hexadecimal string of 64 characters.
* The minimum number of characters required for a Docker ID is 4.
* We can use a shorter ID in docker command if that ID uniquely determined the container. For example, '''docker exec -it 9608 bash''' or even '''docker exec -it 9 bash''' works.


=== Alpine image ===
=== Alpine image ===
* [https://www.makeuseof.com/alpine-linux-explained/ Alpine Linux: A Breath of Fresh Mountain Air for Linux Experts]
* [https://docs.docker.com/samples/library/alpine/ A minimal Docker image based on Alpine Linux with a complete package index and only 5 MB in size!]
* [https://docs.docker.com/samples/library/alpine/ A minimal Docker image based on Alpine Linux with a complete package index and only 5 MB in size!]
* [http://containertutorials.com/alpine/get_started.html Container Tutorials > Getting Started with Alpine]
* [http://containertutorials.com/alpine/get_started.html Container Tutorials > Getting Started with Alpine]
Line 919: Line 1,142:
</pre>
</pre>


=== Check container status (docker status) - CPU, Mem usage ===
=== Check container status (docker status) - CPU, Memory usage ===
[https://github.com/ColinFay/dockerstats docker stats]
<ul>
<li>[https://www.cloudytuts.com/tutorials/docker/how-to-check-memory-and-cpu-utilization-of-docker-container/ How to Check Memory and CPU Utilization of Docker Container], [https://www.howtoforge.com/how-to-check-docker-container-ram-and-cpu-usage/ How to Check Docker Container RAM and CPU Usage]
<pre>
docker stats
</pre>
<li>[https://github.com/ColinFay/dockerstats docker stats]
</ul>


=== Container networking===
=== Container networking===
* https://docs.docker.com/network/
<ul>
* [https://docs.docker.com/network/network-tutorial-standalone/#use-user-defined-bridge-networks Use user-defined bridge networks]
<li>https://docs.docker.com/network/
<li>[https://docs.docker.com/network/network-tutorial-standalone/#use-user-defined-bridge-networks Use user-defined bridge networks]
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
$ docker network create MyNewNetworkName
$ docker network create MyNewNetworkName
Line 936: Line 1,166:
# OR docker network rm MyNewNetworkName
# OR docker network rm MyNewNetworkName
</syntaxhighlight>
</syntaxhighlight>
* https://youtu.be/Ox5mZBwc92E
<li>https://youtu.be/Ox5mZBwc92E
<li>Use '''dig''' to find out the IP and run DNS query. [https://jvns.ca/blog/2021/01/04/docker-compose-is-nice/ Docker Compose: a nice way to set up a dev environment]
<li>[https://dev.to/leandronsp/mastering-the-docker-networking-2h57 Mastering the Docker networking]
</ul>


=== [https://docs.docker.com/network/host/ Host network] ===
=== [https://docs.docker.com/network/host/ Host network] ===
Line 956: Line 1,189:
=== Access Ports on the Host from a Docker Container ===
=== Access Ports on the Host from a Docker Container ===
[https://www.cloudsavvyit.com/6940/how-to-access-ports-on-the-host-from-a-docker-container/ How to Access Ports on the Host from a Docker Container]
[https://www.cloudsavvyit.com/6940/how-to-access-ports-on-the-host-from-a-docker-container/ How to Access Ports on the Host from a Docker Container]
=== Multiple NICs ===
[https://www.reddit.com/r/docker/comments/vi2alw/containers_in_docker_to_use_public_ip_addresses/ containers in docker to use public ip addresses directly]


=== Viewing the Web Application's Logs ===
=== Viewing the Web Application's Logs ===
Line 961: Line 1,197:
$ sudo docker logs -f nostalgic_morse
$ sudo docker logs -f nostalgic_morse
</pre>
</pre>
=== Clear Logs of Running Docker Containers ===
[https://www.howtogeek.com/devops/how-to-clear-logs-of-running-docker-containers/ How to Clear Logs of Running Docker Containers]


=== Looking at our Web Application Container's processes ===
=== Looking at our Web Application Container's processes ===
Line 973: Line 1,212:


=== Obtain the container's IP address, log into a running server ===
=== Obtain the container's IP address, log into a running server ===
PS. '''Portainer''' web interface can show the IP addresses.
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
$ docker inspect <container id> | grep "IPAddress"
$ docker inspect <container id> | grep "IPAddress"
</syntaxhighlight>
</syntaxhighlight>


Then to log into a running server, use
We don't need the IP address if we just want to log into a running server,
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
$ docker exec -it <contianer id> bash
$ docker exec -it <contianer id> bash
</syntaxhighlight>
</syntaxhighlight>
=== How to Secure Docker’s TCP Socket ===
[https://www.cloudsavvyit.com/15079/how-to-secure-dockers-tcp-socket-with-tls/ How to Secure Docker’s TCP Socket with TLS]


=== docker attach ===
=== docker attach ===
Line 995: Line 1,238:
=== docker exec: SSH into a running container ===
=== docker exec: SSH into a running container ===
Run a command in a running container
Run a command in a running container
 
<ul>
* [https://docs.docker.com/engine/reference/commandline/exec/ Usage]: <syntaxhighlight lang='bash'>
<li>[https://docs.docker.com/engine/reference/commandline/exec/ Usage]: <syntaxhighlight lang='bash'>
docker exec [OPTIONS] CONTAINER COMMAND [ARG...]
docker exec [OPTIONS] CONTAINER COMMAND [ARG...]
</syntaxhighlight>
</syntaxhighlight>
* Examples: <syntaxhighlight lang='bash'>
<li>Examples: <syntaxhighlight lang='bash'>
$ docker exec -d ubuntu_bash touch /tmp/execWorks # do st in the background
$ docker exec -d ubuntu_bash touch /tmp/execWorks # do st in the background


Line 1,009: Line 1,252:
$ docker exec -it -w /root ubuntu_bash pwd # change the working directory
$ docker exec -it -w /root ubuntu_bash pwd # change the working directory
</syntaxhighlight>
</syntaxhighlight>
<li>[https://www.cloudsavvyit.com/14541/how-to-run-a-command-on-a-running-docker-container/ How to Run a Command on a Running Docker Container]
<li>[https://thenewstack.io/how-to-use-the-docker-exec-command/ How to Use the Docker exec Command]. nginx container is used as an example.
<pre>
docker run --name docker-nginx -p 8080:80 -d nginx
# method 1. Access the Running Container’s Shell
docker exec -it ID /bin/bash
  apt-get update
  apt-get upgrade -y
  exit
# method 2. Run a Command from Outside the Container
docker exec ID apt-get update &amp;&amp; apt-get upgrade
docker exec ID cat /usr/share/nginx/html/index.html
docker cp index.html ID:/usr/share/nginx/html/
docker exec ID cat /usr/share/nginx/html/index.html
</pre>
</ul>


=== docker cp ===
=== docker cp ===
Line 1,020: Line 1,282:
* https://www.linuxnix.com/docker-start-exited-container/
* https://www.linuxnix.com/docker-start-exited-container/
<pre>
<pre>
$ sudo docker start nostalgic_morse
$ docker start nostalgic_morse
OR
OR
$ sudo docker restart nostalgic_morse
$ docker restart nostalgic_morse
</pre>
For an interactive container, use '''docker start -ai CONTAINER''' which is equal to run "docker start CONTAINER" and "docker attach CONTAINER".
 
=== Rename a container ===
[https://docs.docker.com/engine/reference/commandline/container_rename/ docker container rename]
<pre>
docker container rename CONTAINER NEW_NAME
</pre>
</pre>


=== Inspect container images and their metadata ===
=== Inspect container images and their metadata ===
[https://microbadger.com/images/rocker/r-base MicroBadger]
* [https://www.cloudsavvyit.com/14663/how-to-inspect-a-docker-images-content-without-starting-a-container/ How to Inspect a Docker Image’s Content Without Starting a Container]
* [https://microbadger.com/images/rocker/r-base MicroBadger]


=== Know the container size ===
=== Know the container size ===
Line 1,076: Line 1,346:


'''docker create''' is similar to '''docker run -d''' except the container is never started.
'''docker create''' is similar to '''docker run -d''' except the container is never started.
=== Retrieve docker run command ===
https://stackoverflow.com/a/32774347. See the github page of [https://github.com/lavie/runlike runlike]. So it is better to put the docker run in a stack. Then for example the Portainer has an Editor tab to show the [https://docs.docker.com/compose/compose-file/ compose file].
<pre>
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
    assaflavie/runlike -p CONTAINER_NAME
</pre>
The '''-p''' option splits the output into pretty lines.
=== docker run -it and -d together ===
* [https://stackoverflow.com/a/41918607 Practically, what is the difference between docker run -dit(-itd) vs docker run -d?]
* [https://github.com/docker/docker.github.io/issues/10632 "docker run -dit" vs "docker run -d" #10632]
=== How to Modify the Configuration of Running Docker Containers ===
[https://www.cloudsavvyit.com/15254/how-to-modify-the-configuration-of-running-docker-containers/ How to Modify the Configuration of Running Docker Containers]


== Volume ==
== Volume ==
Line 1,083: Line 1,368:
* [https://www.melvinvivas.com/using-docker-data-volume-with-a-mysql-container/ Using Docker Data Volume with a MySQL container]
* [https://www.melvinvivas.com/using-docker-data-volume-with-a-mysql-container/ Using Docker Data Volume with a MySQL container]
* [https://www.cloudsavvyit.com/7500/what-are-docker-volumes-and-how-do-you-use-them/ What Are Docker Volumes, and How Do You Use Them?]
* [https://www.cloudsavvyit.com/7500/what-are-docker-volumes-and-how-do-you-use-them/ What Are Docker Volumes, and How Do You Use Them?]
=== Examples of host's volume locations ===
<pre>
/home/$USER/docker/$PROJECT/$SUB-DIRECTORY
</pre>
=== Back Up Your Docker Volumes ===
[https://www.howtogeek.com/devops/how-to-back-up-your-docker-volumes/ How to Back Up Your Docker Volumes]


=== Two ways to achieve persistent data ===
=== Two ways to achieve persistent data ===
Line 1,116: Line 1,409:


=== Remove an an unnamed volume ===
=== Remove an an unnamed volume ===
I you want to automatically removes volumes when a container is removed, you can use the '''--rm''' flag when starting the container with the "docker run" command. This flag tells Docker to automatically remove the container and any '''anonymous volumes''' associated with it when the container exits. However, this flag does not affect '''named volumes'''.
If you created an unnamed volume, it can be deleted at the same time as the container with the -v flag. Note that this only works with unnamed volumes.
If you created an unnamed volume, it can be deleted at the same time as the container with the -v flag. Note that this only works with unnamed volumes.
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
docker rm -v container_name
docker rm -v container_name
Line 1,123: Line 1,417:


If the volume is named, it stays present. To remove a named volume, use '''docker volume rm volume_name''' .
If the volume is named, it stays present. To remove a named volume, use '''docker volume rm volume_name''' .
=== Volumes created in docker-compose ===
When you use '''docker-compose''' to create and manage containers, volumes are handled slightly differently than when using the '''docker run''' command.
In a "docker-compose.yml" file, you can specify named volumes using the volumes key at the top level of the file. These volumes are created when you run docker-compose up and are not automatically removed when you stop or remove the containers using docker-compose down.
If you want to remove '''named volumes''' created by docker-compose, you can use the '''-v''' flag with the '''docker-compose down''' command. Here’s an example command that stops and removes all containers defined in a docker-compose.yml file and also removes any named volumes:
<pre>
docker-compose down -v
</pre>
This command stops and removes all containers defined in the docker-compose.yml file and also removes any named volumes specified in the file. All data stored in the volumes will be permanently deleted.
'''Anonymous volumes''' created by docker-compose are automatically removed when you stop and remove the containers using docker-compose down, even if you don’t use the -v flag.


=== [https://docs.docker.com/storage/volumes/#start-a-container-with-a-volume Start a container with a volume] ===
=== [https://docs.docker.com/storage/volumes/#start-a-container-with-a-volume Start a container with a volume] ===
Line 1,141: Line 1,448:
* '''target''' in "--mount" can be replaced by '''destination''' or '''dst'''.
* '''target''' in "--mount" can be replaced by '''destination''' or '''dst'''.
* To use a read-only volume, add the ''',readonly''' option in "--mount" or the ''':ro''' option in "-v".
* To use a read-only volume, add the ''',readonly''' option in "--mount" or the ''':ro''' option in "-v".
* We cannot use "~/" to represent a local directory under HOME. We have to specify a full path in '''docker run'''.


=== A simple example ===
=== A simple example ===
Line 1,182: Line 1,490:


=== Sharing data between containers ===
=== Sharing data between containers ===
[https://www.cloudsavvyit.com/15112/how-to-share-data-between-docker-containers/ How to Share Data Between Docker Containers]
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
docker run -it --name writer -v shared-data:/data alpine /bin/sh
docker run -it --name writer -v shared-data:/data alpine /bin/sh
Line 1,259: Line 1,569:
=== Differences between VOLUME and '-v|--volume' ===
=== Differences between VOLUME and '-v|--volume' ===
https://stackoverflow.com/a/25312719
https://stackoverflow.com/a/25312719
=== Backup and restore container ===
* https://docs.docker.com/storage/volumes/#backup-restore-or-migrate-data-volumes
* [https://stackoverflow.com/a/28145912 Backup a running Docker container?]


=== Container Memory Limits, Setting Available CPUs, Allocating memory and CPU ===
=== Container Memory Limits, Setting Available CPUs, Allocating memory and CPU ===
Line 1,280: Line 1,586:
* [https://docs.docker.com/config/containers/resource_constraints/ Limit a container's resources]
* [https://docs.docker.com/config/containers/resource_constraints/ Limit a container's resources]
* [https://blog.docker.com/2018/12/top-5-post-docker-container-java/ TOP 5 BLOG OF 2018: IMPROVED DOCKER CONTAINER INTEGRATION WITH JAVA 10]
* [https://blog.docker.com/2018/12/top-5-post-docker-container-java/ TOP 5 BLOG OF 2018: IMPROVED DOCKER CONTAINER INTEGRATION WITH JAVA 10]
* [https://www.jottr.org/2022/04/22/parallelly-1.31.1/ parallelly 1.31.1: Better at Inferring Number of CPU Cores with Cgroups and Linux Containers]
* [https://www.thorsten-hans.com/limit-memory-for-docker-containers Docker Container Memory Limits Explained]
* [https://www.howtogeek.com/devops/how-to-set-a-memory-limit-for-docker-containers/ How to Set a Memory Limit for Docker Containers]


== Work with container images ==
== Work with container images ==
Line 1,291: Line 1,600:
# by name
# by name
docker images --format "{{.ID}}\t{{.Size}}\t{{.Repository}}" | sort -k 3  
docker images --format "{{.ID}}\t{{.Size}}\t{{.Repository}}" | sort -k 3  
</pre>
=== List specific columns ===
<pre>
docker images --format 'table {{.Repository}}\t{{.Tag}}\t{{.Size}}'
</pre>
</pre>


Line 1,320: Line 1,634:


=== Create an image interactively using '''commit''' - Example 2 ===
=== Create an image interactively using '''commit''' - Example 2 ===
* [https://www.cloudsavvyit.com/15489/how-to-create-a-docker-image-from-a-running-container/ How to Create a Docker Image From a Running Container]. Commit messages are displayed when you use the '''docker history''' command to view the layers in an image.
* [http://www.techrepublic.com/article/how-to-commit-changes-to-a-docker-image/ How to commit changes to a docker image]
* [http://www.techrepublic.com/article/how-to-commit-changes-to-a-docker-image/ How to commit changes to a docker image]


Line 1,358: Line 1,673:
docker load < out.tar.gz  
docker load < out.tar.gz  
</syntaxhighlight>
</syntaxhighlight>
=== Docker Image Manifest ===
[https://www.cloudsavvyit.com/12330/what-is-a-docker-image-manifest/ What Is a Docker Image Manifest?]


=== Resources allocated to a container using docker? ===
=== Resources allocated to a container using docker? ===
https://stackoverflow.com/questions/16084741/how-do-i-set-resources-allocated-to-a-container-using-docker
https://stackoverflow.com/questions/16084741/how-do-i-set-resources-allocated-to-a-container-using-docker


=== hub.docker.com ===
== hub.docker.com ==
* Note that Dockerfile is optional on https://registry.hub.docker.com.
* Note that Dockerfile is optional on https://registry.hub.docker.com.
* For free account, we can have only [https://docs.docker.com/docker-hub/repos/ 1 free private repository]. If we want to create a new repository online, we can
* For free account, we can have only [https://docs.docker.com/docker-hub/repos/ 1 free private repository]. If we want to create a new repository online, we can
Line 1,399: Line 1,717:
$ curl -H "Accept: application/xml" -H "Content-Type: application/json" -X GET http://localhost:5000/v2/_catalog
$ curl -H "Accept: application/xml" -H "Content-Type: application/json" -X GET http://localhost:5000/v2/_catalog
</syntaxhighlight>
</syntaxhighlight>
* [https://www.howtogeek.com/devops/how-to-login-to-docker-hub-and-private-registries-with-the-docker-cli/ How to Login to Docker Hub and Private Registries With The Docker CLI]
=== Github registry ===
* [https://www.docker.com/blog/docker-support-for-the-new-github-container-registry/ Docker Support for the New GitHub Container Registry]
* [https://docs.github.com/en/packages/guides/migrating-to-github-container-registry-for-docker-images Migrating to GitHub Container Registry for Docker images]
* [https://www.cloudsavvyit.com/6871/how-to-get-started-with-githubs-new-docker-container-registry/ How to Get Started with Github’s New Docker Container Registry]
* [https://nikiforovall.github.io/docker/2020/09/19/publish-package-to-ghcr.html Publish images to GitHub Container Registry (ghcr)]
<pre>
docker pull ghcr.io/OWNER/IMAGE_NAME:TAG
# docker pull registry-url/image-name:tag
</pre>


=== Google cloud registry ===
=== Google cloud registry ===
Line 1,421: Line 1,752:
*** If we use ENTRYPOINT + CMD, ENTRYPOINT defines the command and CMD defines parameters. The example above will run ''ping 8.8.8.8 -c 3''. This form is called the '''exec''' form.
*** If we use ENTRYPOINT + CMD, ENTRYPOINT defines the command and CMD defines parameters. The example above will run ''ping 8.8.8.8 -c 3''. This form is called the '''exec''' form.
* [https://github.com/jamtur01/dockerbook-code The Docker Book]
* [https://github.com/jamtur01/dockerbook-code The Docker Book]
* [https://github.com/rocker-org/rocker rocker (R and RStudio)]
* [https://github.com/Bioconductor/bioc_docker/tree/master/out Bioconductor]


=== Examples of Dockerfile ===
=== Examples of Dockerfile ===
Line 1,464: Line 1,793:
<li> [https://stackoverflow.com/a/45673309 How to give non-root user in Docker container access to a volume mounted on the host] </li>
<li> [https://stackoverflow.com/a/45673309 How to give non-root user in Docker container access to a volume mounted on the host] </li>
</ul></ul>
</ul></ul>
==== Rocker ====
[https://github.com/rocker-org/rocker rocker (R and RStudio)]
==== Bioconductor ====
[https://github.com/Bioconductor/bioconductor_docker Bioconductor]


=== How to use Dockerfile ===
=== How to use Dockerfile ===
https://docs.docker.com/engine/reference/commandline/build/
https://docs.docker.com/engine/reference/commandline/build/


[https://stackoverflow.com/a/55034787 The . simply means "current working directory"].
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
docker build Dockerfile -t arraytools/myimagename
docker build -f Dockerfile -t arraytools/myimagename .


docker build -t [myname] .   
docker build -t [myname] .   
Line 1,484: Line 1,820:
docker container run --rm -it pinger
docker container run --rm -it pinger
</syntaxhighlight>
</syntaxhighlight>
=== Docker Build Args ===
[https://www.howtogeek.com/devops/how-to-use-docker-build-args-to-configure-image-builds/ How to Use Docker Build Args to Configure Image Builds]
=== Clean up after failed builds ===
[https://gist.github.com/kuznero/0c751283ea77177a1db9 Cleanup docker images and containers after failed builds]
<pre>
#!/bin/bash
docker rm $(docker ps -aq) \
  docker rmi $(docker images | grep "^<none>" | awk '{print $3}')
</pre>


=== ENTRYPOINT and CMD ===
=== ENTRYPOINT and CMD ===
Line 1,514: Line 1,861:


=== Use multi-stage builds ===
=== Use multi-stage builds ===
https://docs.docker.com/develop/develop-images/multistage-build/
* https://docs.docker.com/develop/develop-images/multistage-build/
* [https://www.cloudsavvyit.com/9260/what-are-multi-stage-docker-builds/ What Are Multi-Stage Docker Builds?]


With multi-stage builds, we have a single Dockerfile containing multiple FROM instructions. Each FROM instruction is a new '''build stage''' that can easily COPY artifacts from previous '''stages'''.
With multi-stage builds, we have a single Dockerfile containing multiple FROM instructions. Each FROM instruction is a new '''build stage''' that can easily COPY artifacts from previous '''stages'''.
Line 1,613: Line 1,961:


== Assign a static IP to a container ==
== Assign a static IP to a container ==
* [https://www.cloudsavvyit.com/14508/how-to-assign-a-static-ip-to-a-docker-container/ How to Assign a Static IP to a Docker Container]
* https://stackoverflow.com/questions/27937185/assign-static-ip-to-docker-container <syntaxhighlight lang='bash'>
* https://stackoverflow.com/questions/27937185/assign-static-ip-to-docker-container <syntaxhighlight lang='bash'>
# Step 1. create you own docker network (mynet123)
# Step 1. create you own docker network (mynet123)
Line 1,625: Line 1,974:
* https://docs.docker.com/engine/reference/commandline/network_create/#bridge-driver-options
* https://docs.docker.com/engine/reference/commandline/network_create/#bridge-driver-options
* [https://stackoverflow.com/questions/25529386/how-can-i-set-a-static-ip-address-in-a-docker-container Try using the --net=host option] (no a new IP was allocated; not a good solution)
* [https://stackoverflow.com/questions/25529386/how-can-i-set-a-static-ip-address-in-a-docker-container Try using the --net=host option] (no a new IP was allocated; not a good solution)
== Running Multiple Docker Services on the Same Server ==
[https://www.howtogeek.com/devops/how-to-run-multiple-docker-containers-on-different-ip-addresses/ How to Run Multiple Docker Containers on Different IP Addresses]


== Firewall ==
== Firewall ==
Line 1,658: Line 2,010:


== Enabling HTTPS/Let's encrypt ==
== Enabling HTTPS/Let's encrypt ==
* [https://caddyserver.com/ Caddy]
* https://hub.docker.com/r/jrcs/letsencrypt-nginx-proxy-companion
* https://hub.docker.com/r/jrcs/letsencrypt-nginx-proxy-companion
* [https://medium.com/bros/enabling-https-with-lets-encrypt-over-docker-9cad06bdb82b Enabling HTTPS with Let's Encrypt on Docker]
* [https://medium.com/bros/enabling-https-with-lets-encrypt-over-docker-9cad06bdb82b Enabling HTTPS with Let's Encrypt on Docker]
Line 1,680: Line 2,033:
* [https://www.digitalocean.com/community/tutorials/how-to-use-traefik-as-a-reverse-proxy-for-docker-containers-on-ubuntu-18-04 How To Use Traefik as a Reverse Proxy for Docker Containers on Ubuntu 18.04]
* [https://www.digitalocean.com/community/tutorials/how-to-use-traefik-as-a-reverse-proxy-for-docker-containers-on-ubuntu-18-04 How To Use Traefik as a Reverse Proxy for Docker Containers on Ubuntu 18.04]
* [https://www.howtoforge.com/how-to-install-and-use-traefik-as-a-reverse-proxy-with-docker-on-centos-8/ How to Install and Use Traefik as a Reverse Proxy with Docker on CentOS 8]
* [https://www.howtoforge.com/how-to-install-and-use-traefik-as-a-reverse-proxy-with-docker-on-centos-8/ How to Install and Use Traefik as a Reverse Proxy with Docker on CentOS 8]
* [https://www.digitalocean.com/community/tutorials/how-to-use-traefik-v2-as-a-reverse-proxy-for-docker-containers-on-ubuntu-20-04 How To Use Traefik v2 as a Reverse Proxy for Docker Containers on Ubuntu 20.04]
* [https://medium.com/accesto/docker-reverse-proxy-using-traefik-2657c0086a3d Docker reverse proxy using Traefik]
* [https://blog.eleven-labs.com/en/using-traefik-as-a-reverse-proxy/ USING TRAEFIK AS A REVERSE PROXY]
* [https://youtu.be/N5EmluXSbEg How to Install Traefik 2 on OMV and Docker] DB Tech 2020. Cloudflare, Portainer. Airsonic.
* [https://youtu.be/liV3c9m_OX8 Put Wildcard Certificates and SSL on EVERYTHING - Traefik Tutorial] Apr 2021
* Raspberry Pi
** [https://www.alexhyett.com/traefik-vs-nginx-docker-raspberry-pi Traefik vs Nginx for Reverse Proxy with Docker on a Raspberry Pi]
** [https://codecaptured.com/blog/self-hosting-pi-hole-with-docker-and-traefik/ Self-Hosting Pi-hole with Docker and Traefik]
** [https://blog.pankajgarg.com/helloaspnetcoreonpi/ Running ASP.NET core web application on Raspberry Pi]
* [https://youtu.be/liV3c9m_OX8 Put Wildcard Certificates and SSL on EVERYTHING - Traefik Tutorial] (video)


== Nginx reverse proxy ==
== Nginx proxy manager ==
* https://nginxproxymanager.com/
* [https://www.the-digital-life.com/nginx-proxy-manager/ NGINX Proxy Manager – How-To Installation and Configuration]
* [https://youtu.be/CSbgLBcIuwE Access your internal websites! Nginx Reverse Proxy in Home Assistant]
* [https://www.howtoforge.com/how-to-install-and-use-nginx-proxy-manager/ How to Install and Use Nginx Proxy Manager with Docker] 2022
* [https://www.the-digital-life.com/nginx-proxy-manager/ NGINX Proxy Manager – How-To Installation and Configuration] the-digital-life
* https://hub.docker.com/r/jlesage/nginx-proxy-manager, [https://github.com/jlesage/docker-nginx-proxy-manager Github]
* [https://dev.to/domysee/setting-up-a-reverse-proxy-with-nginx-and-docker-compose-29jg Setting up a Reverse-Proxy with Nginx and docker-compose]
* [https://dev.to/domysee/setting-up-a-reverse-proxy-with-nginx-and-docker-compose-29jg Setting up a Reverse-Proxy with Nginx and docker-compose]
* [https://linoxide.com/containers/setup-nginx-reverse-proxy-docker/ How to Setup NGINX as Reverse Proxy Using Docker]. It works.
* [https://linoxide.com/containers/setup-nginx-reverse-proxy-docker/ How to Setup NGINX as Reverse Proxy Using Docker]. It works.
Line 1,696: Line 2,065:
docker: Error response from daemon: Cannot link to /site1_app_1, as it does not belong to the default network.
docker: Error response from daemon: Cannot link to /site1_app_1, as it does not belong to the default network.
</pre>
</pre>
* [https://phoenixnap.com/kb/docker-nginx-reverse-proxy How to Deploy NGINX Reverse Proxy on Docker] (multiple web services)
* [https://www.howtoforge.com/how-to-install-and-use-portainer-for-docker-management-with-nginx-proxy-manager/ How to Install and Use Portainer for Docker management with Nginx Proxy Manager]


== Running multiple web applications on a Docker host ==
== Running multiple web applications on a Docker host ==
Line 1,711: Line 2,082:
** Use VIRTUAL_HOST if there are multiple domains associated with a container
** Use VIRTUAL_HOST if there are multiple domains associated with a container
** SSL and et al
** SSL and et al
== Authentication: Authelia ==
[https://www.youtube.com/watch?v=4UKOh3ssQSU Additional Self-Hosted Security with Authelia on NGINX Proxy Manager] (video)


== GUI apps ==
== GUI apps ==
* [https://www.cloudsavvyit.com/10520/how-to-run-gui-applications-in-a-docker-container/ How to Run GUI Applications in a Docker Container]
* https://github.com/mviereck/x11docker/ which was used by [https://github.com/ehough/docker-kodi Docker-kodi]
* https://github.com/mviereck/x11docker/ which was used by [https://github.com/ehough/docker-kodi Docker-kodi]
* https://pelle.io/2014/07/11/delivering-gui-applications-with-docker/ Forward X11 approach
* https://pelle.io/2014/07/11/delivering-gui-applications-with-docker/ Forward X11 approach
Line 1,720: Line 2,095:
* https://github.com/fcwu/docker-ubuntu-vnc-desktop VNC approach
* https://github.com/fcwu/docker-ubuntu-vnc-desktop VNC approach


=== Docker-OSX ===
=== Firefox example ===
https://github.com/sickcodes/Docker-OSX
[https://www.linkedin.com/pulse/running-gui-applications-docker-container-revathi-dhotre Running GUI Applications in Docker Container]
<pre>
From ubuntu:20.04
RUN apt update
RUN apt install firefox -y
RUN apt install python3-pip -y
RUN pip3 install  notebook
 
CMD /usr/bin/firefox
CMD jupyter-notebook --allow-root
</pre>
<pre>
nano Dockerfile
docker build -t gui .
docker run --env="DISPLAY" --net=host --name=firefox gui
</pre>
It works. However, I need to use '''docker rm -f firefox''' to kill it since Ctrl+c does not work.
 
=== Meld example, save a running container as an image ===
[https://www.janert.org/blog/2022/running-a-gui-application-in-a-docker-container1 Running a GUI Application in a Docker Container]. It works. Below is a modified version for creating the '''meld''' app. I can save file modified by ''meld''. To use the app, I need to place files in the ~/Documents/docker (defined in '''-v'''). Note that the RAM usage is very minimal. Unfortunately on macOS, I got an error something related to Gtk.
<pre>
host> docker image pull ubuntu:jammy  # 22.04
host> docker container run --rm --net host -v /tmp/.X11-unix:/tmp/.X11-unix -it ubuntu:jammy
container# apt update
container# apt install -y meld
host> xhost +local:
container# export DISPLAY=:0
 
host> docker container ls  # find the ID of the running container
host> docker commit <ID> meld
container# exit
 
host> docker container run --rm --net host \
  -v /tmp/.X11-unix:/tmp/.X11-unix \
  -v ~/Documents/docker:/meld \
  -e DISPLAY \
  --name=meld \
  meld meld
</pre>
 
=== R and httpgd package ===
<ul>
<li>[https://nx10.github.io/httpgd/articles/b03_docker.html httpgd Docker vignette], [https://nx10.github.io/httpgd/ installation] from Github.
<li>It works. However, currently "httpgs" is archived in CRAN (2023/1/25). So my temporary solution is
<pre>
$ docker run --rm -it r-base:4.2.2 bash
# apt update
# apt install  libfontconfig1-dev
# R
> install.packages("remotes")
> remotes::install_github("nx10/httpgd")
## note if we try to run 'httpgd::hgd(host = "0.0.0.0", port = 8888)', it does not work.
## The reason is we have not use "-p" option to expose a port in the previous "docker run" command
 
## open another terminal and create a docker image based on the current container
$ docker ps -a | head
$ docker commit CONTAINER_ID httpgd:4.2.2
$ docker run --rm -it -p 8888:8888 httpgd:4.2.2 R
> httpgd::hgd(host = "0.0.0.0", port = 8888)
> plot(1:5)
</pre>
<li>It works when I tested it on a '''remote ubuntu server''' (R 4.4.0 & httpgd 2.0.1) (following the instruction on [https://cran.r-project.org/web/packages/httpgd/vignettes/b03_docker.html Docker vignette]). Either IP or hostname works but the hostname URL link given by httpgd::hgd() needs to be modified to include '''.local'''.
<li>To use it with [https://github.com/Bioconductor/bioconductor_docker?tab=readme-ov-file Bioconductor] (the Bioconductor docker image will use p3m.dev to install binary R packages so it is fast to create images), we can do like this
<pre>
$ docker run --rm -it -p 8888:8888 bioconductor/bioconductor_docker:RELEASE_3_18 R
 
> install.packages("httpgd")
> httpgd::hgd(host = "0.0.0.0", port = 8888)
</pre>
OR use, for example, "bioconductor/bioconductor_docker:RELEASE_3_18" as the base image in the Dockerfile, and follow the same instruction from httpgd vignette to create a docker image.
<pre>
$ nano Dockerfile_httpgd
$ docker build . -f Dockerfile_httpgd -t bioc-httpgd:RELEASE_3_18
$ docker images
$ docker run --rm -it --user rstudio -p 8888:8888 bioc-httpgd:RELEASE_3_18 R
</pre>
<li>[[Biowulf#Singularity|Singularity]]. The following is a definition file that is using the bioconductor image + the '''httpgd''' package.
<syntaxhighlight lang='sh'>
Bootstrap: docker
From: bioconductor/bioconductor_docker:RELEASE_3_18
 
%post
    apt-get update \
    && apt-get install -y --no-install-recommends \
    libfontconfig1-dev \
    && apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/* \
    && install2.r --error --skipinstalled --ncpu -1 \
    httpgd \
    && rm -rf /tmp/downloaded_packages
 
%runscript
    exec /usr/local/bin/R
 
%environment
    export LC_ALL=C
</syntaxhighlight>
<syntaxhighlight lang='sh'>
sudo singularity build bioc.sif bioc.def
singularity run bioc.sif
 
> httpgd::hgd(host = "0.0.0.0", port = 8888)
</syntaxhighlight>
After we copy the URL, we need to modify the IP or hostname.
</ul>
 
=== Docker-OSX ===
https://github.com/sickcodes/Docker-OSX


== Pruning unused resources ==
== Pruning unused resources ==
Line 1,751: Line 2,233:
docker system prune
docker system prune
</syntaxhighlight>
</syntaxhighlight>
== Plugins ==
[https://www.cloudsavvyit.com/15066/how-to-manage-docker-engine-plugins/ How to Manage Docker Engine Plugins]


== Misc ==
== Misc ==
Line 1,777: Line 2,262:


=== Date/Time zone ===
=== Date/Time zone ===
* http://tedwise.com/2015/05/02/setting-the-timezone-in-a-docker-image/
* List of time zone. [https://docs.diladele.com/docker/timezones.html Setting Local Time in Docker]; e.g. "America/New_York" or "US/Eastern".
* https://stackoverflow.com/questions/22800624/will-docker-container-auto-sync-time-with-the-host-machine
* https://stackoverflow.com/questions/22800624/will-docker-container-auto-sync-time-with-the-host-machine
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
Line 1,816: Line 2,301:
* https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-storage-driver
* https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-storage-driver
* http://stackoverflow.com/questions/21016210/function-of-docker-directory-structure-in-var-lib-docker
* http://stackoverflow.com/questions/21016210/function-of-docker-directory-structure-in-var-lib-docker
* [https://foxutech.com/manage-docker-images-local-disk/ How to Manage Docker images on local disk]
* [https://www.howtogeek.com/devops/how-to-store-docker-images-and-containers-on-an-external-drive/ How to Store Docker Images and Containers on an External Drive]


The default is '''/var/lib/docker'''. The location can be changed by modifying the file '''/etc/default/docker'''. Three options if we are tight on the disk space.
The default is '''/var/lib/docker'''. The location can be changed by modifying the file '''/etc/default/docker'''. Three options if we are tight on the disk space.
Line 1,912: Line 2,399:
</pre>
</pre>


== Docker Compose <docker-compose.yaml> ==
== Package CLI Applications ==
Docker Compose can help us out as it allows us to specify a single file in which we can define our entire environment structure and run it with a single command (much like a Vagrantfile works).
[https://www.cloudsavvyit.com/15713/how-to-use-docker-to-package-cli-applications/ How to Use Docker to Package CLI Applications]
 
== Stack ==
* https://www.composerize.com/
* [https://youtu.be/-ttZjGBkLL8 Export Docker Container Settings as Docker Compose Stack], [https://github.com/Red5d/docker-autocompose docker-autocompose] (only x86)


== Docker app ==
Docker App is an experimental Docker feature which lets you build and publish application stacks consisting of multiple containers. It aims to let you share '''Docker Compose''' stacks with the same ease of use as regular Docker containers.
[https://www.cloudsavvyit.com/10673/how-to-use-docker-app-to-containerise-an-entire-application-stack/ How to Use 'Docker App' to Containerise an Entire Application Stack]
== Docker Swarm ==
* https://www.linux.com/learn/how-use-docker-machine-create-swarm-cluster
* [https://www.howtoforge.com/tutorial/ubuntu-docker-swarm-cluster/ How Setup and Configure Docker Swarm Cluster on Ubuntu]
* [https://www.cloudsavvyit.com/13049/what-is-docker-swarm-mode-and-when-should-you-use-it/ What is Docker Swarm Mode and When Should You Use It?]
== Security ==
* [https://cloudberry.engineering/article/dockerfile-security-best-practices/ Docker Security Best Practices from the Dockerfile]
* [https://www.cloudsavvyit.com/12631/how-to-secure-sensitive-data-with-docker-compose-secrets/ How to Secure Sensitive Data With Docker Compose Secrets]
== [https://mobyproject.org/ Moby Project] ==
[https://www.infoworld.com/article/3193904/containers/what-is-dockers-moby-project.html What is Docker's Moby Project?]
== Windows container ==
[https://stackoverflow.com/questions/45380972/how-can-i-run-a-docker-windows-container-on-osx How can I run a docker windows container on osx?]
== When Not to Use Docker ==
[https://www.cloudsavvyit.com/15446/when-not-to-use-docker-cases-where-containers-dont-help/ When Not to Use Docker: Cases Where Containers Don’t Help]
= Docker Compose <docker-compose.yaml> =
Docker Compose can help us out as it allows us to specify a single file in which we can define our entire environment structure and run it with a single command (much like a Vagrantfile works).
* Tabs are not allowed in a Docker Compose YAML file. You should use spaces for indentation instead.
* https://docs.docker.com/compose/ (the example will give an error when "RUN pip install -r requirements.txt")
* https://docs.docker.com/compose/ (the example will give an error when "RUN pip install -r requirements.txt")
*# app.py
*# app.py
Line 1,931: Line 2,449:
** Running [https://github.com/nextcloud/docker nextcloud], [https://blog.ouseful.info/2017/06/16/rolling-your-own-jupyter-and-rstudio-data-analysis-environment-around-apache-drill-using-docker-compose/ Jupyter and RStudio]
** Running [https://github.com/nextcloud/docker nextcloud], [https://blog.ouseful.info/2017/06/16/rolling-your-own-jupyter-and-rstudio-data-analysis-environment-around-apache-drill-using-docker-compose/ Jupyter and RStudio]
** [https://github.com/dceoy/docker-rstudio-server Rstudio]
** [https://github.com/dceoy/docker-rstudio-server Rstudio]
* [https://readmedium.com/docker-compose-for-beginners-working-with-multiple-containers-ee0727aab687 Docker Compose For Beginners: Working With Multiple Containers]
** image, container_name
** image, container_name, environment
** image, container_name, environment, volumes, ports


=== An example from 'Fundamentals of Docker' ===
== YAML validator ==
<syntaxhighlight lang='bash'>
https://codebeautify.org/yaml-validator
git clone https://github.com/fundamentalsofdocker/labs.git
cd labs/ch08
docker-compose up
# Open http://localhost:3000/pet
</syntaxhighlight>
The images do not show up:( The terminal shows what has happened under the hood. So the problem is the http links for images do not exist.


We can also run the application in the background
== Download binary ==
<syntaxhighlight lang='bash'>
<ul>
docker-compose up -d
<li>https://github.com/docker/compose/releases for macOS (x86/arm), Linux (aarch64 or armv6 or armv7).
</syntaxhighlight>
<li>New instruction [https://docs.docker.com/compose/install/linux/ Install the Compose plugin]. In short, we will use '''docker compose''' (new) instead of "docker-compose" (deprecated) command. There is no need to install the original "docker-compose" tool. "docker-compose --version"
</ul>


To stop and clean up the application,
== Difference of "docker compose" and "docker-compose" ==
<syntaxhighlight lang='bash'>
* Docker-compose is the original '''Python-based''' command-line tool that was released in 2014. Docker compose is a newer '''Go-based''' command-line tool that is integrated into the Docker CLI platform and supports the compose-spec. Docker compose is meant to be a drop-in replacement for docker-compose, but it may have some behavior differences and new features. Docker compose is currently a tech preview, but it will eventually replace docker-compose as the recommended way to use Compose.
docker-compose down
</syntaxhighlight>


If we also want to remove the volume for the database
* https://docs.docker.com/compose/migrate/. From July 2023 Compose V1 stopped receiving updates. It’s also no longer available in new releases of Docker Desktop.
<syntaxhighlight lang='bash'>
* [https://forums.docker.com/t/docker-compose-vs-docker-compose/137884 Docker-compose vs docker compose]
docker volume rm ch08_pets-data
* [https://stackoverflow.com/a/66516826 Difference between "docker compose" and "docker-compose"]
</syntaxhighlight>


=== An example from "How to Setup NGINX as Reverse Proxy Using Docker" ===
== Simple examples ==
See [[#Nginx_reverse_proxy|here]]. Only nginx is used.
Create a file '''docker-compose.yml''' and run '''docker-compose up''' after creating the file.
 
'''[https://hub.docker.com/_/hello-world hello-world]''': 9kB
<pre>
version: "3"
services:
  hello:
    image: hello-world
</pre>


=== An example from "Docker Deep Dive" (flask + redis) ===
'''[https://hub.docker.com/_/alpine alpine]''': 7.73MB
'''Note''' that on [https://docs.docker.com/compose/gettingstarted/#step-7-update-the-application Get started with Docker Compose] it mounts the current directory to ''/code'' inside the container. So after we modify ''app.py'', we don't need to copy it to the container.
<pre>
version: "3"
services:
  server:
    image: alpine
    container_name: my_container
    command: sh -c "echo 'hello' && echo 'docker'"
</pre>


Another one [https://takacsmark.com/docker-compose-tutorial-beginners-by-example-basics/ Docker compose tutorial for beginners by example]
'''[https://hub.docker.com/_/nginx Nginx]''': 135MB
<syntaxhighlight lang='bash'>
<pre>
$ git clone https://github.com/nigelpoulton/counter-app.git
mkdir src
$ cd counter-app
echo "Hello world!" > src/index.html
$ ls
</pre>
app.py  docker-compose.yml  Dockerfile  README.md  requirements.txt
<pre>
version: "3"
services:
  client:
    image: nginx
    ports:
      - 8000:80
    volumes:
      - ./src:/usr/share/nginx/html
</pre>


$ cat requirements.txt
== PUID, PGID ==
flask
* [https://docs.linuxserver.io/general/understanding-puid-and-pgid Understanding PUID and PGID] (or the [https://github.com/linuxserver/docker-documentation/blob/master/general/understanding-puid-and-pgid.md source])
* You should use the -e PUID and -e PGID options when creating a container from a Docker image to map the container’s internal user to a user on the host machine. This is useful because Docker runs all of its containers under the '''root''' user domain, which means that processes running inside your containers also run as '''root'''. '''This kind of elevated access is not ideal for day-to-day use and can potentially give applications access to things they shouldn’t.''' By using PUID and PGID, you can ensure that files and directories created during the container’s lifespan are owned by a user on the host machine instead of root.
* In this video [https://youtu.be/oHC6J_aN4eQ?t=137 How to Install Calibre on OMV and Docker], it uses the command '''id admin'''  where "admin" is the portainer user to get PUID (of "admin") and PGID (of "users") to find out the two ids.


$ cat Dockerfile
== Composerize/convert a docker command into a docker compose file ==
FROM python:3.4-alpine
* Copilot/ChatGPT/...
ADD . /code
* https://www.composerize.com/
WORKDIR /code
* [https://ostechnix.com/convert-docker-run-commands-into-docker-compose-files/ Convert Docker Run Commands Into Docker-Compose Files]
RUN pip install -r requirements.txt
CMD ["python", "app.py"]


$ cat docker-compose.yml
== An example from 'Fundamentals of Docker' ==
version: "3.5"
<syntaxhighlight lang='bash'>
services:
git clone https://github.com/fundamentalsofdocker/labs.git
  web-fe:
cd labs/ch08
    build: .
docker-compose up
    command: python app.py
# Open http://localhost:3000/pet
    ports:
</syntaxhighlight>
      - target: 5000
The images do not show up:( The terminal shows what has happened under the hood. So the problem is the http links for images do not exist.
        published: 5000
    networks:
      - counter-net
    volumes:
      - type: volume
        source: counter-vol
        target: /code
  redis:
    image: "redis:alpine"
    networks:
      counter-net:


networks:
We can also run the application in the background
  counter-net:
<syntaxhighlight lang='bash'>
docker-compose up -d
</syntaxhighlight>


volumes:
To stop and clean up the application, [https://www.thegeekstuff.com/2016/04/docker-compose-up-stop-rm/ Howto use docker-compose to Start, Stop, Remove Docker Containers]
  counter-vol:
<syntaxhighlight lang='bash'>
docker-compose down # Stop and remove containers, networks, images, and unnamed volumes
                    # defined in the docker-compose.yml flie
# OR
docker-compose down -v # similar to above but remove named volumes defined in yml file
# OR
docker-compose stop && docker-compose rm -f
docker-compose rm -v
</syntaxhighlight>


$ docker-compose up &
If we also want to remove the volume for the database
<syntaxhighlight lang='bash'>
docker volume rm ch08_pets-data
</syntaxhighlight>


$ docker container ls
== An example from "How to Setup NGINX as Reverse Proxy Using Docker" ==
See [[#Nginx_reverse_proxy|here]]. Only nginx is used.


$ docker network ls
== An example from "Docker Deep Dive" (flask + redis) ==
NETWORK ID          NAME                    DRIVER              SCOPE
'''Note''' that on [https://docs.docker.com/compose/gettingstarted/#step-7-update-the-application Get started with Docker Compose] it mounts the current directory to ''/code'' inside the container. So after we modify ''app.py'', we don't need to copy it to the container.
2acef6dabde6        bridge                  bridge              local
a2d42bc482ff        counterapp_counter-net  bridge              local
e1e093b64282        host                    host                local
7ecd0a6a9ebd        none                    null                local


# Open the browser http://localhost:5000
Another one [https://takacsmark.com/docker-compose-tutorial-beginners-by-example-basics/ Docker compose tutorial for beginners by example]
$ docker-compose ps
<syntaxhighlight lang='bash'>
      Name                      Command              State          Ports       
$ git clone https://github.com/nigelpoulton/counter-app.git
-------------------------------------------------------------------------------------
$ cd counter-app
counterapp_redis_1    docker-entrypoint.sh redis ...  Up      6379/tcp             
$ ls
counterapp_web-fe_1  python app.py                   Up      0.0.0.0:5000->5000/tcp
app.py docker-compose.yml  Dockerfile  README.md  requirements.txt


$ docker-compose stop
$ cat requirements.txt
$ docker-compose ps
flask
# We can see stopping a Compose app does not delete the application


$ docker container ls -a
$ cat Dockerfile
$ docker-compose rm    # delete a stopped Compose app
FROM python:3.4-alpine
                        # images, volumes and source code remain
ADD . /code
$ docker-compose restart
WORKDIR /code
                        # If you made changes to your Compose app since stopping,
RUN pip install -r requirements.txt
                        # these changes will not appear in the restarted app.
CMD ["python", "app.py"]
                        # You need to re-deploy the app to get the changes.
$ docker-compose ps
$ docker-compose down  # stop and delete the app
                        # images, volumes and source code remain
$ docker-compose down --volumes # remove the data volume used by the Redis container
$ docker-compose up -d
$ docker volume ls
$ docker-compose


# We can make changes to files in the volume, from the host side,
$ cat docker-compose.yml
# and have them reflected immediately in the app.
version: "3.5"
$ nano app.py   # do some changes
services:
$ docker volume inspect counterapp_counter-vol | grep Mount
  web-fe:
$ sudo cp app.py \
    build: .
  /var/lib/docker/volumes/counterapp_counter-vol/_data/app.py
    command: python app.py
# Our changes should be reflected
    ports:
      - target: 5000
        published: 5000
    networks:
      - counter-net
    volumes:
      - type: volume
        source: counter-vol
        target: /code
  redis:
    image: "redis:alpine"
    networks:
      counter-net:


$ docker-compose --help
networks:
</syntaxhighlight>
  counter-net:


volumes:
  counter-vol:


=== Docker-Compose persistent data MySQL ===
$ docker-compose up &
https://stackoverflow.com/questions/39175194/docker-compose-persistent-data-mysql


=== Connect to Docker daemon over ssh using docker-compose ===
$ docker container ls
[https://medium.com/@sujaypillai/dockertips-connect-to-docker-daemon-over-ssh-using-docker-compose-f4b189dd8951 #DockerTips: Connect to Docker daemon over ssh using docker-compose]


=== Dockerfile + docker-compose ===
$ docker network ls
[https://stackoverflow.com/a/29487120 Docker Compose vs. Dockerfile - which is better?]
NETWORK ID          NAME                    DRIVER              SCOPE
2acef6dabde6        bridge                  bridge              local
a2d42bc482ff        counterapp_counter-net  bridge              local
e1e093b64282        host                    host                local
7ecd0a6a9ebd        none                    null                local


The Compose file describes the container in its running state, leaving the details on how to build the container to Dockerfiles.
# Open the browser http://localhost:5000
 
$ docker-compose ps
== Docker Swarm ==
      Name                      Command              State          Ports       
* https://www.linux.com/learn/how-use-docker-machine-create-swarm-cluster
-------------------------------------------------------------------------------------
* [https://www.howtoforge.com/tutorial/ubuntu-docker-swarm-cluster/ How Setup and Configure Docker Swarm Cluster on Ubuntu]
counterapp_redis_1    docker-entrypoint.sh redis ...   Up      6379/tcp             
counterapp_web-fe_1  python app.py                    Up      0.0.0.0:5000->5000/tcp


== [https://mobyproject.org/ Moby Project] ==
$ docker-compose stop
[https://www.infoworld.com/article/3193904/containers/what-is-dockers-moby-project.html What is Docker's Moby Project?]
$ docker-compose ps
# We can see stopping a Compose app does not delete the application


= Applications =
$ docker container ls -a
Some popular Linux applications such as: owncloud, lamp, openstack, node.js, roundcube, DLNA/Server, madsonic, webproxy/webfilter, DHCP/DNS-Server like dnsmasq, cloudprint, ...
$ docker-compose rm    # delete a stopped Compose app
                        # images, volumes and source code remain
$ docker-compose restart
                        # If you made changes to your Compose app since stopping,
                        # these changes will not appear in the restarted app.
                        # You need to re-deploy the app to get the changes.
$ docker-compose ps
$ docker-compose down  # stop and delete the app
                        # images, volumes and source code remain
$ docker-compose down --volumes # remove the data volume used by the Redis container
$ docker-compose up -d
$ docker volume ls
$ docker-compose


* [https://kuonsam.miraheze.org/wiki/Setting_Up_a_Docker_Server#Install_Docker-compose Setting Up a Docker Server] (host OS is CentOS)
# We can make changes to files in the volume, from the host side,
# and have them reflected immediately in the app.
$ nano app.py  # do some changes
$ docker volume inspect counterapp_counter-vol | grep Mount
$ sudo cp app.py \
  /var/lib/docker/volumes/counterapp_counter-vol/_data/app.py
# Our changes should be reflected


== Dockerizing Compiled Software ==
$ docker-compose --help
https://ram.tianon.xyz/post/2017/12/26/dockerize-compiled-software.html
</syntaxhighlight>


== Raspberry Pi servers based on Docker ==
== Create Compose Files From Running Docker Containers ==
[https://youtu.be/a6mjt8tWUws Raspberry Pi Server based on Docker, with VPN, Dropbox backup, Influx, Grafana, etc.]
[https://www.makeuseof.com/create-docker-compose-files-from-running-docker-containers/ How to Automatically Create Compose Files From Running Docker Containers]


== Data Science ==
== Docker-Compose persistent data MySQL ==
[[Data_science#Container|Data Science]]
https://stackoverflow.com/questions/39175194/docker-compose-persistent-data-mysql


== Illumina BaseSpace ==
== Connect to Docker daemon over ssh using docker-compose ==
* http://blog.basespace.illumina.com/2013/10/21/introducing-the-native-app-engine/
[https://medium.com/@sujaypillai/dockertips-connect-to-docker-daemon-over-ssh-using-docker-compose-f4b189dd8951 #DockerTips: Connect to Docker daemon over ssh using docker-compose]
* https://developer.basespace.illumina.com/docs/content/documentation/getting-started/overview
* https://developer.basespace.illumina.com/docs/content/documentation/native-apps/manage-docker-image


== Bioinformatics analyses ==
== Dockerfile + docker-compose ==
* [https://www.biorxiv.org/content/early/2017/10/17/204495 Hot-starting software containers for bioinformatics analyses]
[https://stackoverflow.com/a/29487120 Docker Compose vs. Dockerfile - which is better?]  
* [https://academic.oup.com/gigascience/article/6/8/1/3894236 Bio-Docklets: virtualization containers for single-step execution of NGS pipelines]


== Cloud provider ==
The Compose file describes the container in its running state, leaving the details on how to build the container to Dockerfiles.
Docker can be deployed on any cloud provider like Digital Ocean, Amazon Web Services (AWS), Microsoft Azure, Google Cloud Computing.


* https://www.digitalocean.com/community/tutorials/how-to-use-the-digitalocean-docker-application
== How to deploy on remote Docker hosts with docker-compose ==
* http://go.mkb.io/post/118463195802/docker-machine-with-digital-ocean
[https://www.docker.com/blog/how-to-deploy-on-remote-docker-hosts-with-docker-compose/ How to deploy on remote Docker hosts with docker-compose]
* http://networkstatic.net/running-docker-machine-on-digital-ocean/
* http://linoxide.com/linux-how-to/use-docker-machine-cloud-provider/
* [http://www.ithome.com.tw/news/106274 HTC採用AWS的三大原因]: 上市時間縮短、節省成本以及提供創新發想的低成本空間。


== ownCloud ==
== logs ==
* https://hub.docker.com/_/owncloud/
<pre>
* https://doc.owncloud.com/server/10.0/admin_manual/installation/docker/
docker-compose logs -f
* https://blog.securem.eu/serverside/2015/08/25/setting-up-owncloud-server-in-a-docker-container/
# Ctrl + c
* [https://stackoverflow.com/questions/35137944/how-to-put-owncloud-behind-nginx-with-docker How to put ownCloud behind Nginx with Docker?]
</pre>
* [https://www.blaize.net/2017/02/migrating-owncloud-to-docker/ Migrating ownCloud to Docker]
* "You are accessing the server from an untrusted domain" message when I use IP to access the server
** trusted_domains (config.php)
** https://forum.owncloud.org/viewtopic.php?t=34158
** https://blog.securem.eu/serverside/2015/08/25/setting-up-owncloud-server-in-a-docker-container/
** [https://youtu.be/KUZu5XG6cjI Fix ownCloud accessing the server from an untrusted domain]
** '''My simple fix''': use the IP instead of localhost when opening the browser to set up the first time


After I use '''docker run --rm --name myowncloud -p 81:80 owncloud:8.1''', I find I can only access it through http://localhost:81. If I try to access it through another computer by http://xxx.xxx.xx.1:81, the URL is changed back to http://localhost:81 so it failed to load.
= GUI/TUI interface manager =
== [https://github.com/moncho/dry Dry] ==
[https://www.2daygeek.com/dry-an-interactive-cli-manager-for-docker-containers/ Dry – An Interactive CLI Manager For Docker Containers]. The TUI is built on top of [https://github.com/gizak/termui termui]; a cross-platform, easy-to-compile, and fully-customizable terminal dashboard. It is inspired by [https://github.com/yaronn/blessed-contrib blessed-contrib], but purely in Go.


== nextcloud ==
== LazyDocker (TUI) ==
* https://hub.docker.com/_/nextcloud/
* [https://github.com/jesseduffield/lazydocker Github]
* https://github.com/nextcloud/docker
* [https://www.linuxuprising.com/2019/07/lazydocker-new-docker-and-docker.html?m=1 LazyDocker: New Docker And Docker Compose Terminal UI]
* [https://apps.nextcloud.com/ App store]
* It works on ARM devices too.
* [https://doc.owncloud.org/server/8.1/admin_manual/installation/apps_management_installation.html Installing and Managing Apps]
* Video by [https://youtu.be/xb99lgO5Mw0?t=1083 Awesome open source]
* Based on [https://github.com/owncloud owncloud's github repositories] or [https://github.com/nextcloud nextcloud's github repositories], android app has been updated more often than ios app.


I feel nextcloud is a little modern than owncloud though their interface are very similar. I got the same problem (URL is changed from IP to localhost) when I try to access its web from another computer. '''Using the IP instead of localhost to finish the initial setup can fix the error'''.
== Dockly (TUI) ==
[https://www.ostechnix.com/dockly-manage-docker-containers-from-terminal/ Dockly – Manage Docker Containers From Terminal]


Note:
== DockStation ==
* The first we open the webpage, we need to create the admin username/password.
It is not open source. It works with remote Docker containers.
* Nextcloud uses owncloud mobile application.
* Nextcloud includes a small mp4 file. I can use browser to play mp4. Photos files can be opened in browsers too (including slideshow).


=== [https://apps.nextcloud.com/ Nextcloud Apps] ===
* [https://github.com/DockStation/dockstation DockStation]
For mp3, no apps are shipped with nextcloud.  
* [https://www.upnxtblog.com/index.php/2018/01/17/top-6-gui-tools-for-managing-docker-environments/ TOP 6 GUI tools for managing Docker environments] 2018
* [https://www.how2shout.com/tools/4-best-docker-gui-tools-to-manage-containers-graphically.html 4 Best Docker GUI tools to manage containers graphically] 2019


* [https://apps.nextcloud.com/apps/audioplayer Audio Player]. When I play mp3 files, the web interface does not show a progress bar:( <syntaxhighlight lang='bash'>
== DockSTARTer: get started with home server apps running in Docker ==
$ docker run --rm --name mynextcloud -v ~/Downloads/nextcloudapps:/var/www/html/custom_apps -p 81:80 nextcloud
* https://dockstarter.com/
# above will create an empty folder ~/Downloads/nextcloudapps (www-data:root)
* [https://www.makeuseof.com/tag/setup-linux-media-server/ How to Set Up a Linux Media Server in Under One Hour]
$ tar xzvf Downloads/audioplayer-2.3.1.tar.gz -C Downloads/
$ docker cp Downloads/audioplayer mynextcloud:/var/www/html/custom_apps
# Go to nextcloud and '+Apps', a new app 'Audio Player' should have shown up.
</syntaxhighlight>
* [https://apps.nextcloud.com/apps/music Music]. GOOD. The app is from the owncloud team though for some reason the app is not included in owncloud/nextcloud. Yes, a progress bar is shown when I played a song downloaded from Youtube:) <syntaxhighlight lang='bash'>
# No need to create a volume
$ docker run -d --name mycloud -p 81:80 nextcloud


# Open another terminal
== [https://portainer.io/ Portainer]* (nice) ==
# Don't download the zip file from github. Get the zip file from the app store
$ tar xzvf ~/Downloads/music_0.7.0_nc-signed.tar.gz -C ~/Downloads/
$ docker cp Downloads/music mycloud:/var/www/html/custom_apps
$ docker exec mycloud chmod -R 0750 /var/www/html/custom_apps
$ docker exec mycloud chown -R www-data:www-data /var/www/html/custom_apps
</syntaxhighlight> Now open the website. Click the gear icon (Settings) and click "Apps". Find out the 'music' app from the bottom of the list and click the 'Enable' button. A new Music icon will be created next to the Gallery icon at the top. Note: if we want to play music repeatedly, I should not play the music directly from Files. I should click the Music icon at the top and let it start to scan musics. Then once I can play a music from there, it will show two new icons; one is shuffle and the other is loop. To have a persistent data, we can 'run' using <syntaxhighlight lang='bash'>
$ mkdir -p ~/nextcloud/{html,apps,config,data}
$ docker run -d \
    --name mycloud \
    -p 81:80 \
    -v ~/nextcloud/html:/var/www/html \
    -v ~/nextcloud/apps:/var/www/html/custom_apps \
    -v ~/nextcloud/config:/var/www/html/config \
    -v ~/nextcloud/data:/var/www/html/data \
    nextcloud
</syntaxhighlight> If we like to delete the contain and its unamed volume, run <syntaxhighlight lang='bash'>
$ docker stop mycloud; docker rm -v mycloud
</syntaxhighlight> If we run the container again mounting all volumes, all apps,files,configuration,admin's username & password are kept.
* '''PDF viewer'''. PDF files will be downloaded instead of opening on the browser. This can be easily fixed. Click '+Apps' from the personal settings (the response is slow when I tested using Docker). Search for '''PDF viewer'''. Click the 'Enable' button (it will be changed to 'Disable' after that)
* [https://apps.nextcloud.com/apps/drawio draw.io]. After enable it, we can click the "+" sign and choose "Diagram" to create a new diagram (*.xml).
* [https://apps.nextcloud.com/apps/radio Radio]. After enable it, a new "Radio" icon will be shown at the top. Only 20 stations are available from the 'Top'. No response after I clicked "Categories".
 
== Use with R (r-base) & RStudio IDE: Rocker ==
<ul>
<ul>
<li>[https://registry.hub.docker.com/_/r-base/ r-base] (Official image, R version is tagged)
<li>Password: at least 12 characters long</li>
<ul>
<li>[https://github.com/portainer/portainer/issues/5406 Hardware minimum requirements] 100MB RAM. That's why 1GB ram of Raspberry Pi works fine.
<li>[https://github.com/docker-library/docs Github] repository of the Official Docker Images.</li>
<li>[https://docs.portainer.io/start/install-ce/server/docker Install CE], [https://docs.portainer.io/ Documentation]
<li>[https://colinfay.me/docker-r-reproducibility/ An Introduction to Docker for R Users]: how to write your own <Dockerfile>, install packages, run a script and get results.</li>
<pre>
<li>Extensions from r-base. For example, [https://github.com/rocker-org/rocker/issues/119 r-spatial-base]. It also mentions [https://registry.hub.docker.com/u/rocker/ropensci/ ropensci] container is built upon rocker/rstudio. </li>
docker volume create portainer_data
<li>The r-base image does not have pdflatex, git. Need to manually install them.</li>
docker run -d -p 8000:8000 -p 9443:9443 \
<li>Not sure if the Docker Official Image is the same as the one provided by Rocker Project.</li>
  --name portainer \
<li>NOTE: Plotting works by forwarding X11. See rocker [https://github.com/rocker-org/rocker/wiki/Allowing-GUI-windows Wiki] or the command below. Creating graphics files inside a container is still OK 👌; see the example [https://jlintusaari.github.io/2018/07/how-to-compile-rmarkdown-documents-using-docker/ How to compile R Markdown documents using Docker].
  --restart=always \
{{Pre}}
  -v /var/run/docker.sock:/var/run/docker.sock \
docker pull r-base:3.5.3
  -v portainer_data:/data portainer/portainer-ce:latest
docker run -it --rm rocker/drd RD              # a little smaller, 3.6GB for R 4.0
docker run -it --rm rocker/drd R              # good to test the pipe operator (due in R 4.1.0)
docker run -it --rm rocker/r-devel RD          # initial one, larger, 5.7GB for R 4.0
docker run -it --rm rocker/r-devel R          # r-release
docker run -it --rm r-base:3.5.3              # default is root "/"
docker run -v ~/Downloads:/src -it --rm r-base # /src does not exist
docker run -v ~/Downloads:/home/docker -it --rm r-base # /home/docker exists and is empty
docker run -it --rm -u1000:1000 -e DISPLAY=unix$DISPLAY \
          -v /tmp/.X11-unix:/tmp/.X11-unix \
          -v $(pwd):/work -w /work r-base
</pre>
</pre>
By default, Portainer Server will expose the UI over port 9443 and expose a TCP tunnel server over port 8000. The latter is optional and is only required if you plan to use the Edge compute features with Edge agents.
</li>
</li>
</ul>
<li>Access https://localhost:9443 </li>
<li>Portainer itself is a docker application. </li>
<li>[https://docs.portainer.io/advanced/reset-admin Reset the admin user's password].</li>
<li>[https://youtu.be/M365jgJ0O2E How to Update Portainer Fast, Simple, and Easy Guide]. docker stop + docker rm + docker rmi + docker run.
</li>
</li>
</ul>
<li>[https://dbtechreviews.com/2020/08/update-portainer-to-version-2-0-super-easy/ Update Portainer to Version 2.0 – Super Easy!]. When the first time accessing http://localhost:9000, you need to create a password (at least <s>8</s> 12 characters long) for the default user "admin".
* Dirk.eddelbuettel
{{Pre}}
** slides http://dirk.eddelbuettel.com/presentations/
$ docker volume create portainer_data
** http://dirk.eddelbuettel.com/papers/useR2014_keynote.pdf
$ docker run -d -p 9000:9000 -p 8000:8000 --name portainer --restart always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce
** http://dirk.eddelbuettel.com/papers/r_and_docker_sep2014.pdf
</pre>
** [http://dirk.eddelbuettel.com/blog/2014/10/23/ Introducing Rocker: Docker for R]
* [https://youtu.be/2oi4IQF7VnE Raspberry Pi Home Server Episode 5: Remote Access with NGINX Proxy Manager]
** [http://dirk.eddelbuettel.com/blog/2019/08/05/#023_rocker_debug_example Debugging with Docker and Rocker – A Concrete Example helping on macOS]
* [https://youtu.be/ji0kgXhXR6g Raspberry Pi Home Server Episode 8: BitWardenRS - Password Manager with Remote Access] (video)
** [https://arxiv.org/abs/2001.10641 The Rockerverse: Packages and Applications for Containerization with R]
* [https://dbtechreviews.com/2020/07/install-nextcloud-on-a-raspberry-pi-4-with-openmediavault-and-docker/ Install NextCloud on a Raspberry Pi 4 with OpenMediaVault and Docker]
** [https://github.com/rocker-org/rocker/tree/master/r-bspm #29: Easy, Reliable, Fast Linux CRAN Binaries via BSPM]
</li>
* [https://www.rocker-project.org/ The Rocker Project] (<strike>only has the latest version</strike>)
<li>Update from 2.11 to 2.13 (The password must be at least 12 characters long)
** [https://github.com/rocker-org/rocker Github] repository.
<ul>
** [https://github.com/rocker-org/rocker-versioned/blob/master/VERSIONS.md R, MRAN, RStudio Server versions correspondance]
<li>Step 1: [https://dbtechreviews.com/2021/08/how-to-update-portainer-to-the-newest-version/ How to Update Portainer to the Newest Version]
** [https://github.com/rocker-org/rocker-versioned/blob/master/r-dependencies.md What dependencies are included in included in r-ver?]
* [https://github.com/rocker-org/rocker/wiki/Using-the-RStudio-image Using the RStudio image] includes the official instruction for the latest change. PS the default username is '''rstudio'''. <syntaxhighlight lang='bash'>
docker run -d -p 8787:8787 \
  -e PASSWORD=<password> \
  --name rstudio rocker/rstudio
# mount a local directory, change the username. A new directory /src will be created.
docker run -d -p 8787:8787 \
  -v ~/Downloads:/src \
  -e USER=<username>
  -e PASSWORD=<password> \
  --name rstudio rocker/rstudio
 
# Use R, without RStudio
docker run -it -e DISABLE_AUTH=true rocker/rstudio:3.5.3 R
# Use Bash without RStudio, mount the current directory
docker run -it -e DISABLE_AUTH=true -v $(pwd):/home/rstudio/project rocker/rstudio:3.5.3 bash
 
# Fix an error when running library(tidyverse): libxml2.so.2 no such file
$ docker exec -it rstudio bash
# apt update
# apt upgrade
# apt install libcurl4-openssl-dev libssl-dev libxml2-dev
</syntaxhighlight>
** [https://github.com/rocker-org/rocker/wiki/Sharing-files-with-host-machine Sharing files with host machine]
** [http://www.magesblog.com/2014/09/running-rstudio-via-docker-in-cloud.html Running RStudio in cloud]
** [https://sbamin.com/blog/2016/02/running_rstudio_in_docker_environment/ Running RStudio in a docker container]
* [http://ropenscilabs.github.io/r-docker-tutorial/ R Docker tutorial] from ropenscilabs. It covers sharing your analysis.
 
=== Dockerfile ===
Create a new directory and a new file 'Dockerfile' with the content.
 
<pre>
<pre>
FROM debian:testing
sudo docker stop portainer && sudo docker rm portainer
MAINTAINER Dirk Eddelbuettel [email protected]
docker pull portainer/portainer-ce:latest
## Remain current
RUN apt-get update -qq
RUN apt-get dist-upgrade -y
RUN apt-get install -y --no-install-recommends r-base r-base-dev r-recommended littler
RUN ln -s /usr/share/doc/littler/examples/install.r /usr/local/bin/install.r
</pre>
</pre>
 
</li>
NOTE
<li>Step 2: [https://docs.portainer.io/start/upgrade/docker Upgrading on Docker Standalone]
# From [https://hub.docker.com/_/r-base r-base] in DockerHub click the "latest" in the "Supported tags and respective Dockerfile links" section.
{{Pre}}
# I ran into errors when I use the above (short) Dockerfile. But the Dockerfile from rocker (leave out the last line of launching R) works well. The R packages built in the image include 'docopt', 'magrittr', 'stringi', and 'stringr'.
docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest
# Install R package is possible when we launch a container. But we are not able to save the packages?? The [https://github.com/rocker-org/rocker/wiki rocker wiki] also mentions something about installing packages.
# See also [https://github.com/rocker-org/rocker/wiki/How-to-save-data How to save data] in wiki.
 
=== A quick run of an R script ===
<pre>
docker run --rm \
  -v $(pwd):/tmp/working_dir \
  -w /tmp/working_dir \
  rocker/tidyverse:latest \
  Rscript my_script.R
</pre>
</pre>
</li>
</ul>
</ul>
* [https://github.com/portainer/portainer-docs/issues/91 Why port 8000 is needed to be published?], [https://forums.portainer.io/t/what-port-8000-used-for-do-i-need-to-expose-it/210 What port 8000 used for? Do I need to expose it?]
* [https://dzone.com/articles/a-comparison-of-docker-guis A Comparison of Docker GUIs]. A username/password is required to access the page unless we use the '''--no-auth''' option; see [https://portainer.readthedocs.io/en/stable/configuration.html Configuration].
* [https://www.howtoforge.com/tutorial/ubuntu-docker-portainer/ How to Manage Docker Containers using Portainer on Ubuntu 16.04]. Go to Local. It will show the status of all local stacks, containers, volumes and images. Very Cool!
* [https://www.linuxtechi.com/monitor-manage-docker-containers-portainer-part1/ Monitor and Manage Docker Containers with Portainer.io (GUI tool) – Part-1] 2019
* [https://www.cloudsavvyit.com/8911/how-to-get-started-with-portainer-a-web-ui-for-docker/ How to Get Started With Portainer, a Web UI for Docker]
* [https://www.linuxtechi.com/monitor-manage-docker-containers-portainer-io-part-2/ Monitor and Manage Docker Containers with Portainer.io (GUI tool) – Part-2]
* [https://collabnix.com/portainer-vs-rancher/ Portainer Vs Rancher]
* [https://youtu.be/XYcKmPi4McA How-to: Deploy Portainer on MicroK8s] (video)
* [https://www.portainer.io/blog/from-zero-to-production-with-fedora-coreos-portainer-and-wordpress-in-7-easy-steps From Zero to Production with Fedora CoreOS, Portainer, and WordPress in 7 Easy Steps]. Virtualbox was used


=== docker run ===
=== IP address 0.0.0.0 ===
Note that if we are using the Dockerfile above to create an image, we will be dropped to the Linux shell. If we are pulling the rocker/r-base image from Docker Hub, we will be in R console directly. See the last line of Dockerfile on [https://github.com/rocker-org/rocker/blob/master/r-base/Dockerfile github] website for rocker.
[https://www.reddit.com/r/docker/comments/rkq7o8/comment/hpbb13k/ How to setup ip address in portainer to access containers?] Environments > local (or whatever your environment is named) there you set your public ip.


Then run the following to do some exercise (We could possibly replace 21b6a9e8b9e8 with your image ID or use ''rocker/r-base''). For simplicity, we can try the [http://cran.rstudio.com/web/packages/colortools/index.html colortools package] first which does not depend on other packages and there is no need to compile the package.
=== Templates ===
<syntaxhighlight lang='bash'>
* [https://dbtechreviews.com/2020/09/using-application-templates-in-portainer-2-0/ Using Application Templates in Portainer 2.0]
sudo docker build -t debian:testing-add-r . # create an image based on the above Dockerfile
* [https://youtu.be/BwgKd6LohQo Install 80+ Docker Images In Seconds Using This Portainer App Template] (video)
wget http://cran.r-project.org/src/contrib/sanitizers_0.1.0.tar.gz
* [https://www.reddit.com/r/portainer/comments/v3uyo3/best_and_most_uptodate_portainer_app_template_for/?utm_source=share&utm_medium=web2x&context=3 Best and most up-to-date Portainer app template for selfhosting?] 2022-06-03
sudo docker run -v `pwd`:/mytmp -t 21b6a9e8b9e8 \
** [https://github.com/mycroftwilde/portainer_templates mycroftwilde]
    R CMD check --no-manual --no-build-vignettes /mytmp/sanitizers_0.1.0.tar.gz
* [https://youtu.be/BwgKd6LohQo Install 80+ Docker Images In Seconds Using This Portainer App Template] (video, 2020)
sudo docker run -v `pwd`:/mytmp -t 21b6a9e8b9e8 \
** Default on my Portainer 2.0.1 (see logs for version number) https://raw.githubusercontent.com/portainer/templates/master/templates-2.0.json
    Rdevel CMD check --no-manual --no-build-vignettes /mytmp/sanitizers_0.1.0.tar.gz
** https://raw.githubusercontent.com/SelfhostedPro/selfhosted_templates/portainer-2.0/Template/template.json (working), [https://github.com/SelfhostedPro/selfhosted_templates Github]
** https://raw.githubusercontent.com/nashosted/self-hosted-template/master/template.json (working)
** Alternative: [https://github.com/Taisun-Docker/taisun taisun]


sudo docker search eddelbuettel
== Yacht ==
sudo docker pull eddelbuettel/docker-ubuntu-r  # default tag is 'latest'; actually older than the other tags
* https://yacht.sh/Installation/yacht/
sudo docker images eddelbuettel/docker-ubuntu-r # see the tag column
* Yacht's [https://yacht.sh/docs/Pages/Dashboard Dashboard] page shows the resources used by each docker application (it seems Portainer can't)
sudo docker pull eddelbuettel/docker-ubuntu-r:add-r # the tag name can only be obtained from hub.docker.com
* [https://youtu.be/KstIS5DstTE Yacht Installed on Docker - Portainer Alternative] (video)
sudo docker images eddelbuettel/docker-ubuntu-r # see the tag column
* [https://youtu.be/c4rKWrH88F0?t=166 What's on my Home Server? MUST HAVE Services] (video)
sudo docker pull eddelbuettel/docker-ubuntu-r:add-r-devel
sudo docker images eddelbuettel/docker-ubuntu-r # see the tag column
sudo docker run -v `pwd`:/mytmp -t 54d865dbd2c9 R CMD check --no-manual --no-build-vignettes /mytmp/sanitizers_0.1.0.tar.gz


sudo docker run -t -i eddelbuettel/docker-ubuntu-r /bin/bash
== cockpit-docker ==
$ sudo docker images
* [https://www.hostwinds.com/guide/installing-and-using-cockpit-docker-linux-vps/ Installing And Using Cockpit-Docker]
REPOSITORY                    TAG                IMAGE ID            CREATED            VIRTUAL SIZE
* The default port is 9090.  
debian                        testing-add-r      21b6a9e8b9e8        28 minutes ago      572.2 MB
* Not useful for manager docker containers.
ubuntu                        14.04              ed5a78b7b42b        4 days ago          188.3 MB
<pre>
ubuntu                        latest              ed5a78b7b42b        4 days ago          188.3 MB
sudo apt-get -y install cockpit-docker
debian                        testing            88ba2870bfbe        7 weeks ago        154.7 MB
eddelbuettel/docker-ubuntu-r  add-r-devel        c998a74a1fb4        11 weeks ago        460.4 MB
eddelbuettel/docker-ubuntu-r  add-r              54d865dbd2c9        11 weeks ago        460.4 MB
eddelbuettel/docker-ubuntu-r  latest              a7cd5ddeb98e        5 months ago        515.4 MB


sudo docker logs xxxxx                # view the log
sudo systemctl restart cockpit
sudo docker restart xxxxx
</pre>
sudo docker exec -it xxxx /bin/bash  # view any changes in R library
sudo docker stop xxxxx
sudo docker rm xxxxx
</syntaxhighlight>


[https://hub.docker.com/r/dennishazelett/motifbreakr/ This] is another example of using 'docker run' accompanying [https://github.com/Simon-Coetzee/MotifBreakR MotifBreakR] package.
== [https://github.com/crosbymichael/dockerui DockerUI] (Deprecated, Development continues at Portainer) ==
https://github.com/kevana/ui-for-docker. A quick start:
# Run:  <syntaxhighlight lang='bash'>
docker run -d -p 9000:9000 --privileged \
    -v /var/run/docker.sock:/var/run/docker.sock uifd/ui-for-docker
</syntaxhighlight>  where '''-v''' means to bind mount a volume.
# Open your browser to http://<dockerd host ip>:9000


=== Testing R packages ===
Note: Anyone in the local network can access the website without any authentication.
* [http://cran.rstudio.com/web/packages/RColorBrewer/index.html RColorBrewer] (pure R)
* [http://cran.rstudio.com/web/packages/digest/index.html digtest] (need compile, with vignette) [https://github.com/eddelbuettel/digest github source]
* [http://cran.rstudio.com/web/packages/randomForest/index.html randomForest] (need compile & has suggests RColorBrewer)


=== Docker for R Package Development ===
== Rancher ==
http://www.jimhester.com/2017/10/13/docker/
* https://rancher.com/quick-start/. The doc above is not good enough to follow. Search Google for tutorial or get started guide including screenshots.
* [https://www.howtoforge.com/tutorial/ubuntu-rancher-docker-container-manager/ How to Manage Docker Containers with Rancher on Ubuntu 16.04]
* [https://hackernoon.com/creating-a-scalable-application-using-docker-rancher-rancheros-cattle-5feb83a3e743 Creating A Scalable Application Using Docker + Rancher + RancherOs + Cattle]
* Initial setup video https://youtu.be/1EMW1y7529Y
* Training from Rancher Labs https://youtu.be/8K14A_CZFdI. It seems Rancher is complicated but powerful. The whole operation is like Amazon Cloud.
* [https://gist.github.com/lmmendes/fbed32a452cf02d2a1095658795cb3d2 Running Rancher locally (with two hosts) using MacOS Docker and VirtualBox]
* To allow UDP port
<syntaxhighlight lang='bash'>
$ sudo apt-get install ufw
$ sudo ufw allow 4500/udp
$ sudo ufw allow 500/udp
</syntaxhighlight>
* discoposse.com
** [http://discoposse.com/2015/11/08/rancher-part-1-installing-rancher-and-setting-access-control/ Part 1] Installing Rancher and Setting Access Control
** [http://discoposse.com/2015/11/09/rancher-part-2-adding-a-docker-host-to-rancher/ Part 2] Adding a Docker Host to Rancher
** [http://discoposse.com/2015/11/12/rancher-part-3-adding-the-dockerhub-to-our-rancher-registry/ Part 3] Adding the DockerHub to our Rancher Registry
** [http://discoposse.com/2015/11/13/rancher-part-4-using-the-catalog-example-with-glusterfs/ Part 4] Using the Catalog Example with GlusterFS


=== A DevOps Perspective ===
== [https://github.com/tobegit3hub/seagull Seagull] ==
* https://www.opencpu.org/posts/opencpu-with-docker/
https://youtu.be/TuT5gb8oRw8


=== Reproducible ===
* [https://journal.r-project.org/archive/2020/RJ-2020-007/RJ-2020-007.pdf The Rockerverse: Packages and Applications for Containerisation with R] 2020
** [https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1008316 Ten Simple Rules for Writing Dockerfiles for Reproducible Data Science], [https://github.com/nuest/ten-simple-rules-dockerfiles code in github]
** [https://www.theoj.org/joss-papers/joss.01603/10.21105.joss.01603.pdf containerit: Generating Dockerfiles for reproducible research with R], [https://o2r.info/containerit/ containerit] package with GUI via [https://rstudio.github.io/rstudioaddins/ RStudio Addins].
<ul>
<li>A simple example to produce a PDF report from an Rmarkdown file by using [https://hub.docker.com/r/rocker/verse rocker/verse] image:
<pre>
<pre>
docker run -it --rm --name pdmr -v $(pwd):/home/docker -w /home/docker \
docker run -d -p 127.0.0.1:10086:10086 -v /var/run/docker.sock:/var/run/docker.sock tobegit3hub/seagull
      --user rstudio rocker/verse:4.0.3 \
      Rscript -e "rmarkdown::render('Example.Rmd')"
</pre>
</li>
<li>[https://datawookie.netlify.com/blog/2019/08/r-docker-and-checkpoint-a-route-to-reproducibility/ R, Docker and Checkpoint: A Route to Reproducibility]
</li>
<li>[https://cran.r-project.org/web/packages/liftr/index.html liftr]: Persistent reproducible reporting by containerization of R Markdown documents. I am not successful running the example in vignette.
{{Pre}}
> render_docker(input)
Sending build context to Docker daemon  6.144kB
Step 1/8 : FROM rocker/r-base:latest
---> 4e232e9b4097
Step 2/8 : MAINTAINER Nan Xiao <[email protected]>
---> Running in f75404db590a
Removing intermediate container f75404db590a
---> ceae0a54648c
Step 3/8 : RUN  rm -f /var/lib/dpkg/available  && rm -rf  /var/cache/apt/*  && apt-get update -qq  && apt-get install -y
  --no-install-recommends    ca-certificates    libssl-dev    libcurl4-openssl-dev    libxml2-dev    git
---> Running in 8a4324cc7c53
W: Failed to fetch http://deb.debian.org/debian/dists/testing/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch http://http.debian.net/debian/dists/sid/InRelease  Temporary failure resolving 'http.debian.net'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
E: The value 'testing' is invalid for APT::Default-Release as such a release is not available in the sources
</pre>
</pre>
</li>
</ul>


=== Debugging R memory problem ===
The only issue is there is no username/password to protect other people to access the web GUI. The solution of binding to localhost to restrict the access does not work for remote administration.
[https://hub.docker.com/r/wch1/r-debug/ Docker image for debugging R memory problems] (valgrind)


=== Debugging with gcc problem ===
That is, the tool is suitable for home use.
https://twitter.com/eddelbuettel/status/1232341601483182081


=== More examples ===
== [https://kitematic.com/ Kitematic] (Mac, Windows and Ubuntu) ==
* [[#Docker|R -> Docker]]
Owned by Docker. Available for Mac OS X 10.8+ and Windows 7+ (64-bit) and Ubuntu. https://github.com/docker/kitematic/releases/
* [https://www.rplumber.io/docs/hosting.html#docker Plumber] package


=== Building a Repository of Alpine-based Docker Images for R ===
Run containers through a simple, yet powerful graphical user interface.
* [https://velaco.github.io/my-dockerfile-for-r-shiny-based-on-alpine-linux/ Part I]
* [https://velaco.github.io/my-dockerfile-for-r-shiny-based-on-alpine-linux-II/ Part II]


=== RStudio in Docker – now share your R code effortlessly!. Markdown ===
It can not connect to remote docker machines.
https://harini.blog/2019/05/25/rstudio-and-rshiny-in-docker/


It is interesting the Dockerfile uses littler's install2.r to install R packages. See http://dirk.eddelbuettel.com/code/littler.examples.html or [https://hub.docker.com/r/rocker/verse/dockerfile rocker/verse] Dockerfile. But it is not clear how to install private R packages (mount host folder and use install.packages()).
A [https://www.r-bloggers.com/share-your-shiny-apps-with-docker-and-kitematic/ Share your Shiny Apps with Docker and Kitematic!]


Note that the tag name should be lower case; awesomer instead awesomeR.
== [https://github.com/shipyard/shipyard Shipyard] (retired) ==
* [https://www.sumologic.com/blog/code/managing-containers-docker-shipyard/ Managing Containers with Docker Shipyard]
* Three alternatives:
** [http://rancher.com/ Rancher]
** [https://github.com/kevana/ui-for-docker Docker UI]
** [https://portainer.io/ Portainer]


Also got an error when trying to build the image: ''Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease  Temporary failure resolving 'deb.debian.org' ''.
== VS Code ==
* [https://youtu.be/i7ABlHngi1Q Docker For Beginners: From Docker Desktop to Deployment]
* [https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers Visual Studio Code Remote - Containers]
* [https://www.docker.com/blog/how-to-develop-inside-a-container-using-visual-studio-code-remote-containers/ How to Develop Inside a Container Using Visual Studio Code Remote Containers]
* [https://www.cloudsavvyit.com/12837/how-to-edit-code-in-docker-containers-with-visual-studio-code/ How to Edit Code in Docker Containers With Visual Studio Code]


Try it again at home. The ''apt update'' part is OK but I still got a new error: ''Error: installation of package ‘gifski’ had non-zero exit status''
= Applications =
[[Docker_Applications|Docker Applications]]


(Updated 9-18-2020) Try both the long and short commands using the tag '4.0.2' instead of '3.5.1'. Both work. The report file <example_report.pdf> is generated. In this example, the Rmd file is called through an R file. See [https://github.com/jlintusaari/R-docker-report the source code].
== CasaOS ==
Every app is based on a Docker application
* https://casaos.io/
** https://wiki.casaos.io/en/get-started. It also supports arm64, armv7.
** http://casaos.local
** https://docs.zimaboard.com/docs/index.html Default login casaos/casaos. For a new user, the password has to be at least 5 characters.
* [https://youtu.be/FwJByjTdKks Revisiting CasaOS After A Few Months] 2022-6-14
* [https://youtu.be/w44CypRO5l4 Home Servers Have NEVER Been This Easy: CasaOS + ZimaBoard] 4/23/2023
* [https://github.com/xp1ode/new/blob/506bf428592a9604c4d8ca4cc6d0426280805b48/Screenshot%20from%202022-11-15%2006-12-55.png List of apps] 2022/11


=== METACRAN web ===
= Orchestrator =
https://github.com/metacran/metacranweb It works (2019-11-3).
== [https://www.docker.com/kubernetes Kubernete] ==
 
* [https://www.zdnet.com/article/kubernetes-dropping-docker-is-not-that-big-of-a-deal/ Kubernetes dropping Docker is not that big of a deal]
=== Modified Dockerfile ===
* [https://www.freecodecamp.org/news/kubernetes-vs-docker-whats-the-difference-explained-with-examples/ Kubernetes VS Docker: What's the Difference? Explained With Examples]
[https://joachim-gassen.github.io/2020/05/checking-your-package-for-compatibility-with-r-4/ Checking your Package for Compatibility with R 4.0.0]
* [https://kodekloud.com/p/kubernetes-for-the-absolute-beginners-hands-on Kubernetes For The Absolute Beginner - Hands On] from kodekloud.com
 
* [http://docker-k8s-lab.readthedocs.io/en/latest/ Docker Kubernetes Lab Handbook] by Peng Xiao. The github page is on https://github.com/xiaopeng163/docker-k8s-lab.
== [http://www.bioconductor.org/help/docker/ Bioconductor] ==
* [https://youtu.be/2G-A6Knjrvo Introducing Kubernetes] by Marc Sluiter. Kubenetes can be used to schedule, run and manage containers in a cluster of virtual or physical machines.
(2020-1-30)
** Starting by Google in 2014
* https://bioconductor.org/help/docker/
** Pods: smallest unit of deployment. small group of tightly coupled containers. share network and data volumes. routable IP address. multiple pods per node.
* https://hub.docker.com/r/bioconductor/bioconductor_docker
* [https://youtu.be/6xJwQgDnMFE Create a 2-node Kubernetes cluster in 10 minutes]
 
** [https://github.com/kubernetes/dashboard Kubernetes Dashboard]
(2019-10-15)
* [https://youtu.be/_vHTaIJm9uY Kubernetes Webinar Series - Getting Started with Kubernetes]
* [https://hub.docker.com/r/bioconductor/bioconductor_full bioconductor_full]  
** https://thenewstack.io/tutorial-configuring-ultimate-development-environment-kubernetes/
* [https://seandavi.github.io/post/build-linux-r-binary-packages/ Building R Binary Packages for Linux]
** https://github.com/janakiramm/Kubernetes-dev-env
 
* [https://www.oreilly.com/ideas/how-to-manage-docker-containers-in-kubernetes-with-java Deploying Java Applications with Docker and Kubernetes]
[https://medium.com/@nitesh.turaga/how-i-use-bioconductor-with-docker-ca9d17d87771 How I use Bioconductor with Docker], [https://medium.com/@nitesh.turaga/how-i-use-bioconductor-with-docker-part-2-more-memory-faster-bioconductor-with-docker-fd7e69dce2bf Part 2: More memory, faster Bioconductor with Docker]
* [https://blog.alexellis.io/docker-for-mac-with-kubernetes/ First impressions: Docker for Mac with Kubernetes]
 
* [https://github.com/kubernetes/minikube Minikube]: run Kubernetes locally
=== [https://github.com/Bioconductor-notebooks/BiocImageBuilder BiocImageBuilder] ===
** [https://www.howtoforge.com/tutorial/how-to-install-kubernetes-with-minikube-on-ubuntu-1804-lts/ How to Install Kubernetes with Minikube on Ubuntu 18.04 LTS]
[http://www.rna-seqblog.com/reproducible-bioconductor-workflows-using-browser-based-interactive-notebooks-and-containers/ Reproducible Bioconductor workflows using browser-based interactive notebooks and containers]
** [https://opensource.com/article/19/12/kubernetes-namespaces Kubernetes namespaces for beginners].  You need to have Minikube, Helm, and the kubectl command line installed.
 
* [https://www.biorxiv.org/content/10.1101/488643v2?rss=1 Galaxy-Kubernetes integration: scaling bioinformatics workflows in the cloud] Moreno et al 2019
=== Bioc Conference ===
* [https://seandavi.github.io/2019/03/helm-on-google-kubernetes-engine/ Helm on Google Kubernetes Engine] by Sean Davis
* [https://github.com/Bioconductor/BiocWorkshops2019/blob/master/README.md Bioc2019 conference]. [http://biocworkshops2019.bioconductor.org.s3-website-us-east-1.amazonaws.com/ Workshopt material]. Before the meeting (6/20/2019) there are 13 downloads and it bumps to 32 downloads after the meeting for [https://hub.docker.com/r/seandavi/bioc_2019 the docker image].
* [https://microk8s.io/ MicroK8s]
** Download a tarball containing R packages (binary). 605 packages are included. It bundles 605 packages based on R 3.6.0 and Bioconductor 3.10 (BiocManager 1.30.4).
** [https://github.com/CanonicalLtd/multipass Multipass]
** Run RStudio container. All R packages downloaded in the last step are mounted. (user=rstudio, ps=bioc). That is, built-in is at '''/usr/local/lib/R/library''', custom is at '''/usr/local/lib/R/site-library'''. These two locations are what '''.libPaths()''' gives.
** [https://itnext.io/microk8s-on-macos-98f1de3aa63e MicroK8s on MacOS]
* [http://bioc2020.bioconductor.org/workshops BioC 2020]
* [https://www.howtoforge.com/resource-limits-in-kuberenetes/ Resource Limits in Kuberenetes]
** Workshop packages were created using the [https://github.com/seandavi/BuildABiocWorkshop2020?s=09 BuildABiocWorkshop2020] template. From the 'Dockerfile', we see each workshop material is organized as an R package. So each workshop's package is built in the Docker image. There is no need to build the vignette again. NOTE: it takes a while to build the Docker image locally since it requires to compile each R package separately.
* [https://github.com/seandavi/Orchestra?s=09 Orchestra] and [https://eurobioc2020.bioconductor.org/workshops?s=09 European Bioconductor Meeting 2020]
** [https://docs.google.com/presentation/d/1xY4-8rTVAv9Z6fVXIv1fAIlZGtvGPN6G5e-bKKPLEx0/ BioC 2020: Where Software and Biology Connect] Opening Remarks
* [https://www.cloudsavvyit.com/9783/kubernetes-alternatives-to-docker-commands/ Kubernetes Alternatives to Docker Commands]
** Take the [https://github.com/LieberInstitute/recountWorkshop2020 recount2 workshop] for example, we don't need to knit the Rmd file. To view HTML vignette, we type '''browseVignettes(package="recountWorkshop2020") ''' and click the link "HTML". In case of the requested page was not found error, add '''help/''' to the URL right after the hostname, e.g., http://localhost:8787/help/library/recountWorkshop2020/doc/recount-workshop.html. Another way to open the HTML without any tweak is type "help(package = 'recountWorkshop2020')" -> User Guide -> HTML.
* [https://www.makeuseof.com/what-is-kubernetes/ What Is Kubernetes and What Is It Used For?]
** [https://biocasia2020.bioconductor.org/workshops Bioc Asia 2020]
** The vignette may not include the R code. So the Rmd file is still needed to understand the content or do a practice.
 
== Docker for Shiny Server ==
<ul>
<li>https://hub.docker.com/r/rocker/shiny/. Pay attention to the [https://hub.docker.com/r/rocker/shiny/~/dockerfile/ Dockerfile] there.
{{Pre}}
# sudo mkdir /srv/
sudo mkdir /srv/shinylog
sudo mkdir /srv/shinyapps
cd /srv/shinyapps
sudo wget https://raw.githubusercontent.com/rstudio/shiny-examples/master/050-kmeans-example/ui.R
sudo wget https://raw.githubusercontent.com/rstudio/shiny-examples/master/050-kmeans-example/server.R
nmap localhost
docker run -d --restart always -p 3838:3838  \
  -v /srv/shinyapps/:/srv/shiny-server/  \
  -v /srv/shinylog/:/var/log/shiny-server/  \
  rocker/shiny:4.0.3
nmap localhost
</pre>
If the 3838 port method (http://IP:3838) works, we can move on to use Nginx + reverse proxy method; see [[Apache#R.2FShiny_example|Apache->R/Shiny example]] for the instruction.
</li>
</ul>
* [http://docs.rstudio.com/shiny-server/ Shiny Administrator Guide]
* [https://docs.docker.com/engine/reference/commandline/run Docker Reference]. See Engine (docker) CLI from their LHS.
* [http://www.flaviobarros.net/2015/04/30/dockerizing-a-shiny-app/ Dockering a shiny app]
<ul>
<li>[https://biolitika.si/pin-r-package-versions-using-docker-and-renv.html Pin package versions in your production Docker image]. Need to modify Dockerfile to install 'shiny' along with 'renv'. We need to create a directory 'shinyapp' with simple shiny files (eg [https://shiny.rstudio.com/gallery/kmeans-example.html kmeans] example).
{{Pre}}
docker build -t pinrpackageversion .
mkdir shinyapp
wget https://raw.githubusercontent.com/rstudio/shiny-examples/master/050-kmeans-example/ui.R -P shinyapp
wget https://raw.githubusercontent.com/rstudio/shiny-examples/master/050-kmeans-example/server.R -P shinyapp
docker run --name shinyapp --rm -p 3838:3838 pinrpackageversion
</pre>
</li>
<li>Question: what happened if there is a new version of R or shiny-sever?
<syntaxhighlight lang='bash'>
# specify --name so we don't need to use container id to access it later
$ docker run -d --name rshiny -p 81:3838 rocker/shiny


$ docker ps -l
== Kubernete vs Docker swarm ==
* https://platform9.com/blog/kubernetes-docker-swarm-compared/
* [https://www.ithome.com.tw/news/124290 DigitalOcean容器應用開發大調查:近半數開發者已採用]


# Run in container's bash
== k3s: Lightweight Kubernetes ==
# https://stackoverflow.com/questions/30172605/how-to-get-into-a-docker-container
[https://opensource.com/article/20/3/kubernetes-raspberry-pi-k3s Run Kubernetes on a Raspberry Pi with k3s]
# non-interactive
$ docker exec -it rshiny ip addr 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
22: eth0@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 scope global eth0
      valid_lft forever preferred_lft forever


# interactive shell
= Other containers =
$ docker exec -it rshiny bash
== [https://sylabs.io/ Singularity] and HPC systems ==
root@5b78988d5619:/# who
* Old URL at [http://singularity.lbl.gov/ singularity.lbl.gov]
root@5b78988d5619:/# which R
* Singularity enables users to have full control of their environment; Singularity containers let users run applications in a Linux environment of their choosing. No 'sudo' is needed in general unless you want to build a container from a recipe.
/usr/bin/R
* Containers are more like an executable file for you to use
root@5b78988d5619:/# ls /home
* Containers are stored under the current location. It does not have a centre location (like /etc/default/docker if we use docker) to store images.
docker shiny
* Can convert Docker containers to Singularity and run containers directly from Docker Hub
root@5b78988d5619:/# ls /srv/shiny-server
* These bind points cannot be created unless the path already exists within the container. To ensure access to these storage spaces and remedy bind point errors, create these directories in the %post section of your Bootstrap file.
01_hello  02_text  03_reactivity  04_mpg  05_sliders  06_tabsets  07_widgets  08_html  09_upload  10_download  11_timer  index.html  sample-apps
* [https://singularity-hub.org/ Singularity Hub]
root@5b78988d5619:/# R -e 'package.Description("glmnet")'
root@5b78988d5619:/# exit
 
$ docker stop rshiny
$ docker rm rshiny  # remove the container
</syntaxhighlight>
</li>
</ul>


It is also useful to use the '''-v''' option in '''docker run''' so the local directory (eg /srv/shinyapps/) can be mapped to the docker volume (/srv/shiny-server).
Ref:
<syntaxhighlight lang='bash'>
* [https://hpc.nih.gov/apps/singularity.html Singularity] at NIH/Biowulf. It needs to run 'module load singularity' first to load related commands.
# Put your shiny app in host's /srv/shinyapps/appdir,
* NIH-HPC
$ docker run -d -p 81:3838 \
** [https://singularity-tutorial.github.io/ Tutorial] and its [https://github.com/NIH-HPC/Singularity-Tutorial Github] source ('''valuable''')
    -v /srv/shinyapps/:/srv/shiny-server/ \
** [https://github.com/nih-byob/presentations/tree/master/2019/01_bioinformatics_tips Bioinformatics advice I wish I learned 10 years ago]
    -v /srv/shinylog/:/var/log/shiny-server/ \
** [https://github.com/NIH-HPC/singularity-def-files NIH HPC Singularity Definition Files]
    rocker/shiny
* [https://tin6150.github.io/psg/blogger_container_hpc.html DOCKER VS SINGULARITY VS SHIFTER VS UGE CONTAINER EDITION]
</syntaxhighlight>
* [https://github.com/data61/gossamer xenome] case where the build instruction is good for Ubuntu 16.04. If we try to build under Ubuntu 14.04 (tested using docker run), we will get an error message '' CMake 3.1 or higher is required.  You are running version 2.8.12.2''.
And check running status
* https://github.com/NIH-HPC/singularity-examples/blob/master/rnaseq/Singularity.rnaseq
<pre>
$ docker ps -a
</pre>


=== Check opened ports ===
'''Comparison of docker and singularity commands:'''
See [[Linux#What_network_ports_are_open:_lsof_or_netstat|What network ports are open: lsof or netstat]].
{| class="wikitable"
<pre>
|-
$ netstat -ant
! [https://docs.docker.com/edge/engine/reference/commandline/docker/ docker] !! [http://singularity.lbl.gov/user-guide singularity]
Active Internet connections (servers and established)
|-
Proto Recv-Q Send-Q Local Address          Foreign Address        State     
| $ docker pull ubuntu:latest <br/>$ docker pull broadinstitute/gatk3:3.8-0 || $ singularity pull docker://ubuntu:latest<br/>$ singularity pull docker://broadinstitute/gatk3:3.8-0
tcp        0      0 0.0.0.0:22              0.0.0.0:*              LISTEN   
|-
tcp        0      0 127.0.0.1:25            0.0.0.0:*              LISTEN   
| $ docker build -t myname/myapp:latest -f Dockerfile || $ singularity build myapp.sif myapp.def
...
|-
tcp6      0      0 :::80                  :::*                    LISTEN   
| $ docker shell (not exist) || $ singularity shell docker://broadinstitute/gatk3-3.8-0<br/>  $ singularity shell gatk3-3.8-0.img<br/> > ls  # the default location depends on the host system<br/>
tcp6      0      0 :::81                  :::*                    LISTEN   
> ls /usr  # this is from the container <br/><br/>$ singularity shell --bind ~/Downloads:/mnt XXX.img<br/>$ singularity shell docker://ubuntu:latest<br/># container is ephemeral
tcp6      0      0 :::22                  :::*                    LISTEN   
|-
tcp6      0      0 ::1:25                  :::*                    LISTEN
| $ docker run --name test -it ubuntu date <br/><br/># <span style="color: red">The next example is similar to 'singularity exec'</span><br/>$ docker run --rm -i -t \<br/-v $(pwd):/usr/my_data \<br/>  broadinstitute/gatk3:3.8-0 \<br/>  bash /usr/my_data/myscript.sh || $ singularity run gatk3-3.8-0.img date
</pre>
|-
 
| $ docker run --name ubuntu_bash --rm -i -t ubuntu bash<br/>$ docker exec -d ubuntu_bash touch /tmp/execWorks || # <span style="color: red">Most useful</span><br>$ singularity exec gatk3-3.8-0.img java -version <br/>$ singularity exec xxx.img cat /etc/*release<br>$ singularity exec docker://rocker/tidyverse:latest R <br/>$ singularity exec docker://rocker/tidyverse:latest Rscript myScript.R<br/>
=== Blog Explorer ===
|}
https://github.com/nz-stefan/blog-explorer, [https://blog.rstudio.com/2020/07/13/winners-of-the-2nd-shiny-contest/ Winners of the 2nd Annual Shiny Contest]
 
=== Raspberry Pi ===
[https://www.brodrigues.co/blog/2020-09-20-shiny_raspberry/ The Raspberry Pi 4B as a shiny server]


== BioContainers ==
=== Cache ===
* https://biocontainers.pro/#/
When we run ''singularity exec docker://rocker/tidyverse:latest R'', it will save something in the cache in our system.
* https://hub.docker.com/u/biocontainers/
 
* https://github.com/BioContainers
It seems to be OK after I manually delete the directory $HOME/.singularity (tested in Biowulf).
 
=== RStudio ===
* [https://rviews.rstudio.com/2017/03/29/r-and-singularity/ R and Singularity] by Bryan Lewis
* [https://archive.sylabs.io/2018/10/rstudio-ide-and-server/ RStudio – IDE and Server powered by Singularity] by Jason Stover
* https://github.com/nickjer/singularity-rstudio and https://github.com/nickjer/singularity-r
* https://www.rocker-project.org/use/singularity/. It also includes how to run a SLURM job script.
* https://njstem.wordpress.com/2018/08/02/r-script-seurat-with-a-singularity-container-using-slurm/
* https://journal.r-project.org/archive/2017/RJ-2017-065/RJ-2017-065.pdf


== Nginx ==
* [https://www.cloudsavvyit.com/5167/how-to-run-nginx-inside-docker-for-easy-auto-scaling/ How to Run NGINX Inside Docker (for Easy Auto-Scaling)]
* [https://www.digitalocean.com/community/tutorials/how-to-run-nginx-in-a-docker-container-on-ubuntu-14-04 How To Run Nginx in a Docker Container on Ubuntu 14.04]. It teaches to put the whole web site in the host directory and the nginx configuration in the local host. You can edit the default file locally (to serve a new directory), or to use a '''proxy_pass''' to forward the traffic to another app/container like you would with a regular Nginx installation).
* [https://www.digitalocean.com/community/tutorials/docker-explained-how-to-containerize-and-use-nginx-as-a-proxy How To Containerize and Use Nginx as a Proxy]
* [https://blog.docker.com/2015/04/tips-for-deploying-nginx-official-image-with-docker/ TIPS FOR DEPLOYING NGINX (OFFICIAL IMAGE) WITH DOCKER]
* [https://linux.cn/article-8932-1.html Docker 快速入门]
* [https://www.linux.com/learn/intro-to-linux/2017/11/how-install-and-use-docker-linux How to Install and Use Docker on Linux] Reverse-proxy, LetsEncrypt ... built in.
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
docker run --name docker-nginx -p 80:80 nginx
$ singularity exec docker://rocker/tidyverse:latest R
 
$ singularity exec docker://rocker/tidyverse:latest Rscript myScript.R
docker run --name my-nginx \
    -v /host/path/nginx.conf:/etc/nginx/nginx.conf:ro \
    -v /some/html:/usr/share/nginx/html:ro \
    -p 8080:80 \
    -d nginx
</syntaxhighlight>
</syntaxhighlight>


== httpd/apache ==
== [https://github.com/scanon/shifter Shifter] ==
https://hub.docker.com/_/httpd/


* [https://www.tecmint.com/install-apache-web-server-in-a-docker-container/ How to Setup a Simple Apache Web Server in a Docker Container]
= [https://conda.io/docs/ Conda] =
* [https://codingbee.net/docker/docker-an-apache-hello-world-example Docker – An apache ‘hello world’ example]
* [https://www.quora.com/What-is-the-difference-between-conda-and-Docker What is the difference between conda and Docker?]


== Tomcat ==
== Anaconda ==
https://hub.docker.com/_/tomcat/
* [https://hub.docker.com/r/continuumio/anaconda3 Getting started with Anaconda & Docker] using continuumio/anaconda3


== Jenkins for continuous integration ==
== Bioconda ==
* https://www.tutorialspoint.com/jenkins/jenkins_overview.htm
* https://bioconda.github.io/
* https://jenkins.io/doc/book/installing/
* https://anaconda.org/bioconda
* Example: https://dev.ropensci.org/
* https://www.biorxiv.org/content/early/2017/10/27/207092
 
* https://zhuanlan.zhihu.com/p/25085567
== MySQL ==
* [http://slides.com/olexschoen/snakemake-bioconda#/ Snakemake: Reproducible and scalable data analysis]
* https://hub.docker.com/r/mysql/mysql-server/
* https://chrisconlan.com/installing-r-python-anaconda-biologists/. Note that the R version is 3.4.2 but the latest one should be 3.4.3.
* [https://dev.mysql.com/doc/mysql-installation-excerpt/5.7/en/linux-installation-docker.html Deploying MySQL on Linux with Docker]
* [https://davetang.github.io/reproducible_bioinformatics/conda.html Introduction to Conda] Dave Tang
* Note it seems impossible to use the 'grant' command interactively within the MySQL docker container.
* https://dzone.com/articles/using-docker-to-create-a-mysql-server <syntaxhighlight lang='bash'>
docker run --detach --name test-mysql --env="MYSQL_ROOT_PASSWORD=mypassword" mysql
docker inspect test-mysql | grep IPAddress  # 172.17.0.2
 
# https://dev.mysql.com/doc/index-other.html
wget -O - http://downloads.mysql.com/docs/world.sql.gz | gunzip -c > world.sql


mysql -uroot -pmypassword -h 172.17.0.2 -e 'CREATE DATABASE world'
Using docker to install conda (https://conda.io/docs/user-guide/tutorials/index.html)
cat world.sql | mysql -uroot -pmypassword -h 172.17.0.2 world
# zcat world.sql.gz | mysql -uroot -pmypassword -h 172.17.0.2 world
mysql -uroot -pmypassword -h 172.17.0.2 -e 'USE world; SHOW TABLES'
 
mysql -uroot -pmypassword -h 172.17.0.2 -e 'DROP DATABASE world'
</syntaxhighlight>  <syntaxhighlight lang='rsplus'>
# Let's see if we can access the database in R
system("sudo apt install libmysqlclient-dev")
library(RMySQL)
con <- dbConnect(MySQL(), "world", user="root", password="mypassword", host="172.17.0.2")
dbListTables(con)
head(dbReadTable(con, "city"))
head(dbReadTable(con, "country"))
head(dbReadTable(con, "countrylanguage"))
 
dbWriteTable(con, 'iris', iris)
# Error in .local(conn, statement, ...) :
#  could not run statement: The used command is not allowed with this MySQL version
dbDisconnect(con)
</syntaxhighlight>
* [https://youtu.be/w_aVnMmrASE How To Create And Test MySQL Database Server in Docker On Ubuntu Linux] (video). Use Dockerfile.
* [https://severalnines.com/blog/mysql-docker-containers-understanding-basics MySQL Docker Containers: Understanding the basics], https://youtu.be/qJQPckRYNis?t=16m29s <syntaxhighlight lang='bash'>
# Fire up
docker run --detach --name=test-mysql --env="MYSQL_ROOT_PASSWORD=mypassword" mysql
docker ps -a
 
# Connecting to the Container
docker inspect test-mysql | grep IPAddress
# sudo apt install mysql-client
# There are two methods to connect to the container
# method 1. https://hub.docker.com/r/mysql/mysql-server/
# docker exec -it test-mysql mysql -uroot -p
# method 2
mysql -uroot -pmypassword -h 172.17.0.20 -P 3306
docker run --detach --name test-wordpress --link test-mysql:mysql wordpress
docker exec -it test-wordpress bash
$ cat /etc/hosts
 
# Expose the MySQL container to the outside world
# Access the MySQL container directly from the machine’s port 6603
docker run --detach --name=test-mysql --env="MYSQL_ROOT_PASSWORD=mypassword" --publish 6603:3306 mysql
mysql -u root -pmypassword -h 127.0.0.1 -P 6603
 
# Configuration management
mkdir -p /root/container/test-mysql/conf.d
nano /root/container/test-mysql/conf.d/my-custom.cnf
docker run \
    --detach \
    --name=test-mysql \
    --env="MYSQL_ROOT_PASSWORD=mypassword" \
    --publish 6603:3306 \
    --volume=/root/docker/test-mysql/conf.d:/etc/mysql/conf.d \
    mysql
mysql -uroot -pmypassword -h127.0.0.1 -P6603 -e 'show global variables like "max_connections"';
docker run \
    --detach \
    --name=test-mysql \
    --env="MYSQL_ROOT_PASSWORD=mypassword" \
    --publish 6603:3306 \
    mysql \
    --max-connections=200 \
    --character-set-server=utf8mb4 \
    --collation-server=utf8mb4_unicode_ci
 
# Data Storage
mkdir -p /storage/docker/mysql-datadir
docker run \
    --detach \
    --name=test-mysql \
    --env="MYSQL_ROOT_PASSWORD=mypassword" \
    --publish 6603:3306 \
    --volume=/root/docker/test-mysql/conf.d:/etc/mysql/conf.d \
    --volume=/storage/docker/mysql-datadir:/var/lib/mysql \
    mysql
docker inspect test-mysql
ls -al /storage/docker/mysql-datadir/
 
# If a MySQL container runs on top of an existing MySQL datadir,
# the $MYSQL_ROOT_PASSWORD variable should be omitted from the “run” command line;
docker stop test-mysql
docker rm -f test-mysql
docker run -d --name=another-new-mysql -p 6605:3306 -v /storage/docker/mysql-datadir:/var/lib/mysql mysql
docker logs another-new-mysql
</syntaxhighlight>
* [https://codereviewvideos.com/course/docker-tutorial-for-beginners/video/docker-mysql-tutorial Use Docker to run a typical MySQL server] from Code Review Videos <syntaxhighlight lang='bash'>
docker volume create crv_mysql
 
docker run \
    -e MYSQL_ROOT_PASSWORD=my-secret-pw \
    -e MYSQL_DATABASE=devdb \
    -e MYSQL_USER=dbuser \
    -e MYSQL_PASSWORD=dbpassword \
    --mount type=volume,src=crv_mysql,dst=/var/lib/mysql \
    -p 3306:3306 \
    -d \
    mysql:latest
 
# Alternatively we can put username/password in a text file for security
cat > .env <<EOF
MYSQL_ROOT_PASSWORD=my-secret-pw
MYSQL_DATABASE=devdb
MYSQL_USER=dbuser
MYSQL_PASSWORD=dbpassword
EOF
docker run \
    --env-file .env \
    --mount type=volume,src=crv_mysql,dst=/var/lib/mysql \
    -p 3306:3306 \
    -d \
    mysql:latest
</syntaxhighlight>
 
=== Backup and restore ===
https://gist.github.com/spalladino/6d981f7b33f6e0afe6bb
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
# Backup
$ docker run -t -i --name test --net=host ubuntu bash
docker exec CONTAINER /usr/bin/mysqldump -u root --password=root DATABASE > backup.sql
# apt-get update
# apt-get install -y wget bzip2 python
# wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
# wget https://repo.continuum.io/archive/Anaconda2-5.1.0-Linux-x86_64.sh
# bash Miniconda2-latest-Linux-x86_64.sh
# bash Anaconda2-5.1.0-Linux-x86_64.sh
# exit


# Restore
$ docker start test
cat backup.sql | docker exec -i CONTAINER /usr/bin/mysql -u root --password=root DATABASE
$ docker exec -i -t test bash
# conda list  # WORKS!
# conda config --add channels r
# conda config --add channels defaults
# conda config --add channels conda-forge
# conda config --add channels bioconda
# conda install bwa  (Segmentation fault. Core dumped)
# which bwa
/root/anaconda2/bin/bwa
# conda install r  (Only get 3.4.2 but the latest is 3.4.3.)
# conda install bowtie
# bowtie --version
# conda install gatk (https://bioconda.github.io/recipes/gatk/README.html)
  (Due to license restrictions, this recipe cannot distribute and install GATK directly)
  (R is downgraded to 3.2.2:( )
  (Segmentation fault. Core dumped)
# exit
# docker stop test
# docker rm test
</syntaxhighlight>
</syntaxhighlight>


=== Blog ===
Get miniconda image instead of using a Ubuntu image
[https://www.howtoforge.com/tutorial/docker-guide-deploying-ghost-blog-with-mysql-and-traefik-with-docker/ Deploying Ghost Blog with MySQL and Traefik with Docker]
<syntaxhighlight lang='bash'>
 
$ docker pull continuumio/miniconda
== PostgreSQL ==
$ docker run -i -t continuumio/miniconda /bin/bash
* [https://markheath.net/post/exploring-postgresql-with-docker Exploring PostgreSQL with Docker]
# conda install r  (get 3.4.2)
* [https://colinfay.me/clients-db/ clientsdb - A docker image with clients comments]. See [[MySQL#Docker_server_.2B_R_client|here]].
# conda config --add channels bioconda
 
# conda install bwa  (OK, no error)
== Jekyll ==
# conda install gatk  (R was downgraded to 3.2.2, install openjdk 8.0.121)
[https://martinpeck.com/blog/2020/11/05/building-jekyll-with-vscode-devcontainers/ Building Jekyll Sites within VS Code Dev Containers]
# which gatk
 
/opt/conda/bin/gatk
== node.js ==
# gatk -h
[https://medium.com/openmindonline/js-monday-13-dockerizing-node-js-e09d5f8bf945 Dockerizing Node.js]. See the code [https://github.com/arraytools/dockerizenode here].
GATK jar file not found. Have you run "gatk-register"?
<pre>
</syntaxhighlight>
 
Issues:
$ docker build -t my-node-app .
* R version is not up to date
$ docker run -p 3000:3000 my-node-app
* '''So the problem is installing GATK requires an installation of R and the current R was affected.'''
</pre>
 
[https://www.docker.com/blog/how-to-setup-your-local-node-js-development-environment-using-docker/ How To Setup Your Local Node.js Development Environment Using Docker] (not working)
{{Pre}}
$ cd notes-service
$ cat Dockerfile
FROM node:12.18.1


WORKDIR /code
= [https://coreos.com/ CoreOS] =
* Redhat has acquired CoreOS 1/31/2018.
* [https://coreos.com/os/eol/ End-of-life announcement for CoreOS Container Linux]


COPY package.json /code/package.json
== Installation ==
COPY package-lock.json /code/package-lock.json
* [https://coreos.com/os/docs/latest/quickstart.html Quick Start]
* [https://youtu.be/vy6hWsOuCh8  Installing CoreOS on a bare metal or virtual machine] from the user Format C: in youtube
* [https://youtu.be/M0TYqVDnNAI Assigning Static IP to a CoreOS Server]
* [https://www.youtube.com/watch?v=6LOiIH8s3xM Mounting storage in CoreOS]
* [https://www.youtube.com/watch?v=InLmXWgPzBs Installing, Deploying Docker containers with Fig / Docker Compose on boot2docker or CoreOS]
* [https://www.youtube.com/watch?v=S9kfP2WYV_I Running a Private Docker Registry Container]


RUN npm install
We first boot a liveCD from any OS (CentOS works but Ubuntu 16.04 gave errors). In Virtualbox, we choose 'Red Hat' if we use CentOS.


COPY . /code
Once the VM is created. We go to the settings. Create a bridged network or host-only network first (even we can get files from the host without creating a host-only network). Storage: choose CentOS-7.  


CMD [ "node", "server.js" ]
# Get the [https://coreos.com/os/docs/latest/installing-to-disk.html install script] from Github and create <'''coreos_install.sh'''> and chmod +x
# create <'''cloud-config.yaml'''> file which will include ssh_authorized_keys generated from another machine. It should also contain a new token for the cluster from https://discovery.etcd.io/new.
# ls -l /dev/sd*
# run '''sudo ./coreos_install.sh -d /dev/sda -C stable -c cloud-config.yaml'''. It will download the latest stable CoreOS, install to the HD
# Don't leave the VM or it will freeze. Issue '''sudo shutdown -h now''' once we see the word 'Success' at the last line of the output.
# Remove CentOS from the VM storage. Boot the coreOS VM.  


$ docker build -t notes-service .
The new screen shows ''corebm1 login'' with an IP. Go back to another machine and type '''ssh -i /tmp/CoreOSBM_rsa [email protected]'''. Inside CoreOS, we can type '''docker images'''.
$ docker run -it --rm -d -v mongodb:/data/db -v mongodb_config:/data/configdb -p 27017:27017 --network mongodb --name mongodb mongo
$ docker run -it --rm -d --network mongodb --name notes -p 8081:8081 -e SERVER_PORT=8081 -e DATABASE_CONNECTIONSTRING=mongodb://mongodb:27017/yoda_notes notes-service
# this does not work though no errors; omit "-d" to see the error message


$ curl --request POST \
The '''cloud-config.yaml'' file has to follow the format in https://coreos.com/os/docs/latest/cloud-config.html. Use the online validator https://coreos.com/validate/ to correct. At first I use the file from the youtube video. There is no error coming out when I ran the installation script. But I cannot connect to coreOS. The cloud-config.yaml file I use is (pay attention to '-', double quotes and indent characters)
--url http://localhost:8081/services/m/notes \
<pre>
  --header 'content-type: application/json' \
#cloud-config
  --data '{
#
"name”: "this is a note",
# set hostname
"text": "this is a note that I wanted to take while I was working on writing a blog post.",
hostname: CoreBM1
"owner": "peter"
}'
 
</pre>


== Redis ==
# Set ssh key
https://hub.docker.com/_/redis/
ssh_authorized_keys:
  - "ssh-rsa AAAAB3 ..... brb@T3600"


[https://github.com/metacran/metacranweb Web pages for metacran]
coreos:
 
  etcd:
[https://markheath.net/post/exploring-redis-with-docker Exploring Redis with Docker]
    discovery: "https://discovery.etcd.io/d3e95 .... "
 
# sudo ./installos -d /dev/sda -C stable -c cloud-config.yaml
[http://collabnix.com/running-redis-inside-docker-container-on-jetson-nano/ Redis running inside Docker container on NVIDIA Jetson Nano], [https://redis.io/topics/ARM Redis is ideal for IoT and Embedded devices for several reasons].
</pre>
 
== Laravel ==
[https://www.howtoforge.com/tutorial/dockerizing-laravel-with-nginx-mysql-and-docker-compose/ Dockerizing Laravel with Nginx MySQL and Docker Compose on Ubuntu 18.04 LTS]
 
== phpMyAdmin ==
* https://hub.docker.com/r/phpmyadmin/phpmyadmin/
* [http://omarghader.github.io/docker-tutorial-phpmyadmin-and-mysql-server/ phpMyAdmin and MySQL server]


== CoreOS exploration ==
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
docker run --name mysqlphp -e MYSQL_ROOT_PASSWORD=0000 -d mysql
brb@T3600 /tmp $ ssh -i /tmp/id_rsa [email protected]
docker run --name myadmin -d --link mysqlphp:db -p 8080:80 phpmyadmin/phpmyadmin
Enter passphrase for key '/tmp/id_rsa':
</syntaxhighlight>
CoreOS stable (1010.6.0)
 
core@CoreBM1 ~ $
Now go to your browser , tap http://localhost:8080 and login with root/0000.
core@CoreBM1 ~ $ docker images
 
REPOSITORY          TAG                IMAGE ID            CREATED            SIZE
'''ERROR''' mysqli_real_connect(): The server requested authentication method unknown to the client [caching_sha2_password]
core@CoreBM1 ~ $ df -h
 
Filesystem      Size  Used Avail Use% Mounted on
The solution can be found [https://stackoverflow.com/questions/49948350/phpmyadmin-on-mysql-8-0 here]. After the change, I can log in phpMyAdmin.
devtmpfs        1.4G    0  1.4G  0% /dev
<syntaxhighlight lang='bash'>
tmpfs          1.4G    0  1.4G  0% /dev/shm
docker exec -it mysqlphp bash
tmpfs          1.4G  340K  1.4G  1% /run
 
tmpfs          1.4G    0  1.4G  0% /sys/fs/cgroup
mysql -u root -p0000
/dev/sda9        18G  23M  17G  1% /
 
/dev/sda3      985M  589M  345M  64% /usr
ALTER USER root IDENTIFIED WITH mysql_native_password BY '0000';
tmpfs          1.4G    0  1.4G  0% /media
 
/dev/sda1      128M  37M  92M  29% /boot
exit
tmpfs          1.4G    0  1.4G  0% /tmp
exit
/dev/sda6      108M  52K  99M  1% /usr/share/oem
core@CoreBM1 ~ $ free -m
            total      used      free    shared    buffers    cached
Mem:          2713        187      2525          0          9        109
-/+ buffers/cache:        68      2644
Swap:            0          0          0
core@CoreBM1 ~ $ lsb_release -a
-bash: lsb_release: command not found
core@CoreBM1 ~ $ docker pull ubuntu
Using default tag: latest
latest: Pulling from library/ubuntu
f069f1d21059: Pull complete
ecbeec5633cf: Pull complete
ea6f18256d63: Pull complete
54bde7b02897: Pull complete
Digest: sha256:bbfd93a02a8487edb60f20316ebc966ddc7aa123c2e609185450b96971020097
Status: Downloaded newer image for ubuntu:latest
core@CoreBM1 ~ $ docker images
REPOSITORY          TAG                IMAGE ID            CREATED            SIZE
ubuntu              latest              0f192147631d        7 days ago          132.7 MB
core@CoreBM1 ~ $
</syntaxhighlight>
</syntaxhighlight>


== MagicMirror ==
== CoreOS cluster discovery ==
https://github.com/MichMich/MagicMirror/blob/master/README.md#docker
https://coreos.com/os/docs/latest/cluster-discovery.html


== Mail/smtp server ==
== etcd ==
* [https://jzweig.com/blog/setup-your-own-email-server-with-docker/ Setup Your Own Email Server with Docker]
* https://coreos.com/etcd/
* [https://petermeissner.de/blog/2020/09/07/web-send-mail-windows/ Sending Emails with {curl} and Docker]
** https://hub.docker.com/r/bytemark/smtp/


== Mediawiki ==
== fleet ==
* https://www.mediawiki.org/wiki/Docker
* https://coreos.com/using-coreos/clustering/
* https://hub.docker.com/_/mediawiki/ or (better) https://docs.docker.com/samples/library/mediawiki/. The Dockerfile is based on [https://hub.docker.com/_/php/ jessie].
 
* https://github.com/kristophjunge/docker-mediawiki
= TryGhost =
* [https://jzweig.com/blog/setup-mediawiki-and-postgres-with-docker/ Setup MediaWiki and Postgres with Docker] (based on simplyintricate/mediawiki, works on Mint). Visit http://localhost:8080 to see the result.
https://github.com/TryGhost/Ghost
 
= Firecracker =
[https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-less-than-a-second/ Firecracker: start a VM in less than a second]
 
= Self-hosting =
[https://www.reddit.com/r/selfhosted/comments/uvq1x3/tools_and_resources_for_selfhosting/ Tools and Resources for Self-Hosting]
 
= Linux in browser =
<ul>
<li>Webtop, https://hub.docker.com/r/linuxserver/webtop#!
* [[#Play_with_Docker_.28PWD.29|Play with Docker]]
* [https://www.linuxserver.io/blog/2021-05-05-meet-webtops-a-linux-desktop-environment-in-your-browser Meet Webtops A Linux Desktop Environment In Your Browser]
* [https://shownotes.opensourceisawesome.com/webtops/ Webtops - Linux Desktops in Docker]
<pre>
<pre>
$ ls -l ~/wiki_root
---
total 24
version: "2.1"
drwxr-xr-x 3 root     root     4096 Jul 10 10:09 data
services:
drwxrwxr-x 3 brb      brb      4096 Jul 10 09:23 db
  webtop:
drwxrwxr-x 2 www-data www-data 4096 Jul 10 09:23 extensions
    image: lscr.io/linuxserver/webtop:ubuntu-mate
drwxrwxr-x 2 www-data www-data 4096 Jul 10 09:23 images
     container_name: webtop
-rw-rw-r-- 1 www-data www-data 4152 Jul 10 11:12 LocalSettings.php
     security_opt:
      - seccomp:unconfined #optional
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/Chicago
      - SUBFOLDER=/ #optional
      - KEYBOARD=en-us-qwerty #optional
      - TITLE=Webtop #optional
    volumes:
      - ./data:/config
      - /var/run/docker.sock:/var/run/docker.sock #optional
    ports:
      - 3000:3000
    devices:
      - /dev/dri:/dev/dri #optional
    shm_size: "1gb" #optional
    restart: unless-stopped
</pre>
</pre>
* [https://github.com/besnik/tutorials/tree/master/docker-mediawiki Running Mediawiki as Docker container] (based on synctree/mediawiki, works tested on GalliumOS)
** Follow [https://github.com/besnik/tutorials/tree/master/docker-mysql Run MySql inside docker container] first. 1. Setup host to store configuration, data and dumps. The 'dumps' and 'initdb.d' folders are optional 2. Override default mysql configuration 3. Executing scripts and sql on container startup (optional) 4. Starting mysql container
** Visit http://IP_address:8080 to see the mediawiki page running on Docker
<pre>
<pre>
brb@galliumos:~$ tree mysql
$ nano docker-compose.yml
mysql
$ docker-compose up -d
├── conf.d
$ ls data # files in this directory will be shared between the container and the host
│   └── mysql.cnf
# Open http://localhost:3000
├── data
</pre>
│   ├── auto.cnf
If we remove 'ubuntu-mate', it will create alpine linux instead.  
│   ├── ca-key.pem
│   ├── ca.pem
│   ├── client-cert.pem
│   ├── client-key.pem
│   ├── ib_buffer_pool
│   ├── ibdata1
│   ├── ib_logfile0
│   ├── ib_logfile1
│   ├── ibtmp1
│   ├── mediawiki [error opening dir]
│   ├── mysql [error opening dir]
│   ├── performance_schema [error opening dir]
│   ├── private_key.pem
│   ├── public_key.pem
│   ├── server-cert.pem
│   ├── server-key.pem
│   ├── sys [error opening dir]
│   └── wikidb [error opening dir]
├── dumps
└── initdb.d


9 directories, 15 files
In the alpine linux ([https://www.tecmint.com/apk-command-examples/ 13 Apk Commands for Alpine Linux Package Management]), we can use the following commands to install a package. However, meld is not responding from my testing in the setup.
 
<pre>
brb@galliumos:~$ tree -L 2 mediawiki/
sudo apk update
mediawiki/
sudo apk add meld nano htop neofetch
├── config
│   └── LocalSettings.php
├── extensions
│   ├── Cite
│   ├── ConfirmEdit
│   ├── Gadgets
│   ├── ImageMap
│   ├── InputBox
│   ├── Interwiki
│   ├── LocalisationUpdate
│   ├── Nuke
│   ├── ParserFunctions
│   ├── PdfHandler
│   ├── Poem
│   ├── README
│   ├── Renameuser
│   ├── SpamBlacklist
│   ├── SyntaxHighlight_GeSHi
│   ├── TitleBlacklist
│   └── WikiEditor
└── images
    └── README
 
19 directories, 3 files
</pre>
</pre>
* https://linuxconfig.org/mediawiki-easy-deployment-with-docker-container (based on linuxconfig/mediawiki)
</li>
* http://collabnix.com/how-to-setup-mediawiki-in-seconds-using-docker/ (based on appcontainers/mediawiki)
<li>[https://www.kasmweb.com/ Kasm Workspaces], https://hub.docker.com/r/kasmweb/desktop/#!
* bitnami/mediawiki (too complicated)
* [https://youtu.be/go7n0FmNqh4 KASM: Isolated Disposable Remote Workspace Via Your Browser] (video)
* [https://www.packtpub.com/books/content/isolated-mediawiki-setup-raspberry-pi-using-docker-part-1 Isolated MediaWiki Setup on the Raspberry Pi using Docker, Part 1]
* [https://www.youtube.com/watch?v=hXkZVqqAg7c Kasm Workspaces: Your Solution for Remote Desktops?] (video)
* [https://youtu.be/_ur59HHoRGc?t=442 Desktop Apps in Docker Containers with Kasm Workspaces] (you cannot install any software there)
</li>
</ul>


== [https://www.bookstackapp.com/ BookStack] ==
= Podman =
BookStack is a simple, self-hosted, easy-to-use platform for organising and storing information.
* [https://podman.io/docs/installation Podman Installation Instructions]
** [https://ostechnix.com/install-podman-desktop-in-linux/ How To Install Podman Desktop In Linux]
** Raspberry Pi OS use the standard Debian's repositories, so it is fully compatible with Debian's arm64 repository. You can simply follow the steps for Debian to install Podman.
* [https://linuxhandbook.com/docker-vs-podman/ Podman vs docker]:
** One of the main differences between Podman and Docker is their architecture. Docker uses a client-server architecture with a central daemon that manages containers. In contrast, Podman is daemonless and uses a fork-exec model to manage containers.
** '''Podman is designed to run containers without requiring root privileges or the use of sudo'''. This is one of the key differences between Podman and Docker, as Docker requires root privileges to run containers.
** Both Podman and Docker are compatible with the Open Container Initiative (OCI) container specification, which means that they can run the same container images. However, '''Podman is more closely aligned with Kubernetes''' and its native container runtime, while Docker also works with its own orchestration tool, Docker Swarm.
** Podman provides several '''benefits''' over Docker. For example, Podman is daemon-less, which means that if the Docker daemon crashes, the containers are in an uncertain state. This is prevented by making Podman daemon-less. You can also use systemd to manage your containers with Podman, which gives you virtually unlimited configurability compared to Docker. Hooking Podman with systemd allows you to update running containers with minimal downtime and recover from any bad updates.
* Podman is a project from Red Hat
* [https://www.howtogeek.com/devops/getting-started-with-podman-desktop-an-open-source-docker-desktop-alternative/ Getting Started With Podman Desktop, an Open Source Docker Desktop Alternative]
* [https://phoenixnap.com/kb/podman-compose Podman Compose - Managing Containers]
:<syntaxhighlight lang='bash'>
pip3 install podman-compose
</syntaxhighlight>
:But it seems the compatibility is an issue even I tried a small example based on alpine image.
* Nginx example (works)
:<syntaxhighlight lang='bash'>
podman run -it --rm -d -p 8080:80 \
  --name web \
  -v /mnt/Podman/site-content:/usr/share/nginx/html \
  docker.io/libary/nginx
</syntaxhighlight>
* [https://developers.redhat.com/articles/2023/05/23/podman-desktop-now-generally-available Podman Desktop 1.0]: Local container development made easy.


https://github.com/solidnerd/docker-bookstack
* [https://dzone.com/articles/podman-for-docker-users Podman for Docker Users]
** Prerequisites
** Moving Images from Docker to Podman
** Creating a Basic Nuxt.js Project
** Building a Container Image for Your Nuxt.JS Project
** Push Your Podman Image to Quay.io
** '''Run Your Podman Image with Docker'''
** Creating Pods
** Generate a Kubernetes Pod Spec with Podman
** Create a Kubernetes Cluster with Kind (Optional)
** Deploying to Kubernetes
* Podman, Docker and Singularity all support OCI container format images.
* [https://appsilon.com/docker-vs-podman-vs-singularity/ Docker vs. Podman vs. Singularity: Which Containerization Platform is Best for R Shiny?]


== Media server ==
= Resource =
=== Airsonic media server ===
== Internet ==
https://airsonic.github.io/download/
* [https://opensource.com/article/19/11/microservices-cheat-sheet A guide to open source for microservices] from opensource.com
* [https://github.com/rainsworth/osip2019-containerisation-workshop/ Reproducible Research through Containerisation: Docker and Singularity] from rainsworth.
* [https://youtu.be/fqMOX6JJhGo Docker Tutorial for Beginners - A Full DevOps Course on How to Run Applications in Containers] from freeCodeCamp.org


My comment:  
== Books ==
* Added mp3/mp4 files are not recognized.
* The Docker Book: Containerization is the new virtualization
* Cannot play by web browsers
* Orchestrating Docker
* Special clients are needed:(
* Docker Hands on: Deploy, Administer Docker Platform
 
* [https://www.oreilly.com/library/view/learn-docker-/9781788997027/?afsrc=1 Learn Docker - Fundamentals of Docker] 18.x by Gabriel N. Schenker. Release Date: April 2018. https://github.com/fundamentalsofdocker/labs
=== Plex ===
* [https://www.amazon.com/Docker-Deep-Dive-Nigel-Poulton/dp/1521822808 Docker Deep Dive] by by Nigel Poulton. 2017
[https://github.com/plexinc/pms-docker Official Docker container for Plex Media Server]
* Master Docker (2nd, 2017) by Russ McKendrick, Scott Gallagher. Source code on [https://github.com/PacktPublishing/Mastering-Docker-Second-Edition github].


My comment:
== Blogs ==
* After starting, web browser cannot connect to it
* collabnix
** [https://dockerlabs.collabnix.com/ Welcome to Docker Labs]
** [https://dockerlabs.collabnix.com/intermediate/workshop/ Docker Workshop for Intermediate Users]
* http://blog.thoward37.me/articles/where-are-docker-images-stored/
* [http://www.ithome.com.tw/people/109065 Docker容器架構最大儲存挑戰是,你根本不知道資料存在何處]
* [https://blog.quickmediasolutions.com/2017/08/23/running-everything-in-docker.html Running Everything in Docker]. Docker Compose was used. Several applications were illustrated: mediawiki, Jenkins, Gogs & caddy-docker.


=== DockSTARTer: get started with home server apps running in Docker ===
== Tips/trouble shooting ==
* https://dockstarter.com/
* [https://nickjanetakis.com/blog/docker-tip-64-dealing-with-unexplained-docker-errors Dealing with Unexplained Docker Errors] and other [https://nickjanetakis.com/blog/tag/docker-tips-tricks-and-tutorials tips and tricks]
* [https://www.makeuseof.com/tag/setup-linux-media-server/ How to Set Up a Linux Media Server in Under One Hour]
* [https://medium.com/lucjuggery/running-a-container-with-a-non-root-user-e35830d1f42a Running a container with a non root user]
* [https://stackoverflow.com/a/49941240 Error: Cannot start container: port has already been allocated]. '''sudo service docker restart'''


== DNS server ==
== Play with Docker (PWD) ==
[https://linuxconfig.org/protecting-your-privacy-with-firefox-on-linux#h4-1-private-dns Private DNS server container]
* http://labs.play-with-docker.com/
 
* http://training.play-with-docker.com/, [https://training.play-with-docker.com/alacart/ A full list of labs]
[https://www.howtoforge.com/tutorial/how-to-deploy-a-dynamic-dns-server-with-docker-on-debian-10/ How to Deploy a Dynamic DNS Server with Docker on Debian 10]
* https://blog.docker.com/2017/07/best-way-learn-docker-free-play-docker-pwd/
 
* One session is 4 hours. Each session can have multiple instance. Each instance will be allocated 4GB mem. Each docker image is stored in memory.
== Pi-hole ==
* '''df -h''' shows there is only 4.7G space for /var/lib/docker directory
* https://hub.docker.com/r/pihole/pihole/, https://github.com/pi-hole/docker-pi-hole
* It is based on Alpine Linux. To install htop, do '''apk add htop'''. But '''htop''' command shows the resource from the host, not from the user's account.
* [https://pimylifeup.com/raspberry-pi-pi-hole/ Raspberry Pi Pi-Hole]: A Network Wide Ad-blocker
* '''ctrl + insert''' to copy and '''shift + insert''' to paste
* [https://homenetworkguy.com/how-to/install-pihole-on-raspberry-pi-with-docker-and-portainer/ Install Pi-hole on a Raspberry Pi with Docker and Portainer]
* [https://github.com/play-with-docker/play-with-docker/issues/238 connect to a play-with-docker instance]. Answer: You just need to create a random private key. [https://kostislab.blogspot.com/2019/03/play-with-play-with-docker-form-your.html Play with "Play with Docker" from your terminal!].
 
== KeeWeb: password manager compatible with KeePass ==
https://github.com/keeweb/keeweb
 
== Flask ==
[https://www.howtoforge.com/dockerizing-flask-application-using-docker-on-debian-10/ Dockerizing Flask Application Using Docker on Debian 10]
 
== ONLYOFFICE ==
* https://www.howtoforge.com/tutorial/how-to-install-onlyoffice-8.9.0-with-docker-on-ubuntu/
* https://www.howtoforge.com/tutorial/how-to-update-onlyoffice-to-version-95-with-docker/ This involves backing up and restoring MYSQL data.
 
== Gitlab ==
* https://docs.gitlab.com/omnibus/docker/
* https://hub.docker.com/r/gitlab/gitlab-ce/
* [https://www.howtoforge.com/tutorial/how-to-install-gitlab-server-with-docker-on-ubuntu-1804/ How to Install Gitlab Server with Docker on Ubuntu 18.04 LTS]
* [https://www.howtoforge.com/how-to-install-gitlab-with-docker-on-ubuntu-2004/ How to Install GitLab with Docker on Ubuntu 20.04 LTS]
 
== NAS & Docker ==
http://www.ithome.com.tw/tech/96500
 
Some examples
* GetLab專案管理工具
* Jenkins開發版本管理工具
 
== Draw.io for diagrams and more ==
https://github.com/jgraph/drawio/tree/master/etc/docker
<pre>
docker run -d --rm --name="draw" -p 8080:8080 -p 8443:8443 jgraph/drawio
</pre>
 
== Run a benchmark program on ARM devices ==
* [http://www.element14.com/community/people/markfink/blog/2015/02/05/using-docker-on-beaglebone-black?CMP=SOM-EUROPE-MFINK-BBBlack Beaglebone black] as an example. More device examples can be found on [http://www.testing-software.org/internet_of_things.html http://www.testing-software.org/].
 
== Set up a Mastodon Instance on Linux ==
[http://www.makeuseof.com/tag/social-network-set-mastodon-instance-linux/ Your Own Social Network: How to Set Up a Mastodon Instance on Linux]
 
== Bioinformatics tools (Tophat, BWA, et al) ==
https://github.com/dceoy/docker-bio
 
== cbioportal ==
* [https://docs.cbioportal.org/2.1.1-deploy-with-docker-recommended/docker Deploy with Docker]
* https://groups.google.com/g/cbioportal
* docker-compose up is OK, but it requires me to restart the container (I wasn't instructed to do that. Documents need to improve).
<ul>
<li>Import study does not work if we use the docker-compose method. It works if we use '''docker exec -it CONTAINER bash'''.  
{{Pre}}
[brb@localhost cbioportal-docker-compose]$ docker-compose run cbioportal metaImport.py -u http://cbioportal:8080 -s study/lgg_ucsf_2014/ -o
Creating cbioportal-docker-compose_cbioportal_run ... done
Starting validation...
 
CRITICAL: -: No meta files found in study/lgg_ucsf_2014/. Please make sure the directory is the path to the folder containing the files.
ERROR: -: Cancer type needs to be defined for a study. Verify that you have a study file and have defined the cancer type correctly.
ERROR: -: No valid study file detected
</pre>
See also https://github.com/cBioPortal/cbioportal/issues/7972
</li>
<li>The version number shown on http://localhost:8080 is v3.4.17 but the https://www.cbioportal.org/ website shows v3.5.3 (same as the current release version on [https://github.com/cBioPortal/cbioportal Github])</li>
<li>The 2nd field on the 1st row of '''data_cancer_type.txt''' contains the study name (blue color) shown on website </li>
<li>'''meta_study.txt ''' contains the study name (black color) shown on website </li>
<li>study_es_0 (breast, contains mRNA expression) and study_es_3 can be imported successfully when I use '''docker-compose exec -it CONTAINER bash''' and follow [https://docs.cbioportal.org/2.1.2-deploy-without-docker/load-sample-cancer-study Load sample cancer study].</li>
<li>lgg_ucsf_2014 study (brain, CNA) should use the correct path '''docker-compose run cbioportal metaImport.py -u http://cbioportal:8080 -s study/lgg_ucsf_2014/lgg_ucsf_2014/ -o''' </li>
<li>I have tested on CentOS 7 and Ubuntu 20.04 on KVM/QEMU w/ 2 cores, 2/4GB memory, 25GB storage. I can use host's terminal and browser for interaction (i.e. no VNC/Spice server needed).</li>
<li>[http://blog.thehyve.nl/blog/gene-panels-cbioportal-patient-view Gene panels in the cBioPortal patient view] 2020-01-07, and other blogs related to cbioportal from '''thehyve'''.
<ul>
<li>[http://blog.thehyve.nl/blog/simple-data-loading-into-cbioportal-with-docker Simple data loading into cBioPortal with Docker (Updated 2019)] </li>
<li>[http://blog.thehyve.nl/blog/cbioportal-setup-made-easy-using-docker cBioPortal setup made easy using Docker (Updated 2019)] </li>
</ul>
</li>
</ul>
 
== GIVE - Genomic Interactive Visualization Engine ==
https://github.com/Zhong-Lab-UCSD/Genomic-Interactive-Visualization-Engine/blob/master/tutorials/GIVE-Docker.md
 
== GeneTEFlow: Analysing gene and transposable elements expression from RNA-Seq data ==
[https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0232994 GeneTEFlow]: A Nextflow-based pipeline for analysing gene and transposable elements expression from RNA-Seq data. [https://github.com/zhongw2/GeneTEFlow Source code].
 
== [https://www.elastic.co/ Elastic Search] ==
* https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
* https://docs.docker.com/samples/library/elasticsearch/
* https://github.com/triestpa/guttenberg-search
 
== Webpage monitor tools ==
[https://github.com/statping/statping/wiki/Features statping]
 
= Orchestrator =
== [https://www.docker.com/kubernetes Kubernete] ==
* [https://www.zdnet.com/article/kubernetes-dropping-docker-is-not-that-big-of-a-deal/ Kubernetes dropping Docker is not that big of a deal]
* [https://www.freecodecamp.org/news/kubernetes-vs-docker-whats-the-difference-explained-with-examples/ Kubernetes VS Docker: What's the Difference? Explained With Examples]
* [https://kodekloud.com/p/kubernetes-for-the-absolute-beginners-hands-on Kubernetes For The Absolute Beginner - Hands On] from kodekloud.com
* [http://docker-k8s-lab.readthedocs.io/en/latest/ Docker Kubernetes Lab Handbook] by Peng Xiao. The github page is on https://github.com/xiaopeng163/docker-k8s-lab.
* [https://youtu.be/2G-A6Knjrvo Introducing Kubernetes] by Marc Sluiter. Kubenetes can be used to schedule, run and manage containers in a cluster of virtual or physical machines.
** Starting by Google in 2014
** Pods: smallest unit of deployment. small group of tightly coupled containers. share network and data volumes. routable IP address. multiple pods per node.
* [https://youtu.be/6xJwQgDnMFE Create a 2-node Kubernetes cluster in 10 minutes]
** [https://github.com/kubernetes/dashboard Kubernetes Dashboard]
* [https://youtu.be/_vHTaIJm9uY Kubernetes Webinar Series - Getting Started with Kubernetes]
** https://thenewstack.io/tutorial-configuring-ultimate-development-environment-kubernetes/
** https://github.com/janakiramm/Kubernetes-dev-env
* [https://www.oreilly.com/ideas/how-to-manage-docker-containers-in-kubernetes-with-java Deploying Java Applications with Docker and Kubernetes]
* [https://blog.alexellis.io/docker-for-mac-with-kubernetes/ First impressions: Docker for Mac with Kubernetes]
* [https://github.com/kubernetes/minikube Minikube]: run Kubernetes locally
** [https://www.howtoforge.com/tutorial/how-to-install-kubernetes-with-minikube-on-ubuntu-1804-lts/ How to Install Kubernetes with Minikube on Ubuntu 18.04 LTS]
** [https://opensource.com/article/19/12/kubernetes-namespaces Kubernetes namespaces for beginners].  You need to have Minikube, Helm, and the kubectl command line installed.
* [https://www.biorxiv.org/content/10.1101/488643v2?rss=1 Galaxy-Kubernetes integration: scaling bioinformatics workflows in the cloud] Moreno et al 2019
* [https://seandavi.github.io/2019/03/helm-on-google-kubernetes-engine/ Helm on Google Kubernetes Engine] by Sean Davis
* [https://microk8s.io/ MicroK8s]
** [https://github.com/CanonicalLtd/multipass Multipass]
** [https://itnext.io/microk8s-on-macos-98f1de3aa63e MicroK8s on MacOS]
* [https://www.howtoforge.com/resource-limits-in-kuberenetes/ Resource Limits in Kuberenetes]
* [https://github.com/seandavi/Orchestra?s=09 Orchestra] and [https://eurobioc2020.bioconductor.org/workshops?s=09 European Bioconductor Meeting 2020]
 
== Kubernete vs Docker swarm ==
* https://platform9.com/blog/kubernetes-docker-swarm-compared/
* [https://www.ithome.com.tw/news/124290 DigitalOcean容器應用開發大調查:近半數開發者已採用]
 
== k3s: Lightweight Kubernetes ==
[https://opensource.com/article/20/3/kubernetes-raspberry-pi-k3s Run Kubernetes on a Raspberry Pi with k3s]
 
= Other containers =
== [https://sylabs.io/ Singularity] and HPC systems ==
* Old URL at [http://singularity.lbl.gov/ singularity.lbl.gov]
* Singularity enables users to have full control of their environment; Singularity containers let users run applications in a Linux environment of their choosing. No 'sudo' is needed in general unless you want to build a container from a recipe.
* Containers are more like an executable file for you to use
* Containers are stored under the current location. It does not have a centre location (like /etc/default/docker if we use docker) to store images.
* Can convert Docker containers to Singularity and run containers directly from Docker Hub
* These bind points cannot be created unless the path already exists within the container. To ensure access to these storage spaces and remedy bind point errors, create these directories in the %post section of your Bootstrap file.
* [https://singularity-hub.org/ Singularity Hub]
 
Ref:
* [https://hpc.nih.gov/apps/singularity.html Singularity] at NIH/Biowulf. It needs to run 'module load singularity' first to load related commands.
* NIH-HPC
** [https://singularity-tutorial.github.io/ Tutorial] and its [https://github.com/NIH-HPC/Singularity-Tutorial Github] source ('''valuable''')
** [https://github.com/nih-byob/presentations/tree/master/2019/01_bioinformatics_tips Bioinformatics advice I wish I learned 10 years ago]
** [https://github.com/NIH-HPC/singularity-def-files NIH HPC Singularity Definition Files]
* [https://tin6150.github.io/psg/blogger_container_hpc.html DOCKER VS SINGULARITY VS SHIFTER VS UGE CONTAINER EDITION]
* [https://github.com/data61/gossamer xenome] case where the build instruction is good for Ubuntu 16.04. If we try to build under Ubuntu 14.04 (tested using docker run), we will get an error message '' CMake 3.1 or higher is required.  You are running version 2.8.12.2''.
* https://github.com/NIH-HPC/singularity-examples/blob/master/rnaseq/Singularity.rnaseq
 
'''Comparison of docker and singularity commands:'''
{| class="wikitable"
|-
! [https://docs.docker.com/edge/engine/reference/commandline/docker/ docker] !! [http://singularity.lbl.gov/user-guide singularity]
|-
| $ docker pull ubuntu:latest <br/>$ docker pull broadinstitute/gatk3:3.8-0 || $ singularity pull docker://ubuntu:latest<br/>$ singularity pull docker://broadinstitute/gatk3:3.8-0
|-
| $ docker build || $ singularity build
|-
| $ docker shell (not exist) || $ singularity shell docker://broadinstitute/gatk3-3.8-0<br/>  $ singularity shell gatk3-3.8-0.img<br/> > ls  # the default location depends on the host system<br/>
> ls /usr  # this is from the container <br/><br/>$ singularity shell --bind ~/Downloads:/mnt XXX.img<br/>$ singularity shell docker://ubuntu:latest<br/># container is ephemeral
|-
| $ docker run --name test -it ubuntu date <br/><br/># <span style="color: red">The next example is similar to 'singularity exec'</span><br/>$ docker run --rm -i -t \<br/>  -v $(pwd):/usr/my_data \<br/>  broadinstitute/gatk3:3.8-0 \<br/>  bash /usr/my_data/myscript.sh || $ singularity run gatk3-3.8-0.img date
|-
| $ docker run --name ubuntu_bash --rm -i -t ubuntu bash<br/>$ docker exec -d ubuntu_bash touch /tmp/execWorks || # <span style="color: red">Most useful</span><br>$ singularity exec gatk3-3.8-0.img java -version <br/>$ singularity exec xxx.img cat /etc/*release<br>$ singularity exec docker://rocker/tidyverse:latest R <br/>$ singularity exec docker://rocker/tidyverse:latest Rscript myScript.R<br/>
|}
 
=== Cache ===
When we run ''singularity exec docker://rocker/tidyverse:latest R'', it will save something in the cache in our system.
 
It seems to be OK after I manually delete the directory $HOME/.singularity (tested in Biowulf).
 
=== RStudio ===
* [https://rviews.rstudio.com/2017/03/29/r-and-singularity/ R and Singularity] by Bryan Lewis
* [https://archive.sylabs.io/2018/10/rstudio-ide-and-server/ RStudio – IDE and Server powered by Singularity] by Jason Stover
* https://github.com/nickjer/singularity-rstudio and https://github.com/nickjer/singularity-r
* https://www.rocker-project.org/use/singularity/. It also includes how to run a SLURM job script.
* https://njstem.wordpress.com/2018/08/02/r-script-seurat-with-a-singularity-container-using-slurm/
* https://journal.r-project.org/archive/2017/RJ-2017-065/RJ-2017-065.pdf
 
<syntaxhighlight lang='bash'>
$ singularity exec docker://rocker/tidyverse:latest R
$ singularity exec docker://rocker/tidyverse:latest Rscript myScript.R
</syntaxhighlight>
 
== [https://github.com/scanon/shifter Shifter] ==
 
= [https://conda.io/docs/ Conda] =
* [https://hub.docker.com/r/continuumio/anaconda3 Getting started with Anaconda & Docker] using continuumio/anaconda3
* [https://www.quora.com/What-is-the-difference-between-conda-and-Docker What is the difference between conda and Docker?]
 
== Bioconda ==
* https://bioconda.github.io/
* https://anaconda.org/bioconda
* https://www.biorxiv.org/content/early/2017/10/27/207092
* https://zhuanlan.zhihu.com/p/25085567
* [http://slides.com/olexschoen/snakemake-bioconda#/ Snakemake: Reproducible and scalable data analysis]
* https://chrisconlan.com/installing-r-python-anaconda-biologists/. Note that the R version is 3.4.2 but the latest one should be 3.4.3.
 
Using docker to install conda (https://conda.io/docs/user-guide/tutorials/index.html)
<syntaxhighlight lang='bash'>
$ docker run -t -i --name test --net=host ubuntu bash
# apt-get update
# apt-get install -y wget bzip2 python
# wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
# wget https://repo.continuum.io/archive/Anaconda2-5.1.0-Linux-x86_64.sh
# bash Miniconda2-latest-Linux-x86_64.sh
# bash Anaconda2-5.1.0-Linux-x86_64.sh
# exit
 
$ docker start test
$ docker exec -i -t test bash
# conda list  # WORKS!
# conda config --add channels r
# conda config --add channels defaults
# conda config --add channels conda-forge
# conda config --add channels bioconda
# conda install bwa  (Segmentation fault. Core dumped)
# which bwa
/root/anaconda2/bin/bwa
# conda install r  (Only get 3.4.2 but the latest is 3.4.3.)
# conda install bowtie
# bowtie --version
# conda install gatk (https://bioconda.github.io/recipes/gatk/README.html)
  (Due to license restrictions, this recipe cannot distribute and install GATK directly)
  (R is downgraded to 3.2.2:( )
  (Segmentation fault. Core dumped)
# exit
# docker stop test
# docker rm test
</syntaxhighlight>
 
Get miniconda image instead of using a Ubuntu image
<syntaxhighlight lang='bash'>
$ docker pull continuumio/miniconda
$ docker run -i -t continuumio/miniconda /bin/bash
# conda install r  (get 3.4.2)
# conda config --add channels bioconda
# conda install bwa  (OK, no error)
# conda install gatk  (R was downgraded to 3.2.2, install openjdk 8.0.121)
# which gatk
/opt/conda/bin/gatk
# gatk -h
GATK jar file not found. Have you run "gatk-register"?
</syntaxhighlight>
Issues:
* R version is not up to date
* '''So the problem is installing GATK requires an installation of R and the current R was affected.'''
 
= [https://coreos.com/ CoreOS] =
* Redhat has acquired CoreOS 1/31/2018.
* [https://coreos.com/os/eol/ End-of-life announcement for CoreOS Container Linux]
 
== Installation ==
* [https://coreos.com/os/docs/latest/quickstart.html Quick Start]
* [https://youtu.be/vy6hWsOuCh8  Installing CoreOS on a bare metal or virtual machine] from the user Format C: in youtube
* [https://youtu.be/M0TYqVDnNAI Assigning Static IP to a CoreOS Server]
* [https://www.youtube.com/watch?v=6LOiIH8s3xM Mounting storage in CoreOS]
* [https://www.youtube.com/watch?v=InLmXWgPzBs Installing, Deploying Docker containers with Fig / Docker Compose on boot2docker or CoreOS]
* [https://www.youtube.com/watch?v=S9kfP2WYV_I Running a Private Docker Registry Container]
 
We first boot a liveCD from any OS (CentOS works but Ubuntu 16.04 gave errors). In Virtualbox, we choose 'Red Hat' if we use CentOS.
 
Once the VM is created. We go to the settings. Create a bridged network or host-only network first (even we can get files from the host without creating a host-only network). Storage: choose CentOS-7.
 
# Get the [https://coreos.com/os/docs/latest/installing-to-disk.html install script] from Github and create <'''coreos_install.sh'''> and chmod +x
# create <'''cloud-config.yaml'''> file which will include ssh_authorized_keys generated from another machine. It should also contain a new token for the cluster from https://discovery.etcd.io/new.
# ls -l /dev/sd*
# run '''sudo ./coreos_install.sh -d /dev/sda -C stable -c cloud-config.yaml'''. It will download the latest stable CoreOS, install to the HD
# Don't leave the VM or it will freeze. Issue '''sudo shutdown -h now''' once we see the word 'Success' at the last line of the output.
# Remove CentOS from the VM storage. Boot the coreOS VM.
 
The new screen shows ''corebm1 login'' with an IP. Go back to another machine and type '''ssh -i /tmp/CoreOSBM_rsa [email protected]'''. Inside CoreOS, we can type '''docker images'''.
 
The '''cloud-config.yaml'' file has to follow the format in https://coreos.com/os/docs/latest/cloud-config.html. Use the online validator https://coreos.com/validate/ to correct. At first I use the file from the youtube video. There is no error coming out when I ran the installation script. But I cannot connect to coreOS. The cloud-config.yaml file I use is (pay attention to '-', double quotes and indent characters)
<pre>
#cloud-config
#
# set hostname
hostname: CoreBM1
 
# Set ssh key
ssh_authorized_keys:
  - "ssh-rsa AAAAB3 ..... brb@T3600"
 
coreos:
  etcd:
    discovery: "https://discovery.etcd.io/d3e95 .... "
# sudo ./installos -d /dev/sda -C stable -c cloud-config.yaml
</pre>
 
== CoreOS exploration ==
<syntaxhighlight lang='bash'>
brb@T3600 /tmp $ ssh -i /tmp/id_rsa [email protected]
Enter passphrase for key '/tmp/id_rsa':
CoreOS stable (1010.6.0)
core@CoreBM1 ~ $
core@CoreBM1 ~ $ docker images
REPOSITORY          TAG                IMAGE ID            CREATED            SIZE
core@CoreBM1 ~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.4G    0  1.4G  0% /dev
tmpfs          1.4G    0  1.4G  0% /dev/shm
tmpfs          1.4G  340K  1.4G  1% /run
tmpfs          1.4G    0  1.4G  0% /sys/fs/cgroup
/dev/sda9        18G  23M  17G  1% /
/dev/sda3      985M  589M  345M  64% /usr
tmpfs          1.4G    0  1.4G  0% /media
/dev/sda1      128M  37M  92M  29% /boot
tmpfs          1.4G    0  1.4G  0% /tmp
/dev/sda6      108M  52K  99M  1% /usr/share/oem
core@CoreBM1 ~ $ free -m
            total      used      free    shared    buffers    cached
Mem:          2713        187      2525          0          9        109
-/+ buffers/cache:        68      2644
Swap:            0          0          0
core@CoreBM1 ~ $ lsb_release -a
-bash: lsb_release: command not found
core@CoreBM1 ~ $ docker pull ubuntu
Using default tag: latest
latest: Pulling from library/ubuntu
f069f1d21059: Pull complete
ecbeec5633cf: Pull complete
ea6f18256d63: Pull complete
54bde7b02897: Pull complete
Digest: sha256:bbfd93a02a8487edb60f20316ebc966ddc7aa123c2e609185450b96971020097
Status: Downloaded newer image for ubuntu:latest
core@CoreBM1 ~ $ docker images
REPOSITORY          TAG                IMAGE ID            CREATED            SIZE
ubuntu              latest              0f192147631d        7 days ago          132.7 MB
core@CoreBM1 ~ $
</syntaxhighlight>
 
== CoreOS cluster discovery ==
https://coreos.com/os/docs/latest/cluster-discovery.html
 
== etcd ==
* https://coreos.com/etcd/


== fleet ==
* Some applications I've tested.
* https://coreos.com/using-coreos/clustering/
** webtop (OK)
** r-base:3.6.3, r-base:4.1.0, r-base:4.1.1 (OK)
** r-base:4.1.2, r-base:4.2.0 (ERROR: R_HOME ('/usr/lib/R') not found). Maybe the docker version there is too old.


= Resource =
== Alternatives ==
== Internet ==
[https://www.makeuseof.com/best-docker-alternatives/ The 9 Best Docker Alternatives for Container Management]
* [https://opensource.com/article/19/11/microservices-cheat-sheet A guide to open source for microservices] from opensource.com
* [https://github.com/rainsworth/osip2019-containerisation-workshop/ Reproducible Research through Containerisation: Docker and Singularity] from rainsworth.
* [https://youtu.be/fqMOX6JJhGo Docker Tutorial for Beginners - A Full DevOps Course on How to Run Applications in Containers] from freeCodeCamp.org
 
== Books ==
* The Docker Book: Containerization is the new virtualization
* Orchestrating Docker
* Docker Hands on: Deploy, Administer Docker Platform
* [https://www.oreilly.com/library/view/learn-docker-/9781788997027/?afsrc=1 Learn Docker - Fundamentals of Docker] 18.x by Gabriel N. Schenker. Release Date: April 2018. https://github.com/fundamentalsofdocker/labs
* [https://www.amazon.com/Docker-Deep-Dive-Nigel-Poulton/dp/1521822808 Docker Deep Dive] by by Nigel Poulton. 2017
* Master Docker (2nd, 2017) by Russ McKendrick, Scott Gallagher. Source code on [https://github.com/PacktPublishing/Mastering-Docker-Second-Edition github].
 
== Blogs ==
* collabnix
** [https://dockerlabs.collabnix.com/ Welcome to Docker Labs]
** [https://dockerlabs.collabnix.com/intermediate/workshop/ Docker Workshop for Intermediate Users]
* http://blog.thoward37.me/articles/where-are-docker-images-stored/
* [http://www.ithome.com.tw/people/109065 Docker容器架構最大儲存挑戰是,你根本不知道資料存在何處]
* [https://blog.quickmediasolutions.com/2017/08/23/running-everything-in-docker.html Running Everything in Docker]. Docker Compose was used. Several applications were illustrated: mediawiki, Jenkins, Gogs & caddy-docker.
 
== Tips/trouble shooting ==
* [https://nickjanetakis.com/blog/docker-tip-64-dealing-with-unexplained-docker-errors Dealing with Unexplained Docker Errors] and other [https://nickjanetakis.com/blog/tag/docker-tips-tricks-and-tutorials tips and tricks]
* [https://medium.com/lucjuggery/running-a-container-with-a-non-root-user-e35830d1f42a Running a container with a non root user]
 
== Play with Docker (PWD) ==
* http://labs.play-with-docker.com/
* http://training.play-with-docker.com/, [https://training.play-with-docker.com/alacart/ A full list of labs]
* https://blog.docker.com/2017/07/best-way-learn-docker-free-play-docker-pwd/


== Serverless computing ==
== Serverless computing ==

Revision as of 13:16, 28 April 2024

Official web page http://docker.io.

Docker is both a client and a server: the server is a daemon that runs on Linux. The normal approach was that you used docker on the same server the daemon was running on - however it was possible to connect the docker client to a remote docker daemon.

Installation

Which OS to install?

Containers vs virtual machines

KubeVirt

OS containers vs application containers

Differences:

  • OS containers: LXC, OpenVZ, Linux VServer, BSD Jails and Solaris zones. The container acts as VPS.
  • App containers: Docker, Rocket. The container acts as an application.

Current release version

Ubuntu x86 and Mint

One-line script

https://github.com/docker/docker-install, https://docs.docker.com/engine/install/ubuntu/, https://twitter.com/portainerio/status/1650171336864550912

Note that 1) the one-liner is a huge security issue. 2) but how will you add the current user to docker group and then logout and log back in.

$ curl -fsSL https://get.docker.com | bash
# Executing docker install script, commit: e5543d473431b782227f8908005543bb4389b8de
+ sudo -E sh -c 'apt-get update -qq >/dev/null'
[sudo] password for brb: 
+ sudo -E sh -c 'DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null'
+ sudo -E sh -c 'install -m 0755 -d /etc/apt/keyrings'
+ sudo -E sh -c 'curl -fsSL "https://download.docker.com/linux/debian/gpg" | gpg --dearmor --yes -o /etc/apt/keyrings/docker.gpg'
gpg: WARNING: unsafe ownership on homedir '/home/brb/.gnupg'
+ sudo -E sh -c 'chmod a+r /etc/apt/keyrings/docker.gpg'
+ sudo -E sh -c 'echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian bullseye stable" > /etc/apt/sources.list.d/docker.list'
+ sudo -E sh -c 'apt-get update -qq >/dev/null'
+ sudo -E sh -c 'DEBIAN_FRONTEND=noninteractive apt-get install -y -qq docker-ce docker-ce-cli containerd.io docker-compose-plugin docker-ce-rootless-extras docker-buildx-plugin >/dev/null'
+ sudo -E sh -c 'docker version'
Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:08:17 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:17 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.26
  GitCommit:        3dd1e886e55dd695541fdcd67420c2888645a495
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0


To run Docker as a non-privileged user, consider setting up the Docker daemon in rootless mode for your user: dockerd-rootless-setuptool.sh install Visit https://docs.docker.com/go/rootless/ to learn about rootless mode. To run the Docker daemon as a fully privileged service, but granting non-root users access, refer to https://docs.docker.com/go/daemon-access/ WARNING: Access to the remote API on a privileged Docker daemon is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' documentation for details: https://docs.docker.com/go/attack-surface/
$ sudo groupadd docker $ sudo usermod -aG docker $USER $ newgrp docker $ docker run hello-world

This can install docker but you still need "sudo" to run it. See Linux post-installation steps for Docker Engine: 1) Manage Docker as a non-root user, and 2) Configure Docker to start on boot with systemd.

Docker Desktop

Without sudo, Post-installation

To use docker without sudo, follow the instruction on the official guide.

# Add the docker group if it doesn't already exist.
# sudo groupadd docker

# Add your user to the docker group.
sudo usermod -aG docker $USER

# Log out and log in

After running this command, you need to log out and log back in for the changes to take effect. This is because group membership is determined at login time. When you log in, the system reads the group membership information and assigns the appropriate permissions to your user account.

Upgrade Docker Desktop

It seems it does not affect running containers (e.g. RStudio on Mac).

Is it fine to upgrade Docker-ce while a container is running?

Doesn't matter. Your system will stop the container if you update docker.

Is there a way to hibernate a docker container

Live restore

Rate limits for GitHub Apps

Rate limits for GitHub Apps

When I tried several times of docker build, I finally got a message

Downloading GitHub repo XXX/XXXXX@HEAD
Error: Failed to install 'unknown package' from GitHub:
  HTTP error 403.
  API rate limit exceeded for XXX.XX.XXX.X. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)

  Rate limit remaining: 0/60
  Rate limit reset at: 2021-04-12 20:32:28 UTC

  To increase your GitHub API rate limit
  - Use `usethis::browse_github_pat()` to create a Personal Access Token.
  - Use `usethis::edit_r_environ()` and add the token as `GITHUB_PAT`.
Execution halted

CentOS

https://docs.docker.com/engine/installation/linux/docker-ce/centos/

Boot2Docker

For Windows and OS operation systems, we have to use Boot2Docker. Boot2Docker is a local virtual machine with its own network interface and IP address. To find the Boot2Docker IP address you can check the value of the DOCKER_HOST environment variable. You're be prompted to set this variable when you start or install Boot2Docker the first time. You can find the IP address by running boot2docker ip command.

Note that since Windows and OS X don't share a file system as Linux, the command 'docker run' with '-v' flag to mount a local directory into a Docker container will not work with Boot2Docker release prior to 1.3. The support for volumes is now available for OS X but is not yet present for Windows with the release of Boot2Docker 1.3.

Windows

Note many of the information here have not been updated.

Docker can be run on Windows 10 Pro as a native application; see

The information below is based on running Docker on Windows 7.1 and 8. Your processor needs to support hardware virtualization.

  • Windows Installer includes msys-git, Virtualbox, Boot2Docker-cli management tool and Boot2Docker ISO.
  • Installation instruction for Windows OS. It will install Boot2Docker management tool with the boot2docker iso (based on Tiny Core Linux), Virtualbox and MYSYS-git UNIX tools.
  • Docker needs Admin right to be installed. However, Virtualbox can be installed by user's account.
  • If the installer detects a version of VirtualBox installed, the VirtualBox checkbox will not be checked by default (Windows OS). The VirtualBox cannot be used anymore after updating my VB from 4.3.18 to 4.3.20. The error may be related to Windows update according to Virtualbox forum.
Error in supR3HardenedWinReSpawn
  • Note that boot2docker cannot be installed/run inside a Windows guest machine. See this post and my Virtualbox wiki here. If we try to launch boot2docker-vm from Virtualbox, we will see a message "This kernel requires an x86-64 CPU, but only detected an i686 CPU."
  • After I switch back to an old version of virtualbox, every thing works again. I can even install Docker successfully.
    • Boot2Docker Start icon cannot be run directly because Notepad++ will automatically open it. A possible solution is to go to control panel and change default program for .sh file from Notepad++ to C:\Program Files (x86)\Git\bin\bash.exe.
    • The above step does not work well since a terminal appears and disappears quickly.
    • A working approach is to open Git Bash from Start menu. And run /c/Program Files/Boot2DockerforWindows/start.sh (or boot2docker start or boot2docer init)
    • A new VM called 'boot2docker-vm' will be created (we can open VirtualBox Manager to check). But I got an error error in run: Failed to start machine "boot2docker-vm" (run again with -v for details). The VM has an error on Network>Adapter2>VirtualBox Host-Only Ethernet Adapter #2. So I open the setting of <boot2docker-vm>, go to Network > Adapter 2 and change the dropdown list of Name from VirtualBox Host-Only Ethernet Adapter #2 to VirtualBox Host-Only Ethernet Adapter.
    • Now it works either I directly click boot2docker-vm VM from VB Manager or use the command start.sh from Git Bash.

Boot2docker-vm.png

$ # boot2docker is in the PATH variable, so there is not need to cd to the folder.
$ boot2docker start
initializing...
Virtual machine boot2docker-vm already exists

starting...
Waiting for VM and Docker daemon to start...
........o
Started.
Writing c:\Users\brb\.boot2docker\certs\boot2docker-vm\ca.pem
Writing c:\Users\brb\.boot2docker\certs\boot2docker-vm\cert.pem
Writing c:\Users\brb\.boot2docker\certs\boot2docker-vm\key.pem
Docker client does not run on Windows for now. Please use
    "c:\Program files\Boot2Docker for Windows\boot2docker.exe" ssh
to SSH into the VM instead.


192.168.56.101
connecting...
                        ##        .
                  ## ## ##       ==
               ## ## ## ##      ===
           /""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
           \______ o          __/
             \    \        __/
              \____\______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.4.1, build master : 86f7ec8 - Tue Dec 16 23:11:29 UTC 2014

Docker version 1.4.1, build 5bc2ff8
docker@boot2docker:~$ docker
Usage: docker [OPTIONS] COMMAND [arg...]

A self-sufficient runtime for linux containers.

Options:
  --api-enable-cors=false                      Enable CORS headers in the remote
 API
  -b, --bridge=""                              Attach containers to a pre-existi
ng network bridge
...
Run 'docker COMMAND --help' for more information on a command.
docker@boot2docker:~$ docker run hello-world
Unable to find image 'hello-world:latest' locally
hello-world:latest: The image you are pulling has been verified
511136ea3c5a: Pull complete
31cbccb51277: Pull complete
e45a5af57b00: Pull complete
Status: Downloaded newer image for hello-world:latest
Hello from Docker.
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (Assuming it was not already locally available.)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

For more examples and ideas, visit:
 http://docs.docker.com/userguide/

docker@boot2docker:~$ ls
boot2docker, please format-me
docker@boot2docker:~$ pwd
/home/docker
docker@boot2docker:~$ ls /
bin/     dev/     home/    lib/     mnt/     proc/    run/     sys/     usr/
c/       etc/     init     linuxrc  opt/     root/    sbin/    tmp      var/

docker@boot2docker:~$ docker run hello-world
Hello from Docker.
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (Assuming it was not already locally available.)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

For more examples and ideas, visit:
 http://docs.docker.com/userguide/
docker@boot2docker:~$
docker@boot2docker:~$
docker@boot2docker:~$
docker@boot2docker:~$ docker run -it ubuntu bash
Unable to find image 'ubuntu:latest' locally
ubuntu:latest: The image you are pulling has been verified
53f858aaaf03: Pull complete
837339b91538: Pull complete
615c102e2290: Pull complete
b39b81afc8ca: Pull complete
511136ea3c5a: Already exists
Status: Downloaded newer image for ubuntu:latest


root@ea7e3289a01a:/# pwd
/
root@ea7e3289a01a:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs           19G  269M   17G   2% /
none             19G  269M   17G   2% /
tmpfs          1005M     0 1005M   0% /dev
shm              64M     0   64M   0% /dev/shm
/dev/sda1        19G  269M   17G   2% /etc/hosts
tmpfs          1005M     0 1005M   0% /proc/kcore
root@ea7e3289a01a:/# ls
bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
boot  etc  lib   media  opt  root  sbin  sys  usr
root@ea7e3289a01a:/# exit
exit


docker@boot2docker:~$ pwd
/home/docker
docker@boot2docker:~$ ls
boot2docker, please format-me
docker@boot2docker:~$ exit
[Press any key to exit]

brb@NCI-01825357 /c/Program files/Boot2Docker for Windows
$ boot2docker down

brb@NCI-01825357 /c/Program files/Boot2Docker for Windows
$
$ boot2docker --help
Usage: c:\Program files\Boot2Docker for Windows\boot2docker.exe [<options>] <command> [<args>]

Boot2Docker management utility.

Commands:
   init                Create a new Boot2Docker VM.
   up|start|boot       Start VM from any states.
   ssh [ssh-command]   Login to VM via SSH.
   save|suspend        Suspend VM and save state to disk.
   down|stop|halt      Gracefully shutdown the VM.
   restart             Gracefully reboot the VM.
   poweroff            Forcefully power off the VM (may corrupt disk image).
   reset               Forcefully power cycle the VM (may corrupt disk image).
   delete|destroy      Delete Boot2Docker VM and its disk image.
   config|cfg          Show selected profile file settings.
   info                Display detailed information of VM.
   ip                  Display the IP address of the VM's Host-only network.
   shellinit           Display the shell commands to set up the Docker client.
   status              Display current state of VM.
   download            Download Boot2Docker ISO image.
   upgrade             Upgrade the Boot2Docker ISO image (restart if running).
   version             Display version information.

Options:
      --basevmdk="": Path to VMDK to use as base for persistent partition
      --clobber=false: overwrite Docker client binary on boot2docker upgrade
      --dhcp=true: enable VirtualBox host-only network DHCP.
      --dhcpip=192.168.59.99: VirtualBox host-only network DHCP server address.
....
  -v, --verbose=false: display verbose command invocations.
      --vm="boot2docker-vm": virtual machine name.
      --waittime=300: Time in milliseconds to wait between port knocking retries during 'start'
error in run: config error: pflag: help requested

brb@NCI-01825357 /c/Program files/Boot2Docker for Windows

The big picture


                           start.sh                      docker run -it ubuntu bash
Git Bash Git Bash         ---------->  boot2docker-vm       ------------->   ubuntu
                                   docker@boot2docker:
   <-------               <----------                       <------------- 
   boot2docker down           exit                                 exit
   (shutdown boot2docker) (boot2docker-vm is still on)
    |
    |
    |  boot2docker up (start boot2docker)
    |
    |  boot2docker ssh (log into docker acct)
    |
    v
   boot2docker-vm
   docker@boot2docker

Increase boot2docker vmdk space

https://docs.docker.com/articles/b2d_volume_resize/

Install utilities in Boot2docker VM

http://blog.tutum.co/2014/11/05/how-to-use-docker-on-windows/

For example, to install cifs-utils,

wget http://distro.ibiblio.org/tinycorelinux/5.x/x86/tcz/cifs-utils.tcz
tce-load -i cifs-utils.tcz

WSL2

Mac

  • https://docs.docker.com/desktop/mac/
  • Alternatives to Docker Desktop for Mac? Rancher is recommended. 2022-06-08
  • Vagrant method. If you have Mac, you don't have to use boot2docker (iso & its management tool). You can use other Linux which comes with docker pre-installed. See this post.

Raspberry Pi

ARM architeture from hub.docker.com

curl -sSL https://get.docker.com | sh
  • UDOO Quad running Armbian 20.04
    • The instruction on official Docker website does not work
    • The curl command method above does not work
    • sudo apt-get install -y docker.io works (docker -v shows it is 19.03.8). After that, run sudo usermod -aG docker $USER and log out/in.
  • See Odroid magazine 2015 January and 2015 February. Note that the current versions of Docker and Docker Hub are not aware of the architecture for which the image has been built. All standard images are intended for the x86 architecture, and the autobuild feature offered by the Docker registry is only available for x86.
  • NVIDIA Jetson Nano Developer Kit - Introduction, Redis running inside Docker container on NVIDIA Jetson Nano
sudo apt install curl
curl -sSL https://get.docker.com/ | sh

docker-compose

Some examples*

Not I use the arm64 image on my Pi3b+.

Images from https://www.linuxserver.io/. Some indices include number of pulls and stars.

List of tz database time zones

Portainer. The port number is 9000. Note the stack will be deployed using the equivalent of docker-compose. Only Compose file format version 2 is supported at the moment.

Samba. Tested on iOS, Ubuntu & Windows 10.

mkdir -p /mnt/usb/share/{data,backups}
mkdir /mnt/usb/share/data/{alice,bob,documents}
touch /mnt/usb/share/backups/backupsfile
touch /mnt/usb/share/data/bob/bobfile
touch /mnt/usb/share/data/documents/documentfile

docker run -d -p 445:445 \
  -v /mnt/usb/share/data:/share/data \
  -v /mnt/usb/share/backups:/share/backups \
  --name rpi-samba trnape/rpi-samba \
  -u "alice:abc123" \
  -u "bob:secret" \
  -u "guest:guest" \
  -s "Backup directory:/share/backups:rw:alice,bob" \
  -s "Bob (private):/share/data/bob:rw:bob" \
  -s "Documents (readonly):/share/data/documents:ro:guest,alice,bob" 

On Windows, 1) right click on 'This PC' and choose 'Add a network location'. 2) type \\192.168.1.249\ and the dropdown list will populate all available folders. 3) choose the one (e.g. Bob) and then enter the credential. Done. On Ubuntu, just type smb://192.168.1.249/. It will then populate the available folders.

Nginx

mkdir -p /mnt/usb/docker-nginx/html
echo "hello world" >> /mnt/usb/docker-nginx/html/index.html
nano /mnt/usb/docker-nginx/html/sharefile
docker run --name rpi-nginx -p 8086:80 \
  --restart always \
  -v /mnt/usb/docker-nginx/html:/usr/share/nginx/html \
  -d nginx:stable-alpine

# Or a stack file
version: '2'
services:
    nginx:
        container_name: rpi-nginx
        ports:
            - '8086:80'
        restart: always
        volumes:
            - '/mnt/usb/docker-nginx/html:/usr/share/nginx/html'
        image: nginx:stable-alpine

Note consider to use a samba share folder (see above) as a nginx document root.

cp /mnt/usb/docker-nginx/html/* /mnt/usb/share/data/bob/
rm -rf /mnt/usb/docker-nginx/html
ln -s /mnt/usb/share/data/bob/ /mnt/usb/docker-nginx/html

Rpi-monitor. I need to change /dev/vcsm to /dev/vcsm-cma. But the temperature part is not working. I am using 64-bit Raspberry Pi OS and it does not show attached USB disks. The port number is 8888.

code-server

---
version: "2.1"
services:
  code-server:
    image: ghcr.io/linuxserver/code-server
    container_name: code-server
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/New_York
      - PASSWORD=password #optional
      - SUDO_PASSWORD=password #optional
    volumes:
      - /mnt/usb/code-server:/config
    ports:
      - 8443:8443
    restart: unless-stopped

mstream Music streaming. Works great.

emby does not work on arm64. It works on x86 though. Even I copy a mp4 file to movies directory the movie does not show up:(

version: '2.1'
services:
    embyserver:
        container_name: emby
        network_mode: bridge
        restart: always
        environment:
            - VERSION=latest
            - UID=1000
            - GID=1000
            - TZ=America/Denver
        volumes:
            - /media/crucial/emby/config:/config
            - /media/crucial/emby/tv:/mnt/tv
            - /media/crucial/emby/movies:/mnt/movies
        ports:
            - 8096:8096            
        image: 'emby/embyserver:latest'

jellyfin Jellyfin is descended from Emby's 3.5.2 release and ported to the .NET Core framework to enable full cross-platform support. How to Install Jellyfin on Docker with Portainer

plex We can access the plex server via http://IP:32400/web. Note that in the first server setup, we need to add Library' by choosing the new library name (eg Other Videos) shown on plex & the data source (eg /data) so our own media can be found. After we added new media files we can rescan by clicking the vertical 3 dots icon and selecting scan library files. Pi3b+ is still a little weak since I can see all threads are busy when I played a mp4 file.

mkdir -p /mnt/usb/plex/{config,data}
cp FILENAme.mp4 /mnt/usb/plex/data
docker run \
  -d \
  --name plex \
  --net host \
  -p 32400:32400 \
  --restart always \
  --volume /mnt/usb/plex/config:/config \
  --volume /mnt/usb/plex/data:/data \
  greensheep/plex-server-docker-rpi:latest

WARNING: The requested image's platform (linux/arm) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

Nextcloud.

sudo mkdir -p /srv/dev-disk-by-label-Files/Databases/NextCloud
sudo mkdir -p /srv/dev-disk-by-label-Files/Config/Nextcloud

After that, copy and paste the stack into portainer. Wait for a few minutes on RPi3. The port number is 8080. Now we can create the admin username/password such as nextcloud/nextcloud. Click the little triangle next to "Storage and Database". Change to MySQL. In the next part we enter nextcloud/nextcloud/nextcloud/db (note the "db" replaces localhost b/c we use "db" as the service name). Again, wait for a few minutes.

Heimdall (Dashboard for web apps). I keep the PUID (1000) and PGID (1000). The instruction says it is from the admin user account but I don't find admin account? Change the volume to /srv/dev-disk-by-label-Files/Config/Heimdall (use sudo mkdir to create the directory on terminal). Change the port to 83 & remove port 443. Define the endpoint from Portainer -> Endpoints -> local -> Public IP as raspberrypi.local (depending on your hostname). We need to wait a little bit. Now go to the container and find heimdall and click the port in order to open the website correctly (instead 0.0.0.0). I can add apps like nextcloud, portainer, pi-hole, other servers, etc. The Application Type entry has a good list of popular apps and it will pre-populate the button icon and the background color for our app.

taisun The default port is 3000

yacht. The default login is [email protected] and pass. The name shown on portainer is pedantic_hermann

docker volume create yacht
docker run -d -p 8001:8000 -v /var/run/docker.sock:/var/run/docker.sock -v yacht:/config selfhostedpro/yacht

CloudFlare DDNS - Update CloudFlare with Your Dynamic IP Address

WatchTower

bitwardenrs. Use the terminal to create a volume first. The port number is 8100. This is straightforward.

Duplicati for backup.

photoshow. It works. It has a slideshow button. PhotoShow only displays videos in WebM.

R. r-base provide arm64 image but not not 32-bit arm architecture.

# 64-bit OS
docker pull r-base
docker run -it --rm r-base   # enter R directly

rocker/rstudio DOES NOT work on arm64 even I can pull. WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

I build a 32-bit armv7 image for r-base v4.0.2. This image works on either 32-bit or 64-bit arm OS (tested on 32-/64-bit Raspberry Pi and other 32-bit SBC devices).

docker pull arraytools/r402armv7
docker run -it --rm arraytools/r402armv7 R
docker pull r-base
# Using default tag: latest
# latest: Pulling from library/r-base
# no matching manifest for linux/arm/v7 in the manifest list entries

How and Why to Use A Remote Docker Host

Backup

Usage

Basics, docs, cheatsheet, introduction

Note that we need sudo is needed unless it is on a Mac OS.

If docker cannot find an image, it will try to pull it from its repository.

$ sudo docker run -it ubuntu /bin/bash
Unable to find image 'ubuntu' locally
Pulling repository ubuntu
04c5d3b7b065: Download complete 
511136ea3c5a: Download complete 
c7b7c6419568: Download complete 
70c8faa62a44: Download complete 
d735006ad9c1: Download complete 
root@ec83b3ac878d:/# 
purpose command
run a container docker container run -d -p 80:80 httpd
list running cotainer docker container ls
view logs of Docker container docker container logs cranky_cori
identify Docker container process? docker container top cranky_cori
stop Docker container? docker container stop cranky_cori
list stopped or not running Docker containers docker container ls -a
start Docker container docker container start c46f2e9e4690
remove Docker container docker container rm cranky_cori
list Docker images docker images
remove Docker image docker rmi iman/touch

Restart docker daemon

When I try the Chap5 > Continuous integration (Jenkins) of the Docker Book, I found I cannot stop/kill the container. See others' report here. The solution is to restart the docker daemon.

sudo service docker start

After that, I can stop and rm the container.

sudo docker stop jenkins
sudo docker rm jenkins
sudo docker ps -a

images vs containers

$ sudo docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
iman                           latest              6e0f5644b2fd        About a minute ago   460.4 MB
iman/touch                     latest              77b9ac5951c2        4 minutes ago        460.4 MB
<none>                         <none>              aaa75e64ddf0        5 weeks ago          188.3 MB
ouruser/sinatra                v2                  ea8c9f407a8d        5 weeks ago          447 MB
ubuntu                         14.04               ed5a78b7b42b        5 weeks ago          188.3 MB
ubuntu                         latest              ed5a78b7b42b        5 weeks ago          188.3 MB
eddelbuettel/docker-ubuntu-r   add-r-devel-san     3c19d078c5d9        3 months ago         460.4 MB
hello-world                    latest              ef872312fe1b        4 months ago         910 B
training/sinatra               latest              f0f4ab557f95        8 months ago         447 MB

$ sudo docker ps -a
CONTAINER ID IMAGE                                          COMMAND              CREATED        STATUS                   PORTS NAMES
8fbdbcdb5126 iman/touch:latest                              "/bin/bash"          2 minutes ago  Exited (0) 2 minutes ago       thirsty_engelbart   
dc9e82f2c00a eddelbuettel/docker-ubuntu-r:add-r-devel-san   "/bin/bash"          9 minutes ago  Exited (0) 3 minutes ago       kickass_bardeen     
532a90f36aa8 eddelbuettel/docker-ubuntu-r:add-r-devel-san   "/bin/bash"          18 hours ago   Exited (0) 18 hours ago        happy_lalande       
7634024ee0bf eddelbuettel/docker-ubuntu-r:add-r-devel-san   "/bin/bash"          18 hours ago   Exited (0) 18 hours ago        insane_mclean       
14034a9720cb eddelbuettel/docker-ubuntu-r:add-r-devel-san   "/bin/bash"          18 hours ago   Exited (0) 18 hours ago        naughty_lumiere     
ca90954628db eddelbuettel/docker-ubuntu-r:add-r-devel-san   "/bin/bash"          19 hours ago   Exited (130) 18 hours ago      sick_hawking        
8bbdcb7c339f eddelbuettel/docker-ubuntu-r:add-r-devel-san   "/bin/bash"          19 hours ago   Exited (0) 19 hours ago        modest_davinci      
e8e24f80f0dd aaa75e64ddf0                                   "/bin/sh -c 'apt-get 5 weeks ago    Exited (100) 5 weeks ago       berserk_hodgkin     
d41959e0eb55 aaa75e64ddf0                                   "/bin/sh -c 'apt-get 5 weeks ago    Exited (100) 5 weeks ago       jovial_curie        
b408c0e2805b aaa75e64ddf0                                   "/bin/sh -c 'apt-get 5 weeks ago    Exited (100) 5 weeks ago       lonely_tesla        
72a551e4b492 ouruser/sinatra:v2                             "/bin/bash"          5 weeks ago    Exited (0) 5 weeks ago         jolly_meitner       
75fd6cc4658b training/sinatra:latest                        "/bin/bash"          5 weeks ago    Exited (0) 5 weeks ago         evil_yalow          
cc8886f5a02e training/sinatra:latest                        "/bin/bash"          5 weeks ago    Exited (130) 5 weeks ago       elegant_curie       
0585e4f5fecd eddelbuettel/docker-ubuntu-r:add-r-devel-san   "/bin/bash"          5 weeks ago    Exited (0) 5 weeks ago         elated_euclid       
brb@brbweb4:~/Downloads$ 

When we want to delete a container, we use the container's CONTAINER ID or NAME (last column output from docker ps -a). But when we want to delete an image, we use the image's REPOSITORY or IMAGE ID (2nd column output from docker images)

$ sudo docker rm thirsty_engelbart  # iman/touch
$ sudo docker rm dc9e82f2c00a       # eddelbuettel/docker-ubuntu-r:add-r-devel-san
$ sudo docker ps -a   # check to see the container is gone now

$ sudo docker rmi 6e0f5644b2fd
$ sudo docker rmi iman/touch
$ sudo docker images  # check to see the images are gone now

Command line interface, CLI

https://docs.docker.com/engine/reference/commandline/cli/ Docker command line

$ docker

Usage:	docker COMMAND

A self-sufficient runtime for containers

Options:
      --config string      Location of client config files (default "/home/brb/.docker")
  -D, --debug              Enable debug mode
  -H, --host list          Daemon socket(s) to connect to
  -l, --log-level string   Set the logging level ("debug"|"info"|"warn"|"error"|"fatal") (default "info")
      --tls                Use TLS; implied by --tlsverify
      --tlscacert string   Trust certs signed only by this CA (default "/home/brb/.docker/ca.pem")
      --tlscert string     Path to TLS certificate file (default "/home/brb/.docker/cert.pem")
      --tlskey string      Path to TLS key file (default "/home/brb/.docker/key.pem")
      --tlsverify          Use TLS and verify the remote
  -v, --version            Print version information and quit

Management Commands:
  config      Manage Docker configs
  container   Manage containers
  image       Manage images
  network     Manage networks
  node        Manage Swarm nodes
  plugin      Manage plugins
  secret      Manage Docker secrets
  service     Manage services
  swarm       Manage Swarm
  system      Manage Docker
  trust       Manage trust on Docker images
  volume      Manage volumes

Commands:
  attach      Attach local standard input, output, and error streams to a running container
  build       Build an image from a Dockerfile
  commit      Create a new image from a container's changes
  cp          Copy files/folders between a container and the local filesystem
  create      Create a new container
  diff        Inspect changes to files or directories on a container's filesystem
  events      Get real time events from the server
  exec        Run a command in a running container
  export      Export a container's filesystem as a tar archive
  history     Show the history of an image
  images      List images
  import      Import the contents from a tarball to create a filesystem image
  info        Display system-wide information
  inspect     Return low-level information on Docker objects
  kill        Kill one or more running containers
  load        Load an image from a tar archive or STDIN
  login       Log in to a Docker registry
  logout      Log out from a Docker registry
  logs        Fetch the logs of a container
  pause       Pause all processes within one or more containers
  port        List port mappings or a specific mapping for the container
  ps          List containers
  pull        Pull an image or a repository from a registry
  push        Push an image or a repository to a registry
  rename      Rename a container
  restart     Restart one or more containers
  rm          Remove one or more containers
  rmi         Remove one or more images
  run         Run a command in a new container
  save        Save one or more images to a tar archive (streamed to STDOUT by default)
  search      Search the Docker Hub for images
  start       Start one or more stopped containers
  stats       Display a live stream of container(s) resource usage statistics
  stop        Stop one or more running containers
  tag         Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
  top         Display the running processes of a container
  unpause     Unpause all processes within one or more containers
  update      Update configuration of one or more containers
  version     Show the Docker version information
  wait        Block until one or more containers stop, then print their exit codes

Run 'docker COMMAND --help' for more information on a command.

Version, system information

Docker version

$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:51 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:15 2018
  OS/Arch:          linux/amd64
  Experimental:     false

System information.

  • what mode the Docker engine is operating in (swarm mode or not)
  • what storage drive is used for the union filesystem
  • what version of the Linux kernel we have on our host
  • et al
$ docker system info
Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 10
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-33-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.674GiB
Name: t420s
ID: VLWB:6BN3:U7KB:L4T4:GQIB:54F3:YZKJ:PAIR:HEUM:UQIC:XLZU:3IFJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

List resource consumption

$ docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              10                  2                   2.58GB              1.519GB (58%)
Containers          2                   0                   304B                304B (100%)
Local Volumes       2                   0                   314.7MB             314.7MB (100%)
Build Cache         0                   0                   0B                  0B

$ docker system df -v  # more detailed information
# We can use the information to clean up our system

A brief intro to docker virtualization

docker search --help
docker search redis
docker search -s 100 redis
docker pull --help
docker pull ubuntu # download all versions of ubuntu
docker images    # available local container images
docker pull centos:latest
docker run --help
cat /etc/issue   # look at the current distr name before running docker
docker run -it centos:latest /bin/bash
                 # create a container & execute as a sudo

cat /etc/redhat-release
yum
cd /home
touch temp.txt
ls
exit

docker ps   # current running processes
docker ps -a # show all processes including closed
docker restart c85850ed0e13
docker ps   # container c85850ed0e13 is running
docker attach c85850ed0e13 # log into the system

ls /home
exit

docker ps -a
docker rm c85850ed0e13 # delete the container

Note: Following the discussion, using attach can only launch one instance of shell. If we use exec, we can launch multiple instances.

sudo docker exec -i -t c85850ed0e13 bash #by ID
or
$ sudo docker exec -i -t loving_heisenberg bash #by Name

Rootless mode

docker pull

https://docs.docker.com/engine/reference/commandline/pull/

$ docker pull ubuntu:zesty
$ docker run -ti --rm ubuntu:zesty /bin/bash 
# lsb_release -a         
bash: lsb_release: command not found
# cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=17.04
DISTRIB_CODENAME=zesty
DISTRIB_DESCRIPTION="Ubuntu 17.04"
NAME="Ubuntu"
VERSION="17.04 (Zesty Zapus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 17.04"
VERSION_ID="17.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=zesty
UBUNTU_CODENAME=zesty

Update/upgrade images

docker compose pull && docker compose up -d
docker compose up --pull always -d

<none>:<none> images

Exit/detach from a container without stopping it

$ docker container run -it ubuntu:latest /bin/bash
# Ctrl+p, Ctrl+q to exit the container without terminating it
$ docker ps -a # showing the container 70c5aceb5512 is running in the background

# You can reattach your terminal to it with the "docker container exec" command
$ docker container exec -it 70c5aceb5512 bash

How to start a stopped Docker container with a different command

How to start a stopped Docker container with a different command?

Clean shutdown DOCKER containers before reboot

Dockerizing Applications/Detached mode

$ sudo docker run -d --name insane_babbage ubuntu:14.04 /bin/sh -c "while true; do echo hello world; sleep 1; done"
$ sudo docker ps -l
$ sudo docker logs insane_babbage
$ sudo docker stop insane_babbage
$ sudo docker ps

The -d flag tells Docker to run the container and put it in the background, to daemonize it.

According to https://docs.docker.com/engine/reference/run/#detached-vs-foreground, containers started in detached mode exit when the root process used to run the container exits, unless you also specify the --rm option. If you use -d with --rm, the container is removed when it is stopped, exits or when the daemon exits, whichever happens first.

Automatically restart after reboot

https://stackoverflow.com/questions/18786054/how-to-auto-restart-a-docker-container-after-a-reboot-in-coreos

Add a --restart=always parameter. It will always restart a stopped container unless it has been explicitly stopped, such as via a "docker container stop" command. See the following

$ docker run -d --restart always myCustomeDocker

$ docker container run --name neverdie -it --restart always ubuntu /bin/bash
# exit
$ docker ps -a  # the container is still ther
$ docker stop neverdie
$ docker ps -a

Working with Containers

$ sudo docker run -i -t ubuntu /bin/bash
$ sudo docker version
$ sudo docker
$ sudo docker attach --help

Environment variables

Docker container ID

  • The full container ID is a hexadecimal string of 64 characters.
  • The minimum number of characters required for a Docker ID is 4.
  • We can use a shorter ID in docker command if that ID uniquely determined the container. For example, docker exec -it 9608 bash or even docker exec -it 9 bash works.

Alpine image

apk add htop

Running a Web Application

$ sudo docker run -d -P training/webapp python app.py

Alpine linux is 6MB. It is a good OS to run a web application. See the demo here.

Viewing our Web Application Container

$ sudo docker ps -l
$ sudo docker run -d -p 5000:5000 training/webapp python app.py

Check container status (docker status) - CPU, Memory usage

Container networking

Host network

If you use the host network driver for a container, that container’s network stack is not isolated from the Docker host. For instance, if you run a container which binds to port 80 and you use host networking, the container’s application will be available on port 80 on the host’s IP address.

ping, ifconfig and ip commands not found in Ubuntu container

apt update
apt install iputils-ping  # ping 
apt install net-tools     # ifconfig
apt install iproute2      # ip

Network Port Shortcut

$ sudo docker port nostalgic_morse 5000

Access Ports on the Host from a Docker Container

How to Access Ports on the Host from a Docker Container

Multiple NICs

containers in docker to use public ip addresses directly

Viewing the Web Application's Logs

$ sudo docker logs -f nostalgic_morse

Clear Logs of Running Docker Containers

How to Clear Logs of Running Docker Containers

Looking at our Web Application Container's processes

$ sudo docker top nostalgic_morse

Inspecting our Web Application Container

$ sudo docker inspect nostalgic_morse

Obtain the container's IP address, log into a running server

PS. Portainer web interface can show the IP addresses.

$ docker inspect <container id> | grep "IPAddress"

We don't need the IP address if we just want to log into a running server,

$ docker exec -it <contianer id> bash

How to Secure Docker’s TCP Socket

How to Secure Docker’s TCP Socket with TLS

docker attach

Suppose I run docker run -it --user rstudio bioconductor/bioconductor_docker:devel R and I use q() to quit the container. The container is still there. To re-enter the R in the container, I use

docker start XXXXXXXX    # restart it in the background
docker attach XXXXXXXX   # reattach the terminal & stdin

If we want the latest created container, then we use

docker start `docker ps -q -l` && docker attach `docker ps -q -l`

docker exec: SSH into a running container

Run a command in a running container

  • Usage:
    docker exec [OPTIONS] CONTAINER COMMAND [ARG...]
  • Examples:
    $ docker exec -d ubuntu_bash touch /tmp/execWorks # do st in the background
    
    $ docker exec -it ubuntu_bash bash
    
    $ docker exec -it -e VAR=1 ubuntu_bash bash # set an environment variable
    
    $ docker exec -it ubuntu_bash pwd
    $ docker exec -it -w /root ubuntu_bash pwd # change the working directory
  • How to Run a Command on a Running Docker Container
  • How to Use the Docker exec Command. nginx container is used as an example.
    docker run --name docker-nginx -p 8080:80 -d nginx
    
    # method 1. Access the Running Container’s Shell
    docker exec -it ID /bin/bash
      apt-get update
      apt-get upgrade -y
      exit
    
    # method 2. Run a Command from Outside the Container
    docker exec ID apt-get update && apt-get upgrade
    
    docker exec ID cat /usr/share/nginx/html/index.html
    docker cp index.html ID:/usr/share/nginx/html/
    docker exec ID cat /usr/share/nginx/html/index.html
    

docker cp

Copy files/folders between a container and the local filesystem.

Restart an exited Container

$ docker start nostalgic_morse
OR
$ docker restart nostalgic_morse

For an interactive container, use docker start -ai CONTAINER which is equal to run "docker start CONTAINER" and "docker attach CONTAINER".

Rename a container

docker container rename

docker container rename CONTAINER NEW_NAME

Inspect container images and their metadata

Know the container size

docker ps -s

Meaning of two sizes

  • The "size" information shows the amount of data (on disk) that is used for the writable layer of each container
  • The "virtual size" is the amount of disk-space used for the read-only image data used by the container.

Removing our Web Application Container

$ sudo docker stop nostalgic_morse
$ sudo docker rm nostalgic_morse

Note: Always remember that deleting a container is final!

Dockerize an SSH service

https://docs.docker.com/engine/examples/running_ssh_service/#environment-variables

Remove old docker containers

This post on stackoverflow.com.

$ sudo docker ps -a | grep 'weeks ago' | awk '{print $1}' | xargs --no-run-if-empty sudo docker rm

Similarly to remove all exited containers

$ sudo docker ps -a | grep Exit | awk '{print $1}' | xargs sudo docker rm

To kill/stop (not delete) all running containers

$ sudo docker kill $(sudo docker ps -q)

To delete all stopped containers

$ sudo docker rm $(sudo docker ps -a -q)
OR
$ sudo docker rm `sudo docker ps -a -q`

It is also helpful to create bash aliases for these commands by editing ~/.bash_aliases file.

docker create vs docker run

https://stackoverflow.com/questions/37744961/docker-run-vs-create

docker create is similar to docker run -d except the container is never started.

Retrieve docker run command

https://stackoverflow.com/a/32774347. See the github page of runlike. So it is better to put the docker run in a stack. Then for example the Portainer has an Editor tab to show the compose file.

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
    assaflavie/runlike -p CONTAINER_NAME

The -p option splits the output into pretty lines.

docker run -it and -d together

How to Modify the Configuration of Running Docker Containers

How to Modify the Configuration of Running Docker Containers

Volume

Examples of host's volume locations

/home/$USER/docker/$PROJECT/$SUB-DIRECTORY

Back Up Your Docker Volumes

How to Back Up Your Docker Volumes

Two ways to achieve persistent data

Inspect the 'Mountpoint' of a volume

$ docker volume create crv
$ docker volume ls

$ docker run -d \
     --name mycloud \
     -p 81:80 \
     -v apps:/var/www/html/custom_apps \
     nextcloud

# docker inspect is not quite useful. It does not show how the volume was created
# But we can examine (ls, du, ...) the directory contents
$ docker inspect apps   
[
    {
        "CreatedAt": "2018-10-23T09:41:52-04:00",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/apps/_data",
        "Name": "apps",
        "Options": null,
        "Scope": "local"
    }
]

Remove an an unnamed volume

I you want to automatically removes volumes when a container is removed, you can use the --rm flag when starting the container with the "docker run" command. This flag tells Docker to automatically remove the container and any anonymous volumes associated with it when the container exits. However, this flag does not affect named volumes.

If you created an unnamed volume, it can be deleted at the same time as the container with the -v flag. Note that this only works with unnamed volumes.

docker rm -v container_name

If the volume is named, it stays present. To remove a named volume, use docker volume rm volume_name .

Volumes created in docker-compose

When you use docker-compose to create and manage containers, volumes are handled slightly differently than when using the docker run command.

In a "docker-compose.yml" file, you can specify named volumes using the volumes key at the top level of the file. These volumes are created when you run docker-compose up and are not automatically removed when you stop or remove the containers using docker-compose down.

If you want to remove named volumes created by docker-compose, you can use the -v flag with the docker-compose down command. Here’s an example command that stops and removes all containers defined in a docker-compose.yml file and also removes any named volumes:

docker-compose down -v

This command stops and removes all containers defined in the docker-compose.yml file and also removes any named volumes specified in the file. All data stored in the volumes will be permanently deleted.

Anonymous volumes created by docker-compose are automatically removed when you stop and remove the containers using docker-compose down, even if you don’t use the -v flag.

Start a container with a volume

--mount -v
docker run -d \

--name devtest \
--mount source=myvol2,target=/app \
nginx:latest

docker run -d \

--name devtest \
-v myvol2:/app \
nginx:latext

Note

  • target in "--mount" can be replaced by destination or dst.
  • To use a read-only volume, add the ,readonly option in "--mount" or the :ro option in "-v".
  • We cannot use "~/" to represent a local directory under HOME. We have to specify a full path in docker run.

A simple example

From the book "Learn Docerk -Fundamentals of Docker 18.x". Chap 5. Data Volumes and System Management > Creating and mounting data volumes.

# Create a volume
docker volume create my-data
docker volume inspect my-data
# The host folder can be found in the output under 'Mountpoint'
# In my case,
#        "Mountpoint": "/var/lib/docker/volumes/my-data/_data",

# Mount a volume into a container
docker run --name test -it -v my-data:/data alpine /bin/sh
# cd /data
# echo 'some data' > data.txt
# echo 'more data' > data2.txt
# exit
docker inspect my-data
sudo ls /var/lib/docker/volumes/my-data/_data
# We can even try to output the content of say, the second file:
sudo cat /var/lib/docker/volumes/my-data/_data/data2.txt
# We can create a new file in this folder from the host and then use the volume with another container
echo "the file is created on host" > sudo tee /var/lib/docker/volumes/my-data/_data/host-data
# Let's delete the test container and run another one
docker rm test

# This time we are mounting our volume to a different container folder
docker run --name test2 -it -v my-data:/app/data centos:7 /bin/bash
# We are able to see three files:
# ls /app/data

# Remove volumes
docker volume rm my-data # Or 
docker volume rm $(docker volume ls -q)

# Remove all running containers to clean up the system,
docker rm -f $(docker ls -aq)

Sharing data between containers

How to Share Data Between Docker Containers

docker run -it --name writer -v shared-data:/data alpine /bin/sh
# create a file inside it
# echo 'my sample file' > /data/sample.txt
# exit
docker run -it --name reader -v shared-data:/app/data:ro ubuntu:17.04 /bin/bash
# ls -l /app/data

Using host volumes

Use volumes that mount a specific host folder

  • It may be possible for the "docker volume" command to mount a local directory to a volume. See examples in the "docker volume create" documentation.
  • Specifying a directory name instead of giving a volume name in the "docker run" 's -v option
  • Since we are specifying a directory name instead of letting docker to create a new volume, "docker volume ls" will not getting a new volume
docker run -it --name test -v $(pwd)/src:/app/src alpine /bin/sh

# Make a sample to demonstrate how that works
mkdir ~/my-web; cd ~/my-web
echo "<h1>My website</h1>" > index.html

# Create 'Dockerfile'
echo -e 'FROM nginx:alpine
COPY . /usr/share/nginx/html' > Dockerfile

docker image build -t my-website:1.0 .
docker run -d -p 8080:80 --name my-site my-website:1.0

# Open http://localhost:8080. It looks good
# Now modify index.html and refresh the website. It does not refresh
# Let's stop and rm the container and rebuild using a volume
docker rm -f my-site
docker run -d -v $(pwd):/usr/share/nginx/html \
   -p 8080:80 --name my-site my-website:1.0
# Now any changes on index.html will refresh on the website

Define volumes in images

A few samples of volume definition

VOLUME /app/data
VOLUME /app/data, /app/profiles, /app/config
VOLUME {"/app/data", "/app/profiles", "/app/config"]

The first line defines a single volume to be mounted at /app/data.

We can use the docker image inspect command to get information about the volumes defined in the Dockerfile.

docker image pull mongo:3.7
docker image inspect --format='{{json .ContainerConfig.Volumes}}' \
       mongo:3.7 | jq
# {
#   "/data/configdb": {},
#   "/data/db": {}
# }

# now run an instance of MongoDB and inspect the volume information
docker run --name my-mongo -d mongo:3.7
docker inspect --format '{{json .Mounts}}' my-mongo | jq
# [
#  {
#    "Type": "volume",
#    "Name": "535e0138b9a32e89f71380e9e73bb0de64ce0d1cad78fcda0ec1d49e11d76d7a",
#    "Source": "/var/lib/docker/volumes/535e0138b9a32e89f71380e9e73bb0de64ce0d1.../_data",
#    "Destination": "/data/configdb",
#    "Driver": "local",
#    "Mode": "",
#    "RW": true,
#    "Propagation": ""
#  },
#  {
#    "Type": "volume",
# SKIP

Differences between VOLUME and '-v|--volume'

https://stackoverflow.com/a/25312719

Container Memory Limits, Setting Available CPUs, Allocating memory and CPU

docker run \
    -rm \ ## Automatically remove the container when it exits
    --memory=6g \ ## memory limit
    --cpus=1.5 \ ## number of CPUs
    -v /shared/data-store:/home/rstudio/data \
    -v /shared/library-store:/usr/local/lib/R/host-site-library \
    -e PASSWORD=bioc \
    -p 8787:8787 \
         bioconductor/bioconductor_full:devel

Work with container images

List images by size or name

# by size
docker images --format "{{.ID}}\t{{.Size}}\t{{.Repository}}" | sort -k 2 -h

# by name
docker images --format "{{.ID}}\t{{.Size}}\t{{.Repository}}" | sort -k 3 

List specific columns

docker images --format 'table {{.Repository}}\t{{.Tag}}\t{{.Size}}'

Create an image interactively using commit - Example 1

The example is from the book 'Learn Docker - Fundamentals of Docker 18.x'.

docker container run -it --name sample alpine /bin/sh
# apk update && apk add iputils
# ping 127.0.0.1
# exit
docker container ls -a | grep sample
docker container diff sample

We can now use the docker container commit command to persist our modifications and create a new image from them

docker container commit sample my-alpine
docker images ls

If we want to see how our custom image has been built, we can use the history command as follows:

docker image history my-alpine
# IMAGE               CREATED              CREATED BY                                      SIZE    COMMENT
# 0f105057899b        About a minute ago   /bin/sh                                         1.55MB              
# 196d12cf6ab1        4 weeks ago          /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B                  
# <missing>           4 weeks ago          /bin/sh -c #(nop) ADD file:25c10b1d1b41d46a1…   4.41MB

The first layer in the preceding list is the one we just created by adding the iputils package.

Create an image interactively using commit - Example 2

Note that it is better/necessary to put the Dockerfile in an empty directory to avoid the problem of taking a long time to build the image (sending build context to Docker daemon ...GB ) since it will grab files from the current directory.

sudo docker search sinatra
sudo docker pull training/sinatra
sudo docker run -t -i training/sinatra /bin/bash
sudo docker commit -m="Added json gem" -a="Kate Smith" 0b2616b0e5a8 ouruser/sinatra:v2
sudo docker images

mkdir sinatra
cd sinatra
touch Dockerfile
sudo docker build -t="ouruser/sinatra:v2" .
sudo docker push ouruser/sinatra
sudo docker rmi training/sinatra
  • I get an error when I try to launch sinatra on my 32-bit ubuntu (Docker can only be installed through apt-get on 32-bit)
$ sudo docker run -t -i training/sinatra /bin/bash
2014/12/31 02:43:26 exec format error

How to copy Docker images from one host to another without using a repository

https://stackoverflow.com/questions/23935141/how-to-copy-docker-images-from-one-host-to-another-without-using-a-repository

docker save -o out.tar <image name>
# Or better to compress the file
docker save <docker image name> | gzip > out.tar.gz

And restore

docker load -i out.tar
# Or decompress the file
docker load < out.tar.gz

Docker Image Manifest

What Is a Docker Image Manifest?

Resources allocated to a container using docker?

https://stackoverflow.com/questions/16084741/how-do-i-set-resources-allocated-to-a-container-using-docker

hub.docker.com

docker tag local-image:tagname new-repo:tagname
docker login
docker push new-repo:tagname
docker pull phusion/baseimage
docker run -ti phusion/baseimage /bin/bash
  • https://dockerfile.github.io/ which includes dockerfiles for different purposes. The ubuntu-desktop one also works well (client needs a vnc viewer in order to see the desktop).

Set up a private Docker registry

$ curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X GET http://localhost:5000/v2/_catalog
$ OR
$ curl -H "Accept: application/xml" -H "Content-Type: application/json" -X GET http://localhost:5000/v2/_catalog

Github registry

docker pull ghcr.io/OWNER/IMAGE_NAME:TAG

# docker pull registry-url/image-name:tag

Google cloud registry

Using google cloud registry for private docker images

Dockerfile

  • Dockerfile Reference
  • Using Dockerfiles to Automate Building of Images from digitalocean.com.
  • Remember to put the Dockerfile in an empty directory.
  • What goes into a Dockerfile
  • Keywords
    • FROM. If we want to start from scratch, we can use FROM scratch.
    • RUN. The argument for RUN is any valid Linux command.
    • USER. This is useful if we want to create new files with a non-root owner privilege. For example, new files created under a binding directory with a non-root user ownership will belong to the current user in the host system. Here is an example where we use Rmarkdown to create pdf output. The generated pdf file should not be own by root. How to add users to Docker container? Switch users.
    • COPY & ADD.
      • "COPY . /app" will copy all files and folders from the current directory recursively to the /app folder. We can use "ADD" too but "ADD" will automatically unpack tarballs. See What is the difference between the `COPY` and `ADD` commands in a Dockerfile?
      • "ADD sample.tar /app/bin" will unpack the sample.tar' file into the target folder
      • "ADD http://example.com/sample.txt /data/" will copy the remote file sample.txt into the target file
    • WORKDIR. Define the working directory or context that is used when a container is run from the image.
    • CMD & ENTRYPOINT. These two are actually definitions of what will happen when a container is started from the image.
      • Use CMD without ENTRYPOINT: "CMD command param1 param2". This form is called the shell form.
      • If we use ENTRYPOINT + CMD, ENTRYPOINT defines the command and CMD defines parameters. The example above will run ping 8.8.8.8 -c 3. This form is called the exec form.
  • The Docker Book

Examples of Dockerfile

FROM python:2.7
RUN mkdir -p /app
WORKDIR /app
COPY ./requirements.txt /app/
RUN pip install -r requirements.txt
CMD ["python", "main.py"]
  • Another example
FROM alpine:latest
ENTRYPOINT ["ping"]
CMD ["8.8.8.8", "-c", "3"]
FROM debian:testing
RUN useradd docker \
	&& mkdir /home/docker \
	&& chown docker:docker /home/docker \
	&& addgroup docker staff

We can test it by "docker build -t mydebian . " and "docker run --rm -it --user docker -v /tmp:/home/docker mydebian". We can create a new file under /home/docker and the file will be accessible and belongs to the current host user once we quit the container. This actually is a huge security issue.

The same technique does not work on alpine if I try to create a new file in the container.

FROM alpine:latest
# Create a group and user; not useful for creating files in host OS
RUN addgroup -S appgroup && adduser -S appuser -G appgroup \
           && chown appuser:appgroup /home/appuser

"docker build -t myalpine . " and "docker run --rm -it -v ~/Downloads/:/home/appuser:rw --user appuser myalpine". When I use the "id" command in the container, I see it returns 100 in alpine container and 1000 in debian container. The id returns 1000 on my host (Ubuntu/Pop_OS). So the solution is docker run --rm -it -v ~/Downloads/:/home/appuser --user 1000:1000 myalpine. So the local user and the created user home directory in the container are not needed. See

Rocker

rocker (R and RStudio)

Bioconductor

Bioconductor

How to use Dockerfile

https://docs.docker.com/engine/reference/commandline/build/

The . simply means "current working directory".

docker build -f Dockerfile -t arraytools/myimagename .

docker build -t [myname] .  
# Multiple tags
docker build -t arraytools/biospear:latest -t arraytools/biospear:3.6.0 .

In the above example, we can create the image by

docker image build -t pinger .

We can run a container from the pinger image

docker container run --rm -it pinger

Docker Build Args

How to Use Docker Build Args to Configure Image Builds

Clean up after failed builds

Cleanup docker images and containers after failed builds

#!/bin/bash
docker rm $(docker ps -aq) \
  docker rmi $(docker images | grep "^<none>" | awk '{print $3}')

ENTRYPOINT and CMD

The advantage of using ENTRYPOINT + CMD (exec form) instead of using CMD alone (shell form) is we can override the CMD part that I have defined in the Dockerfile.

docker container run --rm -it pinger -w 5 127.0.0.1
# ping the loopback for 5 seconds

If we want to overwrite what's defined in the ENTRYPOINT in the Dockerfile, we need to use the --entrypoint parameter.

docker container run --rm -it --entrypoint /bin/sh pinger
# we'll be inside the container. Type exit to leave the container

When we use the shell form, the ENTRYPOINT is have the default value of /bin/sh -c and whatever is the value of CMD will be passed as a string to the shell command.

Temporary failure resolving 'deb.debian.org' when running "docker build"

Add "--net=host" to the docker build command. See Docker build “Could not resolve 'archive.ubuntu.com'” apt-get fails to install anything

Best practices for writing Dockerfiles

Use multi-stage builds

With multi-stage builds, we have a single Dockerfile containing multiple FROM instructions. Each FROM instruction is a new build stage that can easily COPY artifacts from previous stages.

An example from the "Docker Deep Dive" book.

tag after image was built

$ docker tag <imageID> <newName>/<repoName>:<tagName>

About storage drivers

https://docs.docker.com/storage/storagedriver/#sharing-promotes-smaller-images

Privileged versus Root user in Docker

.dockerignore

Using .dockerignore files to build better Docker images

Dockerfile in One Line

FROM ubuntu

Using This simple Dockerfile and the docker command sudo docker build -t scooby_snacks . will result in

$ sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
ubuntu              15.04               2427658c75a1        42 hours ago        117.5 MB
ubuntu              vivid               2427658c75a1        42 hours ago        117.5 MB
ubuntu              vivid-20150218      2427658c75a1        42 hours ago        117.5 MB
ubuntu              utopic-20150211     78949b1e1cfd        42 hours ago        194.4 MB
ubuntu              utopic              78949b1e1cfd        42 hours ago        194.4 MB
ubuntu              14.10               78949b1e1cfd        42 hours ago        194.4 MB
ubuntu              14.04               2d24f826cb16        42 hours ago        188.3 MB
ubuntu              14.04.2             2d24f826cb16        42 hours ago        188.3 MB
ubuntu              trusty              2d24f826cb16        42 hours ago        188.3 MB
ubuntu              trusty-20150218.1   2d24f826cb16        42 hours ago        188.3 MB
ubuntu              latest              2d24f826cb16        42 hours ago        188.3 MB
scooby_snacks       latest              2d24f826cb16        42 hours ago        188.3 MB
ubuntu              precise             1f80e9ca2ac3        42 hours ago        131.5 MB
ubuntu              precise-20150212    1f80e9ca2ac3        42 hours ago        131.5 MB
ubuntu              12.04.5             1f80e9ca2ac3        42 hours ago        131.5 MB
ubuntu              12.04               1f80e9ca2ac3        42 hours ago        131.5 MB
ubuntu              14.04.1             5ba9dab47459        3 weeks ago         188.3 MB
ubuntu              12.10               c5881f11ded9        8 months ago        172.2 MB
ubuntu              quantal             c5881f11ded9        8 months ago        172.2 MB
ubuntu              13.04               463ff6be4238        8 months ago        169.4 MB
ubuntu              raring              463ff6be4238        8 months ago        169.4 MB
ubuntu              13.10               195eb90b5349        8 months ago        184.7 MB
ubuntu              saucy               195eb90b5349        8 months ago        184.7 MB
ubuntu              10.04               3db9c44f4520        10 months ago       183 MB
ubuntu              lucid               3db9c44f4520        10 months ago       183 MB

List all tags of an image

How can I list all tags for a Docker image on a remote registry?

Tag the image with the git commit ID

$ docker build -t REPOS/IMAGE:$(git rev-parse --verify HEAD)

Run a shell script on host

$ docker run -v /path/to/sample_script.sh:/sample_script.sh \
  --rm ubuntu bash sample_script.sh

# GATK container example
# First we log in interactive and see where is the default location (/usr in this case)
$ docker run --rm -i -t broadinstitute/gatk3:3.8-0 bash
$ cat > tmp.sh << EOF
> pwd
> ls
> java -jar GenomeAnalysisTK.jar --version
> EOF
$ docker run --rm -v $(pwd):/usr/my broadinstitute/gatk3:3.8-0 bash my/tmp.sh
# ALTERNATIVELY, WE CAN PUT OUR SCRIPT IN THE TOP DIRECTORY (Hopefully the name is not duplicated)
$ docker run --rm -v $(pwd)/tmp.sh:/tmp.sh broadinstitute/gatk3:3.8-0 bash /tmp.sh
docker run -d -v$(pwd):/my SOMEIMAGE bash 
docker exec -d Test bash /my/script.sh

Link containers together

Manage data in containers

Assign a static IP to a container

Running Multiple Docker Services on the Same Server

How to Run Multiple Docker Containers on Different IP Addresses

Firewall

Rstudio server not loading, taking too long to respond in browser. On Ubuntu run sudo ufw allow PORTNUMBER.

Docker DNS/internet problem

I got an error on resolving the debian server when I was creating an image from a Dockerfile that needs to run apt update and apt install commands. See RStudio in Docker – now share your R code effortlessly!. The problem happened on my Linux Mint Desktop but not on a VirtualBox VM (Ubuntu 18.04).

Fix Docker's networking DNS config

A temporary solution is to add the --dns option to docker run command. This works well when I use the IP from any one of my 2 DNS servers. It does not work however if I use the IP from google DNS or OpenDNS.

A permanent solution is to create a new file /etc/docker/daemon.json and include the working DNS server IPs (these are obtained through the nmcli command or the NetworkManager GUI; see Query DNS server).

{
    "dns": ["XXX.XX.XX.XX", "YYY.YY.YY.YY"]
}

Then restart the docker service: sudo service docker restart

A quick test on the DNS problem is

docker run --rm busybox nslookup google.com

Working with Docker hub

https://docs.docker.com/userguide/dockerrepos/

Github Actions

Enabling HTTPS/Let's encrypt

Enabling HTTPS by self-sign certificates

traefik: The Cloud Native Application Proxy

Nginx proxy manager

docker: Error response from daemon: Cannot link to /site1_app_1, as it does not belong to the default network.

Running multiple web applications on a Docker host

Authentication: Authelia

Additional Self-Hosted Security with Authelia on NGINX Proxy Manager (video)

GUI apps

Firefox example

Running GUI Applications in Docker Container

From ubuntu:20.04
RUN apt update
RUN apt install firefox -y
RUN apt install python3-pip -y
RUN pip3 install  notebook

CMD /usr/bin/firefox
CMD jupyter-notebook --allow-root
nano Dockerfile
docker build -t gui .
docker run --env="DISPLAY" --net=host --name=firefox gui

It works. However, I need to use docker rm -f firefox to kill it since Ctrl+c does not work.

Meld example, save a running container as an image

Running a GUI Application in a Docker Container. It works. Below is a modified version for creating the meld app. I can save file modified by meld. To use the app, I need to place files in the ~/Documents/docker (defined in -v). Note that the RAM usage is very minimal. Unfortunately on macOS, I got an error something related to Gtk.

host> docker image pull ubuntu:jammy  # 22.04
 
host> docker container run --rm --net host -v /tmp/.X11-unix:/tmp/.X11-unix -it ubuntu:jammy
container# apt update
container# apt install -y meld
host> xhost +local:
container# export DISPLAY=:0

host> docker container ls  # find the ID of the running container
host> docker commit <ID> meld
container# exit

host> docker container run --rm --net host \
  -v /tmp/.X11-unix:/tmp/.X11-unix \
  -v ~/Documents/docker:/meld \
  -e DISPLAY \
  --name=meld \
  meld meld

R and httpgd package

  • httpgd Docker vignette, installation from Github.
  • It works. However, currently "httpgs" is archived in CRAN (2023/1/25). So my temporary solution is
    $ docker run --rm -it r-base:4.2.2 bash
    # apt update
    # apt install  libfontconfig1-dev
    # R
    > install.packages("remotes")
    > remotes::install_github("nx10/httpgd")
    ## note if we try to run 'httpgd::hgd(host = "0.0.0.0", port = 8888)', it does not work.
    ## The reason is we have not use "-p" option to expose a port in the previous "docker run" command
    
    ## open another terminal and create a docker image based on the current container
    $ docker ps -a | head
    $ docker commit CONTAINER_ID httpgd:4.2.2
    $ docker run --rm -it -p 8888:8888 httpgd:4.2.2 R
    > httpgd::hgd(host = "0.0.0.0", port = 8888)
    > plot(1:5)
    
  • It works when I tested it on a remote ubuntu server (R 4.4.0 & httpgd 2.0.1) (following the instruction on Docker vignette). Either IP or hostname works but the hostname URL link given by httpgd::hgd() needs to be modified to include .local.
  • To use it with Bioconductor (the Bioconductor docker image will use p3m.dev to install binary R packages so it is fast to create images), we can do like this
    $ docker run --rm -it -p 8888:8888 bioconductor/bioconductor_docker:RELEASE_3_18 R 
    
    > install.packages("httpgd")
    > httpgd::hgd(host = "0.0.0.0", port = 8888)
    

    OR use, for example, "bioconductor/bioconductor_docker:RELEASE_3_18" as the base image in the Dockerfile, and follow the same instruction from httpgd vignette to create a docker image.

    $ nano Dockerfile_httpgd
    $ docker build . -f Dockerfile_httpgd -t bioc-httpgd:RELEASE_3_18
    $ docker images
    $ docker run --rm -it --user rstudio -p 8888:8888 bioc-httpgd:RELEASE_3_18 R
    
  • Singularity. The following is a definition file that is using the bioconductor image + the httpgd package.
    Bootstrap: docker
    From: bioconductor/bioconductor_docker:RELEASE_3_18
    
    %post
        apt-get update \
        && apt-get install -y --no-install-recommends \
        libfontconfig1-dev \
        && apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/* \
        && install2.r --error --skipinstalled --ncpu -1 \
        httpgd \
        && rm -rf /tmp/downloaded_packages
    
    %runscript
        exec /usr/local/bin/R
    
    %environment
        export LC_ALL=C
    sudo singularity build bioc.sif bioc.def
    singularity run bioc.sif 
    
    > httpgd::hgd(host = "0.0.0.0", port = 8888)

    After we copy the URL, we need to modify the IP or hostname.

Docker-OSX

https://github.com/sickcodes/Docker-OSX

Pruning unused resources

  • Prune containers
docker container prune # remove all containers that are not in ''running'' status
                       # Docker will ask for confirmation before deleting the containers

docker container prune -f
docker container rm -f $(docker container ls -aq) # remove even the running containers
  • Prune images
docker images prune # unused image layers
  • Prune volumes
docker volume prune # unused volumes by at least one container

docker volume prune --filter 'label=demo'
docker volume prune --filter 'label=demo' --filter 'label=test'
  • Prune networks
docker network prune
  • Prune everything
docker system prune

Plugins

How to Manage Docker Engine Plugins

Misc

LXC (raw Linux containers)

LXC vs Docker

Vagrant vs Docker

Date/Time zone

docker run --rm -t -i -v /etc/localtime:/etc/localtime:ro ubuntu date

Access the internet from the container

Run the container with the '--net=host' option

sudo docker run --net=host -it ubuntu /bin/bash

How to transfer/copy an image to another host

How to copy Docker images from one host to another without using a repository

# Step 1: save the Docker image as a tar file:
docker save -o <path for generated tar file> <image name>

# Step 2: copy your image to a new system with regular file transfer tools such as cp or scp. 

# Step 3: After that you will have to load the image into Docker:
docker load -i <path to image tar file>

The tar file size is the same as what we get from 'docker image'. If we use the 'gzip' utility, it can reduce the file size (e.g 2.7GB to 1.1GB).

Or https://stackoverflow.com/a/39716019

# Step 1:
docker save docker-image-name | gzip > my-image.tar.gz
# Step 3:
docker load < my-image.tar.gz

Where are Docker images stored on the host: /var/lib/docker

The default is /var/lib/docker. The location can be changed by modifying the file /etc/default/docker. Three options if we are tight on the disk space.

1. Create a softlinks for the Docker data directory (/var/lib/docker) and for /var/lib/docker/tmp as described at miscellaneous-options. See this. See for how to stop docker daemon on different OS.

sudo service docker stop   # or sudo systemctl stop docker
sudo mv /var/lib/docker /a/new/location
sudo ln -s /a/new/location /var/lib/docker # Create a symbolic link
sudo service docker start  # or sudo systemctl start docker

2. Change the default location to another place. For example,

sudo nano /etc/default/docker
# Add a line DOCKER_OPTS="-g /home/brb/Docker"

Then after running sudo service docker.io restart and then a simple pull sudo docker pull rocker/r-base or sudo docker run --rm -ti rocker/r-base (the Dockerfile of r-base is available on github.com, --rm option means Automatically remove the container when it exits), we will see something like this:

$ docker run --rm -ti rocker/r-base
$ docker images
$ docker -v
Docker version 1.0.1, build 990021a

$ docker -D info | grep Root
 Root Dir: /home/brb/Docker/aufs

Consuming Docker system events

# Open a new terminal
docker system events
# This command is a blocking command. 
# Thus, when you execute it in your terminal session the according session is blocked.

# Open another terminal
docker container run --rm alpine echo "Hello World"

Monitor tools

Docker Machine

Docker Machine is a tool that lets you

  • Install Docker Engine on virtual hosts. You can use Machine (a unified way) to create Docker hosts on your local Mac or Windows box, on your company network, in your data center, or on cloud providers like Azure, AWS, or Digital Ocean. See the comment on here.
  • Provision and manage multiple remote Docker hosts
  • Provision Swarm clusters

Docker machine is not installed in Linux when you install Docker. See the instruction on here to install it.

My feeling is if we just want to play Docker on a local Linux machine, we don't really need to use Docker Machine (it just make life more complicated). But if we are working on Mac/Windows or we want to work on clouds or test on VirtualBox, we shall use Docker machines.

Use Docker-machine to Create Docker Servers. Compare the Docker images on the local machine (server 1) & a new host (server 2) created by docker-machine. Question: 1. how to tell we are in the host/machine environment? 2. how to exit the host environment after we use eval $() command? docker-machine stop MachineName.

$ docker-machine help
$ docker-machine create --driver=virtualbox test
# Follow its hint on the output, issue the following command
$ docker-machine env test
# Follow its hint on the output, issue the following command
$ eval$(docker-machine env test) # will configure the docker CLI to connect to this docker machine 'test'
                                 # This is equivalent to running 4 export commands on the command line
$ docker-machine ls  # Very useful
$ docker-machine stop test
$ docker-machine ip test
$ docker-machine start test
$ docker-machine rm test

Play Docker Machine on Mac with Virtualbox. Docker can be used to create a virtual machine just like Vagrant.

$ docker-machine create -d virtualbox demo
$ docker-machine ls

# first way to access a Docker host
$ docker-machine ssh demo
docker@demo:~$ docker images # empty for now

# second way to access 
$ docker-machine env demo
$ eval $(docker-machine env demo)
$ docker version

RancherOS demo video used the docker-machine command to pull and run the RancherOS.

docker-machine create -d virtualbox --virtualbox-boot2docker-url https://releases.rancher.com/os/latest/rancheros.iso demo
docker-machine ssh demo
ps
docker ps
sudo system-docker ps

sudo ros help
sudo ros console list
sudo ros console switch ubunu
apt-get help

Package CLI Applications

How to Use Docker to Package CLI Applications

Stack

Docker app

Docker App is an experimental Docker feature which lets you build and publish application stacks consisting of multiple containers. It aims to let you share Docker Compose stacks with the same ease of use as regular Docker containers.

How to Use 'Docker App' to Containerise an Entire Application Stack

Docker Swarm

Security

Moby Project

What is Docker's Moby Project?

Windows container

How can I run a docker windows container on osx?

When Not to Use Docker

When Not to Use Docker: Cases Where Containers Don’t Help

Docker Compose <docker-compose.yaml>

Docker Compose can help us out as it allows us to specify a single file in which we can define our entire environment structure and run it with a single command (much like a Vagrantfile works).

YAML validator

https://codebeautify.org/yaml-validator

Download binary

Difference of "docker compose" and "docker-compose"

  • Docker-compose is the original Python-based command-line tool that was released in 2014. Docker compose is a newer Go-based command-line tool that is integrated into the Docker CLI platform and supports the compose-spec. Docker compose is meant to be a drop-in replacement for docker-compose, but it may have some behavior differences and new features. Docker compose is currently a tech preview, but it will eventually replace docker-compose as the recommended way to use Compose.

Simple examples

Create a file docker-compose.yml and run docker-compose up after creating the file.

hello-world: 9kB

version: "3"
services:
  hello:
    image: hello-world

alpine: 7.73MB

version: "3"
services:
  server:
    image: alpine
    container_name: my_container
    command: sh -c "echo 'hello' && echo 'docker'"

Nginx: 135MB

mkdir src
echo "Hello world!" > src/index.html
version: "3"
services:
  client:
    image: nginx
    ports:
      - 8000:80
    volumes:
      - ./src:/usr/share/nginx/html

PUID, PGID

  • Understanding PUID and PGID (or the source)
  • You should use the -e PUID and -e PGID options when creating a container from a Docker image to map the container’s internal user to a user on the host machine. This is useful because Docker runs all of its containers under the root user domain, which means that processes running inside your containers also run as root. This kind of elevated access is not ideal for day-to-day use and can potentially give applications access to things they shouldn’t. By using PUID and PGID, you can ensure that files and directories created during the container’s lifespan are owned by a user on the host machine instead of root.
  • In this video How to Install Calibre on OMV and Docker, it uses the command id admin where "admin" is the portainer user to get PUID (of "admin") and PGID (of "users") to find out the two ids.

Composerize/convert a docker command into a docker compose file

An example from 'Fundamentals of Docker'

git clone https://github.com/fundamentalsofdocker/labs.git
cd labs/ch08
docker-compose up
# Open http://localhost:3000/pet

The images do not show up:( The terminal shows what has happened under the hood. So the problem is the http links for images do not exist.

We can also run the application in the background

docker-compose up -d

To stop and clean up the application, Howto use docker-compose to Start, Stop, Remove Docker Containers

docker-compose down # Stop and remove containers, networks, images, and unnamed volumes
                    # defined in the docker-compose.yml flie
# OR
docker-compose down -v # similar to above but remove named volumes defined in yml file
# OR
docker-compose stop && docker-compose rm -f
docker-compose rm -v

If we also want to remove the volume for the database

docker volume rm ch08_pets-data

An example from "How to Setup NGINX as Reverse Proxy Using Docker"

See here. Only nginx is used.

An example from "Docker Deep Dive" (flask + redis)

Note that on Get started with Docker Compose it mounts the current directory to /code inside the container. So after we modify app.py, we don't need to copy it to the container.

Another one Docker compose tutorial for beginners by example

$ git clone https://github.com/nigelpoulton/counter-app.git
$ cd counter-app
$ ls
app.py  docker-compose.yml  Dockerfile  README.md  requirements.txt

$ cat requirements.txt 
flask

$ cat Dockerfile
FROM python:3.4-alpine
ADD . /code
WORKDIR /code
RUN pip install -r requirements.txt
CMD ["python", "app.py"]

$ cat docker-compose.yml 
version: "3.5"
services:
  web-fe:
    build: .
    command: python app.py
    ports:
      - target: 5000
        published: 5000
    networks:
      - counter-net
    volumes:
      - type: volume
        source: counter-vol
        target: /code
  redis:
    image: "redis:alpine"
    networks:
      counter-net:

networks:
  counter-net:

volumes:
  counter-vol:

$ docker-compose up &

$ docker container ls

$ docker network ls
NETWORK ID          NAME                     DRIVER              SCOPE
2acef6dabde6        bridge                   bridge              local
a2d42bc482ff        counterapp_counter-net   bridge              local
e1e093b64282        host                     host                local
7ecd0a6a9ebd        none                     null                local

# Open the browser http://localhost:5000
$ docker-compose ps
       Name                      Command               State           Ports         
-------------------------------------------------------------------------------------
counterapp_redis_1    docker-entrypoint.sh redis ...   Up      6379/tcp              
counterapp_web-fe_1   python app.py                    Up      0.0.0.0:5000->5000/tcp

$ docker-compose stop
$ docker-compose ps
# We can see stopping a Compose app does not delete the application

$ docker container ls -a
$ docker-compose rm     # delete a stopped Compose app
                        # images, volumes and source code remain
$ docker-compose restart
                        # If you made changes to your Compose app since stopping,
                        # these changes will not appear in the restarted app.
                        # You need to re-deploy the app to get the changes.
$ docker-compose ps
$ docker-compose down   # stop and delete the app
                        # images, volumes and source code remain
$ docker-compose down --volumes # remove the data volume used by the Redis container
$ docker-compose up -d 
$ docker volume ls
$ docker-compose 

# We can make changes to files in the volume, from the host side,
# and have them reflected immediately in the app.
$ nano app.py   # do some changes
$ docker volume inspect counterapp_counter-vol | grep Mount
$ sudo cp app.py \
  /var/lib/docker/volumes/counterapp_counter-vol/_data/app.py
# Our changes should be reflected 

$ docker-compose --help

Create Compose Files From Running Docker Containers

How to Automatically Create Compose Files From Running Docker Containers

Docker-Compose persistent data MySQL

https://stackoverflow.com/questions/39175194/docker-compose-persistent-data-mysql

Connect to Docker daemon over ssh using docker-compose

#DockerTips: Connect to Docker daemon over ssh using docker-compose

Dockerfile + docker-compose

Docker Compose vs. Dockerfile - which is better?

The Compose file describes the container in its running state, leaving the details on how to build the container to Dockerfiles.

How to deploy on remote Docker hosts with docker-compose

How to deploy on remote Docker hosts with docker-compose

logs

docker-compose logs -f
# Ctrl + c

GUI/TUI interface manager

Dry

Dry – An Interactive CLI Manager For Docker Containers. The TUI is built on top of termui; a cross-platform, easy-to-compile, and fully-customizable terminal dashboard. It is inspired by blessed-contrib, but purely in Go.

LazyDocker (TUI)

Dockly (TUI)

Dockly – Manage Docker Containers From Terminal

DockStation

It is not open source. It works with remote Docker containers.

DockSTARTer: get started with home server apps running in Docker

Portainer* (nice)

IP address 0.0.0.0

How to setup ip address in portainer to access containers? Environments > local (or whatever your environment is named) there you set your public ip.

Templates

Yacht

cockpit-docker

sudo apt-get -y install cockpit-docker

sudo systemctl restart cockpit

DockerUI (Deprecated, Development continues at Portainer)

https://github.com/kevana/ui-for-docker. A quick start:

  1. Run:
    docker run -d -p 9000:9000 --privileged \
        -v /var/run/docker.sock:/var/run/docker.sock uifd/ui-for-docker
    where -v means to bind mount a volume.
  2. Open your browser to http://<dockerd host ip>:9000

Note: Anyone in the local network can access the website without any authentication.

Rancher

$ sudo apt-get install ufw
$ sudo ufw allow 4500/udp
$ sudo ufw allow 500/udp
  • discoposse.com
    • Part 1 Installing Rancher and Setting Access Control
    • Part 2 Adding a Docker Host to Rancher
    • Part 3 Adding the DockerHub to our Rancher Registry
    • Part 4 Using the Catalog Example with GlusterFS

Seagull

https://youtu.be/TuT5gb8oRw8

docker run -d -p 127.0.0.1:10086:10086 -v /var/run/docker.sock:/var/run/docker.sock tobegit3hub/seagull

The only issue is there is no username/password to protect other people to access the web GUI. The solution of binding to localhost to restrict the access does not work for remote administration.

That is, the tool is suitable for home use.

Kitematic (Mac, Windows and Ubuntu)

Owned by Docker. Available for Mac OS X 10.8+ and Windows 7+ (64-bit) and Ubuntu. https://github.com/docker/kitematic/releases/

Run containers through a simple, yet powerful graphical user interface.

It can not connect to remote docker machines.

A Share your Shiny Apps with Docker and Kitematic!

Shipyard (retired)

VS Code

Applications

Docker Applications

CasaOS

Every app is based on a Docker application

Orchestrator

Kubernete

Kubernete vs Docker swarm

k3s: Lightweight Kubernetes

Run Kubernetes on a Raspberry Pi with k3s

Other containers

Singularity and HPC systems

  • Old URL at singularity.lbl.gov
  • Singularity enables users to have full control of their environment; Singularity containers let users run applications in a Linux environment of their choosing. No 'sudo' is needed in general unless you want to build a container from a recipe.
  • Containers are more like an executable file for you to use
  • Containers are stored under the current location. It does not have a centre location (like /etc/default/docker if we use docker) to store images.
  • Can convert Docker containers to Singularity and run containers directly from Docker Hub
  • These bind points cannot be created unless the path already exists within the container. To ensure access to these storage spaces and remedy bind point errors, create these directories in the %post section of your Bootstrap file.
  • Singularity Hub

Ref:

Comparison of docker and singularity commands:

docker singularity
$ docker pull ubuntu:latest
$ docker pull broadinstitute/gatk3:3.8-0
$ singularity pull docker://ubuntu:latest
$ singularity pull docker://broadinstitute/gatk3:3.8-0
$ docker build -t myname/myapp:latest -f Dockerfile $ singularity build myapp.sif myapp.def
$ docker shell (not exist) $ singularity shell docker://broadinstitute/gatk3-3.8-0
$ singularity shell gatk3-3.8-0.img
> ls # the default location depends on the host system

> ls /usr # this is from the container

$ singularity shell --bind ~/Downloads:/mnt XXX.img
$ singularity shell docker://ubuntu:latest
# container is ephemeral

$ docker run --name test -it ubuntu date

# The next example is similar to 'singularity exec'
$ docker run --rm -i -t \
-v $(pwd):/usr/my_data \
broadinstitute/gatk3:3.8-0 \
bash /usr/my_data/myscript.sh
$ singularity run gatk3-3.8-0.img date
$ docker run --name ubuntu_bash --rm -i -t ubuntu bash
$ docker exec -d ubuntu_bash touch /tmp/execWorks
# Most useful
$ singularity exec gatk3-3.8-0.img java -version
$ singularity exec xxx.img cat /etc/*release
$ singularity exec docker://rocker/tidyverse:latest R
$ singularity exec docker://rocker/tidyverse:latest Rscript myScript.R

Cache

When we run singularity exec docker://rocker/tidyverse:latest R, it will save something in the cache in our system.

It seems to be OK after I manually delete the directory $HOME/.singularity (tested in Biowulf).

RStudio

$ singularity exec docker://rocker/tidyverse:latest R
$ singularity exec docker://rocker/tidyverse:latest Rscript myScript.R

Shifter

Conda

Anaconda

Bioconda

Using docker to install conda (https://conda.io/docs/user-guide/tutorials/index.html)

$ docker run -t -i --name test --net=host ubuntu bash
# apt-get update
# apt-get install -y wget bzip2 python
# wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
# wget https://repo.continuum.io/archive/Anaconda2-5.1.0-Linux-x86_64.sh
# bash Miniconda2-latest-Linux-x86_64.sh
# bash Anaconda2-5.1.0-Linux-x86_64.sh
# exit

$ docker start test 
$ docker exec -i -t test bash
# conda list  # WORKS!
# conda config --add channels r
# conda config --add channels defaults
# conda config --add channels conda-forge
# conda config --add channels bioconda
# conda install bwa  (Segmentation fault. Core dumped)
# which bwa
/root/anaconda2/bin/bwa
# conda install r   (Only get 3.4.2 but the latest is 3.4.3.)
# conda install bowtie
# bowtie --version
# conda install gatk (https://bioconda.github.io/recipes/gatk/README.html)
   (Due to license restrictions, this recipe cannot distribute and install GATK directly)
   (R is downgraded to 3.2.2:( )
   (Segmentation fault. Core dumped)
# exit
# docker stop test
# docker rm test

Get miniconda image instead of using a Ubuntu image

$ docker pull continuumio/miniconda
$ docker run -i -t continuumio/miniconda /bin/bash
# conda install r   (get 3.4.2)
# conda config --add channels bioconda
# conda install bwa  (OK, no error)
# conda install gatk  (R was downgraded to 3.2.2, install openjdk 8.0.121)
# which gatk
/opt/conda/bin/gatk
# gatk -h
GATK jar file not found. Have you run "gatk-register"?

Issues:

  • R version is not up to date
  • So the problem is installing GATK requires an installation of R and the current R was affected.

CoreOS

Installation

We first boot a liveCD from any OS (CentOS works but Ubuntu 16.04 gave errors). In Virtualbox, we choose 'Red Hat' if we use CentOS.

Once the VM is created. We go to the settings. Create a bridged network or host-only network first (even we can get files from the host without creating a host-only network). Storage: choose CentOS-7.

  1. Get the install script from Github and create <coreos_install.sh> and chmod +x
  2. create <cloud-config.yaml> file which will include ssh_authorized_keys generated from another machine. It should also contain a new token for the cluster from https://discovery.etcd.io/new.
  3. ls -l /dev/sd*
  4. run sudo ./coreos_install.sh -d /dev/sda -C stable -c cloud-config.yaml. It will download the latest stable CoreOS, install to the HD
  5. Don't leave the VM or it will freeze. Issue sudo shutdown -h now once we see the word 'Success' at the last line of the output.
  6. Remove CentOS from the VM storage. Boot the coreOS VM.

The new screen shows corebm1 login with an IP. Go back to another machine and type ssh -i /tmp/CoreOSBM_rsa [email protected]. Inside CoreOS, we can type docker images.

The 'cloud-config.yaml file has to follow the format in https://coreos.com/os/docs/latest/cloud-config.html. Use the online validator https://coreos.com/validate/ to correct. At first I use the file from the youtube video. There is no error coming out when I ran the installation script. But I cannot connect to coreOS. The cloud-config.yaml file I use is (pay attention to '-', double quotes and indent characters)

#cloud-config
#
# set hostname
hostname: CoreBM1

# Set ssh key
ssh_authorized_keys:
  - "ssh-rsa AAAAB3 ..... brb@T3600"

coreos:
  etcd:
    discovery: "https://discovery.etcd.io/d3e95 .... "
# sudo ./installos -d /dev/sda -C stable -c cloud-config.yaml

CoreOS exploration

brb@T3600 /tmp $ ssh -i /tmp/id_rsa [email protected]
Enter passphrase for key '/tmp/id_rsa':
CoreOS stable (1010.6.0)
core@CoreBM1 ~ $
core@CoreBM1 ~ $ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
core@CoreBM1 ~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.4G     0  1.4G   0% /dev
tmpfs           1.4G     0  1.4G   0% /dev/shm
tmpfs           1.4G  340K  1.4G   1% /run
tmpfs           1.4G     0  1.4G   0% /sys/fs/cgroup
/dev/sda9        18G   23M   17G   1% /
/dev/sda3       985M  589M  345M  64% /usr
tmpfs           1.4G     0  1.4G   0% /media
/dev/sda1       128M   37M   92M  29% /boot
tmpfs           1.4G     0  1.4G   0% /tmp
/dev/sda6       108M   52K   99M   1% /usr/share/oem
core@CoreBM1 ~ $ free -m
             total       used       free     shared    buffers     cached
Mem:          2713        187       2525          0          9        109
-/+ buffers/cache:         68       2644
Swap:            0          0          0
core@CoreBM1 ~ $ lsb_release -a
-bash: lsb_release: command not found
core@CoreBM1 ~ $ docker pull ubuntu
Using default tag: latest
latest: Pulling from library/ubuntu
f069f1d21059: Pull complete
ecbeec5633cf: Pull complete
ea6f18256d63: Pull complete
54bde7b02897: Pull complete
Digest: sha256:bbfd93a02a8487edb60f20316ebc966ddc7aa123c2e609185450b96971020097
Status: Downloaded newer image for ubuntu:latest
core@CoreBM1 ~ $ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu              latest              0f192147631d        7 days ago          132.7 MB
core@CoreBM1 ~ $

CoreOS cluster discovery

https://coreos.com/os/docs/latest/cluster-discovery.html

etcd

fleet

TryGhost

https://github.com/TryGhost/Ghost

Firecracker

Firecracker: start a VM in less than a second

Self-hosting

Tools and Resources for Self-Hosting

Linux in browser

Podman

  • Podman Installation Instructions
    • How To Install Podman Desktop In Linux
    • Raspberry Pi OS use the standard Debian's repositories, so it is fully compatible with Debian's arm64 repository. You can simply follow the steps for Debian to install Podman.
  • Podman vs docker:
    • One of the main differences between Podman and Docker is their architecture. Docker uses a client-server architecture with a central daemon that manages containers. In contrast, Podman is daemonless and uses a fork-exec model to manage containers.
    • Podman is designed to run containers without requiring root privileges or the use of sudo. This is one of the key differences between Podman and Docker, as Docker requires root privileges to run containers.
    • Both Podman and Docker are compatible with the Open Container Initiative (OCI) container specification, which means that they can run the same container images. However, Podman is more closely aligned with Kubernetes and its native container runtime, while Docker also works with its own orchestration tool, Docker Swarm.
    • Podman provides several benefits over Docker. For example, Podman is daemon-less, which means that if the Docker daemon crashes, the containers are in an uncertain state. This is prevented by making Podman daemon-less. You can also use systemd to manage your containers with Podman, which gives you virtually unlimited configurability compared to Docker. Hooking Podman with systemd allows you to update running containers with minimal downtime and recover from any bad updates.
  • Podman is a project from Red Hat
  • Getting Started With Podman Desktop, an Open Source Docker Desktop Alternative
  • Podman Compose - Managing Containers
pip3 install podman-compose
But it seems the compatibility is an issue even I tried a small example based on alpine image.
  • Nginx example (works)
podman run -it --rm -d -p 8080:80 \
  --name web \
  -v /mnt/Podman/site-content:/usr/share/nginx/html \
  docker.io/libary/nginx

Resource

Internet

Books

Blogs

Tips/trouble shooting

Play with Docker (PWD)

  • Some applications I've tested.
    • webtop (OK)
    • r-base:3.6.3, r-base:4.1.0, r-base:4.1.1 (OK)
    • r-base:4.1.2, r-base:4.2.0 (ERROR: R_HOME ('/usr/lib/R') not found). Maybe the docker version there is too old.

Alternatives

The 9 Best Docker Alternatives for Container Management

Serverless computing