Docker Applications: Difference between revisions

From 太極
Jump to navigation Jump to search
Line 37: Line 37:
* [https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1009244 Performance and scaling behavior of bioinformatic applications in virtualization environments to create awareness for the efficient use of compute resources] Hanussek, 2021
* [https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1009244 Performance and scaling behavior of bioinformatic applications in virtualization environments to create awareness for the efficient use of compute resources] Hanussek, 2021
* [https://github.com/StaPH-B/docker-builds docker-builds] - Dockerfiles and documentation on tools for public health bioinformatics
* [https://github.com/StaPH-B/docker-builds docker-builds] - Dockerfiles and documentation on tools for public health bioinformatics
* [https://github.com/shmohammadi86/ACTIONet/tree/R-release/mini_docker ACTIONet] and the paper [https://pubmed.ncbi.nlm.nih.gov/36650127/ Transcriptional vulnerabilities of striatal neurons in human and rodent models of Huntington's disease] 2023.


= Cloud provider =
= Cloud provider =

Revision as of 14:04, 24 January 2023

Homelab

App store

Tipi

Dockerizing Compiled Software

Crontab

How to Use Cron With Your Docker Containers

Install macOS

Raspberry Pi servers based on Docker

Data Science

Data Science

Illumina BaseSpace

Bioinformatics analyses

Cloud provider

Docker can be deployed on any cloud provider like Digital Ocean, Amazon Web Services (AWS), Microsoft Azure, Google Cloud Computing.

Dashboard

Dashy

docker run -d \
  -p 8295:80 \
  --volume /home/$USER/dashy/public/conf.yml:/app/public/conf.yml \
  --volume /home/$USER/dashy/icons:/app/public/item-icons/icons \
  --name dashy \
  --restart=unless-stopped \
  lissy93/dashy:latest

Homer

heimdall

https://github.com/linuxserver/docker-heimdall

Dashboard for monitoring system: grafana

After I change the firewall by adding two ports (9090 & 9100), the docker-compose method of grafana works like a charm.

  • Reboot the host OS will not affect the grafana (assuming we include restart: unless-stopped in the 3 containers). But if we modify docker-compose.yml and run docker-compose down; docker-compose up it will result in data lose and we will need to re-create the admin password and all the setup.

ownCloud

After I use docker run --rm --name myowncloud -p 81:80 owncloud:8.1, I find I can only access it through http://localhost:81. If I try to access it through another computer by http://xxx.xxx.xx.1:81, the URL is changed back to http://localhost:81 so it failed to load.

nextcloud

Nextcloud looks more modern than owncloud though their interface are very similar. I got the same problem (URL is changed from IP to localhost) when I try to access its web from another computer. Using the IP instead of localhost to finish the initial setup can fix the error.

Note:

  • The first we open the webpage, we need to create the admin username/password.
  • Nextcloud uses owncloud mobile application.
  • Nextcloud includes a small mp4 file. I can use browser to play mp4. Photos files can be opened in browsers too (including slideshow).

Nextcloud Apps

For mp3, no apps are shipped with nextcloud.

  • Audio Player. When I play mp3 files, the web interface does not show a progress bar:( 
    $ docker run --rm --name mynextcloud -v ~/Downloads/nextcloudapps:/var/www/html/custom_apps -p 81:80 nextcloud
# above will create an empty folder ~/Downloads/nextcloudapps (www-data:root)
$ tar xzvf Downloads/audioplayer-2.3.1.tar.gz -C Downloads/
$ docker cp Downloads/audioplayer mynextcloud:/var/www/html/custom_apps
# Go to nextcloud and '+Apps', a new app 'Audio Player' should have shown up.
  • Music. GOOD. The app is from the owncloud team though for some reason the app is not included in owncloud/nextcloud. Yes, a progress bar is shown when I played a song downloaded from Youtube:) 
    # No need to create a volume
$ docker run -d --name mycloud -p 81:80 nextcloud

# Open another terminal
# Don't download the zip file from github. Get the zip file from the app store
$ tar xzvf ~/Downloads/music_0.7.0_nc-signed.tar.gz -C ~/Downloads/
$ docker cp Downloads/music mycloud:/var/www/html/custom_apps
$ docker exec mycloud chmod -R 0750 /var/www/html/custom_apps
$ docker exec mycloud chown -R www-data:www-data /var/www/html/custom_apps
    Now open the website. Click the gear icon (Settings) and click "Apps". Find out the 'music' app from the bottom of the list and click the 'Enable' button. A new Music icon will be created next to the Gallery icon at the top. Note: if we want to play music repeatedly, I should not play the music directly from Files. I should click the Music icon at the top and let it start to scan musics. Then once I can play a music from there, it will show two new icons; one is shuffle and the other is loop. To have a persistent data, we can 'run' using 
    $ mkdir -p ~/nextcloud/{html,apps,config,data}
$ docker run -d \
    --name mycloud \
    -p 81:80 \
    -v ~/nextcloud/html:/var/www/html \
    -v ~/nextcloud/apps:/var/www/html/custom_apps \
    -v ~/nextcloud/config:/var/www/html/config \
    -v ~/nextcloud/data:/var/www/html/data \
    nextcloud
    If we like to delete the contain and its unamed volume, run 
    $ docker stop mycloud; docker rm -v mycloud
    If we run the container again mounting all volumes, all apps,files,configuration,admin's username & password are kept.
  • PDF viewer. PDF files will be downloaded instead of opening on the browser. This can be easily fixed. Click '+Apps' from the personal settings (the response is slow when I tested using Docker). Search for PDF viewer. Click the 'Enable' button (it will be changed to 'Disable' after that)
  • draw.io. After enable it, we can click the "+" sign and choose "Diagram" to create a new diagram (*.xml).
  • Radio. After enable it, a new "Radio" icon will be shown at the top. Only 20 stations are available from the 'Top'. No response after I clicked "Categories".

FileRun

File manager

File browser

The original image (filebrowser/filebrowser) does not work but linuxserver's image (filebrowser/filebrowser:s6) works fine. 

$ touch filebrowser.db
$ cat settings.json
{
  "port": 80,
  "baseURL": "",
  "address": "",
  "log": "stdout",
  "database": "/database/filebrowser.db",
  "root": "/srv"
}

#  Modify $FULL_PATH1 <- directory to share files
#      &  $FULL_PATH2  <- directory contains filebrowser.db & settings.json

$ docker run \
    --name filebrowser \
    -v $FULL_PATH1:/srv \
    -v $FULL_PATH2/filebrowser.db:/database/filebrowser.db \
    -v $FULL_PATH2/settings.json:/config/settings.json \
    -e PUID=$(id -u) \
    -e PGID=$(id -g) \
    -p 8080:80 \
    filebrowser/filebrowser:s6

FilGator

https://filegator.io/

Seafile

Run a Linux desktop VM with VNC

Run Ubuntu Linux in Docker with Desktop Environment and VNC

Guacamole

Webtop: Linux desktop in a browser

$ sudo apt install /config/Downloads/rstudio-2022.02.3-492-amd64.deb 
the following packages have unmet dependencies:
 rstudio : Depends: libssl1.0.0 but it is not installable or
                    libssl1.0.2 but it is not installable or
                    libssl1.1 but it is not installable

instantbox

IDE

cloud9

https://hub.docker.com/r/linuxserver/cloud9

vs code

Use with R (r-base) & RStudio IDE: Rocker

  • Docker 101 for Data Scientists by RStudio
  • r-base (Official image, R version is tagged), RStudio
    • Managing Users
    • An Introduction to Docker for R Users: how to write your own <Dockerfile>, install packages, run a script and get results.
    • Extensions from r-base. For example, r-spatial-base. It also mentions ropensci container is built upon rocker/rstudio.
    • The r-base image does not have pdflatex, git. Need to manually install them.
    • Not sure if the Docker Official Image is the same as the one provided by Rocker Project.
    • NOTE: Plotting works by forwarding X11. The instruction depends on the host OS. See rocker Wiki or the command below. Creating graphics files inside a container is still OK 👌; see the example How to compile R Markdown documents using Docker. 
      docker pull r-base:3.5.3
docker run -it --rm rocker/drd RD              # a little smaller, 3.6GB for R 4.0
docker run -it --rm rocker/drd R               # good to test the pipe operator (due in R 4.1.0)
docker run -it --rm rocker/r-devel RD          # initial one, larger, 5.7GB for R 4.0
docker run -it --rm rocker/r-devel R           # r-release
docker run -it --rm r-base:3.5.3               # default is root "/"
docker run -it --rm rocker/r-rspm:22.04        # seems the 'latest' tag is missing
docker run -it --rm rocker/r-bspm:22.04        # better than r-rspm in the case of 'tidyverse'
                                               # since bspm can take care of missing system OS libraries
                                               # 'Many' Bioconductor packages like DESeq2/limma/sva are avail
docker run -v ~/Downloads:/src -it --rm r-base # /src does not exist
docker run -v ~/Downloads:/home/docker -it --rm r-base # /home/docker exists and is empty by default
                                              # setwd("/home/docker")
docker run -it --rm -u1000:1000 -e DISPLAY=$DISPLAY \
           -v /tmp/.X11-unix:/tmp/.X11-unix \
           -v $(pwd):/work -w /work r-base

docker run -ti --rm -v "$PWD":/home/docker -w /home/docker \
       -u docker r-base bash   # Non-root user

docker run -ti --rm -v "$PWD":/home/rstudio -w /home/rstudio \
       -u rstudio rocker/rstudio bash  # Non-root user

docker run -it --rm -u rstudio -v $(pwd):/home/rstudio/project \
       -w /home/rstudio/project rocker/tidyverse:3.6.0 bash

Dockerfile

Create a new directory and a new file 'Dockerfile' with the content. 

FROM debian:testing
MAINTAINER Dirk Eddelbuettel [email protected]
## Remain current
RUN apt-get update -qq
RUN apt-get dist-upgrade -y
RUN apt-get install -y --no-install-recommends r-base r-base-dev r-recommended littler
RUN ln -s /usr/share/doc/littler/examples/install.r /usr/local/bin/install.r

NOTE

  1. From r-base in DockerHub click the "latest" in the "Supported tags and respective Dockerfile links" section.
  2. I ran into errors when I use the above (short) Dockerfile. But the Dockerfile from rocker (leave out the last line of launching R) works well. The R packages built in the image include 'docopt', 'magrittr', 'stringi', and 'stringr'.
  3. Install R package is possible when we launch a container. But we are not able to save the packages?? The rocker wiki also mentions something about installing packages.
  4. See also How to save data in wiki.

A quick run of an R script

docker run --rm \
  -v $(pwd):/tmp/working_dir \
  -w /tmp/working_dir \
  rocker/tidyverse:latest \
  Rscript my_script.R

docker run

Note that if we are using the Dockerfile above to create an image, we will be dropped to the Linux shell. If we are pulling the rocker/r-base image from Docker Hub, we will be in R console directly. See the last line of Dockerfile on github website for rocker.

Then run the following to do some exercise (We could possibly replace 21b6a9e8b9e8 with your image ID or use rocker/r-base). For simplicity, we can try the colortools package first which does not depend on other packages and there is no need to compile the package. 

sudo docker build -t debian:testing-add-r . # create an image based on the above Dockerfile
wget http://cran.r-project.org/src/contrib/sanitizers_0.1.0.tar.gz
sudo docker run -v `pwd`:/mytmp -t 21b6a9e8b9e8 \
     R CMD check --no-manual --no-build-vignettes /mytmp/sanitizers_0.1.0.tar.gz
sudo docker run -v `pwd`:/mytmp -t 21b6a9e8b9e8 \
     Rdevel CMD check --no-manual --no-build-vignettes /mytmp/sanitizers_0.1.0.tar.gz

sudo docker search eddelbuettel
sudo docker pull eddelbuettel/docker-ubuntu-r   # default tag is 'latest'; actually older than the other tags
sudo docker images eddelbuettel/docker-ubuntu-r # see the tag column
sudo docker pull eddelbuettel/docker-ubuntu-r:add-r # the tag name can only be obtained from hub.docker.com
sudo docker images eddelbuettel/docker-ubuntu-r # see the tag column
sudo docker pull eddelbuettel/docker-ubuntu-r:add-r-devel
sudo docker images eddelbuettel/docker-ubuntu-r # see the tag column
sudo docker run -v `pwd`:/mytmp -t 54d865dbd2c9 R CMD check --no-manual --no-build-vignettes /mytmp/sanitizers_0.1.0.tar.gz

sudo docker run -t -i eddelbuettel/docker-ubuntu-r /bin/bash
$ sudo docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
debian                         testing-add-r       21b6a9e8b9e8        28 minutes ago      572.2 MB
ubuntu                         14.04               ed5a78b7b42b        4 days ago          188.3 MB
ubuntu                         latest              ed5a78b7b42b        4 days ago          188.3 MB
debian                         testing             88ba2870bfbe        7 weeks ago         154.7 MB
eddelbuettel/docker-ubuntu-r   add-r-devel         c998a74a1fb4        11 weeks ago        460.4 MB
eddelbuettel/docker-ubuntu-r   add-r               54d865dbd2c9        11 weeks ago        460.4 MB
eddelbuettel/docker-ubuntu-r   latest              a7cd5ddeb98e        5 months ago        515.4 MB

sudo docker logs xxxxx                # view the log
sudo docker restart xxxxx
sudo docker exec -it xxxx /bin/bash   # view any changes in R library
sudo docker stop xxxxx
sudo docker rm xxxxx

This is another example of using 'docker run' accompanying MotifBreakR package.

Testing a new R release

R 4.1.0 

docker pull rocker/r-base:4.1.0 

alias dkrr='docker run --rm -it -u1000:1000 -v$(pwd):/work -w /work'
dkrr rocker/r-ubuntu:20.04 bash
dkrr r-base:latest R --version | head -1
dkrr r-base:3.6.3 R --version | head -1

# Assume we are in a directory called 'curse'
# (Yes you may need to add Depends and LaTeX support ...)
# Even the 'survival' package requires pdflatex in 'R CMD build' step.
# A toy package like https://cran.r-project.org/web/packages/QuadRoot/ works
dkrr rocker/r-base:4.1.0 R CMD build .  # this will create curse_1.0.0.tar.gz
dkrr rocker/r-base:4.1.0 R CMD check --no-vignettes --no-manual curse_1.0.0.tar.gz

A closer solution is to use rocker/verse:4.0.4 image. But it gives different errors

  • survival: LaTeX Error: File `fancyvrb.sty' not found.
  • glmnet: dependencies ‘foreach’, ‘shape’ are not available. A workaround solution (need to figure out the dependencies first): 
    $ curl -s https://cran.r-project.org/src/contrib/glmnet_4.1-1.tar.gz | tar xzv 
$ cd glmnet
$ docker run --rm -it -v$(pwd):/work -w /work rocker/verse:4.0.4 bash
# Rscript -e "install.packages(c('foreach', 'shape', 'knitr', 'lars', 'testthat', 'xfun', 'rmarkdown'))"
# su rstudio
$ R CMD build .
$ exit
# exit

    If we don't install the 'Suggests' packages, building will fail when it was trying to build the vignette. A more relaxed solution is adding the option --no-build-vignettes

Testing R packages

Multiple containers

Docker for R Package Development

http://www.jimhester.com/2017/10/13/docker/

A DevOps Perspective

Reproducible

  • A simple example to produce a PDF report from an Rmarkdown file by using rocker/verse image: 
    docker run -it --rm --name pdmr -v $(pwd):/home/docker -w /home/docker \
       --user rstudio rocker/verse:4.0.3 \
       Rscript -e "rmarkdown::render('Example.Rmd')"
  • R, Docker and Checkpoint: A Route to Reproducibility
  • liftr: Persistent reproducible reporting by containerization of R Markdown documents. I am not successful running the example in vignette. 
    > render_docker(input)
Sending build context to Docker daemon  6.144kB
Step 1/8 : FROM rocker/r-base:latest
 ---> 4e232e9b4097
Step 2/8 : MAINTAINER Nan Xiao <[email protected]>
 ---> Running in f75404db590a
Removing intermediate container f75404db590a
 ---> ceae0a54648c
Step 3/8 : RUN  rm -f /var/lib/dpkg/available   && rm -rf  /var/cache/apt/*   && apt-get update -qq   && apt-get install -y 
   --no-install-recommends     ca-certificates     libssl-dev     libcurl4-openssl-dev     libxml2-dev     git
 ---> Running in 8a4324cc7c53
W: Failed to fetch http://deb.debian.org/debian/dists/testing/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch http://http.debian.net/debian/dists/sid/InRelease  Temporary failure resolving 'http.debian.net'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
E: The value 'testing' is invalid for APT::Default-Release as such a release is not available in the sources

Research papers

Debugging R memory problem

Docker image for debugging R memory problems (valgrind)

Debugging with gcc problem

https://twitter.com/eddelbuettel/status/1232341601483182081

More examples

Building a Repository of Alpine-based Docker Images for R

RStudio in Docker – now share your R code effortlessly!. Markdown

https://harini.blog/2019/05/25/rstudio-and-rshiny-in-docker/

It is interesting the Dockerfile uses install2.r (R script with a shebang line) from the littler package to install R packages. See http://dirk.eddelbuettel.com/code/littler.examples.html or rocker/verse Dockerfile. But it is not clear how to install private R packages (mount host folder and use install.packages()).

Note that the tag name should be lower case; awesomer instead awesomeR.

Also got an error when trying to build the image: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease Temporary failure resolving 'deb.debian.org' .

Try it again at home. The apt update part is OK but I still got a new error: Error: installation of package ‘gifski’ had non-zero exit status

(Updated 9-18-2020) Try both the long and short commands using the tag '4.0.2' instead of '3.5.1'. Both work. The report file <example_report.pdf> is generated. In this example, the Rmd file is called through an R file. See the source code.

How to manage R package dependencies for shiny app deployment (docker)

How to manage R package dependencies for shiny app deployment (docker)

METACRAN web

https://github.com/metacran/metacranweb It works (2019-11-3).

Modified Dockerfile

Checking your Package for Compatibility with R 4.0.0

Best Practices for R with Docker

Best Practices for R with Docker

Warning: unable to load shared object 'R_X11.so'

Using rocker/rstudio:4.2 image, I got the following message when I use arrange_ggsurvplots(). 

Warning message:
In grSoftVersion() :
  unable to load shared object '/usr/local/lib/R/modules//R_X11.so':
  libXt.so.6: cannot open shared object file: No such file or directory

See https://github.com/rocker-org/rocker-versioned/issues/234, https://issuekiller.com/issues/rocker-org/rocker/81299432. 

$ ls -l /usr/local/lib/R/modules//R_X11.so
-rwxr-xr-x 1 root root 665128 May  7 00:48 /usr/local/lib/R/modules//R_X11.so

$ ldd /usr/local/lib/R/modules//R_X11.so | grep libX
        libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f74b07bb000)
        libXt.so.6 => not found
        libXrender.so.1 => /usr/lib/x86_64-linux-gnu/libXrender.so.1 (0x00007f74af598000)
        libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f74af583000)
        libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f74ad479000)
        libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f74ad46f000)

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
...

My solution is 

$ docker exec -it CONTAINER bash
# apt update
# apt install libxt-dev

Another case when I use ggsurvplot() 

Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared object '/usr/local/lib/R/site-library/xml2/libs/xml2.so':
  libxml2.so.2: cannot open shared object file: No such file or directory

$ ls -l /usr/local/lib/R/site-library/xml2/libs/
total 1404
-rwxrwxr-x 1 rstudio staff 1437536 Jun  3 12:04 xml2.so

$ ldd /usr/local/lib/R/site-library/xml2/libs/xml2.so
        linux-vdso.so.1 (0x00007ffd369b0000)
        libxml2.so.2 => not found
        libR.so => /usr/local/lib/R/lib/libR.so (0x00007faf7406a000)
 ...

Applying the same method seems to fix the problem.

The above problems happened on my mac computer. When I check the problematic files on my Ubuntu host using ldd, it does not have the same problem (but the path is a little different???) 

# ls -l /usr/local/lib/R/site-library/xml2/libs/
total 1404
-rwxrwxr-x 1 rstudio staff 1437536 May 22 19:29 xml2.so

# ldd /usr/local/lib/R/site-library/xml2/libs/xml2.so
	linux-vdso.so.1 (0x00007fff6f51d000)
	libxml2.so.2 => /lib/x86_64-linux-gnu/libxml2.so.2 (0x00007facc1dd8000)
	libR.so => not found
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007facc1bf6000)
 ...

Similar problem when I try to install tidyverse package on top of a rocker/r-rspm:22.04 container. See also piggy-back on RSPM's system dependency data base #404. To fix it, run the following in the R session 

system("apt update")
system("apt-get install libxml2-dev")

Bioconductor

(2020-1-30)

(2019-10-15)

How I use Bioconductor with Docker, Part 2: More memory, faster Bioconductor with Docker

BiocImageBuilder

Reproducible Bioconductor workflows using browser-based interactive notebooks and containers

Bioc Conference

  • Orchestra
  • Bioc2019 conference. Workshopt material. Before the meeting (6/20/2019) there are 13 downloads and it bumps to 32 downloads after the meeting for the docker image.
    • Download a tarball containing R packages (binary). 605 packages are included. It bundles 605 packages based on R 3.6.0 and Bioconductor 3.10 (BiocManager 1.30.4).
    • Run RStudio container. All R packages downloaded in the last step are mounted. (user=rstudio, ps=bioc). That is, built-in is at /usr/local/lib/R/library, custom is at /usr/local/lib/R/site-library. These two locations are what .libPaths() gives.
  • BioC 2020
    • Workshop packages were created using the BuildABiocWorkshop2020 template. From the 'Dockerfile', we see each workshop material is organized as an R package. So each workshop's package is built in the Docker image. There is no need to build the vignette again. NOTE: it takes a while to build the Docker image locally since it requires to compile each R package separately.
    • BioC 2020: Where Software and Biology Connect Opening Remarks
    • Take the recount2 workshop for example, we don't need to knit the Rmd file. To view HTML vignette, we type browseVignettes(package="recountWorkshop2020") and click the link "HTML". In case of the requested page was not found error, add help/ to the URL right after the hostname, e.g., http://localhost:8787/help/library/recountWorkshop2020/doc/recount-workshop.html. Another way to open the HTML without any tweak is type "help(package = 'recountWorkshop2020')" -> User Guide -> HTML.
    • Bioc Asia 2020
    • The vignette may not include the R code. So the Rmd file is still needed to understand the content or do a practice.

single-cell RNA-Seq

  • Docker image with rstudio for single cell analysis (github), https://hub.docker.com/r/vbarrerab/singlecell-base. Other images: https://github.com/rnakato/docker_singlecell, https://hub.docker.com/r/leanderd/single-cell-analysis 
    docker run -d -p 8787:8787 \
  --name scrna \
  -e USER='rstudio' \
  -e PASSWORD='rstudioSC' \
  -e ROOT=TRUE \
  -v /home/$USER/Documents/scrna:/home/rstudio/projects \
  vbarrerab/singlecell-base:R.4.0.3-BioC.3.11-ubuntu_20.04

    If I accidentally reboot the computer, installed packages were not lost. But it is safer to use docker stop XXX and then run docker start XXX.

  • Image containing rstudio + conda + a set of helpful packages for single cell analysis analysis,
  • docker hub,
  • Dockerfile
  • To use with Portainer, it is better to use composerize to convert the docker run command into a stack. Note Portainer cannot take version 3.x so I change version to 2. A stack/docker-compose.yml file generated by composerize with a modification on version number. 
    version: '2'
services:
    singlecell-base:
        ports:
            - '8787:8787'
        container_name: scrna
        environment:
            - USER=rstudio
            - PASSWORD=rstudioSC
            - ROOT=TRUE
        volumes:
            - '/tmp/scrna:/home/rstudio/projects'
        image: 'vbarrerab/singlecell-base:R.4.0.3-BioC.3.11-ubuntu_20.04'

    Note: the right way to delete a stack is to stop the container, and then delete the container. The final step is to select the stack and remove it.

  • Note that I have 2 pythons installed. One is from the OS (/opt/conda/bin/python) whose version is 3.8.3. The other one is on (/home/rstudio/.conda/envs/sc_env/bin/python) whose version is 3.7.8. The $PATH variable will show differences.
  • Another scRNA-Seq course including a docker image (not tried yet): Analysis of single cell RNA-seq data (ebook, University of Cambridge Bioinformatics training unit) and the paper Tutorial: guidelines for the computational analysis of single-cell RNA sequencing data Andrews 2020.

Cellar: interactive tool for analyzing single-cell omics data

Cellar is an interactive tool for analyzing single-cell omics data. Cellar is built in Python using the Dash framework and relies on several open-source packages.

Nanopore sequencing

DUESSELPORE Webserver and the paper

Docker for Shiny Server

  • Pin package versions in your production Docker image. Need to modify Dockerfile to install 'shiny' along with 'renv'. We need to create a directory 'shinyapp' with simple shiny files (eg kmeans example). 
    docker build -t pinrpackageversion .
mkdir shinyapp
wget https://raw.githubusercontent.com/rstudio/shiny-examples/master/050-kmeans-example/ui.R -P shinyapp
wget https://raw.githubusercontent.com/rstudio/shiny-examples/master/050-kmeans-example/server.R -P shinyapp
docker run --name shinyapp --rm -p 3838:3838 pinrpackageversion
  • Question: what happened if there is a new version of R or shiny-sever? 
    # specify --name so we don't need to use container id to access it later
$ docker run -d --name rshiny -p 81:3838 rocker/shiny 

$ docker ps -l

# Run in container's bash
# https://stackoverflow.com/questions/30172605/how-to-get-into-a-docker-container
# non-interactive
$ docker exec -it rshiny ip addr  
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
22: eth0@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 scope global eth0
       valid_lft forever preferred_lft forever

# interactive shell
$ docker exec -it rshiny bash 
root@5b78988d5619:/# who
root@5b78988d5619:/# which R
/usr/bin/R
root@5b78988d5619:/# ls /home
docker	shiny
root@5b78988d5619:/# ls /srv/shiny-server
01_hello  02_text  03_reactivity  04_mpg  05_sliders  06_tabsets  07_widgets  08_html  09_upload  10_download  11_timer  index.html  sample-apps
root@5b78988d5619:/# R -e 'package.Description("glmnet")'
root@5b78988d5619:/# exit

$ docker stop rshiny
$ docker rm rshiny   # remove the container

It is also useful to use the -v option in docker run so the local directory (eg /srv/shinyapps/) can be mapped to the docker volume (/srv/shiny-server). 

# Put your shiny app in host's /srv/shinyapps/appdir,
$ docker run -d -p 81:3838 \
    -v /srv/shinyapps/:/srv/shiny-server/ \
    -v /srv/shinylog/:/var/log/shiny-server/ \
    rocker/shiny

And check running status 

$ docker ps -a

R packages installation: packrat

Check opened ports

See What network ports are open: lsof or netstat. 

$ netstat -ant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN     
...
tcp6       0      0 :::80                   :::*                    LISTEN     
tcp6       0      0 :::81                   :::*                    LISTEN     
tcp6       0      0 :::22                   :::*                    LISTEN     
tcp6       0      0 ::1:25                  :::*                    LISTEN

Blog Explorer

https://github.com/nz-stefan/blog-explorer, Winners of the 2nd Annual Shiny Contest

Raspberry Pi

The Raspberry Pi 4B as a shiny server

BioContainers

CMS, blog website

Ghost

Proxy

Nginx proxy manager

Nginx

httpd/apache

https://hub.docker.com/_/httpd/

Tomcat

https://hub.docker.com/_/tomcat/

ASP.NET

Forum

How to Install Discourse Forum with Docker on CentOS 8

Jenkins for continuous integration

MySQL

  • https://hub.docker.com/r/mysql/mysql-server/
  • Deploying MySQL on Linux with Docker
  • Note it seems impossible to use the 'grant' command interactively within the MySQL docker container.
  • https://dzone.com/articles/using-docker-to-create-a-mysql-server 
    docker run --detach --name test-mysql --env="MYSQL_ROOT_PASSWORD=mypassword" mysql
docker inspect test-mysql | grep IPAddress  # 172.17.0.2

# https://dev.mysql.com/doc/index-other.html
wget -O - http://downloads.mysql.com/docs/world.sql.gz | gunzip -c > world.sql

mysql -uroot -pmypassword -h 172.17.0.2 -e 'CREATE DATABASE world'
cat world.sql | mysql -uroot -pmypassword -h 172.17.0.2 world
# zcat world.sql.gz | mysql -uroot -pmypassword -h 172.17.0.2 world
mysql -uroot -pmypassword -h 172.17.0.2 -e 'USE world; SHOW TABLES'

mysql -uroot -pmypassword -h 172.17.0.2 -e 'DROP DATABASE world'
    # Let's see if we can access the database in R
system("sudo apt install libmysqlclient-dev")
library(RMySQL)
con <- dbConnect(MySQL(), "world", user="root", password="mypassword", host="172.17.0.2")
dbListTables(con)
head(dbReadTable(con, "city"))
head(dbReadTable(con, "country"))
head(dbReadTable(con, "countrylanguage"))

dbWriteTable(con, 'iris', iris)
# Error in .local(conn, statement, ...) :
#   could not run statement: The used command is not allowed with this MySQL version
dbDisconnect(con)
  • How To Create And Test MySQL Database Server in Docker On Ubuntu Linux (video). Use Dockerfile.
  • MySQL Docker Containers: Understanding the basics, https://youtu.be/qJQPckRYNis?t=16m29s 
    # Fire up
docker run --detach --name=test-mysql --env="MYSQL_ROOT_PASSWORD=mypassword" mysql
docker ps -a

# Connecting to the Container
docker inspect test-mysql | grep IPAddress
# sudo apt install mysql-client
# There are two methods to connect to the container
# method 1. https://hub.docker.com/r/mysql/mysql-server/
# docker exec -it test-mysql mysql -uroot -p
# method 2
mysql -uroot -pmypassword -h 172.17.0.20 -P 3306
docker run --detach --name test-wordpress --link test-mysql:mysql wordpress
docker exec -it test-wordpress bash
$ cat /etc/hosts

# Expose the MySQL container to the outside world
# Access the MySQL container directly from the machine’s port 6603
docker run --detach --name=test-mysql --env="MYSQL_ROOT_PASSWORD=mypassword" --publish 6603:3306 mysql
mysql -u root -pmypassword -h 127.0.0.1 -P 6603

# Configuration management
mkdir -p /root/container/test-mysql/conf.d
nano /root/container/test-mysql/conf.d/my-custom.cnf
docker run \
    --detach \
    --name=test-mysql \
    --env="MYSQL_ROOT_PASSWORD=mypassword" \
    --publish 6603:3306 \
    --volume=/root/docker/test-mysql/conf.d:/etc/mysql/conf.d \
    mysql
mysql -uroot -pmypassword -h127.0.0.1 -P6603 -e 'show global variables like "max_connections"';
docker run \
    --detach \
    --name=test-mysql \
    --env="MYSQL_ROOT_PASSWORD=mypassword" \
    --publish 6603:3306 \
    mysql \
    --max-connections=200 \
    --character-set-server=utf8mb4 \
    --collation-server=utf8mb4_unicode_ci

# Data Storage
mkdir -p /storage/docker/mysql-datadir
docker run \
    --detach \
    --name=test-mysql \
    --env="MYSQL_ROOT_PASSWORD=mypassword" \
    --publish 6603:3306 \
    --volume=/root/docker/test-mysql/conf.d:/etc/mysql/conf.d \
    --volume=/storage/docker/mysql-datadir:/var/lib/mysql \
    mysql
docker inspect test-mysql
ls -al /storage/docker/mysql-datadir/

# If a MySQL container runs on top of an existing MySQL datadir, 
# the $MYSQL_ROOT_PASSWORD variable should be omitted from the “run” command line;
docker stop test-mysql
docker rm -f test-mysql
docker run -d --name=another-new-mysql -p 6605:3306 -v /storage/docker/mysql-datadir:/var/lib/mysql mysql
docker logs another-new-mysql
  • Use Docker to run a typical MySQL server from Code Review Videos 
    docker volume create crv_mysql

docker run \
    -e MYSQL_ROOT_PASSWORD=my-secret-pw \
    -e MYSQL_DATABASE=devdb \
    -e MYSQL_USER=dbuser \
    -e MYSQL_PASSWORD=dbpassword \
    --mount type=volume,src=crv_mysql,dst=/var/lib/mysql \
    -p 3306:3306 \
    -d \
    mysql:latest

# Alternatively we can put username/password in a text file for security
cat > .env <<EOF
MYSQL_ROOT_PASSWORD=my-secret-pw
MYSQL_DATABASE=devdb
MYSQL_USER=dbuser
MYSQL_PASSWORD=dbpassword
EOF
docker run \
    --env-file .env \
    --mount type=volume,src=crv_mysql,dst=/var/lib/mysql \
    -p 3306:3306 \
    -d \
    mysql:latest

Backup and restore

# Backup
docker exec CONTAINER /usr/bin/mysqldump -u root --password=root DATABASE > backup.sql

# Restore
# Method 1. docker exec + command
docker exec -i CONTAINER /usr/bin/mysql -u root --password=root < backup.sql
# Method 2. docker exec + bash
# We can also use docker exec log into the contain and run any mysql commands there
docker exec -it CONTAINER bash
# Method 3. mysql command only
# We directly use mysql command to import database without using docker commands
#   as long as we know the IP address of the container server
docker inspect CONTAINER | grep IPAddress
mysql -uroot -pPASSWORD -h 172.18.0.3 -e 'show databases'
mysql -uroot -pPASSWORD -h 172.18.0.3 < backup.sql

PostgreSQL

Jekyll

Building Jekyll Sites within VS Code Dev Containers

node.js

Dockerizing Node.js. See the code here. 


$ docker build -t my-node-app .
$ docker run -p 3000:3000 my-node-app

How To Setup Your Local Node.js Development Environment Using Docker (not working) 

$ cd notes-service
$ cat Dockerfile 
FROM node:12.18.1

WORKDIR /code

COPY package.json /code/package.json
COPY package-lock.json /code/package-lock.json

RUN npm install

COPY . /code

CMD [ "node", "server.js" ]

$ docker build -t notes-service .
$ docker run -it --rm -d -v mongodb:/data/db -v mongodb_config:/data/configdb -p 27017:27017 --network mongodb --name mongodb mongo
$ docker run -it --rm -d --network mongodb --name notes -p 8081:8081 -e SERVER_PORT=8081 -e DATABASE_CONNECTIONSTRING=mongodb://mongodb:27017/yoda_notes notes-service 
# this does not work though no errors; omit "-d" to see the error message

$ curl --request POST \
--url http://localhost:8081/services/m/notes \
  --header 'content-type: application/json' \
  --data '{
"name”: "this is a note",
"text": "this is a note that I wanted to take while I was working on writing a blog post.",
"owner": "peter"
}'

React

How to Dockerise A React App

Redis

https://hub.docker.com/_/redis/

Web pages for metacran

Exploring Redis with Docker

Redis running inside Docker container on NVIDIA Jetson Nano, Redis is ideal for IoT and Embedded devices for several reasons.

Laravel

Dockerizing Laravel with Nginx MySQL and Docker Compose on Ubuntu 18.04 LTS

phpMyAdmin

docker run --name mysqlphp -e MYSQL_ROOT_PASSWORD=0000 -d mysql
docker run --name myadmin -d --link mysqlphp:db -p 8080:80 phpmyadmin/phpmyadmin

Now go to your browser , tap http://localhost:8080 and login with root/0000.

ERROR mysqli_real_connect(): The server requested authentication method unknown to the client [caching_sha2_password]

The solution can be found here. After the change, I can log in phpMyAdmin. 

docker exec -it mysqlphp bash

mysql -u root -p0000

ALTER USER root IDENTIFIED WITH mysql_native_password BY '0000';

exit
exit

MagicMirror

https://github.com/MichMich/MagicMirror/blob/master/README.md#docker

Mail/smtp server

Mediawiki

  • https://www.mediawiki.org/wiki/Docker
  • https://hub.docker.com/_/mediawiki/ or (better) https://docs.docker.com/samples/library/mediawiki/. The Dockerfile is based on jessie.
  • bitnami/mediawiki (Verified but too complicated, 1M+)
  • https://hub.docker.com/r/ubcctlt/mediawiki (10M+)
  • Hosting Your Own Wiki With MediaWiki and Docker Nov 2020. The instruction is easy to follow.
    • In mediawiki set up we page, we need to enter 'database' for the Database host
    • The database information we entered will go to 'LocalSettings.php' file. But the name of wiki, administrator user name, password will go to the database.
    • the instruction for the 'extensions' part seems too much. We just need to mount local 'extensions' to container's '/var/www/extensions' folder.
    • Remember to run 'docker-compose down && docker-compose up -d' after we modify something
  • docker搭建mediawiki
  • https://github.com/kristophjunge/docker-mediawiki
  • Setup MediaWiki and Postgres with Docker (based on simplyintricate/mediawiki, works on Mint). Visit http://localhost:8080 to see the result. 
    $ ls -l ~/wiki_root
total 24
drwxr-xr-x 3 root     root     4096 Jul 10 10:09 data
drwxrwxr-x 3 brb      brb      4096 Jul 10 09:23 db
drwxrwxr-x 2 www-data www-data 4096 Jul 10 09:23 extensions
drwxrwxr-x 2 www-data www-data 4096 Jul 10 09:23 images
-rw-rw-r-- 1 www-data www-data 4152 Jul 10 11:12 LocalSettings.php
  • Running Mediawiki as Docker container (based on synctree/mediawiki which is not updated anymore, works tested on GalliumOS)
    • Follow Run MySql inside docker container first. 1. Setup host to store configuration, data and dumps. The 'dumps' and 'initdb.d' folders are optional 2. Override default mysql configuration 3. Executing scripts and sql on container startup (optional) 4. Starting mysql container
    • Visit http://IP_address:8080 to see the mediawiki page running on Docker
    brb@galliumos:~$ tree mysql
mysql
├── conf.d
│   └── mysql.cnf
├── data
│   ├── auto.cnf
│   ├── ca-key.pem
│   ├── ca.pem
│   ├── client-cert.pem
│   ├── client-key.pem
│   ├── ib_buffer_pool
│   ├── ibdata1
│   ├── ib_logfile0
│   ├── ib_logfile1
│   ├── ibtmp1
│   ├── mediawiki [error opening dir]
│   ├── mysql [error opening dir]
│   ├── performance_schema [error opening dir]
│   ├── private_key.pem
│   ├── public_key.pem
│   ├── server-cert.pem
│   ├── server-key.pem
│   ├── sys [error opening dir]
│   └── wikidb [error opening dir]
├── dumps
└── initdb.d

9 directories, 15 files

brb@galliumos:~$ tree -L 2 mediawiki/
mediawiki/
├── config
│   └── LocalSettings.php
├── extensions
│   ├── Cite
│   ├── ConfirmEdit
│   ├── Gadgets
│   ├── ImageMap
│   ├── InputBox
│   ├── Interwiki
│   ├── LocalisationUpdate
│   ├── Nuke
│   ├── ParserFunctions
│   ├── PdfHandler
│   ├── Poem
│   ├── README
│   ├── Renameuser
│   ├── SpamBlacklist
│   ├── SyntaxHighlight_GeSHi
│   ├── TitleBlacklist
│   └── WikiEditor
└── images
    └── README

19 directories, 3 files
  • https://linuxconfig.org/mediawiki-easy-deployment-with-docker-container (based on linuxconfig/mediawiki)
  • http://collabnix.com/how-to-setup-mediawiki-in-seconds-using-docker/ (based on appcontainers/mediawiki)
  • Isolated MediaWiki Setup on the Raspberry Pi using Docker, Part 1

BookStack

BookStack is a simple, self-hosted, easy-to-use platform for organising and storing information.

https://github.com/solidnerd/docker-bookstack

Media server

Youtube downloader

Airsonic media server

https://airsonic.github.io/download/

My comment:

  • Added mp3/mp4 files are not recognized.
  • Cannot play by web browsers
  • Special clients are needed:(

Plex

Official Docker container for Plex Media Server

My comment:

  • After starting, web browser cannot connect to it

Jellyfin

DNS server

Private DNS server container

How to Deploy a Dynamic DNS Server with Docker on Debian 10

WireHole

Pi-hole

Authelia

Whoogle

https://github.com/benbusby/whoogle-search

Backup

Duplicati

Python

Python Jupyter including R

docker run --rm -p 8888:8888 \
      -e JUPYTER_ENABLE_LAB=yes \
      -v "$PWD":/home/jovyan/work \
      jupyter/datascience-notebook:r-4.0.3

Follow the screen output to access the app by going to http://IP:8888/?token=XXXXXX. This will take care of the authentication problem.

This includes python3, Julia 1.0.0 and R. However, the R version is pretty old if we don't specify the tag.

Also the method does not work on arm64. WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested.

Keyboard shortcuts:

  • ctrl+Enter to run.
  • alt+Enter to run & insert a cell below.
  • shift+Enter to run and shift to a cell below.
  • Esc: command mode
  • Enter: edit mode

Note by default the ipynb files are saved to /home/jovyan instead of /home/jovyan/work directory. So it's better to click the "work" icon on the LHS of the file panel in order to change the working directory to "work".

Calibre

RSS

Home Assistant

Password manager

Vaultwarden

Vaultwarden - Open Source, self hosted backend for the Bitwarden password manager (video)

KeeWeb: password manager compatible with KeePass

https://github.com/keeweb/keeweb

Padloc: cloud-based password manager

Flask

Dockerizing Flask Application Using Docker on Debian 10

ONLYOFFICE

Gitlab

NAS & Docker

http://www.ithome.com.tw/tech/96500

Some examples

  • GetLab專案管理工具
  • Jenkins開發版本管理工具

Draw.io for diagrams and more

https://github.com/jgraph/drawio/tree/master/etc/docker 

docker run -d --rm --name="draw" -p 8080:8080 -p 8443:8443 jgraph/drawio

Internet Archive

How To Self-host Your Own Internet Archive With ArchiveBox In Linux

Run a benchmark program on ARM devices

Cheat.sh

https://github.com/chubin/cheat.sh#docker

Set up a Mastodon Instance on Linux

Your Own Social Network: How to Set Up a Mastodon Instance on Linux

Bioinformatics tools (Tophat, BWA, et al)

https://github.com/dceoy/docker-bio

cbioportal

  • Import study does not work if we use the docker-compose method. It works if we use docker exec -it CONTAINER bash. 
    [brb@localhost cbioportal-docker-compose]$ docker-compose run cbioportal metaImport.py -u http://cbioportal:8080 -s study/lgg_ucsf_2014/ -o
Creating cbioportal-docker-compose_cbioportal_run ... done
Starting validation...

CRITICAL: -: No meta files found in study/lgg_ucsf_2014/. Please make sure the directory is the path to the folder containing the files.
ERROR: -: Cancer type needs to be defined for a study. Verify that you have a study file and have defined the cancer type correctly.
ERROR: -: No valid study file detected

    See also https://github.com/cBioPortal/cbioportal/issues/7972

  • The version number shown on http://localhost:8080 is v3.4.17 (see .env variable) but the https://www.cbioportal.org/ website shows v3.5.3 (same as the current release version on Github & Docker hub)
  • The 2nd field on the 1st row of data_cancer_type.txt contains the study name (blue color) shown on website
  • meta_study.txt contains the study name (black color) shown on website
  • study_es_0 (breast, contains mRNA expression) and study_es_3 can be imported successfully when I use docker-compose exec -it CONTAINER bash and follow Load sample cancer study.
  • lgg_ucsf_2014 study (brain, CNA) should use the correct path docker-compose run cbioportal metaImport.py -u http://cbioportal:8080 -s study/lgg_ucsf_2014/lgg_ucsf_2014/ -o
  • I have tested on CentOS 7 and Ubuntu 20.04 on KVM/QEMU w/ 2 cores, 2/4GB memory, 25GB storage. I can use host's terminal and browser for interaction (i.e. no VNC/Spice server needed).
  • Gene panels in the cBioPortal patient view 2020-01-07, and other blogs related to cbioportal from thehyve.

GIVE - Genomic Interactive Visualization Engine

https://github.com/Zhong-Lab-UCSD/Genomic-Interactive-Visualization-Engine/blob/master/tutorials/GIVE-Docker.md

GeneTEFlow: Analysing gene and transposable elements expression from RNA-Seq data

GeneTEFlow: A Nextflow-based pipeline for analysing gene and transposable elements expression from RNA-Seq data. Source code.

Elastic Search

Webpage monitor tools

Retrieved from "https://wiki.taichimd.us/index.php?title=Docker_Applications&oldid=35245"