Docker Applications: Difference between revisions

From 太極
Jump to navigation Jump to search
(36 intermediate revisions by the same user not shown)
Line 3: Line 3:
* [https://noted.lol/what-are-your-most-used-self-hosted-applications What are your Most Used Self Hosted Applications?] and [https://www.reddit.com/r/selfhosted/comments/udxlnp/what_are_your_most_used_self_hosted_applications/ more]
* [https://noted.lol/what-are-your-most-used-self-hosted-applications What are your Most Used Self Hosted Applications?] and [https://www.reddit.com/r/selfhosted/comments/udxlnp/what_are_your_most_used_self_hosted_applications/ more]
* [https://youtu.be/9uF2us2PabM?si=ia76EPobqVRohIOy Best Docker Containers for Home Server!] 2022/11 (video)
* [https://youtu.be/9uF2us2PabM?si=ia76EPobqVRohIOy Best Docker Containers for Home Server!] 2022/11 (video)
== Top/best ==
* https://hub.docker.com/
* [https://github.com/veggiemonk/awesome-docker Awesome Docker]
* [https://fleet.linuxserver.io/ LinuxServer.io]. Need to sort by downloads.
* Reddit [https://www.reddit.com/r/homelab/comments/zy5io8/what_are_your_must_docker_containers_for_home_lab/ what are your must docker containers for home lab?]
* [https://www.blackvoid.club/docker-top-10-images-and-solutions/ Docker top (18) images and solutions]
* [https://www.smarthomebeginner.com/best-docker-containers-for-home-server/ 60+ Best Docker Containers for Home Server Beginners 2023]


= Templates =
= Templates =
Line 37: Line 45:


= Bioinformatics analyses =
= Bioinformatics analyses =
* [https://www.biorxiv.org/content/early/2017/10/17/204495 Hot-starting software containers for bioinformatics analyses]
<ul>
* [https://academic.oup.com/gigascience/article/6/8/1/3894236 Bio-Docklets: virtualization containers for single-step execution of NGS pipelines]
<li>[https://www.biorxiv.org/content/early/2017/10/17/204495 Hot-starting software containers for bioinformatics analyses]
* [https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1009244 Performance and scaling behavior of bioinformatic applications in virtualization environments to create awareness for the efficient use of compute resources] Hanussek, 2021
<li>[https://academic.oup.com/gigascience/article/6/8/1/3894236 Bio-Docklets: virtualization containers for single-step execution of NGS pipelines]
* [https://github.com/StaPH-B/docker-builds docker-builds] - Dockerfiles and documentation on tools for public health bioinformatics
<li>[https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1009244 Performance and scaling behavior of bioinformatic applications in virtualization environments to create awareness for the efficient use of compute resources] Hanussek, 2021
* [https://github.com/shmohammadi86/ACTIONet/tree/R-release/mini_docker ACTIONet] and the paper [https://pubmed.ncbi.nlm.nih.gov/36650127/ Transcriptional vulnerabilities of striatal neurons in human and rodent models of Huntington's disease] 2023.
<li>[https://github.com/StaPH-B/docker-builds docker-builds] - Dockerfiles and documentation on tools for public health bioinformatics
<li>[https://github.com/shmohammadi86/ACTIONet/tree/R-release/mini_docker ACTIONet] and the paper [https://pubmed.ncbi.nlm.nih.gov/36650127/ Transcriptional vulnerabilities of striatal neurons in human and rodent models of Huntington's disease] 2023.
<li>[https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-024-05695-9 CREDO: a friendly Customizable, REproducible, DOcker file generator for bioinformatics applications] 2024
 
[[File:CredoError.png|240px]]
</ul>


= Cloud provider =
= Cloud provider =
Line 56: Line 69:
== Dashy ==
== Dashy ==
* https://github.com/Lissy93/dashy
* https://github.com/Lissy93/dashy
* [https://youtu.be/QsQUzutGarA Dashy, a configurable, themable, flexible personal dashboard for your homelab!] (video) and [https://shownotes.opensourceisawesome.com/dashy-powerful-informative/ Notes]
* [https://youtu.be/QsQUzutGarA Dashy, a configurable, themable, flexible personal dashboard for your homelab!] (video) and [https://wiki.opensourceisawesome.com/books/self-hosted-dashboards/page/dashy-powerful-informative-configurable-self-hosting-dashboard Notes]
** https://github.com/walkxcode/dashboard-icons for icons
* Icons are located in ~/dash/icons/dashboard-icons/png directory.
* Icons are located in ~/dash/icons/dashboard-icons/png directory.
* Download an icon from [https://icons8.com/icons/ icons8.com] or [https://uxwing.com/ uxwing.com]. The default 48x48 works fine.
* Better to work on one icon and then save the change. Repeat.
<pre>
<pre>
docker run -d \
docker run -d \
Line 67: Line 83:
   lissy93/dashy:latest
   lissy93/dashy:latest
</pre>
</pre>
== Glance ==
* https://github.com/glanceapp/glance
* [https://www.youtube.com/watch?v=2dGD4C2g8ig Glance Dashboard: Your New Favorite Docker Dashboard]
== Homepage ==
https://github.com/gethomepage/homepage
== Flame ==
https://github.com/pawelmalak/flame


== Homer ==
== Homer ==
Line 102: Line 127:
= Google photo =
= Google photo =
* [https://docs.photoprism.app/getting-started/docker/ PhotoPrism]
* [https://docs.photoprism.app/getting-started/docker/ PhotoPrism]
== Immich: Google photo replacment ==
* [https://github.com/immich-app/immich immich] - High performance self-hosted photo and video backup solution
* [https://www.makeuseof.com/raspberry-pi-immich-google-photos-alternative/ Immich Is a Great Google Photos Alternative You Can Host on Raspberry Pi]
* [https://www.makeuseof.com/raspberry-pi-immich-google-photos-alternative/ Immich Is a Great Google Photos Alternative You Can Host on Raspberry Pi]
* [https://missilepuppy.com/blog/2024/01/09/immich-finally-a-good-self-hosted-google-photos-replacement/ Immich – Finally, A Good Self-Hosted Google Photos Replacement]


= ownCloud =
= ownCloud =
Line 149: Line 178:


== [https://apps.nextcloud.com/ Nextcloud Apps] ==
== [https://apps.nextcloud.com/ Nextcloud Apps] ==
For mp3, no apps are shipped with nextcloud.  
* [https://docs.nextcloud.com/server/15/admin_manual/installation/installation_wizard.html#trusted-domains "access through untrusted domain" error]. On my Casaos, I found the file ''/DATA/AppData/nextcloud/var/www/html/config/config.php'' shows trusted domain is casaos.local:10081. So I just need to change the app's setting in CasaOS to use http://casaos.local:10081 instead of http://IP:10081 to open the page.
 
* [https://apps.nextcloud.com/apps/audioplayer Audio Player]. When I play mp3 files, the web interface does not show a progress bar:( <syntaxhighlight lang='bash'>
* [https://apps.nextcloud.com/apps/audioplayer Audio Player]. When I play mp3 files, the web interface does not show a progress bar:( <syntaxhighlight lang='bash'>
$ docker run --rm --name mynextcloud -v ~/Downloads/nextcloudapps:/var/www/html/custom_apps -p 81:80 nextcloud
$ docker run --rm --name mynextcloud -v ~/Downloads/nextcloudapps:/var/www/html/custom_apps -p 81:80 nextcloud
Line 234: Line 262:
* [https://itsfoss.com/deploy-seafile-server-docker/ How to Deploy Seafile Server with Docker to Host Your Own File Synchronization and Sharing Solution]
* [https://itsfoss.com/deploy-seafile-server-docker/ How to Deploy Seafile Server with Docker to Host Your Own File Synchronization and Sharing Solution]


= Run a Linux desktop VM with VNC =
= Run a Linux desktop =
 
== Run a Linux desktop VM with VNC ==
[https://computingforgeeks.com/run-ubuntu-linux-in-docker-with-desktop-environment-and-vnc/ Run Ubuntu Linux in Docker with Desktop Environment and VNC]
[https://computingforgeeks.com/run-ubuntu-linux-in-docker-with-desktop-environment-and-vnc/ Run Ubuntu Linux in Docker with Desktop Environment and VNC]
== Distrobox ==
* https://github.com/89luca89/distrobox
* [https://www.howtogeek.com/run-any-app-on-any-linux-distro-with-distrobox/ DEB Files on Arch? Run Any App on Any Linux Distro With This Tool]
* [https://youtu.be/Sj064D9ZUl8 Stop Distro Hopping! Use this AWESOME tool on Linux] 7/15/2023
** Linux subsystem in another Linux


= Guacamole =
= Guacamole =
Line 251: Line 287:


== Webtop: Linux desktop in a browser ==
== Webtop: Linux desktop in a browser ==
* https://hub.docker.com/r/linuxserver/webtop/. Ubuntu, Alpine, Arch, and Fedora based Webtop images, Linux in a web browser supporting popular desktop environments.
* https://hub.docker.com/r/linuxserver/webtop/, or https://github.com/linuxserver/docker-webtop. Ubuntu, Alpine, Arch, and Fedora based Webtop images, Linux in a web browser supporting popular desktop environments.
* An example. It seems the host's Disk space, Memory are all passed to VM. In this example, the host's directory "/tmp/downloads" will serve as the HOME directory in VM.
:<syntaxhighlight lang='sh'>
mkdir /tmp/downloads/
docker run -d --name=webtop  \
  -e PUID=1000 -e PGID=1000  \
  -e TZ=US/Eastern  \
  -p 3000:3000  \
  -v /tmp/downloads:/config  \
  --shm-size="1gb"  \
  --restart unless-stopped  \
  linuxserver/webtop:ubuntu-xfce
</syntaxhighlight>
* [https://www.linuxserver.io/blog/2021-05-05-meet-webtops-a-linux-desktop-environment-in-your-browser Meet Webtops A Linux Desktop Environment In Your Browser]
* [https://www.linuxserver.io/blog/2021-05-05-meet-webtops-a-linux-desktop-environment-in-your-browser Meet Webtops A Linux Desktop Environment In Your Browser]
* [https://youtu.be/bQrtnZ1UrQQ Run Linux Desktops in your Browser with Docker and Webtops!] (video) and [https://shownotes.opensourceisawesome.com/webtops/ Notes]
* [https://youtu.be/bQrtnZ1UrQQ Run Linux Desktops in your Browser with Docker and Webtops!] (video) and [https://wiki.opensourceisawesome.com/books/webtops-for-remote-sessions/page/webtops-linux-desktops-in-docker Notes]
* Default username/password is abc/abc
* Default username/password is abc/abc. But we don't need to enter the password when we use "sudo".
* [https://youtu.be/Gd9bvdkIXOQ Linux desktop, inside of a container, inside of a browser??? Yes. A Webtop], [https://youtu.be/Gd9bvdkIXOQ?t=811 privileged: true] is required for installing something like vscode in VM (video) and [https://docs.technotim.live/posts/webtop-container/ Doc]
* [https://youtu.be/Gd9bvdkIXOQ Linux desktop, inside of a container, inside of a browser??? Yes. A Webtop], [https://youtu.be/Gd9bvdkIXOQ?t=811 privileged: true] is required for installing something like vscode in VM (video) and [https://docs.technotim.live/posts/webtop-container/ Doc]
* If I'm not using the default desktop, I need to switch to use some user's account instead of root to run docker-compose command. If I use root to launch ubuntu-kde, for example, the desktop is blank.
* If I'm not using the default desktop, I need to switch to use some user's account instead of root to run docker-compose command. If I use root to launch ubuntu-kde, for example, the desktop is blank.
Line 300: Line 348:
* [https://hub.docker.com/r/gitpod/openvscode-server OpenVSCode Server] 1M+ pulls. [https://github.com/coder/code-server/discussions/4267 Difference to OpenVSCode Server].
* [https://hub.docker.com/r/gitpod/openvscode-server OpenVSCode Server] 1M+ pulls. [https://github.com/coder/code-server/discussions/4267 Difference to OpenVSCode Server].


= Use with R (r-base) & RStudio IDE: Rocker =
= Docker for R (r-base, rstudio) =
<ul>
See [[R_Docker|R Docker]].
<li>[https://solutions.rstudio.com/environments/docker/ Docker 101 for Data Scientists] by RStudio </li>
<li>[https://hub.docker.com/_/r-base r-base] (Official image, R version is tagged), [https://hub.docker.com/r/rocker/rstudio/tags RStudio]
<ul>
<li>The oldest version of R is 3.1.2 (2014-10-31). '''docke run -it --rm r-base:3.1.2'''
<li>[https://www.rocker-project.org/use/managing_users/ Managing Users] </li>
<li>[https://colinfay.me/docker-r-reproducibility/ An Introduction to Docker for R Users]: how to write your own <Dockerfile>, install packages, run a script and get results.</li>
<li>Extensions from r-base. For example, [https://github.com/rocker-org/rocker/issues/119 r-spatial-base]. It also mentions [https://registry.hub.docker.com/u/rocker/ropensci/ ropensci] container is built upon rocker/rstudio. </li>
<li>The r-base image does not have pdflatex, git. Need to manually install them.</li>
<li>Not sure if the Docker Official Image is the same as the one provided by Rocker Project.</li>
<li>NOTE: Plotting works by forwarding X11. The instruction depends on the host OS. See rocker [https://github.com/rocker-org/rocker/wiki/Allowing-GUI-windows Wiki] or the command below. Creating graphics files inside a container is still OK 👌; see the example [https://jlintusaari.github.io/2018/07/how-to-compile-rmarkdown-documents-using-docker/ How to compile R Markdown documents using Docker].
{{Pre}}
docker pull r-base:3.5.3
docker run -it --rm rocker/drd RD              # a little smaller, 3.6GB for R 4.0
docker run -it --rm rocker/drd R              # good to test the pipe operator (due in R 4.1.0)
docker run -it --rm rocker/r-devel RD          # initial one, larger, 5.7GB for R 4.0
docker run -it --rm rocker/r-devel R          # r-release
docker run -it --rm r-base:3.5.3              # default is root "/"
docker run -it --rm rocker/r-rspm:22.04        # seems the 'latest' tag is missing
docker run -it --rm rocker/r-bspm:22.04        # better than r-rspm in the case of 'tidyverse'
                                              # since bspm can take care of missing system OS libraries
                                              # 'Many' Bioconductor packages like DESeq2/limma/sva are avail
docker run -v ~/Downloads:/src -it --rm r-base # /src does not exist
docker run -v ~/Downloads:/home/docker -it --rm r-base # /home/docker exists and is empty by default
                                              # setwd("/home/docker")
docker run -it --rm -u1000:1000 -e DISPLAY=$DISPLAY \
          -v /tmp/.X11-unix:/tmp/.X11-unix \
          -v $(pwd):/work -w /work r-base
 
docker run -it --rm -p 8787:8787 \
      -v $(pwd):/home/rstudio/project \
      -e PASSWORD=mypassword \
      -w /home/rstudio/project rocker/tidyverse
 
# Disable authorization
docker run -it --rm -p 8787:8787 \
      -v $(pwd):/home/rstudio/project \
      -e DISABLE_AUTH=true \
      -w /home/rstudio/project rocker/tidyverse:4.2
</pre>
The "-u" option causes an error "s6-supervise (child): fatal: unable to exec run: Permission denied" (2/19/2023). According to the [https://rocker-project.org/images/versioned/rstudio.html rocker/tidyverse] documentation, '''the non-root default user rstudio is set up as RStudio Server user.''' So it is not needed to use a different username.
<pre>
docker run -ti --rm -v "$PWD":/home/docker -w /home/docker \
      -u docker r-base bash  # Non-root user
 
docker run -ti --rm -v "$PWD":/home/rstudio -w /home/rstudio \
      -u rstudio rocker/rstudio bash  # Non-root user
</pre>
</li>
</ul>
</li>
</ul>
* Dirk.eddelbuettel
** slides http://dirk.eddelbuettel.com/presentations/
** http://dirk.eddelbuettel.com/papers/useR2014_keynote.pdf
** http://dirk.eddelbuettel.com/papers/r_and_docker_sep2014.pdf
** [http://dirk.eddelbuettel.com/blog/2014/10/23/ Introducing Rocker: Docker for R]
** [http://dirk.eddelbuettel.com/blog/2019/08/05/#023_rocker_debug_example Debugging with Docker and Rocker – A Concrete Example helping on macOS]
** [https://arxiv.org/abs/2001.10641 The Rockerverse: Packages and Applications for Containerization with R]
** [https://github.com/rocker-org/rocker/tree/master/r-bspm #29: Easy, Reliable, Fast Linux CRAN Binaries via BSPM], [https://github.com/rocker-org/rocker/tree/master/r-bspm Source] on Github
* [https://www.rocker-project.org/ The Rocker Project] (<strike>only has the latest version</strike>)
** https://github.com/rocker-org/rocker Github repository
** [https://github.com/rocker-org/rocker-versioned/blob/master/VERSIONS.md R, MRAN, RStudio Server versions correspondance]
** [https://github.com/rocker-org/rocker-versioned/blob/master/r-dependencies.md What dependencies are included in included in r-ver?]
* RStudio server recommends 8GB memory. So I cannot use it in Play with Docker.
* [https://github.com/rocker-org/rocker/wiki/Using-the-RStudio-image Using the RStudio image] includes the official instruction for the latest change. PS the default username is '''rstudio'''. <syntaxhighlight lang='bash'>
docker run -d -p 8787:8787 \
  -e PASSWORD=<password> \
  -v "$PWD":/home/rstudio -w /home/rstudio \
  --name rstudio rocker/rstudio
# mount a local directory, change the username. A new directory /src will be created.
docker run -d -p 8787:8787 \
  -v ~/Downloads:/src \
  -e USER=<username>
  -e PASSWORD=<password> \
  --name rstudio rocker/rstudio
 
# Use R, without RStudio
docker run -it -e DISABLE_AUTH=true rocker/rstudio:3.5.3 R
# Use Bash without RStudio, mount the current directory
docker run -it -e DISABLE_AUTH=true -v $(pwd):/home/rstudio/project rocker/rstudio:3.5.3 bash
 
# '-e ROOT=TRUE' so that I can be root in the Terminal and test some installation steps
# before creating my own derivated Dockerfile
docker run -d -p 8787:8787 -e ROOT=TRUE -e DISABLE_AUTH=true rocker/rstudio:devel
 
# Fix an error when running library(tidyverse): libxml2.so.2 no such file
$ docker exec -it rstudio bash
# apt update
# apt upgrade
# apt install libcurl4-openssl-dev libssl-dev libxml2-dev
</syntaxhighlight>
** [https://github.com/rocker-org/rocker/wiki/Sharing-files-with-host-machine Sharing files with host machine]
** [http://www.magesblog.com/2014/09/running-rstudio-via-docker-in-cloud.html Running RStudio in cloud]
** [https://sbamin.com/blog/2016/02/running_rstudio_in_docker_environment/ Running RStudio in a docker container]
* [http://ropenscilabs.github.io/r-docker-tutorial/ R Docker tutorial] from ropenscilabs. It covers sharing your analysis.
 
== Dockerfile ==
Create a new directory and a new file 'Dockerfile' with the content.
 
<pre>
FROM debian:testing
MAINTAINER Dirk Eddelbuettel [email protected]
## Remain current
RUN apt-get update -qq
RUN apt-get dist-upgrade -y
RUN apt-get install -y --no-install-recommends r-base r-base-dev r-recommended littler
RUN ln -s /usr/share/doc/littler/examples/install.r /usr/local/bin/install.r
</pre>
 
NOTE
# From [https://hub.docker.com/_/r-base r-base] in DockerHub click the "latest" in the "Supported tags and respective Dockerfile links" section.
# I ran into errors when I use the above (short) Dockerfile. But the Dockerfile from rocker (leave out the last line of launching R) works well. The R packages built in the image include 'docopt', 'magrittr', 'stringi', and 'stringr'.
# Install R package is possible when we launch a container. But we are not able to save the packages?? The [https://github.com/rocker-org/rocker/wiki rocker wiki] also mentions something about installing packages.
# See also [https://github.com/rocker-org/rocker/wiki/How-to-save-data How to save data] in wiki.
 
== install2.r ==
The '''install2.r''' command can be used to concisely describe the installation of the R package in the Dockerfile. See
[https://rocker-project.org/use/extending.html Rocker project - Extending images]
 
== A quick run of an R script ==
<pre>
docker run --rm \
  -v $(pwd):/tmp/working_dir \
  -w /tmp/working_dir \
  rocker/tidyverse:latest \
  Rscript my_script.R
</pre>
 
== docker run ==
Note that if we are using the Dockerfile above to create an image, we will be dropped to the Linux shell. If we are pulling the rocker/r-base image from Docker Hub, we will be in R console directly. See the last line of Dockerfile on [https://github.com/rocker-org/rocker/blob/master/r-base/Dockerfile github] website for rocker.
 
Then run the following to do some exercise (We could possibly replace 21b6a9e8b9e8 with your image ID or use ''rocker/r-base''). For simplicity, we can try the [http://cran.rstudio.com/web/packages/colortools/index.html colortools package] first which does not depend on other packages and there is no need to compile the package.
<syntaxhighlight lang='bash'>
sudo docker build -t debian:testing-add-r . # create an image based on the above Dockerfile
wget http://cran.r-project.org/src/contrib/sanitizers_0.1.0.tar.gz
sudo docker run -v `pwd`:/mytmp -t 21b6a9e8b9e8 \
    R CMD check --no-manual --no-build-vignettes /mytmp/sanitizers_0.1.0.tar.gz
sudo docker run -v `pwd`:/mytmp -t 21b6a9e8b9e8 \
    Rdevel CMD check --no-manual --no-build-vignettes /mytmp/sanitizers_0.1.0.tar.gz
 
sudo docker search eddelbuettel
sudo docker pull eddelbuettel/docker-ubuntu-r  # default tag is 'latest'; actually older than the other tags
sudo docker images eddelbuettel/docker-ubuntu-r # see the tag column
sudo docker pull eddelbuettel/docker-ubuntu-r:add-r # the tag name can only be obtained from hub.docker.com
sudo docker images eddelbuettel/docker-ubuntu-r # see the tag column
sudo docker pull eddelbuettel/docker-ubuntu-r:add-r-devel
sudo docker images eddelbuettel/docker-ubuntu-r # see the tag column
sudo docker run -v `pwd`:/mytmp -t 54d865dbd2c9 R CMD check --no-manual --no-build-vignettes /mytmp/sanitizers_0.1.0.tar.gz
 
sudo docker run -t -i eddelbuettel/docker-ubuntu-r /bin/bash
$ sudo docker images
REPOSITORY                    TAG                IMAGE ID            CREATED            VIRTUAL SIZE
debian                        testing-add-r      21b6a9e8b9e8        28 minutes ago      572.2 MB
ubuntu                        14.04              ed5a78b7b42b        4 days ago          188.3 MB
ubuntu                        latest              ed5a78b7b42b        4 days ago          188.3 MB
debian                        testing            88ba2870bfbe        7 weeks ago        154.7 MB
eddelbuettel/docker-ubuntu-r  add-r-devel        c998a74a1fb4        11 weeks ago        460.4 MB
eddelbuettel/docker-ubuntu-r  add-r              54d865dbd2c9        11 weeks ago        460.4 MB
eddelbuettel/docker-ubuntu-r  latest              a7cd5ddeb98e        5 months ago        515.4 MB
 
sudo docker logs xxxxx                # view the log
sudo docker restart xxxxx
sudo docker exec -it xxxx /bin/bash  # view any changes in R library
sudo docker stop xxxxx
sudo docker rm xxxxx
</syntaxhighlight>
 
[https://hub.docker.com/r/dennishazelett/motifbreakr/ This] is another example of using 'docker run' accompanying [https://github.com/Simon-Coetzee/MotifBreakR MotifBreakR] package.
 
== Testing a new R release ==
[https://twitter.com/eddelbuettel/status/1384510084622032900 R 4.1.0]
<pre>
docker pull rocker/r-base:4.1.0
 
alias dkrr='docker run --rm -it -u1000:1000 -v$(pwd):/work -w /work'
dkrr rocker/r-ubuntu:20.04 bash
dkrr r-base:latest R --version | head -1
dkrr r-base:3.6.3 R --version | head -1
 
# Assume we are in a directory called 'curse'
# (Yes you may need to add Depends and LaTeX support ...)
# Even the 'survival' package requires pdflatex in 'R CMD build' step.
# A toy package like https://cran.r-project.org/web/packages/QuadRoot/ works
dkrr rocker/r-base:4.1.0 R CMD build .  # this will create curse_1.0.0.tar.gz
dkrr rocker/r-base:4.1.0 R CMD check --no-vignettes --no-manual curse_1.0.0.tar.gz
</pre>
A closer solution is to use ''rocker/verse:4.0.4'' image. But it gives different errors
<ul>
<li>survival: LaTeX Error: File `fancyvrb.sty' not found. </li>
<li>glmnet: dependencies ‘foreach’, ‘shape’ are not available. A workaround solution (need to figure out the dependencies first):
<pre>
$ curl -s https://cran.r-project.org/src/contrib/glmnet_4.1-1.tar.gz | tar xzv
$ cd glmnet
$ docker run --rm -it -v$(pwd):/work -w /work rocker/verse:4.0.4 bash
# Rscript -e "install.packages(c('foreach', 'shape', 'knitr', 'lars', 'testthat', 'xfun', 'rmarkdown'))"
# su rstudio
$ R CMD build .
$ exit
# exit
</pre>
If we don't install the 'Suggests' packages, building will fail when it was trying to build the vignette. A more relaxed solution is adding the option '''--no-build-vignettes'''
</li>
</ul>
 
== rocker/verse images ==
The [https://rocker-project.org/images/ rocker/verse] images contain the '''curl''' utility & '''libcurl4-openssl-dev''' & '''libxml2-dev''' which are needed when I use the '''groundhog''' package to install packages. The tidyverse images don't include these libraries/utility. Tested on 3.6.3.
 
Note that these versioned images may be using MRAN. For example for rocker/verse:3.6.3,
<pre>
> options()$repo
                                            CRAN
"https://mran.microsoft.com/snapshot/2020-04-24"
</pre>
For rocker/verse:4.2.3
<pre>
> options()$repo
                                                            CRAN
"https://packagemanager.posit.co/cran/__linux__/jammy/2023-04-20"
</pre>
For rocker/verse:4.3.0
<pre>
> options()$repo
                                                        CRAN
"https://packagemanager.posit.co/cran/__linux__/jammy/latest"
</pre>
 
== Testing R packages ==
* [http://cran.rstudio.com/web/packages/RColorBrewer/index.html RColorBrewer] (pure R)
* [http://cran.rstudio.com/web/packages/digest/index.html digtest] (need compile, with vignette) [https://github.com/eddelbuettel/digest github source]
* [http://cran.rstudio.com/web/packages/randomForest/index.html randomForest] (need compile & has suggests RColorBrewer)
 
== Multiple containers ==
* When I run multiple containers at the same time with different ports, only 1 container can be logged in OR reload constantly. The problem happened on Linux host and Mac host.
* It seems to be related to '''cookies'''.
** One can get rid of the problem by open a new tab in [https://askubuntu.com/a/1397664 incognito mode], or
** '''use a different domain name''' (e.g. localtest.me or modifying your /etc/hosts file) or different browsers.
* [https://github.com/rocker-org/rocker-versioned2/issues/171 Can't login into servers on different ports on the same computer #171]
* [https://github.com/rstudio/rstudio/issues/9175 Issue with RStudio user authentication cookies #9175]
* [https://stackoverflow.com/q/71856297 Running multiple instances of RStudio in Docker]
 
== Docker for R Package Development ==
http://www.jimhester.com/2017/10/13/docker/
 
== A DevOps Perspective ==
* https://www.opencpu.org/posts/opencpu-with-docker/
 
== Reproducible ==
* [https://journal.r-project.org/archive/2020/RJ-2020-007/RJ-2020-007.pdf The Rockerverse: Packages and Applications for Containerisation with R] 2020
** [https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1008316 Ten Simple Rules for Writing Dockerfiles for Reproducible Data Science], [https://github.com/nuest/ten-simple-rules-dockerfiles code in github]
** [https://www.theoj.org/joss-papers/joss.01603/10.21105.joss.01603.pdf containerit: Generating Dockerfiles for reproducible research with R], [https://o2r.info/containerit/ containerit] package with GUI via [https://rstudio.github.io/rstudioaddins/ RStudio Addins].
<ul>
<li>A simple example to produce a PDF report from an Rmarkdown file by using [https://hub.docker.com/r/rocker/verse rocker/verse] image:
<pre>
docker run -it --rm --name pdmr -v $(pwd):/home/docker -w /home/docker \
      --user rstudio rocker/verse:4.0.3 \
      Rscript -e "rmarkdown::render('Example.Rmd')"
</pre>
</li>
<li>[https://datawookie.netlify.com/blog/2019/08/r-docker-and-checkpoint-a-route-to-reproducibility/ R, Docker and Checkpoint: A Route to Reproducibility]
</li>
<li>[https://cran.r-project.org/web/packages/liftr/index.html liftr]: Persistent reproducible reporting by containerization of R Markdown documents. I am not successful running the example in vignette.
{{Pre}}
> render_docker(input)
Sending build context to Docker daemon  6.144kB
Step 1/8 : FROM rocker/r-base:latest
---> 4e232e9b4097
Step 2/8 : MAINTAINER Nan Xiao <[email protected]>
---> Running in f75404db590a
Removing intermediate container f75404db590a
---> ceae0a54648c
Step 3/8 : RUN  rm -f /var/lib/dpkg/available  && rm -rf  /var/cache/apt/*  && apt-get update -qq  && apt-get install -y
  --no-install-recommends    ca-certificates    libssl-dev    libcurl4-openssl-dev    libxml2-dev    git
---> Running in 8a4324cc7c53
W: Failed to fetch http://deb.debian.org/debian/dists/testing/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch http://http.debian.net/debian/dists/sid/InRelease  Temporary failure resolving 'http.debian.net'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
E: The value 'testing' is invalid for APT::Default-Release as such a release is not available in the sources
</pre>
</li>
</ul>
 
== Research papers ==
* [https://github.com/qianhuiSenn/scRNA_cell_deconv_benchmark Evaluation of Cell Type Annotation R Packages on Single-cell RNA-seq Data] 2020
 
== Debugging R memory problem ==
[https://hub.docker.com/r/wch1/r-debug/ Docker image for debugging R memory problems] using the [https://github.com/wch/r-debug r-debug] container (valgrind)
 
== Debugging with gcc problem ==
https://twitter.com/eddelbuettel/status/1232341601483182081
 
== More examples ==
* [[#Docker|R -> Docker]]
* [https://www.rplumber.io/docs/hosting.html#docker Plumber] package
 
== Building a Repository of Alpine-based Docker Images for R ==
* [https://velaco.github.io/my-dockerfile-for-r-shiny-based-on-alpine-linux/ Part I]
* [https://velaco.github.io/my-dockerfile-for-r-shiny-based-on-alpine-linux-II/ Part II]
 
== RStudio in Docker – now share your R code effortlessly!. Markdown ==
https://harini.blog/2019/05/25/rstudio-and-rshiny-in-docker/
 
It is interesting the Dockerfile uses '''install2.r''' (R script with a shebang line) from the [https://cran.r-project.org/web/packages/littler/ littler] package to install R packages. See http://dirk.eddelbuettel.com/code/littler.examples.html or [https://hub.docker.com/r/rocker/verse/dockerfile rocker/verse] Dockerfile. But it is not clear how to install private R packages (mount host folder and use install.packages()).
 
Note that the tag name should be lower case; awesomer instead awesomeR.
 
Also got an error when trying to build the image: ''Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease  Temporary failure resolving 'deb.debian.org' ''.
 
Try it again at home. The ''apt update'' part is OK but I still got a new error: ''Error: installation of package ‘gifski’ had non-zero exit status''
 
(Updated 9-18-2020) Try both the long and short commands using the tag '4.0.2' instead of '3.5.1'. Both work. The report file <example_report.pdf> is generated. In this example, the Rmd file is called through an R file. See [https://github.com/jlintusaari/R-docker-report the source code].
 
== Dockerized RStudio Server, packages (renv) ==
[https://www.r-bloggers.com/2022/10/homelab-6-dockerized-rstudio-server-packages-persistent-storage-and-ssl-certs/ HomeLab 6: Dockerized RStudio Server, packages, persistent storage and SSL certs]
 
== How to manage R package dependencies for shiny app deployment (docker) ==
[https://community.rstudio.com/t/how-to-manage-r-package-dependencies-for-shiny-app-deployment-docker/18593 How to manage R package dependencies for shiny app deployment (docker)]
 
== METACRAN web ==
https://github.com/metacran/metacranweb It works (2019-11-3).
 
== Modified Dockerfile ==
[https://joachim-gassen.github.io/2020/05/checking-your-package-for-compatibility-with-r-4/ Checking your Package for Compatibility with R 4.0.0]
 
== Best Practices for R with Docker ==
[https://hosting.analythium.io/best-practices-for-r-with-docker/ Best Practices for R with Docker]
 
== Warning: unable to load shared object 'R_X11.so' ==
Using rocker/rstudio:4.2 image, I got the following message when I use arrange_ggsurvplots().
<pre>
Warning message:
In grSoftVersion() :
  unable to load shared object '/usr/local/lib/R/modules//R_X11.so':
  libXt.so.6: cannot open shared object file: No such file or directory
</pre>
See https://github.com/rocker-org/rocker-versioned/issues/234, https://issuekiller.com/issues/rocker-org/rocker/81299432.
{{Pre}}
$ ls -l /usr/local/lib/R/modules//R_X11.so
-rwxr-xr-x 1 root root 665128 May  7 00:48 /usr/local/lib/R/modules//R_X11.so
 
$ ldd /usr/local/lib/R/modules//R_X11.so | grep libX
        libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f74b07bb000)
        libXt.so.6 => not found
        libXrender.so.1 => /usr/lib/x86_64-linux-gnu/libXrender.so.1 (0x00007f74af598000)
        libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f74af583000)
        libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f74ad479000)
        libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f74ad46f000)
 
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
...
</pre>
My solution is
<pre>
$ docker exec -it CONTAINER bash
# apt update
# apt install libxt-dev
</pre>
Another case when I use ggsurvplot()
{{Pre}}
Error in dyn.load(file, DLLpath = DLLpath, ...) :
  unable to load shared object '/usr/local/lib/R/site-library/xml2/libs/xml2.so':
  libxml2.so.2: cannot open shared object file: No such file or directory
</pre>
{{Pre}}
$ ls -l /usr/local/lib/R/site-library/xml2/libs/
total 1404
-rwxrwxr-x 1 rstudio staff 1437536 Jun  3 12:04 xml2.so
 
$ ldd /usr/local/lib/R/site-library/xml2/libs/xml2.so
        linux-vdso.so.1 (0x00007ffd369b0000)
        libxml2.so.2 => not found
        libR.so => /usr/local/lib/R/lib/libR.so (0x00007faf7406a000)
...
</pre>
Applying the same method seems to fix the problem.
 
The above problems happened on my mac computer. When I check the problematic files on my Ubuntu host using ''ldd'', it does not have the same problem (but the path is a little different???)
{{Pre}}
# ls -l /usr/local/lib/R/site-library/xml2/libs/
total 1404
-rwxrwxr-x 1 rstudio staff 1437536 May 22 19:29 xml2.so
 
# ldd /usr/local/lib/R/site-library/xml2/libs/xml2.so
linux-vdso.so.1 (0x00007fff6f51d000)
libxml2.so.2 => /lib/x86_64-linux-gnu/libxml2.so.2 (0x00007facc1dd8000)
libR.so => not found
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007facc1bf6000)
...
</pre>
 
Similar problem when I try to install tidyverse package on top of a rocker/r-rspm:22.04 container. See also [https://github.com/rocker-org/rocker/issues/404 piggy-back on RSPM's system dependency data base #404]. To fix it, run the following in the R session
<pre>
system("apt update")
system("apt-get install libxml2-dev")
</pre>
 
= [http://www.bioconductor.org/help/docker/ Bioconductor] =
(2020-1-30)
<ul>
<li>https://bioconductor.org/help/docker/
<li>https://hub.docker.com/r/bioconductor/bioconductor_docker (support as early as RELEASE_3_10 with R 3.6.3)
<pre>
docker run \
    -e PASSWORD=bioc \
    -p 8787:8787 \
    bioconductor/bioconductor_docker:devel
 
docker run -it --user rstudio bioconductor/bioconductor_docker:RELEASE_3_10 R
 
docker run -it --user rstudio bioconductor/bioconductor_docker:devel bash
</pre>
One case (Bioc 3.14 works with R 4.1.3)
<pre>
docker run -it --name hungry_wiles \
      -v $(pwd):/home/rstudio -w /home/rstudio \
      --user rstudio \
      bioconductor/bioconductor_docker:RELEASE_3_14 R
# q()
# docker restart hungry_wiles
# docker attach hungry_wiles
</pre>
<li>https://github.com/Bioconductor/bioconductor_docker
</ul>
 
(2019-10-15)
* [https://hub.docker.com/r/bioconductor/bioconductor_full bioconductor_full]
* [https://seandavi.github.io/post/build-linux-r-binary-packages/ Building R Binary Packages for Linux]
 
[https://medium.com/@nitesh.turaga/how-i-use-bioconductor-with-docker-ca9d17d87771 How I use Bioconductor with Docker], [https://medium.com/@nitesh.turaga/how-i-use-bioconductor-with-docker-part-2-more-memory-faster-bioconductor-with-docker-fd7e69dce2bf Part 2: More memory, faster Bioconductor with Docker]
 
== [https://github.com/Bioconductor-notebooks/BiocImageBuilder BiocImageBuilder] ==
[http://www.rna-seqblog.com/reproducible-bioconductor-workflows-using-browser-based-interactive-notebooks-and-containers/ Reproducible Bioconductor workflows using browser-based interactive notebooks and containers]
 
== ARM64 ==
[https://blog.bioconductor.org/posts/2023-07-14-linux-arm64-github-actions/ Testing Packages on Linux ARM64 with GitHub Actions]
 
== Bioc Conference ==
* [http://app.orchestra.cancerdatasci.org/ Orchestra]
* [https://github.com/Bioconductor/BiocWorkshops2019/blob/master/README.md Bioc2019 conference]. [http://biocworkshops2019.bioconductor.org.s3-website-us-east-1.amazonaws.com/ Workshopt material]. Before the meeting (6/20/2019) there are 13 downloads and it bumps to 32 downloads after the meeting for [https://hub.docker.com/r/seandavi/bioc_2019 the docker image].
** Download a tarball containing R packages (binary). 605 packages are included. It bundles 605 packages based on R 3.6.0 and Bioconductor 3.10 (BiocManager 1.30.4).
** Run RStudio container. All R packages downloaded in the last step are mounted. (user=rstudio, ps=bioc). That is, built-in is at '''/usr/local/lib/R/library''', custom is at '''/usr/local/lib/R/site-library'''. These two locations are what '''.libPaths()''' gives.
* [http://bioc2020.bioconductor.org/workshops BioC 2020]
** Workshop packages were created using the [https://github.com/seandavi/BuildABiocWorkshop2020?s=09 BuildABiocWorkshop2020] template. From the 'Dockerfile', we see each workshop material is organized as an R package. So each workshop's package is built in the Docker image. There is no need to build the vignette again. NOTE: it takes a while to build the Docker image locally since it requires to compile each R package separately.
** [https://docs.google.com/presentation/d/1xY4-8rTVAv9Z6fVXIv1fAIlZGtvGPN6G5e-bKKPLEx0/ BioC 2020: Where Software and Biology Connect] Opening Remarks
** Take the [https://github.com/LieberInstitute/recountWorkshop2020 recount2 workshop] for example, we don't need to knit the Rmd file. To view HTML vignette, we type '''browseVignettes(package="recountWorkshop2020") ''' and click the link "HTML". In case of the requested page was not found error, add '''help/''' to the URL right after the hostname, e.g., http://localhost:8787/help/library/recountWorkshop2020/doc/recount-workshop.html. Another way to open the HTML without any tweak is type "help(package = 'recountWorkshop2020')" -> User Guide -> HTML.
** [https://biocasia2020.bioconductor.org/workshops Bioc Asia 2020]
** The vignette may not include the R code. So the Rmd file is still needed to understand the content or do a practice.
 
== single-cell RNA-Seq ==
<ul>
<li>[http://bioinformatics.sph.harvard.edu/knowledgebase/scrnaseq/rstudio_sc_docker.html Docker image with rstudio for single cell analysis] ([https://github.com/hbc/knowledgebase/blob/master/scrnaseq/rstudio_sc_docker.md github]), https://hub.docker.com/r/vbarrerab/singlecell-base. Other images: https://github.com/rnakato/docker_singlecell, https://hub.docker.com/r/leanderd/single-cell-analysis
{{Pre}}
docker run -d -p 8787:8787 \
  --name scrna \
  -e USER='rstudio' \
  -e PASSWORD='rstudioSC' \
  -e ROOT=TRUE \
  -v /home/$USER/Documents/scrna:/home/rstudio/projects \
  vbarrerab/singlecell-base:R.4.0.3-BioC.3.11-ubuntu_20.04
</pre>
If I accidentally reboot the computer, installed packages were not lost. But it is safer to use ''docker stop XXX'' and then run ''docker start XXX''.
</li>
<li>Image containing rstudio + conda + a set of helpful packages for single cell analysis analysis, </li>
<li>[https://hub.docker.com/r/vbarrerab/singlecell-base/tags docker hub], </li>
<li>[https://github.com/vbarrera/docker_configuration/blob/master/docker_singlecell/r_4.0.3_bioc_3.11/Dockerfile Dockerfile] </li>
<li>To use with Portainer, it is better to use [https://www.composerize.com/ composerize] to convert the docker run command into a stack. Note Portainer cannot take version 3.x so I change version to 2. A stack/docker-compose.yml file generated by '''composerize''' with a modification on version number.
<pre>
version: '2'
services:
    singlecell-base:
        ports:
            - '8787:8787'
        container_name: scrna
        environment:
            - USER=rstudio
            - PASSWORD=rstudioSC
            - ROOT=TRUE
        volumes:
            - '/tmp/scrna:/home/rstudio/projects'
        image: 'vbarrerab/singlecell-base:R.4.0.3-BioC.3.11-ubuntu_20.04'
</pre>
 
Note: the right way to delete a stack is to stop the container, and then delete the container. The final step is to select the stack and remove it.
</li>
<li>
Note that I have 2 pythons installed. One is from the OS (/opt/conda/bin/python) whose version is 3.8.3. The other one is on (/home/rstudio/.conda/envs/sc_env/bin/python) whose version is 3.7.8. The '''$PATH''' variable will show differences.
 
</li>
<li>Another scRNA-Seq course including a docker image (not tried yet): [https://scrnaseq-course.cog.sanger.ac.uk/website/index.html Analysis of single cell RNA-seq data] (ebook, University of Cambridge Bioinformatics training unit) and the paper [https://www.nature.com/articles/s41596-020-00409-w Tutorial: guidelines for the computational analysis of single-cell RNA sequencing data] Andrews 2020. </li>
</ul>
 
== Cellar: interactive tool for analyzing single-cell omics data ==
[https://github.com/euxhenh/cellar/ Cellar] is an interactive tool for analyzing single-cell omics data. Cellar is built in Python using the Dash framework and relies on several open-source packages.
 
== Nanopore sequencing ==
[https://github.com/thachnguyen/duesselpore DUESSELPORE Webserver] and the [https://www.biorxiv.org/content/10.1101/2021.11.15.468670v1 paper]


= Docker for Shiny Server =
= Docker for Shiny Server =
Line 805: Line 355:
<li>[https://hosting.analythium.io/user-2022-best-practicesfor-shiny-apps-with-docker-and-more/ UseR!2022: Best Practices for Shiny Apps with Docker and More]
<li>[https://hosting.analythium.io/user-2022-best-practicesfor-shiny-apps-with-docker-and-more/ UseR!2022: Best Practices for Shiny Apps with Docker and More]
<li>[https://hosting.analythium.io/containerizing-interactive-r-markdown-documents/ Containerizing Interactive R Markdown Documents] 2022/7
<li>[https://hosting.analythium.io/containerizing-interactive-r-markdown-documents/ Containerizing Interactive R Markdown Documents] 2022/7
<li>By default, Shiny Server looks for either:
* A single-file Shiny app named '''app.R''' in the root of a directory.
* A pair of files named '''ui.R''' and '''server.R''' within a directory.
<li>[https://datawookie.dev/blog/2022/09/using-shiny-server-in-docker/ Using Shiny Server in Docker] 2022/9/7. Hello world example. Need to generate ''Dockerfile''.
<li>[https://datawookie.dev/blog/2022/09/using-shiny-server-in-docker/ Using Shiny Server in Docker] 2022/9/7. Hello world example. Need to generate ''Dockerfile''.
<pre>
<pre>
Line 938: Line 491:
* Best proxy - traefik, NPM, SWAG, caddy or other?
* Best proxy - traefik, NPM, SWAG, caddy or other?
* [https://github.com/linuxserver/docker-swag swag]
* [https://github.com/linuxserver/docker-swag swag]
* Caddy
** [https://github.com/rocker-org/rocker/issues?q=caddy Rocker], [https://www.rocker-project.org/use/networking/ Networking: tips and tricks] from Rocker
** [https://youtu.be/tv4A39yDDXs?t=526 Pi-Hosted : Reverse Proxy with Caddy]. We can play Caddy using IP address.


== Nginx proxy manager ==
== Nginx proxy manager ==
* [https://youtu.be/cjJVmAI1Do4 Putting it All Together - Docker, Docker-Compose, NGinx Proxy Manager, and Domain Routing - How To] (video) and [https://shownotes.opensourceisawesome.com/putting-it-all-together/ Notes] 2020-10-06
* [https://youtu.be/cjJVmAI1Do4 Putting it All Together - Docker, Docker-Compose, NGinx Proxy Manager, and Domain Routing - How To] (video) and [https://shownotes.opensourceisawesome.com/putting-it-all-together/ Notes] 2020-10-06
* [https://youtu.be/UfCkwlPIozw Securing NGinX Proxy Manager - follow up - securing your admin console for this Open Source Software] (video) and [https://shownotes.opensourceisawesome.com/securing-nginx-proxy-manager/ Notes] 2020-10-20
* [https://youtu.be/UfCkwlPIozw Securing NGinX Proxy Manager - follow up - securing your admin console for this Open Source Software] (video) and [https://shownotes.opensourceisawesome.com/securing-nginx-proxy-manager/ Notes] 2020-10-20
== Caddy ==
* [https://github.com/rocker-org/rocker/issues?q=caddy Rocker], [https://www.rocker-project.org/use/networking/ Networking: tips and tricks] from Rocker
* [https://youtu.be/tv4A39yDDXs?t=526 Pi-Hosted : Reverse Proxy with Caddy]. We can play Caddy using IP address.
* [https://linuxiac.com/how-to-set-up-caddy-as-reverse-proxy/ How to Set up Caddy as a Reverse Proxy]
= Website analysis =
* [https://www.howtoforge.com/how-to-install-plausible-analytics-with-docker-on-ubuntu-22-04/ How to Install Plausible Analytics with Docker on Ubuntu 22.04]


= Nginx =
= Nginx =
Line 1,355: Line 913:


https://github.com/solidnerd/docker-bookstack
https://github.com/solidnerd/docker-bookstack
= HDHomeRun =
== tvheadend ==
<ul>
<li>docker-compose.yml. See [https://docs.linuxserver.io/images/docker-tvheadend/ linuxserver/tvheadend]. Open http://localhost:9981 .
<pre>
services:
    tvheadend:
        container_name: tvheadend
        environment:
            - PUID=1000
            - PGID=1000
            - TZ=Etc/UTC
        network_mode: host
        volumes:
            - ./data:/config
            - ./recordings:/recordings
        restart: unless-stopped
        image: lscr.io/linuxserver/tvheadend:latest
</pre>
<li>tvheadend setup. [https://www.youtube.com/watch?v=6Dgjn1IKrFM TVheadend and HDHomeRun using IPTV inputs].
<li>[https://www.youtube.com/watch?v=2Y-E4sQSb94 How to configure tvheadend server].
<li>[https://www.youtube.com/watch?v=Y4gnJRgMB9s How to install Tvheadend server on $40 Android box running Linux]. The configuration allows to scan channels on HDHomeRun.
</ul>


= Media server =
= Media server =
Line 1,366: Line 948:
* [https://jellyfin.org/downloads/clients/ Official clients]
* [https://jellyfin.org/downloads/clients/ Official clients]
* [https://youtu.be/aY732cSkopU ULTIMATE Jellyfin Guide - Hardware Acceleration, Codecs (PART 2) ]
* [https://youtu.be/aY732cSkopU ULTIMATE Jellyfin Guide - Hardware Acceleration, Codecs (PART 2) ]
* Problems:
** [https://www.reddit.com/r/jellyfin/comments/rf0taf/best_way_to_update_your_library_when_you_make/ Best way to update your library when you make changes to existing files?]. Answer: (manual) Settings - Dashboard - Libraries - Libraries - Scan all libraries, (automatic) scheduled tasks - scan media library (every 12 hour).
** My (downloaded) videos got different video titles.


== Youtube downloader ==
== Youtube downloader ==
Line 1,378: Line 963:
# http://localhost:8081
# http://localhost:8081
</pre>
</pre>
The following example value for "YTDL_OPTIONS" embeds English subtitles and chapter markers (for videos that have them), and also changes the permissions on the downloaded video and sets the file modification timestamp to the date of when it was downloaded
{{Pre}}
docker run -d -p 8081:8081 \
  -v /tmp/downloads:/downloads \
  -e YTDL_OPTIONS='{"writesubtitles":true,"subtitleslangs":["en","-live_chat"],"updatetime":false,"postprocessors":[{"key":"Exec","exec_cmd":"chmod 0664","when":"after_move"},{"key":"FFmpegEmbedSubtitle","already_have_subtitle":false},{"key":"FFmpegMetadata","add_chapters":true}]}' \
  --name metube \
  --user 1000:1001 alexta69/metube
</pre>
<li>Chapters and subtitles:
* Testing video: [https://www.youtube.com/watch?v=yG8sFwBISVU Macha Galette].
* In VLC, subtitle is available under "Subtitle" -> "Sub Track" -> "English". The subtitle can be extracted from the downloaded webm files using <syntaxhighlight lang='sh' inline>ffmpeg -i filename.webm -map 0:s:0? subs.srt</syntaxhighlight>. If a video's subtitle is auto-generated by Youtube, then there is no subtitle embedded in the downloaded webm files.
* In VLC, chapters is available under "Playback" -> "Chapter"
<li>[https://github.com/yt-dlp/yt-dlp yt-dlp]
<li>[https://github.com/Tzahi12345/YoutubeDL-Material YoutubeDL-Material]
<li>[https://github.com/Tzahi12345/YoutubeDL-Material YoutubeDL-Material]
</ul>
</ul>
Line 1,390: Line 988:


== Plex ==
== Plex ==
[https://github.com/plexinc/pms-docker Official Docker container for Plex Media Server]
* [https://github.com/plexinc/pms-docker Official Docker container for Plex Media Server]
* https://hub.docker.com/r/linuxserver/plex/
* Live tv subtitles/captions: Server -> Settings -> Languages -> Subtitle mode -> Always enabled -> Save Changes.
* [https://www.plex.tv/plex-pass/ Plex Pass] subscription
** [https://support.plex.tv/articles/225877347-live-tv-dvr/ Live tv DVR]
** [https://support.plex.tv/articles/115002178853-using-hardware-accelerated-streaming/ Using Hardware-Accelerated Streaming]
<pre>
---
services:
  plex:
    image: lscr.io/linuxserver/plex:latest
    container_name: plex
    network_mode: host
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=US/Eastern
      - VERSION=docker
      - PLEX_CLAIM= #optional
    volumes:
      - /tmp/plex/library:/config
      - /tmp/plex/tvseries:/tv
    devices:
      - /dev/dri:/dev/dri
    restart: unless-stopped
</pre>


My comment:
== Sonarr, Radarr, Prowlarr ==
* After starting, web browser cannot connect to it
* [https://www.youtube.com/watch?v=LD8-Qr3B2-o Make Automated Torrent Media Server with Emby, Sonarr, Radarr, Prowlarr, and qBittorrent on Windows!]
* [https://www.youtube.com/watch?v=zfYthZrDpVU How to install and configure Sonarr]


= DNS server =
= DNS server =
Line 1,403: Line 1,027:
* https://github.com/IAmStoxe/wirehole
* https://github.com/IAmStoxe/wirehole
* [https://youtu.be/DOJ39lyx6Js WireHole: WireGuard, Pi-Hole and Unbound in Docker] (video)
* [https://youtu.be/DOJ39lyx6Js WireHole: WireGuard, Pi-Hole and Unbound in Docker] (video)
= Speed test =
* https://myspeed.dev/, http://localhost:5216. I need to open the page in incognito mode when using SSH local forwarding.
* [https://www.youtube.com/watch?v=tBJmhgn3ZOM MySpeed: Speedtest Automation Made Simple!]


= Pi-hole =
= Pi-hole =
Line 1,505: Line 1,133:
* https://hub.docker.com/r/freshrss/freshrss/
* https://hub.docker.com/r/freshrss/freshrss/
* [https://hub.docker.com/r/linuxserver/freshrss/ FreshRSS]
* [https://hub.docker.com/r/linuxserver/freshrss/ FreshRSS]
= Read it later/Pocket =
[https://github.com/omnivore-app/omnivore Omnivore]


= Home Assistant =
= Home Assistant =
Line 1,546: Line 1,177:


= Draw.io for diagrams and more =
= Draw.io for diagrams and more =
https://github.com/jgraph/drawio/tree/master/etc/docker
<ul>
<li>https://github.com/jgraph/drawio/tree/master/etc/docker
<pre>
<pre>
docker run -d --restart=unless-stopped --rm --name="draw" -p 8080:8080 -p 8443:8443 jgraph/drawio
docker run -d --restart=unless-stopped \
  --rm --name="draw" -p 8080:8080 \
  -p 8443:8443 jgraph/drawio
</pre>
</pre>
It seems the daemon parameter '''-d''' does not work.
<li>To allow to save to devices, the app and the file have to be on the same machine.
<li>It is super easy to install draw.io in Nextcloud (just a few clicks). It is very natural to save the file in the Nextcloud storage.
</ul>


= Internet Archive =
= Internet Archive =
Line 1,635: Line 1,271:
** Apps are using ports other than 80. Clicking on an app (eg Snapdrop was changed to use port 88 instead of 80) sometimes opened a tab with a port 80. So the launching is not successfully. We need to click on each app's Settings and find out what the port it uses and modify the port on the Web UI field and 'Save'.
** Apps are using ports other than 80. Clicking on an app (eg Snapdrop was changed to use port 88 instead of 80) sometimes opened a tab with a port 80. So the launching is not successfully. We need to click on each app's Settings and find out what the port it uses and modify the port on the Web UI field and 'Save'.
** Kasm not working.
** Kasm not working.
** Snapdrop. I can't see other devices??
** Snapdrop. I can't see other devices? [https://docs.linuxserver.io/images/docker-pairdrop/ pairdrop] from linuxserver.io works.
** OpenSpeedTest & librespeed/speedtest works but gave quite different results than speedtest.net
** OpenSpeedTest & librespeed/speedtest works but gave quite different results than speedtest.net
** [https://www.helpwire.app/blog/rdesktop-linux/ Rdesktop]. It is [https://youtu.be/jt_C3cStfAA?si=Mp-4UpV30Gn3lKVw&t=22 adding a desktop environment to your server]. So the IP should be IP:3389 when we use Remmina for example. The username/password is abc/abc. For example, I allocate 40GB to CasaOS and the remote desktop will show 40GB disk space for the server. The desktop includes the Chromium browser. We can connect to internet using the Chromium browser.
** [https://www.helpwire.app/blog/rdesktop-linux/ Rdesktop]. It is [https://youtu.be/jt_C3cStfAA?si=Mp-4UpV30Gn3lKVw&t=22 adding a desktop environment to your server]. So the IP should be IP:3389 when we use Remmina for example. The username/password is abc/abc. For example, I allocate 40GB to CasaOS and the remote desktop will show 40GB disk space for the server. The desktop includes the Chromium browser. We can connect to internet using the Chromium browser.

Revision as of 16:03, 3 September 2024

Homelab

Top/best

Templates

App store

Tipi

Dockerizing Compiled Software

Crontab

How to Use Cron With Your Docker Containers

Install macOS

Raspberry Pi servers based on Docker

Data Science

Data Science

Illumina BaseSpace

Bioinformatics analyses

Cloud provider

Docker can be deployed on any cloud provider like Digital Ocean, Amazon Web Services (AWS), Microsoft Azure, Google Cloud Computing.

Dashboard

Dashy

docker run -d \
  -p 8295:80 \
  --volume /home/$USER/dashy/public/conf.yml:/app/public/conf.yml \
  --volume /home/$USER/dashy/icons:/app/public/item-icons/icons \
  --name dashy \
  --restart=unless-stopped \
  lissy93/dashy:latest

Glance

Homepage

https://github.com/gethomepage/homepage

Flame

https://github.com/pawelmalak/flame

Homer

Uptime Kuma

heimdall

https://github.com/linuxserver/docker-heimdall

Dashboard for monitoring system: grafana

After I change the firewall by adding two ports (9090 & 9100), the docker-compose method of grafana works like a charm.

  • Reboot the host OS will not affect the grafana (assuming we include restart: unless-stopped in the 3 containers). But if we modify docker-compose.yml and run docker-compose down; docker-compose up it will result in data lose and we will need to re-create the admin password and all the setup.

Google photo

Immich: Google photo replacment

ownCloud

After I use docker run --rm --name myowncloud -p 81:80 owncloud:8.1, I find I can only access it through http://localhost:81. If I try to access it through another computer by http://xxx.xxx.xx.1:81, the URL is changed back to http://localhost:81 so it failed to load.

nextcloud

Nextcloud looks more modern than owncloud though their interface are very similar. I got the same problem (URL is changed from IP to localhost) when I try to access its web from another computer. Using the IP instead of localhost to finish the initial setup can fix the error.

Note:

  • The first we open the webpage, we need to create the admin username/password.
  • Nextcloud uses owncloud mobile application.
  • Nextcloud includes a small mp4 file. I can use browser to play mp4. Photos files can be opened in browsers too (including slideshow).

Add external storage like SAMBA

To connect to a Samba drive in Nextcloud, you will need to use the External Storage app. This app allows you to mount external storage services, including Samba shares, as directories within your Nextcloud instance. Here are the steps to connect to a Samba drive in Nextcloud:

First, make sure that the External Storage app is installed and enabled on your Nextcloud instance. You can do this by going to the Apps section in the Nextcloud admin panel and searching for the External Storage app. If it is not installed, you can install it from there.

Once the External Storage app is installed and enabled, go to the Settings section in the Nextcloud admin panel and select the External Storages option under the Administration section (Left panel, scroll down to see the app).

On the External Storages page, click on the Add storage button and select SMB / CIFS from the dropdown menu.

In the configuration form that appears, enter the details of your Samba share, including the hostname or IP address of the server, share name, username, and password. You can also specify a subfolder within the share if you only want to mount a specific subfolder.

Once you have entered all of the required information, click on the Save button to save your configuration.

Nextcloud Apps

  • "access through untrusted domain" error. On my Casaos, I found the file /DATA/AppData/nextcloud/var/www/html/config/config.php shows trusted domain is casaos.local:10081. So I just need to change the app's setting in CasaOS to use http://casaos.local:10081 instead of http://IP:10081 to open the page.
  • Audio Player. When I play mp3 files, the web interface does not show a progress bar:(
    $ docker run --rm --name mynextcloud -v ~/Downloads/nextcloudapps:/var/www/html/custom_apps -p 81:80 nextcloud
    # above will create an empty folder ~/Downloads/nextcloudapps (www-data:root)
    $ tar xzvf Downloads/audioplayer-2.3.1.tar.gz -C Downloads/
    $ docker cp Downloads/audioplayer mynextcloud:/var/www/html/custom_apps
    # Go to nextcloud and '+Apps', a new app 'Audio Player' should have shown up.
  • Music. GOOD. The app is from the owncloud team though for some reason the app is not included in owncloud/nextcloud. Yes, a progress bar is shown when I played a song downloaded from Youtube:)
    # No need to create a volume
    $ docker run -d --name mycloud -p 81:80 nextcloud
    
    # Open another terminal
    # Don't download the zip file from github. Get the zip file from the app store
    $ tar xzvf ~/Downloads/music_0.7.0_nc-signed.tar.gz -C ~/Downloads/
    $ docker cp Downloads/music mycloud:/var/www/html/custom_apps
    $ docker exec mycloud chmod -R 0750 /var/www/html/custom_apps
    $ docker exec mycloud chown -R www-data:www-data /var/www/html/custom_apps
    Now open the website. Click the gear icon (Settings) and click "Apps". Find out the 'music' app from the bottom of the list and click the 'Enable' button. A new Music icon will be created next to the Gallery icon at the top. Note: if we want to play music repeatedly, I should not play the music directly from Files. I should click the Music icon at the top and let it start to scan musics. Then once I can play a music from there, it will show two new icons; one is shuffle and the other is loop. To have a persistent data, we can 'run' using
    $ mkdir -p ~/nextcloud/{html,apps,config,data}
    $ docker run -d \
        --name mycloud \
        -p 81:80 \
        -v ~/nextcloud/html:/var/www/html \
        -v ~/nextcloud/apps:/var/www/html/custom_apps \
        -v ~/nextcloud/config:/var/www/html/config \
        -v ~/nextcloud/data:/var/www/html/data \
        nextcloud
    If we like to delete the contain and its unamed volume, run
    $ docker stop mycloud; docker rm -v mycloud
    If we run the container again mounting all volumes, all apps,files,configuration,admin's username & password are kept.
  • PDF viewer. PDF files will be downloaded instead of opening on the browser. This can be easily fixed. Click '+Apps' from the personal settings (the response is slow when I tested using Docker). Search for PDF viewer. Click the 'Enable' button (it will be changed to 'Disable' after that)
  • draw.io. After enable it, we can click the "+" sign and choose "Diagram" to create a new diagram (*.xml).
  • Radio. After enable it, a new "Radio" icon will be shown at the top. Only 20 stations are available from the 'Top'. No response after I clicked "Categories".

Music: Navidrome

Navidrome Is the Perfect Self-Hosted Music Server for Raspberry Pi

FileRun

File manager

File browser

The original image (filebrowser/filebrowser) does not work but linuxserver's image (filebrowser/filebrowser:s6) works fine.

$ touch filebrowser.db
$ cat settings.json
{
  "port": 80,
  "baseURL": "",
  "address": "",
  "log": "stdout",
  "database": "/database/filebrowser.db",
  "root": "/srv"
}

#  Modify $FULL_PATH1 <- directory to share files
#      &  $FULL_PATH2  <- directory contains filebrowser.db & settings.json

$ docker run \
    --name filebrowser \
    -v $FULL_PATH1:/srv \
    -v $FULL_PATH2/filebrowser.db:/database/filebrowser.db \
    -v $FULL_PATH2/settings.json:/config/settings.json \
    -e PUID=$(id -u) \
    -e PGID=$(id -g) \
    -p 8080:80 \
    filebrowser/filebrowser:s6

FilGator

https://filegator.io/

Seafile

Run a Linux desktop

Run a Linux desktop VM with VNC

Run Ubuntu Linux in Docker with Desktop Environment and VNC

Distrobox

Guacamole

Webtop: Linux desktop in a browser

mkdir /tmp/downloads/
docker run -d --name=webtop   \
  -e PUID=1000 -e PGID=1000   \
  -e TZ=US/Eastern   \
  -p 3000:3000   \
  -v /tmp/downloads:/config   \
  --shm-size="1gb"   \
  --restart unless-stopped   \
  linuxserver/webtop:ubuntu-xfce
  • Meet Webtops A Linux Desktop Environment In Your Browser
  • Run Linux Desktops in your Browser with Docker and Webtops! (video) and Notes
  • Default username/password is abc/abc. But we don't need to enter the password when we use "sudo".
  • Linux desktop, inside of a container, inside of a browser??? Yes. A Webtop, privileged: true is required for installing something like vscode in VM (video) and Doc
  • If I'm not using the default desktop, I need to switch to use some user's account instead of root to run docker-compose command. If I use root to launch ubuntu-kde, for example, the desktop is blank.
  • It's cool. I got Ubuntu 22.04 released 1.5 months ago.
  • After I installed 'htop', the htop command shows the cpu & memory & uptime ... from the 'host' system. But docker stats shows webtop (ubuntu-mate) desktop is using around 300MB memory.
  • When I want to copy text from VM to remote client, I need to open the clipboard using the black circle tool on the left-hand side.
  • After I played around, KDE-konsole does not respond to my keyboard. Logged out and logged in again fixes the problem.
  • I got the following errors when I try to install RStudio 2022.02.3 Build 492 on Ubuntu 22.04; this solution Installing R Studio on Ubuntu 22.04 works for me.
$ sudo apt install /config/Downloads/rstudio-2022.02.3-492-amd64.deb 
the following packages have unmet dependencies:
 rstudio : Depends: libssl1.0.0 but it is not installable or
                    libssl1.0.2 but it is not installable or
                    libssl1.1 but it is not installable

instantbox

Firewall/router/Vlan

IDE

cloud9

https://hub.docker.com/r/linuxserver/cloud9

VS Code

Docker for R (r-base, rstudio)

See R Docker.

Docker for Shiny Server

  • Pin package versions in your production Docker image. Need to modify Dockerfile to install 'shiny' along with 'renv'. We need to create a directory 'shinyapp' with simple shiny files (eg kmeans example).
    docker build -t pinrpackageversion .
    mkdir shinyapp
    wget https://raw.githubusercontent.com/rstudio/shiny-examples/master/050-kmeans-example/ui.R -P shinyapp
    wget https://raw.githubusercontent.com/rstudio/shiny-examples/master/050-kmeans-example/server.R -P shinyapp
    docker run --name shinyapp --rm -p 3838:3838 pinrpackageversion
    
  • Question: what happened if there is a new version of R or shiny-sever?
    # specify --name so we don't need to use container id to access it later
    $ docker run -d --name rshiny -p 81:3838 rocker/shiny 
    
    $ docker ps -l
    
    # Run in container's bash
    # https://stackoverflow.com/questions/30172605/how-to-get-into-a-docker-container
    # non-interactive
    $ docker exec -it rshiny ip addr  
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
    22: eth0@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
        link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
        inet 172.17.0.2/16 scope global eth0
           valid_lft forever preferred_lft forever
    
    # interactive shell
    $ docker exec -it rshiny bash 
    root@5b78988d5619:/# who
    root@5b78988d5619:/# which R
    /usr/bin/R
    root@5b78988d5619:/# ls /home
    docker	shiny
    root@5b78988d5619:/# ls /srv/shiny-server
    01_hello  02_text  03_reactivity  04_mpg  05_sliders  06_tabsets  07_widgets  08_html  09_upload  10_download  11_timer  index.html  sample-apps
    root@5b78988d5619:/# R -e 'package.Description("glmnet")'
    root@5b78988d5619:/# exit
    
    $ docker stop rshiny
    $ docker rm rshiny   # remove the container

It is also useful to use the -v option in docker run so the local directory (eg /srv/shinyapps/) can be mapped to the docker volume (/srv/shiny-server).

# Put your shiny app in host's /srv/shinyapps/appdir,
$ docker run -d -p 81:3838 \
    -v /srv/shinyapps/:/srv/shiny-server/ \
    -v /srv/shinylog/:/var/log/shiny-server/ \
    rocker/shiny

And check running status

$ docker ps -a

R packages installation: packrat

Check opened ports

See What network ports are open: lsof or netstat.

$ netstat -ant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN     
...
tcp6       0      0 :::80                   :::*                    LISTEN     
tcp6       0      0 :::81                   :::*                    LISTEN     
tcp6       0      0 :::22                   :::*                    LISTEN     
tcp6       0      0 ::1:25                  :::*                    LISTEN

Blog Explorer

https://github.com/nz-stefan/blog-explorer, Winners of the 2nd Annual Shiny Contest

Raspberry Pi

The Raspberry Pi 4B as a shiny server

BioContainers

CMS, blog website

Ghost

Proxy

  • Best proxy - traefik, NPM, SWAG, caddy or other?
  • swag

Nginx proxy manager

Caddy

Website analysis

Nginx

httpd/apache

https://hub.docker.com/_/httpd/

Tomcat

https://hub.docker.com/_/tomcat/

ASP.NET

Forum

How to Install Discourse Forum with Docker on CentOS 8

Jenkins for continuous integration

MySQL

  • https://hub.docker.com/r/mysql/mysql-server/
  • Deploying MySQL on Linux with Docker
  • Note it seems impossible to use the 'grant' command interactively within the MySQL docker container.
  • https://dzone.com/articles/using-docker-to-create-a-mysql-server
    docker run --detach --name test-mysql --env="MYSQL_ROOT_PASSWORD=mypassword" mysql
    docker inspect test-mysql | grep IPAddress  # 172.17.0.2
    
    # https://dev.mysql.com/doc/index-other.html
    wget -O - http://downloads.mysql.com/docs/world.sql.gz | gunzip -c > world.sql
    
    mysql -uroot -pmypassword -h 172.17.0.2 -e 'CREATE DATABASE world'
    cat world.sql | mysql -uroot -pmypassword -h 172.17.0.2 world
    # zcat world.sql.gz | mysql -uroot -pmypassword -h 172.17.0.2 world
    mysql -uroot -pmypassword -h 172.17.0.2 -e 'USE world; SHOW TABLES'
    
    mysql -uroot -pmypassword -h 172.17.0.2 -e 'DROP DATABASE world'
    # Let's see if we can access the database in R
    system("sudo apt install libmysqlclient-dev")
    library(RMySQL)
    con <- dbConnect(MySQL(), "world", user="root", password="mypassword", host="172.17.0.2")
    dbListTables(con)
    head(dbReadTable(con, "city"))
    head(dbReadTable(con, "country"))
    head(dbReadTable(con, "countrylanguage"))
    
    dbWriteTable(con, 'iris', iris)
    # Error in .local(conn, statement, ...) :
    #   could not run statement: The used command is not allowed with this MySQL version
    dbDisconnect(con)
  • How To Create And Test MySQL Database Server in Docker On Ubuntu Linux (video). Use Dockerfile.
  • MySQL Docker Containers: Understanding the basics, https://youtu.be/qJQPckRYNis?t=16m29s
    # Fire up
    docker run --detach --name=test-mysql --env="MYSQL_ROOT_PASSWORD=mypassword" mysql
    docker ps -a
    
    # Connecting to the Container
    docker inspect test-mysql | grep IPAddress
    # sudo apt install mysql-client
    # There are two methods to connect to the container
    # method 1. https://hub.docker.com/r/mysql/mysql-server/
    # docker exec -it test-mysql mysql -uroot -p
    # method 2
    mysql -uroot -pmypassword -h 172.17.0.20 -P 3306
    docker run --detach --name test-wordpress --link test-mysql:mysql wordpress
    docker exec -it test-wordpress bash
    $ cat /etc/hosts
    
    # Expose the MySQL container to the outside world
    # Access the MySQL container directly from the machine’s port 6603
    docker run --detach --name=test-mysql --env="MYSQL_ROOT_PASSWORD=mypassword" --publish 6603:3306 mysql
    mysql -u root -pmypassword -h 127.0.0.1 -P 6603
    
    # Configuration management
    mkdir -p /root/container/test-mysql/conf.d
    nano /root/container/test-mysql/conf.d/my-custom.cnf
    docker run \
        --detach \
        --name=test-mysql \
        --env="MYSQL_ROOT_PASSWORD=mypassword" \
        --publish 6603:3306 \
        --volume=/root/docker/test-mysql/conf.d:/etc/mysql/conf.d \
        mysql
    mysql -uroot -pmypassword -h127.0.0.1 -P6603 -e 'show global variables like "max_connections"';
    docker run \
        --detach \
        --name=test-mysql \
        --env="MYSQL_ROOT_PASSWORD=mypassword" \
        --publish 6603:3306 \
        mysql \
        --max-connections=200 \
        --character-set-server=utf8mb4 \
        --collation-server=utf8mb4_unicode_ci
    
    # Data Storage
    mkdir -p /storage/docker/mysql-datadir
    docker run \
        --detach \
        --name=test-mysql \
        --env="MYSQL_ROOT_PASSWORD=mypassword" \
        --publish 6603:3306 \
        --volume=/root/docker/test-mysql/conf.d:/etc/mysql/conf.d \
        --volume=/storage/docker/mysql-datadir:/var/lib/mysql \
        mysql
    docker inspect test-mysql
    ls -al /storage/docker/mysql-datadir/
    
    # If a MySQL container runs on top of an existing MySQL datadir, 
    # the $MYSQL_ROOT_PASSWORD variable should be omitted from the “run” command line;
    docker stop test-mysql
    docker rm -f test-mysql
    docker run -d --name=another-new-mysql -p 6605:3306 -v /storage/docker/mysql-datadir:/var/lib/mysql mysql
    docker logs another-new-mysql
  • Use Docker to run a typical MySQL server from Code Review Videos
    docker volume create crv_mysql
    
    docker run \
        -e MYSQL_ROOT_PASSWORD=my-secret-pw \
        -e MYSQL_DATABASE=devdb \
        -e MYSQL_USER=dbuser \
        -e MYSQL_PASSWORD=dbpassword \
        --mount type=volume,src=crv_mysql,dst=/var/lib/mysql \
        -p 3306:3306 \
        -d \
        mysql:latest
    
    # Alternatively we can put username/password in a text file for security
    cat > .env <<EOF
    MYSQL_ROOT_PASSWORD=my-secret-pw
    MYSQL_DATABASE=devdb
    MYSQL_USER=dbuser
    MYSQL_PASSWORD=dbpassword
    EOF
    docker run \
        --env-file .env \
        --mount type=volume,src=crv_mysql,dst=/var/lib/mysql \
        -p 3306:3306 \
        -d \
        mysql:latest

Backup and restore

# Backup
docker exec CONTAINER /usr/bin/mysqldump -u root --password=root DATABASE > backup.sql

# Restore
# Method 1. docker exec + command
docker exec -i CONTAINER /usr/bin/mysql -u root --password=root < backup.sql
# Method 2. docker exec + bash
# We can also use docker exec log into the contain and run any mysql commands there
docker exec -it CONTAINER bash
# Method 3. mysql command only
# We directly use mysql command to import database without using docker commands
#   as long as we know the IP address of the container server
docker inspect CONTAINER | grep IPAddress
mysql -uroot -pPASSWORD -h 172.18.0.3 -e 'show databases'
mysql -uroot -pPASSWORD -h 172.18.0.3 < backup.sql

PostgreSQL

Jekyll

Building Jekyll Sites within VS Code Dev Containers

node.js

Dockerizing Node.js. See the code here.


$ docker build -t my-node-app .
$ docker run -p 3000:3000 my-node-app

How To Setup Your Local Node.js Development Environment Using Docker (not working)

$ cd notes-service
$ cat Dockerfile 
FROM node:12.18.1

WORKDIR /code

COPY package.json /code/package.json
COPY package-lock.json /code/package-lock.json

RUN npm install

COPY . /code

CMD [ "node", "server.js" ]

$ docker build -t notes-service .
$ docker run -it --rm -d -v mongodb:/data/db -v mongodb_config:/data/configdb -p 27017:27017 --network mongodb --name mongodb mongo
$ docker run -it --rm -d --network mongodb --name notes -p 8081:8081 -e SERVER_PORT=8081 -e DATABASE_CONNECTIONSTRING=mongodb://mongodb:27017/yoda_notes notes-service 
# this does not work though no errors; omit "-d" to see the error message

$ curl --request POST \
--url http://localhost:8081/services/m/notes \
  --header 'content-type: application/json' \
  --data '{
"name”: "this is a note",
"text": "this is a note that I wanted to take while I was working on writing a blog post.",
"owner": "peter"
}'

React

How to Dockerise A React App

Redis

https://hub.docker.com/_/redis/

Web pages for metacran

Exploring Redis with Docker

Redis running inside Docker container on NVIDIA Jetson Nano, Redis is ideal for IoT and Embedded devices for several reasons.

Laravel

Dockerizing Laravel with Nginx MySQL and Docker Compose on Ubuntu 18.04 LTS

phpMyAdmin

docker run --name mysqlphp -e MYSQL_ROOT_PASSWORD=0000 -d mysql
docker run --name myadmin -d --link mysqlphp:db -p 8080:80 phpmyadmin/phpmyadmin

Now go to your browser , tap http://localhost:8080 and login with root/0000.

ERROR mysqli_real_connect(): The server requested authentication method unknown to the client [caching_sha2_password]

The solution can be found here. After the change, I can log in phpMyAdmin.

docker exec -it mysqlphp bash

mysql -u root -p0000

ALTER USER root IDENTIFIED WITH mysql_native_password BY '0000';

exit
exit

MagicMirror

https://github.com/MichMich/MagicMirror/blob/master/README.md#docker

Mail/smtp server

Mediawiki

  • https://www.mediawiki.org/wiki/Docker
  • https://hub.docker.com/_/mediawiki/ or (better) https://docs.docker.com/samples/library/mediawiki/. The Dockerfile is based on jessie.
  • bitnami/mediawiki (Verified but too complicated, 1M+)
  • https://hub.docker.com/r/ubcctlt/mediawiki (10M+)
  • Hosting Your Own Wiki With MediaWiki and Docker Nov 2020. The instruction is easy to follow.
    • In mediawiki set up we page, we need to enter 'database' for the Database host
    • The database information we entered will go to 'LocalSettings.php' file. But the name of wiki, administrator user name, password will go to the database.
    • the instruction for the 'extensions' part seems too much. We just need to mount local 'extensions' to container's '/var/www/extensions' folder.
    • Remember to run 'docker-compose down && docker-compose up -d' after we modify something
  • docker搭建mediawiki
  • https://github.com/kristophjunge/docker-mediawiki
  • Setup MediaWiki and Postgres with Docker (based on simplyintricate/mediawiki, works on Mint). Visit http://localhost:8080 to see the result.
    $ ls -l ~/wiki_root
    total 24
    drwxr-xr-x 3 root     root     4096 Jul 10 10:09 data
    drwxrwxr-x 3 brb      brb      4096 Jul 10 09:23 db
    drwxrwxr-x 2 www-data www-data 4096 Jul 10 09:23 extensions
    drwxrwxr-x 2 www-data www-data 4096 Jul 10 09:23 images
    -rw-rw-r-- 1 www-data www-data 4152 Jul 10 11:12 LocalSettings.php
    
  • Running Mediawiki as Docker container (based on synctree/mediawiki which is not updated anymore, works tested on GalliumOS)
    • Follow Run MySql inside docker container first. 1. Setup host to store configuration, data and dumps. The 'dumps' and 'initdb.d' folders are optional 2. Override default mysql configuration 3. Executing scripts and sql on container startup (optional) 4. Starting mysql container
    • Visit http://IP_address:8080 to see the mediawiki page running on Docker
    brb@galliumos:~$ tree mysql
    mysql
    ├── conf.d
    │   └── mysql.cnf
    ├── data
    │   ├── auto.cnf
    │   ├── ca-key.pem
    │   ├── ca.pem
    │   ├── client-cert.pem
    │   ├── client-key.pem
    │   ├── ib_buffer_pool
    │   ├── ibdata1
    │   ├── ib_logfile0
    │   ├── ib_logfile1
    │   ├── ibtmp1
    │   ├── mediawiki [error opening dir]
    │   ├── mysql [error opening dir]
    │   ├── performance_schema [error opening dir]
    │   ├── private_key.pem
    │   ├── public_key.pem
    │   ├── server-cert.pem
    │   ├── server-key.pem
    │   ├── sys [error opening dir]
    │   └── wikidb [error opening dir]
    ├── dumps
    └── initdb.d
    
    9 directories, 15 files
    
    brb@galliumos:~$ tree -L 2 mediawiki/
    mediawiki/
    ├── config
    │   └── LocalSettings.php
    ├── extensions
    │   ├── Cite
    │   ├── ConfirmEdit
    │   ├── Gadgets
    │   ├── ImageMap
    │   ├── InputBox
    │   ├── Interwiki
    │   ├── LocalisationUpdate
    │   ├── Nuke
    │   ├── ParserFunctions
    │   ├── PdfHandler
    │   ├── Poem
    │   ├── README
    │   ├── Renameuser
    │   ├── SpamBlacklist
    │   ├── SyntaxHighlight_GeSHi
    │   ├── TitleBlacklist
    │   └── WikiEditor
    └── images
        └── README
    
    19 directories, 3 files
    
  • https://linuxconfig.org/mediawiki-easy-deployment-with-docker-container (based on linuxconfig/mediawiki)
  • http://collabnix.com/how-to-setup-mediawiki-in-seconds-using-docker/ (based on appcontainers/mediawiki)
  • Isolated MediaWiki Setup on the Raspberry Pi using Docker, Part 1

BookStack

BookStack is a simple, self-hosted, easy-to-use platform for organising and storing information.

https://github.com/solidnerd/docker-bookstack

HDHomeRun

tvheadend

Media server

Jellyfin

Youtube downloader

  • Download YouTube videos, host locally?
  • MeTube (simple but no thumbnails, deleted files seem not deleted from the disk/deleted files from disks are still on the list; no sync),
    mkdir /tmp/downloads # O.W. /tmp/downloads generated by root cannot be written
    docker run -d -p 8081:8081 \
      -v /tmp/downloads:/downloads \
      --user 1000:1001 alexta69/metube
    # http://localhost:8081
    

    The following example value for "YTDL_OPTIONS" embeds English subtitles and chapter markers (for videos that have them), and also changes the permissions on the downloaded video and sets the file modification timestamp to the date of when it was downloaded

    docker run -d -p 8081:8081 \
      -v /tmp/downloads:/downloads \
      -e YTDL_OPTIONS='{"writesubtitles":true,"subtitleslangs":["en","-live_chat"],"updatetime":false,"postprocessors":[{"key":"Exec","exec_cmd":"chmod 0664","when":"after_move"},{"key":"FFmpegEmbedSubtitle","already_have_subtitle":false},{"key":"FFmpegMetadata","add_chapters":true}]}' \
      --name metube \
      --user 1000:1001 alexta69/metube
    
  • Chapters and subtitles:
    • Testing video: Macha Galette.
    • In VLC, subtitle is available under "Subtitle" -> "Sub Track" -> "English". The subtitle can be extracted from the downloaded webm files using ffmpeg -i filename.webm -map 0:s:0? subs.srt. If a video's subtitle is auto-generated by Youtube, then there is no subtitle embedded in the downloaded webm files.
    • In VLC, chapters is available under "Playback" -> "Chapter"
  • yt-dlp
  • YoutubeDL-Material

Airsonic media server

https://airsonic.github.io/download/

My comment:

  • Added mp3/mp4 files are not recognized.
  • Cannot play by web browsers
  • Special clients are needed:(

Plex

---
services:
  plex:
    image: lscr.io/linuxserver/plex:latest
    container_name: plex
    network_mode: host
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=US/Eastern
      - VERSION=docker
      - PLEX_CLAIM= #optional
    volumes:
      - /tmp/plex/library:/config
      - /tmp/plex/tvseries:/tv
    devices:
      - /dev/dri:/dev/dri
    restart: unless-stopped

Sonarr, Radarr, Prowlarr

DNS server

Private DNS server container

How to Deploy a Dynamic DNS Server with Docker on Debian 10

WireHole

Speed test

Pi-hole

Authelia

Whoogle

https://github.com/benbusby/whoogle-search

Backup

Duplicati

Python

Python Jupyter including R

docker run --rm -p 8888:8888 \
      -e JUPYTER_ENABLE_LAB=yes \
      -v "$PWD":/home/jovyan/work \
      --name jupyter \
      jupyter/datascience-notebook:r-4.3.1

Follow the screen output to access the app by going to http://IP:8888/?token=XXXXXX. This will take care of the authentication problem.

If we use 'docker compose up -d' method below, we need to use 'docker logs jupyter' to get the URL which includes the token password.

version: '3.3'
services:
    datascience-notebook:
        ports:
            - '8888:8888'
        environment:
            - JUPYTER_ENABLE_LAB=yes
        volumes:
            - '"$PWD":/home/jovyan/work'
        container_name: jupyter
        restart: unless-stopped
        image: 'jupyter/datascience-notebook:r-4.3.1'

This includes python3, Julia 1.0.0 and R. However, the R version is pretty old if we don't specify the tag.

Also the method does not work on arm64. WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested.

Keyboard shortcuts:

  • ctrl+Enter to run.
  • alt+Enter to run & insert a cell below.
  • shift+Enter to run and shift to a cell below.
  • Esc: command mode
  • Enter: edit mode

Note by default the ipynb files are saved to /home/jovyan instead of /home/jovyan/work directory. So it's better to click the "work" icon on the LHS of the file panel in order to change the working directory to "work".

Calibre

RSS

Read it later/Pocket

Omnivore

Home Assistant

Password manager

Vaultwarden

Vaultwarden - Open Source, self hosted backend for the Bitwarden password manager (video)

KeeWeb: password manager compatible with KeePass

https://github.com/keeweb/keeweb

Padloc: cloud-based password manager

Flask

Dockerizing Flask Application Using Docker on Debian 10

ONLYOFFICE

Gitlab

NAS & Docker

http://www.ithome.com.tw/tech/96500

Some examples

  • GetLab專案管理工具
  • Jenkins開發版本管理工具

Memo/note

Draw.io for diagrams and more

  • https://github.com/jgraph/drawio/tree/master/etc/docker
    docker run -d --restart=unless-stopped \
       --rm --name="draw" -p 8080:8080 \
       -p 8443:8443 jgraph/drawio
    
  • To allow to save to devices, the app and the file have to be on the same machine.
  • It is super easy to install draw.io in Nextcloud (just a few clicks). It is very natural to save the file in the Nextcloud storage.

Internet Archive

How To Self-host Your Own Internet Archive With ArchiveBox In Linux

Run a benchmark program on ARM devices

Cheat.sh

https://github.com/chubin/cheat.sh#docker

Set up a Mastodon Instance on Linux

Bioinformatics tools (Tophat, BWA, et al)

https://github.com/dceoy/docker-bio

cbioportal

  • Import study does not work if we use the docker-compose method. It works if we use docker exec -it CONTAINER bash.
    [brb@localhost cbioportal-docker-compose]$ docker-compose run cbioportal metaImport.py -u http://cbioportal:8080 -s study/lgg_ucsf_2014/ -o
    Creating cbioportal-docker-compose_cbioportal_run ... done
    Starting validation...
    
    CRITICAL: -: No meta files found in study/lgg_ucsf_2014/. Please make sure the directory is the path to the folder containing the files.
    ERROR: -: Cancer type needs to be defined for a study. Verify that you have a study file and have defined the cancer type correctly.
    ERROR: -: No valid study file detected
    

    See also https://github.com/cBioPortal/cbioportal/issues/7972

  • The version number shown on http://localhost:8080 is v3.4.17 (see .env variable) but the https://www.cbioportal.org/ website shows v3.5.3 (same as the current release version on Github & Docker hub)
  • The 2nd field on the 1st row of data_cancer_type.txt contains the study name (blue color) shown on website
  • meta_study.txt contains the study name (black color) shown on website
  • study_es_0 (breast, contains mRNA expression) and study_es_3 can be imported successfully when I use docker-compose exec -it CONTAINER bash and follow Load sample cancer study.
  • lgg_ucsf_2014 study (brain, CNA) should use the correct path docker-compose run cbioportal metaImport.py -u http://cbioportal:8080 -s study/lgg_ucsf_2014/lgg_ucsf_2014/ -o
  • I have tested on CentOS 7 and Ubuntu 20.04 on KVM/QEMU w/ 2 cores, 2/4GB memory, 25GB storage. I can use host's terminal and browser for interaction (i.e. no VNC/Spice server needed).
  • Gene panels in the cBioPortal patient view 2020-01-07, and other blogs related to cbioportal from thehyve.

GIVE - Genomic Interactive Visualization Engine

https://github.com/Zhong-Lab-UCSD/Genomic-Interactive-Visualization-Engine/blob/master/tutorials/GIVE-Docker.md

GeneTEFlow: Analysing gene and transposable elements expression from RNA-Seq data

GeneTEFlow: A Nextflow-based pipeline for analysing gene and transposable elements expression from RNA-Seq data. Source code.

Elastic Search

Webpage monitor tools

CasaOS

  • https://casaos.io/, Wiki (Very limited)
  • https://blog.casaos.io/ Blog and new release announcement
  • Self-Deploying-Applications from Zimaboard.
  • I installed CasaOS in Proxmox using Debian 12 LXC. I allocated 40GB space and 4GB ram. The memory can be increased without a need to shut down the LXC.
  • Files app is similar to NextCloud but it can connect to cloud storage like Google Drive, Dropbox and OneDrive. It can also connect to other network storage something like smb://192.168.1.1
  • I guess we are supposed to use CasaOS in a full screen model. The backward button in the browser will possibly quit CasaOS.
  • Portainer, Uptime Kuma, Wireguard Easy are some of apps in the App Store.
  • CasaOS can expand their App Store.
    • Custom installer is another option; we can use Docker Compose or Docker CLI.
    • "Add Source". Store list. After I copy/paste CasaOS-LinuxServer-AppStore https://casaos-appstore.paodayag.dev/linuxserver.zip and click "Add +" button it immediately shows 226 apps. That is cool! It includes Firefox, Chromium, Opera. I installed FF but when I use the web interface of FF on my Android tablet, I could not scroll. FreshRSS not working on my tablets. Firefox and Chromium are hard to use on my tablet.
    • Apps are using ports other than 80. Clicking on an app (eg Snapdrop was changed to use port 88 instead of 80) sometimes opened a tab with a port 80. So the launching is not successfully. We need to click on each app's Settings and find out what the port it uses and modify the port on the Web UI field and 'Save'.
    • Kasm not working.
    • Snapdrop. I can't see other devices? pairdrop from linuxserver.io works.
    • OpenSpeedTest & librespeed/speedtest works but gave quite different results than speedtest.net
    • Rdesktop. It is adding a desktop environment to your server. So the IP should be IP:3389 when we use Remmina for example. The username/password is abc/abc. For example, I allocate 40GB to CasaOS and the remote desktop will show 40GB disk space for the server. The desktop includes the Chromium browser. We can connect to internet using the Chromium browser.
    • Webtop is based on KasmVNC. Webtops is a project where you can setup a variety of Linux desktops, with a variety of Desktop Environments in Docker, and access them through a web browser! The screen/desktop is synchronized across devices. So we can use our tablet device as a monitor and our mouse and keyboard are still connected to our desktop computer. Interesting! But no internet connection? The font size on the desktop changes with the connected devices. The default distro is Arch Linux.
    • AdGuard: when I installed it, the default port for dashboard is 80. I changed it to 3000. But after the setup, the port 3000 is not working. Check the setting from CasaOS, it shows 3001. So the change of port in the initial set up was not taken (I ssh into the server and use ss to check open ports) though admin account was saved. The port 3001 is actually working. It works when I tested on a Android tablet (use only AdGuard IP as the only DNS, not including 1.1.1.1 or else). See Adguard Home Installed on Docker video for AdGuard setup.
    • ddclient, Duckdns, Duplicati, healthchecks (A watchdog for your cron jobs).
  • CasaOS+Tailscale: Seamless Home Network Integration + Remote access to your data