Python: Difference between revisions
(16 intermediate revisions by the same user not shown) | |||
Line 28: | Line 28: | ||
== Install, setup == | == Install, setup == | ||
=== | |||
=== Alias === | |||
[https://www.freecodecamp.org/news/how-to-fix-python-installation-errors-on-mac/ How to Fix Common Python Installation Errors on macOS] | |||
<pre> | |||
nano ~/.bash_profile | |||
# or | |||
nano ~/.zshrc | |||
</pre> | |||
Add the following line | |||
<pre> | |||
alias python=python3 | |||
</pre> | |||
=== Ubuntu === | === Ubuntu === | ||
Line 53: | Line 62: | ||
* On my 2021 mac Ventura, the default python3 is at "/usr/bin". But when we try to run 'python3', it asked to install the command line developer tools. After the installation we can use python3. Still, there is no "~/Library/Python" directory. | * On my 2021 mac Ventura, the default python3 is at "/usr/bin". But when we try to run 'python3', it asked to install the command line developer tools. After the installation we can use python3. Still, there is no "~/Library/Python" directory. | ||
== | == Multiple python versions == | ||
[https://www.freecodecamp.org/news/manage-multiple-python-versions-and-virtual-environments-venv-pyenv-pyvenv-a29fb00c296f/ How to manage multiple Python versions and virtual environments] 2018. | |||
== | === conda/mamba === | ||
<pre> | |||
conda create --name myenv python=3.6.12 | |||
conda activate myenv | |||
= | mamba create --name myenv python=3.6.12 | ||
mamba activate myenv | |||
</pre> | |||
=== pyenv === | |||
* https://github.com/pyenv/pyenv | |||
* [https://github.com/posit-dev/positron/wiki Positron wiki] | |||
* [https://opensource.com/article/20/4/pyenv How to use pyenv to run multiple versions of Python on a Mac] | |||
* [https://ubuntushell.com/install-multiple-python-versions-on-ubuntu/ How to Install Multiple Python Versions on Ubuntu Using Pyenv] | |||
<pre> | |||
pyenv install 3.6.12 | |||
pyenv virtualenv 3.6.12 myenv | |||
pyenv activate myenv | |||
</pre> | |||
=== virtualenv === | |||
<pre> | |||
pip install virtualenv | |||
virtualenv -p python3.6 myenv | |||
source myenv/bin/activate | |||
</pre> | |||
=== venv (python 3.3+) === | |||
<ul> | |||
<li>For Python 3, '''venv''' is generally more commonly used because it is included in the Python standard library starting from Python 3.3, making it more convenient and straightforward to use. However, '''virtualenv''' is still popular, especially among developers who need more advanced features or compatibility with older Python versions. It offers more flexibility and can be used with both Python 2 and Python 3. | |||
<ul> | |||
<li>The primary purpose of '''venv''' is to create isolated environments for managing Python packages and dependencies, but not python itself. For instance, <syntaxhighlight lang='sh' inline>python3.10 -m venv myenv</syntaxhighlight>. | |||
<li>On the other hand, '''virtualenv''' can indeed be used to control the Python version for your virtual environments. For instance, <syntaxhighlight lang='sh' inline>virtualenv -p /usr/bin/python3.10 myenv</syntaxhighlight>. | |||
</ul> | |||
<li>[https://www.howtogeek.com/start-project-python-virtual-environments/ Don’t Make This Mistake When You Start Your Python Project] | |||
<li>[https://python-code.dev/articles/236456323 Understanding Python's Virtual Environment Landscape: venv vs. virtualenv, Wrapper Mania, and Dependency Control] | |||
[https:// | |||
= | <li>Here is another example | ||
<syntaxhighlight lang='bash'> | |||
~/github/PUREE$ python3 -m venv myenv | |||
The virtual environment was not created successfully because ensurepip is not | |||
available. On Debian/Ubuntu systems, you need to install the python3-venv | |||
package using the following command. | |||
apt install python3.10-venv | |||
... | |||
~/github/PUREE$ sudo apt install python3.10-venv | |||
~/github/PUREE$ python3 -m venv myenv | |||
~/github/PUREE$ source myenv/bin/activate | |||
(myenv) ~/github/PUREE$ which python | |||
/home/brb/github/PUREE/myenv/bin/python | |||
(myenv) ~/github/PUREE$ pip freeze > requirements.txt | |||
(myenv) ~/github/PUREE$ deactivate | |||
~/github/PUREE$ | |||
~/github/PUREE$ ls myenv/bin | |||
activate activate.fish f2py f2py3.10 pip pip3.10 python3 wheel | |||
activate.csh Activate.ps1 f2py3 normalizer pip3 python python3.10 | |||
</syntaxhighlight> | </syntaxhighlight> | ||
< | </li> | ||
</ul> | </ul> | ||
== | == Online compiler == | ||
* https:// | * [https://www.python.org/shell/ Python.org]. It seems this has most modules. | ||
* [https:// | * [https://www.tutorialspoint.com/online_python_compiler.php Tutorialspoint] | ||
* [https://onecompiler.com/python OneCompiler] | |||
=== | == IDE == | ||
* [https://www.jetbrains.com/pycharm/ PyCharm] | |||
** Pycharm was used by [https://youtu.be/rfscVS0vtbw Learn Python - Full Course for Beginners ] (freeCodeCamp.org) | |||
** [https://stackoverflow.com/questions/7681431/run-python-source-code-line-by-line Run a code line by line] by changing the keyboard shortcuts from Settings -> Keymap -> Other. | |||
** To run the current file, right click the tab and select Run XXX. (Frustrated) | |||
* [http://thonny.org/ Thonny] | |||
* [https://www.spyder-ide.org/ Spyder] | |||
* RStudio | |||
** Create a file (xxx.py) | |||
** Click the terminal tab. Type 'python' (or ipython3). | |||
** Use Ctrl/CMD + Alt + Enter to run your python code line by line or a chunk. | |||
=== Visual Studio Code === | |||
* [https://youtu.be/h1sAzPojKMg Get started with Jupyter Notebooks in less than 4 minutes] (video) | |||
* [https://code.visualstudio.com/docs/datascience/jupyter-notebooks Jupyter Notebooks in VS Code], [https://code.visualstudio.com/docs/languages/python Python in Visual Studio Code] | |||
* [https://ithome.com.tw/news/156795 VS Code Python擴充套件開始不預裝Jupyter擴充套件] | |||
The ipynb file can contain figures. | |||
[https://github.com/immunogenomics/harmony2019/tree/master/notebooks This (Harmony Manuscript)] has several notebook files where the code in ipynb files were written in R, not Python. | |||
I can use [https://stackoverflow.com/a/61707097 vsc] to open a ipynb file. | |||
Conversion | |||
* Rmd to ipynb | |||
** [https://github.com/mkearney/rmd2jupyter rmd2jupyter] package | |||
** [https://codes.correlaid.org/first%20steps/r/jupyter/2021/03/02/Convert-Rmd-files-to-Jupyter-Notebook.html How to convert Rmd to ipynb notebook]: '''Jupytext''' and '''notedown'''. | |||
** [https://vatlab.github.io/sos-docs/index.html Script of Scripts (SoS) ] | |||
* ipynb to Rmd | |||
** [https://rmarkdown.rstudio.com/docs/reference/convert_ipynb.html Convert ipynb to Rmd] | |||
=== nbdev === | |||
* [https://github.com/fastai/nbdev nbdev] | |||
* [https://towardsdatascience.com/jupyter-is-now-a-full-fledged-ide-c99218d33095 Jupyter is now a full-fledged IDE] Literate programming is now a reality through nbdev and the new visual debugger for Jupyter. | |||
=== | === Emacs === | ||
[https://stackoverflow.com/a/7053298 Emacs Shell mode: how to send region to shell?] | |||
=== | == JupyterLab == | ||
[https:// | * [https://stackoverflow.com/a/52392304 What is the difference between Jupyter Notebook and JupyterLab?] | ||
* Jupyter Notebook (classic) and JupyterLab are both web-based interactive computing environments for working with data and code, but they have some key differences in terms of their user interface, features, and capabilities. | |||
* JupyterLab is a more modern and powerful tool than Jupyter Notebook, and is recommended for users who want a more flexible and feature-rich interface for working with data and code. However, Jupyter Notebook remains a popular and widely used tool, particularly for working with Jupyter notebooks. | |||
* [https://towardsdatascience.com/7-reasons-why-you-should-use-jupyterlab-for-data-science-7c2a3db8755a 7 Reasons Why You Should Use Jupyterlab for Data Science] | |||
=== | === Some resources === | ||
* https:// | * [https://www.dataschool.io/cloud-services-for-jupyter-notebook/ Six easy ways to run your Jupyter Notebook in the cloud] | ||
* | * [http://www.win-vector.com/blog/2020/03/cross-methods-are-a-leak-variance-trade-off/ Cross-Methods are a Leak/Variance Trade-Off] | ||
* [https://opensource.com/article/20/11/daily-journal-jupyter Journal five minutes a day with Jupyter] | |||
* [https://www.dataquest.io/blog/jupyter-notebook-tutorial/ How to Use Jupyter Notebook in 2020: A Beginner’s Tutorial] | |||
[https:// | * [https://www.makeuseof.com/get-started-with-jupyter-notebook/ Get Started With Jupyter Notebook: A Tutorial] | ||
* [https://datalya.com/blog/python-data-science/jupyter-notebook-command-mode-keyboard-shortcuts Jupyter Notebook Command Mode Keyboard Shortcuts] | |||
** Enter: edit mode | |||
** Esc: command mode | |||
** Ctrl-Enter: run cell | |||
** '''Shift-Enter''': run current cell, and select cell below | |||
** '''Alt-Enter''': run cell, insert a cell below | |||
** Y: to code | |||
** M: to markdown | |||
** 1: to insert heading 1 | |||
** 2,3,4,5,6: to insert heading 2,3,4,5,6 | |||
=== Online tools rendering ipynb === | |||
* Github | |||
* NBViewer (nbviewer.jupyter.org) | |||
* Google Colaboratory (colab.research.google.com) | |||
* Binder (mybinder.org) | |||
* Kaggle Notebooks (kaggle.com) | |||
* Azure Notebooks (notebooks.azure.com) | |||
* Datalore (datalore.jetbrains.com) | |||
* Deepnote (deepnote.com) | |||
* CoCalc (cocalc.com) | |||
=== Different installation methods === | |||
< | <ul> | ||
<li>https://pypi.org/project/jupyterlab/ | |||
</ | <li>https://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html | ||
<li>[https://www.howtoforge.com/guide-to-install-jupyterlab-on-debian-12/ Guide to Install JupyterLab on Debian 12]. Hint: require node.js. '''Node.js''', is used for building and managing JupyterLab’s '''JavaScript''' dependencies. Many Jupyter extensions require having working '''npm''' (which comes with Node.js) and '''jlpm''' commands, which are required for downloading useful Jupyter extensions or other JavaScript dependencies. Node.js itself is built with '''GYP''', a cross-platform build tool written in Python, which is another reason why Python is needed. | |||
<li>[https://www.howtoforge.com/how-to-install-jupyterlab-on-rocky-linux-9/ How to Install JupyterLab on Rocky Linux 9] | |||
<syntaxhighlight lang='sh'> | |||
[https:// | mkdir -p ~/project; cd ~/project | ||
python3 -m venv myenv # '-m venv' means to run venv module as a script | |||
source myenv/bin/activate | |||
pip3 install jupyter | |||
which jupyter | |||
jupyter --version | |||
jupyter server --generate-config | |||
jupyter server password | |||
jupyter lab --generate-config | |||
jupyter lab --show-config | |||
= | sudo firewall-cmd --add-port=8888/tcp | ||
https://www. | jupyter lab --ip 192.168.5.120 | ||
# http://192.168.5.120:8888/ | |||
</syntaxhighlight> | |||
<Li>[https://www.linuxcapable.com/how-to-install-jupyter-notebook-on-ubuntu-linux/ How to Install Jupyter Notebook on Ubuntu 24.04, 22.04 or 20.04] | |||
</ul> | |||
== | === JupyterLab Desktop === | ||
* https://github.com/jupyterlab/jupyterlab-desktop | |||
* [https://blog.jupyter.org/introducing-the-new-jupyterlab-desktop-bca1982bdb23 Introducing the new JupyterLab Desktop!] 2/9/2023 | |||
=== | === pip/pip3 Jupyter === | ||
< | <ul> | ||
<li>https://jupyter.org/ | |||
</ | <pre> | ||
which python3 | |||
# /usr/bin/python3 | |||
pip3 install jupyterlab | |||
jupyter-lab | |||
< | # http://localhost:8888/lab | ||
# The current directory will be available on the file browser panel in JupyterLab. | |||
</ | </pre> | ||
On Mac, it shows the following when I run 'pip3 install jupyterlab' | |||
{{Pre}} | |||
Installing collected packages: pip | |||
WARNING: The scripts pip, pip3 and pip3.9 are installed in '/Users/XXX/Library/Python/3.9/bin' which is not on PATH. | |||
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. | |||
Successfully installed pip-23.0 | |||
WARNING: You are using pip version 21.2.4; however, version 23.0 is available. | |||
You should consider upgrading via the '/Library/Developer/CommandLineTools/usr/bin/python3 -m pip install --upgrade pip' command. | |||
</pre> | |||
That is, I need to use '''/Users/XXX/Library/Python/3.9/bin/jupyter-lab''' to launch '''jupyter-lab''' OR add the path to ".zshrc"; see [https://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html jupyterlab doc]. | |||
</li> | |||
</ul> | |||
=== | === conda + Jupyter === | ||
[https://stackoverflow.com/a/ | * [https://stackoverflow.com/a/58068850 How to use Jupyter notebooks in a conda environment?] Option 1 works for me. | ||
< | * https://github.com/dylkot/cNMF/#analyze_simulated_example_data.ipynb | ||
<pre> | |||
</ | conda install --yes jupyterlab && conda clean --yes --all | ||
</pre> | |||
== | === IPython shell === | ||
<ul> | |||
<li>[https://opensource.com/article/21/3/ipython-shell-jupyter-notebooks Why I love using the IPython shell and Jupyter notebooks] | |||
<pre> | |||
ipython # Shell | |||
jupyter notebook # auto open the browser | |||
</pre> | |||
</li> | |||
<li>[https://medium.com/analytics-vidhya/why-switch-to-jupyterlab-from-jupyter-notebook-c6d98362945b Why switch to JupyterLab from jupyter-notebook?] | |||
</li> | |||
</ul> | |||
=== | === Extract python code from Jupyter notebook === | ||
* https:// | * [https://stackoverflow.com/a/56832942 Get only the code out of Jupyter Notebook]. '''nbconvert''' or '''jq'''. Or File -> Download as -> Python (.py) — this should export all code cells as single .py file. | ||
* [https://win-vector.com/2022/04/30/separating-code-from-presentation-in-jupyter-notebooks/ Separating Code from Presentation in Jupyter Notebooks] | |||
* [https:// | |||
=== | === Execute Javascript in a Jupyter Notebook === | ||
[https://linuxtldr.com/run-javascript-jupyter-notebook/ How to Execute Javascript in a Jupyter Notebook on Linux] | |||
=== Google colab === | |||
* https://colab.research.google.com/ | |||
* An example [https://github.com/MariaNattestad/pca-on-genotypes Mini bioinformatics project: PCA on genotypes] | |||
=== R programming === | |||
[https://developers.refinitiv.com/en/article-catalog/article/setup-jupyter-notebook-r Setup Jupyter Notebook for R] or [https://dzone.com/articles/using-r-on-jupyternbspnotebook Using R on Jupyter Notebook] | |||
To use R in JupyterLab, you will first need to install the IRkernel package in your R environment using the following command: | |||
<pre> | |||
</ | install.packages('IRkernel') | ||
</pre> | |||
Once you have installed the IRkernel package, you can register it with JupyterLab using the following command in your R console: | |||
<pre> | <pre> | ||
IRkernel::installspec() | |||
</pre> | </pre> | ||
After you have registered the kernel, you can start a new Jupyter notebook or JupyterLab session and select the "R" kernel from the kernel dropdown menu. This will allow you to run R code in JupyterLab, including data analysis, visualization, and other tasks. | |||
=== Xeus-R: a future-proof Jupyter kernel for R === | |||
[https://blog.jupyter.org/meet-xeus-r-a-future-proof-jupyter-kernel-for-r-1adc5fdd09ab Meet Xeus-R: a future-proof Jupyter kernel for R] | |||
== | == Run Jupyter Notebooks on an Apple M1 Mac == | ||
* [https:// | * [https://blog.roboflow.com/how-to-run-jupyter-notebooks-on-a-mac-m1/ How to Run Jupyter Notebooks on an Apple M1 Mac] | ||
* [https:// | * [https://towardsdatascience.com/how-to-easily-set-up-python-on-any-m1-mac-5ea885b73fab How to Easily Set Up Python on Any M1 Mac] | ||
* [https://alexmanrique.com/blog/development/2021/03/05/installing-jupyter-in-macbook-air-m1.html Installing Jupyter in Macbook Air M1] | |||
==== | == Cheat sheet == | ||
https:// | * http://datasciencefree.com/python.pdf | ||
* https://s3.amazonaws.com/assets.datacamp.com/blog_assets/PythonForDataScience.pdf | |||
==== | == The Most Frequently Asked Questions About Python Programming == | ||
https://www.makeuseof.com/tag/python-programming-faq/ | |||
==== | == Running == | ||
=== Interactively === | |||
Use Ctrl+d to quit. | |||
=== | === How to run a python script file === | ||
<syntaxhighlight lang='bash'> | |||
< | python mypython.py | ||
</syntaxhighlight> | |||
=== Run python statements from a command line === | |||
Use '''-c''' (command) option. | |||
<syntaxhighlight lang='bash'> | <syntaxhighlight lang='bash'> | ||
python -c "import psutil" | |||
</syntaxhighlight> | |||
=== run python source code line by line === | |||
[https://stackoverflow.com/a/7681464 run python source code line by line] | |||
<syntaxhighlight lang='bash'> | |||
python -m pdb <script.py> | |||
</syntaxhighlight> | </syntaxhighlight> | ||
== Install a new module == | |||
* See an example of installing [[Anders2013#HTSeq|HTSeq]]. | |||
=== Module != Package === | |||
* A Python '''module''' is a single file containing Python code, while a '''package''' is a collection of modules organized in a specific way. | |||
* A module has a filename with the suffix '''.py''' added. | |||
=== PyPI/Python Package Index === | |||
* https://en.wikipedia.org/wiki/Python_Package_Index | |||
* [https://pypi.python.org/pypi The Python Package Index (PyPI)] is the definitive list of packages (or modules) | |||
* [https://opensource.com/article/20/3/pip-linux-mac-windows How to install pip to manage PyPI packages easily] | |||
* [https://opensource.com/article/19/9/pypi-guide Introducing the guide to 7 essential PyPI libraries and how to use them] | |||
[https:// | |||
* [https://opensource.com/article/20/ | |||
* [https:// | |||
=== | === pip === | ||
'''pip''', use PyPI as the default source for packages and their dependencies. | |||
As an example, [https://pypi.org/project/motioneye/ motionEye] can be installed by pip install or pip2 install; see its [https://github.com/ccrisan/motioneye/wiki/Installation wiki] and source code on Github. | |||
<syntaxhighlight lang='bash'> | <syntaxhighlight lang='bash'> | ||
sudo apt-get install python-pip | |||
pip --version | |||
pip install SomePackage | |||
# | pip show --files SomePackage | ||
pip install --upgrade SomePackage | |||
pip install --upgrade https://github.com/Lasagne/Lasagne/archive/master.zip | |||
pip install ‐‐upgrade pip # Upgrade itself | |||
pip uninstall SomePackage | |||
sudo apt install python3-pip | |||
pip3 --version | |||
</syntaxhighlight> | </syntaxhighlight> | ||
==== Upgrade packages ==== | |||
[https://itsfoss.com/upgrade-pip-packages/ How to Upgrade Python Packages with Pip] | |||
<li> | ==== requirements.txt ==== | ||
<li> | <ul> | ||
<li>[https://learnpython.com/blog/python-requirements-file/ The Python Requirements File and How to Create it] | |||
<pre> | |||
pip freeze > requirements.txt | |||
</pre> | |||
<li>An example | |||
<pre> | |||
tensorflow==2.3.1 | |||
uvicorn==0.12.2 | |||
fastapi==0.63.0 | |||
</pre> | |||
We can use '''pip freeze''' or '''pip list''' to verify available packages in an environment. | |||
</ul> | </ul> | ||
=== | ==== List installed packages and their versions, location/directory ==== | ||
<pre> | |||
pip3 list -v | |||
</pre> | |||
< | On my Ubuntu 20.04, the packages installed by '''pip3''' is located in ''~/.local/lib/python3.8/site-packages/''. It does not matter where I issued the ''pip3 install'' command. | ||
==== The danger of upgrading pip ==== | |||
</ | * [https://stackoverflow.com/questions/49836676/error-after-upgrading-pip-cannot-import-name-main Error after upgrading pip]: cannot import name 'main' | ||
* [https://askubuntu.com/a/783442 You should consider upgrading via the 'pip install --upgrade pip' command] | |||
- | |||
=== | ==== Don't use sudo + pip ==== | ||
https://askubuntu.com/questions/802544/is-sudo-pip-install-still-a-broken-practice | |||
=== | ==== "--user" option in pip ==== | ||
* It is not recommended to use sudo before calling pip on Linux (actually can we?). This is because using sudo can cause permissions issues and can potentially damage your system12. Instead, you can install packages locally using the '''--user''' flag. | |||
* [https://askubuntu.com/a/641194 Upgrade python packages with pip: use "sudo" or "--user"? ] | |||
* If you need to install packages system-wide, you can use virtual environments instead of sudo. Virtual environments allow you to create isolated Python environments that do not interfere with the system Python installation | |||
</ | * [https://github.com/pypa/pip/issues/4186 Permission denied: '/usr/local/lib/python2.7/dist-packages/pip'] | ||
<pre style="white-space: pre-wrap; /* CSS 3 */ white-space: -moz-pre-wrap; /* Mozilla, since 1999 */ white-space: -pre-wrap; /* Opera 4-6 */ white-space: -o-pre-wrap; /* Opera 7 */ word-wrap: break-word; /* IE 5.5+ */ " > | |||
$ pip install Pygments | |||
... | |||
OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/Pygments-2.2.0.dist-info' | |||
/usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning. | |||
InsecurePlatformWarning | |||
$ pip install --user Pygments | |||
Collecting Pygments | |||
Using cached Pygments-2.2.0-py2.py3-none-any.whl | |||
Installing collected packages: Pygments | |||
Successfully installed Pygments-2.2.0 | |||
</pre> | |||
=== | ==== pip -t option ==== | ||
We can force to install a package in user's directory (i.e. a package is already installed in the global directory /usr/lib/python3/dist-packages but some applications cannot find it). [https://stackoverflow.com/questions/17216689/pip-install-python-package-into-a-specific-directory-other-than-the-default-inst Pip install python package into a specific directory other than the default install location] | |||
<pre> | |||
pip3 install -t ~/.local/bin/python3.10/site-packages pytz | |||
</pre> | |||
==== virtualenv ==== | |||
Python “Virtual Environments” allows us to install a Python package in an isolated location, rather than installing it globally. | |||
python- | <ul> | ||
<li>[https://www.ostechnix.com/manage-python-packages-using-pip/ How To Manage Python Packages Using Pip]. | |||
First Create a new project folder and cd to the project folder in your terminal. | |||
<syntaxhighlight lang='bash'> | <syntaxhighlight lang='bash'> | ||
$ pip | # Python 2 | ||
$ sudo pip install virtualenv | |||
$ virtualenv <DIR_NAME> | |||
$ source <DIR_NAME>/bin/activate | |||
(<DIR_NAME>) ~$ which python | |||
.... | |||
$ deactivate | |||
# For Python 3, https://docs.python.org/3/tutorial/venv.html it is more common to use venv instead | |||
< | $ python3 -m venv <DIR_NAME> # DIR_NAME is also called an environment | ||
> | $ source <DIR_NAME>/bin/activate | ||
> | (<DIR_NAME>) ~$ which python | ||
.... | |||
$ deactivate | |||
</syntaxhighlight> | </syntaxhighlight> | ||
<li> | |||
[https://youtu.be/N5vscPTWKOk Python Tutorial: virtualenv and why you should use virtual environments]. '''pip freeze'''. | |||
<pre> | |||
pip list | |||
pip freeze --local > requirements.txt | |||
... | |||
pip install -r requirements.txt | |||
pip list | |||
</pre> | |||
</li> | |||
</ul> | |||
* [https://opensource.com/article/20/10/learn-python-ebook Learn Python by creating a video game] | |||
* [https://www.pythonforbeginners.com/basics/how-to-use-python-virtualenv How to use Python virtualenv] | |||
* [https://www.dabapps.com/blog/introduction-to-pip-and-virtualenv-python/ A non-magical introduction to Pip and Virtualenv for Python beginners] | |||
* Alternative to ''virtualenv'' we need to add "--user" to the '''pip''' command. See the installation guide of [https://lasagne.readthedocs.io/en/latest/user/installation.html#python-pip lasagne] or [https://stackoverflow.com/a/15912917 easy_install or pip as a limited user?] | |||
==== pipenv ==== | |||
* [https://www.makeuseof.com/pipenv-python-environment/ Why Use Pipenv to Create a Python Environment?] | |||
=== | * Pipenv is a Python virtualenv management tool that supports a multitude of systems and nicely bridges the gaps between pip, pyenv and virtualenv. It automatically creates and manages a virtualenv for your projects, as well as adds/removes packages from your Pipfile as you install/uninstall packages. It also generates the ever-important '''Pipfile.lock''', which is used to produce deterministic builds. | ||
https:// | * https://pipenv.pypa.io/en/latest/pipfile/ | ||
.. | |||
=== | ==== Poetry ==== | ||
* https://python-poetry.org/ | |||
* Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you. Poetry offers a '''poetry.lock''' lockfile to ensure repeatable installs, and can build your project for distribution. | |||
==== pipx (alternative to pip3) ==== | |||
<ul> | |||
* | <li>Pipx is a tool that helps you install and run end-user applications written in Python. It is similar to macOS’s brew, JavaScript’s npx, and Linux’s apt. | ||
* Pipx is focused on installing and managing Python packages that can be '''run from the command line directly''' as applications. | |||
* pipx is made specifically for application installation, as it adds isolation yet still makes the apps available in your shell: pipx creates an '''isolated environment''' for each application and its associated packages. | |||
<syntaxhighlight lang='bash'> | |||
python3 -m pip install --user pipx | |||
python3 -m pipx ensurepath | |||
# OR sudo apt install pipx | |||
# Or pip3 install pipx | |||
pipx install <package_name> # no sudo needed | |||
< | pipx list | ||
pipx uninstall <package_name> | |||
</syntaxhighlight> | </syntaxhighlight> | ||
<li>I need to find an alternative to '''pip''' utility because of a problem when I used pip command. '''Error: externally-managed-environment''' | |||
* [https://www.omgubuntu.co.uk/2023/04/pip-install-error-externally-managed-environment-fix 3 Ways to Solve Pip Install Error on Ubuntu 23.04] | |||
* See the example of installing [[Linux#Asciinema_.26_agg|Asciinema & agg]] | |||
* I can see "asciinema" was installed under ~/.local/bin directory that's because "pipx ensurepath" adds ~/.local/bin to ~/.bashrc file. | |||
<li>Official website [https://pypa.github.io/pipx/pipx] - Install and Run Python Applications in Isolated Environments, [https://github.com/pypa/pipx github] | |||
<li>pipx vs pip | |||
* pipx is made specifically for application installation and adds isolation yet still makes the apps available in your shell. '''pipx creates an isolated environment for each application''' and its associated packages. On the other hand, '''pip is a general-purpose package installer for both libraries and apps with no environment isolation'''. | |||
* If you want to install an application that has dependencies that conflict with other applications or libraries on your system, you can use pipx to create an isolated environment for that application and its dependencies. This way, '''you can avoid conflicts between different versions of the same package'''. | |||
* On my Ubuntu, pip installs packages to /usr/local/lib/python3.8/dist-packages/. | |||
<li>[https://www.ostechnix.com/pipx-install-and-run-python-applications-in-isolated-environments/ Pipx – Install And Run Python Applications In Isolated Environments] | |||
<li>[https://opensource.com/article/21/7/python-pipx Run Python applications in virtual environments] | |||
</ul> | |||
=== | === python setup.py === | ||
If a package has been bundled by its creator using the standard approach to | |||
bundling modules (with Python’s distutils tool), all you need to do is download | |||
the package, uncompress it and type: | |||
<syntaxhighlight lang='bash'> | |||
python setup.py build | |||
sudo python setup.py install | |||
</syntaxhighlight> | |||
For Python 2, the packages are installed under '''/usr/local/lib/python2.7/dist-packages/'''. | |||
<syntaxhighlight lang='bash'> | <syntaxhighlight lang='bash'> | ||
$ ls -l /usr/local/lib/python2.7/dist-packages/ | |||
total 12 | |||
-rw-r--r-- 1 root staff 273 Jan 12 13:45 easy-install.pth | |||
drwxr-sr-x 4 root staff 4096 Jan 12 13:45 HTSeq-0.6.1p1-py2.7-linux-x86_64.egg | |||
drwxr-sr-x 4 root staff 4096 Jan 12 13:42 pysam-0.9.1.4-py2.7-linux-x86_64.egg | |||
</syntaxhighlight> | |||
=== python setup.py bdist_wheel === | |||
* [https://stackoverflow.com/a/65480764 Why does python setup.py bdist_wheel creates a build folder?] | |||
* [https://realpython.com/python-wheels/ What Are Python Wheels and Why Should You Care?] The purpose of creating a wheel file in Python is to package and distribute your code in a way that makes it easier for others to install and use your code. | |||
* [https://docs.python.org/3/distutils/builtdist.html Creating Built Distributions] | |||
* [https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#id67 Wheels] | |||
* In Python programming, “wheel” refers to a built-package format for Python. It is designed to contain all the files for a PEP 376 compatible install in a way that is very close to the on-disk format. | |||
* [https://www.geeksforgeeks.org/what-is-a-python-wheel/# What is a Python wheel?] | |||
* To create a wheel file in Python. | |||
** First, make sure you have the wheel package installed: pip install wheel | |||
** Navigate to the directory containing your package and run the following command: python setup.py bdist_wheel | |||
** This will create a .whl file in the dist directory of your package. (people can use [https://stackoverflow.com/a/28300854 '''pip''' to install the project]) | |||
=== Get a list of installed modules === | |||
http://stackoverflow.com/questions/739993/how-can-i-get-a-list-of-locally-installed-python-modules | |||
====== | <syntaxhighlight lang='bash'> | ||
pydoc modules | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Not helpful. See the '''pip list''' command. | |||
== | === Check installed packages' versions === | ||
If you install packages through '''pip''', use | |||
< | <syntaxhighlight lang='bash'> | ||
$ pip list | |||
</ | ... | ||
pyOpenSSL (0.13.1) | |||
pyparsing (2.0.1) | |||
pysam (0.10.0) | |||
python-dateutil (1.5) | |||
pytz (2013.7) | |||
rudix (2016.12.13) | |||
scipy (0.13.0b1) | |||
setuptools (1.1.6) | |||
singledispatch (3.4.0.3) | |||
six (1.4.1) | |||
tornado (4.4.2) | |||
vboxapi (1.0) | |||
xattr (0.6.4) | |||
zope.interface (4.1.1) | |||
</syntaxhighlight> | |||
And more information about a package by using '''pip show PACKAGE'''. | |||
<syntaxhighlight lang='bash'> | |||
$ pip show pysam | |||
Name: pysam | |||
Version: 0.10.0 | |||
Summary: pysam | |||
Home-page: https://github.com/pysam-developers/pysam | |||
Author: Andreas Heger | |||
Author-email: [email protected] | |||
License: MIT | |||
Location: /Users/XXX/Library/Python/2.7/lib/python/site-packages | |||
Requires: | |||
</syntaxhighlight> | |||
= | The following method works whether the package is installed by source or binary package | ||
<syntaxhighlight lang='python'> | |||
>>> import pysam | |||
>>> print(pysam.__version__) | |||
0.10.0 | |||
>>> print pysam.__version__ | |||
0.10.0 | |||
</syntaxhighlight> | |||
See http://hammelllab.labsites.cshl.edu/tetoolkit-faq/ | |||
== | === Install a specific version of package through pip === | ||
https://stackoverflow.com/questions/5226311/installing-specific-package-versions-with-pip | |||
== | For example, pysam package was actively released. But the new release (0.11.2.2) may introduce some bugs. So I have to install an older version (0.10.0 works for me on Mac El Capitan and Sierra). | ||
<syntaxhighlight lang='bash'> | |||
$ sudo -H pip uninstall pysam | |||
Uninstalling pysam-0.11.2.2: | |||
...... | |||
$ sudo -H pip install pysam==0.10.0 | |||
Collecting pysam==0.10.0 | |||
Downloading pysam-0.10.0.tar.gz (2.3MB) | |||
100% |████████████████████████████████| 2.3MB 418kB/s | |||
Installing collected packages: pysam | |||
Running setup.py install for pysam ... done | |||
Successfully installed pysam-0.10.0 | |||
</syntaxhighlight> | |||
== | === warning: Please check the permissions and owner of that directory === | ||
I got this message when I use root to run the 'sudo pip install PACKAGE' command. | |||
See | |||
* http://stackoverflow.com/questions/27870003/pip-install-please-check-the-permissions-and-owner-of-that-directory | |||
* http://askubuntu.com/questions/578869/python-pip-permissions | |||
=== python3-pip installed but pip3 command not found? === | |||
<syntaxhighlight lang='bash'> | |||
sudo apt-get remove python3-pip; sudo apt-get install python3-pip | |||
<syntaxhighlight lang=' | |||
</syntaxhighlight> | </syntaxhighlight> | ||
=== | === DeepSurv example === | ||
https://github.com/jaredleekatzman/DeepSurv | |||
<syntaxhighlight lang='bash'> | |||
git clone https://github.com/jaredleekatzman/DeepSurv.git | |||
sudo cp /usr/bin/pip /usr/bin/pip.bak | |||
sudo nano /usr/bin/pip # See https://stackoverflow.com/a/50187211 more detail | |||
# Method 1 for Theano | |||
sudo pip install theano | |||
# Method 2 for Theano | |||
pip install --user --upgrade https://github.com/Theano/Theano/archive/master.zip | |||
pip install --user --upgrade https://github.com/Lasagne/Lasagne/archive/master.zip | |||
cd DeepSurv/ | |||
pip install . --user | |||
sudo apt install python-pytest | |||
pip install h5py --user | |||
sudo pip uninstall protobuf # https://stackoverflow.com/a/33623372 | |||
pip install protobuf --user | |||
sudo apt install python-tk | |||
py.test | |||
============ test session starts =========== | |||
platform linux2 -- Python 2.7.12, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 | |||
rootdir: /home/brb/github/DeepSurv, inifile: | |||
collected 7 items | |||
tests/test_deepsurv.py ....... | |||
== | ========== 7 passed in 5.77 seconds ======== | ||
</syntaxhighlight> | </syntaxhighlight> | ||
== | == How to list all installed modules == | ||
<pre> | <pre> | ||
help('modules') # the output is not pretty | |||
</pre> | </pre> | ||
== | == Comment == | ||
# Use the comment symbol # for a single line | |||
# Use a delimiter “”” on each end of the comment. '''Attention''': [https://stackoverflow.com/questions/675442/how-to-comment-out-a-block-of-code-in-python Don't use triple-quotes] | |||
[http://www.zentut.com/python-tutorial/python-comments/ Python Comments] from zentut.com. | |||
=== Docstring === | |||
* https://en.wikipedia.org/wiki/Docstring | |||
* Python Developer's Guide [https://www.python.org/dev/peps/pep-0257/ Docstring Conventions] | |||
== | == Try / Except == | ||
[https://pythonbasics.org/try-except/ Try and Except in Python] | |||
< | <pre> | ||
try: | |||
number = int(input("Enter a number: ")) | |||
print(number) | |||
except: | |||
print("Invalid Input") | |||
</pre> | |||
</ | |||
== if __name__ == "__main__": == | |||
* [https://stackoverflow.com/a/419185 What does if __name__ == “__main__”: do?] | |||
** [https://stackoverflow.com/a/419986 Simplest example] | |||
** [https://stackoverflow.com/a/419189 A more complicated example] | |||
* [https://www.freecodecamp.org/news/whats-in-a-python-s-name-506262fe61e8/ What’s in a (Python’s) __name__?] | |||
== How to Get the Current Directory in Python == | |||
[https://www.makeuseof.com/how-to-get-the-current-directory-in-python/ How to Get the Current Directory in Python] | |||
== Import a compiled C module == | |||
* An [http://www.swig.org/tutorial.html example] based on SWIG compiler. | |||
== string and string operators == | |||
Reference: | |||
# Python for Genomic Data Science from coursera. | |||
# [https://www.codementor.io/mgalarny/python-hello-world-and-string-manipulation-gdgwd8ymp Python Hello World and String Manipulation] | |||
* Use double quote instead of single quote to define a string | |||
* Use triple double quotes """ to write a [https://docs.python.org/3/tutorial/introduction.html long string spanning multiple lines] or [http://stackoverflow.com/questions/7696924/multiline-comments-in-python comments in a python script] | |||
* if dna="gatagc", then | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
dna[0]='g' | |||
dna[-1]='c' (start counting from the right) | |||
dna[-2]='g' | |||
dna[0:3]='gat' (the end always excluded) | |||
dna[:3]='gat' | |||
dna[2:]='tgc' | |||
len(dna)=6 | |||
type(dna) | |||
print(dna) | |||
dna.count('c') | |||
dna.upper() | |||
dna.find('ag')=3 (only the first occurrence of 'ag' is reported) | |||
dna.find('17', 2) (start looking from pos 17) | |||
dna.rfind('ag') ( search backwards in string) | |||
dna.islower() (True) | |||
dna.isupper() (False) | |||
dna.replace('a', 'A') | |||
print(dna.upper().isupper()) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
=== Format === | |||
[https://docs.python.org/3.8/library/string.html#format-specification-mini-language Format Specification Mini-Language] | |||
== Regular expression == | |||
[https://www.makeuseof.com/regular-expressions-python/ The Beginner’s Guide to Regular Expressions With Python] | |||
== User's input == | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
dna=raw_input("Enter a DNA sequence: ") # python 2 | |||
dna=input("Enter a DNA sequence: ") # python 3 | |||
</syntaxhighlight> | </syntaxhighlight> | ||
To convert a user's input (a string) to others | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
int(x, [, base]) | |||
flaot(x) | |||
str(x) #converts x to a string | |||
str(65) # '65' | |||
chr(x) # converts an integer to a character | |||
chr(65) # 'A' | |||
</syntaxhighlight> | </syntaxhighlight> | ||
== | == Print == | ||
[https://stackoverflow.com/a/6183002 Why is parenthesis in print voluntary in Python 2.7?] | |||
== Fancy Output == | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
print("THE DNA's GC content is ", gc, "%") # gives too many digits following the dot | |||
print("THE DNA's GC content is %5.3f %%" % " % gc) | |||
# the percent operator separating the formatting string and the value to | |||
# replace the format placeholder | |||
print("%d" % 10.6) # 10 | |||
print("%e" % 10.6) # 10.060000e+01 | |||
print("%s" % dna) # gatagc | |||
</syntaxhighlight> | </syntaxhighlight> | ||
== | == Type == | ||
A | [https://docs.python.org/3/library/functions.html Built-in Functions], [https://www.golinuxcloud.com/python-type-of-variable/#Conclusion How to check type of variable (object) in Python] | ||
<pre> | |||
type(object) | |||
</pre> | |||
== List == | |||
A list is an ordered set of values | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
gene_expr=['gene', 5.16e-08, 0.001385, 7.33e-08] | |||
print(gene_expr[2] | |||
gene_expr[0]='Lif' | |||
</syntaxhighlight> | </syntaxhighlight> | ||
= | Slice a list (it will create a new list) | ||
<syntaxhighlight lang='python'> | |||
gene_expr[-3:] # [5.16e-08, 0.001385, 7.33e-08] | |||
gene_expr[1:3] = [6.09e-07] | |||
</syntaxhighlight> | |||
Clear the list | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
gene_expr[]=[] | |||
</syntaxhighlight> | </syntaxhighlight> | ||
=== List functions === | |||
Size of the list | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
len(gene_expr) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Delete an element | |||
<syntaxhighlight lang='python'> | |||
del gene_expr[1] | |||
</syntaxhighlight> | |||
Extend/append to a list | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
gene_expr).extend([5.16e-08, 0.00123]) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Count the number of times an element appears in a list | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
print(gene_expr.count('Lif'), gene_expr.count('gene')) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Reverse all elements in a list | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
gene_expr.reverse() | |||
print(gene_expr) | |||
help(list) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Lists as Stacks | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
stack=['a', 'b', 'c', 'd'] | |||
stack.append('e') | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Sorting lists | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
mylist=[3, 31, 123, 1, 5] | |||
sorted(mylist) | |||
mylist # not changed | |||
mylist.sort() | |||
mylist=['c', 'g', 'T', 'a', 'A'] | |||
mylist.sort() | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Don't change an element in a string! | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
motif = 'nacggggtc' | |||
motif[0] = 'a' # ERROR | |||
</syntaxhighlight> | </syntaxhighlight> | ||
== Tuples == | |||
A tuple consists of a number of values separated by commas, and is another standard sequence data type, like strings and lists. | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
t=1,2,3 | |||
t | |||
t=(1,2,3) # we may input tuples with or without surrounding parentheses | |||
</syntaxhighlight> | </syntaxhighlight> | ||
== Sets == | |||
A set is an unordered collection with no duplicate elements. | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
brca1={'DNA repair', 'zine ion binding'} | |||
brca2={protein binding', 'H4 histone'} | |||
brca1 | brca2 | |||
brca1 & brca2 | |||
brca1 - brca2 | |||
</syntaxhighlight> | </syntaxhighlight> | ||
== Dictionaries == | |||
A '''dictionary''' is an unordered set of ''key'' and ''value'' pairs, with the requirement that the keys are unique (within on dictionary). | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
TF_motif = {'SP1' : 'gggcgg', | |||
'C/EBP' : 'attgcgcaat', | |||
'ATF' : 'tgacgtca', | |||
'c-Myc' : 'cacgtg', | |||
'Oct-1' : 'atgcaaat'} | |||
# Access | |||
print("The recognition sequence for the ATF transcription is %s." % TF_motif['ATF']) | |||
# Update | |||
TF_motif['AP-1'] = 'tgagtca' | |||
# Delete | |||
del TF_motif['SP1'] | |||
# Size of a list | |||
len(TF_motif) | |||
# Get a list of all the 'keys' in a dictionary | |||
list(TF_motif.keys()) | |||
# Get a list of all the 'values' | |||
list(TF_motif.values()) | |||
# sort | |||
sorted(TF_motif.keys()) | |||
sorted(TF_motif.values()) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
We can retrieve data from dictionaries using the '''items()''' method. | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
for name,seq in seqs.item(): | |||
print(name, seq) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
In summary, '''strings''', '''lists''' and '''dictionaries''' are most useful data types for bioinformatics. | |||
== '''if''' statement == | |||
< | <syntaxhighlight lang='python'> | ||
dna=input('Enter DNA sequence: ') | |||
. | if 'n' in dna : | ||
nbases=dna.count('n') | |||
print("dna sequence has %d undefined bases " % nbases) | |||
if condtion 1: | |||
do action 1 | |||
elif condition 2: | |||
do action 2 | |||
else: | |||
</ | do action 3 | ||
</syntaxhighlight> | |||
=== | == Logical operators == | ||
Use and, or, not. | |||
<syntaxhighlight lang='python'> | |||
dna=input('Enter DNA sequence: ') | |||
if 'n' in dna or 'N' in dna: | |||
nbases=dna.count('n')+dna.count('N') | |||
print("dna sequence has %d undefined bases " % nbases) | |||
else: | |||
print("dna sequence has no undefined bases) | |||
</syntaxhighlight> | |||
== Loops == | |||
while | |||
<syntaxhighlight lang='python'> | |||
dna=input('Enter DNA sequence:') | |||
pos=dna.find('gt', 0) | |||
while pos>-1 : | |||
print("Donar splice site candidate at position %d" %pos) | |||
pos=dna.find('gt', pos+1) | |||
</syntaxhighlight> | |||
for | |||
<syntaxhighlight lang='python'> | |||
motifs=["attccgt", "aggggggttttttcg", "gtagc"] | |||
for m in motifs: | |||
print(m, len(m)) | |||
</syntaxhighlight> | |||
> | |||
</ | |||
range | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
for i in range(4): | |||
print(i) | |||
print( | for i in range(1,10,2): | ||
print(i) | |||
print( | </syntaxhighlight> | ||
Problem: find all characters in a given protein sequence are valid amino acids. | |||
<syntaxhighlight lang='python'> | |||
protein='SDVIHRYKUUPAKSHGWYVCJRSRFTWMVWWRFRSCRA' | |||
for i in range(len(protein)): | |||
if protein[i] not in 'ABCDEFGHIKLMNPQRSTVWXYZ': | |||
print("this is not a valid protein sequence!") | |||
break | |||
</syntaxhighlight> | </syntaxhighlight> | ||
continue | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
protein='SDVIHRYKUUPAKSHGWYVCJRSRFTWMVWWRFRSCRA' | |||
corrected_protein='' | |||
for i in range(len(protein)): | |||
if protein[i] not in 'ABCDEFGHIKLMNPQRSTVWXYZ': | |||
continue | |||
corrected_protein=corrected_protein+protein[i] | |||
print("COrrected protein seq is %s" % corrected_protein) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
else Statement used with loops | |||
* If used with a for loop, the else statement is executed when the loop has exhausted iterating the list | |||
* If used with a while loop, the else statement is executed when the condition becomes false | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
# Find all prime numbers smaller than a given integer | |||
N=10 | |||
for y in range(2, N): | |||
for x in range(2, y): | |||
if y %x == 0: | |||
print(y, 'equals', x, '*', y//x) | |||
break | |||
else: | |||
// loop fell through without finding a factor | |||
print(y, 'is a prime number') | |||
</syntaxhighlight> | |||
The '''pass''' statement is a placeholder | |||
<syntaxhighlight lang='python'> | |||
if motif not in dna: | |||
pass | |||
else: | |||
some_function_here() | |||
</syntaxhighlight> | </syntaxhighlight> | ||
== Functions == | |||
[https://opensource.com/article/19/7/get-modular-python-functions Get modular with Python function] | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
def | def function_name(arguments) : | ||
return | function_code_block | ||
return output | |||
</syntaxhighlight> | </syntaxhighlight> | ||
For example, | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
def | def gc(dna) : | ||
" | "this function computes the gc perc of a dna seq" | ||
nbases=dna.count('n')+dna.count('n') | |||
gcpercent=float(dna.count('c')+dna.count('C')+dna.count('g) | |||
+dna.count('G'))*100.0/(len(dna)-nbases) | |||
return gcpercent | |||
gc('AAAAGTNNAGTCC') | |||
help(gc) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
=== | === SyntaxError: invalid syntax === | ||
https://stackoverflow.com/a/11890194 | |||
On the Python shell add an empty line at the end of function definition. Eg | |||
</ | <pre> | ||
>>> def fun(a): | |||
... return a+1 | |||
... | |||
>>> fun(9) | |||
10 | |||
>>> exit() | |||
</pre> | |||
On a python script | |||
< | <pre> | ||
def | def fun(a): | ||
return a+1 | |||
print fun(9) | |||
</pre> | |||
</ | |||
== | === Debug functions === | ||
https://stackoverflow.com/a/4929267 | |||
You can launch a Python program through [https://docs.python.org/3/library/pdb.html pdb] by using '''pdb myscript.py''' or '''python -m pdb myscript.py''' | |||
< | <pre> | ||
$ cat debug.py | |||
def fun(a): | |||
a= a*2 | |||
a= a*3 | |||
return a+1 | |||
def | print fun(5) | ||
return | |||
$ python -m pdb debug.py | |||
> /home/pi/Downloads/debug.py(1)<module>() | |||
-> def fun(a): | |||
(Pdb) b fun | |||
* | Breakpoint 1 at /home/pi/Downloads/debug.py:1 | ||
* | (Pdb) c | ||
> /home/pi/Downloads/debug.py(2)fun() | |||
-> a= a*2 | |||
(Pdb) n | |||
> /home/pi/Downloads/debug.py(3)fun() | |||
-> a= a*3 | |||
(Pdb) | |||
</ | > /home/pi/Downloads/debug.py(4)fun() | ||
-> return a+1 | |||
(Pdb) p a | |||
30 | |||
(Pdb) n | |||
--Return-- | |||
> /home/pi/Downloads/debug.py(4)fun()->31 | |||
-> return a+1 | |||
(Pdb) exit | |||
</pre> | |||
=== Boolean functions === | |||
Problem: checks if a given dna seq contains an in-frame stop condon | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
dna=input("Enter a dna seq: ") | |||
if (has_stop_codon(dna)) : | |||
print("input seq has an in frame stop codon.") | |||
else : | |||
print("input seq has no in frame stop codon.") | |||
def has_stop_codon(dna) : | |||
"This function checks if given dna seq has in frame stop codons." | |||
stop_codon_found=False | |||
stop_codons=['tga', 'tag', 'taa'] | |||
for i in range(0, len(dna), 3) : | |||
codon=dna[i:i+3].lower() | |||
if codon in stop_codons : | |||
stop_codon_found=True | |||
break | |||
return stop_codon_found | |||
</syntaxhighlight> | </syntaxhighlight> | ||
=== Function default parameter values === | |||
Suppose the has_stop_codon function also accepts a frame argument (equal to 0, 1, or 2) which specifies in what frame we want to look for stop codons. | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
def has_stop_codon(dna, frame=0) : | |||
"This function checks if given dna seq has in frame stop codons." | |||
stop_codon_found=False | |||
stop_codons=['tga', 'tag', 'taa'] | |||
for i in range(frame, len(dna), 3) : | |||
codon=dna[i:i+3].lower() | |||
if codon in stop_codons : | |||
stop_codon_found=True | |||
break | |||
return stop_codon_found | |||
dna="atgagcggccggct" | |||
</syntaxhighlight> | has_stop_codon(dna) # False | ||
has_stop_codon(dna, 0) # False | |||
has_stop_codon(dna, 1) # True | |||
has_stop_codon(frame=0, dna=dna) | |||
</syntaxhighlight> | |||
=== More examples === | |||
Reverse complement of a dna sequence | |||
<syntaxhighlight lang='python'> | |||
def reversecomplement(seq): | |||
"""Return the reverse complement of the dna string.""" | |||
seq = reverse_string(seq) | |||
seq = complement(seq) | |||
return seq | |||
reversecomplement('CCGGAAGAGCTTACTTAG') | |||
</syntaxhighlight> | </syntaxhighlight> | ||
To reverse a string | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
def reverse_string(seq): | |||
return seq[::-1] | |||
reverse_string(dna) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Complement a DNA Sequence | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
def complement(dna): | |||
"""Return the complementary sequence string.""" | |||
basecomplement = {'A':'T', 'C':'G', 'G':'C', 'T':'A', | |||
'N':'N', 'a':t', 'c':'g', 'g':'c', 't':'a', 'n':'n'} # dictionary | |||
letters = list(dna) # list comprehensions | |||
letters = [basecomplement[base] for base in letters] | |||
return ''.join(letters) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
=== Split and Join functions === | |||
<syntaxhighlight lang='python'> | |||
sentence="enzymes and other proteins come in many shapes" | |||
sentence.split() # split on all whitespaces | |||
sentence.split('and') # use 'and' as the separator | |||
'-'.join(['enzymes', 'and', 'other', 'proteins', 'come', 'in', 'many', 'shapes']) | |||
</syntaxhighlight> | |||
=== Variable number of function arguments === | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
def newfunction(fi, se, th, *rest): | |||
print("First: %s" % fi) | |||
print( | print("Second: %s" % se) | ||
print("Third: %s" % th) | |||
print("Rest... %s" % rest) | |||
return | |||
</syntaxhighlight> | |||
== Modules and packages == | |||
* [https://www.programiz.com/python-programming/modules Python Modules] | |||
* [https://www.w3schools.com/python/python_modules.asp Python Modules] from w3schools | |||
* [https://realpython.com/python-import/ Python import: Advanced Techniques and Tips] | |||
* [https://docs.python.org/3/py-modindex.html Python Module Index] | |||
'''Packages''' group multiple '''modules''' under on name, by using "dotted module names". For example, the module name A.B designates a submodule named B in a package named A. See [https://stackoverflow.com/questions/7948494/whats-the-difference-between-a-python-module-and-a-python-package What's the difference between a Python module and a Python package?] | |||
<dnautil.py> | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
#!/usr/bin/python | |||
""" | |||
dnautil module contains a few useful functions for dna seq | |||
""" | |||
def gc(dna) : | |||
blah | |||
blah | |||
return gcpercent | |||
</syntaxhighlight> | </syntaxhighlight> | ||
When a module is imported, Python first searches for a built-in module with that name. | |||
If built-in module is not found, Python then searches for a file obtained by | |||
adding the extension .py to the name of the module that it's imported: | |||
* in your current directory, | |||
* the directory where Python has been installed | |||
* in a path, i.e., a colon(':') separated list of file paths, stored in the environment variable PYTHONPATH. | |||
You can use the '''sys.path''' variable from the '''sys''' built-in module to check the list of all directories where Python look for files | |||
<syntaxhighlight lang='python'> | |||
import sys | |||
sys.path | |||
</syntaxhighlight> | |||
If the sys.path variable does not contains the directory where you put your module you can extend it: | |||
< | <syntaxhighlight lang='python'> | ||
sys.path.append("/home/$USER/python") | |||
sys.path | |||
</syntaxhighlight> | |||
</ | |||
=== Using modules (from PACKAGE/DIRNAME/FILENAME import CLASS) === | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
import | from math import * | ||
print(floor(3.7)) | |||
import dnautil | |||
dnautil.gc(dna) | dna="atgagggctaggt" | ||
gc(dna) # gc is not defined | |||
dnautil.gc(dna) # Good | |||
</syntaxhighlight> | </syntaxhighlight> | ||
= | Import Names from a Module | ||
<syntaxhighlight lang='python'> | |||
from dnautil import * | |||
gc(dna) # OK | |||
from dnautil import gc, has_stop_codon | |||
</syntaxhighlight> | </syntaxhighlight> | ||
[https://opensource.com/article/19/7/get-modular-python-functions Get modular with Python functions] & [https://opensource.com/article/19/7/get-modular-python-classes Learn object-oriented programming with Python] from opensource.com. | |||
from | |||
=== from...import vs import vs import...as === | |||
<ul> | |||
<li>[https://vinesmsuic.github.io/2020/06/05/python-import-vs-fromimport/ Difference between 'import' and 'from...import' in Python] </li> | |||
<li>[https://core-electronics.com.au/tutorials/import-from-as-python.html Import, From and As Keywords in Python] </li> | |||
<li>[https://stackoverflow.com/a/21547572 `from … import` vs `import .`] </li> | |||
<li>[http://www.wellho.net/mouth/418_Difference-between-import-and-from-in-Python.html Difference between import and from in Python]. | |||
Python's '''import''' loads a Python module into its own namespace, so that you have to add the module name followed by a dot in front of references to any names from the imported module that you refer to: | |||
<syntaxhighlight lang='python'> | |||
import feathers | |||
duster = feathers.ostrich("South Africa") | |||
</syntaxhighlight> | |||
'''from''' loads a Python module into the current namespace, so that you can refer to it without the need to mention the module name again: | |||
<syntaxhighlight lang='python'> | |||
from feathers import * | |||
duster = ostrich("South Africa") | |||
</syntaxhighlight> | |||
<ul> | |||
<li>Question: Why are both import and from provided? Can't I always use from? | |||
Answer: If you were to load a lot of modules using from, you would find sooner or later that there was a conflict of names; from is fine for a small program but if it was used throughout a big program, you would hit problems from time to time | |||
</li> | |||
<li> | |||
Question: Should I always use import then? | |||
Answer: No ... '''use import most of the time, but use from is you want to refer to the members of a module many, many times in the calling code'''; that way, you save yourself having to write "feather." (in our example) time after time, but yet you don't end up with a cluttered namespace. You could describe this approach as being the best of both worlds. | |||
</li> | |||
</ul> | |||
<li>[https://stackoverflow.com/a/22245722 from … import OR import … as for modules] </li> | |||
<li>Some examples: | |||
<syntaxhighlight lang='python'> | |||
from numpy import array # Run file; load specific 'attribute' | |||
arr = array([1,2,3]) # Use name directly: no need to qualify | |||
print(arr) # print [1 2 3] | |||
from math import pi | |||
pi # 3.141592653589793 | |||
math.pi # NameError: name 'math' is not defined | |||
</syntaxhighlight> | </syntaxhighlight> | ||
VS | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
import numpy # Run file; load module as a whole | |||
arr = numpy.array([1,2,3]) # Use its attribute names: '.' to qualify | |||
print(arr) # print [1 2 3] | |||
import math | |||
math.pi # 3.141592653589793 | |||
dir(math) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
VS | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
import numpy as np | |||
dir(np) | |||
import math as m | |||
m.pi # 3.141592653589793 | |||
</syntaxhighlight> | </syntaxhighlight> | ||
</li> | |||
<li>[https://github.com/qianhuiSenn/scRNA_cell_deconv_benchmark/blob/master/Scripts/Result_preprocessing/metrics_calculation_example.ipynb scRNA_cell_deconv_benchmark] example. | |||
</li> | |||
</ul> | |||
=== help === | |||
< | <pre> | ||
from AAA import BBB | |||
help(BBB) | |||
help(BBB.FunctionName) | |||
import BBB as CCC | |||
help(CCC) | |||
</pre> | |||
=== Packages & __init__.py === | |||
... | Each package in Python is a directory which MUST contain a special file __init__.py. This file can be empty and it indicates that the directory it contains is a Python package, so it can be imported the same way a module can be imported. https://docs.python.org/2/tutorial/modules.html | ||
... | |||
Example: suppose you have several modules dnautil.py, rnautil.py , and proteinutil.py. You want to group them in a package called "bioseq" which processes all types of biological sequences. The structure of the package: | |||
<pre> | <pre> | ||
bioseq/ | |||
__init__.py | |||
dnautil.py | |||
rnautil.py | |||
proteinutil.py | |||
fasta/ | |||
__init__.py | |||
fastautil.py | |||
fastq/ | |||
__init__.py | |||
fastqutil.py | |||
</pre> | </pre> | ||
Loading from packages: | |||
<syntaxhighlight lang='python'> | |||
import bioseq.dnautil | |||
bioseq.dnautil.gc(dna) | |||
from bioseq import dnautil | |||
dnautil.gc(dna) | |||
from bioseq.fasta.fastautil import fastqseqread | |||
<syntaxhighlight | </syntaxhighlight> | ||
=== Example === | |||
[https://www.youtube.com/watch?v=rfscVS0vtbw&t=14257s Building a Multiple Choice Quiz] by freeCodeCamp.org | |||
'''QuestionFile.py''' | |||
<syntaxhighlight lang='python'> | |||
class Question: | |||
def __init__(self, prompt, answer): | |||
self.prompt = prompt | |||
self.answer = answer | |||
</syntaxhighlight> | </syntaxhighlight> | ||
'''app.py''' | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
import | from QuestionFile import Question | ||
question_prompts = [ | |||
"What color are apples?\n(a) Red/Green\n(b) Purple\n(c) Orange\n\n", | |||
"What color are Bananas?\n(a) Teal\n(b) Magenta\n(c) Yellow\n\n", | |||
""" | "What color are strawberries?\n(a) Yellow\n(b) Red\n(c) Blue\n\n" | ||
] | |||
questions = [ | |||
Question(question_prompts[0], "a"), | |||
Question(question_prompts[1], "c"), | |||
Question(question_prompts[2], "b") | |||
] | |||
def run_test(question): | |||
score = 0 | |||
for question in questions: | |||
print(" | answer = input(question.prompt) | ||
if answer == question.answer: | |||
score += 1 | |||
print("You got " + str(score) + " /" + str(len(questions))+ " correct") | |||
run_test(questions) | |||
</syntaxhighlight> | |||
Run the program by '''python3 app.py''' | |||
== Files - Communicate with the outside == | |||
* [https://www.pythonforbeginners.com/files/reading-and-writing-files-in-python Reading and Writing Files in Python] | |||
* [https://realpython.com/read-write-files-python/#iterating-over-each-line-in-the-file Reading and Writing Files in Python (Guide)] | |||
<syntaxhighlight lang='python'> | |||
f=open('myfile', 'r') # read | |||
f=open('myfile') | |||
f=open('myfile', 'w') # write | |||
f=open('myfile', 'a') # append | |||
</syntaxhighlight> | |||
=== Take care if a file does not exists === | |||
<syntaxhighlight lang='python'> | |||
try: | |||
f = open('myfile') | |||
except IOError: | |||
print("the file myfile does not exist!!") | |||
</syntaxhighlight> | |||
=== Reading === | |||
<syntaxhighlight lang='python'> | |||
for line in f: | |||
print(line) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Change positions within a file object | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
f.seek(0) # go to the beginning of the file | |||
f.read() | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Read a single line | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
f.seek(0) | |||
f.readline() | |||
</syntaxhighlight> | </syntaxhighlight> | ||
== | Write into a file | ||
<syntaxhighlight lang='python'> | |||
f=open("/home/$USER/myfile, 'a) | |||
f.write("this is a new line") | |||
f.close() | |||
>>> with open("file.txt", "w") as f: | |||
... f.write(str(object)) | |||
... | |||
</syntaxhighlight> | |||
[https://stackoverflow.com/a/16999000 Importing large tab-delimited .txt file into Python] | |||
<pre> | |||
# R | |||
write.table(iris[1:10,], file="iris.txt", sep="\t", quote=F, row.names=F) | |||
# Python | |||
import csv | |||
print( | with open('iris.txt') as f: | ||
reader = csv.reader(f, delimiter="\t") | |||
d = list(reader) | |||
print(d[0][2]) | |||
print(d[1][2]) | |||
# Shell | |||
< | $ python test_csv.py | ||
Petal.Length | |||
1.4 | |||
</pre> | |||
If the data are all numerical, we can use the numpy package. | |||
<pre> | |||
# R | |||
write.table(iris[1:10, 1:4], | |||
file="~/Downloads/iris2.txt", | |||
sep="\t", quote=F, row.names=F, col.names=F) | |||
# Python | |||
import numpy as np | |||
d = np.loadtxt('iris2.txt', delimiter="\t") | |||
print(d[0][2]) | |||
print(d[1][2]) | |||
# Shell | |||
$ python test_csv2.py | |||
1.4 | |||
1.4 | |||
</pre> | |||
=== Read text file from a URL === | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
import urllib.request | |||
url = "http://textfiles.com/adventure/aencounter.txt" | |||
file = urllib.request.urlopen(url) | |||
for line in file: | |||
print(line.decode('utf-8')) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
* [https://docs.python.org/3.0/library/urllib.request.html urllib.request — extensible library for opening URLs] | |||
* [https://www.guru99.com/accessing-internet-data-with-python.html Python Internet Access using Urllib.Request and urlopen()] | |||
=== Command line arguments === | |||
Suppose we run 'python processfasta.py myfile.fa' in the command line, then | |||
<syntaxhighlight lang='python'> | <syntaxhighlight lang='python'> | ||
import sys | |||
print(sys.argv) # ['processfasta.py', 'myfile.fa'] | |||
</syntaxhighlight> | |||
More completely | |||
<syntaxhighlight lang='python'> | |||
#!/usr/bin/python | |||
""" | |||
processfasta.py builds a dictionary with all sequences from a FASTA file. | |||
""" | |||
import sys | |||
filename=sys.argv[1] | |||
= | try: | ||
f = open(filename) | |||
except IOError: | |||
print("File %s does not exist!" % filename) | |||
</syntaxhighlight> | |||
= | Parsing command line arguments with '''getopt'''. Suppose we want to store in the dictionary the sequences bigger than a given length provided in the command line: 'processfasta.py -l 250 myfile.fa' | ||
<syntaxhighlight lang='python'> | |||
#!/usr/bin/python | |||
import sys | |||
import getopt | |||
def usage(): | |||
print """ | |||
processfasta.py: reads a FASTA file and builds a | |||
dictionary with all sequence bigger than a given length | |||
processfasta.py [-h] [-l <length>] <filename> | |||
-h print this message | |||
-l <length> filter all sequences with a length | |||
< | smaller than <length> | ||
(default <length>=0) | |||
<filename> the file has to be in FASTA format | |||
< | |||
= | o, a = getopt.getopt(sys.argv[1:], '1:h') | ||
[ | opts = {} # empty dictionary | ||
== | seqlen=0; | ||
for k,v in o: | |||
opts[k] = v | |||
if 'h' in opts.keys(): # he means the user wants help | |||
usage(); sys.exit() | |||
if len(a) < 1: | |||
usage(); sys.exit("input fasta file is missing") | |||
if 'l' in opts.keys(): | |||
if opts['l'] <0 : | |||
print("length of seq should be positive!"); sys.exit(0); | |||
seqlen=opts['l'] | |||
</syntaxhighlight> | </syntaxhighlight> | ||
== | === stdin and stdout === | ||
<syntaxhighlight lang='python'> | |||
sys.stdin.read() | |||
sys.stdout.write("Some useful ouput.\n") | |||
sys.stderr.write("Warning: input file was not found\n") | |||
</syntaxhighlight> | |||
</ | |||
== | === Call external programs === | ||
<syntaxhighlight lang='python'> | |||
import subprocess | |||
subprocess.call('["ls", "-l"]) # return code indicates the success or failure of the execution | |||
subprocess.call('["tophat", "genome_mouse_idx", "PE_reads_1.fq.gz", "PE_reads_2.fq.gz"]) | |||
</syntaxhighlight> | |||
== | == Exceptions == | ||
[https://www.thegeekstuff.com/2019/05/python-try-except-examples/ 5 Python Examples to Handle Exceptions using try, except and finally] | |||
== Debugging == | |||
[https://www.makeuseof.com/debug-python-code/ How to Debug Your Python Code] | |||
python - | |||
== [http://biopython.org/wiki/Main_Page Biopython] & Pubmed == | |||
* Parsers for various bioinformatics file formats (FASTA, Genbank) | |||
* Access to online services like NCBI Entrez or Pubmed databases | |||
* Interfaces to common bioinformatics programs such as BLAST, Clustalw and others. | |||
<syntaxhighlight lang='python'> | |||
import Bio | |||
print(Bio.__version__) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Running BLAST over the internet | |||
from | <syntaxhighlight lang='python'> | ||
from Bio.Blast import NCBIWWW | |||
fasta_string = open("myseq.fa").read() | |||
result_handle = NCBIWWW.qblast("blastn":, "nt", fasta_string) | |||
# blastn is the program to use | |||
# nt is the database to search against | |||
# default output is xml | |||
help(NCBIWWW.qblast) | |||
</syntaxhighlight> | |||
The BLAST record | |||
<syntaxhighlight lang='python'> | |||
from Bio.Blast import NCBIXML | |||
blast_record = NCBIXML.read(result_handle) | |||
</syntaxhighlight> | |||
Parse BLAST output | |||
<syntaxhighlight lang='python'> | |||
len(blast_record.alignments) | |||
E_VALUE_THRESH = 0.01 | |||
for alignment in blas_record.alignments: | |||
for hsp in alignment.hsps: | |||
if hsp.expect < E_VALUE_THRESH: | |||
print('***Alignment***') | |||
print('sequence:', alignment.title) | |||
print('length:', alignment.length) | |||
print('e value:', hsp.expect) | |||
print(hsp.query) | |||
print(hsp.match) | |||
print(hsp.sbjct) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
More help with Biopython | |||
* Biopython tutorial and cookbook: http://biopython.org/DIST/docs/tutorial/Tutorial.html | |||
* Biopython FAQ: http://biopython.org/DIST/docs/tutorial/Tutorial.html#htoc5 | |||
== pubmed_parser == | |||
[https://github.com/titipata/pubmed_parser Parser for Pubmed Open-Access XML Subset and MEDLINE XML Dataset] | |||
== pyTest == | |||
* https://wiki.python.org/moin/PyTest | |||
* https://docs.python-guide.org/writing/tests/ | |||
== | == pyc file == | ||
[https://stackoverflow.com/a/3918716 What is the difference between .py and .pyc files? [duplicate]]. I observe it can cause a problem when I want to modify a python file but it keeps using the old pyc file so my change is not used (Raspbery Pi e-ink example). | |||
== | == Shutdown or restart OS == | ||
Below is tested on Raspbian | |||
<syntaxhighlight lang='python'> | |||
import os | |||
os.system('sudo shutdown -h now') | |||
</syntaxhighlight> | |||
= | = Popular python libraries = | ||
[https://pythontips.com/2013/07/30/20-python-libraries-you-cant-live-without/ 20 Python libraries you can’t live without] | |||
== psutil == | |||
* | * [https://programtalk.com/python-examples/psutil.cpu_percent/ psutil.cpu_percent() examples]. Inspired by the e-ink example from Raspberry Pi. | ||
* https:// | * https://github.com/arvydas/blinkstick-python/wiki/Example%3A-Display-CPU-usage | ||
* https://www.liaoxuefeng.com/wiki/1016959663602400/1183565811281984 | |||
<syntaxhighlight lang='python'> | |||
# pip install psutil --user | |||
for x in range(10): | |||
psutil.cpu_percent(interval=1) | |||
</syntaxhighlight> | |||
== | == [http://www.numpy.org/ numpy] == | ||
* | * [https://engineering.ucsb.edu/~shell/che210d/numpy.pdf An introduction to Numpy and Scipy] | ||
* [https:// | * https://docs.scipy.org/doc/numpy-dev/user/quickstart.html | ||
* [https://s3.amazonaws.com/assets.datacamp.com/blog_assets/Numpy_Python_Cheat_Sheet.pdf Cheat sheets] | |||
* [https://www.geeksforgeeks.org/program-to-find-the-sum-of-each-row-and-each-column-of-a-matrix/ Program to find the Sum of each Row and each Column of a Matrix] | |||
== | == pandas == | ||
[https://www. | [https://www.makeuseof.com/pandas-manipulate-dataframes/ 30 pandas Commands for Manipulating DataFrames] | ||
Write a pandas dataframe to a text file using [https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html to.csv()]. https://stackoverflow.com/a/41514539 | |||
https:// | <pre> | ||
a.to_csv('xgboost.txt', header=True, index=True, sep='\t') | |||
</pre> | |||
== | == [https://www.scipy.org/ scipy] == | ||
[ | * [http://blog.nextgenetics.net/?e=94 Hypergeometric test] | ||
* [http://www2.stat.duke.edu/~ar182/rr/examples-gallery/PermutationTest.html Permutation Test] | |||
== | == seaborn == | ||
* | * https://seaborn.pydata.org/ | ||
* [https:// | * [https://github.com/georgezakinih/exploratory-data-analysis Examples of performaing Explorator Data Analysis for few public clinical data sets] | ||
* ChatGPT [https://twitter.com/arjunrajlab/status/1650134944641871878 GPT-4] | |||
= | == matplotlib == | ||
== | * https://matplotlib.org/users/installing.html | ||
https:// | * [https://www.makeuseof.com/draw-graphs-jupyter-notebook/ How to Draw Graphs in Jupyter Notebook] | ||
= | Installation. <syntaxhighlight lang='bash'> | ||
python -m pip install -U pip | |||
python -m pip install -U matplotlib | |||
# https://stackoverflow.com/a/50328517 | |||
sudo apt-get install python3.5-tk | |||
</syntaxhighlight> | |||
Example. <syntaxhighlight lang='python'> | |||
from sklearn import datasets | |||
iris = datasets.load_iris() | |||
import matplotlib.pyplot as plt | |||
iris = iris.data | |||
# Scatterplot | |||
plt.scatter(iris[:,1], iris[:,2]) | |||
plt.show() | |||
[ | # Boxplot | ||
plot.boxplot(iris[:,1]) | |||
plt.show() | |||
# Histogram | |||
plt.hist(iris[:,1]) | |||
plt.show() | |||
</syntaxhighlight> | |||
== scikit-learn == | |||
[https://scikit-learn.org/stable/index.html scikit-learn]: Machine Learning in Python | |||
Installation. <syntaxhighlight lang='bash'> | |||
pip install -U scikit-learn | |||
</syntaxhighlight> | |||
Example. | |||
python | <syntaxhighlight lang='python'> | ||
$ python | |||
>>> from sklearn import datasets | |||
>>> iris = datasets.load_iris() | |||
>>> digits = datasets.load_digits() | |||
</syntaxhighlight> | |||
== PyTorch == | |||
* https://pytorch.org/ | |||
* [https://github.com/PyTorchLightning/pytorch-lightning PyTorch Lightning] - The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate. | |||
== feedparser == | |||
[https://fedoramagazine.org/never-miss-magazines-article-build-rss-notification-system/ Never miss a Magazine article — build your own RSS notification system] | |||
== Boto == | |||
A Python interface to Amazon Web Services | |||
* http://docs.pythonboto.org/en/latest/ | |||
* https://hpc.nih.gov/training/handouts/object_storage_class_2018_oct.pdf | |||
https:// | == PIL, Pillow == | ||
* Installation <syntaxhighlight lang='bash'>sudo apt install python-imaging </syntaxhighlight> | |||
* [https://stackoverflow.com/a/24103766 How I can load a font file with PIL.ImageFont.truetype without specifying the absolute path?] | |||
== | == plotnine == | ||
[https://www.r-bloggers.com/2020/11/python-and-r-part-2-visualizing-data-with-plotnine/ Python and R – Part 2: Visualizing Data with Plotnine] | |||
== nltk: Natural Language Toolkit == | |||
https://www.nltk.org/ | |||
== pygame == | |||
[https://opensource.com/article/20/10/learn-python-ebook Learn Python by creating a video game] | |||
== scanpy == | |||
* [https://github.com/theislab/scanpy scanpy] and the [https://scanpy.readthedocs.io/en/stable/installation.html installation instruction] | |||
. | * [https://github.com/chriscainx/mnnpy mnnpy] | ||
= Trouble shooting = | |||
== ImportError: cannot import name main when running pip == | |||
https://stackoverflow.com/a/50187211 | |||
== | == Error: externally-managed-environment == | ||
See [[Python#pipx_.28alternative_to_pip3.29|pipx]] | |||
= | == TypeError: ‘module’ object is not callable == | ||
I was trying to run "bbknn.py" from [https://github.com/oxwang/fda_scRNA-seq/tree/master/4_Batch_correction/Code here]. | |||
[https://www.thecrazyprogrammer.com/2020/11/typeerror-module-object-is-not-callable.html Solve “TypeError: ‘module’ object is not callable” in Python], [https://stackoverflow.com/a/4534443 TypeError: 'module' object is not callable] | |||
The problem is I have a file called "bbknn.py" and I have "import bbknn" in the code. It will confuse python. The solution is to rename my script file "bbknn.py" ('''avoid MODULE.py''') to other name like "bbknnDemo.py". | |||
[https://www.quora.com/When-I-import-my-module-in-python-it-automatically-runs-all-of-the-defined-functions-inside-of-it-How-do-I-prevent-it-from-auto-executing-my-functions-but-still-allow-me-to-call-them-in-my-main-script When I import my module in python, it automatically runs all of the defined functions inside of it. How do I prevent it from auto executing my functions, but still allow me to call them in my main script?] | |||
[https://www. | |||
== | == Illegal instruction == | ||
I got this error after I called python3 -c 'import scanpy'. [https://hpc.nih.gov/apps/python.html Python on Biowulf]. | |||
<pre> | |||
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh | |||
TMPDIR=/tmp bash Miniconda3-latest-Linux-x86_64.sh -p ~/conda -b | |||
source ~/conda/etc/profile.d/conda.sh # ~/conda/condabin is added to PATH | |||
conda activate base | |||
python -V # Python 3.9.4 | |||
pip3 install - | conda create -n project1 pandas numpy scipy -y | ||
conda activate project1 | |||
pip3 install scanpy bbknn | |||
ls ~/conda/envs/project1/lib/python3.9/site-packages | |||
# bbknn and scanpy are there | |||
python3 -c 'import scanpy' | |||
# Illegal instruction | |||
conda info --env | |||
conda deactivate | |||
conda remove --all -n project1 -y | |||
conda deactivate | |||
</pre> | </pre> | ||
= | == No matching distribution found for XXX == | ||
[https:// | Got an error ''No matching distribution found for lasagne==0.2.dev1'' when I ran '' 'pip install .' '' on [https://github.com/jaredleekatzman/DeepSurv DeepSurv]. | ||
https://github.com/imatge-upc/saliency-salgan-2017/issues/29 | |||
= | == Python AttributeError: 'module' object has no attribute 'SSL_ST_INIT' == | ||
See https://stackoverflow.com/a/52398193. I got this message after I ran ''sudo pip install --upgrade cryptography'' and ''pip show cryptography''. The reason I try to upgrade cryptography is the following message | |||
<pre> | <pre> | ||
$ pip show protobuf | |||
/home/brb/.local/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py:83: | |||
RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown. | |||
warnings.warn(warning, RequestsDependencyWarning) | |||
Name: protobuf | |||
... | |||
</pre> | </pre> | ||
And OpenSSL & pyOpenSSL-0.15.1.egg-inf are under /usr/lib/python2.7/dist-packages directory on my Ubuntu 16.04. | |||
Note the following solutions do not work | |||
<pre> | <pre> | ||
$ sudo pip uninstall pyopenssl | |||
$ sudo pip install pyOpenSSL==16.2.0 | |||
</pre> | </pre> | ||
I always get an error message | |||
<pre> | |||
... | |||
File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 118, in <module> | |||
SSL_ST_INIT = _lib.SSL_ST_INIT | |||
AttributeError: 'module' object has no attribute 'SSL_ST_INIT' | |||
< | |||
</pre> | </pre> | ||
And a quick solution is to do '''sudo rm -r /usr/local/lib/python2.7/dist-packages/OpenSSL'''. I also did '''sudo pip install pyopenssl''' but I did not follow [https://stackoverflow.com/a/53195423 this answer] ('''sudo apt install --reinstall python-openssl'''). | |||
== /usr/bin/env: ‘python’: No such file or directory == | |||
On [https://askubuntu.com/a/1234598 Ubuntu 20.04], | |||
<pre> | <pre> | ||
--- | sudo apt-get install python-is-python3 | ||
</pre> | |||
This solved an error when I used [https://yt-dl.org/ youtube-dl]. | |||
= Projects based on python = | |||
* [http://kevinmehall.net/p/pithos/ pithos] Pandora on linux | |||
* Many Raspberry Pi GPIO projects | |||
* [http://csbio.unc.edu/genescissors/instruction.html GeneScissors] It also requires pip and scikit-learn packages. | |||
* [http://keepnote.org KeepNote] It depends on Python 2.X, [http://www.sqlite.org sqlite] and [http://www.pygtk.org PyGTK]. | |||
* [http://www.zim-wiki.org Zim] It depends on Python, Gtk and the python-gtk bindings. | |||
* [http://www.giuspen.com/cherrytree Cherrytree] It depends on Python2, Python-gtk2, Python-gtksourceview2, p7zip-full, python-enchant and python-dbus. | |||
= Send emails = | |||
* [https://support.google.com/accounts/answer/6010255?hl=en&utm_source=google-account&utm_medium=profile-less-secure-apps-card Less secure apps & your Google Account]. To help keep your account secure, from May 30, 2022, Google no longer supports the use of third-party apps or devices which ask you to sign in to your Google Account using only your username and password. | |||
** [https://pythonassets.com/posts/send-email-via-gmail-and-smtp/ Send Email via Gmail and SMTP] Use an '''App Password''' 2022/9. Click on Security -> 2-Step Verification (You may need to enter your PW first). Scroll to the bottom of the page, and you'll see the "App passwords" section. You can delete/create app passwords but you can't view any existing passwords. | |||
* [https://www.makeuseof.com/python-send-email/ How to Send Automated Email Messages in Python 3] 2021/3 | |||
* [https:// | |||
* [https:// | |||
* [https://www. | |||
= | = GUI programming = | ||
[https://www.raspberrypi.org/blog/create-graphical-user-interfaces-with-python/ New book: Create Graphical User Interfaces with Python] | |||
== | == Qt for GUI development == | ||
* | * http://zetcode.com/gui/pyqt4/ | ||
* | * http://wiki.wildsong.biz/index.php/PyQt Create GUI in Qt Designer and convert/use it in PyQt. | ||
== | = Python 3 = | ||
* [https://github.com/ | * Python 2.7 will not be maintained past 2020. See https://pythonclock.org/. | ||
* [https:// | * [https://github.com/arogozhnikov/python3_with_pleasure Migrating to Python 3 with pleasure] | ||
* [https://github.com/wesm/pydata-book Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython] by Wes McKinney. | |||
= | == pip3 == | ||
Use '''pip3''' instead of '''pip''' for Python 3. For example, | |||
<syntaxhighlight lang='bash'> | |||
pip3 install --upgrade pip | |||
pip3 install -U scikit-learn | |||
pip3 install -U matplotlib | |||
</syntaxhighlight> | |||
== | == http.server == | ||
[https:// | [https://developers.google.com/web/tools/chrome-devtools/workspaces/ Edit Files With Workspaces]. The 'http.server' module is contained in python3. | ||
<pre> | <pre> | ||
cd ~/website | |||
python3 -m http.server | |||
</pre> | |||
= C vs Python = | |||
[https://www.makeuseof.com/c-python-core-differences/ C vs. Python: The Key Differences] | |||
= | = R and Python: reticulate package = | ||
* [https:// | * [https://blog.rstudio.com/2018/03/26/reticulate-r-interface-to-python/ reticulate: R interface to Python] (2018). [https://rstudio.github.io/reticulate Latest version]. | ||
<ul> | <ul> | ||
<li>[https:// | <li>[https://support.rstudio.com/hc/en-us/articles/360023654474-Installing-and-Configuring-Python-with-RStudio Installing and Configuring Python with RStudio] | ||
<ul> | <ul> | ||
<li> | <li>The instruction is based on ''virtualenv''. But I'm following Biowulf's Python [https://hpc.nih.gov/apps/python.html#envs miniconda instruction] to create a new project/environment. One caveat is I need to run ''source ~/$USER/conda/etc/profile.d/conda.sh'' each time before I start R in order to make conda available OR I need to set the [https://rstudio.github.io/reticulate/reference/miniconda_path.html RETICULATE_MINICONDA_PATH] variable (see below). </li> | ||
<li> | <li>The conda-related reticulate functions include conda_create(), [https://www.rdocumentation.org/packages/reticulate/versions/1.20/topics/use_python use_condaenv()], conda_install(), conda_list(), conda_remove() </li> | ||
<li> | <li>Use '''py_config()''' to check the current python path and other python versions found. </li> | ||
<li>My example | |||
<pre> | <pre> | ||
library(reticulate) | |||
# Assume I followed Biowulf's instruction to create 'project1' | |||
[ | Sys.setenv(RETICULATE_MINICONDA_PATH = "~/conda") | ||
conda_list() | |||
use_condaenv("project1", required=T) | |||
py_config() | |||
</pre> | |||
</li> | |||
</ul> | |||
</li> | |||
</ul> | |||
* [[Rstudio#Python|RStudio -> Python]] | |||
* https://cran.r-project.org/web/packages/reticulate/index.html, [https://github.com/rstudio/reticulate Github] | |||
** Using Python in R markdown | |||
** Importing Python modules and call its functions directly from R — '''import()''' function | |||
** Sourcing Python scripts — '''source_python()''' function | |||
** Python REPL — The '''repl_python()''' function creates an interactive Python console within R. | |||
<ul> | |||
<li>[https://rstudio.github.io/reticulate/articles/versions.html Python Version Configuration]. Suppose I have installed miniconda and create a new environment called 'project1'. Then after calling '''source ~/conda/etc/profile.d/conda.sh''' I can start in R | |||
<pre> | |||
library(reticulate) | |||
use_condaenv("project1", required = TRUE) | |||
</pre> | |||
</li> | |||
</ul> | |||
* On my macOS, even I have python3 installed, it still asks to install miniconda (/Users/$USER/Library/r-miniconda). So I get another version of Python3 in '''/Users/$USER/Library/r-miniconda/envs/r-reticulate/bin/python'''. | |||
* I found RStudio IDE is better than PyCharm and Thonny editors. | |||
* Install Python packages https://rstudio.github.io/reticulate/articles/python_packages.html | |||
** Better to have [https://www.anaconda.com/distribution/ anaconda3] installed. 2.26G space is required on macOS. | |||
** Direct running py_install("pandas") would ask me to upgrade virtualenv | |||
** Running virtualenv_create("r-reticulate") and then py_install("pandas") works | |||
* Cheat sheet https://ugoproto.github.io/ugo_r_doc/pdf/reticulate.pdf | |||
* [https://www.brodrigues.co/blog/2018-12-30-reticulate/ R or Python? Why not both? Using Anaconda Python within R with {reticulate}] | |||
* [https://www.listendata.com/2018/03/run-python-from-r.html?m=1 Run Python from R] | |||
* [https://www.statworx.com/de/blog/r-and-python-using-reticulate-to-get-the-best-of-both-worlds/ R and Python: Using reticulate to get the best of both worlds]. Note | |||
** [https://rstudio.github.io/reticulate/articles/r_markdown.html RStudio v1.2 preview release includes support for using reticulate to execute Python chunks within R Notebooks] | |||
** Error from my execution: ''ValueError: 'RBF' is not in list'' | |||
* [https://rviews.rstudio.com/2019/03/18/the-reticulate-package-solves-the-hardest-problem-in-data-science-people/ The reticulate package solves the hardest problem in data science: people] | |||
* [https://rviews.rstudio.com/2019/06/10/reticulate-virtualenv-and-python-in-linux/ reticulate, virtualenv, and Python in Linux] | |||
* Bugs | |||
** [https://stackoverflow.com/a/49556037 Pass Python objects to R]: Works. Or use py_run_string() | |||
** [https://stackoverflow.com/a/52542230 Cannot pass R variables to Python]: use source_python() | |||
* [https://github.com/matloff/R-vs.-Python-for-Data-Science R vs Python for data science] by Norm Matloff. | |||
* [https://bensstats.wordpress.com/2020/11/05/rvspython-5-1-making-the-game-even-with-pythons-best-practices/ RvsPython #5.1: Making the Game even with Python’s Best Practices] | |||
* [https://bensstats.wordpress.com/2020/11/04/rvspython-5-using-monte-carlo-to-simulate-%CF%80/ RvsPython #5: Using Monte Carlo To Simulate π] | |||
* [https://www.business-science.io/learn-r/2020/04/20/setup-python-in-r-with-rmarkdown.html How to Run Python's Scikit-Learn in R in 5 minutes] | |||
<ul> | |||
<li>Test python and markdown files | |||
{{Pre}} | |||
def add_three(x): | |||
z = x + 3 | |||
return z | |||
</pre> | </pre> | ||
<pre> | |||
--- | |||
title: "R Notebook" | |||
output: html_notebook | |||
--- | |||
```{r} | |||
library(reticulate) | |||
py_discover_config() | |||
x <- 5 | |||
source_python("test.py") | |||
y <- add_three(x) | |||
print(y) | |||
``` | |||
Pass R variables to Python. Works | |||
```{python} | |||
a = 7 | |||
print(r.x) | |||
``` | |||
Pass python variables to R. Works. | |||
```{r} | |||
py$a | |||
py_run_string("y = 10"); py$y | |||
``` | |||
</pre> | |||
</li> | |||
</ul> | |||
* [https://hutsons-hacks.info/reticulate-webinar-r-and-python-a-happy-union Reticulate webinar – R and Python – a happy union] | |||
* [https://datascienceplus.com/linking-r-and-python-to-retrieve-financial-data-and-plot-a-candlestick/ Linking R and Python to retrieve financial data and plot a candlestick] | |||
* [https://www.r-bloggers.com/2022/04/getting-started-with-python-using-r-and-reticulate/ Getting started with Python using R and reticulate] | |||
== How to quit python == | |||
Type '''exit''' and hit Enter. See https://rstudio.github.io/reticulate/. | |||
== R vs Python == | |||
* [https://www.business-science.io/business/2021/07/12/R-is-for-research-Python-is-for-production.html R is for Research, Python is for Production] | |||
* [https://datasciencetut.com/difference-between-r-and-python/ Difference between R and Python] | |||
== Call R from Python == | |||
* [https://github.com/kpj/rwrap rwrap] Seamlessly integrate R packages into Python. | |||
* [https://pypi.org/project/rpy2/ rpy2] | |||
= | = Conda, Anaconda, miniconda = | ||
* [[Docker#Conda|Docker]] | |||
* [https://hpc.nih.gov/apps/python.html Python on Biowulf]. Users who need stable, reproducible environments are encouraged to install miniconda in their data directory and create their own private environments. | |||
* [ | * [https://towardsdatascience.com/a-guide-to-conda-environments-bc6180fc533 The Definitive Guide to Conda Environments] | ||
* | |||
* | |||
conda | |||
== | == Private environment == | ||
[https://hpc.nih.gov/docs/diy_installation/conda.html Conda on Biowulf] & [https://github.com/conda-forge/miniforge#mambaforge mambaforge] | |||
== | == Transfer a conda environment to another computer: YAML files == | ||
[https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html Managing environments] | |||
<pre> | |||
# computer 1 | |||
conda env export > environment.yml | |||
# computer 2 | |||
conda env create -f environment.yml | |||
<pre> | </pre> | ||
conda | == Conda environment create, activate, deactivate, info (see a list) == | ||
< | [https://conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-conda Getting started with conda]. More details are in [https://conda.io/projects/conda/en/latest/user-guide/tasks/index.html Tasks]. | ||
<syntaxhighlight lang='bash'> | |||
conda --version | |||
# Manage environment | |||
conda info --envs # see a list of environments. | |||
# The active environment is the one with an asterisk (*) | |||
# create a new environment | |||
conda create --name myenv | |||
# remove an environment | |||
conda remove --name myenv --all | |||
# Manage Python | |||
conda create --name snakes python=3.5 | |||
conda | conda activate snowflakes # activate | ||
# | conda info --envs | ||
base | python --version | ||
conda activate # Change your current environment back to the default (base) | |||
conda deactivate # exit any python virtualenv | |||
# Managing packages | |||
conda search beautifulsoup4 | |||
conda | conda install beautifulsoup4 | ||
conda | conda list | ||
# Updating Anaconda or Miniconda | |||
conda update conda | |||
</syntaxhighlight> | |||
</ | |||
== Anaconda == | |||
* [https://research.computing.yale.edu/sites/default/files/files/anaconda.pdf Introduction to Anaconda]. Simplifies installation of Python packages | |||
** Platform-independent package manager | |||
** Doesn’t require administrative privileges | |||
** Installs non-Python library dependencies (MKL, HDF5, Boost) | |||
** Provides ”virtual environment” capabilities | |||
< | ** Many channels exist that support additional packages | ||
* [https://docs.anaconda.com/anaconda/install/mac-os/ Install Anaconda on macOS]. Better to use the command line method in order to install it to the user's directory. The new python can be manually loaded into the shell by using '''source ~/.bash_profile'''. Like Ubuntu, ananconda3 is installed under ~/ directory. In addition, '''Anaconda-Navigator''' is available under Finder -> Applications. | |||
<ul> | |||
<li>[https://www.digitalocean.com/community/tutorials/how-to-install-the-anaconda-python-distribution-on-ubuntu-16-04 How To Install the Anaconda Python Distribution on Ubuntu 16.04]. As we can see Anaconda3 will be installed under '''/home/$USER/anaconda3'''. | |||
<ul> | |||
<li>Download '''Anaconda3-2018.12-Linux-x86_64.sh''' from https://www.anaconda.com/distribution/#download-section </li> | |||
<li>bash Anaconda3-2018.12-Linux-x86_64.sh </li> | |||
<li>There is a question: <span style="color: red">Do you wish the installer to initialize Anaconda3</span>. If you answer Yes, it will run '''conda init''' & modify ~/.bashrc file. # <span style="color: red"> This will overwrite system's Python</span>. So the default python/python3 will now be in /home/$USER/anaconda3/bin/. | |||
<pre> | |||
Do you wish the installer to initialize Anaconda3 | |||
by running conda init? [yes|no] | |||
[no] >>> yes | |||
no change /home/brb/anaconda3/condabin/conda | |||
no change /home/brb/anaconda3/bin/conda | |||
no change /home/brb/anaconda3/bin/conda-env | |||
no change /home/brb/anaconda3/bin/activate | |||
no change /home/brb/anaconda3/bin/deactivate | |||
no change /home/brb/anaconda3/etc/profile.d/conda.sh | |||
no change /home/brb/anaconda3/etc/fish/conf.d/conda.fish | |||
no change /home/brb/anaconda3/shell/condabin/Conda.psm1 | |||
no change /home/brb/anaconda3/shell/condabin/conda-hook.ps1 | |||
no change /home/brb/anaconda3/lib/python3.8/site-packages/xontrib/conda.xsh | |||
no change /home/brb/anaconda3/etc/profile.d/conda.csh | |||
modified /home/brb/.bashrc | |||
== | ==> For changes to take effect, close and re-open your current shell. <== | ||
If you'd prefer that conda's base environment not be activated on startup, | |||
set the auto_activate_base parameter to false: | |||
conda config --set auto_activate_base false | |||
< | </pre> | ||
If I choose not to modify .bashrc file, | |||
<pre> | <pre> | ||
conda create -n r_env r-essentials r-base | Do you wish the installer to initialize Anaconda3 | ||
conda activate r_env | by running conda init? [yes|no] | ||
</pre> | [no] >>> no | ||
<li>Difference of install a package using '''install.packages()''' function in R and using the '''conda install''' command? | |||
* The install.packages() function in R and the conda install command are two different ways to install R packages. The install.packages() function is used to install packages from the Comprehensive R Archive Network (CRAN), while the conda install command is used to install packages from the Anaconda repository. | You have chosen to not have conda modify your shell scripts at all. | ||
* One key difference between the two methods is that conda can manage dependencies across multiple programming languages, while install.packages() only manages dependencies within R. | To activate conda's base environment in your current shell session: | ||
* Another difference is that conda allows you to create and manage multiple isolated environments, each with its own set of packages. This can be useful if you want to have different versions of packages available for different projects. With install.packages(), all packages are installed in the same global library, which can make it more difficult to manage dependencies and avoid conflicts. | |||
<li> | eval "$(/home/brb/anaconda3/bin/conda shell.YOUR_SHELL_NAME hook)" | ||
[https://towardsdatascience.com/a-guide-to-conda-environments-bc6180fc533 The Definitive Guide to Conda Environments], [https://docs.anaconda.com/anaconda/user-guide/tasks/using-r-language/ Using R language with Anaconda]. '''Environments''' created with ''conda create'' live by default in the '''envs/''' folder of your Conda directory, whose path will look something like ''/Users/user-name/miniconda3/envs'' or ''/Users/user-name/anaconda3/envs''. | |||
<pre> | To install conda's shell functions for easier access, first activate, then: | ||
Activate conda base Create a new env Activate a new env Deactivate an env | |||
----------------------------> (base) -----------------> -------------------> (r-env) -----------------> (base) | conda init | ||
eval $(conda shell.bash hook)" conda create r-env conda activate r-env conda deactivate | |||
</pre> | If you'd prefer that conda's base environment not be activated on startup, | ||
<pre> | set the auto_activate_base parameter to false: | ||
$ eval "$(/home/brb/anaconda3/bin/conda shell.bash hook)" | |||
(base) $ mkdir mypythonproj; cd mypythonproj # This step seems not necessary | conda config --set auto_activate_base false | ||
(base) $ conda create -n r-env r-base | |||
... | Thank you for installing Anaconda3! | ||
# | |||
# To activate this environment, use | </pre></li> | ||
# | <li>'''Anaconda-Navigator''' (including jupyter notebook, Spyder IDE, ...) can be launched by typing '''anaconda-navigator''' in a terminal </li> | ||
# $ conda activate r-env | </ul> | ||
# | </li> | ||
# To deactivate an active environment, use | </ul> | ||
# | * [https://opensource.com/article/18/4/getting-started-anaconda-python Getting started with Anaconda Python for data science] | ||
# $ conda deactivate | * Differences: | ||
(base) $ conda activate r-env | ** [https://stackoverflow.com/questions/30034840/what-are-the-differences-between-conda-and-anaconda What are the differences between Conda and Anaconda] | ||
(r-env) $ ls anaconda3/envs | ** [https://www.quora.com/What-is-the-comparison-among-conda-vs-pip-vs-anaconda What is the comparison among conda vs pip vs anaconda?] | ||
r-env | ** [https://bioconda.github.io/faqs.html#conda-anaconda-minconda What’s the difference between Anaconda, conda, and Miniconda?] | ||
(r-env) $ conda install r-essentials | * Comparions: | ||
(r-env) $ which R | ** [https://conda.io/docs/ Conda]: an open source package management system and environment management system | ||
/home/brb/anaconda3/envs/r-env/bin/R | ** [https://conda.io/miniconda.html Miniconda], which is a smaller alternative to Anaconda that is just conda and its dependencies. Once you have Miniconda, you can easily install Anaconda into it with '''conda install anaconda'''. | ||
(r-env) $ ls -la # Still Empty | ** [https://anaconda.org/ Anaconda]: Anaconda is a set of about a hundred packages including conda, numpy, scipy, ipython notebook, and so on. | ||
(r-env) $ R --version | *** [https://anaconda.org/r/rstudio RStudio] | ||
R version 3.4.3 (2017-11-30) -- "Kite-Eating Tree" | *** [https://bioconda.github.io/ Bioconda] is a '''channel''' for the conda package manager specializing in bioinformatics software. | ||
# Note that the current R version should be 4.0.3 | * [https://docs.anaconda.com/anaconda/install/uninstall/ Uninstall] | ||
(r-env) $ conda env list | * Used in [https://github.com/MaxSalm/pdxBlacklist pdxBlacklist] | ||
base /home/brb/anaconda3 | * [https://stringfestanalytics.com/what-is-open-source-distribution/ What is an open source software distribution?] | ||
r-env * /home/brb/anaconda3/envs/r-env | |||
(r-env) $ conda deactivate | == Miniconda == | ||
(base) $ | * https://docs.conda.io/en/latest/miniconda.html As you can see miniconda installers were separated by the Python version. | ||
</pre> | * [https://ostechnix.com/how-to-install-miniconda-in-linux/ How To Install Miniconda In Linux] 2021. It includes Install Miniconda interactively, '''unattended installation''', '''Update Miniconda''', and '''Uninstall Miniconda'''. If you've chosen the default location, the installer will display “PREFIX=/var/home/<user>/miniconda3”. To manually activate conda's base environment, do '''/home/<user>/miniconda3/etc/profile.d/conda.sh''' where we assume miniconda is installed under /home/<user>/miniconda3 directory. | ||
It seems to be better to save the environment inside a project directory. So using '''python -m venv /path/to/new/environment''' method is preferred. You can also use '''conda create --prefix /path/to/new/environment'''. Placing environments outside of the default env/ folder comes with some drawbacks. Read the document of 'The Definitive Guide to Conda Environments'. | * [https://youtu.be/bbIG5d3bOmk Miniconda Installation for macOS users] 2019. At the end of installation, we see if we don't want conda's base environment to be activated on start up, we can do '''conda config --set auto_activate_base false''' | ||
</li> | * See also [https://hpc.nih.gov/apps/python.html Python on Biowulf] about how to specify '''prefix'''. | ||
<li> [https://conda-forge.org/ conda-forge channel], [https://conda-forge.org/docs/user/introduction.html A brief introduction], https://anaconda.org/conda-forge/r-base. Following the instruction seems to mess things up though the conda-forge says the latest version is 4.0.3 (3 years late). | * We can add/install a module to an existing environment. See [https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/background_install/miniconda.html#sidenote Miniconda: Python(s) in a convenient setup]. | ||
{{Pre}} | :<syntaxhighlight lang='bash'> | ||
$ eval "$(/home/brb/anaconda3/bin/conda shell.bash hook)" | conda install -n <env_name> <package> | ||
(base) $ conda install -c conda-forge r-base | conda create -n myenv python=3 # create a new environment named “myenv” with Python 3 installed | ||
... | # after that, use "conda activate myenv" and use "conda install numpy" to install the numpy | ||
## Package Plan ## | </syntaxhighlight> | ||
=== Install and "conda init" === | |||
* Windows: screenshots are included [https://katiekodes.com/setup-python-windows-miniconda/ Setting up Python on Windows with Miniconda by Anaconda] & [https://docs.anaconda.com/free/anaconda/install/windows/ Anaconda documentation]. The default is not to add Anaconda to my PATH environment variable. | |||
* Ubuntu: [https://varhowto.com/install-miniconda-ubuntu-20-04/ How to Install Miniconda on Ubuntu 20.04]. After installation, PATH variable will prepend '''~/miniconda3/condabin''' which contains only 1 file: '''conda'''. | |||
* '''conda init''' | |||
** Running conda init initializes conda for shell interaction by writing some shell code in the relevant startup scripts of your shell (e.g~/.bashrc) 1. This allows the conda command to interact more closely with the shell context and provides a cleaner PATH manipulation and snappier responses in some conda commands. The main advantage of running conda init is that it enables the use of the [https://docs.conda.io/projects/conda/en/latest/dev-guide/deep-dives/activation.html conda activate and conda deactivate] commands, which are used to activate and deactivate conda environments. | |||
** We only need to call "conda init" once no matter after we install conda how many conda environments we will work. | |||
** One disadvantage of running conda init is that it can sometimes cause issues if the initialization is not done correctly or if there are conflicts with other configurations in your [https://stackoverflow.com/questions/58388190/conda-init-doesnt-work-in-bash-on-windows shell startup scripts]. However, these issues can usually be resolved by troubleshooting and making the necessary changes to your configurat | |||
=== conda environment === | |||
A conda environment is a directory that contains a specific collection of conda packages that you have installed. | |||
You need to create an environment first before you can activate it. The conda activate command does not create an environment for you, it only activates an existing one. | |||
<pre> | |||
conda create --name myenv | |||
conda activate myenv | |||
</pre> | |||
Q: Where are the environments located? A: Conda environments are typically stored in the envs subdirectory of your Anaconda installation directory. For example, if you have an environment named myenv, it would be located in a directory like ~/anaconda3/envs/myenv. The exact path can be found by using '''conda env list''' command. | |||
Q: How do I list all existing environments? A: To list all existing conda environments, you can use the conda env list or conda info --envs command. Here’s how you do it: | |||
<pre> | |||
conda env list | |||
# | |||
base * /opt/conda | |||
DrivR-Base /opt/conda/envs/DrivR-Base | |||
</pre> | |||
Q: How to quit a conda environment? | |||
<pre> | |||
conda deactivate # Return to base | |||
conda deactivate # Exit base | |||
</pre> | |||
Q: Check the disk space used by a specific conda environment. | |||
<pre> | |||
du -sh /path/to/conda/envs/your_enviornment_name | |||
</pre> | |||
Q: How to delete a conda environemnt, | |||
<pre> | |||
conda deactivate | |||
conda env remove --name your_environment_name | |||
# OR | |||
conda remove --name your_environment_name --all | |||
</pre> | |||
=== Install all anaconda packages === | |||
* https://stackoverflow.com/a/52316549 '''conda install anaconda''' | |||
* https://docs.conda.io/en/latest/miniconda.html '''conda create -n py3k anaconda python=3''' | |||
* how much space is needed for installing anaconda? The minimum disk space required for installing Anaconda is 3 GB, but it is recommended to have at least 6 GB of free disk space available. | |||
=== Uninstall miniconda === | |||
# rm -rf ~/miniconda3 | |||
# nano ~/.bash_profile and delete conda initialize block | |||
=== What's the purpose of the “base” (for best practices) in Anaconda? === | |||
https://stackoverflow.com/a/56504279 | |||
=== Does Conda replace the need for virtualenv? === | |||
[https://stackoverflow.com/a/34398794 Yes]. Conda is not limited to Python but can be used for other languages too. | |||
== GCC/gFortran == | |||
* '''conda install gcc''' | |||
* Using [https://anaconda.org/conda-forge/gfortran/ conda-forge] channel - '''conda install -c conda-forge gfortran''' | |||
== Using R language with Anaconda == | |||
<ul> | |||
<li>[https://docs.anaconda.com/free/anaconda/packages/using-r-language/ Using R language with Anaconda] | |||
<pre> | |||
conda create -n r_env r-essentials r-base | |||
conda activate r_env | |||
</pre> | |||
<li>Difference of install a package using '''install.packages()''' function in R and using the '''conda install''' command? | |||
* The install.packages() function in R and the conda install command are two different ways to install R packages. The install.packages() function is used to install packages from the Comprehensive R Archive Network (CRAN), while the conda install command is used to install packages from the Anaconda repository. | |||
* One key difference between the two methods is that conda can manage dependencies across multiple programming languages, while install.packages() only manages dependencies within R. | |||
* Another difference is that conda allows you to create and manage multiple isolated environments, each with its own set of packages. This can be useful if you want to have different versions of packages available for different projects. With install.packages(), all packages are installed in the same global library, which can make it more difficult to manage dependencies and avoid conflicts. | |||
<li> | |||
[https://towardsdatascience.com/a-guide-to-conda-environments-bc6180fc533 The Definitive Guide to Conda Environments], [https://docs.anaconda.com/anaconda/user-guide/tasks/using-r-language/ Using R language with Anaconda]. '''Environments''' created with ''conda create'' live by default in the '''envs/''' folder of your Conda directory, whose path will look something like ''/Users/user-name/miniconda3/envs'' or ''/Users/user-name/anaconda3/envs''. | |||
<pre> | |||
Activate conda base Create a new env Activate a new env Deactivate an env | |||
----------------------------> (base) -----------------> -------------------> (r-env) -----------------> (base) | |||
eval $(conda shell.bash hook)" conda create r-env conda activate r-env conda deactivate | |||
</pre> | |||
<pre> | |||
$ eval "$(/home/brb/anaconda3/bin/conda shell.bash hook)" | |||
(base) $ mkdir mypythonproj; cd mypythonproj # This step seems not necessary | |||
(base) $ conda create -n r-env r-base | |||
... | |||
# | |||
# To activate this environment, use | |||
# | |||
# $ conda activate r-env | |||
# | |||
# To deactivate an active environment, use | |||
# | |||
# $ conda deactivate | |||
(base) $ conda activate r-env | |||
(r-env) $ ls anaconda3/envs | |||
r-env | |||
(r-env) $ conda install r-essentials | |||
(r-env) $ which R | |||
/home/brb/anaconda3/envs/r-env/bin/R | |||
(r-env) $ ls -la # Still Empty | |||
(r-env) $ R --version | |||
R version 3.4.3 (2017-11-30) -- "Kite-Eating Tree" | |||
# Note that the current R version should be 4.0.3 | |||
(r-env) $ conda env list | |||
base /home/brb/anaconda3 | |||
r-env * /home/brb/anaconda3/envs/r-env | |||
(r-env) $ conda deactivate | |||
(base) $ | |||
</pre> | |||
It seems to be better to save the environment inside a project directory. So using '''python -m venv /path/to/new/environment''' method is preferred. You can also use '''conda create --prefix /path/to/new/environment'''. Placing environments outside of the default env/ folder comes with some drawbacks. Read the document of 'The Definitive Guide to Conda Environments'. | |||
</li> | |||
<li> [https://conda-forge.org/ conda-forge channel], [https://conda-forge.org/docs/user/introduction.html A brief introduction], https://anaconda.org/conda-forge/r-base. Following the instruction seems to mess things up though the conda-forge says the latest version is 4.0.3 (3 years late). | |||
{{Pre}} | |||
$ eval "$(/home/brb/anaconda3/bin/conda shell.bash hook)" | |||
(base) $ conda install -c conda-forge r-base | |||
... | |||
## Package Plan ## | |||
environment location: /home/brb/anaconda3 | environment location: /home/brb/anaconda3 | ||
added / updated specs: | added / updated specs: | ||
- r-base | - r-base | ||
... | ... | ||
Downloading and Extracting Packages | Downloading and Extracting Packages | ||
r-base-3.2.2 ... | r-base-3.2.2 ... | ||
(base) $ R --version | (base) $ R --version | ||
/home/brb/anaconda3/lib/R/bin/exec/R: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such file or directory | /home/brb/anaconda3/lib/R/bin/exec/R: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such file or directory | ||
(base) $ which R | (base) $ which R | ||
/home/brb/anaconda3/bin/R | /home/brb/anaconda3/bin/R | ||
</pre> | </pre> | ||
</li> | </li> | ||
</ul> | |||
== Run R with Jupyter notebook == | |||
<ul> | |||
<li>[https://marketsplash.com/tutorials/r/how-to-use-r-in-jupyter-notebooks/ How To Use R In Jupyter Notebooks: A Step-By-Step Approach] | |||
<pre> | |||
mkdir project; cd project | |||
python3 -m venv myenv | |||
source myenv/bin/activate | |||
pip3 install jupyter | |||
# Same terminal | |||
R # this is from the global environment | |||
# so the local environment for R does not work | |||
</pre> | |||
<pre> | |||
install.packages('IRkernel') # Install IRkernel from within R | |||
# Make sure build-essential has been installed before | |||
# running install.packages(). | |||
IRkernel::installspec() # Make IRkernel available to JupyterLab | |||
q() | |||
</pre> | |||
After running these commands in R, you should be able to select R as a kernel when creating a new notebook in JupyterLab. | |||
<pre> | |||
jupyter lab # automatically launch jupyter in a browser | |||
Ctrl+c # stop | |||
deactivate | |||
</pre> | |||
<li>[https://docs.anaconda.com/anaconda/navigator/tutorials/r-lang/ Using the R programming language in Jupyter Notebook] (Anaconda) | |||
<li>[https://developers.refinitiv.com/en/article-catalog/article/setup-jupyter-notebook-r Setup Jupyter Notebook for R] (Windows OS, no conda) | |||
<li>[https://datatofish.com/r-jupyter-notebook/ How to Add R to Jupyter Notebook (full steps)] using Anaconda | |||
<li>[https://www.storybench.org/install-r-jupyter-notebook/ How to install R on a Jupyter notebook] using homebrew | |||
<li>[http://www.rebeccabarter.com/blog/2017-11-17-ggplot2_tutorial/ ggplot2: Mastering the basics] & [https://github.com/rlbarter/ggplot2-thw Jupyter Notebook]. To set up the Jupyter environment, see the [[Docker#Python_Jupyter_including_R|Docker method]]. | |||
<pre> | |||
docker run --rm -p 8888:8888 \ | |||
-e JUPYTER_ENABLE_LAB=yes \ | |||
-v "$PWD":/home/jovyan \ | |||
jupyter/datascience-notebook:r-4.0.3 | |||
</pre> | |||
We first have to use "git clone https://github.com/rlbarter/ggplot2-thw.git" to download the repo and "cd ggplot2-thw". Then after opening http://IP:8888/?token=XXXXXXX we will see "ggplot2.ipynb" on the left panel. Double click the file will open it on the Notebook. | |||
</li> | |||
</ul> | |||
== Example 1: GEO2RNAseq == | |||
[[Genome#GEO2RNAseq|GEO2RNAseq]] | |||
== Example 2: p-NET == | |||
[https://github.com/marakeby/pnet_prostate_paper Biologically informed deep neural network for prostate cancer classification and discovery] and the [https://www.nature.com/articles/s41586-021-03922-4 paper] 2021. | |||
= Mamba = | |||
<ul> | |||
<li>'''Mamba''' is a high-performance package manager that is fully compatible with '''Conda''', the package management system widely used in the Python ecosystem. It was developed to provide a faster and more efficient alternative to Conda, addressing some of the performance issues, especially in terms of dependency resolution and package installation speed. | |||
<li> https://github.com/mamba-org/mamba The Fast Cross-Platform Package Manager | |||
<li> [https://hpc.nih.gov/apps/python.html#envs Biowulf]. Mambaforge: a derivative of miniconda that includes mamba and uses the conda-forge channel in place of the defaults channel | |||
<li> [https://youtu.be/yeXDyF6_VwQ How to install Mamba on Ubuntu 21.10], [https://youtu.be/q0qn8EOOP6c How to install R using Mamba], [https://youtu.be/rlTXjOXjJGE How to install RStudio on Ubuntu 21.10 with R installed using Mamba] | |||
<li> Debian 12 | |||
<ul> | |||
<li>(This step is unnecessary, miniforge includes Conda already) Install [https://docs.anaconda.com/anaconda/install/linux/ anaconda] on debian. I choose 'no' at the final question. | |||
<syntaxhighlight lang='sh'> | |||
sudo apt-get install libgl1-mesa-glx libegl1-mesa libxrandr2 libxrandr2 \ | |||
libxss1 libxcursor1 libxcomposite1 libasound2 libxi6 libxtst6 | |||
curl -O https://repo.anaconda.com/archive/Anaconda3-2024.06-1-Linux-x86_64.sh | |||
bash Anaconda3-2024.06-1-Linux-x86_64.sh | |||
# Anaconda3 will now be installed into this location: /home/$USER/anaconda3 | |||
# Do you wish to update your shell profile to automatically initialize conda? no | |||
# Log out and log in again | |||
# verify conda | |||
conda list | |||
conda activate base | |||
conda deactivate | |||
</syntaxhighlight> | |||
<li>Install [https://mamba.readthedocs.io/en/latest/installation/mamba-installation.html mamba]. Note that I choose 'yes' at the final question. | |||
<syntaxhighlight lang='sh'> | |||
curl -O https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh | |||
bash Miniforge3-$(uname)-$(uname -m).sh | |||
# press q to exit the agreement | |||
# Miniforge3 will now be installed into this location: | |||
# /home/brb/miniforge3 | |||
# To activate this environment, use: | |||
# | |||
# micromamba activate /home/brb/miniforge3 | |||
# | |||
# Or to execute a single command in this environment, use: | |||
# | |||
# micromamba run -p /home/brb/miniforge3 mycommand | |||
# | |||
# ... | |||
# You can undo this by running `conda init --reverse $SHELL`? [yes|no] | |||
# [no] >>> yes <----- IMPORTANT; o.w. mamba will not be available | |||
# You have chosen to not have conda modify your shell scripts at all. | |||
# To activate conda's base environment in your current shell session: | |||
# | |||
# eval "$(/home/brb/miniforge3/bin/conda shell.YOUR_SHELL_NAME hook)" | |||
# | |||
# To install conda's shell functions for easier access, first activate, then: | |||
# | |||
# conda init | |||
# | |||
# Thank you for installing Miniforge3! | |||
</syntaxhighlight> | |||
<li>Using mamba. Mamba commands are the same as Conda commands, so you can seamlessly switch between using the two. | |||
<syntaxhighlight lang='sh'> | |||
mamba create -n myenv python=3.6.12 | |||
mamba activate myenv | |||
mamba install numpy=1.19.2 pandas=1.1.3 | |||
pip list | |||
mamba deactivate | |||
# deactivating an environment does not delete it; it simply changes your working context. | |||
mamba env list | |||
mamba env remove -n myenv | |||
$ which mamba | |||
/home/brb/miniforge3/condabin/mamba | |||
$ which conda | |||
/home/brb/miniforge3/condabin/conda | |||
</syntaxhighlight> | |||
</ul> | </ul> | ||
</ul> | </ul> | ||
= Web framework = | = Web framework = |
Latest revision as of 11:07, 12 September 2024
Basic
Resources
- https://www.tutorialspoint.com/python/
- https://docs.python.org/3/tutorial/index.html
- https://www.learnpython.org/ (contains a Run button for each example)
- https://www.w3schools.com/python/ (contains a Run button for each example)
- Think Python (Free Ebook)
- Python教程
- Python 程式設計-李明昌 免費電子書
- Python for R users
- Python Data Science Handbook by Jake VanderPlas
- The Hitchhiker’s Guide to Python!
- Python 3 for Scientists
- Another Book on Data Science Learn R and Python in Parallel
- Fluent Python. github for source code.
- A dozen ways to learn Python from opensource.com
- Introduction to Programming (with Python) - a webinar from NIAID
- Learn Python - Full Course for Beginners by freeCodeCamp.org
- Real Python Tutorials
- Primer on Python Decorators. A decorator takes a function, extends it and returns.
- Top articles for learning Python in 2020
- Python Programming And Numerical Methods: A Guide For Engineers And Scientists
- Learn to Code Python Free With These Courses and Apps
- 20 Python Functions You Should Know
Python end of life
https://endoflife.date/python or https://devguide.python.org/. By default, the end-of-life is scheduled 5 years after the first release, but can be adjusted by the release manager of each branch.
Install, setup
Alias
How to Fix Common Python Installation Errors on macOS
nano ~/.bash_profile # or nano ~/.zshrc
Add the following line
alias python=python3
Ubuntu
How to Install the Latest Python Version on Ubuntu Linux
Mac
- Installing Python 3 on Mac OS X, How to Fix Permissions on Home-brew on MacOS High Sierra. I use soft link to point python to pythons (/usr/local/bin). I need to quit and restart iTerm2.
- How to set up virtual environments for Python on MacOS
- pip installation
# check pip version python -m pip --version # install curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py python get-pip.py # Upgrading pip python -m pip install -U pip
- On my 2018 mac, the default python3 is at "/usr/local/bin/pyhton3". In "~/Library/Python" directory, it has "2.7", "3.8" and "3.9".
- On my 2021 mac Ventura, the default python3 is at "/usr/bin". But when we try to run 'python3', it asked to install the command line developer tools. After the installation we can use python3. Still, there is no "~/Library/Python" directory.
Multiple python versions
How to manage multiple Python versions and virtual environments 2018.
conda/mamba
conda create --name myenv python=3.6.12 conda activate myenv mamba create --name myenv python=3.6.12 mamba activate myenv
pyenv
- https://github.com/pyenv/pyenv
- Positron wiki
- How to use pyenv to run multiple versions of Python on a Mac
- How to Install Multiple Python Versions on Ubuntu Using Pyenv
pyenv install 3.6.12 pyenv virtualenv 3.6.12 myenv pyenv activate myenv
virtualenv
pip install virtualenv virtualenv -p python3.6 myenv source myenv/bin/activate
venv (python 3.3+)
- For Python 3, venv is generally more commonly used because it is included in the Python standard library starting from Python 3.3, making it more convenient and straightforward to use. However, virtualenv is still popular, especially among developers who need more advanced features or compatibility with older Python versions. It offers more flexibility and can be used with both Python 2 and Python 3.
- The primary purpose of venv is to create isolated environments for managing Python packages and dependencies, but not python itself. For instance,
python3.10 -m venv myenv
. - On the other hand, virtualenv can indeed be used to control the Python version for your virtual environments. For instance,
virtualenv -p /usr/bin/python3.10 myenv
.
- The primary purpose of venv is to create isolated environments for managing Python packages and dependencies, but not python itself. For instance,
- Don’t Make This Mistake When You Start Your Python Project
- Understanding Python's Virtual Environment Landscape: venv vs. virtualenv, Wrapper Mania, and Dependency Control
- Here is another example
~/github/PUREE$ python3 -m venv myenv The virtual environment was not created successfully because ensurepip is not available. On Debian/Ubuntu systems, you need to install the python3-venv package using the following command. apt install python3.10-venv ... ~/github/PUREE$ sudo apt install python3.10-venv ~/github/PUREE$ python3 -m venv myenv ~/github/PUREE$ source myenv/bin/activate (myenv) ~/github/PUREE$ which python /home/brb/github/PUREE/myenv/bin/python (myenv) ~/github/PUREE$ pip freeze > requirements.txt (myenv) ~/github/PUREE$ deactivate ~/github/PUREE$ ~/github/PUREE$ ls myenv/bin activate activate.fish f2py f2py3.10 pip pip3.10 python3 wheel activate.csh Activate.ps1 f2py3 normalizer pip3 python python3.10
Online compiler
- Python.org. It seems this has most modules.
- Tutorialspoint
- OneCompiler
IDE
- PyCharm
- Pycharm was used by Learn Python - Full Course for Beginners (freeCodeCamp.org)
- Run a code line by line by changing the keyboard shortcuts from Settings -> Keymap -> Other.
- To run the current file, right click the tab and select Run XXX. (Frustrated)
- Thonny
- Spyder
- RStudio
- Create a file (xxx.py)
- Click the terminal tab. Type 'python' (or ipython3).
- Use Ctrl/CMD + Alt + Enter to run your python code line by line or a chunk.
Visual Studio Code
- Get started with Jupyter Notebooks in less than 4 minutes (video)
- Jupyter Notebooks in VS Code, Python in Visual Studio Code
- VS Code Python擴充套件開始不預裝Jupyter擴充套件
The ipynb file can contain figures.
This (Harmony Manuscript) has several notebook files where the code in ipynb files were written in R, not Python.
I can use vsc to open a ipynb file.
Conversion
- Rmd to ipynb
- rmd2jupyter package
- How to convert Rmd to ipynb notebook: Jupytext and notedown.
- Script of Scripts (SoS)
- ipynb to Rmd
nbdev
- nbdev
- Jupyter is now a full-fledged IDE Literate programming is now a reality through nbdev and the new visual debugger for Jupyter.
Emacs
Emacs Shell mode: how to send region to shell?
JupyterLab
- What is the difference between Jupyter Notebook and JupyterLab?
- Jupyter Notebook (classic) and JupyterLab are both web-based interactive computing environments for working with data and code, but they have some key differences in terms of their user interface, features, and capabilities.
- JupyterLab is a more modern and powerful tool than Jupyter Notebook, and is recommended for users who want a more flexible and feature-rich interface for working with data and code. However, Jupyter Notebook remains a popular and widely used tool, particularly for working with Jupyter notebooks.
- 7 Reasons Why You Should Use Jupyterlab for Data Science
Some resources
- Six easy ways to run your Jupyter Notebook in the cloud
- Cross-Methods are a Leak/Variance Trade-Off
- Journal five minutes a day with Jupyter
- How to Use Jupyter Notebook in 2020: A Beginner’s Tutorial
- Get Started With Jupyter Notebook: A Tutorial
- Jupyter Notebook Command Mode Keyboard Shortcuts
- Enter: edit mode
- Esc: command mode
- Ctrl-Enter: run cell
- Shift-Enter: run current cell, and select cell below
- Alt-Enter: run cell, insert a cell below
- Y: to code
- M: to markdown
- 1: to insert heading 1
- 2,3,4,5,6: to insert heading 2,3,4,5,6
Online tools rendering ipynb
- Github
- NBViewer (nbviewer.jupyter.org)
- Google Colaboratory (colab.research.google.com)
- Binder (mybinder.org)
- Kaggle Notebooks (kaggle.com)
- Azure Notebooks (notebooks.azure.com)
- Datalore (datalore.jetbrains.com)
- Deepnote (deepnote.com)
- CoCalc (cocalc.com)
Different installation methods
- https://pypi.org/project/jupyterlab/
- https://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html
- Guide to Install JupyterLab on Debian 12. Hint: require node.js. Node.js, is used for building and managing JupyterLab’s JavaScript dependencies. Many Jupyter extensions require having working npm (which comes with Node.js) and jlpm commands, which are required for downloading useful Jupyter extensions or other JavaScript dependencies. Node.js itself is built with GYP, a cross-platform build tool written in Python, which is another reason why Python is needed.
- How to Install JupyterLab on Rocky Linux 9
mkdir -p ~/project; cd ~/project python3 -m venv myenv # '-m venv' means to run venv module as a script source myenv/bin/activate pip3 install jupyter which jupyter jupyter --version jupyter server --generate-config jupyter server password jupyter lab --generate-config jupyter lab --show-config sudo firewall-cmd --add-port=8888/tcp jupyter lab --ip 192.168.5.120 # http://192.168.5.120:8888/
- How to Install Jupyter Notebook on Ubuntu 24.04, 22.04 or 20.04
JupyterLab Desktop
pip/pip3 Jupyter
- https://jupyter.org/
which python3 # /usr/bin/python3 pip3 install jupyterlab jupyter-lab # http://localhost:8888/lab # The current directory will be available on the file browser panel in JupyterLab.
On Mac, it shows the following when I run 'pip3 install jupyterlab'
Installing collected packages: pip WARNING: The scripts pip, pip3 and pip3.9 are installed in '/Users/XXX/Library/Python/3.9/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Successfully installed pip-23.0 WARNING: You are using pip version 21.2.4; however, version 23.0 is available. You should consider upgrading via the '/Library/Developer/CommandLineTools/usr/bin/python3 -m pip install --upgrade pip' command.
That is, I need to use /Users/XXX/Library/Python/3.9/bin/jupyter-lab to launch jupyter-lab OR add the path to ".zshrc"; see jupyterlab doc.
conda + Jupyter
- How to use Jupyter notebooks in a conda environment? Option 1 works for me.
- https://github.com/dylkot/cNMF/#analyze_simulated_example_data.ipynb
conda install --yes jupyterlab && conda clean --yes --all
IPython shell
- Why I love using the IPython shell and Jupyter notebooks
ipython # Shell jupyter notebook # auto open the browser
- Why switch to JupyterLab from jupyter-notebook?
Extract python code from Jupyter notebook
- Get only the code out of Jupyter Notebook. nbconvert or jq. Or File -> Download as -> Python (.py) — this should export all code cells as single .py file.
- Separating Code from Presentation in Jupyter Notebooks
Execute Javascript in a Jupyter Notebook
How to Execute Javascript in a Jupyter Notebook on Linux
Google colab
R programming
Setup Jupyter Notebook for R or Using R on Jupyter Notebook
To use R in JupyterLab, you will first need to install the IRkernel package in your R environment using the following command:
install.packages('IRkernel')
Once you have installed the IRkernel package, you can register it with JupyterLab using the following command in your R console:
IRkernel::installspec()
After you have registered the kernel, you can start a new Jupyter notebook or JupyterLab session and select the "R" kernel from the kernel dropdown menu. This will allow you to run R code in JupyterLab, including data analysis, visualization, and other tasks.
Xeus-R: a future-proof Jupyter kernel for R
Meet Xeus-R: a future-proof Jupyter kernel for R
Run Jupyter Notebooks on an Apple M1 Mac
- How to Run Jupyter Notebooks on an Apple M1 Mac
- How to Easily Set Up Python on Any M1 Mac
- Installing Jupyter in Macbook Air M1
Cheat sheet
- http://datasciencefree.com/python.pdf
- https://s3.amazonaws.com/assets.datacamp.com/blog_assets/PythonForDataScience.pdf
The Most Frequently Asked Questions About Python Programming
https://www.makeuseof.com/tag/python-programming-faq/
Running
Interactively
Use Ctrl+d to quit.
How to run a python script file
python mypython.py
Run python statements from a command line
Use -c (command) option.
python -c "import psutil"
run python source code line by line
run python source code line by line
python -m pdb <script.py>
Install a new module
- See an example of installing HTSeq.
Module != Package
- A Python module is a single file containing Python code, while a package is a collection of modules organized in a specific way.
- A module has a filename with the suffix .py added.
PyPI/Python Package Index
- https://en.wikipedia.org/wiki/Python_Package_Index
- The Python Package Index (PyPI) is the definitive list of packages (or modules)
- How to install pip to manage PyPI packages easily
- Introducing the guide to 7 essential PyPI libraries and how to use them
pip
pip, use PyPI as the default source for packages and their dependencies.
As an example, motionEye can be installed by pip install or pip2 install; see its wiki and source code on Github.
sudo apt-get install python-pip pip --version pip install SomePackage pip show --files SomePackage pip install --upgrade SomePackage pip install --upgrade https://github.com/Lasagne/Lasagne/archive/master.zip pip install ‐‐upgrade pip # Upgrade itself pip uninstall SomePackage sudo apt install python3-pip pip3 --version
Upgrade packages
How to Upgrade Python Packages with Pip
requirements.txt
- The Python Requirements File and How to Create it
pip freeze > requirements.txt
- An example
tensorflow==2.3.1 uvicorn==0.12.2 fastapi==0.63.0
We can use pip freeze or pip list to verify available packages in an environment.
List installed packages and their versions, location/directory
pip3 list -v
On my Ubuntu 20.04, the packages installed by pip3 is located in ~/.local/lib/python3.8/site-packages/. It does not matter where I issued the pip3 install command.
The danger of upgrading pip
- Error after upgrading pip: cannot import name 'main'
- You should consider upgrading via the 'pip install --upgrade pip' command
Don't use sudo + pip
https://askubuntu.com/questions/802544/is-sudo-pip-install-still-a-broken-practice
"--user" option in pip
- It is not recommended to use sudo before calling pip on Linux (actually can we?). This is because using sudo can cause permissions issues and can potentially damage your system12. Instead, you can install packages locally using the --user flag.
- Upgrade python packages with pip: use "sudo" or "--user"?
- If you need to install packages system-wide, you can use virtual environments instead of sudo. Virtual environments allow you to create isolated Python environments that do not interfere with the system Python installation
- Permission denied: '/usr/local/lib/python2.7/dist-packages/pip'
$ pip install Pygments ... OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/Pygments-2.2.0.dist-info' /usr/local/lib/python2.7/dist-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning $ pip install --user Pygments Collecting Pygments Using cached Pygments-2.2.0-py2.py3-none-any.whl Installing collected packages: Pygments Successfully installed Pygments-2.2.0
pip -t option
We can force to install a package in user's directory (i.e. a package is already installed in the global directory /usr/lib/python3/dist-packages but some applications cannot find it). Pip install python package into a specific directory other than the default install location
pip3 install -t ~/.local/bin/python3.10/site-packages pytz
virtualenv
Python “Virtual Environments” allows us to install a Python package in an isolated location, rather than installing it globally.
- How To Manage Python Packages Using Pip.
First Create a new project folder and cd to the project folder in your terminal.
# Python 2 $ sudo pip install virtualenv $ virtualenv <DIR_NAME> $ source <DIR_NAME>/bin/activate (<DIR_NAME>) ~$ which python .... $ deactivate # For Python 3, https://docs.python.org/3/tutorial/venv.html it is more common to use venv instead $ python3 -m venv <DIR_NAME> # DIR_NAME is also called an environment $ source <DIR_NAME>/bin/activate (<DIR_NAME>) ~$ which python .... $ deactivate
-
Python Tutorial: virtualenv and why you should use virtual environments. pip freeze.
pip list pip freeze --local > requirements.txt ... pip install -r requirements.txt pip list
- Learn Python by creating a video game
- How to use Python virtualenv
- A non-magical introduction to Pip and Virtualenv for Python beginners
- Alternative to virtualenv we need to add "--user" to the pip command. See the installation guide of lasagne or easy_install or pip as a limited user?
pipenv
- Why Use Pipenv to Create a Python Environment?
- Pipenv is a Python virtualenv management tool that supports a multitude of systems and nicely bridges the gaps between pip, pyenv and virtualenv. It automatically creates and manages a virtualenv for your projects, as well as adds/removes packages from your Pipfile as you install/uninstall packages. It also generates the ever-important Pipfile.lock, which is used to produce deterministic builds.
- https://pipenv.pypa.io/en/latest/pipfile/
Poetry
- https://python-poetry.org/
- Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you. Poetry offers a poetry.lock lockfile to ensure repeatable installs, and can build your project for distribution.
pipx (alternative to pip3)
- Pipx is a tool that helps you install and run end-user applications written in Python. It is similar to macOS’s brew, JavaScript’s npx, and Linux’s apt.
- Pipx is focused on installing and managing Python packages that can be run from the command line directly as applications.
- pipx is made specifically for application installation, as it adds isolation yet still makes the apps available in your shell: pipx creates an isolated environment for each application and its associated packages.
python3 -m pip install --user pipx python3 -m pipx ensurepath # OR sudo apt install pipx # Or pip3 install pipx pipx install <package_name> # no sudo needed pipx list pipx uninstall <package_name>
- I need to find an alternative to pip utility because of a problem when I used pip command. Error: externally-managed-environment
- 3 Ways to Solve Pip Install Error on Ubuntu 23.04
- See the example of installing Asciinema & agg
- I can see "asciinema" was installed under ~/.local/bin directory that's because "pipx ensurepath" adds ~/.local/bin to ~/.bashrc file.
- Official website [1] - Install and Run Python Applications in Isolated Environments, github
- pipx vs pip
- pipx is made specifically for application installation and adds isolation yet still makes the apps available in your shell. pipx creates an isolated environment for each application and its associated packages. On the other hand, pip is a general-purpose package installer for both libraries and apps with no environment isolation.
- If you want to install an application that has dependencies that conflict with other applications or libraries on your system, you can use pipx to create an isolated environment for that application and its dependencies. This way, you can avoid conflicts between different versions of the same package.
- On my Ubuntu, pip installs packages to /usr/local/lib/python3.8/dist-packages/.
- Pipx – Install And Run Python Applications In Isolated Environments
- Run Python applications in virtual environments
python setup.py
If a package has been bundled by its creator using the standard approach to bundling modules (with Python’s distutils tool), all you need to do is download the package, uncompress it and type:
python setup.py build sudo python setup.py install
For Python 2, the packages are installed under /usr/local/lib/python2.7/dist-packages/.
$ ls -l /usr/local/lib/python2.7/dist-packages/ total 12 -rw-r--r-- 1 root staff 273 Jan 12 13:45 easy-install.pth drwxr-sr-x 4 root staff 4096 Jan 12 13:45 HTSeq-0.6.1p1-py2.7-linux-x86_64.egg drwxr-sr-x 4 root staff 4096 Jan 12 13:42 pysam-0.9.1.4-py2.7-linux-x86_64.egg
python setup.py bdist_wheel
- Why does python setup.py bdist_wheel creates a build folder?
- What Are Python Wheels and Why Should You Care? The purpose of creating a wheel file in Python is to package and distribute your code in a way that makes it easier for others to install and use your code.
- Creating Built Distributions
- Wheels
- In Python programming, “wheel” refers to a built-package format for Python. It is designed to contain all the files for a PEP 376 compatible install in a way that is very close to the on-disk format.
- What is a Python wheel?
- To create a wheel file in Python.
- First, make sure you have the wheel package installed: pip install wheel
- Navigate to the directory containing your package and run the following command: python setup.py bdist_wheel
- This will create a .whl file in the dist directory of your package. (people can use pip to install the project)
Get a list of installed modules
http://stackoverflow.com/questions/739993/how-can-i-get-a-list-of-locally-installed-python-modules
pydoc modules
Not helpful. See the pip list command.
Check installed packages' versions
If you install packages through pip, use
$ pip list ... pyOpenSSL (0.13.1) pyparsing (2.0.1) pysam (0.10.0) python-dateutil (1.5) pytz (2013.7) rudix (2016.12.13) scipy (0.13.0b1) setuptools (1.1.6) singledispatch (3.4.0.3) six (1.4.1) tornado (4.4.2) vboxapi (1.0) xattr (0.6.4) zope.interface (4.1.1)
And more information about a package by using pip show PACKAGE.
$ pip show pysam Name: pysam Version: 0.10.0 Summary: pysam Home-page: https://github.com/pysam-developers/pysam Author: Andreas Heger Author-email: [email protected] License: MIT Location: /Users/XXX/Library/Python/2.7/lib/python/site-packages Requires:
The following method works whether the package is installed by source or binary package
>>> import pysam >>> print(pysam.__version__) 0.10.0 >>> print pysam.__version__ 0.10.0
See http://hammelllab.labsites.cshl.edu/tetoolkit-faq/
Install a specific version of package through pip
https://stackoverflow.com/questions/5226311/installing-specific-package-versions-with-pip
For example, pysam package was actively released. But the new release (0.11.2.2) may introduce some bugs. So I have to install an older version (0.10.0 works for me on Mac El Capitan and Sierra).
$ sudo -H pip uninstall pysam Uninstalling pysam-0.11.2.2: ...... $ sudo -H pip install pysam==0.10.0 Collecting pysam==0.10.0 Downloading pysam-0.10.0.tar.gz (2.3MB) 100% |████████████████████████████████| 2.3MB 418kB/s Installing collected packages: pysam Running setup.py install for pysam ... done Successfully installed pysam-0.10.0
warning: Please check the permissions and owner of that directory
I got this message when I use root to run the 'sudo pip install PACKAGE' command.
See
- http://stackoverflow.com/questions/27870003/pip-install-please-check-the-permissions-and-owner-of-that-directory
- http://askubuntu.com/questions/578869/python-pip-permissions
python3-pip installed but pip3 command not found?
sudo apt-get remove python3-pip; sudo apt-get install python3-pip
DeepSurv example
https://github.com/jaredleekatzman/DeepSurv
git clone https://github.com/jaredleekatzman/DeepSurv.git sudo cp /usr/bin/pip /usr/bin/pip.bak sudo nano /usr/bin/pip # See https://stackoverflow.com/a/50187211 more detail # Method 1 for Theano sudo pip install theano # Method 2 for Theano pip install --user --upgrade https://github.com/Theano/Theano/archive/master.zip pip install --user --upgrade https://github.com/Lasagne/Lasagne/archive/master.zip cd DeepSurv/ pip install . --user sudo apt install python-pytest pip install h5py --user sudo pip uninstall protobuf # https://stackoverflow.com/a/33623372 pip install protobuf --user sudo apt install python-tk py.test ============ test session starts =========== platform linux2 -- Python 2.7.12, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 rootdir: /home/brb/github/DeepSurv, inifile: collected 7 items tests/test_deepsurv.py ....... ========== 7 passed in 5.77 seconds ========
How to list all installed modules
help('modules') # the output is not pretty
Comment
- Use the comment symbol # for a single line
- Use a delimiter “”” on each end of the comment. Attention: Don't use triple-quotes
Python Comments from zentut.com.
Docstring
- https://en.wikipedia.org/wiki/Docstring
- Python Developer's Guide Docstring Conventions
Try / Except
try: number = int(input("Enter a number: ")) print(number) except: print("Invalid Input")
if __name__ == "__main__":
How to Get the Current Directory in Python
How to Get the Current Directory in Python
Import a compiled C module
- An example based on SWIG compiler.
string and string operators
Reference:
- Python for Genomic Data Science from coursera.
- Python Hello World and String Manipulation
- Use double quote instead of single quote to define a string
- Use triple double quotes """ to write a long string spanning multiple lines or comments in a python script
- if dna="gatagc", then
dna[0]='g' dna[-1]='c' (start counting from the right) dna[-2]='g' dna[0:3]='gat' (the end always excluded) dna[:3]='gat' dna[2:]='tgc' len(dna)=6 type(dna) print(dna) dna.count('c') dna.upper() dna.find('ag')=3 (only the first occurrence of 'ag' is reported) dna.find('17', 2) (start looking from pos 17) dna.rfind('ag') ( search backwards in string) dna.islower() (True) dna.isupper() (False) dna.replace('a', 'A') print(dna.upper().isupper())
Format
Format Specification Mini-Language
Regular expression
The Beginner’s Guide to Regular Expressions With Python
User's input
dna=raw_input("Enter a DNA sequence: ") # python 2 dna=input("Enter a DNA sequence: ") # python 3
To convert a user's input (a string) to others
int(x, [, base]) flaot(x) str(x) #converts x to a string str(65) # '65' chr(x) # converts an integer to a character chr(65) # 'A'
Why is parenthesis in print voluntary in Python 2.7?
Fancy Output
print("THE DNA's GC content is ", gc, "%") # gives too many digits following the dot print("THE DNA's GC content is %5.3f %%" % " % gc) # the percent operator separating the formatting string and the value to # replace the format placeholder print("%d" % 10.6) # 10 print("%e" % 10.6) # 10.060000e+01 print("%s" % dna) # gatagc
Type
Built-in Functions, How to check type of variable (object) in Python
type(object)
List
A list is an ordered set of values
gene_expr=['gene', 5.16e-08, 0.001385, 7.33e-08] print(gene_expr[2] gene_expr[0]='Lif'
Slice a list (it will create a new list)
gene_expr[-3:] # [5.16e-08, 0.001385, 7.33e-08] gene_expr[1:3] = [6.09e-07]
Clear the list
gene_expr[]=[]
List functions
Size of the list
len(gene_expr)
Delete an element
del gene_expr[1]
Extend/append to a list
gene_expr).extend([5.16e-08, 0.00123])
Count the number of times an element appears in a list
print(gene_expr.count('Lif'), gene_expr.count('gene'))
Reverse all elements in a list
gene_expr.reverse() print(gene_expr) help(list)
Lists as Stacks
stack=['a', 'b', 'c', 'd'] stack.append('e')
Sorting lists
mylist=[3, 31, 123, 1, 5] sorted(mylist) mylist # not changed mylist.sort() mylist=['c', 'g', 'T', 'a', 'A'] mylist.sort()
Don't change an element in a string!
motif = 'nacggggtc' motif[0] = 'a' # ERROR
Tuples
A tuple consists of a number of values separated by commas, and is another standard sequence data type, like strings and lists.
t=1,2,3 t t=(1,2,3) # we may input tuples with or without surrounding parentheses
Sets
A set is an unordered collection with no duplicate elements.
brca1={'DNA repair', 'zine ion binding'} brca2={protein binding', 'H4 histone'} brca1 | brca2 brca1 & brca2 brca1 - brca2
Dictionaries
A dictionary is an unordered set of key and value pairs, with the requirement that the keys are unique (within on dictionary).
TF_motif = {'SP1' : 'gggcgg', 'C/EBP' : 'attgcgcaat', 'ATF' : 'tgacgtca', 'c-Myc' : 'cacgtg', 'Oct-1' : 'atgcaaat'} # Access print("The recognition sequence for the ATF transcription is %s." % TF_motif['ATF']) # Update TF_motif['AP-1'] = 'tgagtca' # Delete del TF_motif['SP1'] # Size of a list len(TF_motif) # Get a list of all the 'keys' in a dictionary list(TF_motif.keys()) # Get a list of all the 'values' list(TF_motif.values()) # sort sorted(TF_motif.keys()) sorted(TF_motif.values())
We can retrieve data from dictionaries using the items() method.
for name,seq in seqs.item(): print(name, seq)
In summary, strings, lists and dictionaries are most useful data types for bioinformatics.
if statement
dna=input('Enter DNA sequence: ') if 'n' in dna : nbases=dna.count('n') print("dna sequence has %d undefined bases " % nbases) if condtion 1: do action 1 elif condition 2: do action 2 else: do action 3
Logical operators
Use and, or, not.
dna=input('Enter DNA sequence: ') if 'n' in dna or 'N' in dna: nbases=dna.count('n')+dna.count('N') print("dna sequence has %d undefined bases " % nbases) else: print("dna sequence has no undefined bases)
Loops
while
dna=input('Enter DNA sequence:') pos=dna.find('gt', 0) while pos>-1 : print("Donar splice site candidate at position %d" %pos) pos=dna.find('gt', pos+1)
for
motifs=["attccgt", "aggggggttttttcg", "gtagc"] for m in motifs: print(m, len(m))
range
for i in range(4): print(i) for i in range(1,10,2): print(i)
Problem: find all characters in a given protein sequence are valid amino acids.
protein='SDVIHRYKUUPAKSHGWYVCJRSRFTWMVWWRFRSCRA' for i in range(len(protein)): if protein[i] not in 'ABCDEFGHIKLMNPQRSTVWXYZ': print("this is not a valid protein sequence!") break
continue
protein='SDVIHRYKUUPAKSHGWYVCJRSRFTWMVWWRFRSCRA' corrected_protein='' for i in range(len(protein)): if protein[i] not in 'ABCDEFGHIKLMNPQRSTVWXYZ': continue corrected_protein=corrected_protein+protein[i] print("COrrected protein seq is %s" % corrected_protein)
else Statement used with loops
- If used with a for loop, the else statement is executed when the loop has exhausted iterating the list
- If used with a while loop, the else statement is executed when the condition becomes false
# Find all prime numbers smaller than a given integer N=10 for y in range(2, N): for x in range(2, y): if y %x == 0: print(y, 'equals', x, '*', y//x) break else: // loop fell through without finding a factor print(y, 'is a prime number')
The pass statement is a placeholder
if motif not in dna: pass else: some_function_here()
Functions
Get modular with Python function
def function_name(arguments) : function_code_block return output
For example,
def gc(dna) : "this function computes the gc perc of a dna seq" nbases=dna.count('n')+dna.count('n') gcpercent=float(dna.count('c')+dna.count('C')+dna.count('g) +dna.count('G'))*100.0/(len(dna)-nbases) return gcpercent gc('AAAAGTNNAGTCC') help(gc)
SyntaxError: invalid syntax
https://stackoverflow.com/a/11890194
On the Python shell add an empty line at the end of function definition. Eg
>>> def fun(a): ... return a+1 ... >>> fun(9) 10 >>> exit()
On a python script
def fun(a): return a+1 print fun(9)
Debug functions
https://stackoverflow.com/a/4929267
You can launch a Python program through pdb by using pdb myscript.py or python -m pdb myscript.py
$ cat debug.py def fun(a): a= a*2 a= a*3 return a+1 print fun(5) $ python -m pdb debug.py > /home/pi/Downloads/debug.py(1)<module>() -> def fun(a): (Pdb) b fun Breakpoint 1 at /home/pi/Downloads/debug.py:1 (Pdb) c > /home/pi/Downloads/debug.py(2)fun() -> a= a*2 (Pdb) n > /home/pi/Downloads/debug.py(3)fun() -> a= a*3 (Pdb) > /home/pi/Downloads/debug.py(4)fun() -> return a+1 (Pdb) p a 30 (Pdb) n --Return-- > /home/pi/Downloads/debug.py(4)fun()->31 -> return a+1 (Pdb) exit
Boolean functions
Problem: checks if a given dna seq contains an in-frame stop condon
dna=input("Enter a dna seq: ") if (has_stop_codon(dna)) : print("input seq has an in frame stop codon.") else : print("input seq has no in frame stop codon.") def has_stop_codon(dna) : "This function checks if given dna seq has in frame stop codons." stop_codon_found=False stop_codons=['tga', 'tag', 'taa'] for i in range(0, len(dna), 3) : codon=dna[i:i+3].lower() if codon in stop_codons : stop_codon_found=True break return stop_codon_found
Function default parameter values
Suppose the has_stop_codon function also accepts a frame argument (equal to 0, 1, or 2) which specifies in what frame we want to look for stop codons.
def has_stop_codon(dna, frame=0) : "This function checks if given dna seq has in frame stop codons." stop_codon_found=False stop_codons=['tga', 'tag', 'taa'] for i in range(frame, len(dna), 3) : codon=dna[i:i+3].lower() if codon in stop_codons : stop_codon_found=True break return stop_codon_found dna="atgagcggccggct" has_stop_codon(dna) # False has_stop_codon(dna, 0) # False has_stop_codon(dna, 1) # True has_stop_codon(frame=0, dna=dna)
More examples
Reverse complement of a dna sequence
def reversecomplement(seq): """Return the reverse complement of the dna string.""" seq = reverse_string(seq) seq = complement(seq) return seq reversecomplement('CCGGAAGAGCTTACTTAG')
To reverse a string
def reverse_string(seq): return seq[::-1] reverse_string(dna)
Complement a DNA Sequence
def complement(dna): """Return the complementary sequence string.""" basecomplement = {'A':'T', 'C':'G', 'G':'C', 'T':'A', 'N':'N', 'a':t', 'c':'g', 'g':'c', 't':'a', 'n':'n'} # dictionary letters = list(dna) # list comprehensions letters = [basecomplement[base] for base in letters] return ''.join(letters)
Split and Join functions
sentence="enzymes and other proteins come in many shapes" sentence.split() # split on all whitespaces sentence.split('and') # use 'and' as the separator '-'.join(['enzymes', 'and', 'other', 'proteins', 'come', 'in', 'many', 'shapes'])
Variable number of function arguments
def newfunction(fi, se, th, *rest): print("First: %s" % fi) print("Second: %s" % se) print("Third: %s" % th) print("Rest... %s" % rest) return
Modules and packages
- Python Modules
- Python Modules from w3schools
- Python import: Advanced Techniques and Tips
- Python Module Index
Packages group multiple modules under on name, by using "dotted module names". For example, the module name A.B designates a submodule named B in a package named A. See What's the difference between a Python module and a Python package?
<dnautil.py>
#!/usr/bin/python """ dnautil module contains a few useful functions for dna seq """ def gc(dna) : blah blah return gcpercent
When a module is imported, Python first searches for a built-in module with that name.
If built-in module is not found, Python then searches for a file obtained by adding the extension .py to the name of the module that it's imported:
- in your current directory,
- the directory where Python has been installed
- in a path, i.e., a colon(':') separated list of file paths, stored in the environment variable PYTHONPATH.
You can use the sys.path variable from the sys built-in module to check the list of all directories where Python look for files
import sys sys.path
If the sys.path variable does not contains the directory where you put your module you can extend it:
sys.path.append("/home/$USER/python") sys.path
Using modules (from PACKAGE/DIRNAME/FILENAME import CLASS)
from math import * print(floor(3.7)) import dnautil dna="atgagggctaggt" gc(dna) # gc is not defined dnautil.gc(dna) # Good
Import Names from a Module
from dnautil import * gc(dna) # OK from dnautil import gc, has_stop_codon
Get modular with Python functions & Learn object-oriented programming with Python from opensource.com.
from...import vs import vs import...as
- Difference between 'import' and 'from...import' in Python
- Import, From and As Keywords in Python
- `from … import` vs `import .`
- Difference between import and from in Python.
Python's import loads a Python module into its own namespace, so that you have to add the module name followed by a dot in front of references to any names from the imported module that you refer to:
import feathers duster = feathers.ostrich("South Africa")
from loads a Python module into the current namespace, so that you can refer to it without the need to mention the module name again:
from feathers import * duster = ostrich("South Africa")
- Question: Why are both import and from provided? Can't I always use from? Answer: If you were to load a lot of modules using from, you would find sooner or later that there was a conflict of names; from is fine for a small program but if it was used throughout a big program, you would hit problems from time to time
- Question: Should I always use import then? Answer: No ... use import most of the time, but use from is you want to refer to the members of a module many, many times in the calling code; that way, you save yourself having to write "feather." (in our example) time after time, but yet you don't end up with a cluttered namespace. You could describe this approach as being the best of both worlds.
- from … import OR import … as for modules
- Some examples:
from numpy import array # Run file; load specific 'attribute' arr = array([1,2,3]) # Use name directly: no need to qualify print(arr) # print [1 2 3] from math import pi pi # 3.141592653589793 math.pi # NameError: name 'math' is not defined
VS
import numpy # Run file; load module as a whole arr = numpy.array([1,2,3]) # Use its attribute names: '.' to qualify print(arr) # print [1 2 3] import math math.pi # 3.141592653589793 dir(math)
VS
import numpy as np dir(np) import math as m m.pi # 3.141592653589793
- scRNA_cell_deconv_benchmark example.
help
from AAA import BBB help(BBB) help(BBB.FunctionName) import BBB as CCC help(CCC)
Packages & __init__.py
Each package in Python is a directory which MUST contain a special file __init__.py. This file can be empty and it indicates that the directory it contains is a Python package, so it can be imported the same way a module can be imported. https://docs.python.org/2/tutorial/modules.html
Example: suppose you have several modules dnautil.py, rnautil.py , and proteinutil.py. You want to group them in a package called "bioseq" which processes all types of biological sequences. The structure of the package:
bioseq/ __init__.py dnautil.py rnautil.py proteinutil.py fasta/ __init__.py fastautil.py fastq/ __init__.py fastqutil.py
Loading from packages:
import bioseq.dnautil bioseq.dnautil.gc(dna) from bioseq import dnautil dnautil.gc(dna) from bioseq.fasta.fastautil import fastqseqread
Example
Building a Multiple Choice Quiz by freeCodeCamp.org
QuestionFile.py
class Question: def __init__(self, prompt, answer): self.prompt = prompt self.answer = answer
app.py
from QuestionFile import Question question_prompts = [ "What color are apples?\n(a) Red/Green\n(b) Purple\n(c) Orange\n\n", "What color are Bananas?\n(a) Teal\n(b) Magenta\n(c) Yellow\n\n", "What color are strawberries?\n(a) Yellow\n(b) Red\n(c) Blue\n\n" ] questions = [ Question(question_prompts[0], "a"), Question(question_prompts[1], "c"), Question(question_prompts[2], "b") ] def run_test(question): score = 0 for question in questions: answer = input(question.prompt) if answer == question.answer: score += 1 print("You got " + str(score) + " /" + str(len(questions))+ " correct") run_test(questions)
Run the program by python3 app.py
Files - Communicate with the outside
f=open('myfile', 'r') # read f=open('myfile') f=open('myfile', 'w') # write f=open('myfile', 'a') # append
Take care if a file does not exists
try: f = open('myfile') except IOError: print("the file myfile does not exist!!")
Reading
for line in f: print(line)
Change positions within a file object
f.seek(0) # go to the beginning of the file f.read()
Read a single line
f.seek(0) f.readline()
Write into a file
f=open("/home/$USER/myfile, 'a) f.write("this is a new line") f.close() >>> with open("file.txt", "w") as f: ... f.write(str(object)) ...
Importing large tab-delimited .txt file into Python
# R write.table(iris[1:10,], file="iris.txt", sep="\t", quote=F, row.names=F) # Python import csv with open('iris.txt') as f: reader = csv.reader(f, delimiter="\t") d = list(reader) print(d[0][2]) print(d[1][2]) # Shell $ python test_csv.py Petal.Length 1.4
If the data are all numerical, we can use the numpy package.
# R write.table(iris[1:10, 1:4], file="~/Downloads/iris2.txt", sep="\t", quote=F, row.names=F, col.names=F) # Python import numpy as np d = np.loadtxt('iris2.txt', delimiter="\t") print(d[0][2]) print(d[1][2]) # Shell $ python test_csv2.py 1.4 1.4
Read text file from a URL
import urllib.request url = "http://textfiles.com/adventure/aencounter.txt" file = urllib.request.urlopen(url) for line in file: print(line.decode('utf-8'))
- urllib.request — extensible library for opening URLs
- Python Internet Access using Urllib.Request and urlopen()
Command line arguments
Suppose we run 'python processfasta.py myfile.fa' in the command line, then
import sys print(sys.argv) # ['processfasta.py', 'myfile.fa']
More completely
#!/usr/bin/python """ processfasta.py builds a dictionary with all sequences from a FASTA file. """ import sys filename=sys.argv[1] try: f = open(filename) except IOError: print("File %s does not exist!" % filename)
Parsing command line arguments with getopt. Suppose we want to store in the dictionary the sequences bigger than a given length provided in the command line: 'processfasta.py -l 250 myfile.fa'
#!/usr/bin/python import sys import getopt def usage(): print """ processfasta.py: reads a FASTA file and builds a dictionary with all sequence bigger than a given length processfasta.py [-h] [-l <length>] <filename> -h print this message -l <length> filter all sequences with a length smaller than <length> (default <length>=0) <filename> the file has to be in FASTA format o, a = getopt.getopt(sys.argv[1:], '1:h') opts = {} # empty dictionary seqlen=0; for k,v in o: opts[k] = v if 'h' in opts.keys(): # he means the user wants help usage(); sys.exit() if len(a) < 1: usage(); sys.exit("input fasta file is missing") if 'l' in opts.keys(): if opts['l'] <0 : print("length of seq should be positive!"); sys.exit(0); seqlen=opts['l']
stdin and stdout
sys.stdin.read() sys.stdout.write("Some useful ouput.\n") sys.stderr.write("Warning: input file was not found\n")
Call external programs
import subprocess subprocess.call('["ls", "-l"]) # return code indicates the success or failure of the execution subprocess.call('["tophat", "genome_mouse_idx", "PE_reads_1.fq.gz", "PE_reads_2.fq.gz"])
Exceptions
5 Python Examples to Handle Exceptions using try, except and finally
Debugging
Biopython & Pubmed
- Parsers for various bioinformatics file formats (FASTA, Genbank)
- Access to online services like NCBI Entrez or Pubmed databases
- Interfaces to common bioinformatics programs such as BLAST, Clustalw and others.
import Bio print(Bio.__version__)
Running BLAST over the internet
from Bio.Blast import NCBIWWW fasta_string = open("myseq.fa").read() result_handle = NCBIWWW.qblast("blastn":, "nt", fasta_string) # blastn is the program to use # nt is the database to search against # default output is xml help(NCBIWWW.qblast)
The BLAST record
from Bio.Blast import NCBIXML blast_record = NCBIXML.read(result_handle)
Parse BLAST output
len(blast_record.alignments) E_VALUE_THRESH = 0.01 for alignment in blas_record.alignments: for hsp in alignment.hsps: if hsp.expect < E_VALUE_THRESH: print('***Alignment***') print('sequence:', alignment.title) print('length:', alignment.length) print('e value:', hsp.expect) print(hsp.query) print(hsp.match) print(hsp.sbjct)
More help with Biopython
- Biopython tutorial and cookbook: http://biopython.org/DIST/docs/tutorial/Tutorial.html
- Biopython FAQ: http://biopython.org/DIST/docs/tutorial/Tutorial.html#htoc5
pubmed_parser
Parser for Pubmed Open-Access XML Subset and MEDLINE XML Dataset
pyTest
pyc file
What is the difference between .py and .pyc files? [duplicate]. I observe it can cause a problem when I want to modify a python file but it keeps using the old pyc file so my change is not used (Raspbery Pi e-ink example).
Shutdown or restart OS
Below is tested on Raspbian
import os os.system('sudo shutdown -h now')
Popular python libraries
20 Python libraries you can’t live without
psutil
- psutil.cpu_percent() examples. Inspired by the e-ink example from Raspberry Pi.
- https://github.com/arvydas/blinkstick-python/wiki/Example%3A-Display-CPU-usage
- https://www.liaoxuefeng.com/wiki/1016959663602400/1183565811281984
# pip install psutil --user for x in range(10): psutil.cpu_percent(interval=1)
numpy
- An introduction to Numpy and Scipy
- https://docs.scipy.org/doc/numpy-dev/user/quickstart.html
- Cheat sheets
- Program to find the Sum of each Row and each Column of a Matrix
pandas
30 pandas Commands for Manipulating DataFrames
Write a pandas dataframe to a text file using to.csv(). https://stackoverflow.com/a/41514539
a.to_csv('xgboost.txt', header=True, index=True, sep='\t')
scipy
seaborn
- https://seaborn.pydata.org/
- Examples of performaing Explorator Data Analysis for few public clinical data sets
- ChatGPT GPT-4
matplotlib
Installation.
python -m pip install -U pip python -m pip install -U matplotlib # https://stackoverflow.com/a/50328517 sudo apt-get install python3.5-tk
Example.
from sklearn import datasets iris = datasets.load_iris() import matplotlib.pyplot as plt iris = iris.data # Scatterplot plt.scatter(iris[:,1], iris[:,2]) plt.show() # Boxplot plot.boxplot(iris[:,1]) plt.show() # Histogram plt.hist(iris[:,1]) plt.show()
scikit-learn
scikit-learn: Machine Learning in Python
Installation.
pip install -U scikit-learn
Example.
$ python >>> from sklearn import datasets >>> iris = datasets.load_iris() >>> digits = datasets.load_digits()
PyTorch
- https://pytorch.org/
- PyTorch Lightning - The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate.
feedparser
Never miss a Magazine article — build your own RSS notification system
Boto
A Python interface to Amazon Web Services
- http://docs.pythonboto.org/en/latest/
- https://hpc.nih.gov/training/handouts/object_storage_class_2018_oct.pdf
PIL, Pillow
- Installation
sudo apt install python-imaging
- How I can load a font file with PIL.ImageFont.truetype without specifying the absolute path?
plotnine
Python and R – Part 2: Visualizing Data with Plotnine
nltk: Natural Language Toolkit
pygame
Learn Python by creating a video game
scanpy
- scanpy and the installation instruction
- mnnpy
Trouble shooting
ImportError: cannot import name main when running pip
https://stackoverflow.com/a/50187211
Error: externally-managed-environment
See pipx
TypeError: ‘module’ object is not callable
I was trying to run "bbknn.py" from here.
Solve “TypeError: ‘module’ object is not callable” in Python, TypeError: 'module' object is not callable
The problem is I have a file called "bbknn.py" and I have "import bbknn" in the code. It will confuse python. The solution is to rename my script file "bbknn.py" (avoid MODULE.py) to other name like "bbknnDemo.py".
Illegal instruction
I got this error after I called python3 -c 'import scanpy'. Python on Biowulf.
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh TMPDIR=/tmp bash Miniconda3-latest-Linux-x86_64.sh -p ~/conda -b source ~/conda/etc/profile.d/conda.sh # ~/conda/condabin is added to PATH conda activate base python -V # Python 3.9.4 conda create -n project1 pandas numpy scipy -y conda activate project1 pip3 install scanpy bbknn ls ~/conda/envs/project1/lib/python3.9/site-packages # bbknn and scanpy are there python3 -c 'import scanpy' # Illegal instruction conda info --env conda deactivate conda remove --all -n project1 -y conda deactivate
No matching distribution found for XXX
Got an error No matching distribution found for lasagne==0.2.dev1 when I ran 'pip install .' on DeepSurv.
https://github.com/imatge-upc/saliency-salgan-2017/issues/29
Python AttributeError: 'module' object has no attribute 'SSL_ST_INIT'
See https://stackoverflow.com/a/52398193. I got this message after I ran sudo pip install --upgrade cryptography and pip show cryptography. The reason I try to upgrade cryptography is the following message
$ pip show protobuf /home/brb/.local/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown. warnings.warn(warning, RequestsDependencyWarning) Name: protobuf ...
And OpenSSL & pyOpenSSL-0.15.1.egg-inf are under /usr/lib/python2.7/dist-packages directory on my Ubuntu 16.04.
Note the following solutions do not work
$ sudo pip uninstall pyopenssl $ sudo pip install pyOpenSSL==16.2.0
I always get an error message
... File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 118, in <module> SSL_ST_INIT = _lib.SSL_ST_INIT AttributeError: 'module' object has no attribute 'SSL_ST_INIT'
And a quick solution is to do sudo rm -r /usr/local/lib/python2.7/dist-packages/OpenSSL. I also did sudo pip install pyopenssl but I did not follow this answer (sudo apt install --reinstall python-openssl).
/usr/bin/env: ‘python’: No such file or directory
On Ubuntu 20.04,
sudo apt-get install python-is-python3
This solved an error when I used youtube-dl.
Projects based on python
- pithos Pandora on linux
- Many Raspberry Pi GPIO projects
- GeneScissors It also requires pip and scikit-learn packages.
- KeepNote It depends on Python 2.X, sqlite and PyGTK.
- Zim It depends on Python, Gtk and the python-gtk bindings.
- Cherrytree It depends on Python2, Python-gtk2, Python-gtksourceview2, p7zip-full, python-enchant and python-dbus.
Send emails
- Less secure apps & your Google Account. To help keep your account secure, from May 30, 2022, Google no longer supports the use of third-party apps or devices which ask you to sign in to your Google Account using only your username and password.
- Send Email via Gmail and SMTP Use an App Password 2022/9. Click on Security -> 2-Step Verification (You may need to enter your PW first). Scroll to the bottom of the page, and you'll see the "App passwords" section. You can delete/create app passwords but you can't view any existing passwords.
- How to Send Automated Email Messages in Python 3 2021/3
GUI programming
New book: Create Graphical User Interfaces with Python
Qt for GUI development
- http://zetcode.com/gui/pyqt4/
- http://wiki.wildsong.biz/index.php/PyQt Create GUI in Qt Designer and convert/use it in PyQt.
Python 3
- Python 2.7 will not be maintained past 2020. See https://pythonclock.org/.
- Migrating to Python 3 with pleasure
- Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython by Wes McKinney.
pip3
Use pip3 instead of pip for Python 3. For example,
pip3 install --upgrade pip pip3 install -U scikit-learn pip3 install -U matplotlib
http.server
Edit Files With Workspaces. The 'http.server' module is contained in python3.
cd ~/website python3 -m http.server
C vs Python
C vs. Python: The Key Differences
R and Python: reticulate package
- Installing and Configuring Python with RStudio
- The instruction is based on virtualenv. But I'm following Biowulf's Python miniconda instruction to create a new project/environment. One caveat is I need to run source ~/$USER/conda/etc/profile.d/conda.sh each time before I start R in order to make conda available OR I need to set the RETICULATE_MINICONDA_PATH variable (see below).
- The conda-related reticulate functions include conda_create(), use_condaenv(), conda_install(), conda_list(), conda_remove()
- Use py_config() to check the current python path and other python versions found.
- My example
library(reticulate) # Assume I followed Biowulf's instruction to create 'project1' Sys.setenv(RETICULATE_MINICONDA_PATH = "~/conda") conda_list() use_condaenv("project1", required=T) py_config()
- RStudio -> Python
- https://cran.r-project.org/web/packages/reticulate/index.html, Github
- Using Python in R markdown
- Importing Python modules and call its functions directly from R — import() function
- Sourcing Python scripts — source_python() function
- Python REPL — The repl_python() function creates an interactive Python console within R.
- Python Version Configuration. Suppose I have installed miniconda and create a new environment called 'project1'. Then after calling source ~/conda/etc/profile.d/conda.sh I can start in R
library(reticulate) use_condaenv("project1", required = TRUE)
- On my macOS, even I have python3 installed, it still asks to install miniconda (/Users/$USER/Library/r-miniconda). So I get another version of Python3 in /Users/$USER/Library/r-miniconda/envs/r-reticulate/bin/python.
- I found RStudio IDE is better than PyCharm and Thonny editors.
- Install Python packages https://rstudio.github.io/reticulate/articles/python_packages.html
- Better to have anaconda3 installed. 2.26G space is required on macOS.
- Direct running py_install("pandas") would ask me to upgrade virtualenv
- Running virtualenv_create("r-reticulate") and then py_install("pandas") works
- Cheat sheet https://ugoproto.github.io/ugo_r_doc/pdf/reticulate.pdf
- R or Python? Why not both? Using Anaconda Python within R with {reticulate}
- Run Python from R
- R and Python: Using reticulate to get the best of both worlds. Note
- RStudio v1.2 preview release includes support for using reticulate to execute Python chunks within R Notebooks
- Error from my execution: ValueError: 'RBF' is not in list
- The reticulate package solves the hardest problem in data science: people
- reticulate, virtualenv, and Python in Linux
- Bugs
- Pass Python objects to R: Works. Or use py_run_string()
- Cannot pass R variables to Python: use source_python()
- R vs Python for data science by Norm Matloff.
- RvsPython #5.1: Making the Game even with Python’s Best Practices
- RvsPython #5: Using Monte Carlo To Simulate π
- How to Run Python's Scikit-Learn in R in 5 minutes
- Test python and markdown files
def add_three(x): z = x + 3 return z
--- title: "R Notebook" output: html_notebook --- ```{r} library(reticulate) py_discover_config() x <- 5 source_python("test.py") y <- add_three(x) print(y) ``` Pass R variables to Python. Works ```{python} a = 7 print(r.x) ``` Pass python variables to R. Works. ```{r} py$a py_run_string("y = 10"); py$y ```
- Reticulate webinar – R and Python – a happy union
- Linking R and Python to retrieve financial data and plot a candlestick
- Getting started with Python using R and reticulate
How to quit python
Type exit and hit Enter. See https://rstudio.github.io/reticulate/.
R vs Python
Call R from Python
Conda, Anaconda, miniconda
- Docker
- Python on Biowulf. Users who need stable, reproducible environments are encouraged to install miniconda in their data directory and create their own private environments.
- The Definitive Guide to Conda Environments
Private environment
Transfer a conda environment to another computer: YAML files
# computer 1 conda env export > environment.yml # computer 2 conda env create -f environment.yml
Conda environment create, activate, deactivate, info (see a list)
Getting started with conda. More details are in Tasks.
conda --version # Manage environment conda info --envs # see a list of environments. # The active environment is the one with an asterisk (*) # create a new environment conda create --name myenv # remove an environment conda remove --name myenv --all # Manage Python conda create --name snakes python=3.5 conda activate snowflakes # activate conda info --envs python --version conda activate # Change your current environment back to the default (base) conda deactivate # exit any python virtualenv # Managing packages conda search beautifulsoup4 conda install beautifulsoup4 conda list # Updating Anaconda or Miniconda conda update conda
Anaconda
- Introduction to Anaconda. Simplifies installation of Python packages
- Platform-independent package manager
- Doesn’t require administrative privileges
- Installs non-Python library dependencies (MKL, HDF5, Boost)
- Provides ”virtual environment” capabilities
- Many channels exist that support additional packages
- Install Anaconda on macOS. Better to use the command line method in order to install it to the user's directory. The new python can be manually loaded into the shell by using source ~/.bash_profile. Like Ubuntu, ananconda3 is installed under ~/ directory. In addition, Anaconda-Navigator is available under Finder -> Applications.
- How To Install the Anaconda Python Distribution on Ubuntu 16.04. As we can see Anaconda3 will be installed under /home/$USER/anaconda3.
- Download Anaconda3-2018.12-Linux-x86_64.sh from https://www.anaconda.com/distribution/#download-section
- bash Anaconda3-2018.12-Linux-x86_64.sh
- There is a question: Do you wish the installer to initialize Anaconda3. If you answer Yes, it will run conda init & modify ~/.bashrc file. # This will overwrite system's Python. So the default python/python3 will now be in /home/$USER/anaconda3/bin/.
Do you wish the installer to initialize Anaconda3 by running conda init? [yes|no] [no] >>> yes no change /home/brb/anaconda3/condabin/conda no change /home/brb/anaconda3/bin/conda no change /home/brb/anaconda3/bin/conda-env no change /home/brb/anaconda3/bin/activate no change /home/brb/anaconda3/bin/deactivate no change /home/brb/anaconda3/etc/profile.d/conda.sh no change /home/brb/anaconda3/etc/fish/conf.d/conda.fish no change /home/brb/anaconda3/shell/condabin/Conda.psm1 no change /home/brb/anaconda3/shell/condabin/conda-hook.ps1 no change /home/brb/anaconda3/lib/python3.8/site-packages/xontrib/conda.xsh no change /home/brb/anaconda3/etc/profile.d/conda.csh modified /home/brb/.bashrc ==> For changes to take effect, close and re-open your current shell. <== If you'd prefer that conda's base environment not be activated on startup, set the auto_activate_base parameter to false: conda config --set auto_activate_base false
If I choose not to modify .bashrc file,
Do you wish the installer to initialize Anaconda3 by running conda init? [yes|no] [no] >>> no You have chosen to not have conda modify your shell scripts at all. To activate conda's base environment in your current shell session: eval "$(/home/brb/anaconda3/bin/conda shell.YOUR_SHELL_NAME hook)" To install conda's shell functions for easier access, first activate, then: conda init If you'd prefer that conda's base environment not be activated on startup, set the auto_activate_base parameter to false: conda config --set auto_activate_base false Thank you for installing Anaconda3!
- Anaconda-Navigator (including jupyter notebook, Spyder IDE, ...) can be launched by typing anaconda-navigator in a terminal
- Getting started with Anaconda Python for data science
- Differences:
- Comparions:
- Conda: an open source package management system and environment management system
- Miniconda, which is a smaller alternative to Anaconda that is just conda and its dependencies. Once you have Miniconda, you can easily install Anaconda into it with conda install anaconda.
- Anaconda: Anaconda is a set of about a hundred packages including conda, numpy, scipy, ipython notebook, and so on.
- Uninstall
- Used in pdxBlacklist
- What is an open source software distribution?
Miniconda
- https://docs.conda.io/en/latest/miniconda.html As you can see miniconda installers were separated by the Python version.
- How To Install Miniconda In Linux 2021. It includes Install Miniconda interactively, unattended installation, Update Miniconda, and Uninstall Miniconda. If you've chosen the default location, the installer will display “PREFIX=/var/home/<user>/miniconda3”. To manually activate conda's base environment, do /home/<user>/miniconda3/etc/profile.d/conda.sh where we assume miniconda is installed under /home/<user>/miniconda3 directory.
- Miniconda Installation for macOS users 2019. At the end of installation, we see if we don't want conda's base environment to be activated on start up, we can do conda config --set auto_activate_base false
- See also Python on Biowulf about how to specify prefix.
- We can add/install a module to an existing environment. See Miniconda: Python(s) in a convenient setup.
conda install -n <env_name> <package> conda create -n myenv python=3 # create a new environment named “myenv” with Python 3 installed # after that, use "conda activate myenv" and use "conda install numpy" to install the numpy
Install and "conda init"
- Windows: screenshots are included Setting up Python on Windows with Miniconda by Anaconda & Anaconda documentation. The default is not to add Anaconda to my PATH environment variable.
- Ubuntu: How to Install Miniconda on Ubuntu 20.04. After installation, PATH variable will prepend ~/miniconda3/condabin which contains only 1 file: conda.
- conda init
- Running conda init initializes conda for shell interaction by writing some shell code in the relevant startup scripts of your shell (e.g~/.bashrc) 1. This allows the conda command to interact more closely with the shell context and provides a cleaner PATH manipulation and snappier responses in some conda commands. The main advantage of running conda init is that it enables the use of the conda activate and conda deactivate commands, which are used to activate and deactivate conda environments.
- We only need to call "conda init" once no matter after we install conda how many conda environments we will work.
- One disadvantage of running conda init is that it can sometimes cause issues if the initialization is not done correctly or if there are conflicts with other configurations in your shell startup scripts. However, these issues can usually be resolved by troubleshooting and making the necessary changes to your configurat
conda environment
A conda environment is a directory that contains a specific collection of conda packages that you have installed.
You need to create an environment first before you can activate it. The conda activate command does not create an environment for you, it only activates an existing one.
conda create --name myenv conda activate myenv
Q: Where are the environments located? A: Conda environments are typically stored in the envs subdirectory of your Anaconda installation directory. For example, if you have an environment named myenv, it would be located in a directory like ~/anaconda3/envs/myenv. The exact path can be found by using conda env list command.
Q: How do I list all existing environments? A: To list all existing conda environments, you can use the conda env list or conda info --envs command. Here’s how you do it:
conda env list # base * /opt/conda DrivR-Base /opt/conda/envs/DrivR-Base
Q: How to quit a conda environment?
conda deactivate # Return to base conda deactivate # Exit base
Q: Check the disk space used by a specific conda environment.
du -sh /path/to/conda/envs/your_enviornment_name
Q: How to delete a conda environemnt,
conda deactivate conda env remove --name your_environment_name # OR conda remove --name your_environment_name --all
Install all anaconda packages
- https://stackoverflow.com/a/52316549 conda install anaconda
- https://docs.conda.io/en/latest/miniconda.html conda create -n py3k anaconda python=3
- how much space is needed for installing anaconda? The minimum disk space required for installing Anaconda is 3 GB, but it is recommended to have at least 6 GB of free disk space available.
Uninstall miniconda
- rm -rf ~/miniconda3
- nano ~/.bash_profile and delete conda initialize block
What's the purpose of the “base” (for best practices) in Anaconda?
https://stackoverflow.com/a/56504279
Does Conda replace the need for virtualenv?
Yes. Conda is not limited to Python but can be used for other languages too.
GCC/gFortran
- conda install gcc
- Using conda-forge channel - conda install -c conda-forge gfortran
Using R language with Anaconda
- Using R language with Anaconda
conda create -n r_env r-essentials r-base conda activate r_env
- Difference of install a package using install.packages() function in R and using the conda install command?
- The install.packages() function in R and the conda install command are two different ways to install R packages. The install.packages() function is used to install packages from the Comprehensive R Archive Network (CRAN), while the conda install command is used to install packages from the Anaconda repository.
- One key difference between the two methods is that conda can manage dependencies across multiple programming languages, while install.packages() only manages dependencies within R.
- Another difference is that conda allows you to create and manage multiple isolated environments, each with its own set of packages. This can be useful if you want to have different versions of packages available for different projects. With install.packages(), all packages are installed in the same global library, which can make it more difficult to manage dependencies and avoid conflicts.
-
The Definitive Guide to Conda Environments, Using R language with Anaconda. Environments created with conda create live by default in the envs/ folder of your Conda directory, whose path will look something like /Users/user-name/miniconda3/envs or /Users/user-name/anaconda3/envs.
Activate conda base Create a new env Activate a new env Deactivate an env ----------------------------> (base) -----------------> -------------------> (r-env) -----------------> (base) eval $(conda shell.bash hook)" conda create r-env conda activate r-env conda deactivate
$ eval "$(/home/brb/anaconda3/bin/conda shell.bash hook)" (base) $ mkdir mypythonproj; cd mypythonproj # This step seems not necessary (base) $ conda create -n r-env r-base ... # # To activate this environment, use # # $ conda activate r-env # # To deactivate an active environment, use # # $ conda deactivate (base) $ conda activate r-env (r-env) $ ls anaconda3/envs r-env (r-env) $ conda install r-essentials (r-env) $ which R /home/brb/anaconda3/envs/r-env/bin/R (r-env) $ ls -la # Still Empty (r-env) $ R --version R version 3.4.3 (2017-11-30) -- "Kite-Eating Tree" # Note that the current R version should be 4.0.3 (r-env) $ conda env list base /home/brb/anaconda3 r-env * /home/brb/anaconda3/envs/r-env (r-env) $ conda deactivate (base) $
It seems to be better to save the environment inside a project directory. So using python -m venv /path/to/new/environment method is preferred. You can also use conda create --prefix /path/to/new/environment. Placing environments outside of the default env/ folder comes with some drawbacks. Read the document of 'The Definitive Guide to Conda Environments'.
- conda-forge channel, A brief introduction, https://anaconda.org/conda-forge/r-base. Following the instruction seems to mess things up though the conda-forge says the latest version is 4.0.3 (3 years late).
$ eval "$(/home/brb/anaconda3/bin/conda shell.bash hook)" (base) $ conda install -c conda-forge r-base ... ## Package Plan ## environment location: /home/brb/anaconda3 added / updated specs: - r-base ... Downloading and Extracting Packages r-base-3.2.2 ... (base) $ R --version /home/brb/anaconda3/lib/R/bin/exec/R: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such file or directory (base) $ which R /home/brb/anaconda3/bin/R
Run R with Jupyter notebook
- How To Use R In Jupyter Notebooks: A Step-By-Step Approach
mkdir project; cd project python3 -m venv myenv source myenv/bin/activate pip3 install jupyter # Same terminal R # this is from the global environment # so the local environment for R does not work
install.packages('IRkernel') # Install IRkernel from within R # Make sure build-essential has been installed before # running install.packages(). IRkernel::installspec() # Make IRkernel available to JupyterLab q()
After running these commands in R, you should be able to select R as a kernel when creating a new notebook in JupyterLab.
jupyter lab # automatically launch jupyter in a browser Ctrl+c # stop deactivate
- Using the R programming language in Jupyter Notebook (Anaconda)
- Setup Jupyter Notebook for R (Windows OS, no conda)
- How to Add R to Jupyter Notebook (full steps) using Anaconda
- How to install R on a Jupyter notebook using homebrew
- ggplot2: Mastering the basics & Jupyter Notebook. To set up the Jupyter environment, see the Docker method.
docker run --rm -p 8888:8888 \ -e JUPYTER_ENABLE_LAB=yes \ -v "$PWD":/home/jovyan \ jupyter/datascience-notebook:r-4.0.3
We first have to use "git clone https://github.com/rlbarter/ggplot2-thw.git" to download the repo and "cd ggplot2-thw". Then after opening http://IP:8888/?token=XXXXXXX we will see "ggplot2.ipynb" on the left panel. Double click the file will open it on the Notebook.
Example 1: GEO2RNAseq
Example 2: p-NET
Biologically informed deep neural network for prostate cancer classification and discovery and the paper 2021.
Mamba
- Mamba is a high-performance package manager that is fully compatible with Conda, the package management system widely used in the Python ecosystem. It was developed to provide a faster and more efficient alternative to Conda, addressing some of the performance issues, especially in terms of dependency resolution and package installation speed.
- https://github.com/mamba-org/mamba The Fast Cross-Platform Package Manager
- Biowulf. Mambaforge: a derivative of miniconda that includes mamba and uses the conda-forge channel in place of the defaults channel
- How to install Mamba on Ubuntu 21.10, How to install R using Mamba, How to install RStudio on Ubuntu 21.10 with R installed using Mamba
- Debian 12
- (This step is unnecessary, miniforge includes Conda already) Install anaconda on debian. I choose 'no' at the final question.
sudo apt-get install libgl1-mesa-glx libegl1-mesa libxrandr2 libxrandr2 \ libxss1 libxcursor1 libxcomposite1 libasound2 libxi6 libxtst6 curl -O https://repo.anaconda.com/archive/Anaconda3-2024.06-1-Linux-x86_64.sh bash Anaconda3-2024.06-1-Linux-x86_64.sh # Anaconda3 will now be installed into this location: /home/$USER/anaconda3 # Do you wish to update your shell profile to automatically initialize conda? no # Log out and log in again # verify conda conda list conda activate base conda deactivate
- Install mamba. Note that I choose 'yes' at the final question.
curl -O https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh bash Miniforge3-$(uname)-$(uname -m).sh # press q to exit the agreement # Miniforge3 will now be installed into this location: # /home/brb/miniforge3 # To activate this environment, use: # # micromamba activate /home/brb/miniforge3 # # Or to execute a single command in this environment, use: # # micromamba run -p /home/brb/miniforge3 mycommand # # ... # You can undo this by running `conda init --reverse $SHELL`? [yes|no] # [no] >>> yes <----- IMPORTANT; o.w. mamba will not be available # You have chosen to not have conda modify your shell scripts at all. # To activate conda's base environment in your current shell session: # # eval "$(/home/brb/miniforge3/bin/conda shell.YOUR_SHELL_NAME hook)" # # To install conda's shell functions for easier access, first activate, then: # # conda init # # Thank you for installing Miniforge3!
- Using mamba. Mamba commands are the same as Conda commands, so you can seamlessly switch between using the two.
mamba create -n myenv python=3.6.12 mamba activate myenv mamba install numpy=1.19.2 pandas=1.1.3 pip list mamba deactivate # deactivating an environment does not delete it; it simply changes your working context. mamba env list mamba env remove -n myenv $ which mamba /home/brb/miniforge3/condabin/mamba $ which conda /home/brb/miniforge3/condabin/conda
- (This step is unnecessary, miniforge includes Conda already) Install anaconda on debian. I choose 'no' at the final question.
Web framework
Flask
- Flask (web framework) Flask is a micro web framework written in Python. It is classified as a microframework because it does not require particular tools or libraries.
- https://palletsprojects.com/p/flask/
- How to Install Flask with Python 3 on Ubuntu 18.04
- Data Science for Startups: Containers Building reproducible setups for machine learning
- Raspberry Pi
- Raspberry Pi System Stats. To access the web page, use for example http://192.168.1.104:5000/cpu or http://192.168.1.104:5000/disk or http://192.168.1.104:5000/memory.
- Build a Python Web Server with Flask
- Python WebServer With Flask and Raspberry Pi