Python

Python processors are installed from the outset on the NIG supercomputer. You can also install Python processors using a virtual environment such as venv, miniconda, etc. with user permissions.

Using Python processors already installed on the NIG supercomputer

Python processors are installed from the outset on the NIG supercomputer. The procedure for checking the installed Python version and the installed packages is as follows.

$ python --version
Python 3.10.12
$ pip list
Package                                          Version
------------------------------------------------ ------------------
matplotlib                                       3.5.1
numpy                                            1.21.5
pandas                                           1.3.5
scikit-learn                                     0.23.2
scipy                                            1.8.0

Using a virtual environment such as venv, miniconda, etc.

To use a package or version that is not installed, set up a virtual Python environment in your home directory. This page describes four approaches to setting up a virtual environment using venv, virtualenv, pyenv and Miniconda. Choose the approach that best suits your purpose and usage.

Virtual Env Tool	Description
venv	official Python Virtual Env Tool, installation with pip supported
virtualenv	similar to venv, Python2 supported
pyenv	arbitrary version of Python can be intorduced, package management per version provided
Miniconda	provides the management of Python version and libraries per virt env, used to install Jupyter Lab

venv

venv is the official Pyhton tool for virtual environment. You need to use it to install and use packages that are not installed.

$ python -m venv ~/venv_p310
$ source ~/venv_p310/bin/activate
$ python --version
Python 3.10.12

After you activate your virtual environment, use pip to install required libraries

$ pip install torch

To terminate your virtual environment, use deactivate.

$ deactivate

Launch Python with Job scheduler

You need to run Python after you activate your virtual environment in your job script.

Grid Engine

$ cat launch_python.sh
#!/bin/bash
#$ -cwd
#$ -V
#$ -l short
#$ -l s_vmem=16G
#$ -l mem_req=16G
#$ -N user_tensorflow
#$ -S /bin/bash
source ${HOME}/venv_p310/bin/activate
python tensorflow-testing.py
deactivate
$ qsub launch_python.sh

Slurm

$ cat launch_python.sh
#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -c 1
#SBATCH --mem-per-cpu=16G
#SBATCH -t 0-01:00:00
#SBATCH -J user_tensorflow
source ${HOME}/venv_p310/bin/activate
python tensorflow-testing.py
deactivate
$ sbatch launch_python.sh

Refer to the official web page for the details venv --- Create Virtual Envi

virtualenv

Basically, you need virtualenv in order to use Python2.

$ virtualenv -p python2.7 ~/p27
$ source ~/p27/bin/activate
$ python --version
Python 2.7.18

After you activate your virtual environment, use pip to install required libraries.

$ pip install torch

To terminate your virtual environment, use deactivate.

$ deactivate

Launch Python with Job scheduler

You need to run Python after you activate your virtual environment in your job script.

Grid Engine

$ cat launch_python.sh
#!/bin/bash
#$ -cwd
#$ -V
#$ -l short
#$ -l s_vmem=16G
#$ -l mem_req=16G
#$ -N user_tensorflow
#$ -S /bin/bash
source ${HOME}/p27/bin/activate
python tensorflow-testing.py
deactivate
$ qsub launch_python.sh

Slurm

$ cat launch_python.sh
#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -c 1
#SBATCH --mem-per-cpu=16G
#SBATCH -t 0-01:00:00
#SBATCH -J user_tensorflow
source ${HOME}/p27/bin/activate
python tensorflow-testing.py
deactivate
$ sbatch launch_python.sh

Refer to the official web page for the details virtualenv User Guide.

pyenv

You need to use pyenv to install and use the versions of Python that are not installed. You can manage packages per Python version. You can activate your virtual environment for all directories (global) or any given directory (local).

$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv
$ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
$ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
$ echo 'eval "$(pyenv init -)"' >> ~/.bashrc
$ source ~/.bashrc

You need to confirme what versions of Python are available and you can install the required version.

$ pyenv install --list
Available versions:
  3.10.13
  3.11.7
  3.12.0
  3.12.1
  3.12.2
$ pyenv install 3.12.2

After you install Python, you need to activate the version of Python. You can set global version and local version. The global version affects all directries and local version only affects the current directory.

$ pyenv global 3.12.1
$ mkdir ~/p3122;cd $_;pyenv local 3.12.2
$ python --version
Python 3.12.2

After you enter your virtual environment, use pip to install required libraries.

$ pip install torch

Launch Python with Job scheduler

You need to run Python after you move to your virtual environment in your job script.

Grid Engine

$ cat launch_python.sh
#!/bin/bash
#$ -cwd
#$ -V
#$ -l short
#$ -l s_vmem=16G
#$ -l mem_req=16G
#$ -N user_tensorflow
#$ -S /bin/bash
WORKDIR=${HOME}/p3122
cd ${WORKDIR}
python tensorflow-testing.py
$ qsub launch_python.sh

Slurm

$ cat launch_python.sh
#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -c 1
#SBATCH --mem-per-cpu=16G
#SBATCH -t 0-01:00:00
#SBATCH -J user_tensorflow
WORKDIR=${HOME}/p3122
cd ${WORKDIR}
python tensorflow-testing.py
$ sbatch launch_python.sh

Refer to the official web page for the details pyenv.

Miniconda

Miniconda is needed if you need to use a version of Python that is not installed on the sysytem and need to use packages that are not installed on the system and need to manage packages per virutal environment with the same version of Python.

$ mkdir ~/miniconda3
$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
$ sh ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
$ rm -f ~/miniconda3/miniconda.sh
$ ~/miniconda3/bin/conda init bash
$ source ~/.bashrc

You need to set the conda-forge repository to the default repository and confirm whether it is added to .condarc.

$ conda config --add channels conda-forge
$ conda config --set channel_priority strict
$ vim ~/.condarc

The base environment starts by default. It may be safe to disable this behavior to prevent you from setting up unintended environment.

(base) $ conda deactivate
$ conda config --set auto_activate_base false

Confirm what versions of Python are available and create a virtual environment with the required version of Python.

$ conda search -f python
Loading channels: done
# Name                       Version           Build  Channel
python                        3.9.18      h955ad1f_0  pkgs/main
python                       3.10.13      h955ad1f_0  pkgs/main
python                        3.11.7      h955ad1f_0  pkgs/main
python                        3.12.1      h996f2a0_0  pkgs/main
$ conda create --name py312 python=3.12.1
$ conda activate py312
$ python --version
Python 3.12.0

Install required packages by using conda install.

$ conda install pytorch

To terminate your virtual environmnet, use deactivate.

$ conda deactivate

Launch Python with Job scheduler

You need to run Python after you activate your virtual environment in your job script.

Grid Engine

$ cat launch_python.sh
#!/bin/bash
#$ -cwd
#$ -V
#$ -l short
#$ -l s_vmem=16G
#$ -l mem_req=16G
#$ -N user_tensorflow
#$ -S /bin/bash
source ${HOME}/miniconda3/etc/profile.d/conda.sh
conda activate py312
python tensorflow-testing.py
conda deactivate
$ qsub launch_python.sh

Slurm

$ cat launch_python.sh
#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -c 1
#SBATCH --mem-per-cpu=16G
#SBATCH -t 0-01:00:00
#SBATCH -J user_tensorflow
source ${HOME}/miniconda3/etc/profile.d/conda.sh
conda activate py312
python tensorflow-testing.py
conda deactivate
$ sbatch launch_python.sh

Refer to the official web page for the details Installing on Linux conda 4.10.3.post11+888309718 documentation.