Python
Python processors are installed from the outset on the NIG supercomputer. You can also install Python processors using a virtual environment such as venv, miniconda, etc. with user permissions.
Using Python processors already installed on the NIG supercomputer
Python processors are installed from the outset on the NIG supercomputer. The procedure for checking the installed Python version and the installed packages is as follows.
$ python --version
Python 3.10.12
$ pip list
Package Version
------------------------------------------------ ------------------
matplotlib 3.5.1
numpy 1.21.5
pandas 1.3.5
scikit-learn 0.23.2
scipy 1.8.0
Using a virtual environment such as venv, miniconda, etc.
To use a package or version that is not installed, set up a virtual Python environment in your home directory. This page describes four approaches to setting up a virtual environment using venv, virtualenv, pyenv and Miniconda. Choose the approach that best suits your purpose and usage.
| Virtual Env Tool | Description |
|---|---|
| venv | official Python Virtual Env Tool, installation with pip supported |
| virtualenv | similar to venv, Python2 supported |
| pyenv | arbitrary version of Python can be intorduced, package management per version provided |
| Miniconda | provides the management of Python version and libraries per virt env, used to install Jupyter Lab |
venv
venv is the official Pyhton tool for virtual environment. You need to use it to install and use packages that are not installed.
$ python -m venv ~/venv_p310
$ source ~/venv_p310/bin/activate
$ python --version
Python 3.10.12
After you activate your virtual environment, use pip to install required libraries
$ pip install torch
To terminate your virtual environment, use deactivate.
$ deactivate
Launch Python with Job scheduler
You need to run Python after you activate your virtual environment in your job script.
Grid Engine
$ cat launch_python.sh
#!/bin/bash
#$ -cwd
#$ -V
#$ -l short
#$ -l s_vmem=16G
#$ -l mem_req=16G
#$ -N user_tensorflow
#$ -S /bin/bash
source ${HOME}/venv_p310/bin/activate
python tensorflow-testing.py
deactivate
$ qsub launch_python.sh
Slurm
$ cat launch_python.sh
#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -c 1
#SBATCH --mem-per-cpu=16G
#SBATCH -t 0-01:00:00
#SBATCH -J user_tensorflow
source ${HOME}/venv_p310/bin/activate
python tensorflow-testing.py
deactivate
$ sbatch launch_python.sh
Refer to the official web page for the details venv --- Create Virtual Envi
virtualenv
Basically, you need virtualenv in order to use Python2.
$ virtualenv -p python2.7 ~/p27
$ source ~/p27/bin/activate
$ python --version
Python 2.7.18
After you activate your virtual environment, use pip to install required libraries.
$ pip install torch
To terminate your virtual environment, use deactivate.
$ deactivate
Launch Python with Job scheduler
You need to run Python after you activate your virtual environment in your job script.
Grid Engine
$ cat launch_python.sh
#!/bin/bash
#$ -cwd
#$ -V
#$ -l short
#$ -l s_vmem=16G
#$ -l mem_req=16G
#$ -N user_tensorflow
#$ -S /bin/bash
source ${HOME}/p27/bin/activate
python tensorflow-testing.py
deactivate
$ qsub launch_python.sh
Slurm
$ cat launch_python.sh
#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -c 1
#SBATCH --mem-per-cpu=16G
#SBATCH -t 0-01:00:00
#SBATCH -J user_tensorflow
source ${HOME}/p27/bin/activate
python tensorflow-testing.py
deactivate
$ sbatch launch_python.sh
Refer to the official web page for the details virtualenv User Guide.
pyenv
You need to use pyenv to install and use the versions of Python that are not installed. You can manage packages per Python version. You can activate your virtual environment for all directories (global) or any given directory (local).
$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv
$ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
$ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
$ echo 'eval "$(pyenv init -)"' >> ~/.bashrc
$ source ~/.bashrc
You need to confirme what versions of Python are available and you can install the required version.
$ pyenv install --list
Available versions:
3.10.13
3.11.7
3.12.0
3.12.1
3.12.2
$ pyenv install 3.12.2
After you install Python, you need to activate the version of Python. You can set global version and local version. The global version affects all directries and local version only affects the current directory.
$ pyenv global 3.12.1
$ mkdir ~/p3122;cd $_;pyenv local 3.12.2
$ python --version
Python 3.12.2
After you enter your virtual environment, use pip to install required libraries.
$ pip install torch
Launch Python with Job scheduler
You need to run Python after you move to your virtual environment in your job script.
Grid Engine
$ cat launch_python.sh
#!/bin/bash
#$ -cwd
#$ -V
#$ -l short
#$ -l s_vmem=16G
#$ -l mem_req=16G
#$ -N user_tensorflow
#$ -S /bin/bash
WORKDIR=${HOME}/p3122
cd ${WORKDIR}
python tensorflow-testing.py
$ qsub launch_python.sh
Slurm
$ cat launch_python.sh
#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -c 1
#SBATCH --mem-per-cpu=16G
#SBATCH -t 0-01:00:00
#SBATCH -J user_tensorflow
WORKDIR=${HOME}/p3122
cd ${WORKDIR}
python tensorflow-testing.py
$ sbatch launch_python.sh
Refer to the official web page for the details pyenv.
Miniconda
Miniconda is needed if you need to use a version of Python that is not installed on the sysytem and need to use packages that are not installed on the system and need to manage packages per virutal environment with the same version of Python.
$ mkdir ~/miniconda3
$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
$ sh ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
$ rm -f ~/miniconda3/miniconda.sh
$ ~/miniconda3/bin/conda init bash
$ source ~/.bashrc
You need to set the conda-forge repository to the default repository and confirm whether it is added to .condarc.
$ conda config --add channels conda-forge
$ conda config --set channel_priority strict
$ vim ~/.condarc
The base environment starts by default. It may be safe to disable this behavior to prevent you from setting up unintended environment.
(base) $ conda deactivate
$ conda config --set auto_activate_base false
Confirm what versions of Python are available and create a virtual environment with the required version of Python.
$ conda search -f python
Loading channels: done
# Name Version Build Channel
python 3.9.18 h955ad1f_0 pkgs/main
python 3.10.13 h955ad1f_0 pkgs/main
python 3.11.7 h955ad1f_0 pkgs/main
python 3.12.1 h996f2a0_0 pkgs/main
$ conda create --name py312 python=3.12.1
$ conda activate py312
$ python --version
Python 3.12.0
Install required packages by using conda install.
$ conda install pytorch
To terminate your virtual environmnet, use deactivate.
$ conda deactivate
Launch Python with Job scheduler
You need to run Python after you activate your virtual environment in your job script.
Grid Engine
$ cat launch_python.sh
#!/bin/bash
#$ -cwd
#$ -V
#$ -l short
#$ -l s_vmem=16G
#$ -l mem_req=16G
#$ -N user_tensorflow
#$ -S /bin/bash
source ${HOME}/miniconda3/etc/profile.d/conda.sh
conda activate py312
python tensorflow-testing.py
conda deactivate
$ qsub launch_python.sh
Slurm
$ cat launch_python.sh
#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -c 1
#SBATCH --mem-per-cpu=16G
#SBATCH -t 0-01:00:00
#SBATCH -J user_tensorflow
source ${HOME}/miniconda3/etc/profile.d/conda.sh
conda activate py312
python tensorflow-testing.py
conda deactivate
$ sbatch launch_python.sh
Refer to the official web page for the details Installing on Linux conda 4.10.3.post11+888309718 documentation.