Site Policy
2017 年 03 月 26 日

Software configuration

About the OS

As of March 2014, our Supercomputer System has been using Linux as its OS, and the Thin compute nodes use the RedHat Enterprise 6 distribution. This is common to all compute nodes.
Phase1 System and Phase2 System differ in OS version as bellow.
For this reason, the program which has moved Phase2 system from the Phase1 system, may be necessary to re-compile.

SystemOSOS Version
Phase1 System    Red Hat Enterprise Linux 6    6.1
Phase2 System Red Hat Enterprise Linux 6 6.4

 

About the job management system

Our Supercomputer System utilizes Univa Grid Engine (UGE) as the job management system. UGE is a commercial product derived from the open source software Sun Grid Engine6.2U5 owned by the former Sun Micro Systems, and so its underlying system concept, user command system, and so forth, are identical to those of Sun Grid Engine. For basic procedures for using the system utilizing UGE, please reference How to use the system.

Job-related upper limit settings in UGE

The number of job slots that can be used simultaneously by one user is as follows:

Setting itemSetting value
Upper limit on the number of job slots (accounts for general research) 500
Upper limit on the number of job slots (accounts for large-scale use) 1500
Upper limit on the number of jobs to enter per user 5000
Upper limit on the number of tasks to enter per user 75000

By applying for an expansion of computer resources, the upper limit on the number of job slots can be expanded to the level of accounts for large-scale use. Please make an application if necessary.

Queue configuration

The following queues are set up depending on the type of compute node used and the period of use (as of May 3, 2014)

The queue (short.q) for short period job was added in December 2014.

Phase1 System

Queue nameNumber of job slotsMaximum memoryUpper limit for execution timePurposeOptions for queue specification
week_hdd.q 1472 64G 14 days When there is no resource request in two weeks of the execution period No specification (default)
month_gpu.q 248 32G 62 days Use of GPU -l month -l gpu
month_medium.q 160 2T 62 days Use of Medium compute node -l month -l medium
month_fat.q 768 10T 62 days Use of Fat compute node -l month -l fat
debug.q 48 64G 24 hours For debugging and operation check -l debug
login.q 192 64G Unlimited Used to execute qlogin from the gateway node  
short.q 744 32G 24 hours Short period job -l short 

Phase2 System

Compeared with the Phase2 System, month_phi.q queue that has high parallel execution has been added.

Queue nameNumber of job slotsMaximum memoryUpper limit for execution timePurposeOptions for queue specification
week_hdd.q 1020 64G 14 days When there is no resource request in two weeks of the execution period No specification (default)
week_ssd.q 640 64G 14 days Used when use of SSD is desired -l ssd
month_phi.q 600 64G 62 days Enter jobs that utilize Co-processor(Xeon Phi) into this queue. -l month -l phi
month_gpu.q 310 32G 62 days Use of GPU -l month -l gpu
month_medium.q 640 2T 62 days Use of Medium compute node -l month -l medium
debug.q 80 64G 24 hours For debugging and operation check -l debug (-l gpu / -l phi)
login.q 420 64G Unlimited Used to execute qlogin from the gateway node  
short.q 930 32G 24 hours Short period job -l short 

  **The option of a debugging queue of gpu node  ....  -l debug -l gpu 
    The option of a debugging queue of xeon phi node  ....  -l debug -l phi

  **Maximum memory  .... The maximum value that can be specified in -l mem_req

For specific procedures for entering jobs and the job management system, please see How to use the system.

File transfer software

For data transfer between the system and an external entity, sftp, scp, and aspera can be used. For specific transfer procedures, see How to transfer file.

Programming environment

Compiler environment

The following three types of compilers can be used as the development environment in the NIG Cluster:

Package nameVersionSupported languageLocation of installationDocument URL (outside the site)
Intel Composer XE Linux 16.0.1(XE2016) C,C++,Fortran77,90,95,2003,2008 /opt/intel/xe_2016/ URL
PGI Accelerator CDK Cluster Development Kit 16.4 C,C++,C99,FORTRAN77,90,95,2003,HPF /opt/pgi/linux86-64/current/ URL
Gnu Compiler(OSS) 4.8.5 C,C++,Objective-C,Fortran77,90,2003,2008,Java,Ada /usr/local/bin/ URL

Please see Programming environment for procedures.

MPI environment

OpenMPI can be used as the MPI environment. Please see Programming environment for procedures.

Scientific computation library

The following can be utilized. Please see Programming environment for details.

Package nameBundle productsLocation of installationIntel
Intel Cluster Studio Intel MKL, Intel IPP, Intel TBB, Intel MPI, Intel Trace Analyzer/Collector /opt/intel
PGI Accelerator CDK Cluster Development Kit PGDBG, PGPROF, ACML, ScaLAPACK( v1.7) /opt/pgi

GPU utilization environment

As described above, one unit of GPGPU Tesla2090 is equipped for 64-node Thin nodes. CUDA (current version 7.5 as of June 2016) by Nvidia is installed as the environment being used. In addition, the GPU driver version is 352.39  (as of June 2016). OpenCL is not currently installed. For specific procedures for using the environment, please see Programming environment.

Profiler

It is possible to use pgprof, which is included in the PGI compiler environment. Please see Programming environment for details.

Shell environment

The only login shell for the system is bash. However, bash (sh), tcsh (csh), and zsh can be used for job scripts and so forth. The shell included in the distribution is used. It is possible to change the login shell. Since this system manages its users using LDAP, changes shall be made according to the procedures described in system utilization TIPS instead of chsh.

Java environment

Java can be used; however, only the 64-bit version of Java is installed. For details about the procedures, please see Programming environment.

Script languages that can be used by general users

The script language environments that can be used by general users at the login nodes or compute nodes are as follows:

LanguageVersionInstallation path
ruby 2.3.0 /usr/local/bin/ruby
python 2.7.2/3.5.1 /usr/local/bin/python
perl 5.22.1 /usr/local/pkg/perl

Resources related to bioinformatics

The following databases are available. Access here for details.