How to use Archaea tools(formerly HCPtools)

When you transfer files to and from the NIG supercomputer, you can use scp or sftp, which are widely used as the file transfer software. But their transfer speed is slow when a large numbers of files are transfered over long distances.

For high-speed file transfer over long distances, the file transfer software Aspera is available on the general analysis division, and Archaea tools(formerly HCPtools) is available on the personal genome analysis division on the NIG supercomputer.

Reference

• For official manuals for each product, 🔗see the official Bytix page 'Documents'.
• For the latest version, 🔗refer to the Bytix official page 'Download "latest version"'.
• FAQ (HCP tools)

Renaming of software

In October 2022, the HCPtools software provider launched the brand name 'Bytix' as a brand name for data transfer systems and the product name was changed from 'HCPtools' to 'Archaea tools'.

After the product name change, you can still use the same commands that you have been using for HCPtools.

For more information on the change, 🔗refer to the Bytix official page 'Product name change etc.'

Installing the client software

For using Archaea tools(formerly HCPtools), all user must install that client software on user's client computer. Refer to the following documents depending on the environment of your client computer.

• Linux (including for Windows WSL2 environments)
  • 🔗Ubuntu Linux
  • 🔗CentOS 7
• 🔗Mac OS
• 🔗Windows

How to setup the configuration file

To transfer data between the NIG supercomputer, a configuration file for Archaea tools (formerly HCPtools) must be setup in your home directory. So download the configuration file for the NIG supercomputer from github with the git clone command.

Linux (including for Windows WSL2 environments)

cd $HOME
git clone https://github.com/nig-sc/Bytix_Archaea/ .hcp

Mac OS

cd $HOME
git clone https://github.com/nig-sc/Bytix_Archaea/ .hcp

Windows (PowerShell)

cd $HOME
git clone https://github.com/nig-sc/Bytix_Archaea/ _hcp

The following files are created when you git clone.

$ tree .hcp
.hcp
├── README.md
├── hchmod.conf
├── hchown.conf
├── hcp-common.conf
├── hcp-ls.conf
├── hcp.conf
├── hln.conf
├── hmkdir.conf
├── hmv.conf
├── hpwd.conf
├── hrm.conf
└── hsync.conf

1 directory, 12 files

Write the absolute path of the private key in the configuration file hcp-common.conf as follows.

PrivateKeyFile /home/username/.ssh/id_rsa    # Specify the private key
AcceptableCryptMethod   PLAIN                # Encryption: None
AcceptableDigestMethod  SHA256               # Digest format: SHA256
DisableDataIntegrityChecking yes             # Allow no digest format

Mac OS

PrivateKeyFile /Users/youraccount/.ssh/id_rsa    # Specify the private key

Windows (PowerShell)

PrivateKeyFile C:\Users\youraccount\.ssh/id_rsa    # Specify the private key

The file transfer with Archaea tools(formerly HCPtools)

SSL-VPN connection to the personal genome analysis division

For file transfer to and from the personal genome analysis division, establish the SSL-VPN connection between the client computer and the personal genome compartment of the NIG supercomputer.

How to establish it: 「How to Login (the personal genome analysis division)」>「How to establish the SSL-VPN connection」

Upload files to the NIG supercomputer

Start a terminal emulator on the user's client computer and execute the following command.

Linux (including for Windows WSL2 environments)

hcp --user username --hpfp \
   /home/username/your_file.txt \
   gwa.ddbj.nig.ac.jp:/home/your_account-pg/some_directory/your_file.txt

Mac OS

hcp --user username --hpfp \
   /Users/username/your_file.txt \
   gwa.ddbj.nig.ac.jp:/home/your_account-pg/some_directory/your_file.txt

Windows (PowerShell)

hcp --user username --hpfp \
    C:\Users\username\your_file.txt \
    gwa.ddbj.nig.ac.jp:/home/your_account-pg/some_directory/your_file.txt

Download files from the NIG supercomputer

Start a terminal emulator on the user's client computer and execute the following command.。

Linux (including for Windows WSL2 environments)

hcp --user username --hpfp  \
   gwa.ddbj.nig.ac.jp:/home/your_account-pg/some_directory/your_file.txt \
  /home/username/your_file.txt

Mac OS

hcp --user username --hpfp  \
   gwa.ddbj.nig.ac.jp:/home/your_account-pg/some_directory/your_file.txt \
   /Users/username/your_file.txt

Windows (PowerShell)

hcp --user username --hpfp  \
    gwa.ddbj.nig.ac.jp:/home/your_account-pg/some_directory/your_file.txt \
    C:\Users\username\your_file.txt

NOTE.

For the first time when you transfer data, the following message will be displayed. Enter yes.

Are you sure you want to continue connecting [yes/no] ?

Frequently used options for file transfer

For more information of options, refer to 🔗the official manual.

• --hpfp : Specify UDP (HpFP2) communication to speed up communication over long distances
  • Without this option, TCP communication ehich uesd usually widely is performed.
• -p : preserve information about file permission
• -R : copy all files under each directory, recursively
• -r : resume the previous copy
  • For more information, see 🔗the command reference.
• -y : verify blocks of data transfered by message digest
• -z : compress blocks of data transfered

Other Commands

command	feature
hrm	delete files on the server
hcp-ls	list files on the server
hmkdir	create directories on the server
hpwd	retrieve the working directory on the server
hmv	move files on the server
hlm	create symbolic links, etc. on the server
hchmod	change the file's permission on the server
hchown	change the file's owner on the server
hsync	synchronize files on the server

For more information, refer to 🔗the official manual.