Skip to main content

How to use Archaea tools

To upload and download files to and from the NIG supercomputer, you can generally use widely-used file transfer software such as scp or sftp. However, these tools tend to experience slower transfer speeds when transferring large files over long distances.

Archaea tools, developed by Clalink Technology, Inc., enable high-speed communication at 100 Gbps even over long-distance connections.

Reference

Overview of Usage Steps​

To transfer data using Archaea tools, follow the steps below:

  1. Install the Archaea client software on your local machine
  2. Place the provided configuration file
  3. Execute the data transfer command to upload or download your files

Installing the client software​

On the NIG supercomputer, Archaea tools can be used to transfer data between the user's local machine and the supercomputer.

To use Archaea tools, you need to install the client software on your client machine which is the machine that communicates with the NIG supercomputer (typically, your personal computer).

Refer to the following official documentation for installation instructions, depending on your operating environment.

How to setup the configuration file​

To transfer data to and from the NIG supercomputer, you need to place a configuration file for Archaea tools (formerly HCPtools) in the home directory of your client machine.

Instructions on how to write the configuration file can be found in the official documentation at the following link. for the most up-to-date information, refer to the official documentation.

Since there are many configuration files and manual setup can be complex,
we recommend cloning the pre-configured settings for the NIG supercomputer from GitHub to your client machine’s home directory using git clone. (Do not place the configuration files in your home directory of NIG supercomputer.)

Linux (including for Windows WSL2 environments)

cd $HOME
git clone https://github.com/nig-sc/Bytix_Archaea/ .hcp

Mac OS

cd $HOME
git clone https://github.com/nig-sc/Bytix_Archaea/ .hcp

Windows (PowerShell)

cd $HOME
git clone https://github.com/nig-sc/Bytix_Archaea/ _hcp

The following files are created when you git clone.

$ tree .hcp
.hcp
├── README.md
├── hchmod.conf
├── hchown.conf
├── hcp-common.conf
├── hcp-ls.conf
├── hcp.conf
├── hln.conf
├── hmkdir.conf
├── hmv.conf
├── hpwd.conf
├── hrm.conf
└── hsync.conf

1 directory, 12 files

Next, add the absolute path to your private key (used when logging into the NIG supercomputer)
to the configuration file hcp-common.conf as shown below:

hcp-common.conf (macOS)

PrivateKeyFile /Users/youraccount/.ssh/id_ed25519    # Specify the private key
AcceptableCryptMethod   PLAIN                # Encryption: None
AcceptableDigestMethod  SHA256               # Digest format: SHA256
DisableDataIntegrityChecking yes             # Allow no digest format

hcp-common.conf (Windows PowerShell)

PrivateKeyFile C:\Users\youraccount\.ssh/id_ed25519    # Specify the private key
AcceptableCryptMethod   PLAIN                # Encryption: None
AcceptableDigestMethod  SHA256               # Digest format: SHA256
DisableDataIntegrityChecking yes             # Allow no digest format

hcp-common.conf (Linux)

PrivateKeyFile /home/youraccount/.ssh/id_ed25519    # Specify the private key
AcceptableCryptMethod   PLAIN                # Encryption: None
AcceptableDigestMethod  SHA256               # Digest format: SHA256
DisableDataIntegrityChecking yes             # Allow no digest format

The file transfer with Archaea tools(formerly HCPtools)​

Upload files to the NIG supercomputer​

To transfer files from your local machine to the NIG supercomputer,
launch a terminal emulator on your client machine and execute the following command:

hcp --user youraccountname --hpfp \
   your_file.txt \
   dtn1.ddbj.nig.ac.jp:/home/your_account/

Download files from the NIG supercomputer​

To transfer files from the NIG supercomputer to your local machine,
launch a terminal emulator on your client machine and execute the following command:

If you would like to transfer a file with the same name as on the supercomputer to the current directory on your local machine (represented by the final . in the command), use the following:

hcp --user youraccountname --hpfp  \
    dtn1.ddbj.nig.ac.jp:/home/your_account/your_file.txt \
    .

To recursively copy an entire directory from the NIG supercomputer to the current directory on your local machine (represented by the final . in the command),
add the -R option as shown below:

hcp -R --user youraccountname --hpfp  \
    dtn1.ddbj.nig.ac.jp:/home/your_account/data_dir \
    .

NOTE.​

For the first time when you transfer data, the following message will be displayed. Enter yes.

Are you sure you want to continue connecting [yes/no] ?

Frequently used options for file transfer​

For more information of options, refer to 🔗the official manual.

  • --hpfp : Specify UDP (HpFP2) communication to speed up communication over long distances
    • Without this option, TCP communication ehich uesd usually widely is performed.
  • -p : preserve information about file permission
  • -R : copy all files under each directory, recursively
  • -r : resume the previous copy
  • -y : verify blocks of data transfered by message digest
  • -z : compress blocks of data transfered

Other Commands​

commandfeature
hrmdelete files on the server
hcp-lslist files on the server
hmkdircreate directories on the server
hpwdretrieve the working directory on the server
hmvmove files on the server
hlmcreate symbolic links, etc. on the server
hchmodchange the file's permission on the server
hchownchange the file's owner on the server
hsyncsynchronize files on the server

For more information, refer to 🔗the official manual.

Last updated on
RSS FeedJSON Feed