Swestore-irods
Contents
National Storage using iRODS
NOTA: THIS PAGE IS OUT OF DATE! IT WILL BE SOON DELETED.
Getting access
- Apply for storage
- Please follow the instructions on the Apply for storage on SweStore page.
- iRODS user authentication
- SweStore's iRODS system uses Yubikey one-time passwords (OTP). With a simple touch of a button, a 44 character one-time password is generated and sent to the system. The user will be provided with a SweStore yubikey.
- Yubikey has a status as pilot now. It can be changed in the future.
- iRODS usage Pilot.
- How to acquire a SweStore yubikey - Please send an email to support@swestore.se and provide the shipping address to where the yubikey should be sent.
- Yubikey has a status as pilot now. It can be changed in the future.
Support
If you have any issues using SweStore please do not hesitate to contact support@swestore.se.
Better not to use filename with single quotes. (There were problems with these but they had been fixed.
Usage monitoring
Supported clients
- iDrop web - Point your Web browser to iweb.swestore.se
- E-iRODS iCommands 3.0 - Command line client Download E-iRODS icommands
SweStore iRODS uses PAM authentication and SweStore yubikeys. With a simple touch of a button, a 44 character one-time password is generated and sent to the system.
Web GUI (iDrop web)
Please see the specific documentation for iDrop web.
Community iRODS version 3.3
The community iRODS client version 3.3 also should work, with PAM authentication.
It is available from SDSC.
Please install the OpenSSL include files and libraries:
$ sudo apt-get install libssl-dev (debian based system) # yum install openssl-devel (redhat-based systems)
Download irods 3.3 from http://irods.sdsc.edu/download.html and unpack the tar.gz archive.
Please enable the following defines in the Makefile iRODS/config/config.mk.in
PAM_AUTH = 1 PAM_AUTH_NO_EXTEND = 1 USE_SSL = 1
Please run irodssetup to compile the irods community client with PAM authentication.
SweStore iRODS usage documentation
To use the system you need to have the E-iRODS command line client installed or using iDROP web.
Command line client
For Linux systems the iRODS command line client is available as an installable package for various Linux platforms from the e-iRODS website downloads section.
The command line client is natural to use for Unix users. There are versions of the usual ls, rm, mv, mkdir, pwd, rsync commands prefixed with an i for iRODS, i.e. irm, imv, imkdir etc.
As expected iput and iget move files to and from the irods system. All these commands print short help when using the -h option.
iCommands environment file
There is an environment file .irodsEnv in the .irods subdirectory of the home directory ($HOME/.irods/.irodsEnv) which contains information where and how to access the iRODS metadata (iCAT) server.
It looks like (placeholders are in <>):
irodsHost 'irods.swestore.se' irodsPort 1247 irodsDefResource 'snicdefResc' irodsHome '/snicZone/proj/<PROJECT_NAME>' irodsCwd '/snicZone/proj/<PROJECT_NAME>' irodsUserName '<USERNAME>' irodsZone 'snicZone' irodsAuthScheme 'PAM'
The iCAT server is irods.swestore.se. The default irods zone name is snicZone. The default resource is snicdefResc. It is best to set the home directory to the same as the project directory, which would be a subdirectory under the /snicZone/proj directory tree.
Yubikey instructions
Prerequisite: A correct iCommands environment file, see above for instructions.
- Insert the yubikey in an available USB-slot in your computer.
- Type iinit
- Touch the conductive surface on the yubikey to send an one-time password to the system.
$ iinit Enter your current PAM (system) password: $ ils /snicZone/proj/<projectname>: $
After that we can use the usual iCommands for 8 hours.
More details on the iCommands are available at https://www.irods.org/index.php/icommands
iCommands
Having initialized the session as described above we can use tie iRODS versions of the basic Unix commands. The project directory is under /snicZone/proj, all members of the project should have write access to this directory. We can use the command
icd /snicZone/proj/projectname
to move to the project directory, or to change to an another project directory when we are members of more than one project.
All commands give short help when invoked with the -h flag.
To put files files into the iRODS system we can use:
iput localfile irodsfile
or, to put a whole directory tree:
iput -r localdirectory irodscollection
To load large amout of data it might be more advantageous to use
irsync -r localdirectory irodscollection
It might be a good idea to use -K so then checksums will be computed, stored and checked.
To create directories (collections in iRODSspeak) we use:
imkdir collection
as it would be expected.
To get those files back we can use
iget irodsfile localfile
or
irsync -r irodscollection localdirectory
To remove files we use:
irm
or
irm -r
Removing files like that would put the files into the trashcan (path: /snicZone/trash/). Time to time we would need to empty the trashcan, using
irmtrash
Using iCommands on SNIC HPC clusters
On SNIC-clusters the icommands command line tools are either available in the PATH or by adding the irods module, e.g.
- module load irods
- If the irods commands are not available at the SNIC HPC cluster, please contact support@swestore.se
We also need to setup the iCommands environment file $HOME/.irods/.irodsEnv
Storage Project directory structure
Your storage project is available at /snicZone/proj/<PROJECT NAME>
/snicZone/home/<USERNAME> is just a small home directory.
iDROP web client
See the iDrop web specific page.
Upstream documentation
Detailed documentation, papers and resources are available from the E-iRODS web site