|
(Tag: Redirect target changed) |
| (19 intermediate revisions by 6 users not shown) |
| Line 1: |
Line 1: |
| − | = National Storage using iRODS =
| + | #REDIRECT[[Swestore iRODS is decommissioned]] |
| − | [[Category:Storage]]
| |
| − | [[Category:SweStore]]
| |
| − | | |
| − | == Getting access ==
| |
| − | ; Apply for storage
| |
| − | : Please follow the instructions on the [[Apply for storage on SweStore]] page.
| |
| − | | |
| − | ;iRODS user authentication
| |
| − | :SweStore's iRODS system uses [http://www.yubico.com/products/yubikey-hardware/yubikey/ Yubikey] one-time passwords (OTP). With a simple touch of a button, a 44 character one-time password is generated and sent to the system. The user will be provided with a SweStore yubikey.
| |
| − | :Yubikey has a status as pilot now. It can be changed in the future.
| |
| − | | |
| − | ; iRODS usage <span style="color:#FF0000"> Pilot. </span><br> - How to acquire a SweStore yubikey
| |
| − | :Please send an email to [mailto:support@swestore.se?subject=Yubikey support@swestore.se] and provide the shipping address to where the yubikey should be sent.<br>
| |
| − | :Yubikey has a status as pilot now. It can be changed in the future.
| |
| − | | |
| − | == Support ==
| |
| − | | |
| − | If you have any issues using SweStore please do not hesitate to contact [mailto:support@swestore.se support@swestore.se].
| |
| − | | |
| − | <span style="color:#FF0000"> Be careful with filenames. Don't use filename with '. </span><br>
| |
| − | <span style="color:#FF0000"> We have memory issues with many files or bigfiles transferring now. Be careful when moving files bigger than 50GB and 20000 files at one time. </span>
| |
| − | | |
| − | == Usage monitoring ==
| |
| − | * [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]
| |
| − | | |
| − | == Supported clients ==
| |
| − | | |
| − | : iDrop web - Point your Web browser to [https://iweb.swestore.se iweb.swestore.se]
| |
| − | : E-iRODS iCommands 3.0 - Command line client [ftp://ftp.renci.org/pub/eirods/releases/3.0 Download E-iRODS icommands]
| |
| − | | |
| − | SweStore iRODS uses PAM authentication and SweStore yubikeys. With a simple touch of a button, a 44 character one-time password is generated and sent to the system.
| |
| − | | |
| − | | |
| − | === Web GUI (iDrop web) ===
| |
| − | Please see the specific documentation for [[iDrop web]].
| |
| − | | |
| − | === Community iRODS version 3.3 ===
| |
| − | The community iRODS client version 3.3 also should work, with PAM authentication.<br>
| |
| − | Please install the OpenSSL include files and libraries:
| |
| − | <pre>
| |
| − | $ sudo apt-get install libssl-dev (debian based system)
| |
| − | # yum install openssl-devel (redhat-based systems)
| |
| − | </pre>
| |
| − | Download irods 3.3 from https://www.irods.org/index.php/Downloads and unpack the tar.gz archive.
| |
| − | | |
| − | Please enable the following defines in the Makefile iRODS/config/config.mk.in
| |
| − | <pre>
| |
| − | PAM_AUTH = 1
| |
| − | PAM_AUTH_NO_EXTEND = 1
| |
| − | USE_SSL = 1
| |
| − | </pre>
| |
| − | Please run irodssetup to compile the irods community client with PAM authentication.
| |
| − | | |
| − | == SweStore iRODS usage documentation ==
| |
| − | | |
| − | To use the system you need to have the E-iRODS command line client installed or using iDROP web.
| |
| − | | |
| − | === Command line client ===
| |
| − | | |
| − | For Linux systems the iRODS command line client is available as an installable package for various
| |
| − | Linux platforms from the e-iRODS website downloads section.
| |
| − | | |
| − | The command line client is natural to use for Unix users.
| |
| − | There are versions of the usual ls, rm, mv, mkdir, pwd, rsync
| |
| − | commands prefixed with an i for iRODS, i.e. irm, imv, imkdir etc.
| |
| − | | |
| − | As expected iput and iget move files to and from the irods system.
| |
| − | All these commands print short help when using the -h option.
| |
| − | | |
| − | ==== iCommands environment file ====
| |
| − | | |
| − | There is an environment file .irodsEnv in the .irods subdirectory
| |
| − | of the home directory ($HOME/.irods/.irodsEnv) which contains information where and how
| |
| − | to access the iRODS metadata (iCAT) server.
| |
| − | | |
| − | It looks like (placeholders are in <>):
| |
| − | <pre>
| |
| − | irodsHost 'irods.swestore.se'
| |
| − | irodsPort 1247
| |
| − | irodsDefResource 'snicdefResc'
| |
| − | irodsHome '/snicZone/proj/<PROJECT_NAME>'
| |
| − | irodsCwd '/snicZone/proj/<PROJECT_NAME>'
| |
| − | irodsUserName '<USERNAME>'
| |
| − | irodsZone 'snicZone'
| |
| − | irodsAuthScheme 'PAM'
| |
| − | </pre>
| |
| − | | |
| − | The iCAT server is irods.swestore.se.
| |
| − | The default irods zone name is snicZone.
| |
| − | The default resource is snicdefResc.
| |
| − | It is best to set the home directory to the same as the
| |
| − | project directory, which would be a subdirectory under
| |
| − | the /snicZone/proj directory tree.
| |
| − | | |
| − | ==== Yubikey instructions ====
| |
| − | | |
| − | Prerequisite: A correct iCommands environment file, see above for instructions.
| |
| − | | |
| − | # Insert the yubikey in an available USB-slot in your computer.
| |
| − | # Type iinit
| |
| − | # Touch the conductive surface on the yubikey to send an one-time password to the system.
| |
| − | | |
| − | <pre>
| |
| − | | |
| − | $ iinit
| |
| − | Enter your current PAM (system) password:
| |
| − | $ ils
| |
| − | /snicZone/proj/<projectname>:
| |
| − | $
| |
| − | </pre>
| |
| − | | |
| − | After that we can use the usual iCommands for 8 hours.
| |
| − | | |
| − | More details on the iCommands are available at
| |
| − | https://www.irods.org/index.php/icommands
| |
| − | | |
| − | ==== iCommands ====
| |
| − | | |
| − | Having initialized the session as described above we can use tie iRODS versions
| |
| − | of the basic Unix commands. The project directory is under /snicZone/proj, all
| |
| − | members of the project should have write access to this directory. We can use
| |
| − | the command
| |
| − | <pre>
| |
| − | icd /snicZone/proj/projectname
| |
| − | </pre>
| |
| − | to move to the project directory, or to change to an another project directory
| |
| − | when we are members of more than one project.
| |
| − | | |
| − | All commands give short help when invoked with the -h flag.
| |
| − | | |
| − | To put files files into the iRODS system we can use:
| |
| − | <pre>
| |
| − | iput localfile irodsfile
| |
| − | </pre>
| |
| − | or, to put a whole directory tree:
| |
| − | <pre>
| |
| − | iput -r localdirectory irodscollection
| |
| − | </pre>
| |
| − | | |
| − | To load large amout of data it might be more advantageous to use
| |
| − | <pre>
| |
| − | irsync -r localdirectory irodscollection
| |
| − | </pre>
| |
| − | It might be a good idea to use -K so then checksums will be computed,
| |
| − | stored and checked.
| |
| − | | |
| − | To create directories (collections in iRODSspeak) we use:
| |
| − | <pre>
| |
| − | imkdir collection
| |
| − | </pre>
| |
| − | as it would be expected.
| |
| − | | |
| − | To get those files back we can use
| |
| − | <pre>
| |
| − | iget irodsfile localfile
| |
| − | </pre>
| |
| − | or
| |
| − | <pre>
| |
| − | irsync -r irodscollection localdirectory
| |
| − | </pre>
| |
| − | | |
| − | To remove files we use:
| |
| − | <pre>
| |
| − | irm
| |
| − | </pre>
| |
| − | or
| |
| − | <pre>
| |
| − | irm -r
| |
| − | </pre>
| |
| − | | |
| − | Removing files like that would put the files into the trashcan (path: /snicZone/trash/).
| |
| − | Time to time we would need to empty the trashcan, using
| |
| − | <pre>
| |
| − | irmtrash
| |
| − | </pre>
| |
| − | | |
| − | ==== Using iCommands on SNIC HPC clusters ====
| |
| − | | |
| − | On SNIC-clusters the icommands command line tools are either available in the PATH or by adding the irods module, e.g.
| |
| − | : module load irods
| |
| − | :If the irods commands are not available at the SNIC HPC cluster, please contact support@swestore.se
| |
| − | We also need to setup the iCommands environment file $HOME/.irods/.irodsEnv
| |
| − | | |
| − | === Storage Project directory structure ===
| |
| − | | |
| − | Your storage project is available at /snicZone/proj/<PROJECT NAME>
| |
| − | | |
| − | /snicZone/home/<USERNAME> is just a small home directory.
| |
| − | | |
| − | === iDROP web client ===
| |
| − | | |
| − | See the [[iDrop web]] specific page.
| |
| − | | |
| − | === Upstream documentation ===
| |
| − | Detailed documentation, papers and resources are available from
| |
| − | the [http://www.eirods.org E-iRODS web site]
| |
| − | | |
| − | [http://www.irods.org Community iRODS]
| |
| − | | |
| − | [https://groups.google.com/d/forum/irod-chat User forum]
| |
