|
|
| (6 intermediate revisions by 2 users not shown) |
| Line 1: |
Line 1: |
| − | = Swestore iRODS =
| + | #REDIRECT[[Swestore iRODS is decommissioned]] |
| − | Swestore iRODS is distributed across two SNIC centres NSC and PDC. Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple crash of a storage element to losing an entire site while still providing access to the stored data.
| |
| − | | |
| − | = Using Swestore iRODS =
| |
| − | | |
| − | == Basic information ==
| |
| − | | |
| − | For basic information on how to apply for storage or how to become a member of an existing project, please check the main [[Swestore]] page.
| |
| − | | |
| − | == Download and upload data==
| |
| − | | |
| − | === From the command line ===
| |
| − | There are several command line tools capable of using the protocols provided by Swestore iRODS. For interactive usage on SNIC clusters we recommend using the iCommands which should be installed on all SNIC resources. There are too many commands to document fully here.
| |
| − | | |
| − | * Please check full and updated documentation at iRODS official [https://docs.irods.org/master/icommands/user/ Documentation].
| |
| − | * For the command line client, iRODS iCommands, please check iRODS [https://irods.org/download/ Download] page.
| |
| − | ** Please check "Binary Distributions" and instructions for configuring your package manager to include irods APT, YUM or ZYPPER repository at [https://packages.irods.org Repository of iRODS packages ].
| |
| − | *** You can check here an example of [[iRODS iCommands installation on CentOS 7]] from repository.
| |
| − | *** You can check here an example of [[iRODS iCommands installation on Ubuntu 16.04]] from repository (the same procedure is valid for Ubuntu 18.04).
| |
| − | *** You can check here an example of [[iRODS iCommands installation on Ubuntu 20.04]] from repository
| |
| − | *** You can check here an example of [[IRODS iCommands installation on Windows]] from repository.
| |
| − | | |
| − | * Configuration of the icommands requires a file called irods_environment.json to be placed in a subdirectory .irods/ of your home directory (e.g. ~/.irods/irods_environment.json).
| |
| − | You must edit this with a text editor (not MS word, but notepad for Windows, textEdit for Mac or any Unix editor) and put your SNIC username in the place for <YOUR_SNIC_USER_NAME>.:
| |
| − | | |
| − | {
| |
| − | "irods_host": "irods-login.swestore.se",
| |
| − | "irods_port": 2432,
| |
| − | "irods_default_resource": "pdc-gpfs",
| |
| − | "irods_home": "/snic.se/home/<YOUR_SNIC_USER_NAME>",
| |
| − | "irods_cwd": "/snic.se/home/<YOUR_SNIC_USER_NAME>",
| |
| − | "irods_user_name": "<YOUR_SNIC_USER_NAME>",
| |
| − | "irods_zone_name": "snic.se",
| |
| − | "irods_client_server_negotiation": "request_server_negotiation",
| |
| − | "irods_client_server_policy": "CS_NEG_REFUSE",
| |
| − | "irods_encryption_key_size": 32,
| |
| − | "irods_encryption_salt_size": 8,
| |
| − | "irods_encryption_num_hash_rounds": 16,
| |
| − | "irods_encryption_algorithm": "AES-256-CBC",
| |
| − | "irods_default_hash_scheme": "SHA256",
| |
| − | "irods_match_hash_policy": "compatible",
| |
| − | "irods_authentication_scheme": "PAM"
| |
| − | }
| |
| − | | |
| − | First issue
| |
| − | iinit
| |
| − | which will ask for your iRODS password. You can specify a longer time limit
| |
| − | for the session using the <code>--ttl</code> option, <code>iinit -h --ttl</code>
| |
| − | will print help using this. The default generally will suffice.
| |
| − | | |
| − | Use
| |
| − | ils
| |
| − | just to check if you can access your home directory. Please use the project
| |
| − | directory to store files, which will be a subdirectory under
| |
| − | <code>//snic.se/projects/<SNIC_PROJECT_NAME></code>.
| |
| − | | |
| − | To transfer data you can use
| |
| − | <code>irsync</code> like:
| |
| − | irsync -v -r -K local_directory i:irods_directory
| |
| − | where
| |
| − | -v for verbose.
| |
| − | -r for recursive copy of the whole directory tree.
| |
| − | -K to compute and check checksums. This involves higher CPU usage
| |
| − | but gives confidence for the integrity of the data stored.
| |
| − | i.e.:
| |
| − | irsync -r -K mydir i:/snic.se/projects/projectdir
| |
| − | If, for whatever reason the transfer had been interrupted the second
| |
| − | invocation of the command with the same arguments should pick up and
| |
| − | continue from that point.
| |
| − | | |
| − | The <code>ihelp</code> command will print a short description of the iRODS
| |
| − | commands.
| |
| − | | |
| − | === From GUI client ===
| |
| − | Graphical User Interface (GUI) clients are known to work on some operating systems.
| |
| − | | |
| − | ==== Access using WebDAV (map a network drive) ====
| |
| − | [https://en.wikipedia.org/wiki/WebDAV WebDAV] endpoint for Windows 10, Linux and OS X is https://snic-irods-webdav.pdc.kth.se:8443/snic.se/
| |
| − | | |
| − | ===== Mac OS X =====
| |
| − | {| class="wikitable"
| |
| − | !Accessing Swestore-iRODS via WebDAV on Mac OS X
| |
| − | |- style="text-align:center;"
| |
| − | | |
| − | | WebDAV is a method for remotely managing files over the internet. With WebDAV, you can access files stored in Swestore iRODS project by using the same interface as you do with your local files.
| |
| − | Once you connect via WebDAV, and you authenticate, you can view and manage the folders and files stored in your project area. You can perform same actions – drag and drop, rename, delete – that you would on files and folders stored on your own computer.
| |
| − | | |
| − | |-
| |
| − | | This is OS standard WebDAV connection · "Connect to server" function in Finder :
| |
| − | To access Swestore iRODS via WebDAV you'll need to authenticate with your username and password.
| |
| − | 1.Click the 'Finder' icon
| |
| − | [[File:MacOSX Finder.png |none|250px]]
| |
| − | 2.In 'Finder', locate the menu bar at the top and then navigate to Go > Connect to Server.
| |
| − | [[File:ConnectToServer.png |none|250px]]
| |
| − | 3.In the 'Connect to Server' dialog box, enter the URL to Swestore iRODS WebDAV endpoint, in the 'Server Address' field.
| |
| − | | |
| − | 4.Click Connect.
| |
| − | | |
| − | Once the connection is made, you are prompted for your WebDAV username/password.
| |
| − | | |
| − | Once you’ve connected, the WebDAV directory is mounted in your 'Finder' and on your desktop.
| |
| − | | |
| − | You can now start adding, editing, deleting files in this directory from the comfort of your computer.
| |
| − | | |
| − | [[File:Mounted webdav GUI.png |none|350px]]
| |
| − |
| |
| − | |-
| |
| − | | |
| − | | This is an example how to mount with iRODS resource via "mount_webdav" command in your terminal
| |
| − | If you prefer to access your Swestore iRODS files via command line, you can mount the WebDAV-enabled server directory via '''mount_webdav''' command
| |
| − | [[File:MacOSX Terminal WebDAV mount.png |none|350px]]
| |
| − | | |
| − | |}
| |
| − | | |
| − | ===== Windows =====
| |
| − | | |
| − | {| class="wikitable"
| |
| − | !Accessing Swestore-iRODS via WebDAV on Windows
| |
| − | |- style="text-align:center;"
| |
| − | | |
| − | | WebDAV is a method for remotely managing files over the internet. With WebDAV, you can access files stored in Swestore iRODS project by using the same interface as you do with your local files.
| |
| − | Once you connect via WebDAV, and you authenticate, you can view and manage the folders and files stored in your project area. You can perform same actions – drag and drop, rename, delete – that you would on files and folders stored on your own computer.
| |
| − | | |
| − | |-
| |
| − | | How to map the Swestore iRODS to a Windows computer
| |
| − | 1. You should have received an email telling you about your Swestore iRODS username
| |
| − | | |
| − | 2. We’ll use "'''s_user'''" for example
| |
| − | | |
| − | 3. We'll assume that you have set your password at SNIC SUPR for the Swestore iRODS account
| |
| − | | |
| − | 4. Open “File Explorer”
| |
| − | | |
| − | 5. Right-click “Network”
| |
| − | | |
| − | [[File:Screenshot 2020-12-08 at 09.55.18.png |250px]]
| |
| − | | |
| − | 6. Click “Map network drive”
| |
| − | | |
| − | 7. Click the hyperlinked "Connect to a Web site that you can use to store your documents and pictures."
| |
| − | | |
| − | 8. Click "Next" two times (while choosing "Choose a custom network location")
| |
| − | | |
| − | 9. On "Specify the location of your website" enter https://snic-irods-webdav.pdc.kth.se:8443
| |
| − | | |
| − | [[File:Screenshot 2020-12-08 at 10.03.52.png |250px]]
| |
| − | | |
| − | 10. Enter username and password, e.g. s_user and password
| |
| − | | |
| − | [[File:Screenshot 2020-12-08 at 10.04.36.png |250px]]
| |
| − | | |
| − | 11. You can choose to name your new network drive with something meaningful
| |
| − | | |
| − | 11. You should now see the folder mapped to File Explorer!
| |
| − | | |
| − | [[File:Screenshot 2020-12-08 at 10.08.15.png |250px]]
| |
| − |
| |
| − | | |
| − | |}
| |
| − | | |
| − | === From a web browser ===
| |
| − | | |
| − | {| class="wikitable"
| |
| − | !colspan="6"|Davrods is an Apache WebDAV interface to iRODS
| |
| − | |-
| |
| − | |rowspan="3"|[https://github.com/UtrechtUniversity/davrods Davrods]
| |
| − | |Projects
| |
| − | |https://snic-irods-webdav.pdc.kth.se:8443/snic.se/projects/
| |
| − | | To browse private data you'll need to authenticate with your username and password.
| |
| − | |rowspan="3"|This interface is intended as '''simple directory index''' interface. It's purpose is to list, view, download, publish, share data. It is NOT intended to upload data.
| |
| − | [[File:Swestore-iRODS simple webdav dir listing.png |center |thumb|350px]]
| |
| − | |-
| |
| − | |Home
| |
| − | |https://snic-irods-webdav.pdc.kth.se:8443/snic.se/home/
| |
| − | | To browse private data you'll need to authenticate with your username and password.
| |
| − | |-
| |
| − | |Public
| |
| − | |http://snic-irods-webdav.pdc.kth.se:8888/snic.se/home/public/
| |
| − | | This web interface is used to share data with "world" and no authentication is required. <pre style="color: red">NOTE: different port 8888 </pre>
| |
| − | | |
| − | |-
| |
| − | | [https://github.com/irods-contrib/metalnx-web Metalnx]
| |
| − | | Login
| |
| − | | https://snic-irods-web.pdc.kth.se/emc-metalnx-web/login/
| |
| − | |To access SNIC iRODS Zone you'll need to authenticate with your username and password.
| |
| − | | Metalnx is a web application designed to work alongside iRODS. It is a graphical user interface and serves as a client that authenticates to an existing iRODS Zone.
| |
| − | [[File:Screenshot 2020-03-04 at 14.09.10.png |center |thumb|350px]]
| |
| − | | |
| − | |}
| |
| − | | |
| − | = Share your data =
| |
| − | | |
| − | If you want to make some of your project folders world readable (i.e. make them available for download for external collaborators who don't have Swestore account), you can follow two different approaches. The first one is to use Metalnx graphical interface; the second is to use command line interface.
| |
| − | | |
| − | '''In both cases we are giving [[read]] permissions to [[anonymous]] user!'''
| |
| − | | |
| − | '''Let's assume you want to share your "sharefolder" with your external collaborators'''
| |
| − | [vitlacil@t04n27 ~]$ ils /snic.se/projects/operations/sharefolder/
| |
| − | /snic.se/projects/operations/sharefolder:
| |
| − | FitSM-0_Overview_and_vocabulary.pdf
| |
| − | FitSM-1_Requirements.pdf
| |
| − | FitSM_Advanced_Training_SOC_V2.5.pdf
| |
| − | FitSM_Advanced_Training_SPD_V2.5.pdf
| |
| − | FitSM_Advanced_Training_SPD_V2.5_0.pptx
| |
| − | FitSM_Expert_Training_V1.4.pdf
| |
| − | FitSM_Foundation_Training_V2.10.pdf
| |
| − | FitSM_Foundation_Training_V2.10.pptx
| |
| − | FitSM_Foundation_Training_V2.11.pdf
| |
| − | FitSM_Guide_Identifying_Services_v1.0.pdf
| |
| − | FitSM_Sample_Corporate_SLA_v1.0.docx
| |
| − | FitSM_Sample_Service_Portfolio_Catalogue_v2.0_final.xlsx
| |
| − | FitSM_Template_SLA_1.0.docx
| |
| − | FitSM_Teplate_Basic_Service_Report_1.0.docx
| |
| − | C- /snic.se/projects/operations/sharefolder/README
| |
| − | C- /snic.se/projects/operations/sharefolder/test
| |
| − | [vitlacil@t04n27 ~]$
| |
| − | | |
| − | * '''Use Metalnx, web application graphical interface, to change permissions on the folder'''
| |
| − | 1. Go to Metalnx login page and authenticate -> https://snic-irods-web.pdc.kth.se/emc-metalnx-web/login/
| |
| − | | |
| − | 2. Navigate to folder to share and select "Permissions" from dropdown box at the end of the path
| |
| − | | |
| − | N.B. When under heavy load, web interface can be slow. Please have patience, we are looking into this.
| |
| − | | |
| − | See picture:
| |
| − | | |
| − | [[File:Folder sharing.png |350px]] | |
| − | | |
| − | 3. Now you can add '''read''' permission to '''anonymous''' user. Consider also if you are going to apply it to subcollections and files
| |
| − | | |
| − | [[File:Folder Permissions.png |350px]]
| |
| − | | |
| − | 4. Now your folder permissions should list read permission to anonymous user
| |
| − | | |
| − | [[File:Permissions listing.png |350px]]
| |
| − | | |
| − | 4. Now your files are visible and downloadable via http://snic-irods-webdav.pdc.kth.se:8888/snic.se/projects/operations/sharefolder
| |
| − | <pre style="color: red">NOTE: different port 8888 </pre>
| |
| − | | |
| − | | |
| − | [[File:List of shared files.png |350px]]
| |
| − | | |
| − | * '''Use command line interface to change permission on folder'''
| |
| − | | |
| − | [vitlacil@t04n27 ~]$ ichmod -r read anonymous /snic.se/projects/operations/sharefolder
| |
| − | | |
| − | = Publish your data =
| |
| − | | |
| − | For publishing your data via Swestore iRODS, please get in touch with Swestore support at [https://supr.snic.se/support/ Swestore Support].
| |
| − | | |
| − | = Transfer your data =
| |
| − | | |
| − | A good way to transfer data from and to Swestore iRODS is to use the [https://docs.irods.org/master/icommands/user/#irsync irsync] command to synchronise a local directory with iRODS (similar to the Unix rsync command). It can be used to make an exact copy of a directory hierarchy on a local disk within iRODS, or retrieve an exact copy of a directory hierarchy already stored in iRODS. It may also be used to create an exact copy of a file or directory within iRODS.
| |
| − | | |
| − | NOTE: iRODS paths are identified with an i: prefix in the irsync command.
| |
| − | | |
| − | For example, if you have created a directory within iRODS called “/snic.se/projects/myproject/data”, and you wish to retrieve an exact copy of that “data” directory on your local filesystem, run the command:
| |
| − | $ irsync -vrK i:/snic.se/projects/myproject/data /tmp/data
| |
| − | | |
| − | = Using Swestore iRODS at PDC =
| |
| − | | |
| − | * [https://www.pdc.kth.se/support/documents/data_management/swestore.html PDC Support pages for Swestore (iRODS)]
| |
| − | * [https://www.pdc.kth.se/support/documents/data_management/file_transfer_swestore.html PDC Support pages about Swestore (iRODS) client on PDC transfer node]
| |
| − | | |
| − | = Using Swestore iRODS at NSC =
| |
| − | | |
| − | [https://snicdocs.nsc.liu.se/wiki/Swestore-iRODS-at-NSC Here you can read more about the NSC specific aspects]
| |
