http://docs.snic.se/w/api.php?action=feedcontributions&feedformat=atom&user=Krishnaveni+Chitrapu+%28NSC%29 2026-05-02T01:59:21Z User contributions MediaWiki 1.31.10 http://docs.snic.se/w/index.php?title=User:Krishnaveni_Chitrapu_(NSC)&diff=7732 2021-03-01T15:05:30Z

<p>Krishnaveni Chitrapu (NSC): </p> <hr /> <div>{{systems expert info<br /> |first name=Krishnaveni<br /> |last name=C<br /> |centre=NSC<br /> |fields=Software development<br /> |general activities=dCache development.<br /> &lt;!-- |image=name of an uploaded image file<br /> |office=Semicolon separated lines; of visiting address; newlines<br /> are ignored<br /> |phone=Semicolon separated phone numbers (with optional <br /> explanation); newlines are ignored--&gt;<br /> }}<br /> <br /> == Quick facts ==<br /> * System Developer at NSC<br /> * Programming Experience<br /> <br /> == Expertise ==<br /> * [[expertise::JAVA]]<br /> * [[expertise::J2EE]]<br /> * [[expertise::PL/SQL]]<br /> * [[expertise::Python]]<br /> <br /> == Projects ==<br /> * [[project::BILS websearch]]<br /> * [[project::Fido]]<br /> * [[project::Integration of SUPR with Swestore (dCache and iRODS)]]<br /> * [[project::NDGF - NT1 (dCache Developer)]]</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=User:Krishnaveni_Chitrapu_(NSC)&diff=7720 2021-02-17T14:03:11Z

<p>Krishnaveni Chitrapu (NSC): </p> <hr /> <div>{{systems expert info<br /> |first name=Krishnaveni<br /> |last name=C<br /> |centre=NSC<br /> |fields=Software development<br /> |general activities=dCache development.<br /> &lt;!-- |image=name of an uploaded image file<br /> |office=Semicolon separated lines; of visiting address; newlines<br /> are ignored<br /> |phone=Semicolon separated phone numbers (with optional <br /> explanation); newlines are ignored--&gt;<br /> }}<br /> <br /> == Quick facts ==<br /> * System Developer at NSC<br /> * Programming Experience<br /> <br /> == Expertise ==<br /> * [[expertise::JAVA]]<br /> * [[expertise::J2EE]]<br /> * [[expertise::PL/SQL]]<br /> * [[expertise::Python]]<br /> <br /> == Projects ==<br /> * [[project::BILS websearch]]<br /> * [[project::Fido]]<br /> * [[project::Integration of SUPR with Swestore (dCache and iRODS)]]</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=User:Krishnaveni_Chitrapu_(NSC)&diff=7719 2021-02-17T13:15:51Z

<p>Krishnaveni Chitrapu (NSC): </p> <hr /> <div>{{systems expert info<br /> |first name=Krishnaveni<br /> |last name=Chitrapu<br /> |centre=NSC<br /> |fields=Software development<br /> |general activities=dCache development.<br /> &lt;!-- |image=name of an uploaded image file<br /> |office=Semicolon separated lines; of visiting address; newlines<br /> are ignored<br /> |phone=Semicolon separated phone numbers (with optional <br /> explanation); newlines are ignored--&gt;<br /> }}<br /> <br /> == Quick facts ==<br /> * System Developer at NSC<br /> * Programming Experience<br /> <br /> == Expertise ==<br /> * [[expertise::JAVA]]<br /> * [[expertise::J2EE]]<br /> * [[expertise::PL/SQL]]<br /> * [[expertise::Python]]<br /> <br /> == Projects ==<br /> * [[project::BILS websearch]]<br /> * [[project::Fido]]<br /> * [[project::Integration of SUPR with Swestore (dCache and iRODS)]]</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=User:Krishnaveni_Chitrapu_(NSC)&diff=7718 2021-02-17T13:15:26Z

<p>Krishnaveni Chitrapu (NSC): </p> <hr /> <div>{{systems expert info<br /> |first name=Krishnaveni<br /> |last name=Chitrapu<br /> |centre=NSC<br /> |fields=Software development<br /> |general activities=dCache development.<br /> &lt;!-- |image=name of an uploaded image file<br /> |office=Semicolon separated lines; of visiting address; newlines<br /> are ignored<br /> |phone=Semicolon separated phone numbers (with optional <br /> explanation); newlines are ignored--&gt;<br /> }}<br /> <br /> == Quick facts ==<br /> * System Developer at NSC<br /> * Programming Experience<br /> <br /> == Expertise ==<br /> * [[expertise::JAVA]]<br /> * [[expertise::J2EE]]<br /> * [[expertise::PL/SQL]]<br /> * [[expertise::Python]]<br /> <br /> == Projects ==<br /> * [[project::BILS websearch]]<br /> * [[project::Fido]]<br /> * [[Integration of SUPR with Swestore (dCache and iRODs)]]</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=Swestore-dCache&diff=6966 2018-12-14T10:58:07Z

<p>Krishnaveni Chitrapu (NSC): /* Apply for storage (for the PI) */</p> <hr /> <div>[[Category:Storage]]<br /> [[Category:Swestore]]<br /> [[Category:Swestore]]<br /> [[Category:Swestore user guide]]<br /> <br /> Swestore is Research Data Storage Infrastructure operated by the Swedish National Infrastructure for Computing (SNIC). <br /> <br /> The resources provided by Swestore are made available through open procedures such that the best Swedish research is supported and new research is facilitated. The purpose of Swestore allocations, granted by Swedish National Allocations Committee (SNAC), is to provide large scale data storage for “live” or “working” research data, also known as active research data. <br /> <br /> Due to the VR funding, the free allocations on Swestore have some usage limitations.<br /> * Swestore is NOT supposed to be used for backups and such requests for allocation will be rejected. Please, check with your university (home institution/organisation) IT department about backup services, strategies and policies in place. If such services do not exist or if you can’t access them for different reasons, please contact us at support@swestore.se;<br /> * Swestore is NOT supposed to be used as archiving service, long-term storage or repository for “static” data. Once data is no longer in the process of change, and decision on which data should be retained shared and/or preserved has been taken, data should be moved toward appropriate data services. The higher education institutions are responsible for archiving and long-term preservation of research data produced by researchers employed by them;<br /> <br /> ; Glossary<br /> ; Active (Research) data<br /> : is data that is being worked on as part of research project and therefore subject to change. The files containing data will need to be accessed and amended or updated as new data is gathered or processed.<br /> ; Static (Research) data: is data that is no longer in the process of change and it can be prepared for preservation and reuse.<br /> ; Backup<br /> : is a copy of the digital data to be stored and used as a replacement in case the main copy is either deleted or corrupted.<br /> ; Archive<br /> :is a service to record, organise, and store (digital) items in optimal conditions, with standardised labelling to ensure their longevity and continued access. The service is based on application of metadata, archiving policies, records management, and digital preservation actions. Archivists make decisions on selection and retention of items which are usually governed by supporting policies.<br /> <br /> <br /> Swestore is distributed across the SNIC centres [http://www.c3se.chalmers.se/ C3SE], [http://www.hpc2n.umu.se/ HPC2N], [http://www.lunarc.lu.se/ Lunarc], [http://www.nsc.liu.se/ NSC] and [http://www.uppmax.uu.se Uppmax]. Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple crash of a storage element to losing an entire site while still providing access to the stored data. <br /> <br /> One of the major advantages to the distributed nature of Swestore is the excellent aggregated transfer rates possible. This is achieved by bypassing a central node and having transfers going directly to/from the storage elements if the selected transfer protocol allows it. Swestore can achieve aggregated transfer rates in excess of 100 Gigabit per second, but in practice this is limited by connectivity to the end user, each university or a limited number of files (typically max 1 Gbit/s per file/connection).<br /> <br /> To protect against silent data corruption the dCache storage system checksums all stored data and periodically verifies the data using this checksum.<br /> <br /> The dCache system does NOT yet provide protection against user errors like inadvertent file deletions.<br /> <br /> = Getting access to Swestore=<br /> These steps are required for all PI:s and users of Swestore regardless of how the storage area later is accessed.<br /> <br /> ==Apply for storage (for the PI)==<br /> All normal Swestore projects are managed using the SNIC User and Project Repository portal, SUPR. Please follow the instructions on the [http://www.snic.se/resources/swestore/ Apply for storage on Swestore] page to apply for storage.<br /> <br /> ==Acquire an eScience client certificate (for all users)==<br /> Swestore user authentication is currently done by user/client certificates. A valid certificate is needed by all project members.<br /> <br /> If you already have a valid grid or eScience certificate you don't need to acquire another one.<br /> <br /> Acquire a certificate by following the ''Requesting a certificate'' instruction on the [[Grid certificates#Requesting a certificate|certificate documentation page]]. This step can be performed while waiting for the storage application to be approved.<br /> <br /> ==Manage your eScience client certificate (for all users)==<br /> There are multiple suppliers of certificates. Depending on the supplier this step varies a bit.<br /> :; For Digicert certificates<br /> :: If intending to access Swestore from a SNIC resource, please make sure you also [[Exporting_a_client_certificate|export the certificate]], transfer it to the intended SNIC resource and [[Preparing_a_client_certificate|prepare it for use with grid tools]].<br /> :; For Nordugrid certificates<br /> :: Please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br /> <br /> ==Register your eScience client certificate in SUPR (for all users)==<br /> <br /> All project members '''have''' to register in SUPR and be added to the approved project by the PI. '''All users also have to register their certificate in SUPR.''' This information is used by Swestore to authenticate the users when accessing the storage area. Registering the certificate is easy though. Make sure your certificate is stored in your browser, log in to [https://supr.snic.se SUPR ], click &quot;Personal Information&quot; in the left menu, click &quot;Register Client Certificate&quot; and follow the instructions. Please wait for up to 10 minutes for this information to be distributed to Swestore.<br /> <br /> = Using Swestore =<br /> == Download and upload data ==<br /> ===From the command line===<br /> <br /> There are several command line tools capable of using the protocols provided by Swestore. For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br /> <br /> As an integration point for building scripts and automated systems we suggest using the curl program and library.<br /> <br /> :; ARC<br /> :: Please see the instructions for [[Accessing Swestore with the ARC client]]. '''Recommended method when logged in on SNIC resources'''.<br /> :; lftp<br /> :: See the instructions for [[Accessing Swestore with lftp]].<br /> :; cURL<br /> :: Please see the instructions for [[Accessing Swestore with cURL]]. We suggest using this as integration point for building scripts and automated systems.<br /> :; globus-url-copy<br /> :: Please see the instructions for [[Accessing Swestore with globus-url-copy]].<br /> :; gfal2<br /> :: (not documented yet)<br /> :; https://duck.sh/<br /> :: Command-line client for Windows/macOS (not documented yet)<br /> <br /> === Using a GUI client ===<br /> <br /> Graphical User Interface (GUI) clients are known to work on some operating systems.<br /> <br /> :; Cyberduck<br /> :: Please see the instructions for [[Accessing Swestore using Cyberduck]] '''Recommended method on Windows and macOS'''<br /> <br /> === From a web browser ===<br /> Swestore is accessible in your web browser as a simple directory index interface at https://webdav.swestore.se/. To browse private data you need to have your certificate installed in your browser (see above). Projects are organized under the &lt;code&gt;/snic&lt;/code&gt; directory as &lt;code&gt;&lt;nowiki&gt;https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/&lt;/nowiki&gt;&lt;/code&gt;.<br /> <br /> == Enabled access protocols ==<br /> A design criteria for Swestore is to provide the storage over a number of standardized and public protocols. There is no vendor specific client needed for access. <br /> <br /> :; GridFTP<br /> :: Also called gsiftp. Well supported within Swestore.<br /> :: Features: Transfer checksums. Writes goes directly to storage pools for high speed transfers.<br /> :: Access URL: gsiftp://gsiftp.swestore.se/<br /> :; HTTP/WebDAV<br /> :: Contender for being the recommended protocol for Swestore.<br /> :: Features: Direct support in web browsers. Redirected reads and writes if special url is used and client supports it.<br /> :: Access URL: http://webdav.swestore.se/ (unauthenticated, read-only, non-redirected)<br /> :: Access URL: https://webdav.swestore.se/ (authenticated, read-write, non-redirected)<br /> :: Access URL: http://webdav.swestore.se:1080/ (unauthenticated, read-only, redirected)<br /> :: Access URL: https://webdav.swestore.se:1443/ (authenticated, read-write, redirected)<br /> :; NFSv4.1<br /> :: Used by some communities, not recommended for general use<br /> :; SRM - Storage Resource Manager<br /> :: Used by some communities, not recommended for general use.<br /> :: Access URL: srm://srm.swegrid.se/<br /> :; Xrootd<br /> :: Used by some communities, not recommended for general use<br /> :; DCAP<br /> :: Used by some communities, not recommended for general use<br /> <br /> == Tools and scripts ==<br /> <br /> There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br /> <br /> * [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer<br /> * Transfer script, [[SweStore/swetrans_arc|swetrans_arc]], provided by Adam Peplinski / Philipp Schlatter<br /> * [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br /> <br /> == Slides and more ==<br /> <br /> [http://docs.snic.se/wiki/Swestore/Lund_Seminar_Apr18 Slides and material from seminar for Lund users on April 18th]<br /> <br /> == Usage monitoring ==<br /> * On the project page in SUPR: https://supr.snic.se<br /> * On the monitoring server for Swestore: http://status.swestore.se/munin/monitor/monitor/ (only accessible from .se-domains.<br /> <br /> = Accessing SUPR-TEST System and test-irods System =<br /> For projects who would like to test irods and how to create irods projects in SUPR-TEST System. &lt;br&gt;<br /> Detailed Instructions can be found at<br /> https://snicdocs.nsc.liu.se/wiki/SNIC_storage/supr_test_irods<br /> <br /> = Support =<br /> <br /> If you have any issues using Swestore, please do not hesitate to contact [mailto:support@swestore.se support@swestore.se].</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=SNIC_storage/supr_test_irods&diff=6836 2017-09-29T13:41:39Z

<p>Krishnaveni Chitrapu (NSC): /* Using SUPR-Test System for projects */</p> <hr /> <div>=== Using SUPR-Test System for projects ===<br /> <br /> The SUPR-Test System can be accessed via<br /> <br /> https://disposer.c3se.chalmers.se/supr-test<br /> <br /> If its a new person, click on Register New Person Button &lt;br&gt;<br /> <br /> One can<br /> <br /> 1) Register via SWAMID<br /> <br /> 2) Register without SWAMID<br /> <br /> If you are a PI, login to SUPR-Test<br /> <br /> 1) Click on Rounds on the left Panel and select MAX IV Swestore 2017.<br /> <br /> 2) Click on Create new Proposal for MAX IV Swestore 2017 button.<br /> <br /> 3) Give a project title and Create proposal.Alternatively you can clone an existing project from the dropdown.<br /> <br /> 5) The directory name for the project should be simple and not contain any spaces.It would the project/directory name in irods<br /> <br /> 6) PI can add Co-Investigators to the proposal. Alternatively they can be added as members once the proposal is approved.<br /> <br /> 7) Proposal is submitted and once approved a project is created.<br /> <br /> 8) A Python script is run in the background which picks the project and user data and creates corresponding project and users in test-irods.<br /> <br /> 9) User can follow the below instructions to access test-irods.<br /> <br /> === Client setup to use the test-irods system ===<br /> <br /> The iRODS system 'test-irods' is a test system we use to <br /> test upgrades and functionality. It a collection of<br /> Xen virtual machines running CentOS 6.9 and iRODS 4.1.8<br /> at the moment. There is one iCAT server and two iRODS<br /> storage server nodes.<br /> <br /> To access the system it is preferable to use the same<br /> version of the client as the server but theoretically<br /> it should also work with other versions of the client as<br /> well.<br /> <br /> Easiest and most tested is the Unix command line client.<br /> On a cluster like Aurora most probably this is going to<br /> be the one to be used.<br /> <br /> <br /> ==== Installing the iRODS iCommands client ====<br /> <br /> To install the latest 4.2.1 command line client the<br /> instructions are on the iRODS web site at:<br /> https://irods.org/download/<br /> and<br /> https://packages.irods.org/<br /> .<br /> <br /> On a CentOS Linux variant as root:<br /> Get the signing key.<br /> rpm --import https://packages.irods.org/irods-signing-key.asc<br /> <br /> Get the repository description file.<br /> wget -qO - https://packages.irods.org/renci-irods.yum.repo &gt;/etc/yum.repos.d/renci-irods.yum.repo<br /> <br /> Install the package.<br /> yum install irods-icommands<br /> It will ask to confirm to import the iRODS package signing key<br /> and then ask to confirm going ahead with installing the package<br /> and the dependencies. The command line client executables will go<br /> under &lt;code&gt;/bin&lt;/code&gt;.<br /> <br /> The iRODS client will need to be able to talk to the iRODS iCAT server<br /> and the resource servers at ports 1247 and 20000..20199. (Those are<br /> the defaults.) Firewall / packet filtering should allow outgoing connections<br /> to those ports.<br /> <br /> <br /> ==== Configuring the iRODS iCommands client ====<br /> <br /> To be able to use the iRODS command line client a user has to have<br /> a configuration file in place which describes how to access the<br /> iCAT server. The file has &lt;code&gt;json&lt;/code&gt; format (with the latest<br /> versions.)<br /> <br /> An example file would look like this:<br /> &lt;pre&gt;<br /> {<br /> &quot;irods_host&quot;: &quot;&lt;irodshost&gt;&quot;,<br /> &quot;irods_port&quot;: &lt;port&gt;,<br /> &quot;irods_default_resource&quot;: &quot;&lt;resourcename&gt;&quot;,<br /> &quot;irods_home&quot;: &quot;/&lt;zonename&gt;/home/&lt;username&gt;&quot;,<br /> &quot;irods_cwd&quot;: &quot;/&lt;zonename&gt;/home/&lt;username&gt;&quot;,<br /> &quot;irods_user_name&quot;: &quot;&lt;username&gt;&quot;,<br /> &quot;irods_zone_name&quot;: &quot;&lt;zonename&gt;&quot;,<br /> &quot;irods_client_server_negotiation&quot;: &quot;request_server_negotiation&quot;,<br /> &quot;irods_client_server_policy&quot;: &quot;CS_NEG_REFUSE&quot;,<br /> &quot;irods_encryption_key_size&quot;: 32,<br /> &quot;irods_encryption_salt_size&quot;: 8,<br /> &quot;irods_encryption_num_hash_rounds&quot;: 16,<br /> &quot;irods_encryption_algorithm&quot;: &quot;AES-256-CBC&quot;,<br /> &quot;irods_default_hash_scheme&quot;: &quot;SHA256&quot;,<br /> &quot;irods_match_hash_policy&quot;: &quot;compatible&quot;,<br /> &quot;irods_server_control_plane_port&quot;: 1248,<br /> &quot;irods_server_control_plane_key&quot;: &quot;TEMPORARY__32byte_ctrl_plane_key&quot;,<br /> &quot;irods_server_control_plane_encryption_num_hash_rounds&quot;: 16,<br /> &quot;irods_server_control_plane_encryption_algorithm&quot;: &quot;AES-256-CBC&quot;,<br /> &quot;irods_maximum_size_for_single_buffer_in_megabytes&quot;: 32,<br /> &quot;irods_default_number_of_transfer_threads&quot;: 4,<br /> &quot;irods_transfer_buffer_size_for_parallel_transfer_in_megabytes&quot;: 4<br /> }<br /> &lt;/pre&gt;<br /> Names in the angle brackets are placeholders to be replaced with the<br /> actual values (without the angle brackets as in the file below).<br /> irodshost is the fully qualified name of the irods host (with the dots).<br /> port is the port the iCAT server is using, the default is 1247<br /> resourcename is the name of the default resource to be used<br /> zonename is the name of the iRODS zone<br /> If not sure about these please ask a system administrator.<br /> <br /> In out case:<br /> &lt;pre&gt;<br /> {<br /> &quot;irods_host&quot;: &quot;test-irods.nsc.liu.se&quot;,<br /> &quot;irods_port&quot;: 1247,<br /> &quot;irods_default_resource&quot;: &quot;demoResc&quot;,<br /> &quot;irods_home&quot;: &quot;/tempZone/home/s_jonli&quot;,<br /> &quot;irods_cwd&quot;: &quot;/tempZone/home/s_jonli&quot;,<br /> &quot;irods_user_name&quot;: &quot;s_jonli&quot;,<br /> &quot;irods_zone_name&quot;: &quot;tempZone&quot;,<br /> &quot;irods_client_server_negotiation&quot;: &quot;request_server_negotiation&quot;,<br /> &quot;irods_client_server_policy&quot;: &quot;CS_NEG_REFUSE&quot;,<br /> &quot;irods_encryption_key_size&quot;: 32,<br /> &quot;irods_encryption_salt_size&quot;: 8,<br /> &quot;irods_encryption_num_hash_rounds&quot;: 16,<br /> &quot;irods_encryption_algorithm&quot;: &quot;AES-256-CBC&quot;,<br /> &quot;irods_default_hash_scheme&quot;: &quot;SHA256&quot;,<br /> &quot;irods_match_hash_policy&quot;: &quot;compatible&quot;,<br /> &quot;irods_server_control_plane_port&quot;: 1248,<br /> &quot;irods_server_control_plane_key&quot;: &quot;TEMPORARY__32byte_ctrl_plane_key&quot;,<br /> &quot;irods_server_control_plane_encryption_num_hash_rounds&quot;: 16,<br /> &quot;irods_server_control_plane_encryption_algorithm&quot;: &quot;AES-256-CBC&quot;,<br /> &quot;irods_maximum_size_for_single_buffer_in_megabytes&quot;: 32,<br /> &quot;irods_default_number_of_transfer_threads&quot;: 4,<br /> &quot;irods_transfer_buffer_size_for_parallel_transfer_in_megabytes&quot;: 4<br /> }<br /> &lt;/pre&gt;<br /> To use PAM authentication (that is with the Yubikey) you will need one<br /> additional line:<br /> &lt;pre&gt;<br /> &quot;irods_authentication_scheme&quot;: &quot;PAM&quot;,<br /> &lt;/pre&gt;<br /> to be inserted _before_ the line with<br /> &lt;code&gt;irods_transfer_buffer_size_for_parallel_transfer_in_megabytes&lt;/code&gt;.<br /> Please note the comma at the end of the line.<br /> <br /> The configuration file needs to be placed in the directory &lt;code&gt;.irods&lt;/code&gt;<br /> under the home directory, named &lt;code&gt;irods_environment.json&lt;/code&gt;.<br /> <br /> To be able to use the iRODS system first the user has to be authenticated.<br /> This is done using the &lt;code&gt;iinit&lt;/code&gt; command.<br /> iinit<br /> It will ask for a password, which should have been already sent in an e-mail.<br /> For the users who are given a Yubikey the device should be inserted into<br /> a USB port at this point and the gold plated button should be pressed<br /> for a few seconds. In case the authentication was successful the<br /> command prompt is returned.<br /> <br /> From then on the iRODS iCommands should be available to use.<br /> <br /> <br /> ==== Using iCommands ====<br /> <br /> All icommands give help using the command line switch &lt;code&gt;-h&lt;/code&gt;.<br /> The most useful commands are:<br /> iinit<br /> ils<br /> iget<br /> iput<br /> irsync<br /> <br /> <br /> For more information see:<br /> https://irods.org/uploads/2016/06/irods_beginner_training_2016.pdf<br /> from Chapter 5.2 (page 16) in the document. This is an introduction<br /> on how to use iRODS.<br /> <br /> There is a more detailed description at<br /> https://docs.irods.org/4.2.1/icommands/user/</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=SNIC_storage/supr_test_irods&diff=6835 2017-09-29T13:40:44Z

<p>Krishnaveni Chitrapu (NSC): Created page with &quot;=== Using SUPR-Test System for projects === https://disposer.c3se.chalmers.se/supr-test If its a new person, click on Register New Person Button &lt;br&gt; One can register 1) Reg...&quot;</p> <hr /> <div>=== Using SUPR-Test System for projects ===<br /> <br /> https://disposer.c3se.chalmers.se/supr-test<br /> <br /> If its a new person, click on Register New Person Button &lt;br&gt;<br /> <br /> One can register <br /> <br /> 1) Register via SWAMID<br /> <br /> 2) Register without SWAMID<br /> <br /> If you are a PI, login to SUPR-Test<br /> <br /> 1) Click on Rounds on the left Panel and select MAX IV Swestore 2017.<br /> <br /> 2) Click on Create new Proposal for MAX IV Swestore 2017 button.<br /> <br /> 3) Give a project title and Create proposal.Alternatively you can clone an existing project from the dropdown.<br /> <br /> 5) The directory name for the project should be simple and not contain any spaces.It would the project/directory name in irods<br /> <br /> 6) PI can add Co-Investigators to the proposal. Alternatively they can be added as members once the proposal is approved.<br /> <br /> 7) Proposal is submitted and once approved a project is created.<br /> <br /> 8) A Python script is run in the background which picks the project and user data and creates corresponding project and users in test-irods.<br /> <br /> 9) User can follow the below instructions to access test-irods.<br /> <br /> <br /> === Client setup to use the test-irods system ===<br /> <br /> The iRODS system 'test-irods' is a test system we use to <br /> test upgrades and functionality. It a collection of<br /> Xen virtual machines running CentOS 6.9 and iRODS 4.1.8<br /> at the moment. There is one iCAT server and two iRODS<br /> storage server nodes.<br /> <br /> To access the system it is preferable to use the same<br /> version of the client as the server but theoretically<br /> it should also work with other versions of the client as<br /> well.<br /> <br /> Easiest and most tested is the Unix command line client.<br /> On a cluster like Aurora most probably this is going to<br /> be the one to be used.<br /> <br /> <br /> ==== Installing the iRODS iCommands client ====<br /> <br /> To install the latest 4.2.1 command line client the<br /> instructions are on the iRODS web site at:<br /> https://irods.org/download/<br /> and<br /> https://packages.irods.org/<br /> .<br /> <br /> On a CentOS Linux variant as root:<br /> Get the signing key.<br /> rpm --import https://packages.irods.org/irods-signing-key.asc<br /> <br /> Get the repository description file.<br /> wget -qO - https://packages.irods.org/renci-irods.yum.repo &gt;/etc/yum.repos.d/renci-irods.yum.repo<br /> <br /> Install the package.<br /> yum install irods-icommands<br /> It will ask to confirm to import the iRODS package signing key<br /> and then ask to confirm going ahead with installing the package<br /> and the dependencies. The command line client executables will go<br /> under &lt;code&gt;/bin&lt;/code&gt;.<br /> <br /> The iRODS client will need to be able to talk to the iRODS iCAT server<br /> and the resource servers at ports 1247 and 20000..20199. (Those are<br /> the defaults.) Firewall / packet filtering should allow outgoing connections<br /> to those ports.<br /> <br /> <br /> ==== Configuring the iRODS iCommands client ====<br /> <br /> To be able to use the iRODS command line client a user has to have<br /> a configuration file in place which describes how to access the<br /> iCAT server. The file has &lt;code&gt;json&lt;/code&gt; format (with the latest<br /> versions.)<br /> <br /> An example file would look like this:<br /> &lt;pre&gt;<br /> {<br /> &quot;irods_host&quot;: &quot;&lt;irodshost&gt;&quot;,<br /> &quot;irods_port&quot;: &lt;port&gt;,<br /> &quot;irods_default_resource&quot;: &quot;&lt;resourcename&gt;&quot;,<br /> &quot;irods_home&quot;: &quot;/&lt;zonename&gt;/home/&lt;username&gt;&quot;,<br /> &quot;irods_cwd&quot;: &quot;/&lt;zonename&gt;/home/&lt;username&gt;&quot;,<br /> &quot;irods_user_name&quot;: &quot;&lt;username&gt;&quot;,<br /> &quot;irods_zone_name&quot;: &quot;&lt;zonename&gt;&quot;,<br /> &quot;irods_client_server_negotiation&quot;: &quot;request_server_negotiation&quot;,<br /> &quot;irods_client_server_policy&quot;: &quot;CS_NEG_REFUSE&quot;,<br /> &quot;irods_encryption_key_size&quot;: 32,<br /> &quot;irods_encryption_salt_size&quot;: 8,<br /> &quot;irods_encryption_num_hash_rounds&quot;: 16,<br /> &quot;irods_encryption_algorithm&quot;: &quot;AES-256-CBC&quot;,<br /> &quot;irods_default_hash_scheme&quot;: &quot;SHA256&quot;,<br /> &quot;irods_match_hash_policy&quot;: &quot;compatible&quot;,<br /> &quot;irods_server_control_plane_port&quot;: 1248,<br /> &quot;irods_server_control_plane_key&quot;: &quot;TEMPORARY__32byte_ctrl_plane_key&quot;,<br /> &quot;irods_server_control_plane_encryption_num_hash_rounds&quot;: 16,<br /> &quot;irods_server_control_plane_encryption_algorithm&quot;: &quot;AES-256-CBC&quot;,<br /> &quot;irods_maximum_size_for_single_buffer_in_megabytes&quot;: 32,<br /> &quot;irods_default_number_of_transfer_threads&quot;: 4,<br /> &quot;irods_transfer_buffer_size_for_parallel_transfer_in_megabytes&quot;: 4<br /> }<br /> &lt;/pre&gt;<br /> Names in the angle brackets are placeholders to be replaced with the<br /> actual values (without the angle brackets as in the file below).<br /> irodshost is the fully qualified name of the irods host (with the dots).<br /> port is the port the iCAT server is using, the default is 1247<br /> resourcename is the name of the default resource to be used<br /> zonename is the name of the iRODS zone<br /> If not sure about these please ask a system administrator.<br /> <br /> In out case:<br /> &lt;pre&gt;<br /> {<br /> &quot;irods_host&quot;: &quot;test-irods.nsc.liu.se&quot;,<br /> &quot;irods_port&quot;: 1247,<br /> &quot;irods_default_resource&quot;: &quot;demoResc&quot;,<br /> &quot;irods_home&quot;: &quot;/tempZone/home/s_jonli&quot;,<br /> &quot;irods_cwd&quot;: &quot;/tempZone/home/s_jonli&quot;,<br /> &quot;irods_user_name&quot;: &quot;s_jonli&quot;,<br /> &quot;irods_zone_name&quot;: &quot;tempZone&quot;,<br /> &quot;irods_client_server_negotiation&quot;: &quot;request_server_negotiation&quot;,<br /> &quot;irods_client_server_policy&quot;: &quot;CS_NEG_REFUSE&quot;,<br /> &quot;irods_encryption_key_size&quot;: 32,<br /> &quot;irods_encryption_salt_size&quot;: 8,<br /> &quot;irods_encryption_num_hash_rounds&quot;: 16,<br /> &quot;irods_encryption_algorithm&quot;: &quot;AES-256-CBC&quot;,<br /> &quot;irods_default_hash_scheme&quot;: &quot;SHA256&quot;,<br /> &quot;irods_match_hash_policy&quot;: &quot;compatible&quot;,<br /> &quot;irods_server_control_plane_port&quot;: 1248,<br /> &quot;irods_server_control_plane_key&quot;: &quot;TEMPORARY__32byte_ctrl_plane_key&quot;,<br /> &quot;irods_server_control_plane_encryption_num_hash_rounds&quot;: 16,<br /> &quot;irods_server_control_plane_encryption_algorithm&quot;: &quot;AES-256-CBC&quot;,<br /> &quot;irods_maximum_size_for_single_buffer_in_megabytes&quot;: 32,<br /> &quot;irods_default_number_of_transfer_threads&quot;: 4,<br /> &quot;irods_transfer_buffer_size_for_parallel_transfer_in_megabytes&quot;: 4<br /> }<br /> &lt;/pre&gt;<br /> To use PAM authentication (that is with the Yubikey) you will need one<br /> additional line:<br /> &lt;pre&gt;<br /> &quot;irods_authentication_scheme&quot;: &quot;PAM&quot;,<br /> &lt;/pre&gt;<br /> to be inserted _before_ the line with<br /> &lt;code&gt;irods_transfer_buffer_size_for_parallel_transfer_in_megabytes&lt;/code&gt;.<br /> Please note the comma at the end of the line.<br /> <br /> The configuration file needs to be placed in the directory &lt;code&gt;.irods&lt;/code&gt;<br /> under the home directory, named &lt;code&gt;irods_environment.json&lt;/code&gt;.<br /> <br /> To be able to use the iRODS system first the user has to be authenticated.<br /> This is done using the &lt;code&gt;iinit&lt;/code&gt; command.<br /> iinit<br /> It will ask for a password, which should have been already sent in an e-mail.<br /> For the users who are given a Yubikey the device should be inserted into<br /> a USB port at this point and the gold plated button should be pressed<br /> for a few seconds. In case the authentication was successful the<br /> command prompt is returned.<br /> <br /> From then on the iRODS iCommands should be available to use.<br /> <br /> <br /> ==== Using iCommands ====<br /> <br /> All icommands give help using the command line switch &lt;code&gt;-h&lt;/code&gt;.<br /> The most useful commands are:<br /> iinit<br /> ils<br /> iget<br /> iput<br /> irsync<br /> <br /> <br /> For more information see:<br /> https://irods.org/uploads/2016/06/irods_beginner_training_2016.pdf<br /> from Chapter 5.2 (page 16) in the document. This is an introduction<br /> on how to use iRODS.<br /> <br /> There is a more detailed description at<br /> https://docs.irods.org/4.2.1/icommands/user/</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=Swestore-dCache&diff=6834 2017-09-29T13:35:58Z

<p>Krishnaveni Chitrapu (NSC): /* Accessing SUPR-TEST System and test-irods System */</p> <hr /> <div>[[Category:Storage]]<br /> [[Category:Swestore]]<br /> [[Category:Swestore]]<br /> [[Category:Swestore user guide]]<br /> <br /> Swestore is Research Data Storage Infrastructure operated by the Swedish National Infrastructure for Computing (SNIC). <br /> <br /> The resources provided by Swestore are made available through open procedures such that the best Swedish research is supported and new research is facilitated. The purpose of Swestore allocations, granted by Swedish National Allocations Committee (SNAC), is to provide large scale data storage for “live” or “working” research data, also known as active research data. <br /> <br /> Due to the VR funding, the free allocations on Swestore have some usage limitations.<br /> * Swestore is NOT supposed to be used for backups and such requests for allocation will be rejected. Please, check with your university (home institution/organisation) IT department about backup services, strategies and policies in place. If such services do not exist or if you can’t access them for different reasons, please contact us at support@swestore.se;<br /> * Swestore is NOT supposed to be used as archiving service, long-term storage or repository for “static” data. Once data is no longer in the process of change, and decision on which data should be retained shared and/or preserved has been taken, data should be moved toward appropriate data services. The higher education institutions are responsible for archiving and long-term preservation of research data produced by researchers employed by them;<br /> <br /> ; Glossary<br /> ; Active (Research) data<br /> : is data that is being worked on as part of research project and therefore subject to change. The files containing data will need to be accessed and amended or updated as new data is gathered or processed.<br /> ; Static (Research) data: is data that is no longer in the process of change and it can be prepared for preservation and reuse.<br /> ; Backup<br /> : is a copy of the digital data to be stored and used as a replacement in case the main copy is either deleted or corrupted.<br /> ; Archive<br /> :is a service to record, organise, and store (digital) items in optimal conditions, with standardised labelling to ensure their longevity and continued access. The service is based on application of metadata, archiving policies, records management, and digital preservation actions. Archivists make decisions on selection and retention of items which are usually governed by supporting policies.<br /> <br /> <br /> Swestore is distributed across the SNIC centres [http://www.c3se.chalmers.se/ C3SE], [http://www.hpc2n.umu.se/ HPC2N], [http://www.lunarc.lu.se/ Lunarc], [http://www.nsc.liu.se/ NSC] and [http://www.uppmax.uu.se Uppmax]. Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple crash of a storage element to losing an entire site while still providing access to the stored data. <br /> <br /> One of the major advantages to the distributed nature of Swestore is the excellent aggregated transfer rates possible. This is achieved by bypassing a central node and having transfers going directly to/from the storage elements if the selected transfer protocol allows it. Swestore can achieve aggregated transfer rates in excess of 100 Gigabit per second, but in practice this is limited by connectivity to the end user, each university or a limited number of files (typically max 1 Gbit/s per file/connection).<br /> <br /> To protect against silent data corruption the dCache storage system checksums all stored data and periodically verifies the data using this checksum.<br /> <br /> The dCache system does NOT yet provide protection against user errors like inadvertent file deletions.<br /> <br /> = Getting access to Swestore=<br /> These steps are required for all PI:s and users of Swestore regardless of how the storage area later is accessed.<br /> <br /> ==Apply for storage (for the PI)==<br /> All normal Swestore projects are managed using the SNIC User and Project Repository portal, SUPR. Please follow the instructions on the [http://www.snic.se/apply-for-resources/instructions-for-swestore-allocations Apply for storage on Swestore] page to apply for storage.<br /> <br /> ==Acquire an eScience client certificate (for all users)==<br /> Swestore user authentication is currently done by user/client certificates. A valid certificate is needed by all project members.<br /> <br /> If you already have a valid grid or eScience certificate you don't need to acquire another one.<br /> <br /> Acquire a certificate by following the ''Requesting a certificate'' instruction on the [[Grid certificates#Requesting a certificate|certificate documentation page]]. This step can be performed while waiting for the storage application to be approved.<br /> <br /> ==Manage your eScience client certificate (for all users)==<br /> There are multiple suppliers of certificates. Depending on the supplier this step varies a bit.<br /> :; For Digicert certificates<br /> :: If intending to access Swestore from a SNIC resource, please make sure you also [[Exporting_a_client_certificate|export the certificate]], transfer it to the intended SNIC resource and [[Preparing_a_client_certificate|prepare it for use with grid tools]].<br /> :; For Nordugrid certificates<br /> :: Please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br /> <br /> ==Register your eScience client certificate in SUPR (for all users)==<br /> <br /> All project members '''have''' to register in SUPR and be added to the approved project by the PI. '''All users also have to register their certificate in SUPR.''' This information is used by Swestore to authenticate the users when accessing the storage area. Registering the certificate is easy though. Make sure your certificate is stored in your browser, log in to [https://supr.snic.se SUPR ], click &quot;Personal Information&quot; in the left menu, click &quot;Register Client Certificate&quot; and follow the instructions. Please wait for up to 10 minutes for this information to be distributed to Swestore.<br /> <br /> = Using Swestore =<br /> == Download and upload data ==<br /> ===From the command line===<br /> <br /> There are several command line tools capable of using the protocols provided by Swestore. For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br /> <br /> As an integration point for building scripts and automated systems we suggest using the curl program and library.<br /> <br /> :; ARC<br /> :: Please see the instructions for [[Accessing Swestore with the ARC client]]. '''Recommended method when logged in on SNIC resources'''.<br /> :; lftp<br /> :: See the instructions for [[Accessing Swestore with lftp]].<br /> :; cURL<br /> :: Please see the instructions for [[Accessing Swestore with cURL]]. We suggest using this as integration point for building scripts and automated systems.<br /> :; globus-url-copy<br /> :: Please see the instructions for [[Accessing Swestore with globus-url-copy]].<br /> :; gfal2<br /> :: (not documented yet)<br /> :; https://duck.sh/<br /> :: Command-line client for Windows/macOS (not documented yet)<br /> <br /> === Using a GUI client ===<br /> <br /> Graphical User Interface (GUI) clients are known to work on some operating systems.<br /> <br /> :; Cyberduck<br /> :: Please see the instructions for [[Accessing Swestore using Cyberduck]] '''Recommended method on Windows and macOS'''<br /> <br /> === From a web browser ===<br /> Swestore is accessible in your web browser as a simple directory index interface at https://webdav.swestore.se/. To browse private data you need to have your certificate installed in your browser (see above). Projects are organized under the &lt;code&gt;/snic&lt;/code&gt; directory as &lt;code&gt;&lt;nowiki&gt;https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/&lt;/nowiki&gt;&lt;/code&gt;.<br /> <br /> == Enabled access protocols ==<br /> A design criteria for Swestore is to provide the storage over a number of standardized and public protocols. There is no vendor specific client needed for access. <br /> <br /> :; GridFTP<br /> :: Also called gsiftp. Well supported within Swestore.<br /> :: Features: Transfer checksums. Writes goes directly to storage pools for high speed transfers.<br /> :: Access URL: gsiftp://gsiftp.swestore.se/<br /> :; HTTP/WebDAV<br /> :: Contender for being the recommended protocol for Swestore.<br /> :: Features: Direct support in web browsers. Redirected reads and writes if special url is used and client supports it.<br /> :: Access URL: http://webdav.swestore.se/ (unauthenticated, read-only, non-redirected)<br /> :: Access URL: https://webdav.swestore.se/ (authenticated, read-write, non-redirected)<br /> :: Access URL: http://webdav.swestore.se:1080/ (unauthenticated, read-only, redirected)<br /> :: Access URL: https://webdav.swestore.se:1443/ (authenticated, read-write, redirected)<br /> :; NFSv4.1<br /> :: Used by some communities, not recommended for general use<br /> :; SRM - Storage Resource Manager<br /> :: Used by some communities, not recommended for general use.<br /> :: Access URL: srm://srm.swegrid.se/<br /> :; Xrootd<br /> :: Used by some communities, not recommended for general use<br /> :; DCAP<br /> :: Used by some communities, not recommended for general use<br /> <br /> == Tools and scripts ==<br /> <br /> There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br /> <br /> * [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer<br /> * Transfer script, [[SweStore/swetrans_arc|swetrans_arc]], provided by Adam Peplinski / Philipp Schlatter<br /> * [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br /> <br /> == Slides and more ==<br /> <br /> [http://docs.snic.se/wiki/Swestore/Lund_Seminar_Apr18 Slides and material from seminar for Lund users on April 18th]<br /> <br /> == Usage monitoring ==<br /> * On the project page in SUPR: https://supr.snic.se<br /> * On the monitoring server for Swestore: http://status.swestore.se/munin/monitor/monitor/ (only accessible from .se-domains.<br /> <br /> = Accessing SUPR-TEST System and test-irods System =<br /> For projects who would like to test irods and how to create irods projects in SUPR-TEST System. &lt;br&gt;<br /> Detailed Instructions can be found at<br /> https://snicdocs.nsc.liu.se/wiki/SNIC_storage/supr_test_irods<br /> <br /> = Support =<br /> <br /> If you have any issues using Swestore, please do not hesitate to contact [mailto:support@swestore.se support@swestore.se].</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=Swestore-dCache&diff=6833 2017-09-29T13:35:26Z

<p>Krishnaveni Chitrapu (NSC): /* Accessing SUPR-TEST System and test-irods System */</p> <hr /> <div>[[Category:Storage]]<br /> [[Category:Swestore]]<br /> [[Category:Swestore]]<br /> [[Category:Swestore user guide]]<br /> <br /> Swestore is Research Data Storage Infrastructure operated by the Swedish National Infrastructure for Computing (SNIC). <br /> <br /> The resources provided by Swestore are made available through open procedures such that the best Swedish research is supported and new research is facilitated. The purpose of Swestore allocations, granted by Swedish National Allocations Committee (SNAC), is to provide large scale data storage for “live” or “working” research data, also known as active research data. <br /> <br /> Due to the VR funding, the free allocations on Swestore have some usage limitations.<br /> * Swestore is NOT supposed to be used for backups and such requests for allocation will be rejected. Please, check with your university (home institution/organisation) IT department about backup services, strategies and policies in place. If such services do not exist or if you can’t access them for different reasons, please contact us at support@swestore.se;<br /> * Swestore is NOT supposed to be used as archiving service, long-term storage or repository for “static” data. Once data is no longer in the process of change, and decision on which data should be retained shared and/or preserved has been taken, data should be moved toward appropriate data services. The higher education institutions are responsible for archiving and long-term preservation of research data produced by researchers employed by them;<br /> <br /> ; Glossary<br /> ; Active (Research) data<br /> : is data that is being worked on as part of research project and therefore subject to change. The files containing data will need to be accessed and amended or updated as new data is gathered or processed.<br /> ; Static (Research) data: is data that is no longer in the process of change and it can be prepared for preservation and reuse.<br /> ; Backup<br /> : is a copy of the digital data to be stored and used as a replacement in case the main copy is either deleted or corrupted.<br /> ; Archive<br /> :is a service to record, organise, and store (digital) items in optimal conditions, with standardised labelling to ensure their longevity and continued access. The service is based on application of metadata, archiving policies, records management, and digital preservation actions. Archivists make decisions on selection and retention of items which are usually governed by supporting policies.<br /> <br /> <br /> Swestore is distributed across the SNIC centres [http://www.c3se.chalmers.se/ C3SE], [http://www.hpc2n.umu.se/ HPC2N], [http://www.lunarc.lu.se/ Lunarc], [http://www.nsc.liu.se/ NSC] and [http://www.uppmax.uu.se Uppmax]. Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple crash of a storage element to losing an entire site while still providing access to the stored data. <br /> <br /> One of the major advantages to the distributed nature of Swestore is the excellent aggregated transfer rates possible. This is achieved by bypassing a central node and having transfers going directly to/from the storage elements if the selected transfer protocol allows it. Swestore can achieve aggregated transfer rates in excess of 100 Gigabit per second, but in practice this is limited by connectivity to the end user, each university or a limited number of files (typically max 1 Gbit/s per file/connection).<br /> <br /> To protect against silent data corruption the dCache storage system checksums all stored data and periodically verifies the data using this checksum.<br /> <br /> The dCache system does NOT yet provide protection against user errors like inadvertent file deletions.<br /> <br /> = Getting access to Swestore=<br /> These steps are required for all PI:s and users of Swestore regardless of how the storage area later is accessed.<br /> <br /> ==Apply for storage (for the PI)==<br /> All normal Swestore projects are managed using the SNIC User and Project Repository portal, SUPR. Please follow the instructions on the [http://www.snic.se/apply-for-resources/instructions-for-swestore-allocations Apply for storage on Swestore] page to apply for storage.<br /> <br /> ==Acquire an eScience client certificate (for all users)==<br /> Swestore user authentication is currently done by user/client certificates. A valid certificate is needed by all project members.<br /> <br /> If you already have a valid grid or eScience certificate you don't need to acquire another one.<br /> <br /> Acquire a certificate by following the ''Requesting a certificate'' instruction on the [[Grid certificates#Requesting a certificate|certificate documentation page]]. This step can be performed while waiting for the storage application to be approved.<br /> <br /> ==Manage your eScience client certificate (for all users)==<br /> There are multiple suppliers of certificates. Depending on the supplier this step varies a bit.<br /> :; For Digicert certificates<br /> :: If intending to access Swestore from a SNIC resource, please make sure you also [[Exporting_a_client_certificate|export the certificate]], transfer it to the intended SNIC resource and [[Preparing_a_client_certificate|prepare it for use with grid tools]].<br /> :; For Nordugrid certificates<br /> :: Please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br /> <br /> ==Register your eScience client certificate in SUPR (for all users)==<br /> <br /> All project members '''have''' to register in SUPR and be added to the approved project by the PI. '''All users also have to register their certificate in SUPR.''' This information is used by Swestore to authenticate the users when accessing the storage area. Registering the certificate is easy though. Make sure your certificate is stored in your browser, log in to [https://supr.snic.se SUPR ], click &quot;Personal Information&quot; in the left menu, click &quot;Register Client Certificate&quot; and follow the instructions. Please wait for up to 10 minutes for this information to be distributed to Swestore.<br /> <br /> = Using Swestore =<br /> == Download and upload data ==<br /> ===From the command line===<br /> <br /> There are several command line tools capable of using the protocols provided by Swestore. For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br /> <br /> As an integration point for building scripts and automated systems we suggest using the curl program and library.<br /> <br /> :; ARC<br /> :: Please see the instructions for [[Accessing Swestore with the ARC client]]. '''Recommended method when logged in on SNIC resources'''.<br /> :; lftp<br /> :: See the instructions for [[Accessing Swestore with lftp]].<br /> :; cURL<br /> :: Please see the instructions for [[Accessing Swestore with cURL]]. We suggest using this as integration point for building scripts and automated systems.<br /> :; globus-url-copy<br /> :: Please see the instructions for [[Accessing Swestore with globus-url-copy]].<br /> :; gfal2<br /> :: (not documented yet)<br /> :; https://duck.sh/<br /> :: Command-line client for Windows/macOS (not documented yet)<br /> <br /> === Using a GUI client ===<br /> <br /> Graphical User Interface (GUI) clients are known to work on some operating systems.<br /> <br /> :; Cyberduck<br /> :: Please see the instructions for [[Accessing Swestore using Cyberduck]] '''Recommended method on Windows and macOS'''<br /> <br /> === From a web browser ===<br /> Swestore is accessible in your web browser as a simple directory index interface at https://webdav.swestore.se/. To browse private data you need to have your certificate installed in your browser (see above). Projects are organized under the &lt;code&gt;/snic&lt;/code&gt; directory as &lt;code&gt;&lt;nowiki&gt;https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/&lt;/nowiki&gt;&lt;/code&gt;.<br /> <br /> == Enabled access protocols ==<br /> A design criteria for Swestore is to provide the storage over a number of standardized and public protocols. There is no vendor specific client needed for access. <br /> <br /> :; GridFTP<br /> :: Also called gsiftp. Well supported within Swestore.<br /> :: Features: Transfer checksums. Writes goes directly to storage pools for high speed transfers.<br /> :: Access URL: gsiftp://gsiftp.swestore.se/<br /> :; HTTP/WebDAV<br /> :: Contender for being the recommended protocol for Swestore.<br /> :: Features: Direct support in web browsers. Redirected reads and writes if special url is used and client supports it.<br /> :: Access URL: http://webdav.swestore.se/ (unauthenticated, read-only, non-redirected)<br /> :: Access URL: https://webdav.swestore.se/ (authenticated, read-write, non-redirected)<br /> :: Access URL: http://webdav.swestore.se:1080/ (unauthenticated, read-only, redirected)<br /> :: Access URL: https://webdav.swestore.se:1443/ (authenticated, read-write, redirected)<br /> :; NFSv4.1<br /> :: Used by some communities, not recommended for general use<br /> :; SRM - Storage Resource Manager<br /> :: Used by some communities, not recommended for general use.<br /> :: Access URL: srm://srm.swegrid.se/<br /> :; Xrootd<br /> :: Used by some communities, not recommended for general use<br /> :; DCAP<br /> :: Used by some communities, not recommended for general use<br /> <br /> == Tools and scripts ==<br /> <br /> There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br /> <br /> * [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer<br /> * Transfer script, [[SweStore/swetrans_arc|swetrans_arc]], provided by Adam Peplinski / Philipp Schlatter<br /> * [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br /> <br /> == Slides and more ==<br /> <br /> [http://docs.snic.se/wiki/Swestore/Lund_Seminar_Apr18 Slides and material from seminar for Lund users on April 18th]<br /> <br /> == Usage monitoring ==<br /> * On the project page in SUPR: https://supr.snic.se<br /> * On the monitoring server for Swestore: http://status.swestore.se/munin/monitor/monitor/ (only accessible from .se-domains.<br /> <br /> = Accessing SUPR-TEST System and test-irods System =<br /> For projects who would like to test irods and how to create irods projects in SUPR-TEST System<br /> Detailed Instructions can be found at<br /> https://snicdocs.nsc.liu.se/wiki/SNIC_storage/supr_test_irods<br /> <br /> = Support =<br /> <br /> If you have any issues using Swestore, please do not hesitate to contact [mailto:support@swestore.se support@swestore.se].</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=Swestore-dCache&diff=6832 2017-09-29T13:33:00Z

<p>Krishnaveni Chitrapu (NSC): /* Accessing SUPR-TEST System and test-irods System */</p> <hr /> <div>[[Category:Storage]]<br /> [[Category:Swestore]]<br /> [[Category:Swestore]]<br /> [[Category:Swestore user guide]]<br /> <br /> Swestore is Research Data Storage Infrastructure operated by the Swedish National Infrastructure for Computing (SNIC). <br /> <br /> The resources provided by Swestore are made available through open procedures such that the best Swedish research is supported and new research is facilitated. The purpose of Swestore allocations, granted by Swedish National Allocations Committee (SNAC), is to provide large scale data storage for “live” or “working” research data, also known as active research data. <br /> <br /> Due to the VR funding, the free allocations on Swestore have some usage limitations.<br /> * Swestore is NOT supposed to be used for backups and such requests for allocation will be rejected. Please, check with your university (home institution/organisation) IT department about backup services, strategies and policies in place. If such services do not exist or if you can’t access them for different reasons, please contact us at support@swestore.se;<br /> * Swestore is NOT supposed to be used as archiving service, long-term storage or repository for “static” data. Once data is no longer in the process of change, and decision on which data should be retained shared and/or preserved has been taken, data should be moved toward appropriate data services. The higher education institutions are responsible for archiving and long-term preservation of research data produced by researchers employed by them;<br /> <br /> ; Glossary<br /> ; Active (Research) data<br /> : is data that is being worked on as part of research project and therefore subject to change. The files containing data will need to be accessed and amended or updated as new data is gathered or processed.<br /> ; Static (Research) data: is data that is no longer in the process of change and it can be prepared for preservation and reuse.<br /> ; Backup<br /> : is a copy of the digital data to be stored and used as a replacement in case the main copy is either deleted or corrupted.<br /> ; Archive<br /> :is a service to record, organise, and store (digital) items in optimal conditions, with standardised labelling to ensure their longevity and continued access. The service is based on application of metadata, archiving policies, records management, and digital preservation actions. Archivists make decisions on selection and retention of items which are usually governed by supporting policies.<br /> <br /> <br /> Swestore is distributed across the SNIC centres [http://www.c3se.chalmers.se/ C3SE], [http://www.hpc2n.umu.se/ HPC2N], [http://www.lunarc.lu.se/ Lunarc], [http://www.nsc.liu.se/ NSC] and [http://www.uppmax.uu.se Uppmax]. Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple crash of a storage element to losing an entire site while still providing access to the stored data. <br /> <br /> One of the major advantages to the distributed nature of Swestore is the excellent aggregated transfer rates possible. This is achieved by bypassing a central node and having transfers going directly to/from the storage elements if the selected transfer protocol allows it. Swestore can achieve aggregated transfer rates in excess of 100 Gigabit per second, but in practice this is limited by connectivity to the end user, each university or a limited number of files (typically max 1 Gbit/s per file/connection).<br /> <br /> To protect against silent data corruption the dCache storage system checksums all stored data and periodically verifies the data using this checksum.<br /> <br /> The dCache system does NOT yet provide protection against user errors like inadvertent file deletions.<br /> <br /> = Getting access to Swestore=<br /> These steps are required for all PI:s and users of Swestore regardless of how the storage area later is accessed.<br /> <br /> ==Apply for storage (for the PI)==<br /> All normal Swestore projects are managed using the SNIC User and Project Repository portal, SUPR. Please follow the instructions on the [http://www.snic.se/apply-for-resources/instructions-for-swestore-allocations Apply for storage on Swestore] page to apply for storage.<br /> <br /> ==Acquire an eScience client certificate (for all users)==<br /> Swestore user authentication is currently done by user/client certificates. A valid certificate is needed by all project members.<br /> <br /> If you already have a valid grid or eScience certificate you don't need to acquire another one.<br /> <br /> Acquire a certificate by following the ''Requesting a certificate'' instruction on the [[Grid certificates#Requesting a certificate|certificate documentation page]]. This step can be performed while waiting for the storage application to be approved.<br /> <br /> ==Manage your eScience client certificate (for all users)==<br /> There are multiple suppliers of certificates. Depending on the supplier this step varies a bit.<br /> :; For Digicert certificates<br /> :: If intending to access Swestore from a SNIC resource, please make sure you also [[Exporting_a_client_certificate|export the certificate]], transfer it to the intended SNIC resource and [[Preparing_a_client_certificate|prepare it for use with grid tools]].<br /> :; For Nordugrid certificates<br /> :: Please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br /> <br /> ==Register your eScience client certificate in SUPR (for all users)==<br /> <br /> All project members '''have''' to register in SUPR and be added to the approved project by the PI. '''All users also have to register their certificate in SUPR.''' This information is used by Swestore to authenticate the users when accessing the storage area. Registering the certificate is easy though. Make sure your certificate is stored in your browser, log in to [https://supr.snic.se SUPR ], click &quot;Personal Information&quot; in the left menu, click &quot;Register Client Certificate&quot; and follow the instructions. Please wait for up to 10 minutes for this information to be distributed to Swestore.<br /> <br /> = Using Swestore =<br /> == Download and upload data ==<br /> ===From the command line===<br /> <br /> There are several command line tools capable of using the protocols provided by Swestore. For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br /> <br /> As an integration point for building scripts and automated systems we suggest using the curl program and library.<br /> <br /> :; ARC<br /> :: Please see the instructions for [[Accessing Swestore with the ARC client]]. '''Recommended method when logged in on SNIC resources'''.<br /> :; lftp<br /> :: See the instructions for [[Accessing Swestore with lftp]].<br /> :; cURL<br /> :: Please see the instructions for [[Accessing Swestore with cURL]]. We suggest using this as integration point for building scripts and automated systems.<br /> :; globus-url-copy<br /> :: Please see the instructions for [[Accessing Swestore with globus-url-copy]].<br /> :; gfal2<br /> :: (not documented yet)<br /> :; https://duck.sh/<br /> :: Command-line client for Windows/macOS (not documented yet)<br /> <br /> === Using a GUI client ===<br /> <br /> Graphical User Interface (GUI) clients are known to work on some operating systems.<br /> <br /> :; Cyberduck<br /> :: Please see the instructions for [[Accessing Swestore using Cyberduck]] '''Recommended method on Windows and macOS'''<br /> <br /> === From a web browser ===<br /> Swestore is accessible in your web browser as a simple directory index interface at https://webdav.swestore.se/. To browse private data you need to have your certificate installed in your browser (see above). Projects are organized under the &lt;code&gt;/snic&lt;/code&gt; directory as &lt;code&gt;&lt;nowiki&gt;https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/&lt;/nowiki&gt;&lt;/code&gt;.<br /> <br /> == Enabled access protocols ==<br /> A design criteria for Swestore is to provide the storage over a number of standardized and public protocols. There is no vendor specific client needed for access. <br /> <br /> :; GridFTP<br /> :: Also called gsiftp. Well supported within Swestore.<br /> :: Features: Transfer checksums. Writes goes directly to storage pools for high speed transfers.<br /> :: Access URL: gsiftp://gsiftp.swestore.se/<br /> :; HTTP/WebDAV<br /> :: Contender for being the recommended protocol for Swestore.<br /> :: Features: Direct support in web browsers. Redirected reads and writes if special url is used and client supports it.<br /> :: Access URL: http://webdav.swestore.se/ (unauthenticated, read-only, non-redirected)<br /> :: Access URL: https://webdav.swestore.se/ (authenticated, read-write, non-redirected)<br /> :: Access URL: http://webdav.swestore.se:1080/ (unauthenticated, read-only, redirected)<br /> :: Access URL: https://webdav.swestore.se:1443/ (authenticated, read-write, redirected)<br /> :; NFSv4.1<br /> :: Used by some communities, not recommended for general use<br /> :; SRM - Storage Resource Manager<br /> :: Used by some communities, not recommended for general use.<br /> :: Access URL: srm://srm.swegrid.se/<br /> :; Xrootd<br /> :: Used by some communities, not recommended for general use<br /> :; DCAP<br /> :: Used by some communities, not recommended for general use<br /> <br /> == Tools and scripts ==<br /> <br /> There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br /> <br /> * [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer<br /> * Transfer script, [[SweStore/swetrans_arc|swetrans_arc]], provided by Adam Peplinski / Philipp Schlatter<br /> * [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br /> <br /> == Slides and more ==<br /> <br /> [http://docs.snic.se/wiki/Swestore/Lund_Seminar_Apr18 Slides and material from seminar for Lund users on April 18th]<br /> <br /> == Usage monitoring ==<br /> * On the project page in SUPR: https://supr.snic.se<br /> * On the monitoring server for Swestore: http://status.swestore.se/munin/monitor/monitor/ (only accessible from .se-domains.<br /> <br /> = Accessing SUPR-TEST System and test-irods System =<br /> For projects who would like to test irods and how to create irods projects in SUPR-TEST System<br /> <br /> = Support =<br /> <br /> If you have any issues using Swestore, please do not hesitate to contact [mailto:support@swestore.se support@swestore.se].</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=Swestore-dCache&diff=6831 2017-09-29T13:31:43Z

<p>Krishnaveni Chitrapu (NSC): </p> <hr /> <div>[[Category:Storage]]<br /> [[Category:Swestore]]<br /> [[Category:Swestore]]<br /> [[Category:Swestore user guide]]<br /> <br /> Swestore is Research Data Storage Infrastructure operated by the Swedish National Infrastructure for Computing (SNIC). <br /> <br /> The resources provided by Swestore are made available through open procedures such that the best Swedish research is supported and new research is facilitated. The purpose of Swestore allocations, granted by Swedish National Allocations Committee (SNAC), is to provide large scale data storage for “live” or “working” research data, also known as active research data. <br /> <br /> Due to the VR funding, the free allocations on Swestore have some usage limitations.<br /> * Swestore is NOT supposed to be used for backups and such requests for allocation will be rejected. Please, check with your university (home institution/organisation) IT department about backup services, strategies and policies in place. If such services do not exist or if you can’t access them for different reasons, please contact us at support@swestore.se;<br /> * Swestore is NOT supposed to be used as archiving service, long-term storage or repository for “static” data. Once data is no longer in the process of change, and decision on which data should be retained shared and/or preserved has been taken, data should be moved toward appropriate data services. The higher education institutions are responsible for archiving and long-term preservation of research data produced by researchers employed by them;<br /> <br /> ; Glossary<br /> ; Active (Research) data<br /> : is data that is being worked on as part of research project and therefore subject to change. The files containing data will need to be accessed and amended or updated as new data is gathered or processed.<br /> ; Static (Research) data: is data that is no longer in the process of change and it can be prepared for preservation and reuse.<br /> ; Backup<br /> : is a copy of the digital data to be stored and used as a replacement in case the main copy is either deleted or corrupted.<br /> ; Archive<br /> :is a service to record, organise, and store (digital) items in optimal conditions, with standardised labelling to ensure their longevity and continued access. The service is based on application of metadata, archiving policies, records management, and digital preservation actions. Archivists make decisions on selection and retention of items which are usually governed by supporting policies.<br /> <br /> <br /> Swestore is distributed across the SNIC centres [http://www.c3se.chalmers.se/ C3SE], [http://www.hpc2n.umu.se/ HPC2N], [http://www.lunarc.lu.se/ Lunarc], [http://www.nsc.liu.se/ NSC] and [http://www.uppmax.uu.se Uppmax]. Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple crash of a storage element to losing an entire site while still providing access to the stored data. <br /> <br /> One of the major advantages to the distributed nature of Swestore is the excellent aggregated transfer rates possible. This is achieved by bypassing a central node and having transfers going directly to/from the storage elements if the selected transfer protocol allows it. Swestore can achieve aggregated transfer rates in excess of 100 Gigabit per second, but in practice this is limited by connectivity to the end user, each university or a limited number of files (typically max 1 Gbit/s per file/connection).<br /> <br /> To protect against silent data corruption the dCache storage system checksums all stored data and periodically verifies the data using this checksum.<br /> <br /> The dCache system does NOT yet provide protection against user errors like inadvertent file deletions.<br /> <br /> = Getting access to Swestore=<br /> These steps are required for all PI:s and users of Swestore regardless of how the storage area later is accessed.<br /> <br /> ==Apply for storage (for the PI)==<br /> All normal Swestore projects are managed using the SNIC User and Project Repository portal, SUPR. Please follow the instructions on the [http://www.snic.se/apply-for-resources/instructions-for-swestore-allocations Apply for storage on Swestore] page to apply for storage.<br /> <br /> ==Acquire an eScience client certificate (for all users)==<br /> Swestore user authentication is currently done by user/client certificates. A valid certificate is needed by all project members.<br /> <br /> If you already have a valid grid or eScience certificate you don't need to acquire another one.<br /> <br /> Acquire a certificate by following the ''Requesting a certificate'' instruction on the [[Grid certificates#Requesting a certificate|certificate documentation page]]. This step can be performed while waiting for the storage application to be approved.<br /> <br /> ==Manage your eScience client certificate (for all users)==<br /> There are multiple suppliers of certificates. Depending on the supplier this step varies a bit.<br /> :; For Digicert certificates<br /> :: If intending to access Swestore from a SNIC resource, please make sure you also [[Exporting_a_client_certificate|export the certificate]], transfer it to the intended SNIC resource and [[Preparing_a_client_certificate|prepare it for use with grid tools]].<br /> :; For Nordugrid certificates<br /> :: Please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br /> <br /> ==Register your eScience client certificate in SUPR (for all users)==<br /> <br /> All project members '''have''' to register in SUPR and be added to the approved project by the PI. '''All users also have to register their certificate in SUPR.''' This information is used by Swestore to authenticate the users when accessing the storage area. Registering the certificate is easy though. Make sure your certificate is stored in your browser, log in to [https://supr.snic.se SUPR ], click &quot;Personal Information&quot; in the left menu, click &quot;Register Client Certificate&quot; and follow the instructions. Please wait for up to 10 minutes for this information to be distributed to Swestore.<br /> <br /> = Using Swestore =<br /> == Download and upload data ==<br /> ===From the command line===<br /> <br /> There are several command line tools capable of using the protocols provided by Swestore. For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br /> <br /> As an integration point for building scripts and automated systems we suggest using the curl program and library.<br /> <br /> :; ARC<br /> :: Please see the instructions for [[Accessing Swestore with the ARC client]]. '''Recommended method when logged in on SNIC resources'''.<br /> :; lftp<br /> :: See the instructions for [[Accessing Swestore with lftp]].<br /> :; cURL<br /> :: Please see the instructions for [[Accessing Swestore with cURL]]. We suggest using this as integration point for building scripts and automated systems.<br /> :; globus-url-copy<br /> :: Please see the instructions for [[Accessing Swestore with globus-url-copy]].<br /> :; gfal2<br /> :: (not documented yet)<br /> :; https://duck.sh/<br /> :: Command-line client for Windows/macOS (not documented yet)<br /> <br /> === Using a GUI client ===<br /> <br /> Graphical User Interface (GUI) clients are known to work on some operating systems.<br /> <br /> :; Cyberduck<br /> :: Please see the instructions for [[Accessing Swestore using Cyberduck]] '''Recommended method on Windows and macOS'''<br /> <br /> === From a web browser ===<br /> Swestore is accessible in your web browser as a simple directory index interface at https://webdav.swestore.se/. To browse private data you need to have your certificate installed in your browser (see above). Projects are organized under the &lt;code&gt;/snic&lt;/code&gt; directory as &lt;code&gt;&lt;nowiki&gt;https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/&lt;/nowiki&gt;&lt;/code&gt;.<br /> <br /> == Enabled access protocols ==<br /> A design criteria for Swestore is to provide the storage over a number of standardized and public protocols. There is no vendor specific client needed for access. <br /> <br /> :; GridFTP<br /> :: Also called gsiftp. Well supported within Swestore.<br /> :: Features: Transfer checksums. Writes goes directly to storage pools for high speed transfers.<br /> :: Access URL: gsiftp://gsiftp.swestore.se/<br /> :; HTTP/WebDAV<br /> :: Contender for being the recommended protocol for Swestore.<br /> :: Features: Direct support in web browsers. Redirected reads and writes if special url is used and client supports it.<br /> :: Access URL: http://webdav.swestore.se/ (unauthenticated, read-only, non-redirected)<br /> :: Access URL: https://webdav.swestore.se/ (authenticated, read-write, non-redirected)<br /> :: Access URL: http://webdav.swestore.se:1080/ (unauthenticated, read-only, redirected)<br /> :: Access URL: https://webdav.swestore.se:1443/ (authenticated, read-write, redirected)<br /> :; NFSv4.1<br /> :: Used by some communities, not recommended for general use<br /> :; SRM - Storage Resource Manager<br /> :: Used by some communities, not recommended for general use.<br /> :: Access URL: srm://srm.swegrid.se/<br /> :; Xrootd<br /> :: Used by some communities, not recommended for general use<br /> :; DCAP<br /> :: Used by some communities, not recommended for general use<br /> <br /> == Tools and scripts ==<br /> <br /> There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br /> <br /> * [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer<br /> * Transfer script, [[SweStore/swetrans_arc|swetrans_arc]], provided by Adam Peplinski / Philipp Schlatter<br /> * [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br /> <br /> == Slides and more ==<br /> <br /> [http://docs.snic.se/wiki/Swestore/Lund_Seminar_Apr18 Slides and material from seminar for Lund users on April 18th]<br /> <br /> == Usage monitoring ==<br /> * On the project page in SUPR: https://supr.snic.se<br /> * On the monitoring server for Swestore: http://status.swestore.se/munin/monitor/monitor/ (only accessible from .se-domains.<br /> <br /> = Accessing SUPR-TEST System and test-irods System =<br /> = Support =<br /> <br /> If you have any issues using Swestore, please do not hesitate to contact [mailto:support@swestore.se support@swestore.se].</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=Swestore-dCache&diff=6134 2015-07-13T07:47:09Z

<p>Krishnaveni Chitrapu (NSC): /* Getting access */</p> <hr /> <div>[[Category:Storage]]<br /> [[Category:SweStore]]<br /> SNIC is building a storage infrastructure to complement the computational resources.<br /> <br /> Many forms of automated measurements can produce large amounts of data. In scientific areas such as high energy physics (the Large Hadron Collider at CERN), climate modeling, bioinformatics, bioimaging etc., the demands for storage are increasing dramatically. To serve these and other user communities, SNIC has appointed a working group to design a storage strategy, taking into account the needs on many levels and creating a unified storage infrastructure, which is now being implemented.<br /> <br /> Swestore is in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], [http://www.bioimaging.se/swedish_bioimaging_network/Welcome.html Bioimage Sweden], [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ NaturHistoriska RiksMuseet].<br /> <br /> = National storage &quot;SweStore&quot;=<br /> The Swestore Nationally Accessible Storage, commonly called just Swestore, is a robust, flexible and expandable long term storage system aimed at storing large amounts of data produced by various Swedish research projects. It is based on the [http://www.dcache.org dCache] and [http://www.irods.org iRODS]<br /> storage systems.<br /> <br /> Swestore is distributed across the SNIC centres [http://www.c3se.chalmers.se/ C3SE], [http://www.hpc2n.umu.se/ HPC2N], [http://www.lunarc.lu.se/ Lunarc], [http://www.nsc.liu.se/ NSC], [http://www.pdc.kth.se PDC] and [http://www.uppmax.uu.se Uppmax]. Data is stored in two copies with each copy at a different SNIC centre. This enables the system to cope with a multitude of issues ranging from a simple crash of a storage element to losing an entire site while still providing access to the stored data. <br /> <br /> One of the major advantages to the distributed nature of dCache and iRODS ([[Swestore-irods]]) is the excellent aggregated transfer rates possible. This is achieved by bypassing a central node and having transfers going directly to/from the storage elements if the protocol allows it. The Swestore Nationally Accessible Storage system can achieve aggregated transfer rates in excess of 100 Gigabit per second, but in practice this is limited by connectivity to each University (usually 10 Gbit/s) or a limited number of files (typically<br /> max 1 Gbit/s per file/connection).<br /> <br /> To protect against silent data corruption the dCache storage system checksums all stored data and periodically verifies the data using this checksum.<br /> <br /> The dCache system does NOT yet provide protection against user errors like inadvertent file deletions and so on. The [[Swestore-irods]] system provides this protection. Deleted files are moved to a trashcan.<br /> <br /> == Getting access ==<br /> <br /> SweStore currently has a lack of resources. New projects will not be allocated until Autumn 2015. Existing projects on SweStore that are in need of more storage capacity are strongly encouraged to review their present data to see whether there is a potential to compress or delete data. In case you have any questions, please contact the SNIC office (office@snic.se).<br /> <br /> ; Apply for storage<br /> : Please follow the instructions on the [[Apply for storage on SweStore]] page.<br /> : iRODS is in &lt;span style=&quot;color:#FF0000&quot;&gt; pilot phase &lt;/span&gt;&lt;br&gt;<br /> : The iRODS system dosen't have the uptime and performance that our production system have.<br /> : We are still working with iRODS to get it into production.<br /> <br /> ;Difference between dCache and iRODS user authentication<br /> :SweStore's dCache system uses eScience client certificates.<br /> :SweStore's iRODS system uses [http://www.yubico.com/products/yubikey-hardware/yubikey/ Yubikey] one-time passwords (OTP). With a simple touch of a button, a 44 character one-time password is generated and sent to the system. The user will be provided with a SweStore yubikey.<br /> :Yubikey has a status as pilot now. It can be changed in the future.<br /> <br /> ; dCache usage - How to acquire an eScience client certificate<br /> : Follow the instructions on [[Grid_certificates#Requesting_a_certificate|Requesting a certificate]] to get your client certificate. This step can be performed while waiting for the storage application to be approved and processed. Of course, if you already have a valid eScience certificate you don't need to acquire another one.<br /> :; For Terena certificates<br /> :: If intending to access SweStore from a SNIC resource, please make sure you also [[Exporting_a_client_certificate|export the certificate]], transfer it to the intended SNIC resource and [[Preparing_a_client_certificate|prepare it for use with grid tools]] (not necessarily needed with ARC 3.x, see [[Grid_certificates#Creating_a_proxy_certificate_using_the_Firefox.2FThunderbird_credential_store|proxy certificates using Firefox credential store]]).<br /> :; For Nordugrid certificates<br /> :: Please make sure to also [[Requesting_a_grid_certificate_from_the_Nordugrid_CA#Installing_the_certificate_in_your_browser|install your client certificate in your browser]].<br /> :; Request membership in the SweGrid VO<br /> :: Follow the instructions on [[Grid_certificates#Requesting_membership_in_the_SweGrid_VO|Requesting membership in the SweGrid VO]] to get added to the SweGrid Virtual Organisation (VO) and request membership to your allocated storage project.<br /> <br /> ; iRODS usage - How to acquire a SweStore yubikey<br /> :Please send an email to [mailto:support@swestore.se?subject=Yubikey support@swestore.se] and provide the shipping address to where the yubikey should be sent.&lt;br&gt;<br /> :Yubikey has a status as pilot now. It can be changed in the future.<br /> <br /> == Support == <br /> <br /> If you have any issues using SweStore please do not hesitate to contact [mailto:support@swestore.se support@swestore.se].<br /> <br /> == dCache ==<br /> <br /> === Access protocols ===<br /> ; Currently supported protocols<br /> : GridFTP - gsiftp://gsiftp.swestore.se/<br /> : Storage Resource Manager - srm://srm.swegrid.se/<br /> : Hypertext Transfer Protocol (read-only), Web Distributed Authoring and Versioning - http://webdav.swestore.se/ (unauthenticated), https://webdav.swestore.se/<br /> : NFS4.1<br /> <br /> For authentication eScience certificates are used, which provides a higher level of security than legacy username/password schemes.<br /> <br /> === Download and upload data ===<br /> ; Interactive browsing and manipulation of single files<br /> : SweStore is accessible in your web browser in two ways, as a simple and reliable directory index interface at https://webdav.swestore.se/ and with a richer interactive file manager at https://webdav.swestore.se/browser/. '''Note''' that the interactive file manager has a lot of features and functions not supported in SweStore, only the basic file transfer features are supported.<br /> : To browse private data you need to have your certificate installed in your browser (default with Terena certificates, see above). Projects are organized under the &lt;code&gt;/snic&lt;/code&gt; directory as &lt;code&gt;&lt;nowiki&gt;https://webdav.swestore.se/snic/YOUR_PROJECT_NAME/&lt;/nowiki&gt;&lt;/code&gt;.<br /> ; Upload and delete data interactively or with automation<br /> There are several tools that are capable of using the protocols provided by SweStore national storage.<br /> For interactive usage on SNIC clusters we recommend using the ARC tools which should be installed on all SNIC resources.<br /> As an integration point for building scripts and automated systems we suggest using the curl program and library.<br /> : Use the ARC client. Please see the instructions for [[Accessing SweStore national storage with the ARC client]]. '''Recommended''' method when logged in on SNIC resources.<br /> : Use lftp. Please see the instructions for [[Accessing SweStore national storage with lftp]].<br /> : Use cURL. Please see the instructions for [[Accessing SweStore national storage with cURL]].<br /> : Use globus-url-copy. Please see the instructions for [[Accessing SweStore national storage with globus-url-copy]].<br /> <br /> === Tools and scripts ===<br /> <br /> There exists a number of tools and utilities developed externally that can be useful. Here are some links:<br /> <br /> * [https://github.com/samuell/arc_tools ARC_Tools] - Convenience scripts for the arc client (Only a recursive rmdir so far).<br /> * [http://sourceforge.net/projects/arc-gui-clients ARC Graphical Clients] - Contains the ARC Storage Explorer (SweStore supported development).<br /> * Transfer script, [[SweStore/swetrans_arc|swetrans_arc]], provided by Adam Peplinski / Philipp Schlatter<br /> * [http://www.nordugrid.org/documents/SWIG-wrapped-ARC-Python-API.pdf Documentation of the ARC Python API (PDF)]<br /> <br /> === Slides and more ===<br /> <br /> [http://docs.snic.se/wiki/Swestore/Lund_Seminar_Apr18 Slides and material from seminar for Lund users on April 18th]<br /> <br /> === Usage monitoring ===<br /> * [http://status.swestore.se/munin/monitor/monitor/ Per Project Monitoring of Swestore usage]<br /> <br /> == iRODS ==<br /> <br /> Documentation of the SNIC iRODS system: [[Swestore-irods]].</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=User:Krishnaveni_Chitrapu_(NSC)&diff=6062 2015-03-24T19:24:00Z

<p>Krishnaveni Chitrapu (NSC): /* Expertise */</p> <hr /> <div>{{systems expert info<br /> |first name=Krishnaveni<br /> |last name=Chitrapu<br /> |centre=NSC<br /> |fields=Software Development<br /> |general activities=dCache development.<br /> &lt;!-- |image=name of an uploaded image file<br /> |office=Semicolon separated lines; of visiting address; newlines<br /> are ignored<br /> |phone=Semicolon separated phone numbers (with optional <br /> explanation); newlines are ignored--&gt;<br /> }}<br /> <br /> == Quick facts ==<br /> * System Developer at NSC since May 2012 <br /> * Programming Experience<br /> <br /> == Expertise ==<br /> * [[expertise::JAVA]]<br /> * [[expertise::J2EE]]<br /> * [[expertise::PL/SQL]]<br /> * [[expertise::Python]]<br /> <br /> == Projects ==<br /> * [[project::BILS websearch]]<br /> * [[project::Fido]]</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=Accessing_Swestore_with_cURL&diff=6058 2015-03-12T15:06:55Z

<p>Krishnaveni Chitrapu (NSC): /* Optional parameters */</p> <hr /> <div>[[Category:SweStore]]<br /> [[Category:SweStore user guide]]<br /> [[SweStore|&lt; SweStore]]<br /> <br /> <br /> This guide outlines the procedure for using cURL to access files through the WebDav door of dCache.<br /> <br /> == Essential parameters ==<br /> <br /> --capath /etc/grid-security/certificates<br /> The certificate bundle provided through --capath is required in order for cURL to accept the server certificates the door presents. If the certificate bundle is not available, the -k flag may be passed to allow untrusted server certificates.<br /> <br /> --cert /tmp/x509up_u1234<br /> --cert (or -E) names the proxy certificate generated by arcproxy or similar tools, which is a single PEM file consisting of the client certificate, the proxy key and the proxy certificate. The name will vary based on the user issuing it.<br /> grid-proxy-init (and thus arcproxy) will put the certificate in /tmp by default and name it according to the pattern x509up_u&lt;NumericUID&gt;. The -out parameter to grid-proxy-init takes a location to store the certificate in if the default is not sufficient.<br /> <br /> --location<br /> --location (or -L) instructs cURL to follow HTTP redirects, in this case the 302 redirects that the dCache door uses to direct clients to different storage nodes.<br /> <br /> == Required parameters ==<br /> <br /> --tls or -1<br /> Since sslv3 Version is disabled on servers due to POODLE: SSLv3 vulnerability (CVE-2014-3566) but some of the cURL Versions are trying to connect to SSLv3 instead of TLS and failing to connect to with an error along the lines of &lt;tt&gt;curl: (35) Unknown SSL protocol error in connection to ...&lt;/tt&gt;.So this parameter is recommended to be used with cURL till the problem is solved.<br /> <br /> Ex:- curl --tls --location --capath /etc/grid-security/certificates<br /> <br /> == Sample invocations ==<br /> <br /> Downloads the file 'file-to-download.ext':<br /> curl --location --capath /etc/grid-security/certificates --cert /tmp/x509up_u1234 -O https://webdav.swegrid.se/target/path/file-to-download.ext<br /> <br /> Upload the file 'source.file' as 'uploaded.ext':<br /> curl --location --capath /etc/grid-security/certificates --cert /tmp/x509up_u1234 -T ~/source.file https://webdav.swegrid.se/target/path/uploaded.ext<br /> <br /> = Credits =<br /> <br /> This guide was written by Lars Viklund</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=Apply_for_storage_on_Swestore&diff=5890 2014-06-27T13:44:51Z

<p>Krishnaveni Chitrapu (NSC): /* Application instructions */</p> <hr /> <div>[[Category:SweStore]]<br /> [[Category:SweStore user guide]]<br /> [[SweStore|&lt; SweStore]]<br /> <br /> The SweStore nationally accessible storage is available for researchers financed by VR (which includes all researchers using SNIC compute resources) and FORMA.<br /> <br /> = Research Communities =<br /> <br /> SweStore is also in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], [http://www.bioimaging.se/swedish_bioimaging_network/Welcome.html Bioimage], [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ Naturhistoriska Riksmuseet]. If any of these cover your research area, first read their information on applying for SweStore storage.<br /> <br /> == UPPNEX ==<br /> If you are a member of an UPPNEX project on UPPMAX, an iRODS service is already available and configured for you, please see http://www.uppmax.uu.se/faq/how-to-move-files-to-swestore-using-irods for more information on how to use it.<br /> <br /> == Other communities ==<br /> Unless you are instructed otherwise, submit an application to SweStore as outlined below.<br /> <br /> = Application instructions =<br /> <br /> Application for Swestore need more fields now. Soon Swestore application will be moving to SUPR System. Once it is moved, these fields can be filled in SUPR instead.<br /> <br /> Send an email to [mailto:support@swestore.se support@swestore.se] <br /> <br /> Please include the following information in the application:&lt;br&gt;<br /> * Whether you want the regular/production SweStore storage (based on dCache) or the new iRODS based storage (currently in pre-production pilot phase).<br /> ** If you apply for iRODS-storage: please provide a shipping address to where your yubikey should be sent.<br /> <br /> * Principal Investigator (PI) - (Account Responsible)<br /> ** Name<br /> ** Job title/position<br /> ** E-mail address<br /> ** Phone no<br /> ** Citizenship<br /> ** Department<br /> ** Organization<br /> ** Address<br /> ** Postal Code<br /> ** City<br /> <br /> * Technical Administrator (the Primary contact)<br /> ** Name<br /> ** Job title/position<br /> ** E-mail address<br /> ** Phone no<br /> ** Citizenship<br /> ** Department<br /> ** Organization<br /> ** Address<br /> ** Postal Code<br /> ** City<br /> <br /> * Project Classification code according to VR classification codes http://classification.vr.se<br /> * Associated SNIC compute resource allocation (if any)<br /> * Project Summary and Purpose for the storage: A short description of the project and type of data. (Max 250 words)<br /> * Funding agency<br /> * Describe how the data is produced (ex:- computer simulation, experimentation, observation, sensors, digitization of data)<br /> * Describe by whom the data is produced, especially in case this is not within project itself<br /> * Required storage capacity: Preferably a maximum size, but if this is not currently determinable, please calculate a starting size and expansion by time period. <br /> ** '''NOTE''' that applications larger than 10TB takes longer to process as they go through SNIC approval.<br /> * Please indicate the expected total storage volume (in TeraBytes) per year and how this is spread across storage media<br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> ! Type<br /> ! 2014<br /> ! 2015<br /> ! 2016<br /> ! 2017<br /> ! 2018<br /> |-<br /> | Online,Ready-to-use Data<br /> | TBytes<br /> | TBytes<br /> | TBytes<br /> | TBytes<br /> | TBytes<br /> |-<br /> | Backup,Longer-Latency Data<br /> | TBytes<br /> | TBytes<br /> | TBytes<br /> | TBytes<br /> | TBytes<br /> |}<br /> <br /> * Describe how frequently the data will be uploaded to Swestore (once, rarely, continuously)<br /> * Describe which communities should be able to access the data (ex:- a set of people, VO, part of VO, publicly available)<br /> * Describe what kind of access the users should have to the data (read only, read and write)<br /> * Suggested project name / project title: This will be used as root directory name for your storage.<br /> ** '''NOTE''' that this name is long-lived and will persist. It is not coupled to the lifetime of SNIC compute time allocations.<br /> ** We recommend a project name not tied to a person.<br /> ** Additionally, we recommend that the name is not a common word or a term easily confusable with other current or future research efforts.<br /> ** It is a good idea to select a name that's short and easy to type.<br /> ** The name is limited to lower-case letters a-z, digits 0-9, hyphens - and underscores _.</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=Apply_for_storage_on_Swestore&diff=5889 2014-06-27T12:03:21Z

<p>Krishnaveni Chitrapu (NSC): /* Application instructions */</p> <hr /> <div>[[Category:SweStore]]<br /> [[Category:SweStore user guide]]<br /> [[SweStore|&lt; SweStore]]<br /> <br /> The SweStore nationally accessible storage is available for researchers financed by VR (which includes all researchers using SNIC compute resources) and FORMA.<br /> <br /> = Research Communities =<br /> <br /> SweStore is also in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], [http://www.bioimaging.se/swedish_bioimaging_network/Welcome.html Bioimage], [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ Naturhistoriska Riksmuseet]. If any of these cover your research area, first read their information on applying for SweStore storage.<br /> <br /> == UPPNEX ==<br /> If you are a member of an UPPNEX project on UPPMAX, an iRODS service is already available and configured for you, please see http://www.uppmax.uu.se/faq/how-to-move-files-to-swestore-using-irods for more information on how to use it.<br /> <br /> == Other communities ==<br /> Unless you are instructed otherwise, submit an application to SweStore as outlined below.<br /> <br /> = Application instructions =<br /> <br /> Application for Swestore need more fields now. Soon Swestore application will be moving to SUPR System. Once it is moved, these fields can be filled in SUPR instead.<br /> <br /> Send an email to [mailto:support@swestore.se support@swestore.se] <br /> <br /> Please include the following information in the application:&lt;br&gt;<br /> * Whether you want the regular/production SweStore storage (based on dCache) or the new iRODS based storage (currently in pre-production pilot phase).<br /> ** If you apply for iRODS-storage: please provide a shipping address to where your yubikey should be sent.<br /> <br /> * Principal Investigator (PI) - (Account Responsible)<br /> ** Name<br /> ** Job title/position<br /> ** E-mail address<br /> ** Phone no<br /> ** Citizenship<br /> ** Department<br /> ** Organization<br /> ** Address<br /> ** Postal Code<br /> ** City<br /> <br /> * Technical Administrator (the Primary contact)<br /> ** Name<br /> ** Job title/position<br /> ** E-mail address<br /> ** Phone no<br /> ** Citizenship<br /> ** Department<br /> ** Organization<br /> ** Address<br /> ** Postal Code<br /> ** City<br /> <br /> * Project Classification code according to VR classification codes http://classification.vr.se<br /> * Associated SNIC compute resource allocation (if any)<br /> * Project Summary and Purpose for the storage: A short description of the project and type of data. (Max 250 words)<br /> * Funding agency<br /> * Describe how the data is produced (ex:- computer simulation, experimentation, observation, sensors, digitization of data)<br /> * Describe by whom the data is produced, especially in case this is not within project itself<br /> * Required storage capacity: Preferably a maximum size, but if this is not currently determinable, please calculate a starting size and expansion by time period. <br /> ** '''NOTE''' that applications larger than 10TB takes longer to process as they go through SNIC approval.<br /> * Describe how frequently the data will be uploaded to Swestore (once, rarely, continuously)<br /> * Describe which communities should be able to access the data (ex:- a set of people, VO, part of VO, publicly available)<br /> * Describe what kind of access the users should have to the data (read only, read and write)<br /> * Suggested project name / project title: This will be used as root directory name for your storage.<br /> ** '''NOTE''' that this name is long-lived and will persist. It is not coupled to the lifetime of SNIC compute time allocations.<br /> ** We recommend a project name not tied to a person.<br /> ** Additionally, we recommend that the name is not a common word or a term easily confusable with other current or future research efforts.<br /> ** It is a good idea to select a name that's short and easy to type.<br /> ** The name is limited to lower-case letters a-z, digits 0-9, hyphens - and underscores _.</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=Apply_for_storage_on_Swestore&diff=5888 2014-06-27T11:56:49Z

<p>Krishnaveni Chitrapu (NSC): /* Application instructions */</p> <hr /> <div>[[Category:SweStore]]<br /> [[Category:SweStore user guide]]<br /> [[SweStore|&lt; SweStore]]<br /> <br /> The SweStore nationally accessible storage is available for researchers financed by VR (which includes all researchers using SNIC compute resources) and FORMA.<br /> <br /> = Research Communities =<br /> <br /> SweStore is also in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], [http://www.bioimaging.se/swedish_bioimaging_network/Welcome.html Bioimage], [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ Naturhistoriska Riksmuseet]. If any of these cover your research area, first read their information on applying for SweStore storage.<br /> <br /> == UPPNEX ==<br /> If you are a member of an UPPNEX project on UPPMAX, an iRODS service is already available and configured for you, please see http://www.uppmax.uu.se/faq/how-to-move-files-to-swestore-using-irods for more information on how to use it.<br /> <br /> == Other communities ==<br /> Unless you are instructed otherwise, submit an application to SweStore as outlined below.<br /> <br /> = Application instructions =<br /> <br /> Application for Swestore need more fields now. Soon Swestore application will be moving to SUPR System. Once it is moved, these fields can be filled in SUPR instead.<br /> <br /> Send an email to [mailto:support@swestore.se support@swestore.se] <br /> <br /> Please include the following information in the application:&lt;br&gt;<br /> * Whether you want the regular/production SweStore storage (based on dCache) or the new iRODS based storage (currently in pre-production pilot phase).<br /> ** If you apply for iRODS-storage: please provide a shipping address to where your yubikey should be sent.<br /> <br /> * Principal Investigator (PI) - (Account Responsible)<br /> ** Name<br /> ** Job title/position<br /> ** E-mail address<br /> ** Phone no<br /> ** Citizenship<br /> ** Department<br /> ** Organization<br /> ** Address<br /> ** Postal Code<br /> ** City<br /> <br /> * Technical Administrator (the Primary contact)<br /> ** Name<br /> ** Job title/position<br /> ** E-mail address<br /> ** Phone no<br /> ** Citizenship<br /> ** Department<br /> ** Organization<br /> ** Address<br /> ** Postal Code<br /> ** City<br /> <br /> * Project Classification code according to VR classification codes http://classification.vr.se<br /> * Associated SNIC compute resource allocation (if any)<br /> * Project Summary and Purpose for the storage: A short description of the project and type of data. (Max 250 words)<br /> * Funding agency<br /> * Describe how the data is produced (ex:- computer simulation, experimentation, observation, sensors, digitization of data)<br /> * Describe by whom the data is produced, especially in case this is not within project itself<br /> * Required storage capacity: Preferably a maximum size, but if this is not currently determinable, please calculate a starting size and expansion by time period. '''NOTE''' that applications larger than 10TB takes longer to process.<br /> * Describe how frequently the data will be uploaded to Swestore (once, rarely, continuously)<br /> * Describe which communities should be able to access the data (ex:- a set of people, VO, part of VO, publicly available)<br /> * Describe what kind of access the users should have to the data (read only, read and write)<br /> * Suggested project name / project title: This will be used as root directory name for your storage.<br /> # '''NOTE''' that this name is long-lived and will persist. It is not coupled to the lifetime of SNIC compute time allocations.<br /> # We recommend a project name not tied to a person.<br /> # Additionally, we recommend that the name is not a common word or term easily confusable with other current or future research efforts.<br /> # It is a good idea to select a name that's short and easy to type.<br /> # The name is limited to lower-case letters a-z, digits 0-9, hyphens - and underscores _.</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=Apply_for_storage_on_Swestore&diff=5887 2014-06-27T11:55:14Z

<p>Krishnaveni Chitrapu (NSC): /* Application instructions */</p> <hr /> <div>[[Category:SweStore]]<br /> [[Category:SweStore user guide]]<br /> [[SweStore|&lt; SweStore]]<br /> <br /> The SweStore nationally accessible storage is available for researchers financed by VR (which includes all researchers using SNIC compute resources) and FORMA.<br /> <br /> = Research Communities =<br /> <br /> SweStore is also in collaboration with [http://www.ecds.se/ ECDS], [http://snd.gu.se/ SND], [http://www.bioimaging.se/swedish_bioimaging_network/Welcome.html Bioimage], [http://www.bils.se/ BILS], [http://www.uppnex.uu.se/ UPPNEX],[http://wlcg.web.cern.ch/ WLCG], [http://www.nrm.se/ Naturhistoriska Riksmuseet]. If any of these cover your research area, first read their information on applying for SweStore storage.<br /> <br /> == UPPNEX ==<br /> If you are a member of an UPPNEX project on UPPMAX, an iRODS service is already available and configured for you, please see http://www.uppmax.uu.se/faq/how-to-move-files-to-swestore-using-irods for more information on how to use it.<br /> <br /> == Other communities ==<br /> Unless you are instructed otherwise, submit an application to SweStore as outlined below.<br /> <br /> = Application instructions =<br /> <br /> Send an email to [mailto:support@swestore.se support@swestore.se] <br /> <br /> Please include the following information in the application:&lt;br&gt;<br /> * Whether you want the regular/production SweStore storage (based on dCache) or the new iRODS based storage (currently in pre-production pilot phase).<br /> ** If you apply for iRODS-storage: please provide a shipping address to where your yubikey should be sent.<br /> <br /> * Principal Investigator (PI) - (Account Responsible)<br /> ** Name<br /> ** Job title/position<br /> ** E-mail address<br /> ** Phone no<br /> ** Citizenship<br /> ** Department<br /> ** Organization<br /> ** Address<br /> ** Postal Code<br /> ** City<br /> <br /> * Technical Administrator (the Primary contact)<br /> ** Name<br /> ** Job title/position<br /> ** E-mail address<br /> ** Phone no<br /> ** Citizenship<br /> ** Department<br /> ** Organization<br /> ** Address<br /> ** Postal Code<br /> ** City<br /> <br /> * Project Classification code according to VR classification codes http://classification.vr.se<br /> * Associated SNIC compute resource allocation (if any)<br /> * Project Summary and Purpose for the storage: A short description of the project and type of data. (Max 250 words)<br /> * Funding agency<br /> * Describe how the data is produced (ex:- computer simulation, experimentation, observation, sensors, digitization of data)<br /> * Describe by whom the data is produced, especially in case this is not within project itself<br /> * Required storage capacity: Preferably a maximum size, but if this is not currently determinable, please calculate a starting size and expansion by time period. '''NOTE''' that applications larger than 10TB takes longer to process.<br /> * Describe how frequently the data will be uploaded to Swestore (once, rarely, continuously)<br /> * Describe which communities should be able to access the data (ex:- a set of people, VO, part of VO, publicly available)<br /> * Describe what kind of access the users should have to the data (read only, read and write)<br /> * Suggested project name / project title: This will be used as root directory name for your storage.<br /> # '''NOTE''' that this name is long-lived and will persist. It is not coupled to the lifetime of SNIC compute time allocations.<br /> # We recommend a project name not tied to a person.<br /> # Additionally, we recommend that the name is not a common word or term easily confusable with other current or future research efforts.<br /> # It is a good idea to select a name that's short and easy to type.<br /> # The name is limited to lower-case letters a-z, digits 0-9, hyphens - and underscores _.</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=BILS_websearch&diff=4733 2013-02-13T12:43:52Z

<p>Krishnaveni Chitrapu (NSC): </p> <hr /> <div>{{project info<br /> |description=BILS Web Search<br /> |fields=Software Development<br /> |financing=<br /> |active=no<br /> |start date=2012-06-04<br /> |end date=2013-02-04<br /> }}<br /> <br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}<br /> <br /> If you feel the need to specify roles within the project, this is a good place to do so.</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=BILS_websearch&diff=4685 2013-02-04T15:25:07Z

<p>Krishnaveni Chitrapu (NSC): </p> <hr /> <div>{{project info<br /> |description=BILS Web Search<br /> |fields=Software Development<br /> |financing=<br /> |active=no<br /> |start date=2012-06-04<br /> |end date=2013-02-04<br /> }}<br /> <br /> Describe your project here, using your document structure of choice and in as much detail as you deem necessary. Look at other project for guidance. <br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}<br /> <br /> If you feel the need to specify roles within the project, this is a good place to do so.</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=User:Krishnaveni_Chitrapu_(NSC)&diff=4684 2013-02-04T15:21:55Z

<p>Krishnaveni Chitrapu (NSC): </p> <hr /> <div>{{systems expert info<br /> |first name=Krishnaveni<br /> |last name=Chitrapu<br /> |centre=NSC<br /> |fields=Software Development<br /> |general activities=dCache development.<br /> &lt;!-- |image=name of an uploaded image file<br /> |office=Semicolon separated lines; of visiting address; newlines<br /> are ignored<br /> |phone=Semicolon separated phone numbers (with optional <br /> explanation); newlines are ignored--&gt;<br /> }}<br /> <br /> == Quick facts ==<br /> * System Developer at NSC since May 2012 <br /> * Programming Experience<br /> <br /> == Expertise ==<br /> * [[expertise::JAVA]]<br /> * [[expertise::J2EE]]<br /> * [[expertise::PL/SQL]]<br /> <br /> == Projects ==<br /> * [[project::BILS websearch]]</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=User:Krishnaveni_Chitrapu_(NSC)&diff=4683 2013-02-04T15:21:23Z

<p>Krishnaveni Chitrapu (NSC): </p> <hr /> <div>{{systems expert info<br /> |first name=Krishnaveni<br /> |last name=Chitrapu<br /> |centre=NSC<br /> |fields=Software Development<br /> |general activities=part of dCache development.; <br /> &lt;!-- |image=name of an uploaded image file<br /> |office=Semicolon separated lines; of visiting address; newlines<br /> are ignored<br /> |phone=Semicolon separated phone numbers (with optional <br /> explanation); newlines are ignored--&gt;<br /> }}<br /> <br /> == Quick facts ==<br /> * System Developer at NSC since May 2012 <br /> * Programming Experience<br /> <br /> == Expertise ==<br /> * [[expertise::JAVA]]<br /> * [[expertise::J2EE]]<br /> * [[expertise::PL/SQL]]<br /> <br /> == Projects ==<br /> * [[project::BILS websearch]]</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=User:Krishnaveni_Chitrapu_(NSC)&diff=4681 2013-01-31T09:36:38Z

<p>Krishnaveni Chitrapu (NSC): </p> <hr /> <div>{{systems expert info<br /> |first name=Krishnaveni<br /> |last name=Chitrapu<br /> |centre=NSC<br /> |fields=Software Development<br /> &lt;!-- |other activities=Semicolon separated list of major distractions.; Like other part time duties.; Use minimal *but complete* sentences.;<br /> |image=name of an uploaded image file<br /> |office=Semicolon separated lines; of visiting address; newlines<br /> are ignored<br /> |phone=Semicolon separated phone numbers (with optional <br /> explanation); newlines are ignored--&gt;<br /> }}<br /> <br /> == Quick facts ==<br /> * Or any other heading. <br /> * This part is clear text and very much up to you.<br /> * Bullet lists are good <br /> * for the lazy<br /> <br /> == Expertise ==<br /> * [[expertise::JAVA]]<br /> * [[expertise::J2EE]]<br /> * [[expertise::PL/SQL]]<br /> <br /> == Projects ==<br /> * [[project::BILS websearch]]</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=BILS_websearch&diff=4680 2013-01-31T09:34:13Z

<p>Krishnaveni Chitrapu (NSC): </p> <hr /> <div>{{project info<br /> |description=BILS Web Search<br /> |research project=(ok well this project does not have a research project association, but if it did you could state it like so: SeRC: Electron structure solved once and for all.)<br /> |fields=Software Development<br /> |financing=<br /> |active=yes<br /> |start date=2011-05-01<br /> |end date=<br /> }}<br /> <br /> Describe your project here, using your document structure of choice and in as much detail as you deem necessary. Look at other project for guidance. <br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}<br /> <br /> If you feel the need to specify roles within the project, this is a good place to do so.</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=BILS_websearch&diff=4679 2013-01-31T09:29:05Z

<p>Krishnaveni Chitrapu (NSC): Created page with &quot;{{project info |description=BILS Web Search |research project=(ok well this project does not have a research project association, but if it did you could state it like so: SeRC: ...&quot;</p> <hr /> <div>{{project info<br /> |description=BILS Web Search<br /> |research project=(ok well this project does not have a research project association, but if it did you could state it like so: SeRC: Electron structure solved once and for all.)<br /> |fields=Computational materials science<br /> |financing=SNIC<br /> |active=yes<br /> |start date=2011-05-01<br /> |end date=<br /> }}<br /> <br /> Describe your project here, using your document structure of choice and in as much detail as you deem necessary. Look at other project for guidance. <br /> <br /> == Members ==<br /> {{#ask: [[Category:Person]] [[project::{{PAGENAME}}]]<br /> |?centre<br /> |?role<br /> |?field<br /> }}<br /> <br /> If you feel the need to specify roles within the project, this is a good place to do so.</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=User:Krishnaveni_Chitrapu_(NSC)&diff=4678 2013-01-31T09:23:09Z

<p>Krishnaveni Chitrapu (NSC): </p> <hr /> <div>{{systems expert info<br /> |first name=Krishnaveni<br /> |last name=Chitrapu<br /> |centre=NSC<br /> |fields=Software Development<br /> &lt;!-- |other activities=Semicolon separated list of major distractions.; Like other part time duties.; Use minimal *but complete* sentences.;<br /> |image=name of an uploaded image file<br /> |office=Semicolon separated lines; of visiting address; newlines<br /> are ignored<br /> |phone=Semicolon separated phone numbers (with optional <br /> explanation); newlines are ignored--&gt;<br /> }}<br /> <br /> == Quick facts ==<br /> * Or any other heading. <br /> * This part is clear text and very much up to you.<br /> * Bullet lists are good <br /> * for the lazy<br /> <br /> == Expertise ==<br /> * [[expertise::JAVA]]<br /> * [[expertise::J2EE]]<br /> * [[expertise::PL/SQL]]</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=User:Krishnaveni_Chitrapu_(NSC)&diff=4677 2013-01-31T09:22:37Z

<p>Krishnaveni Chitrapu (NSC): </p> <hr /> <div>{{systems expert info<br /> |first name=Krishnaveni<br /> |last name=Chitrapu<br /> |centre=NSC<br /> |fields=Software Development<br /> &lt;!-- |other activities=Semicolon separated list of major distractions.; Like other part time duties.; Use minimal *but complete* sentences.;<br /> |image=name of an uploaded image file<br /> |office=Semicolon separated lines; of visiting address; newlines<br /> are ignored<br /> |phone=Semicolon separated phone numbers (with optional <br /> explanation); newlines are ignored<br /> }}--&gt;<br /> <br /> == Quick facts ==<br /> * Or any other heading. <br /> * This part is clear text and very much up to you.<br /> * Bullet lists are good <br /> * for the lazy<br /> <br /> == Expertise ==<br /> * [[expertise::JAVA]]<br /> * [[expertise::J2EE]]<br /> * [[expertise::PL/SQL]]</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=User:Krishnaveni_Chitrapu_(NSC)&diff=4676 2013-01-31T09:22:02Z

<p>Krishnaveni Chitrapu (NSC): </p> <hr /> <div>{{systems expert info<br /> |first name=Krishnaveni<br /> |last name=Chitrapu<br /> |centre=NSC<br /> |fields=Software Development<br /> &lt;--|other activities=Semicolon separated list of major distractions.; Like other part time duties.; Use minimal *but complete* sentences.;<br /> |image=name of an uploaded image file<br /> |office=Semicolon separated lines; of visiting address; newlines<br /> are ignored<br /> |phone=Semicolon separated phone numbers (with optional <br /> explanation); newlines are ignored<br /> }}--&gt;<br /> <br /> == Quick facts ==<br /> * Or any other heading. <br /> * This part is clear text and very much up to you.<br /> * Bullet lists are good <br /> * for the lazy<br /> <br /> == Expertise ==<br /> * [[expertise::JAVA]]<br /> * [[expertise::J2EE]]<br /> * [[expertise::PL/SQL]]</div>

Krishnaveni Chitrapu (NSC) http://docs.snic.se/w/index.php?title=User:Krishnaveni_Chitrapu_(NSC)&diff=4675 2013-01-31T09:20:49Z

<p>Krishnaveni Chitrapu (NSC): Created page with &quot;{{systems expert info |first name=Krishnaveni |last name=Chitrapu |centre=NSC |fields=Software Development |other activities=Semicolon separated list of major distractions.; Like...&quot;</p> <hr /> <div>{{systems expert info<br /> |first name=Krishnaveni<br /> |last name=Chitrapu<br /> |centre=NSC<br /> |fields=Software Development<br /> |other activities=Semicolon separated list of major distractions.; Like other part time duties.; Use minimal *but complete* sentences.;<br /> |image=name of an uploaded image file<br /> |office=Semicolon separated lines; of visiting address; newlines<br /> are ignored<br /> |phone=Semicolon separated phone numbers (with optional <br /> explanation); newlines are ignored<br /> }}<br /> <br /> == Quick facts ==<br /> * Or any other heading. <br /> * This part is clear text and very much up to you.<br /> * Bullet lists are good <br /> * for the lazy<br /> <br /> == Expertise ==<br /> * [[expertise::JAVA]]<br /> * [[expertise::J2EE]]<br /> * [[expertise::PL/SQL]]</div>

Krishnaveni Chitrapu (NSC)