Difference between revisions of "User:Torben Rasmussen (NSC)/SweGrid old/Using national storage resources"

From SNIC Documentation
Jump to: navigation, search
(Introduction)
 
(17 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
= Introduction =  
 
= Introduction =  
 +
 +
(UNDER DEVELOPMENT)
  
 
The SNIC national storage infrastructure is implemented using dCache, which is a distributed storage system presenting the user with a single filesystem view. The front-end host of the system is srm://srm.swegrid.se which is the machine presenting the filesystem for the user.  
 
The SNIC national storage infrastructure is implemented using dCache, which is a distributed storage system presenting the user with a single filesystem view. The front-end host of the system is srm://srm.swegrid.se which is the machine presenting the filesystem for the user.  
  
The system uses VOMS proxy certificates for authentication. Please see, , for information on how to generate a voms proxy certificate.
+
The system uses VOMS proxy certificates for authentication. Please see, [[Grid certificates#VOMS certificates|VOMS Certificates]] , for information on how to generate a voms proxy certificate.
 +
 
 +
= Copying files =
 +
 
 +
Copying files to and from resources is accomplished using the '''arccp''' command. The command can handle many of existing protocols, such as: ''http, https, httpg, ftp, gsiftp, lfc, rls, srm''.
 +
 
 +
== Copying single files ==
 +
 
 +
Copying single files is accomplished in the same way as using the normal '''cp''' command as shown in the following example:
 +
 
 +
<pre>[jonas@localhost jonas2]$ arccp archive.tar.gz srm://srm.swegrid.se/ops/</pre>
 +
 
 +
== Recursive copying ==
 +
 
 +
Recursive copying is accomplished using the '''--recursive''' option. The argument to the option determines the depth of the recursive copy.
 +
 
 +
<pre>
 +
[jonas@localhost swestore]$ arccp --recursive=3 jonas2/ srm://srm.swegrid.se/ops/jonas/jonas2/
 +
</pre>
  
 
= Listing files =  
 
= Listing files =  
  
= Copying files =
+
Listing files on a resources is done using the '''arcls''' command. In the simplest form the command just takes a URL as input and displays names and directories without any extra information as shown in the following example:
 +
 
 +
<pre>
 +
[jonas@localhost jonas2]$ arcls srm://srm.swegrid.se/ops/
 +
motd.1
 +
bla
 +
generated
 +
ops
 +
motd.f343
 +
GridFTP-1.0.20-linux.gtk.x86.zip
 +
test23
 +
test24
 +
ronstestfile
 +
ronstestfile3
 +
testfile-put-1296854868-6468a2d5df07.txt
 +
</pre>
 +
 
 +
Additional information can be listed by adding the '''--long''' option.
 +
 
 +
<pre>
 +
[jonas@localhost swestore]$ arcls --long srm://srm.swegrid.se/ops
 +
<Name> <Type> <Size> <Creation> <Validity> <CheckSum> <Latency>
 +
motd.1 file 354 2008-06-05 12:28:23 (n/a) adler32:582d7718 NEARLINE
 +
bla dir 512 2008-08-22 12:23:49 (n/a) (n/a) NEARLINE
 +
generated dir 512 2008-11-17 15:07:39 (n/a) (n/a) NEARLINE
 +
ops dir 512 2010-01-21 11:26:00 (n/a) (n/a) NEARLINE
 +
motd.f343 file 436 2010-01-08 14:35:40 (n/a) adler32:0fed94f2 ONLINE
 +
GridFTP-1.0.20-linux.gtk.x86.zip file 30479220 2011-03-09 09:49:20 (n/a) adler32:5c67686b ONLINE
 +
test23 file 0 2011-03-04 09:44:44 (n/a) adler32:00000001 ONLINE
 +
test24 file 0 2011-03-04 09:45:07 (n/a) adler32:00000001 ONLINE
 +
ronstestfile file 4 2010-11-23 17:46:13 (n/a) adler32:03cd014b ONLINE
 +
ronstestfile3 file 4 2010-11-23 17:49:07 (n/a) adler32:03cd014b ONLINE
 +
testfile-put-1296854868-6468a2d5df07.txt file 20 2011-02-04 22:28:47 (n/a) adler32:1a400272 ONLINE
 +
</pre>
 +
 
 +
Metadatainformation on a specific file can be listed by specifying the '''-m''' or '''--metadata''' option. In the following example the metadata information of the '''test23''' file is shown:
 +
 
 +
<pre>
 +
[jonas@localhost swestore]$ arcls --metadata srm://srm.swegrid.se/ops/test23
 +
/ops/test23
 +
accessperm:rw-r-----
 +
checksum:adler32:00000001
 +
ctime:2011-03-04 09:44:44
 +
filestoragetype:PERMANENT
 +
group:25001
 +
latency:ONLINE
 +
lifetimeassigned:PT1S
 +
lifetimeleft:PT1S
 +
mtime:2011-03-04 09:44:45
 +
owner:25001
 +
path:/ops/test23
 +
size:0
 +
spacetokens:
 +
type:file
 +
</pre>
 +
 
 +
= Creating directories =  
 +
 
 +
There are no command line tool yet for creating directories. This command will probably be added in a upcoming release. In the meantime the following procedure can be used:
 +
 
 +
<pre>
 +
[jonas@localhost swestore]$ mkdir mydir
 +
[jonas@localhost swestore]$ touch mydir/dummyfile
 +
[jonas@localhost swestore]$ arccp ./mydir/ srm://srm.swegrid.se/ops/mydir/
 +
[jonas@localhost swestore]$ arcls srm://srm.swegrid.se/ops/mydir
 +
dummyfile
 +
</pre>
 +
 
 +
The dummmy file can be removed using the arcrm command described in the following sections.
  
 
= Removing files =
 
= Removing files =
  
= Creating directories =
+
<pre>[jonas@localhost swestore]$ arcrm srm://srm.swegrid.se/ops/mydir/dummyfile</pre>
 +
 
 +
= Removing directories =  
 +
 
 +
To remove directories they have to be empty. To indicate a directory the URL should end with a '''/'''.
 +
 
 +
<pre>[jonas@localhost swestore]$ arcrm srm://srm.swegrid.se/ops/mydir/</pre>

Latest revision as of 14:31, 8 July 2011

Introduction

(UNDER DEVELOPMENT)

The SNIC national storage infrastructure is implemented using dCache, which is a distributed storage system presenting the user with a single filesystem view. The front-end host of the system is srm://srm.swegrid.se which is the machine presenting the filesystem for the user.

The system uses VOMS proxy certificates for authentication. Please see, VOMS Certificates , for information on how to generate a voms proxy certificate.

Copying files

Copying files to and from resources is accomplished using the arccp command. The command can handle many of existing protocols, such as: http, https, httpg, ftp, gsiftp, lfc, rls, srm.

Copying single files

Copying single files is accomplished in the same way as using the normal cp command as shown in the following example:

[jonas@localhost jonas2]$ arccp archive.tar.gz srm://srm.swegrid.se/ops/

Recursive copying

Recursive copying is accomplished using the --recursive option. The argument to the option determines the depth of the recursive copy.

[jonas@localhost swestore]$ arccp --recursive=3 jonas2/ srm://srm.swegrid.se/ops/jonas/jonas2/

Listing files

Listing files on a resources is done using the arcls command. In the simplest form the command just takes a URL as input and displays names and directories without any extra information as shown in the following example:

[jonas@localhost jonas2]$ arcls srm://srm.swegrid.se/ops/
motd.1
bla
generated
ops
motd.f343
GridFTP-1.0.20-linux.gtk.x86.zip
test23
test24
ronstestfile
ronstestfile3
testfile-put-1296854868-6468a2d5df07.txt

Additional information can be listed by adding the --long option.

[jonas@localhost swestore]$ arcls --long srm://srm.swegrid.se/ops
<Name> <Type> <Size> <Creation> <Validity> <CheckSum> <Latency>
motd.1 file 354 2008-06-05 12:28:23 (n/a) adler32:582d7718 NEARLINE
bla dir 512 2008-08-22 12:23:49 (n/a) (n/a) NEARLINE
generated dir 512 2008-11-17 15:07:39 (n/a) (n/a) NEARLINE
ops dir 512 2010-01-21 11:26:00 (n/a) (n/a) NEARLINE
motd.f343 file 436 2010-01-08 14:35:40 (n/a) adler32:0fed94f2 ONLINE
GridFTP-1.0.20-linux.gtk.x86.zip file 30479220 2011-03-09 09:49:20 (n/a) adler32:5c67686b ONLINE
test23 file 0 2011-03-04 09:44:44 (n/a) adler32:00000001 ONLINE
test24 file 0 2011-03-04 09:45:07 (n/a) adler32:00000001 ONLINE
ronstestfile file 4 2010-11-23 17:46:13 (n/a) adler32:03cd014b ONLINE
ronstestfile3 file 4 2010-11-23 17:49:07 (n/a) adler32:03cd014b ONLINE
testfile-put-1296854868-6468a2d5df07.txt file 20 2011-02-04 22:28:47 (n/a) adler32:1a400272 ONLINE

Metadatainformation on a specific file can be listed by specifying the -m or --metadata option. In the following example the metadata information of the test23 file is shown:

[jonas@localhost swestore]$ arcls --metadata srm://srm.swegrid.se/ops/test23
/ops/test23
accessperm:rw-r-----
checksum:adler32:00000001
ctime:2011-03-04 09:44:44
filestoragetype:PERMANENT
group:25001
latency:ONLINE
lifetimeassigned:PT1S
lifetimeleft:PT1S
mtime:2011-03-04 09:44:45
owner:25001
path:/ops/test23
size:0
spacetokens:
type:file

Creating directories

There are no command line tool yet for creating directories. This command will probably be added in a upcoming release. In the meantime the following procedure can be used:

[jonas@localhost swestore]$ mkdir mydir
[jonas@localhost swestore]$ touch mydir/dummyfile
[jonas@localhost swestore]$ arccp ./mydir/ srm://srm.swegrid.se/ops/mydir/
[jonas@localhost swestore]$ arcls srm://srm.swegrid.se/ops/mydir
dummyfile

The dummmy file can be removed using the arcrm command described in the following sections.

Removing files

[jonas@localhost swestore]$ arcrm srm://srm.swegrid.se/ops/mydir/dummyfile

Removing directories

To remove directories they have to be empty. To indicate a directory the URL should end with a /.

[jonas@localhost swestore]$ arcrm srm://srm.swegrid.se/ops/mydir/