|
|
| (9 intermediate revisions by 2 users not shown) |
| Line 1: |
Line 1: |
| − | [[Category:Swestore]] | + | #REDIRECT[[Swestore Documentation Moved]] |
| − | [[Category:Swestore user guide]]
| |
| − | [[Swestore|< Swestore]]
| |
| − | | |
| − | | |
| − | == Introduction ==
| |
| − | | |
| − | lftp is a file transfer tool that understands a range of protocols from plain old FTP, HTTP, SCP to more esoteric ones like WebDAV and BitTorrent.
| |
| − | | |
| − | The benefit over tools like cURL is that it has interactive traversal of the directory hierarchy, as well as powerful mass-transfer functionality like the mirror command.
| |
| − | | |
| − | Out of the protocols mentioned above, the ones that aligns most with the access doors that Swestore offers are the WebDAV over HTTP/HTTPS protocols.
| |
| − | | |
| − | Authentication against Swestore over the WebDAV door is done with client certificates over HTTPS, where there is a choice of either using the real client certificate or by using a RFC or VOMS proxy certificate generated from the real certificate.
| |
| − | | |
| − | Proxy certificates are preferred as they are valid for a limited period of time, especially as lftp does not seem to offer any way to enter any passphrase to unlock keys, so in order to use a protected real certificate, it will have to be unlocked in advance.
| |
| − | | |
| − | Notable is that while file uploads are securely tunneled inside a SSL connection, downloads are in the plain from storage nodes. As such, the usual guidelines about sensitive data holds, where you should aim to have your data encrypted before transmitting it.
| |
| − | | |
| − | == Required software versions ==
| |
| − | | |
| − | The versions of lftp and its major dependency GNUTLS that have been verified to work with proxy certificates are:
| |
| − | | |
| − | {| class="wikitable"
| |
| − | |+ Tested versions
| |
| − | |-
| |
| − | | '''Tested on''' || || Ubuntu 13.10
| |
| − | |-
| |
| − | | '''lftp''' || 4.3.3 || 4.4.9
| |
| − | |-
| |
| − | | '''gnutls''' || 2.12.0 || 2.12.23
| |
| − | |-
| |
| − | | '''libnettle (for building gnutls)''' || 2.4 || 2.7.1
| |
| − | |}
| |
| − | | |
| − | | |
| − | This gnutls version is the absolute minimum version that will work, any version prior to that (2.10.5 and below) will not be able to connect to the door.
| |
| − | | |
| − | This means that the gnutls version in Ubuntu oneric and older, Scientific Linux/RHEL/CentOS 6.1 and older will not work.
| |
| − | | |
| − | Ubuntu precise has a sufficiently new gnutls and will work out of the box.
| |
| − | | |
| − | == lftp settings ==
| |
| − | | |
| − | While running the <code>lftp</code> program, there are several settings that need to be configured in order to successfully connect and interact with Swestore:
| |
| − | <nowiki>
| |
| − | set ssl:ca-file /etc/grid-security/certificates/TERENA-eScience-SSL-CA.pem
| |
| − | set ssl:check-hostname true
| |
| − | set ssl:verify-certificate true</nowiki>
| |
| − | | |
| − | and
| |
| − | <nowiki>
| |
| − | set ssl:key-file /tmp/x509up_u1234
| |
| − | set ssl:cert-file /tmp/x509up_u1234</nowiki>
| |
| − | | |
| − | where the last two are to indicate the filename of the proxy certificate generated by the <code>arcproxy</code> or <code>grid-proxy-init</code> tool. The trailing digits in the filename will vary based on the user ID (uid) of the local user, which you can find out in a terminal shell by running the <code>id</code> program.
| |
| − | | |
| − | The CA file is for verifying the identity of the server and is strongly recommended that both certificate and hostname verification is in effect to ensure that the server communicated with is the intended target machine.
| |
| − | | |
| − | They can be stored in the configuration file named ~/.lftp/rc together with any other commands you wish to perform during lftp startup.
| |
| − | | |
| − | | |
| − | '''Note!'''
| |
| − | | |
| − | Due to a bug in GnuTLS the verification of host certificates doesn't seem to work. Lftp will return "cd: Fatal error: Certificate verification: Not trusted" when the connection is opened. Please use:
| |
| − | set ssl:verify-certificate false
| |
| − | until this is solved.
| |
| − | | |
| − | == Usage ==
| |
| − | | |
| − | Assuming that you've managed to obtain a working lftp binary, there are some quirks that lftp has together with the Swestore WebDAV door.
| |
| − | | |
| − | When giving a directory path to a command, it should end with a trailing slash to indicate that it is a directory. If this is omitted, the client will get a redirection response that the tool doesn't handle properly.
| |
| − | | |
| − | All the commands mentioned in this section and the previous configuration section are commands inside of a running <code>lftp</code> program.
| |
| − | | |
| − | Some sample tasks that can be achieved with lftp is retrieving or uploading single files or whole directory trees.
| |
| − | | |
| − | The command to connect to the door is:
| |
| − | <nowiki>
| |
| − | open https://webdav.swestore.se/</nowiki>
| |
| − | | |
| − | after which you can navigate around with the use of the 'cd' command:
| |
| − | <nowiki>
| |
| − | cd snic/project_name_here/foo/</nowiki>
| |
| − | | |
| − | Individual files can be manipulated using the ''get'' and ''put'' commands and the ''mget'' and ''mirror'' commands can transfer multiple files and whole trees, respectively.
| |
| − | | |
| − | The program has interactive help for any command through the ''help'' command.
| |
| − | | |
| − | For mirroring the flags ''-R'' and ''-c'' are particularly relevant as ''-R'' controls the direction of the operation - if it is present the transfer is mirroring '''to''' the server, otherwise it's mirroring '''from''' the server.
| |
| − | | |
| − | ''-c'' indicates that the operation should resume whenever possible which may improve synchronization time if you know that any partial files present on the other side are identical to the local files.
| |
| − | | |
| − | For example, the command
| |
| − | <nowiki>mirror -c A B</nowiki>
| |
| − | | |
| − | will download all of the remote directory ''A'' into the local directory named ''B''.
| |
| − | | |
| − | The command
| |
| − | <nowiki>mirror -cR C D</nowiki>
| |
| − | | |
| − | will upload all of the local directory ''C'' into the remote directory named ''D''. Note that the role of the directories is reversed compared to the previous example.
| |
| − | | |
| − | == Build instructions ==
| |
| − | | |
| − | Building lftp and its dependencies from scratch does not require any particular build flags but you might want to install it into a private destination (prefix) to avoid it interfering with system-provided libraries.
| |
| − | | |
| − | libnettle depends on GMP and lftp depends on readline and gperf, the distribution packages for those are sufficient on Ubuntu and CentOS 5/7 and 6.2.
| |
| − | | |
| − | These instructions assume that the ''bash'' shell is used when building, ''tcsh'' and other shells will have slightly different syntax for environment variables.
| |
| − | <nowiki>
| |
| − | export LFTP_PREFIX="${HOME}/local"
| |
| − | export PKG_CONFIG_PATH="${LFTP_PREFIX}/lib/pkgconfig:${PKG_CONFIG_PATH}"
| |
| − | export CPPFLAGS="-I${LFTP_PREFIX}/include"
| |
| − | export LDFLAGS="-L${LFTP_PREFIX}/lib -Wl,-R${LFTP_PREFIX}/lib"</nowiki>
| |
| − | | |
| − | Start by extracting the source distributions:
| |
| − | <nowiki>
| |
| − | tar xzf ~/Downloads/nettle-2.4.tar.gz
| |
| − | tar xJf ~/Downloads/gnutls-3.0.12.tar.xz
| |
| − | tar xJf ~/Downloads/lftp-4.3.5.tar.xz</nowiki>
| |
| − | | |
| − | You might want to avoid building a shared gnutls, so passing
| |
| − | <nowiki>
| |
| − | --enable-static --disable-shared</nowiki>
| |
| − | | |
| − | on the gnutls configure command line might be a good idea.
| |
| − | | |
| − | libnettle is a dependency for gnutls, and gnutls is a dependency for lftp, so we build them in that order:
| |
| − | <nowiki>
| |
| − | mkdir ${LFTP_PREFIX}
| |
| − | | |
| − | pushd nettle-2.4
| |
| − | ./configure --prefix=${LFTP_PREFIX} &&
| |
| − | make && make install
| |
| − | popd
| |
| − | | |
| − | pushd gnutls-3.0.12
| |
| − | ./configure --prefix=${LFTP_PREFIX} --without-p11-kit &&
| |
| − | make && make install
| |
| − | popd
| |
| − | | |
| − | pushd lftp-4.3.5
| |
| − | ./configure --prefix=${LFTP_PREFIX} &&
| |
| − | make && make install
| |
| − | popd</nowiki>
| |
| − | | |
| − | If the platform already has development files for p11-kit there is no harm in letting it use them (it allows gnutls to understand PKCS-style certificates), but it's nothing that lftp can utilize so it's not considered a dependency in this document.
| |
| − | | |
| − | After the build process completes, a lftp binary will exist in ''${LFTP_PREFIX}/bin'' and depend on the gnutls shared library in ''${LFTP_PREFIX}/lib'' if you did not build it statically.
| |
| − | | |
| − | = Credits =
| |
| − | | |
| − | This guide was written by Lars Viklund
| |
