Accessing Swestore with lftp

Introduction

Written by: Lars Viklund

lftp is a file transfer tool that understands a range of protocols from plain old FTP, HTTP, SCP to more esoteric ones like WebDAV and BitTorrent.

The benefit over tools like cURL is that it has interactive traversal of the directory hierarchy, as well as powerful mass-transfer functionality like the mirror command.

Out of the protocols mentioned above, the ones that aligns most with the access doors that Swestore offers are the WebDAV over HTTP/HTTPS protocols.

Authentication against Swestore over the WebDAV door is done with client certificates over HTTPS, where there is a choice of either using the real client certificate or by using a RFC or VOMS proxy certificate generated from the real certificate.

Proxy certificates are preferred as they are valid for a limited period of time, especially as lftp does not seem to offer any way to enter any passphrase to unlock keys, so in order to use a protected real certificate, it will have to be unlocked in advance.

Notable is that while file uploads are securely tunneled inside a SSL connection, downloads are in the plain from storage nodes. As such, the usual guidelines about sensitive data holds, where you should aim to have your data encrypted before transmitting it.

Required software versions

The versions of lftp and its major dependency GNUTLS that have been verified to work with proxy certificates are:

• lftp 4.3.3
• gnutls 2.12.0
• libnettle 2.4 (for building gnutls)

This gnutls version is the absolute minimum version that will work, any version prior to that (2.10.5 and below) will not be able to connect to the door.

This means that the gnutls version in Ubuntu oneric and older, Scientific Linux/RHEL/CentOS 6.1 and older will not work.

Ubuntu precise has a sufficiently new gnutls and will work out of the box.

lftp settings

The settings in lftp that are needed to interface with Swestore are:

 set ssl:ca-file /etc/grid-security/certificates/NorduGrid.pem
 set ssl:check-hostname true
 set ssl:verify-certificate true

and either

 set ssl:key-file /home/you/private/userkey-unlocked.pem
 set ssl:cert-file /home/you/private/usercert.pem

or

 set ssl:key-file /tmp/x509up_u1234
 set ssl:cert-file /tmp/x509up_u1234

where the former set of files is for when you wish to use a passphrase-less regular certificate and where the latter will use a proxy certificate generated by the arcproxy tool.

The CA file is for verifying the identity of the server and is strongly recommended that both certificate and hostname verification is in effect to ensure that the server communicated with is the intended target machine.

They can be stored in the configuration file named ~/.lftp/rc together with any other commands you wish to perform during lftp startup.

Usage

Assuming that you've managed to obtain a working lftp binary, there are some quirks that lftp has together with the Swestore WebDAV door.

When giving a directory path to a command, it should end with a trailing slash to indicate that it is a directory. If this is omitted, the client will get a redirection response that the tool doesn't handle properly.

Some sample tasks that can be achieved with lftp is retrieving or uploading single files or whole directory trees.

The command to connect to the door is:

 open https://webdav.swestore.se/

after which you can navigate around with the use of the 'cd' command:

 cd snic/project_name_here/foo/

Individual files can be manipulated using the get and put commands and the mget and mirror commands can transfer multiple files and whole trees, respectively.

The program has interactive help for any command through the help command.

For mirroring the flags -R and -c are particularly relevant as -R controls the direction of the operation - if it is present the transfer is mirroring to the server, otherwise it's mirroring from the server.

-c indicates that the operation should resume whenever possible which may improve synchronization time if you know that any partial files present on the other side are identical to the local files.

For example, the command

mirror -c A B

will download all of the remote directory A into the local directory named B.

The command

mirror -cR C D

will upload all of the local directory C into the remote directory named D. Note that the role of the directories is reversed compared to the previous example.

Build instructions

Building lftp and its dependencies from scratch does not require any particular build flags but you might want to install it into a private destination (prefix) to avoid it interfering with system-provided libraries.

libnettle depends on GMP and lftp depends on readline and gperf, the distribution packages for those are sufficient on Ubuntu and CentOS 5/7 and 6.2.

These instructions assume that the bash shell is used when building, tcsh and other shells will have slightly different syntax for environment variables.

 export LFTP_PREFIX="${HOME}/local"
 export PKG_CONFIG_PATH="${LFTP_PREFIX}/lib/pkgconfig:${PKG_CONFIG_PATH}"
 export CPPFLAGS="-I${LFTP_PREFIX}/include"
 export LDFLAGS="-L${LFTP_PREFIX}/lib -Wl,-R${LFTP_PREFIX}/lib"

Start by extracting the source distributions:

 tar xzf ~/Downloads/nettle-2.4.tar.gz
 tar xJf ~/Downloads/gnutls-3.0.12.tar.xz
 tar xJf ~/Downloads/lftp-4.3.5.tar.xz

You might want to avoid building a shared gnutls, so passing

 --enable-static --disable-shared

on the gnutls configure command line might be a good idea.

libnettle is a dependency for gnutls, and gnutls is a dependency for lftp, so we build them in that order:

 mkdir ${LFTP_PREFIX}

 pushd nettle-2.4
 ./configure --prefix=${LFTP_PREFIX} &&
 make && make install
 popd

 pushd gnutls-3.0.12
 ./configure --prefix=${LFTP_PREFIX} --without-p11-kit &&
 make && make install
 popd

 pushd lftp-4.3.5
 ./configure --prefix=${LFTP_PREFIX} &&
 make && make install
 popd

If the platform already has development files for p11-kit there is no harm in letting it use them (it allows gnutls to understand PKCS-style certificates), but it's nothing that lftp can utilize so it's not considered a dependency in this document.

After the build process completes, a lftp binary will exist in ${LFTP_PREFIX}/bin and depend on the gnutls shared library in ${LFTP_PREFIX}/lib if you did not build it statically.