Globus transfer for zstash extract

[golaz]

Given the success of #154, it would be great to have the option of using Globus for extracting files.

The idea is that a user could be on Chrysalis and retrieve files from a zstash archive on NERSC HPSS. This would greatly reduce the number of steps required. (Currently a user must manually transfer zstash files, or extract on NERSC and then manually transfer the files to Chrysalis).

For extraction, we only need to implement blocking Globus transfers. I think much of the infrastructure that was put in place for zstash create can be reused for zstash extract.

[lukaszlacinski]

@golaz #154 integrates Globus with all zstash commands including zstash extract. However, --non-blocking is supported only by zstash create.

[golaz]

@lukaszlacinski : I did not realize that you had already implemented all these features. This is wonderful!

I ran a few tests today from Chrysalis accessing a zstash archive on NERSC HPSS. Everything worked as expected.

Below is some of the functionality I tested (ls, extract, check; single and multiple workers):

zstash ls --hpss=globus://nersc/home/f/forsyth/E3SMv2/v2.LR.historical_0101 archive/rest/*

zstash extract --keep --hpss=globus://nersc/home/f/forsyth/E3SMv2/v2.LR.historical_0101 archive/rest/2015-01-01-00000/*

zstash extract --keep --workers=2 --hpss=globus://nersc/home/f/forsyth/E3SMv2/v2.LR.historical_0101 archive/rest/*

zstash check -v --keep --workers=2 --hpss=globus://nersc/home/f/forsyth/E3SMv2/v2.LR.historical_0101 000000.tar 000001.tar

zstash check -v --keep --workers=3 --hpss=globus://nersc/home/f/forsyth/E3SMv2/v2.LR.historical_0101 000002.tar 000003.tar 000004.tar

Thanks for implementing the Globus functionality. It will make life much so easier for zstash users.

[forsyth2]

@lukaszlacinski @golaz This is great! However, I'd like to note that we should update tests/test_globus.py in #154 to include check and/or extract. (We could update the tests in a later pull request though if we want to get the create functionality merged right away).

[forsyth2]

@golaz Do we want to advertise/officially support the Globus extract capability with the in-progress release (v1.2.0)? #154 doesn't include information on extract in docs/source/usage.rst nor is extract tested in tests/test_globus.py. I think it may be better to make an issue (or reopen this one) to address these two things before we officially support it.

[forsyth2]

I created #192 to address these points.