diff --git a/README.md b/README.md index 2c8a8a4..8fd937e 100644 --- a/README.md +++ b/README.md @@ -266,7 +266,47 @@ from the YAML string taking precedence over values from the YAML file. #### Transfer Adapters -TBD +Git LFS servers and clients can implement and negotiate different [transfer adapters] +(https://github.com/git-lfs/git-lfs/blob/master/docs/api/basic-transfers.md). Typically, +Git LFS will only define a `basic` transfer mode and support that. `basic` is simple +and efficient for direct-to-storage uploads for backends that support uploading using +a single `PUT` request. + +To support more complex, and especially multi-part uploads (uploads done using more +than one HTTP request, each with a different part of a large file) directly to backends +that support that, Giftless adds support for a non-standard `multipart-basic` transfer +mode. Note that this can only work with specific backends that support this type of +functionality. + +##### Enabling Multipart Transfer Mode + +You can enable multipart transfers by adding the following lines to your Giftless config +file: + +```yaml +TRANSFER_ADAPTERS: + # Add the following lines: + multipart-basic: + factory: giftless.transfer.multipart:factory + options: + storage_class: giftless.storage.azure:AzureBlobsStorage + storage_options: + connection_string: "somesecretconnectionstringhere" + container_name: my-multipart-storage +``` + +You must specify a `storage_class` that supports multipart transfers (implements the `MultipartStorage` +interface). Currently, these are: +* `giftless.storage.azure:AzureBlobsStorage` - Azure Blob Storage + +The following additional options are available for `multipart-basic` transfer adapter: +* `action_lifetime` - The maximal lifetime in seconds for signed multipart actions; Because multipart +uploads tend to be of very large files and can easily take hours to complete, we recommend setting this +to a few hours; The default is 6 hours. +* `max_part_size` - Maximal length in bytes of a single part upload. The default is 10MB. + +See the specific storage adapter for additional backend-specific configuration options to be added under +`storage_options`. #### Authenticators diff --git a/giftless/transfer/multipart.py b/giftless/transfer/multipart.py index 7cb4b97..52c7cea 100644 --- a/giftless/transfer/multipart.py +++ b/giftless/transfer/multipart.py @@ -10,6 +10,7 @@ from giftless.util import get_callable DEFAULT_PART_SIZE = 10240000 # 10mb +DEFAULT_ACTION_LIFETIME = 6 * 3600 # 6 hours class MultipartTransferAdapter(PreAuthorizingTransferAdapter, ViewProvider): @@ -75,7 +76,8 @@ def _check_object(self, prefix: str, oid: str, size: int): raise exc.InvalidObject('Object size does not match') -def factory(storage_class, storage_options, action_lifetime: int, max_part_size: int = DEFAULT_PART_SIZE): +def factory(storage_class, storage_options, action_lifetime: int = DEFAULT_ACTION_LIFETIME, + max_part_size: int = DEFAULT_PART_SIZE): """Factory for multipart transfer adapter with storage """ try: