Merge pull request #18 from tomdesair/feature/generic-upload-id-factory

Support custom UploadIdFactory implementations that are not UUID-based

README.md

@@ -13,7 +13,7 @@ You can add the latest stable version of this library to your application using
     <dependency>
       <groupId>me.desair.tus</groupId>
       <artifactId>tus-java-server</artifactId>
-      <version>1.0.0-1.3</version>
+      <version>1.0.0-2.0</version>
     </dependency>
 
 The main entry point of the library is the `me.desair.tus.server.TusFileUploadService.process(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse)` method. You can call this method inside a `javax.servlet.http.HttpServlet`, a `javax.servlet.Filter` or any REST API controller of a framework that gives you access to `HttpServletRequest` and `HttpServletResponse` objects. In the following list, you can find some example implementations:
@@ -49,20 +49,25 @@ The first step is to create a `TusFileUploadService` object using its constructo
 * `withDownloadFeature()`: Enable the unofficial `download` extension that also allows you to download uploaded bytes.
 * `addTusExtension(TusExtension)`: Add a custom (application-specific) extension that implements the `me.desair.tus.server.TusExtension` interface. For example you can add your own extension that checks authentication and authorization policies within your application for the user doing the upload.
 * `disableTusExtension(String)`: Disable the `TusExtension` for which the `getName()` method matches the provided string. The default extensions have names "creation", "checksum", "expiration", "concatenation", "termination" and "download". You cannot disable the "core" feature.
+* `withUploadIdFactory(UploadIdFactory)`: Provide a custom `UploadIdFactory` implementation that should be used to generate identifiers for the different uploads. The default implementation generates identifiers using a UUID (`UUIDUploadIdFactory`). Another example implementation of a custom ID factory is the system-time based `TimeBasedUploadIdFactory` class.
 
 
-For now this library only provides file system-based storage and locking options. You can however provide your own implementation of a `UploadStorageService` and `UploadLockingService` using the methods `withUploadStorageService(UploadStorageService)` and `withUploadLockingService(UploadLockingService)` in order to support different types of upload storage.
+For now this library only provides filesystem based storage and locking options. You can however provide your own implementation of a `UploadStorageService` and `UploadLockingService` using the methods `withUploadStorageService(UploadStorageService)` and `withUploadLockingService(UploadLockingService)` in order to support different types of upload storage.
 
 ### 2. Processing an upload
 To process an upload request you have to pass the current `javax.servlet.http.HttpServletRequest` and `javax.servlet.http.HttpServletResponse` objects to the `me.desair.tus.server.TusFileUploadService.process()` method. Typical places were you can do this are inside Servlets, Filters or REST API Controllers (see [examples](#quick-start-and-examples)).
 
 Optionally you can also pass a `String ownerKey` parameter. The `ownerKey` can be used to have a hard separation between uploads of different users, groups or tenants in a multi-tenant setup. Examples of `ownerKey` values are user ID's, group names, client ID's...
 
 ### 3. Retrieving the uploaded bytes and metadata within the application
-Once the upload has been completed by the user, the business logic layer of your application needs to retrieve and do something with the uploaded bytes. This can be achieved by using the `me.desair.tus.server.TusFileUploadService.getUploadedBytes(String uploadURL)` method. The passed `uploadURL` value should be the upload url used by the client to which the file was uploaded. Therefor your application should pass the upload URL of completed uploads to the backend. Optionally, you can also pass an `ownerKey` value to this method in case your application chooses to process uploads using owner keys. Examples of values that can be used as an `ownerKey` are: an internal user identifier, a session ID, the name of the subpart of your application...
+Once the upload has been completed by the user, the business logic layer of your application needs to retrieve and do something with the uploaded bytes. For example it could read the contents of the file, or move the uploaded bytes to their final persistent storage location. Retrieving the uploaded bytes in the backend can be achieved by using the `me.desair.tus.server.TusFileUploadService.getUploadedBytes(String uploadURL)` method. The passed `uploadURL` value should be the upload url used by the client to which the file was uploaded. Therefor your application should pass the upload URL of completed uploads to the backend. Optionally, you can also pass an `ownerKey` value to this method in case your application chooses to process uploads using owner keys. Examples of values that can be used as an `ownerKey` are: an internal user identifier, a session ID, the name of the subpart of your application...
+
+Using the `me.desair.tus.server.TusFileUploadService.getUploadInfo(String uploadURL)` method you can retrieve metadata about a specific upload process. This includes metadata provided by the client as well as metadata kept by the library like creation timestamp, creator ip-address list, upload length... The method `UploadInfo.getId()` will return the unique identifier of this upload encapsulated in an `UploadId` instance. The original (custom generated) identifier object of this upload can be retrieved using `UploadId.getOriginalObject()`. A URL safe string representation of the identifier is returned by `UploadId.toString()`. It is highly recommended to consult the [JavaDoc of both classes](https://tus.desair.me/).
 
 ### 4. Upload cleanup
-After having processed the uploaded bytes on the server backend, it's important to cleanup the uploaded bytes. This can be done by calling the `me.desair.tus.server.TusFileUploadService.deleteUpload(String uploadURI)` method. This will remove the uploaded bytes and any associated upload information from the storage backend. Alternatively, a client can also remove an (in-progress) upload using the [termination extension](https://tus.io/protocols/resumable-upload.html#termination). 
+After having processed the uploaded bytes on the server backend (e.g. copy them to their final persistent location), it's important to cleanup the (temporary) uploaded bytes. This can be done by calling the `me.desair.tus.server.TusFileUploadService.deleteUpload(String uploadURI)` method. This will remove the uploaded bytes and any associated upload information from the storage backend. Alternatively, a client can also remove an (in-progress) upload using the [termination extension](https://tus.io/protocols/resumable-upload.html#termination). 
+
+Next to removing uploads after they have been completed and processed by the backend, it is also recommended to schedule a regular maintenance task to clean up any expired uploads or locks. Cleaning up expired uploads and locks can be achieved using the `me.desair.tus.server.TusFileUploadService.cleanup()` method.
 
 ## Compatible Client Implementations
 This tus protocol implementation has been [tested](https://github.com/tomdesair/tus-java-server-spring-demo) with the [Uppy file upload client](https://uppy.io/). This repository also contains [many automated integration tests](https://github.com/tomdesair/tus-java-server/blob/master/src/test/java/me/desair/tus/server/ITTusFileUploadService.java) that validate the tus protocol server implementation using plain HTTP requests. So in theory this means we're compatible with any tus 1.0.0 compliant client.
@@ -71,4 +76,4 @@ This tus protocol implementation has been [tested](https://github.com/tomdesair/
 This artifact is versioned as `A.B.C-X.Y` where `A.B.C` is the version of the implemented tus protocol (currently 1.0.0) and `X.Y` is the version of this library. 
 
 ## Contributing
-This library comes without any warranty and is released under a [MIT license](https://github.com/tomdesair/tus-java-server/blob/master/LICENSE). If you encounter any bugs or if you have an idea for a useful improvement you are welcome to [open a new issue](https://github.com/tomdesair/tus-java-server/issues) or to [create a pull request](https://github.com/tomdesair/tus-java-server/pulls) with the proposed implementation. Please note that any contributed code needs to be accompanied by automated unit and/or integration tests and comply with the [define code-style](https://github.com/tomdesair/tus-java-server/blob/master/checkstyle.xml).
+This library comes without any warranty and is released under a [MIT license](https://github.com/tomdesair/tus-java-server/blob/master/LICENSE). If you encounter any bugs or if you have an idea for a useful improvement you are welcome to [open a new issue](https://github.com/tomdesair/tus-java-server/issues) or to [create a pull request](https://github.com/tomdesair/tus-java-server/pulls) with the proposed implementation. Please note that any contributed code needs to be accompanied by automated unit and/or integration tests and comply with the [defined code-style](https://github.com/tomdesair/tus-java-server/blob/master/checkstyle.xml).

pom.xml

@@ -4,7 +4,7 @@
 
     <groupId>me.desair.tus</groupId>
     <artifactId>tus-java-server</artifactId>
-    <version>1.0.0-1.4-SNAPSHOT</version>
+    <version>1.0.0-2.0-SNAPSHOT</version>
     <packaging>jar</packaging>
 
     <name>${project.groupId}:${project.artifactId}</name>
@@ -178,11 +178,8 @@
                 <configuration>
                     <!-- Sets the VM argument line used when unit tests are run. -->
                     <argLine>${surefireArgLine}</argLine>
-                    <!-- Disable SLF4j logging on Maven builds -->
-                    <systemPropertyVariables>
-                        <org.slf4j.simpleLogger.defaultLogLevel>off</org.slf4j.simpleLogger.defaultLogLevel>
-                        <org.slf4j.simpleLogger.showDateTime>true</org.slf4j.simpleLogger.showDateTime>
-                    </systemPropertyVariables>
+                    <!-- Sonar test count workaround -->
+                    <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
                 </configuration>
             </plugin>
 
@@ -202,11 +199,6 @@
                     <argLine>${failsafeArgLine}</argLine>
                     <!-- Sonar test count workaround -->
                     <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
-                    <!-- Disable SLF4j logging on Maven builds -->
-                    <systemPropertyVariables>
-                        <org.slf4j.simpleLogger.defaultLogLevel>off</org.slf4j.simpleLogger.defaultLogLevel>
-                        <org.slf4j.simpleLogger.showDateTime>true</org.slf4j.simpleLogger.showDateTime>
-                    </systemPropertyVariables>
                 </configuration>
                 <executions>
                     <execution>

src/main/java/me/desair/tus/server/HttpHeader.java

@@ -92,6 +92,12 @@ public class HttpHeader {
      */
     public static final String TUS_CHECKSUM_ALGORITHM = "Tus-Checksum-Algorithm";
 
+    /**
+     * The X-Forwarded-For (XFF) HTTP header field is a common method for identifying the originating IP address of a
+     * client connecting to a web server through an HTTP proxy or load balancer.
+     */
+    public static final String X_FORWARDED_FOR = "X-Forwarded-For";
+
     private HttpHeader() {
         //This is an utility class to hold constants
     }

src/main/java/me/desair/tus/server/TusFileUploadService.java

@@ -17,6 +17,7 @@
 import me.desair.tus.server.exception.TusException;
 import me.desair.tus.server.expiration.ExpirationExtension;
 import me.desair.tus.server.termination.TerminationExtension;
+import me.desair.tus.server.upload.UUIDUploadIdFactory;
 import me.desair.tus.server.upload.UploadIdFactory;
 import me.desair.tus.server.upload.UploadInfo;
 import me.desair.tus.server.upload.UploadLock;
@@ -44,7 +45,7 @@ public class TusFileUploadService {
 
     private UploadStorageService uploadStorageService;
     private UploadLockingService uploadLockingService;
-    private UploadIdFactory idFactory = new UploadIdFactory();
+    private UploadIdFactory idFactory = new UUIDUploadIdFactory();
     private final LinkedHashMap<String, TusExtension> enabledFeatures = new LinkedHashMap<>();
     private final Set<HttpMethod> supportedHttpMethods = EnumSet.noneOf(HttpMethod.class);
     private boolean isThreadLocalCacheEnabled = false;
@@ -93,6 +94,24 @@ public TusFileUploadService withMaxUploadSize(Long maxUploadSize) {
         return this;
     }
 
+    /**
+     * Provide a custom {@link UploadIdFactory} implementation that should be used to generate identifiers for
+     * the different uploads. Example implementation are {@link me.desair.tus.server.upload.UUIDUploadIdFactory} and
+     * {@link me.desair.tus.server.upload.TimeBasedUploadIdFactory}.
+     *
+     * @param uploadIdFactory The custom {@link UploadIdFactory} implementation
+     * @return The current service
+     */
+    public TusFileUploadService withUploadIdFactory(UploadIdFactory uploadIdFactory) {
+        Validate.notNull(uploadIdFactory, "The UploadIdFactory cannot be null");
+        String previousUploadURI = this.idFactory.getUploadURI();
+        this.idFactory = uploadIdFactory;
+        this.idFactory.setUploadURI(previousUploadURI);
+        this.uploadStorageService.setIdFactory(this.idFactory);
+        this.uploadLockingService.setIdFactory(this.idFactory);
+        return this;
+    }
+
     /**
      * Provide a custom {@link UploadStorageService} implementation that should be used to store uploaded bytes and
      * metadata ({@link UploadInfo}).
@@ -108,7 +127,7 @@ public TusFileUploadService withUploadStorageService(UploadStorageService upload
         uploadStorageService.setIdFactory(this.idFactory);
         //Update the upload storage service
         this.uploadStorageService = uploadStorageService;
-        prepareCacheIfEnable();
+        prepareCacheIfEnabled();
         return this;
     }
 
@@ -125,7 +144,7 @@ public TusFileUploadService withUploadLockingService(UploadLockingService upload
         uploadLockingService.setIdFactory(this.idFactory);
         //Update the upload storage service
         this.uploadLockingService = uploadLockingService;
-        prepareCacheIfEnable();
+        prepareCacheIfEnabled();
         return this;
     }
 
@@ -138,9 +157,9 @@ public TusFileUploadService withUploadLockingService(UploadLockingService upload
      */
     public TusFileUploadService withStoragePath(String storagePath) {
         Validate.notBlank(storagePath, "The storage path cannot be blank");
-        withUploadStorageService(new DiskStorageService(idFactory, storagePath));
-        withUploadLockingService(new DiskLockingService(idFactory, storagePath));
-        prepareCacheIfEnable();
+        withUploadStorageService(new DiskStorageService(storagePath));
+        withUploadLockingService(new DiskLockingService(storagePath));
+        prepareCacheIfEnabled();
         return this;
     }
 
@@ -152,7 +171,7 @@ public TusFileUploadService withStoragePath(String storagePath) {
      */
     public TusFileUploadService withThreadLocalCache(boolean isEnabled) {
         this.isThreadLocalCacheEnabled = isEnabled;
-        prepareCacheIfEnable();
+        prepareCacheIfEnabled();
         return this;
     }
 
@@ -450,7 +469,7 @@ private void updateSupportedHttpMethods() {
         }
     }
 
-    private void prepareCacheIfEnable() {
+    private void prepareCacheIfEnabled() {
         if (isThreadLocalCacheEnabled && uploadStorageService != null && uploadLockingService != null) {
             ThreadLocalCachedStorageAndLockingService service =
                     new ThreadLocalCachedStorageAndLockingService(
@@ -461,4 +480,5 @@ private void prepareCacheIfEnable() {
             this.uploadLockingService = service;
         }
     }
+
 }

src/main/java/me/desair/tus/server/concatenation/ConcatenationPostRequestHandler.java

@@ -44,7 +44,7 @@ public void process(HttpMethod method, TusServletRequest servletRequest,
                 //reset the length, just to be sure
                 uploadInfo.setLength(null);
                 uploadInfo.setUploadType(UploadType.CONCATENATED);
-                uploadInfo.setConcatenationParts(Utils.parseConcatenationIDsFromHeader(uploadConcatValue));
+                uploadInfo.setConcatenationPartIds(Utils.parseConcatenationIDsFromHeader(uploadConcatValue));
 
                 uploadStorageService.getUploadConcatenationService().merge(uploadInfo);
 

src/main/java/me/desair/tus/server/core/CorePatchRequestHandler.java

@@ -55,6 +55,11 @@ public void process(HttpMethod method, TusServletRequest servletRequest,
         if (found) {
             servletResponse.setHeader(HttpHeader.UPLOAD_OFFSET, Objects.toString(uploadInfo.getOffset()));
             servletResponse.setStatus(HttpServletResponse.SC_NO_CONTENT);
+
+            if (!uploadInfo.isUploadInProgress()) {
+                log.info("Upload with ID {} at location {} finished successfully",
+                        uploadInfo.getId(), servletRequest.getRequestURI());
+            }
         } else {
             log.error("The patch request handler could not find the upload for URL " + servletRequest.getRequestURI()
                 + ". This means something is really wrong the request validators!");

src/main/java/me/desair/tus/server/creation/CreationPostRequestHandler.java

@@ -47,11 +47,12 @@ public void process(HttpMethod method, TusServletRequest servletRequest,
         servletResponse.setHeader(HttpHeader.LOCATION, url);
         servletResponse.setStatus(HttpServletResponse.SC_CREATED);
 
-        log.debug("Create upload location {}", url);
+        log.info("Created upload with ID {} at {} for ip address {} with location {}",
+                info.getId(), info.getCreationTimestamp(), info.getCreatorIpAddresses(), url);
     }
 
     private UploadInfo buildUploadInfo(HttpServletRequest servletRequest) {
-        UploadInfo info = new UploadInfo();
+        UploadInfo info = new UploadInfo(servletRequest);
 
         Long length = Utils.getLongHeader(servletRequest, HttpHeader.UPLOAD_LENGTH);
         if (length != null) {

src/main/java/me/desair/tus/server/upload/TimeBasedUploadIdFactory.java

@@ -0,0 +1,33 @@
+package me.desair.tus.server.upload;
+
+import java.io.Serializable;
+
+import org.apache.commons.lang3.StringUtils;
+
+/**
+ * Alternative {@link UploadIdFactory} implementation that uses the current system time to generate ID's.
+ * Since time is not unique, this upload ID factory should not be used in busy, clustered production systems.
+ */
+public class TimeBasedUploadIdFactory extends UploadIdFactory {
+
+    @Override
+    protected Serializable getIdValueIfValid(String extractedUrlId) {
+        Long id = null;
+
+        if (StringUtils.isNotBlank(extractedUrlId)) {
+            try {
+                id = Long.parseLong(extractedUrlId);
+            } catch (NumberFormatException ex) {
+                id = null;
+            }
+        }
+
+        return id;
+    }
+
+    @Override
+    public synchronized UploadId createId() {
+        return new UploadId(System.currentTimeMillis());
+    }
+
+}

src/main/java/me/desair/tus/server/upload/UUIDUploadIdFactory.java

@@ -0,0 +1,29 @@
+package me.desair.tus.server.upload;
+
+import java.io.Serializable;
+import java.util.UUID;
+
+/**
+ * Factory to create unique upload IDs. This factory can also parse the upload identifier
+ * from a given upload URL.
+ */
+public class UUIDUploadIdFactory extends UploadIdFactory {
+
+    @Override
+    protected Serializable getIdValueIfValid(String extractedUrlId) {
+        UUID id = null;
+        try {
+            id = UUID.fromString(extractedUrlId);
+        } catch (IllegalArgumentException ex) {
+            id = null;
+        }
+
+        return id;
+    }
+
+    @Override
+    public synchronized UploadId createId() {
+        return new UploadId(UUID.randomUUID());
+    }
+
+}