Skip to content

Commit

Permalink
Started adding test suite. Started adding code for allowing users to …
Browse files Browse the repository at this point in the history
…update and delete existing files (#22) and started refactoring to allow for multiple range headers (#16)
  • Loading branch information
zachsa committed Jun 26, 2023
1 parent 7f6f9e8 commit 5906416
Show file tree
Hide file tree
Showing 21 changed files with 715 additions and 239 deletions.
17 changes: 9 additions & 8 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,15 @@ yarn-error.log*
lerna-debug.log*

# Local dev environment
/test/*
!/test/
/test/mnt1/*
!/test/mnt1/
!/test/mnt1/README.md
/test/mnt2/*
!/test/mnt2/
!/test/mnt2/README.md
/test-mnts/*
!/test-mnts/
!/test-mnts/README.md
/test-mnts/mnt1/*
!/test-mnts/mnt1/
!/test-mnts/mnt1/README.md
/test-mnts/mnt2/*
!/test-mnts/mnt2/
!/test-mnts/mnt2/README.md

# Diagnostic reports (https://nodejs.org/api/report.html)
report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
Expand Down
11 changes: 10 additions & 1 deletion .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,12 @@
{
"cSpell.words": ["Dockerized", "geotiff", "mbuffer", "mnemosyne", "noindex", "somisana", "Zarr"]
"cSpell.words": [
"Dockerized",
"geotiff",
"keepalive",
"mbuffer",
"mnemosyne",
"noindex",
"somisana",
"Zarr"
]
}
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM node:20.3.0-alpine
FROM node:20.3.1-alpine

ARG TC=UTC
ARG NODE_ENV=production
Expand Down
34 changes: 17 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ Tools that understand cloud-optimized formats (i.e. COGs, Zarrs, etc.) should wo
curl \
-X GET \
--keepalive-time 1200 \
https://<domain>/filename.tif
http://localhost:3000/filename.tif
```

**_Download partial file via cURL_**
Expand All @@ -67,7 +67,7 @@ curl \
curl \
-H "Range bytes=12-20" \
-X GET \
https://<domain>/filename.tif
http://localhost:3000/filename.tif
```

### Customize GET requests via URL params
Expand All @@ -93,7 +93,7 @@ Any folder that includes an `index.html` file will be served as a website. One p

```js
// Client side JavaScript
fetch('https://<domain>/directory?json')
fetch('http://localhost:3000/directory?json')
.then(res => res.json())
.then(json => {
document.getElementsByTagName('body')[0].append(JSON.stringify(json))
Expand All @@ -108,7 +108,7 @@ Any files/folder in the exposed volume will be served. To upload files to the se

### cURL examples

Here are some examples using `cURL` (and some notes):
Here are some examples using `cURL` (and some notes) using the default authentication token on localhost. (Look at server logs for authentication tokens):

- For `PUT` requests I find it's necessary to pipe the output to `cat`, since `cURL` won't print output to stdout (who knows why) when using `POST` and `PUT` requests
- The `-T` header means 'transfer file'. Use this instead of `--data-binary`, since the latter will try to load your entire file into memory before sending
Expand All @@ -126,10 +126,10 @@ cat ./some/local/cog.tiff \
--progress-bar \
--keepalive-time 1200 \
-X PUT \
-H "Authorization: Bearer <token>" \
-H "Authorization: Bearer f7efafd138da71cdcb0aa4767da9b6ee:1cc83802e7a4a831b17efa9ffecf4822" \
-H "Content-Type: application/octet-stream" \
--data-binary @- \
https://<domain>/some/deep/nested/directory/cog.tif \
http://localhost:3000/some/deep/nested/directory/cog.tif \
| cat
```

Expand All @@ -142,14 +142,14 @@ curl \
--progress-bar \
--keepalive-time 1200 \
-X PUT \
-H "Authorization: Bearer <token>" \
-H "Authorization: Bearer f7efafd138da71cdcb0aa4767da9b6ee:1cc83802e7a4a831b17efa9ffecf4822" \
-T \
./some/local/cog.tiff \
https://<domain>/some/deep/nested/directory/cog.tif \
http://localhost:3000/some/deep/nested/directory/cog.tif \
| cat
```

And then that file can be retrieved at `https://<domain>/some/deep/nested/directory/cog.tif`.
And then that file can be retrieved at `http://localhost:3000/some/deep/nested/directory/cog.tif`.

#### Chunked uploads

Expand All @@ -163,11 +163,11 @@ cat ./some/local/cog.tiff \
--progress-bar \
--keepalive-time 1200 \
-X PUT \
-H "Authorization: Bearer <token>" \
-H "Authorization: Bearer f7efafd138da71cdcb0aa4767da9b6ee:1cc83802e7a4a831b17efa9ffecf4822" \
-H "Content-Type: application/octet-stream" \
-T \
- \
https://<domain>/some/deep/nested/directory/cog.tif \
http://localhost:3000/some/deep/nested/directory/cog.tif \
| cat
```

Expand All @@ -184,9 +184,9 @@ mbuffer \
--keepalive-time 1200 \
-X PUT \
-H "Content-Type: application/octet-stream" \
-H "Authorization: Bearer <token>" \
-H "Authorization: Bearer f7efafd138da71cdcb0aa4767da9b6ee:1cc83802e7a4a831b17efa9ffecf4822" \
--data-binary @- \
https://<domain>/some/deep/nested/directory/cog.tif \
http://localhost:3000/some/deep/nested/directory/cog.tif \
| cat
```

Expand All @@ -203,10 +203,10 @@ find \
--progress-bar \
--keepalive-time 1200 \
-X PUT \
-H "Authorization: Bearer <token>" \
-H "Authorization: Bearer f7efafd138da71cdcb0aa4767da9b6ee:1cc83802e7a4a831b17efa9ffecf4822" \
--create-dirs \
-T {} \
https://<domain>/some/nested/directory/{} \; \
http://localhost:3000/some/nested/directory/{} \; \
| cat
```

Expand All @@ -217,10 +217,10 @@ The equivalent to the `cURL` utility on Windows Platform is the `Invoke-RestMeth
cd /to/the/directory/with/your/file
$FILENAME = "some-file.tiff"
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Authorization", "Bearer <token>")
$headers.Add("Authorization", "Bearer f7efafd138da71cdcb0aa4767da9b6ee:1cc83802e7a4a831b17efa9ffecf4822")
Invoke-RestMethod `
-Uri "https://<domain>/some/nested/directory/$FILENAME" `
-Uri "http://localhost:3000/some/nested/directory/$FILENAME" `
-Method Put `
-InFile "./$FILENAME" `
-Headers $headers `
Expand Down
19 changes: 12 additions & 7 deletions chompfile.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,25 +5,30 @@ extensions = ['[email protected]:prettier']

[[task]]
name = 'start'
env = { TZ='UTC', NODE_ENV='development'}
deps = [ 'src/**/*.js' ]
env = { TZ = 'UTC', NODE_ENV = 'development' }
deps = ["test"]
run = """
node \
--trace-warnings \
bin/mnemosyne.js \
--key TjWnZr4u7w!z%C*F-JaNdRgUkXp2s5v8 \
--login local \
--login username \
--login username2 \
--volume ./test \
--volume ./test/mnt1 \
--volume ./test/mnt2
--volume ./test-mnts \
--volume ./test-mnts/mnt1 \
--volume ./test-mnts/mnt2
"""

[[task]]
name = 'test'
deps = ['src/**/*.js', "test/**/*.js"]
run = "mocha"

[[task]]
name = 'lint'
run = "eslint ."


[[task]]
name = 'prettier'
template = 'prettier'
Expand All @@ -44,4 +49,4 @@ find . \
-name package-lock.json \
-type f \
-delete
"""
"""
Loading

0 comments on commit 5906416

Please sign in to comment.