Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Migrate S3 Client from AWS SDK v2 (deprecated) to v3 #221

Merged
merged 60 commits into from
Jan 1, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
98cf010
migrate initialization and createBucket function
vahidalizad Aug 5, 2024
6f49645
install aws s3 v3 packages
vahidalizad Aug 5, 2024
889ca3e
add credentials to set-up tests
vahidalizad Aug 5, 2024
a9a0d46
migrate endpoint test to v3 pattern
vahidalizad Aug 5, 2024
ed0fd9e
migrate aws functions to v3
vahidalizad Aug 7, 2024
dfe0fb2
fix testing of presignedUrl
vahidalizad Aug 7, 2024
06f8483
fix mock s3 client for the create file tests
vahidalizad Aug 7, 2024
771b533
add some tests inline to debug
vahidalizad Aug 7, 2024
f493800
fix output of getFileData to be a full Buffer
vahidalizad Aug 7, 2024
e5dcf37
add fileStream tests
vahidalizad Aug 7, 2024
0897b90
change handleFileStream response handling
vahidalizad Aug 7, 2024
3ff1c77
setup fake adapter for testing to work offline as well
vahidalizad Aug 7, 2024
92d4c10
Update index.js
mtrezza Aug 7, 2024
bf36a95
remove dotenv package
vahidalizad Aug 7, 2024
da2c3e1
reduce the wating time on resolving the function getSignedUrl
vahidalizad Aug 7, 2024
2eef4a1
remove unnecessary initialization comments
vahidalizad Aug 7, 2024
05b2035
fix handleFileStream error handling
vahidalizad Aug 10, 2024
0d3e766
fix test getFileStream tests to run offline
vahidalizad Aug 10, 2024
d4062cb
remove deasync and use async getFileLocation
vahidalizad Aug 10, 2024
a69cb5b
sync dependencies with parse-server and add prettier config
vahidalizad Oct 10, 2024
a5928f3
remove unused dependencies
vahidalizad Oct 10, 2024
5ab5614
add integration tests
vahidalizad Oct 10, 2024
bc08dfb
prettier format all files
vahidalizad Oct 10, 2024
1704457
update packages
vahidalizad Oct 10, 2024
1f1257d
fix coverage directory
vahidalizad Oct 10, 2024
d738c85
add ci command
vahidalizad Oct 10, 2024
e50978d
add integration test command
vahidalizad Oct 10, 2024
3fbe88a
revert @semantic dependencies
vahidalizad Oct 10, 2024
a71372a
Merge branch 'master' into aws-sdk-v3-async
vahidalizad Oct 22, 2024
82bef30
fix merge conflicts of prettier and file names
vahidalizad Oct 22, 2024
b3c2dad
fix lint
vahidalizad Oct 22, 2024
3024336
Update README.md
mtrezza Oct 22, 2024
917f3c4
remove parse server 6 compat
mtrezza Oct 22, 2024
b52723b
run integration tests only for parse server >=7
mtrezza Oct 22, 2024
5d39edd
refactor mock
mtrezza Oct 22, 2024
bf7bfc2
fix mock
mtrezza Oct 22, 2024
daee5e6
remove Parse import
mtrezza Oct 22, 2024
1d5964f
add support for other endpoints
vahidalizad Dec 9, 2024
22071e9
test: add command instance check for s3 send calls
vahidalizad Dec 9, 2024
fe83a76
add test for credentials options
vahidalizad Dec 10, 2024
f4de1fb
add the test for handling stream errors
vahidalizad Dec 10, 2024
ed79c7c
Update index.js
mtrezza Dec 11, 2024
4089ed2
Update index.js
mtrezza Dec 11, 2024
b80db7d
Update spec/test.spec.js
mtrezza Dec 11, 2024
7161e70
Update index.js
mtrezza Dec 11, 2024
8a1163a
Update index.js
mtrezza Dec 11, 2024
fa13535
Update index.js
mtrezza Dec 11, 2024
81f4486
Update spec/test.spec.js
mtrezza Dec 11, 2024
843fef4
Update spec/test.spec.js
mtrezza Dec 11, 2024
1910b59
Update spec/test.spec.js
mtrezza Dec 11, 2024
f4fcda6
hotfix error interpreting index.js
vahidalizad Dec 11, 2024
4a58f6b
remove some mock s3 clients
vahidalizad Dec 11, 2024
74232a4
fix async expectations in unit tests
vahidalizad Dec 11, 2024
f264067
add badges to README
mtrezza Dec 15, 2024
5871b42
Update README.md
mtrezza Dec 15, 2024
77244c6
fix issue #230
vahidalizad Dec 24, 2024
498b043
fix unit tests of bucket operations
vahidalizad Dec 24, 2024
6503652
add credentials tests
vahidalizad Dec 24, 2024
bb83ccb
update readme to add breaking changes
vahidalizad Dec 24, 2024
becb402
fix readme links
vahidalizad Dec 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .eslintrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,4 @@
"globals": {
"Parse": true
}
}
}
3 changes: 0 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,6 @@ jobs:
- name: Parse Server 7, Node.js 22
NODE_VERSION: 22.4.1
PARSE_SERVER_VERSION: 7
- name: Parse Server 6
NODE_VERSION: 18
PARSE_SERVER_VERSION: 6
fail-fast: false
name: ${{ matrix.name }}
timeout-minutes: 15
Expand Down
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,4 +177,4 @@ ___



\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)*
\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)*
109 changes: 96 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@
[![Coverage](https://img.shields.io/codecov/c/github/parse-community/parse-server-s3-adapter/master.svg)](https://codecov.io/github/parse-community/parse-server-s3-adapter?branch=master)
[![auto-release](https://img.shields.io/badge/%F0%9F%9A%80-auto--release-9e34eb.svg)](https://github.com/parse-community/parse-server-s3-adapter/releases)

[![Parse Server](https://img.shields.io/badge/Parse_Server-7.0-169CEE.svg?style=flat&logo=data:image/svg%2bxml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz48c3ZnIGlkPSJ1dWlkLTg2MzQ2MDY1LTNjNjQtNDBlYy1hNmQ0LWUyNzZmM2E0Y2U5MiIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2aWV3Qm94PSIwIDAgMTExMy42NiAxMDk5LjQyIj48ZyBpZD0idXVpZC04MWRmNWUyZC04YWQyLTQwMmEtYTNlZS1hYjE2YTQ5NjhhNjciPjxwYXRoIGQ9Ik00ODUuMDMsNzg1LjE0Yy04MC4zMSwwLTE2MC42MS0uMDktMjQwLjkyLjA3LTE5LjY5LjA0LTM4Ljk2LDIuODUtNTYuODksMTEuODYtMzAuNjEsMTUuMzktNDYuMjQsNDAuODYtNTAuNTEsNzQuMTctMS43OCwxMy44Ny0xLjA3LDI3LjUzLDIuNCw0MS4wNyw5Ljg4LDM4LjYxLDQwLjk3LDYzLDgwLjc3LDYzLjYxLDE0LjQ4LjIyLDI4LjYtMS4xMyw0Mi4xOC02LjU3LDIyLjIxLTguODksMzcuNjgtMjQuNjksNDcuNTUtNDYuMjIsNS43LTEyLjQ0LDguNzgtMjUuNiwxMC4wMy0zOS4yMS43LTcuNjUsMS4zNC04LjM5LDkuMDctOC40LDM5LjExLS4wNiw3OC4yMi0uMDYsMTE3LjMzLDAsNy44MS4wMSw4LjcuNzcsOC4yNSw4LjYxLTEuNSwyNS45LTYuMjYsNTEuMTktMTQuOTUsNzUuNjgtOS44OSwyNy44OC0yNC41Miw1Mi45Ni00NC44OCw3NC40OS0xMi4wNiwxMi43NS0yNS44NiwyMy41LTQxLDMyLjM2LTI3LjYxLDE2LjE3LTU3LjU2LDI1LjU0LTg5LjIxLDI5LjYzLTE2LjAzLDIuMDctMzIuMTMsMy41Ni00OC4zMiwyLjk5LTUxLjA1LTEuODEtOTguMTktMTUuMzItMTM4LjkyLTQ3LjM1LTI5LjE4LTIyLjk0LTUwLjIzLTUxLjkxLTYzLjE2LTg2LjczQzQuNDksOTQwLjAzLjIsOTE0LjAyLDAsODg3LjEyYy0uMi0yNy4zOSwzLjIzLTU0LjA2LDEyLjA0LTgwLjAxLDE2LjE1LTQ3LjU1LDQ2LjA0LTg0LjIyLDg4LjM3LTExMC44NSwzMy41LTIxLjA4LDcwLjM1LTMyLjQxLDEwOS41MS0zNi43NiwxOC45My0yLjEsMzcuOTEtMi43OCw1Ni45NS0yLjc4LDE0Ni4wMS4wNiwyOTIuMDItLjE0LDQzOC4wMy4xNCw0MC43OC4wOCw3OS44OC03LjIsMTE3LjEzLTIzLjY0LDUxLjQ0LTIyLjcsOTEuNi01OC4yNSwxMTkuNzUtMTA3LjA4LDE5LjE3LTMzLjI3LDI5Ljk3LTY5LjE0LDMzLjU2LTEwNy4zNSw0LjI0LTQ1LjEyLS42My04OS4xNy0xNi44LTEzMS40Ni0yOS4xNS03Ni4xOS04My4xMS0xMjUuOTUtMTYxLjc0LTE0OC41NS0zMC42OC04LjgxLTYyLjExLTExLjExLTkzLjc0LTkuMDMtNTAuMzEsMy4zMS05Ni41MiwxOC45LTEzNy4wOCw0OS40MS0yNi45OCwyMC4zLTQ4Ljg5LDQ1LjI3LTY1LjkxLDc0LjQ3LTIzLjY0LDQwLjU2LTM2LjIsODQuNTgtNDEuMzYsMTMxLTIuMDUsMTguNDItMi45OSwzNi44NS0yLjkzLDU1LjM4LjEzLDM4LjA3LjA0LDc2LjEzLjA0LDExNC4yLDAsMi4zNS4xLDQuNy0uMDgsNy4wNC0uMzYsNC44Ny0xLjIzLDUuNjktNi4yMiw2LjA4LTEuODIuMTQtMy42NS4wNy01LjQ3LjA3LTM3LjU1LDAtNzUuMDksMC0xMTIuNjQsMC0xLjU2LDAtMy4xMy4wNS00LjY5LS4wNC01Ljk2LS4zMi02Ljc1LTEuMDgtNy4xMS02LjgyLS4wNi0xLjA0LS4wNC0yLjA5LS4wNC0zLjEzLjAyLTQ1LjYzLS44NC05MS4yOC4yOC0xMzYuODgsMS44MS03My44NSwxNi43My0xNDQuODQsNTAuNTQtMjExLjE0LDIxLjE3LTQxLjUxLDQ4LjY0LTc4LjQsODMuMi0xMDkuNzEsNDEuMzktMzcuNDksODguOTYtNjQuMjcsMTQyLjM5LTgwLjcxLDMwLjU1LTkuNCw2MS43NC0xNS4zNSw5My41My0xNy42NSw4MC4yMS01Ljc5LDE1Ny4wNSw2Ljg1LDIyOC42LDQ0Ljg3LDYzLjExLDMzLjU0LDExMi4wMSw4MS44OCwxNDYuNTUsMTQ0LjU1LDI0LjczLDQ0Ljg3LDM5LjE3LDkyLjk2LDQ1LjU3LDE0My43MSwzLjcxLDI5LjM4LDQuMjIsNTguODcsMi4yOSw4OC4yMS00LjU0LDY5LjI2LTI1LjQxLDEzMy4zOS02NC41LDE5MS4xMS0zNS41MSw1Mi40Mi04MS43Miw5Mi44OC0xMzcuNjgsMTIyLjM4LTQ1LjQ5LDIzLjk4LTkzLjg4LDM4LjY1LTE0NC43NSw0NS4zNC0xOS4zOCwyLjU1LTM4Ljg3LDMuNzQtNTguNDYsMy43LTc0LjA1LS4xNi0xNDguMS0uMDYtMjIyLjE0LS4wNloiIHN0eWxlPSJmaWxsOiNlMGUwZTA7Ii8+PC9nPjwvc3ZnPg==)](https://github.com/parse-community/parse-server/releases)
[![Node Version](https://img.shields.io/badge/nodejs-18,_20,_22-green.svg?logo=node.js&style=flat)](https://nodejs.org)

[![npm latest version](https://img.shields.io/npm/v/@parse/s3-files-adapter.svg)](https://www.npmjs.com/package/@parse/s3-files-adapter)

---
Expand All @@ -28,6 +31,9 @@ The official AWS S3 file storage adapter for Parse Server. See [Parse Server S3
- [Adding Metadata and Tags](#adding-metadata-and-tags)
- [Compatibility with other Storage Providers](#compatibility-with-other-storage-providers)
- [Digital Ocean Spaces](#digital-ocean-spaces)
- [Breaking Changes From v2 to v3](#breaking-changes-from-v2-to-v3)
- [Best Practices](#best-practices)
- [Why the Change](#why-the-change)


# Getting Started
Expand All @@ -40,21 +46,24 @@ The official AWS S3 file storage adapter for Parse Server. See [Parse Server S3

### Parse Server

| Version | End-of-Life | Compatible |
|---------|---------------|------------|
| <=5 | December 2023 | ✅ Yes |
| 6 | December 2024 | ✅ Yes |
| 7 | December 2025 | ✅ Yes |
Parse Server S3 Adapter is compatible with the following versions of Parse Server.

| Parse Server Version | End-of-Life | Compatible |
|----------------------|---------------|------------|
| <=5 | December 2023 | ❌ No |
| 6 | December 2024 | ❌ No |
| <7.3.0 | December 2025 | ❌ No |
| >=7.3.0 | December 2025 | ✅ Yes |

### Node.js

This product is continuously tested with the most recent releases of Node.js to ensure compatibility. We follow the [Node.js Long Term Support plan](https://github.com/nodejs/Release) and only test against versions that are officially supported and have not reached their end-of-life date.
Parse Server S3 Adapter is continuously tested with the most recent releases of Node.js to ensure compatibility. We follow the [Node.js Long Term Support plan](https://github.com/nodejs/Release) and only test against versions that are officially supported and have not reached their end-of-life date.

| Version | Latest Version | End-of-Life | Compatible |
|------------|----------------|-------------|------------|
| Node.js 18 | 18.20.4 | April 2025 | ✅ Yes |
| Node.js 20 | 20.15.1 | April 2026 | ✅ Yes |
| Node.js 22 | 22.4.1 | April 2027 | ✅ Yes |
| Node.js Version | End-of-Life | Compatible |
|-----------------|-------------|------------|
| 18 | April 2025 | ✅ Yes |
| 20 | April 2026 | ✅ Yes |
| 22 | April 2027 | ✅ Yes |

## AWS Credentials

Expand Down Expand Up @@ -277,7 +286,6 @@ var S3Adapter = require("@parse/s3-files-adapter");
var AWS = require("aws-sdk");

//Configure Digital Ocean Spaces EndPoint
const spacesEndpoint = new AWS.Endpoint(process.env.SPACES_ENDPOINT);
var s3Options = {
bucket: process.env.SPACES_BUCKET_NAME,
baseUrl: process.env.SPACES_BASE_URL,
Expand All @@ -290,7 +298,7 @@ var s3Options = {
s3overrides: {
accessKeyId: process.env.SPACES_ACCESS_KEY,
secretAccessKey: process.env.SPACES_SECRET_KEY,
endpoint: spacesEndpoint
endpoint: process.env.SPACES_ENDPOINT
}
};

Expand All @@ -307,3 +315,78 @@ var api = new ParseServer({
filesAdapter: s3Adapter
});
```


# Breaking Changes from v2 to v3

1. **Old Method (No Longer Supported)**:
```javascript
const options = {
bucket: 'bucket-1',
s3overrides: {
accessKeyId: 'access-key',
secretAccessKey: 'secret-key'
}
};
```

2. **New Methods (Required)**:
Credentials must now be passed either:

- **Inside the `s3overrides.credentials` key**:
```javascript
const options = {
bucket: 'bucket-1',
s3overrides: {
credentials: {
accessKeyId: 'access-key',
secretAccessKey: 'secret-key'
}
}
};
```

- **Directly in the root object**:
```javascript
const options = {
bucket: 'bucket-1',
credentials: {
accessKeyId: 'access-key',
secretAccessKey: 'secret-key'
}
};
```

## Best Practices

1. **Use Environment Variables for Credentials**:
Storing credentials directly in code can be insecure. Instead, use environment variables with the AWS SDK's built-in support for credential resolution:
```javascript
const options = {
bucket: 'bucket-1',
s3overrides: {
// The SDK will automatically load credentials from the environment
}
};
```

2. **Prefer AWS IAM Roles (for EC2, ECS, or Lambda)**:
If running in AWS-managed environments, use IAM roles to automatically provide temporary credentials:
- No need to pass `credentials` manually; the SDK resolves them automatically.

3. **Use the AWS Credential Provider Chain**:
Leverage the SDK's support for multiple credential sources:
```javascript
import { fromIni } from '`aws-sdk/credential-providers';

const options = {
bucket: 'bucket-1',
s3overrides: {
credentials: fromIni({ profile: 'your-profile-name' })
}
};
```

## Why the Change

The updated approach adheres to AWS SDK v3's modular and secure design, improving compatibility with advanced credential management techniques and security best practices.
Loading
Loading