feat(s3): replicating objects

[badmintoncryer]

Issue # (if applicable)

Closes #1680.

Reason for this change

AWS S3 supports configuring object replication , but the s3.Bucket construct does not support it.

Description of changes

Added replicationRules to BucketProps.

Replication configuration version

There are two versions of replication configuration. This PR uses only the V2 replication configuration to enable the specification of the Filter element and S3 Replication Time Control (S3 RTC).

To use V2 replication configuration, this PR explicitly specifies Filter.Prefix property.

        const prefix = rule.prefixFilter ?? '';
        const filter = isAndFilter ? {
          and: {
            prefix,
            tagFilters: rule.tagFilter,
          },
        } : {
          prefix,
        };

V2 replication configuration has some restriction:

• Must specify DeleteMarkerReplication

ReplicationStack | 4/7 | 9:22:08 PM | CREATE_FAILED        | AWS::S3::Bucket  | SourceBucket (SourceBucketDDD2130A) Resource handler returned message:
Delete marker replication is not supported if any Tag filter is specified. Please refer to S3 Developer Guide for more information. (Service: S3, Status Code: 400, Request ID: XXX, Extended Request ID: XXX)

• Must specify Priority

ReplicationStack | 4/7 | 9:12:08 PM | CREATE_FAILED        | AWS::S3::Bucket  | SourceBucket (SourceBucketDDD2130A) Resource handler returned message:
Priority must be specified for this version of Cross Region Replication configuration schema. Please refer to S3 Developer Guide for more information. (Service: S3, Status Code: 400, Request ID: XXX, Extended Request ID: XXX)

These restriction is not documented but there are some posts about these points.

• https://repost.aws/questions/QUiEc8wFE_Q16fX5WG-YWnrA/cloudformation-support-for-s3-replication-to-multiple-destination-buckets

To resolve these problems,I made the priority required and explicitly set the deleteMarkerReplication.

       const prefix = rule.prefixFilter ?? ''; // set empty string to use V2 replication configuration
        const filter = isAndFilter ? {
          and: {
            prefix,
            tagFilters: rule.tagFilter,
          },
        } : {
          prefix,
        };

        return {
          id: rule.id,
          priority: rule.priority,
          status: 'Enabled',
          destination: {
            bucket: rule.destination.bucket.bucketArn,
            account: rule.destination.account,
            storageClass: rule.storageClass?.toString(),
            accessControlTranslation: rule.destination.accessControlTransition ? {
              owner: 'Destination',
            } : undefined,
            encryptionConfiguration: rule.kmsKey ? {
              replicaKmsKeyId: rule.kmsKey.keyArn,
            } : undefined,
            replicationTime: rule.replicationTimeControl !== undefined ? {
              status: rule.replicationTimeControl ? 'Enabled' : 'Disabled',
              time: {
                minutes: 15,
              },
            } : undefined,
            metrics: rule.replicationTimeControlMetrics !== undefined ? {
              status: rule.replicationTimeControlMetrics ? 'Enabled' : 'Disabled',
              eventThreshold: {
                minutes: 15,
              },
            } : undefined,
          },
          filter,
          // To avoid deploy error when there are multiple replication rules with undefined deleteMarkerReplication,
          // CDK explicitly set the deleteMarkerReplication if it is undefined.
          deleteMarkerReplication: {
            status: rule.deleteMarkerReplication ? 'Enabled' : 'Disabled',
          },
          sourceSelectionCriteria,
        };

IAM permission

There is a documentation to setup IAM permissions for service role.

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "s3:GetReplicationConfiguration",
            "s3:ListBucket"
         ],
         "Resource":[
            "arn:aws:s3:::SRC-BUCKET"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:GetObjectVersionForReplication",
            "s3:GetObjectVersionAcl",
            "s3:GetObjectVersionTagging"
         ],
         "Resource":[
            "arn:aws:s3:::SRC-BUCKET/*"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:ReplicateObject",
            "s3:ReplicateDelete",
            "s3:ReplicateTags"
         ],
         "Resource":"arn:aws:s3:::DST-BUCKET/*"
      }
   ]
}

However, there are discrepancies between the automatically generated IAM policies in the management console and the IAM policies in the documentation.

Generated Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetReplicationConfiguration",
                "s3:GetObjectVersionForReplication",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTagging",
                "s3:GetObjectRetention",
                "s3:GetObjectLegalHold"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::SRC-BUCKET",
                "arn:aws:s3:::SRC-BUCKET/*"
            ]
        },
        {
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags",
                "s3:GetObjectVersionTagging",
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Effect": "Allow",
            "Condition": {
                "StringLikeIfExists": {
                    "s3:x-amz-server-side-encryption": [
                        "aws:kms",
                        "aws:kms:dsse",
                        "AES256"
                    ]
                }
            },
            "Resource": [
                "arn:aws:s3:::DST-BUCKET/*"
            ]
        },
        {
            "Action": [
                "kms:Decrypt"
            ],
            "Effect": "Allow",
            "Condition": {
                "StringLike": {
                    "kms:ViaService": "s3.ap-northeast-1.amazonaws.com",
                    "kms:EncryptionContext:aws:s3:arn": [
                        "arn:aws:s3:::SRC-BUCKET/*"
                    ]
                }
            },
            "Resource": [
                "arn:aws:kms:ap-northeast-1:123456789012:key/hogehuga"
            ]
        },
        {
            "Action": [
                "kms:Encrypt"
            ],
            "Effect": "Allow",
            "Condition": {
                "StringLike": {
                    "kms:ViaService": [
                        "s3.ap-northeast-1.amazonaws.com"
                    ],
                    "kms:EncryptionContext:aws:s3:arn": [
                        "arn:aws:s3:::DST-BUCKET*"
                    ]
                }
            },
            "Resource": [
                "arn:aws:kms:ap-northeast-1:123456789012:key/hogefuga"
            ]
        }
    ]
}

I adopted the policy from the document. I look forward to hearing your thoughts on this matter.

Description of how you validated changes

Added both unit and integ tests.

Checklist

• My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[aws-cdk-automation]

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

[shikha372]

Hi @badmintoncryer , Thank you for submitting this PR, as this feature is adding some additional pemissions for s3, we'll need to have an internal security review while we go through this PR. Will keep you posted !!

[mergify]

update

✅ Branch has been successfully updated

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[badmintoncryer]

@mergify update

[mergify]

update

✅ Branch has been successfully updated

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[badmintoncryer]

@mergify update

[mergify]

update

✅ Branch has been successfully updated

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[badmintoncryer]

@mergify update

[mergify]

update

☑️ Nothing to do

• #commits-behind > 0 [📌 update requirement]
• queue-position = -1 [📌 update requirement]
• -closed [📌 update requirement]
• -conflict [📌 update requirement]

[mrgrain]

@Mergifyio dequeue

[mergify]

This pull request has been removed from the queue for the following reason: pull request dequeued.

Pull request #30966 has been dequeued by a dequeue command

You should look at the reason for the failure and decide if the pull request needs to be fixed or if you want to requeue it.

If you want to requeue this pull request, you need to post a comment with the text: @mergifyio requeue

[mergify]

dequeue

✅ The pull request has been removed from the queue default-squash

[mrgrain]

Dequeued because Codecov failed and this is blocking the queue.

[badmintoncryer]

@mrgrain Do you know why CodeQL and Codecov fail? I'm only merging the main branch, but these jobs sometimes succeed and sometimes fail.

[mrgrain]

@Mergifyio requeue

[mergify]

requeue

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

[mrgrain]

@badmintoncryer Just a intermittent issue. I'm re-queuing the PR now.

[aws-cdk-automation]

AWS CodeBuild CI Report

• CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
• Commit ID: f543e0f
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.