Amplify build issue when deploying TEAM in delegated admin account

[tawoyinfa]

Describe the bug
There is an ongoing issue when deploying TEAM in a delegated admin account. The issue is caused by a failure in creating a cloudtrail lake event datastore as part of the initial build even though the amplify role has the right permissions.

The build error is shown below:

2024-04-30T00:47:12.142Z [INFO]: CREATE_IN_PROGRESS myEventDataStore AWS::CloudTrail::EventDataStore Tue Apr 30 2024 00:47:08 GMT+0000 (Coordinated Universal Time) CREATE_FAILED myEventDataStore AWS::CloudTrail::EventDataStore Tue Apr 30 2024 00:47:09 GMT+0000 (Coordinated Universal Time) Resource handler returned message: "Invalid request provided: User: arn:aws:sts::xxxxxxxx:assumed-role/TEAM-IDC-APP-AmplifyRole-8ABQF4FsKQN2/BuildSession is not authorized to access this resource (Service: CloudTrail, Status Code: 400, Request ID: ec49a0a9-56ab-4eb5-b549-b53a90f29419)" (RequestToken: 7ee30e5b-6baa-5ca2-cc38-1a8dfb1a2733, HandlerErrorCode: InvalidRequest) CREATE_FAILED amplify-teamidcapp-main-04403-customcloudtrailLake-1M9T6PQ9GXKSF AWS::CloudFormation::Stack Tue Apr 30 2024 00:47:09 GMT+0000 (Coordinated Universal Time) The following resource(s) failed to create: [myEventDataStore].

We are currently investigating this issue with the service team and would provide updates accordingly.

As a workaround, create the Cloudtrail lake event datastore in the delegated admin account outside of TEAM and pass the ARN to the parameter.sh file as described below:

[powerfei]

Is there update on this issue? What's the ETA?

[pwhite1989]

We are also facing this issue

[SachinCldCvr]

I am trying to deploy the above solution for the org having ~300 active users but not able to because of the above error.
when can we have the above bug fixed ?

Thanks

[julian-price]

I hit this issue and was able to use the workaround.
My manually created CloudTrail event data store used mostly default settings - i.e. management read and write events. I had to run ./destroy.sh, update the parameters.sh with the arn and then run ./deploy.sh again as the Amplify app was in a state where it could not be updated or redeployed.

[tawoyinfa]

Still working internally to resolve this issue. It looks like it is related to some recent changes in the cloudformation service and I am working with the team to get a resolution.

@robbycuenot see the workaround described above

[robbycuenot]

Thanks for linking me here @tawoyinfa , this workaround worked for me

[asamboninub8]

I hit this issue and was able to use the workaround. My manually created CloudTrail event data store used mostly default settings - i.e. management read and write events. I had to run ./destroy.sh, update the parameters.sh with the arn and then run ./deploy.sh again as the Amplify app was in a state where it could not be updated or redeployed.

Could you please provide more information about updating the parameters.sh with the arn?
Thank you!

[julian-price]

Could you please provide more information about updating the parameters.sh with the arn? Thank you!

Sure. In the parameters.sh file there is a parameter like this:

CLOUDTRAIL_AUDIT_LOGS=read_write

After you manually create your CloudTrail event data store, you need to grab its ARN from the console (https://us-east-1.console.aws.amazon.com/cloudtrailv2/home?region=us-east-1#/lake/eventDataStores and then click to open your event data store) and then use it as the value to the parameter instead of read_write (below, I've used a dummy value, based on the template TEAM_ACCOUNT and REGION parameters):

CLOUDTRAIL_AUDIT_LOGS="arn:aws:cloudtrail:us-east-1:123456789101:eventdatastore/gs3d5r3s-d113-v0f9-d8a0-wsi99aq20d"

[asamboninub8]

Could you please provide more information about updating the parameters.sh with the arn? Thank you!

Sure. In the parameters.sh file there is a parameter like this:

CLOUDTRAIL_AUDIT_LOGS=read_write

After you manually create your CloudTrail event data store, you need to grab its ARN from the console (https://us-east-1.console.aws.amazon.com/cloudtrailv2/home?region=us-east-1#/lake/eventDataStores and then click to open your event data store) and then use it as the value to the parameter instead of read_write (below, I've used a dummy value, based on the template TEAM_ACCOUNT and REGION parameters):

CLOUDTRAIL_AUDIT_LOGS="arn:aws:cloudtrail:us-east-1:123456789101:eventdatastore/gs3d5r3s-d113-v0f9-d8a0-wsi99aq20d"

Thank you so much! I'm going to give it a try.

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 7 days it will automatically be closed.

[wwebster-rr]

Is the plan to have this fixed so an upgrading from 1.1.1 to 1.1.3 wouldnt need these manual steps or whats the ideal scenario I guess. We are on 1.1.1 currently but if upgrading to 1.1.2 means losing all past audit logs in the app I would prefer to hold off.

[landsamboni]

Hi,

I would like some clarification regarding where to create the CloudTrail Lake organization event datastore.

In the documentation, it suggests that the event datastore should be created in the TEAM account, but another section mentions creating it in the delegated admin account outside of TEAM. Could you please confirm if the datastore should be created in the organization’s master account, or if it can be created in the TEAM account?

Additionally, I have already deployed the application and created the event datastore in the TEAM account, but I am not seeing any logs. If the correct approach is to create the datastore in the master account, do I need to delete everything and redeploy the entire application? Or is there a way to update the parameter.sh file with the new CloudTrail Lake organization event datastore ARN from the master account without redeploying the entire application?

Thank you for your assistance.

[jfevre]

atastore should be created in the TEAM account, but another section mentions creating it in the delegated admin account outside of TEAM. Could you please confirm if the datastore should be created in the organization’s master account, or if it can be created in the TEAM account?

I am also running into this issue would be good to get some more clarification

[landsamboni]

atastore should be created in the TEAM account, but another section mentions creating it in the delegated admin account outside of TEAM. Could you please confirm if the datastore should be created in the organization’s master account, or if it can be created in the TEAM account?

I am also running into this issue would be good to get some more clarification

It needs to be created in the Master. I tested it.

[jfevre]

atastore should be created in the TEAM account, but another section mentions creating it in the delegated admin account outside of TEAM. Could you please confirm if the datastore should be created in the organization’s master account, or if it can be created in the TEAM account?

I am also running into this issue would be good to get some more clarification

It needs to be created in the Master. I tested it.

did you have to tear it down and redeploy ? I've tried to just change it but am still not seeing activity logs

[landsamboni]

yes, complete redeploy

[sbrown-tecracer]

If your TEAM account is the Org delegated Administrator account for CloudTrail, then creating the Lake would suffice in this account would it not?
This is part of the init.sh anyway...unless of course you are deploying Team in the Management account itself.

Deploy to Management Account -> Lake needs to be here
Deploy TEAM to a delegated CloudTrail Account -> lake needs to be here

Could someone please clarify this?

[filipchelev-rldatix]

Solution and explanation:
You can use delegated admin account to create NEW Event data store.
During the creation of the new data store, please set "Include all accounts in my organization" to Yes. This will create the Event data store in the Management account and will receive the Management account arn (even though it is created and visible in the delegated admin account).
This is by AWS Event data store design (When a delegated administrator creates an organization event data store, the organization event data store exists in the management account for the organization. - https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-lake-organizations.html).

Trying just to enable "Include all accounts in my organization" on the existing Event data store, is not possible, because the existing Event data store is created in delegated admin account.