Reconsider the default aws:skipMetadataCheck setting

[jaxxstorm]

In #1288 we disabled the metadata check in order to speed Pulumi up.

Unfortunately, this breaks Pulumi when running with an EC2 instance role, meaning lots of scenarios like self hosted CI/CD runners will not work.

It's possible of course to fix this by setting the environment variable AWS_EC2_METADATA_DISABLED=false or pulumi config set aws:skipMetadataApiCheck false, but the it's extremely difficult to figure that out unless you inherently know about this setting. This default setting seems to be violating the principal of least surprise and is causing a headache for those users who are setting Pulumi up in CI/CD.

To rectify this, we need to either:

1. Provide concrete information back to the user about how to fix this. This might involve intercepting the AWS SDK calls and injecting information about why they're getting this information
2. Consider setting the default value back to false

it might be possible to intercept the error message, which currently looks a bit like this:

Error: NoCredentialProviders: no valid providers in chain
    caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
    SharedCredsLoad: failed to load profile, .
    EC2RoleRequestError: no EC2 instance role found
    caused by: RequestCanceled: EC2 IMDS access disabled via AWS_EC2_METADATA_DISABLED env var

But this comes from the AWS SDK, so we'd need to trap the error

[SharpEdgeMarshall]

ATM is not possible to override through ENV VAR, for some reason setting AWS_EC2_METADATA_DISABLED=false doesn't work.

We had to change all our Providers and add aws:skipMetadataApiCheck false

#1636

[seunggs]

Hi @SharpEdgeMarshall - I'm running into this issue as I'm trying to implement Pulumi Automation API. Is the only solution then to create an explicit AWS provider with { skipMetadataApiCheck: false } in the CustomResourcesOptions? Or is there a simpler solution?

[SharpEdgeMarshall]

@seunggs there are no simpler solutions that i know atm sorry.

[seunggs]

No worries - I just misunderstood the solution. I didn't know you meant adding skipMetadataApiCheck: false as Pulumi config. That's a simple solution! :) Thanks for sharing this.

[SharpEdgeMarshall]

Pay attention that if you put it in the config it works only for the default provider.

[seunggs]

Oh didn't know that - thanks for pointing that out.

[blarghmatey]

This is still a problem. It would be great if we can get the default changed back, or at least fix the fact that the environment variable isn't being picked up properly by the TF provider at the bottom of the chain.

[ringods]

This will at least make it more visible from the AWS Classic registry page: pulumi/registry#1313

[jf]

for the record, I also got hit with needing to set aws:skipCredentialsValidation to true in order for my instance profile to work. Doh...

[robotrapta]

AFAICT this problem manifests with the unhelpful error message

    Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or "AWS_EC2_METADATA_DISABLED" environment variable

Dropping this here to help lost people like previous-me looking for an answer. The answer is:

pulumi config set aws:skipMetadataApiCheck false

[christrt9]

After some testing this is happening only for stack 'dev'

I get this error

    error: rpc error: code = Unknown desc = unable to validate AWS credentials.
    Details: no valid credential sources for Pulumi AWS Classic found.
AWS Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, expect HTTP transport, got <nil>

After i add this
aws:skipMetadataApiCheck: "false"

still the same error

    error: rpc error: code = Unknown desc = unable to validate AWS credentials.
    Details: no valid credential sources for Pulumi AWS Classic found.
    AWS Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, expect HTTP transport, got <nil>

I try this

aws:skipCredentialsValidation: "true" aws:skipMetadataApiCheck: "false"

but i get this error

  aws:iam:Role (AWSLambdaExecutionRole):
    error: 1 error occurred:
    	* configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.
    
    Please see https://registry.terraform.io/providers/hashicorp/aws
    for more information about providing credentials.
    
    AWS Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, expect HTTP transport, got <nil>

[Masonjrc]

I'm also having this issue, trying to use Github Actions to deploy a pulumi stack.

I've committed my Pulumi.dev.yaml file which contains:

secretsprovider: awskms://alias/dev-pulumi-key
encryptedkey: <MY ENCRYPTED KEY>
config:
  aws:region: ap-southeast-2

then in my github actions workflow I'm using these steps

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v3
      with:
        aws-region: ap-southeast-2
        role-to-assume: ${{ secrets.GITHUB_AWS_ROLE }}
        role-duration-seconds: 900

    - name: Preview Pulumi Changes
      uses: pulumi/actions@v4
      with:
        command: preview
        stack-name: dev
        cloud-url: s3://dev-pulumi-state-backend
        work-dir: ./infra
        secrets-provider: awskms://alias/dev-pulumi-key

I've confirmed that my AWS role and credentials are working as I'm able to list the kms keys and s3 bucket.
But playing around with the config, using the aws:skipMetadataApiCheck: false, and aws:skipCredentialsValidation: true doesn't seem to do anything.
I keep seeing this error:

Error: Command failed with exit code 255: pulumi preview --exec-agent pulumi/actions@v3 --color auto --exec-kind auto.local --event-log /tmp/automation-logs-preview-bnZW35/eventlog.txt --stack dev --non-interactive
  error: constructing secrets manager of type "cloud": secrets (code=Unknown): operation error KMS: Decrypt, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, failed to get API token, cannot get API token, operation error ec2imds: getToken, http response error StatusCode: 400, request to EC2 IMDS failed

Any ideas where to go from here?

[christrt9]

@Masonjrc Can you create another stack and test it again ? I was having the same issue only with the dev stack.

[Masonjrc]

@r00t9 Wow that worked, thanks so much!
Nearly fell off my chair when I saw the green tick... I've been stuck on this for hours.

I guess my pulumi state files must somehow be misconfigured in s3? I might try and purge my dev stack and start again. Were you able to get your dev going again?

[Masonjrc]

Purging my dev stack and restarting worked...

So just for anyone else who comes across this, my issue was that when I created the stack I followed the callout instruction given here and gave the secrets provider with a profile specified.

pulumi stack init dev --secrets-provider="awskms://alias/my-secret-key?region=ap-southeast-2&awssdk=v2&profile=dev"

Removing the profile argument in the query string fixed it, you can check your <stack>.json file in your storage backend to see what you've set up. Hope this helps someone

[PierreQcMontreal]

Wanna enforce this, its still an issue even with aws:skipCredentialsValidation: "true" aws:skipMetadataApiCheck: "false"