Panic when updating an aws.ssm.Parameter value #2017

cnunciato · 2022-06-24T14:45:25Z

Given the following program:

import * as aws from "@pulumi/aws";

const someParam = new aws.ssm.Parameter("some-param", {
    name: "/cnunciato/shared-config-ts/dev/someParam",
    value: "some-value",
    type: "String",
});

... the initial update succeeds, but subsequent updates that change the parameter's value property fail:

$ pulumi up
...

Updating (dev)

View Live: https://app.pulumi.com/cnunciato/shared-config-ts/dev/updates/3

     Type                  Name                  Status                  Info
     pulumi:pulumi:Stack   shared-config-ts-dev  **failed**              1 error; 28 messages
 ~   └─ aws:ssm:Parameter  some-param            **updating failed**     [diff: ~value]; 1 error
 
Diagnostics:
  aws:ssm:Parameter (some-param):
    error: error reading from server: EOF
 
  pulumi:pulumi:Stack (shared-config-ts-dev):
    panic: interface conversion: interface {} is nil, not map[string]interface {}
    goroutine 64 [running]:
    github.com/hashicorp/go-cty/cty.Value.GetAttr({{{0xb8be390?, 0xc0005d0930?}}, {0x0?, 0x0?}}, {0xaab1393, 0x4})
        /home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/cty/value_ops.go:711 +0x2ed
    github.com/hashicorp/terraform-provider-aws/internal/service/ssm.resourceParameterUpdate(0xc000d3b500, {0x95882a0?, 0xc00068e000})
        /home/runner/go/pkg/mod/github.com/pulumi/[email protected]/internal/service/ssm/parameter.go:282 +0x3e9
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0xb8bd5c8?, {0xb8bd5c8?, 0xc0000740b8?}, 0xd?, {0x95882a0?, 0xc00068e000?})
        /home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:719 +0x178
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc001673860, {0xb8bd5c8, 0xc0000740b8}, 0xc00172f5f0, 0xc000d3b080, {0x95882a0, 0xc00068e000})
        /home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:837 +0x605
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.v2Provider.Apply({0xc000d42000?}, {0xaafc3ae, 0x11}, {0xb8be438?, 0xc00188c920}, {0xb8c3ab0?, 0xc000d3b080})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfshim/sdk-v2/provider.go:112 +0x19b
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Update(0xc000246fc0, {0xb8bd638?, 0xc001e71e60?}, 0xc000d3a780)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/provider.go:1052 +0x86f
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Update_Handler.func1({0xb8bd638, 0xc001e71e60}, {0xa80a860?, 0xc000d3a780})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider.pb.go:2650 +0x7b
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0xb8bd638, 0xc001e71a70}, {0xa80a860, 0xc000d3a780}, 0xc0019c4940, 0xc000d36450)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/go/otgrpc/server.go:57 +0x3f9
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Update_Handler({0xa945000?, 0xc000246fc0}, {0xb8bd638, 0xc001e71a70}, 0xc001e365a0, 0xc0018f0f00)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider.pb.go:2652 +0x138
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc0006fce00, {0xb8c3ce8, 0xc001357a00}, 0xc001e786c0, 0xc001a2ae70, 0x124640d0, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1283 +0xcfd
    google.golang.org/grpc.(*Server).handleStream(0xc0006fce00, {0xb8c3ce8, 0xc001357a00}, 0xc001e786c0, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1620 +0xa1b
    google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:922 +0x98
    created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:920 +0x28a
 
    error: update failed
 
Resources:
    1 unchanged

Repros with other languages as well. (Initially noticed with YAML.) Does not repro in TF.

The text was updated successfully, but these errors were encountered:

stack72 · 2022-06-24T17:39:49Z

Hi @cnunciato

We are working on pulumi/pulumi-terraform-bridge#548 to fix this up!

This is also a case of needing work because in v4.20.0 of terraform-provider-aws, LaunchConfig changed to use the GetRawConfig which would have caused panics any time that resource was used

Paul

afa4 · 2022-07-08T19:15:49Z

Is there any workaround to bypass this problem, while fix is not deployed?

cnunciato · 2022-07-09T04:33:36Z

@afa4 Apologies for the delay here, the team's actively working to get this fixed. The current workaround is to downgrade to a previous version of the AWS provider. It looks like 5.8.0 introduced the issue, and I've confirmed that 5.7.2 works, so depending on which language you're using, you'll want to do something like this (e.g., for Node.js):

npm install @pulumi/[email protected]
pulumi up

Example run with repro and correction for reference:

➜  pulumi up
Previewing update (dev)

     Type                  Name     Plan       Info
     pulumi:pulumi:Stack   ssm-dev             
 ~   └─ aws:ssm:Parameter  param    update     [diff: ~value]
 
Resources:
    ~ 1 to update
    1 unchanged

Do you want to perform this update? yes
Updating (dev)

     Type                  Name     Status                  Info
     pulumi:pulumi:Stack   ssm-dev  **failed**              1 error; 28 messages
 ~   └─ aws:ssm:Parameter  param    **updating failed**     [diff: ~value]; 1 error
 
Diagnostics:
  pulumi:pulumi:Stack (ssm-dev):
    panic: interface conversion: interface {} is nil, not map[string]interface {}
    ...
    error: update failed
 
  aws:ssm:Parameter (param):
    error: error reading from server: EOF
 
Resources:
    1 unchanged

Duration: 2s

➜  npm install @pulumi/[email protected]

changed 1 package, and audited 117 packages in 2s

➜  pulumi up                    
Previewing update (dev)

     Type                  Name     Plan       Info
     pulumi:pulumi:Stack   ssm-dev             
 ~   └─ aws:ssm:Parameter  param    update     [diff: ~value]
 
Resources:
    ~ 1 to update
    1 unchanged

Do you want to perform this update? yes
Updating (dev)

     Type                  Name     Status      Info
     pulumi:pulumi:Stack   ssm-dev              
 ~   └─ aws:ssm:Parameter  param    updated     [diff: ~value]
 
Resources:
    ~ 1 updated
    1 unchanged

Duration: 4s

Fixes: #2015 Fixes: #2019 Fixes: #2017 Fixes: #2042 Please Note: this also temporarily stops the usage of d.GetRawConfig in the underlying TF provider as this is not yet supported in the TF bridge

AleksandrAksenov · 2022-07-12T22:09:56Z

Hi

For python downgrade from 5.9.2 to 5.7.2 doesn't work properly.

error: Program failed with an unhandled exception:
error: Traceback (most recent call last):
  File "/opt/homebrew/bin/pulumi-language-python-exec", line 107, in <module>
    loop.run_until_complete(coro)
  File "/opt/homebrew/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "./venv/lib/python3.9/site-packages/pulumi/runtime/stack.py", line 126, in run_in_stack
    await run_pulumi_func(lambda: Stack(func))
  File "./venv/lib/python3.9/site-packages/pulumi/runtime/stack.py", line 49, in run_pulumi_func
    func()
  File "./venv/lib/python3.9/site-packages/pulumi/runtime/stack.py", line 126, in <lambda>
    await run_pulumi_func(lambda: Stack(func))
  File "./venv/lib/python3.9/site-packages/pulumi/runtime/stack.py", line 149, in __init__
    func()
  File "/opt/homebrew/bin/pulumi-language-python-exec", line 106, in <lambda>
    coro = pulumi.runtime.run_in_stack(lambda: runpy.run_path(args.PROGRAM, run_name='__main__'))
  File "/opt/homebrew/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 305, in run_path
    return _run_code(code, mod_globals, init_globals,
  File "/opt/homebrew/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "./aws/app-api/./__main__.py", line 32, in <module>
    aws_caller_identity = aws.get_caller_identity()
  File "./venv/lib/python3.9/site-packages/pulumi_aws/get_caller_identity.py", line 103, in get_caller_identity
    __ret__ = pulumi.runtime.invoke('aws:index/getCallerIdentity:getCallerIdentity', __args__, opts=opts, typ=GetCallerIdentityResult).value
  File "./venv/lib/python3.9/site-packages/pulumi/runtime/invoke.py", line 166, in invoke
    raise invoke_error
Exception: invoke of aws:index/getCallerIdentity:getCallerIdentity failed: invocation of aws:index/getCallerIdentity:getCallerIdentity returned an error: unable to validate AWS AccessKeyID and/or SecretAccessKey - see https://pulumi.io/install/aws.html for details on configuration

error: an unhandled error occurred: Program exited with non-zero exit code: 1

AleksandrAksenov · 2022-07-12T22:16:42Z

For python version 5.10.0 works perfectly.

patwritescode · 2022-07-31T00:34:17Z

5.10.0 fixed this for me for node/typescript as well. much appreciated.

BernalCarlos · 2023-04-13T22:12:35Z

This is happening for me on version 5.35.0

josevromo · 2023-07-25T08:44:55Z

Happening with "@pulumi/aws": "^5.42.0"

epeters3 · 2024-01-12T22:50:38Z

Happening with @pulumi/[email protected].

epeters3 · 2024-01-15T19:16:36Z

I set overwrite: true [1] on the aws.ssm.Parameter definition and that solved the problem: I was able to update.

epeters3 · 2024-01-16T17:58:43Z

Scratch that. When I tried a second update with overwrite: true, I got the error again.

iwahbe · 2024-01-17T02:13:41Z

@epeters3 Can you post a program that replicates this panic on the most recent pulumi-aws version? I tried to repro, but I don't see a panic.

t0yv0 · 2024-03-05T18:00:45Z


t0yv0@Antons-MacBook-Pro> pulumi up --yes                                                                                                                                                                                                                                                                                                                                                ~/tmp/2024/03/aws-2017
Previewing update (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/anton-pulumi-corp/aws-2017/dev/previews/316fd999-a2be-4c0c-b135-db2461291d9a

     Type                 Name          Plan     
     pulumi:pulumi:Stack  aws-2017-dev           

Resources:
    2 unchanged

Updating (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/anton-pulumi-corp/aws-2017/dev/updates/2

     Type                 Name          Status     
     pulumi:pulumi:Stack  aws-2017-dev             

Resources:
    2 unchanged

Duration: 4s

CLI          
Version      3.99.0
Go Version   go1.21.5
Go Compiler  gc

Plugins
NAME    VERSION
aws     6.24.2
awsx    2.5.0
docker  4.5.1
docker  3.6.1
nodejs  unknown

Host     
OS       darwin
Version  14.2.1
Arch     x86_64

This project is written in nodejs: executable='/Users/t0yv0/bin/node' version='v18.18.2'

Current Stack: anton-pulumi-corp/aws-2017/dev

TYPE                         URN
pulumi:pulumi:Stack          urn:pulumi:dev::aws-2017::pulumi:pulumi:Stack::aws-2017-dev
pulumi:providers:aws         urn:pulumi:dev::aws-2017::pulumi:providers:aws::default_6_24_2
aws:ssm/parameter:Parameter  urn:pulumi:dev::aws-2017::aws:ssm/parameter:Parameter::some-param


Found no pending operations associated with dev

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/anton-pulumi-corp
User           anton-pulumi-corp
Organizations  anton-pulumi-corp, moolumi, pulumi
Token type     personal

Dependencies:
NAME            VERSION
@pulumi/aws     6.24.2
@pulumi/awsx    2.5.0
@pulumi/pulumi  3.108.1
@types/node     18.19.21

Pulumi locates its logs in /var/folders/gk/cchgxh512m72f_dmkcc3d09h0000gp/T/com.apple.shortcuts.mac-helper// by default

t0yv0 · 2024-03-05T18:01:35Z

Folks I think we should close this one because it works as stated, but let's open new issues with repro programs if anything is still not working as expected. Thanks a lot!

stack72 removed the needs-triage Needs attention from the triage team label Jun 24, 2022

stack72 self-assigned this Jun 24, 2022

stack72 added this to the 0.74 milestone Jun 24, 2022

stack72 mentioned this issue Jun 24, 2022

Ensure bridge is aware of GetRawConfig pulumi/pulumi-terraform-bridge#548

Closed

lukehoban mentioned this issue Jun 24, 2022

panic: interface conversion: interface {} is nil, not map[string]interface {} while updating AWS SSM parameter pulumi/pulumi#9907

Closed

stack72 modified the milestones: 0.74, 0.75 Jun 27, 2022

lukehoban added the p1 A bug severe enough to be the next item assigned to an engineer label Jun 30, 2022

stack72 mentioned this issue Jul 10, 2022

Upgrade to v4.22.0 of the AWS Terraform provider #2045

Merged

stack72 closed this as completed in #2045 Jul 11, 2022

pulumi-bot added the resolution/fixed This issue was fixed label Jul 11, 2022

cnunciato reopened this Jan 13, 2024

cnunciato unassigned stack72 Jan 13, 2024

cnunciato removed p1 A bug severe enough to be the next item assigned to an engineer resolution/fixed This issue was fixed labels Jan 13, 2024

cnunciato added the needs-triage Needs attention from the triage team label Jan 13, 2024

iwahbe added needs-repro Needs repro steps before it can be triaged or fixed and removed needs-triage Needs attention from the triage team labels Jan 17, 2024

t0yv0 added the resolution/no-repro This issue wasn't able to be reproduced label Mar 5, 2024

t0yv0 closed this as completed Mar 5, 2024

pulumi-bot assigned stack72 Mar 5, 2024

pulumi-bot added the resolution/fixed This issue was fixed label Mar 5, 2024

t0yv0 modified the milestones: 0.75, 0.101 Mar 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Panic when updating an aws.ssm.Parameter value #2017

Panic when updating an aws.ssm.Parameter value #2017

cnunciato commented Jun 24, 2022 •

edited

Loading

stack72 commented Jun 24, 2022

afa4 commented Jul 8, 2022 •

edited

Loading

cnunciato commented Jul 9, 2022 •

edited

Loading

AleksandrAksenov commented Jul 12, 2022

AleksandrAksenov commented Jul 12, 2022

patwritescode commented Jul 31, 2022

BernalCarlos commented Apr 13, 2023

josevromo commented Jul 25, 2023

epeters3 commented Jan 12, 2024

epeters3 commented Jan 15, 2024

epeters3 commented Jan 16, 2024

iwahbe commented Jan 17, 2024

t0yv0 commented Mar 5, 2024

t0yv0 commented Mar 5, 2024

Panic when updating an aws.ssm.Parameter value #2017

Panic when updating an aws.ssm.Parameter value #2017

Comments

cnunciato commented Jun 24, 2022 • edited Loading

stack72 commented Jun 24, 2022

afa4 commented Jul 8, 2022 • edited Loading

cnunciato commented Jul 9, 2022 • edited Loading

AleksandrAksenov commented Jul 12, 2022

AleksandrAksenov commented Jul 12, 2022

patwritescode commented Jul 31, 2022

BernalCarlos commented Apr 13, 2023

josevromo commented Jul 25, 2023

epeters3 commented Jan 12, 2024

epeters3 commented Jan 15, 2024

epeters3 commented Jan 16, 2024

iwahbe commented Jan 17, 2024

t0yv0 commented Mar 5, 2024

t0yv0 commented Mar 5, 2024

cnunciato commented Jun 24, 2022 •

edited

Loading

afa4 commented Jul 8, 2022 •

edited

Loading

cnunciato commented Jul 9, 2022 •

edited

Loading