(aws-lambda-python-alpha): The command "export" does not exist - Poetry

[sltmyr]

Describe the bug

Our python poetry lambda deployments started failing with the error message The command "export" does not exist. I guess this is due to the recent poetry 2.0 release (https://github.com/python-poetry/poetry/releases/tag/2.0.0) that removes the export command and relies on the poetry-plugin-export plugin instead. The issue can be solved by using a custom build image that installs the plugin, this is the Dockerfile that worked for me: https://github.com/aws/aws-cdk/pull/32761/files.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

Lambda bundling succeeds.

Current Behavior

Bundling fails with The command "export" does not exist

Reproduction Steps

new lambda.PythonLayerVersion(this, "TestLayer", {
  entry: path.join(__dirname, "../layer"),
  description: "Test Lambda Layer with Poetry dependencies",
  compatibleRuntimes: [cdk.aws_lambda.Runtime.PYTHON_3_11],
});

Possible Solution

Install plugin in build image, see https://github.com/aws/aws-cdk/pull/32761/files

Additional Information/Context

No response

CDK CLI Version

2.142.1

Framework Version

No response

Node.js Version

18.0.0

OS

MacOS

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

Hi

I didn't get it. Where is the export command in your provided sample? Is it in the Dockerfile? If yes, can you share a minimal Dockerfile sample?

new lambda.PythonLayerVersion(this, "TestLayer", {
  entry: path.join(__dirname, "../layer"),
  description: "Test Lambda Layer with Poetry dependencies",
  compatibleRuntimes: [cdk.aws_lambda.Runtime.PYTHON_3_11],
});

And this sounds like an issue from poetry 2.0 and requires a plugin from poetry to work it around?

[igorsimko]

Hey @pahud, please check which version of aws-cdk you're using (not cli, but aws-cdk dependency in your project).

There was fix for installing correct version of poetry during lambda build in
95f8cef#diff-72edaa63ace26f9b6ba27bf4e9420a69f29a4fbe9e574e0c5a1ca269e5b5691a so you need at least version v2.93.0.

In my case I was using older version of aws-cdk where poetry version wasn't locked and with recent release of poetry (v2.0.0) bundling of lambda broke because it installed poetry version 2 where poetry export was dropped.

[sr-rossanderson]

I'm seeing similar issues in a couple of my AWS codepipelines now. I've not been able to find any way to pin the poetry version being used by the cdk synth command.

Also see this thread:

https://stackoverflow.com/questions/79335563/new-poetry-version-causing-issues-when-running-cdk-diff-due-to-the-aws-image/79338983#79338983

[pahud]

Thank you @igorsimko @sr-rossanderson. Looks like this issue is similar to #26823

Let me verify again.

[pahud]

Hi

Looks at this

aws-cdk/packages/@aws-cdk/aws-lambda-python-alpha/lib/Dockerfile

Line 9 in bbdd42c

ARG POETRY_VERSION=1.5.1

Looks like current POETRY_VERSION in the vended Dockerfile is 1.5.1. This means when you bundle your layer using the vended Dockerfile, 1.5.1 should be used, rather than 2.0 or above. Can you elaborate on why you are seeing this error?

I am using the 2.174.1 both CLI and lib.

 % npx cdk --version        
2.174.1 (build f353fc7)
hunhsieh@f84d898d8bae issue-triage % grep aws-cdk-lib package.json

    "aws-cdk-lib": "2.174.1",

[pahud]

@igorsimko

There was fix for installing correct version of poetry during lambda build in
95f8cef#diff-72edaa63ace26f9b6ba27bf4e9420a69f29a4fbe9e574e0c5a1ca269e5b5691a so you need at least version v2.93.0.

In my case I was using older version of aws-cdk where poetry version wasn't locked and with recent release of poetry (v2.0.0) bundling of lambda broke because it installed poetry version 2 where poetry export was dropped.

Yes the version pinning starts from 2.93.0 so you can't use 2.0 since then.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[ibqn]

is there any workaround for using poetry version 2.0.0?

I tried to use

const lambda = new PythonFunction(this, 'Lambda', {
            entry: path.resolve(...),
            runtime: lambda.Runtime.PYTHON_3_12,
            architecture: lambda.Architecture.ARM_64,
            bundling: {
                buildArgs: {
                    POETRY_VERSION: '2.0.0',
                },
            },
            ...
 })

however, that does not seem to help

[sr-rossanderson]

Although my overal cdk version was showing as 2.143.0, after sending logs over to AWS they discovered that I was using v2.92.0 of @aws-cdk/aws-lambda-python-alpha in a couple of my stacks. Uplifted @aws-cdk/aws-lambda-python-alpha to latest and things are looking better now.

[igorsimko]

@ibqn

is there any workaround for using poetry version 2.0.0?

If updating cdk is not an option for you then you can simply patch it on your end. Instead of using python lambda construct, you can just use DockerImageFunction and use something like CDK's Dockerfile

[ibqn]

@ibqn

is there any workaround for using poetry version 2.0.0?

If updating cdk is not an option for you then you can simply patch it on your end. Instead of using python lambda construct, you can just use DockerImageFunction and use something like CDK's Dockerfile

latest version of cdk does not solve this problem, i.e. locked version of poetry 1.5.1 in the dockerfile is not able to use new format of poetry 2.0 lock file.

on the other side when using the latest poerty version as mentioned above the export command is no longer included.

...
 Error: docker exited with status 1
--> Command: docker run --rm -u "502:20" -v "/...:/asset-input:delegated" -v "/.../cdk.out/asset.7edda80864b6f3bebdd458400267b3b8addd6e792b7340c5fb92551ff263c1c9:/asset-output:delegated" -w "/asset-input" cdk-da53f057a981fe2a2056d6da878d5f9c1b6d219897b44a26704be8e2efcfaeeb bash -c "rsync -rLv /asset-input/ /asset-output && cd /asset-output && poetry export --without-hashes --with-credentials --format requirements.txt --output requirements.txt && python -m pip install -r requirements.txt -t /asset-output"
    at AssetStaging.bundle (/.../node_modules/.pnpm/[email protected][email protected]/node_modules/aws-cdk-lib/core/lib/asset-staging.js:2:619)
...

[ibqn]

@igorsimko @pahud therefore, please address this issue. thanks

[pahud]

Hi

As I mentioned earlier in this thread
Given

aws-cdk/packages/@aws-cdk/aws-lambda-python-alpha/lib/Dockerfile

Line 9 in bbdd42c

ARG POETRY_VERSION=1.5.1

we generally have two options:

1. pass a POETRY_VERSION buildArg using to a custom value on bundling to override the default 1.5.1 in Dockerfile
2. use a custom Dockerfile

I think it's possible via Custom Bundling but please note CDK team does not fully test the 2.0 and we still recommend the default 1.5.1. Feel free to share how it works for you in this thread and we welcome any feedbacks here.

[ibqn]

Thanks, I understand that.

however, is if not save to include something like that

# Install the poetry-plugin-export plugin, needed for poetry versions >= 2.0.0
    poetry self add poetry-plugin-export -q && \

to the docker file. that should not do anything for versions starting from poetry=1.5.1 till 2.0.0 and will also include desired export plugin starting from versions 2.0.0.

is it not a good compromise i.e. is there any risk?

[sltmyr]

Thanks for the discussion everyone! Similar to @sr-rossanderson I’ve had an older version of @aws-cdk/aws-lambda-python-alpha in the stack. Upgrading to the latest version fixes things.
@pahud Regarding your question for feedback: I’ve been using a custom build image based on this file: https://github.com/aws/aws-cdk/pull/32761/files and the deployments work without any issues, so I suppose supporting Poetry 2.0 is possible. Of course, I only tested it in our setup and I'm not sure what else you need to check.
I guess only passing the POETRY_VERSION buildArg will not work because then we are be back at the original error of the missing plugin that needs to be installed first.

[pahud]

@sltmyr I see.

So instead of passing the ARG override, the Dockerfile still needs

poetry self add poetry-plugin-export

Right?

It would be very struggling if CDK should bump that support as the default version but we should always allow users to use a custom Dockerfile whenever possible. Looking at the package README, I don't see any sample that allows us to use a custom Dockerfile. I guess the freedom using custom Dockerfile might be the way to go rather than just update or bump the vended one. What do you think?

[pahud]

OK. I just noticed PythonLayerVersionProps allows you to specify bundling options, from where you can specify your own image. This indicates you should be allowed to provide your own bundling image with:

DockerImage.fromBuild(entry);

And hence using your own Dockerfile. In this case, you have the full freedom to customize your own docker image for the bundling. Is that something your are looking for?

[sltmyr]

Yes, this is how I used my custom build image

new PythonFunction(this, `Name`, {
    ...
    bundling: {
      image: DockerImage.fromRegistry("<URL-TO-DOCKER-IMAGE>"),
    }
})

It's great to have this option, it's just one more thing you need to know about, set up, and re-check after updates. In my eyes it would be nice to use the new poetry version in the default image so you do not need to set up a custom build image when setting up a new python lambda with the default poetry version.

[delaine-eb]

I am having this issue and solved it by using a custom command in the bundling step. It would be great if the poetry version was pinned automatically in the bundling step.

new PythonFunction(this, `Name`, {
    ...
    bundling: {
      command: [
          'bash', '-c',
          'pip install poetry==1.5.1 && poetry install --no-dev --no-interaction --no-ansi && cp -r . /asset-output'
      ],
      user: 'root'
    }
})