(pipelines): improve execution time for pipelines

[henrist]

Deploying with CDK Pipelines adds considerable (in our opinion) time comparing to our previous custom Jenkins pipelines or other custom implementations.

As an example, when we deploy our React web application directly to S3 and trigger an CloudFront invalidation, it takes only a few seconds to complete the whole process. Using CDK Pipelines, performing the S3 deployment using CloudFormation Custom Resource (using webapp-deploy-lambda I've written), takes about 8-10 minutes for the full (deployment) pipeline to complete for two environments (without any test actions).

This is a quite considerable difference causing friction about moving into CDK Pipelines.

Every pipeline seems to have a "baseline" of about 4 minutes "wasted". On top of this comes time added by multiple stages, stack references causing sequential actions, test/validation actions, and of cause whatever the application consists of being deployed with CloudFormation (consider that part out of scope).

Use Case

We work closely up to production, commiting and deploying frequently. Having shorter execution time for pipelines would help us deliver changes faster and provide a faster feedback loop.

Shorter execution time also helps if we introduce problems that needs to be fixed, by using a roll-forward strategy.

We also support deploying branches to the development environment by using the pipeline, but this is usually not feasable due to the duration it takes to perform. Note that we often have the CDK-app and application sources in separate repos, and have built features on top of CDK Pipelines to join these codebases/artifacts as part of a pipeline. Example: https://github.com/capraconsulting/webapp-baseline is deployed by https://github.com/capralifecycle/liflig-cdk-app-reference

Proposed Solution

I'm a bit unsure what can be done about this, so I'm very open for ideas and suggestions, or alternative approaches. Maybe there are some other recommendations I'm missing out on?

Some ideas:

• CodeBuild often waits for queue/provisioning time. Could the synth, self-mutate and asset publishing actions be done from a Lambda instead? (Except for Docker assets)
• Could we somehow skip self-mutate if there is no change?
• Is it possible to optimize the CloudFormation Prepare actions, or possibly avoid using change-sets if possible for some stacks? (I guess not - due to comment in Support cdk deploy stack update without change set - speed up deployments #8493)
• Lift the problem to CloudFormation team to make CloudFormation itself faster to react

CodePipeline itself seems slow to complete actions. This is hard to do much about without swapping it out with something else, such as Step Functions (which would likely be considered a new module), which I guess it out of scope.

Other

From my observation every pipeline has overhead such as:

• Synth - adds about 1 minute (or more for larger apps)
• SelfMutate - adds about 1 minute even if no changes
• Assets - adds about 1 minute
• CloudFormation Prepare - adds about 30-90 seconds (very variable from my observations) - and I'm a bit confused why this would even take 60 seconds for a "no change"

It seems CloudFormation takes about 30 seconds to detect that an action is actually complete. So if a CodeBuild job has a duration of 30 seconds, the CodePipeline duration of that action typically ends up as 60 seconds. I've seen this both with CodeBuild and Lambda actions, which is pretty annoying when having many actions in sequence. The CloudFormation Deploy action however, often takes only 1 second if there are no changes.

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

---

This is a 🚀 Feature Request

[kjpgit]

I just want to +1 this and say I tried pipelines for a day and gave up because it is so slow. For the last year I have run cdk deploy either from my local instance or from jenkins and it is 100x faster than the pipelines. It uses caching (for docker and npm), and is done in seconds, with the only wait being cloudformation.

I don't need a self mutating pipeline especially if it slows things down all the time.

I've seen a custom built solution (lambci) with github webhook -> lambda -> eventbridge -> lambda that run in seconds. The codebuild provisioning time, and lack of caching by default in simpleSynth is really bad. One of our toy projects had 14 buildAsset actions each using codebuild, just to build a few lambdas in pipelines. That is insanely expensive.

[straygar]

Re: CodeBuild provision time - I've been annoyed by this issue as well, and was able to reduce it significantly (~250s -> 30s) by using a different buildImage than the default LinuxBuildImage.STANDARD_1_0 (not sure if this applies to any CDK pipelines jobs, but for custom build jobs this was great help).

Also - it would be amazing to merge all asset deployments into the same job if possible. Running a separate one for each zip/image you want to upload is a bit overkill IMO.

[hoegertn]

I already created the PR to merge all asset deployments into one job. It awaits review.

[henrist]

For the reference - that's #13803

Regarding the CodeBuild image in CDK Pipelines: The image aws/codebuild/standard:4.0 is used. According to AWS docs this should be cached. We would probably see a lot higher provisioning time if it was not, since the image itself is almost 9 GiB. I checked one of my recent assets jobs and it took 19 sec for provisioning - still high in my opinion though. A custom image with FROM alpine still takes 17-18 sec to provision in CodeBuild.

Examle used to test provisioning time of CodeBuild

import * as cdk from "@aws-cdk/core"
import * as codebuild from "@aws-cdk/aws-codebuild"

export class CodebuildTestStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props)

    new codebuild.Project(this, "Project", {
      environment: {
        buildImage: codebuild.LinuxBuildImage.fromAsset(this, "Image", {
          directory: "example",
        }),
      },
      buildSpec: codebuild.BuildSpec.fromObject({
        version: "0.2",
        phases: {
          build: {
            commands: ['echo "Hello world!"'],
          },
        },
      }),
    })
  }
}

Dockerfile

FROM alpine

[rix0rrr]

I love and support this discussion, and I think it's a good one to have. For the record, I also don't know what the best solution would be, but if you come up with tangible solutions that fit into the Pipelines framework, then I'd love to hear them.

If your conclusion is that CDK Pipelines needs to work drastically differently, I think a better solution would probably be to start a competing construct library and publish it. If yours gets more popular than ours, we can always re-asses to make that one the default :).

In any case, an open discussion ticket is not what our issue tracker is for. This is not a forum. Try the Slack channel instead, or GitHub Discussions, or the AWS Forums. This issue is not immediately actionable, so it doesn't belong here.

I will keep it open to encourage the discussion, but we're not likely to move on this ourselves soon.

[Aayush-gupta10]

Hi, do we any update on this issue, can we make our cdk pipeline fast by avoiding assets build steps for those assets which have no change?

[TheRealAmazonKendra]

This issue doesn't seem to have a viable set of action items associated with it so I'm going to convert it to a discussion to retain the good tips in here. If you believe this change was made in error, please feel free to open a new issue.

[jplauri]

One might find this PR interesting and relevant to this discussion: #24584