[Proposal] CBI: Container Builder Interface (Docker, BuildKit, Buildah, kaniko, img, Google Cloud Container Builder, OpenShift Source-to-Image...)

[AkihiroSuda]

I'm working on a project called CBI (Container Builder Interface), which provides a vendor-neutral interface for building container images on top of a Kubernetes cluster: https://github.com/containerbuilding/cbi

The following plugins are implemented so far:

Plugin	Backend	Dockerfile	cloudbuild.yaml	OpenShift S2I	BuildKit LLB
docker	Docker	Yes ✅
buildkit	BuildKit	Yes ✅			Planned
buildah	Buildah	Yes ✅
kaniko	kaniko	Yes ✅
img	img	Yes ✅
gcb	Google Cloud Container Builder	Yes ✅	Yes ✅
s2i	OpenShift Source-to-Image (S2I)			Yes ✅

I'd like to propose adding CBI support to build-related tools such as Skaffold, so that new builders and artifact languages can be easily integrated to these tools without reinventing the wheel.

Integration to Skaffold

Here is a POC of CBI integration to Skaffold, with some examples: AkihiroSuda@55fb5ed

examples/getting-started: Dockerfile + CBI BuildKit plugin

apiVersion: skaffold/v1alpha2
kind: Config
build:
  artifacts:
  - imageName: gcr.io/k8s-skaffold/skaffold-example
deploy:
  kubectl:
    manifests:
      - k8s-*
profiles:
  - name: gcb
    build:
      googleCloudBuild:
        projectId: k8s-skaffold
  - name: cbi
    build:
      cbi:
        buildJobTemplate:
          spec:
            registry:
              secretRef:
                name: my-docker-registry-credential
            pluginSelector: "plugin.name=buildkit"
# supported plugins for Dockerfile: docker,buildkit,buildah,kaniko,img,gcb

The cbi builder requires build.cbi.buildJobTemplate to be set to a template of CBI BuildJob CRD ( https://github.com/containerbuilding/cbi/blob/master/pkg/apis/cbi/v1alpha1/types.go ).

Usually, a user only needs to set build.cbi.buildJobTemplate.spec.registry.secretRef.name to be a name of Docker registry secret created with kubectl create secret docker-registry ....
Other fields are set automatically.
The user can also set ...spec.pluginSelector to specify the CBI plugin explicitly.

The workspace is uploaded to a temporary nginx on the Kubernetes cluster using kubectl cp.
(So, unlike the standard Skaffold kaniko builder, CBI kaniko plugin does not require GCS.)

As CBI supports several build context providers such as Git, GCS, S3, SFTP, WebDAV, and even Dropbox (by using Rclone), we could also instrument Skaffold+CBI to access such remote workspaces.

examples/cbi-s2i: OpenShift Source-to-Image artifact + CBI S2I plugin

apiVersion: skaffold/v1alpha2
kind: Config
build:
  artifacts:
  - imageName: gcr.io/k8s-skaffold/skaffold-example
    workspace: .
    s2i: {}
  cbi:
    buildJobTemplate:
      spec:
        registry:
          secretRef:
            name: my-docker-registry-credential
        language:
          kind: S2I
          s2i:
            baseImage: centos/python-36-centos7

CBI can also support non-Dockerfile artifact language as the OpenShift S2I (#518) example above.

We could also add Bazel (on cluster)/Box/Jib support to CBI. (#479 #241 #241 )

cc @r2d4 @dlorenc @mattmoor

[dlorenc]

Nice! A few overall questions.

What does cbi currently require to run in the cluster? Is there any type of daemon, or is the whole lifecycle managed during a single build?

The cbi types are currently v1alpha1. How much do you expect these to change?

[AkihiroSuda]

Is there any type of daemon,

Yes, there is a CRD controller daemon called cbid.
cbid watches creation of BuildJob CRD, and creates batch/v1.Job manifests correspondingly, using plugins.

Flow:

• Skaffold submits BuildJob CRD manifests to Kubernetes API server.
• cbid watches creation of the BuildJob CRD manifests, and transfers them to CBI plugins via CBI Plugin API (gRPC).
• CBI plugins generates PodTemplateSpec objects and send backs to the cbid. Note that neither cbid nor CBI plugins do not build images by themselves.
• cbid creates and submits batch/v1.Job manifests using the PodTemplateSpec generated by the plugins. PodTemplateSpec is like cbipluginhelper prepare-httpcontext http://foo/bar.gz /context && docker build -t gcr.io/foo/bar /context && docker push gcr.io/foo/bar. The name of the batch/v1.Job is set to BuildJob.Status.Job. If BuildJob is deleted, the underlying batch/v1.Job is automatically deleted as well (via OwnerReference).

The cbi types are currently v1alpha1. How much do you expect these to change?

Likely to change, and proposals for changes are highly welcome 😃 .

For v1alpha1, I'm likely to add:

• spec.registry.allowInsecure bool
• spec.jobLifecycle.BackoffLimit int (BuildJob CRD: support setting batch/v1.Job.Spec.BackoffLimit containerbuilding/cbi#53)
• spec.language.dockerfile.targetStage string (Add image digest to BuildJob.Status containerbuilding/cbi#65)
• spec.language.dockerfile.buildArgs map[string]string
• spec.language.dockerfile.buildSecrets map[string]string (Define and implement BuildSecret spec containerbuilding/cbi#28) (maybe)
• status.digest sha256string (maybe)

For breaking changes, I will bump-up the API version e.g. v1alpha2, and try to write converters.

Please be noted that bump-up of CBI API version does not require bump-up of Skaffold API version.
To support multiple CBI API version in a Skaffold config, BuildJobTemplate in the Scaffold config spec is defined as yaml.MapSlice: AkihiroSuda@55fb5ed#diff-b69ba46e5de62302191cb945810dd218R108
The yaml.MapSlice is marshaled again and unmarshalled to concrete cbivNalphaM.BuildJob using the APIVersion value.

[r2d4]

I like this idea. I think we'll have to think through our artifact-builder compatibility matrix finally (not all artifact types can be built with all builders) and how we reconcile our existing GCB and Kaniko integrations with the ones in the CBI, but I don't see those as blockers.

[AkihiroSuda]

how we reconcile our existing GCB and Kaniko integrations with the ones in the CBI

The existing Scaffold-native builders can be kept and are still useful for those don't want CBI dependency.

[r2d4]

@AkihiroSuda what do you think about just accepting a CBI yaml file instead of the full spec?

apiVersion: skaffold/v1alpha2
kind: Config
build:
  artifacts:
  - imageName: gcr.io/k8s-skaffold/skaffold-example
    workspace: .
    s2i: {}
  cbi:
    buildJobTemplate: cbi.yaml

[AkihiroSuda]

what do you think about just accepting a CBI yaml file instead of the full spec?

SGTM. Maybe we can have both buildJobTemplate (yaml.MapSlice ) and buildJobTemplateFile (string)?

[r2d4]

@AkihiroSuda sorry the delay in getting back to you. We'd happily accept a PR for either

[AkihiroSuda]

Thanks! I will open PR, probably after DockerCon.

[AkihiroSuda]

Sorry for delay, but before opening the PR, let me look into whether we can consolidate into Knative: knative/build#284

[AkihiroSuda]

As Knative Build (#851) turned our to share the same goal as CBI, I'm closing this proposal and going to port over CBI features to Knative Build.