Skip to content

Commit

Permalink
Merge pull request #106 from radiofrance/improve-dx
Browse files Browse the repository at this point in the history 
Improve user experience
  • Loading branch information
@graillus
graillus authored May 23, 2022
2 parents 770845d + 0dc9f92 commit 066b2a1
Show file tree
Hide file tree
Showing 24 changed files with 649 additions and 1,058 deletions.
127 changes: 92 additions & 35 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,64 +1,121 @@
DIB (Docker Image Builder) is a tool to build multiple Docker images hosted within the same repository.
DIB: Docker Image Builder
=========================

![CI Status](https://img.shields.io/github/workflow/status/radiofrance/dib/CI?label=CI&logo=github%20actions&logoColor=fff)
[![codecov](https://codecov.io/gh/radiofrance/dib/branch/main/graph/badge.svg)](https://codecov.io/gh/radiofrance/dib)
![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/radiofrance/dib?sort=semver)

# Concepts
DIB is a tool designed to help build multiple Docker images defined within a directory, possibly having dependencies
with one another.

First, DIB creates a graph representation of all Dockerfiles in the repository with their dependencies.
Imagine you have the following Dockerfiles in your repository (you can see this example in the `test/fixtures` directory).
Each image in a subfolder depends on the image in its parent directory (e.g. `sub-image` has a `FROM bullseye` instruction)
## Features

- Build all your Docker images with a single command.
- Only build images when something has changed since last build.
- Supports dependencies between images, builds will be queued until all parent images are built.
- Run test suites on images, and ensure the tests pass before promoting images.
- Multiple build backends supported (Docker/BuildKit, Kaniko)
- Multiple executors supported (Shell, Docker, Kubernetes)

## How it works

DIB recursively parses all Dockerfiles found within a directory, and builds a dependency graph. It computes a unique
hash from each image build context and Dockerfile (plus the hashes from images dependencies if any). This hash is then
converted to human-readable tag, which will make the final image tag.

When an image build context, Dockerfile, or any parent image changes, the hash changes (as well as the human-readable
tag) and DIB knows the image needs to be rebuilt. If the tag is already present on the registry, DIB considers there is
nothing to do as the image has already been built and pushed. This mechanism allows to only build what is necessary.

Example with a simple directory structure:

```
debian
├── Dockerfile # Image: debian-bullseye
└── nginx
└── Dockerfile # Image: nginx
```
└── bullseye
├── Dockerfile
├── external-parent
│ └── Dockerfile
├── multistage
│ └── Dockerfile
├── skipbuild
│ └── Dockerfile
└── sub-image
└── Dockerfile

The parent `debian-bullseye` image depends on an external image, not managed by DIB :

```dockerfile
# debian/Dockerfile
FROM debian:bullseye
LABEL name="debian-bullseye"
```

Then, DIB will check the git diff between the current version of your repository and the previous one.
To figure out the name of the image to build, DIB uses the `name` label (`debian-bullseye` here). The image name is then
appended to the configured registry URL (we'll use `gcr.io/project` in examples). The target image DIB will build here
is `gcr.io/project/debian-bullseye`.

For the `nginx` image, we need to extend the `gcr.io/project/debian-bullseye` image :

To calculate this "previous" version, DIB use a file called `.docker-version` stored in the docker build directory.
The content of this file is a unique hash of the docker directoy, generated by `dib hash`.
```dockerfile
# debian/nginx/Dockerfile
FROM gcr.io/project/debian-bullseye:DIB_MANAGED_VERSION
LABEL name="nginx"
```

Then, for each dockerfile, if its was changed since the last update, it is taggued for rebuild. The tag of this newly
built image is the result of `dib hash`. If the image is unchanged, a new tag is created from the previous tag.
The `DIB_MANAGED_VERSION` placeholder tells DIB it should use the latest `debian-bullseye` image built by DIB itself.
DIB will always use the latest built image, based on the current filesystem state. If the `debian-bullseye`
image changed, it will be rebuilt first, then `nginx` will also be rebuilt because it depends on it.

# Installation
## Installation

First, install the cli
### With Go install:

```
go install github.com/radiofrance/dib@latest
```

Dib uses a "referential" image to store metadata about previous runs. This image needs to be present on the remote
registry. A bootstrap dockerfile is present in the `init` directory with the instructions to build it.
### Download binaries:

Binaries are available to download from the [GitHub releases](https://github.com/radiofrance/dib/releases) page.

## Usage

# Initialisation
Check `dib --help` for command usage.

DIB requires a "meta" docker image to store metadata on previous dib builds. By default, dib expects an image named
`dib-referential` to be present on the registry. If you choose to use another name than `dib-referential`, you will need
to set it in your command-line, with `--referential-image`, or in the dib config file.
## Configuration

The `init` folder helps setup this image. It contains a simple minimalist Dockerfile.
DIB can be configured either by command-line flags, environment variables or configuration file.

```bash
docker build -t <your_registry_url>/dib-referential init
docker push <your_registry_url>/dib-referential
The command-line flags have the highest priority, then environment variables, then config file. This means you can set
default values in the configuration file, and then override with environment variables of command-line flags.

### Command-line flags

Example:

```shell
dib build --registry-url=gcr.io/project path/to/images/dir
```

### Environment variables

Environment variables must be prefixed by `DIB_` followed by the capitalized, snake_cased flag name.

Example:

```shell
export DIB_REGISTRY_URL=gcr.io/project
dib build path/to/images/dir
```

# Usage
### Configuration file

A `.dib.yaml` config file is expected in the current working directory. You can change the file location with
the `--config` (`-c`) flag.

Check `dib --help` for command usage
The YAML keys must be camelCased flag names.

Example

```yaml
# .dib.yaml
registryUrl: gcr.io/project
```
# License
## License
dib is released under the [CeCILL V2.1 License](https://cecill.info/licences/Licence_CeCILL_V2.1-en.txt)
114 changes: 30 additions & 84 deletions cmd/build.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,8 @@ package cmd

import (
"context"
"errors"
"fmt"
"os"
"path"
"strings"

"github.com/aws/aws-sdk-go-v2/config"
"github.com/radiofrance/dib/dib"
Expand All @@ -19,7 +16,6 @@ import (
"github.com/radiofrance/dib/ratelimit"
"github.com/radiofrance/dib/registry"
"github.com/radiofrance/dib/types"
versn "github.com/radiofrance/dib/version"
"github.com/sirupsen/logrus"
"github.com/spf13/cobra"
kube "gitlab.com/radiofrance/kubecli"
Expand All @@ -29,15 +25,13 @@ const (
backendDocker = "docker"
backendKaniko = "kaniko"

placeholderNonExistent = "non-existent"
junitReportsDirectory = "dist/testresults/goss"
junitReportsDirectory = "dist/testresults/goss"
)

type buildOpts struct {
// Root options
BuildPath string `mapstructure:"build_path"`
RegistryURL string `mapstructure:"registry_url"`
ReferentialImage string `mapstructure:"referential_image"`
BuildPath string `mapstructure:"build_path"`
RegistryURL string `mapstructure:"registry_url"`

// Build specific options
DisableGenerateGraph bool `mapstructure:"no_graph"`
Expand Down Expand Up @@ -105,6 +99,8 @@ var buildCmd = &cobra.Command{
For each image, if any file part of its docker context has changed, the image will be rebuilt.
Otherwise, dib will create a new tag based on the previous tag`,
Run: func(cmd *cobra.Command, args []string) {
bindPFlagsSnakeCase(cmd.Flags())

opts := buildOpts{}
hydrateOptsFromViper(&opts)

Expand All @@ -123,34 +119,34 @@ Otherwise, dib will create a new tag based on the previous tag`,
},
}

//nolint:lll
func init() {
rootCmd.AddCommand(buildCmd)

buildCmd.Flags().Bool("dry-run", false, "Simulate what would happen without actually doing anything dangerous.")
buildCmd.Flags().Bool("force-rebuild", false, "Forces rebuilding the entire image graph, without regarding if the target version already exists.")
buildCmd.Flags().Bool("no-graph", false, "Disable generation of graph during the build process.")
buildCmd.Flags().Bool("no-tests", false, "Disable execution of tests during the build process.")
buildCmd.Flags().Bool("no-junit", false, "Disable generation of junit reports when running tests")
buildCmd.Flags().Bool("release", false, "This flag will cause all images to be "+
"retagged with the tags defined by the 'dib.extra-tags' Dockerfile's label, the referential "+
"image will also be tagged ")
buildCmd.Flags().Bool("local-only", false, "Build docker images locally, do not push on remote registry")
buildCmd.Flags().StringP("backend", "b", backendDocker, fmt.Sprintf("Build Backend used to run image builds. Supported backends: %v", supportedBackends))
buildCmd.Flags().Int("rate-limit", 1, "Concurrent number of build that can run simultaneously")

bindPFlagsSnakeCase(buildCmd.Flags())
buildCmd.Flags().Bool("dry-run", false,
"Simulate what would happen without actually doing anything dangerous.")
buildCmd.Flags().Bool("force-rebuild", false,
"Forces rebuilding the entire image graph, without regarding if the target version already exists.")
buildCmd.Flags().Bool("no-graph", false,
"Disable generation of graph during the build process.")
buildCmd.Flags().Bool("no-tests", false,
"Disable execution of tests during the build process.")
buildCmd.Flags().Bool("no-junit", false,
"Disable generation of junit reports when running tests")
buildCmd.Flags().Bool("release", false,
"Enable release mode to tag all images with extra tags found in the `dib.extra-tags` Dockerfile labels.")
buildCmd.Flags().Bool("local-only", false,
"Build docker images locally, do not push on remote registry")
buildCmd.Flags().StringP("backend", "b", backendDocker,
fmt.Sprintf("Build Backend used to run image builds. Supported backends: %v", supportedBackends))
buildCmd.Flags().Int("rate-limit", 1, ""+
"Concurrent number of builds that can run simultaneously")
}

func doBuild(opts buildOpts) error {
workingDir, err := getWorkingDir()
if err != nil {
return fmt.Errorf("failed to get current working directory: %w", err)
}
dockerDir, err := findDockerRootDir(workingDir, opts.BuildPath)
if err != nil {
return err
}

gcrRegistry, err := registry.NewRegistry(opts.RegistryURL, opts.DryRun)
if err != nil {
Expand Down Expand Up @@ -178,7 +174,7 @@ func doBuild(opts buildOpts) error {
case backendDocker:
builder = dockerBuilderTagger
case backendKaniko:
builder = createKanikoBuilder(opts, shell, workingDir, dockerDir)
builder = createKanikoBuilder(opts, shell, workingDir)
default:
logrus.Fatalf("Invalid backend \"%s\": not supported", opts.Backend)
}
Expand All @@ -192,28 +188,11 @@ func doBuild(opts buildOpts) error {

logrus.Infof("Building images in directory \"%s\"", path.Join(workingDir, opts.BuildPath))

currentVersion, err := versn.CheckDockerVersionIntegrity(path.Join(workingDir, dockerDir))
if err != nil {
return fmt.Errorf("cannot find current version: %w", err)
}

previousVersion, diffs, err := versn.GetDiffSinceLastDockerVersionChange(
workingDir, shell, gcrRegistry, path.Join(dockerDir, versn.DockerVersionFilename),
path.Join(opts.RegistryURL, opts.ReferentialImage))
if err != nil {
if errors.Is(err, versn.ErrNoPreviousBuild) {
previousVersion = placeholderNonExistent
} else {
return fmt.Errorf("cannot find previous version: %w", err)
}
}

logrus.Debug("Generate DAG")
DAG := dib.GenerateDAG(path.Join(workingDir, opts.BuildPath), opts.RegistryURL)
logrus.Debug("Generate DAG -- Done")

err = dib.Plan(DAG, gcrRegistry, diffs, previousVersion, currentVersion,
opts.Release, opts.ForceRebuild, !opts.DisableRunTests)
err = dib.Plan(DAG, gcrRegistry, opts.ForceRebuild, !opts.DisableRunTests)
if err != nil {
return err
}
Expand All @@ -223,19 +202,9 @@ func doBuild(opts buildOpts) error {
return err
}

if opts.Release {
err = dib.Retag(DAG, tagger)
if err != nil {
return err
}

// We retag the referential image to explicit this commit was build using dib
if err := tagger.Tag(
fmt.Sprintf("%s:%s", path.Join(opts.RegistryURL, opts.ReferentialImage), "latest"),
fmt.Sprintf("%s:%s", path.Join(opts.RegistryURL, opts.ReferentialImage), currentVersion),
); err != nil {
return err
}
err = dib.Retag(DAG, tagger, opts.Release)
if err != nil {
return err
}

if !opts.DisableGenerateGraph {
Expand All @@ -248,38 +217,15 @@ func doBuild(opts buildOpts) error {
return nil
}

// findDockerRootDir iterates over the BuildPath to find the first matching directory containing
// a .docker-version file. We consider this directory as the root docker directory containing all the dockerfiles.
func findDockerRootDir(workingDir, buildPath string) (string, error) {
searchPath := buildPath
for {
_, err := os.Stat(path.Join(workingDir, searchPath, versn.DockerVersionFilename))
if err == nil {
return searchPath, nil
}
if !errors.Is(err, os.ErrNotExist) {
return "", err
}

dir, _ := path.Split(searchPath)
dir = strings.TrimSuffix(dir, "/")
if dir == "" {
return "", fmt.Errorf("searching for docker root dir failed, no directory in %s "+
"contains a %s file", buildPath, versn.DockerVersionFilename)
}
searchPath = dir
}
}

func createKanikoBuilder(opts buildOpts, shell exec.Executor, workingDir, dockerDir string) *kaniko.Builder {
func createKanikoBuilder(opts buildOpts, shell exec.Executor, workingDir string) *kaniko.Builder {
var (
err error
executor kaniko.Executor
contextProvider kaniko.ContextProvider
)

if opts.LocalOnly {
executor = createKanikoDockerExecutor(shell, path.Join(workingDir, dockerDir), opts.Kaniko)
executor = createKanikoDockerExecutor(shell, workingDir, opts.Kaniko)
contextProvider = kaniko.NewLocalContextProvider()
} else {
executor, err = createKanikoKubernetesExecutor(opts.Kaniko)
Expand Down
Loading

0 comments on commit 066b2a1

Please sign in to comment.