diff --git a/README.md b/README.md index 7cd3531a..b97b3453 100644 --- a/README.md +++ b/README.md @@ -1,64 +1,121 @@ -DIB (Docker Image Builder) is a tool to build multiple Docker images hosted within the same repository. +DIB: Docker Image Builder +========================= ![CI Status](https://img.shields.io/github/workflow/status/radiofrance/dib/CI?label=CI&logo=github%20actions&logoColor=fff) [![codecov](https://codecov.io/gh/radiofrance/dib/branch/main/graph/badge.svg)](https://codecov.io/gh/radiofrance/dib) ![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/radiofrance/dib?sort=semver) -# Concepts +DIB is a tool designed to help build multiple Docker images defined within a directory, possibly having dependencies +with one another. -First, DIB creates a graph representation of all Dockerfiles in the repository with their dependencies. -Imagine you have the following Dockerfiles in your repository (you can see this example in the `test/fixtures` directory). -Each image in a subfolder depends on the image in its parent directory (e.g. `sub-image` has a `FROM bullseye` instruction) +## Features +- Build all your Docker images with a single command. +- Only build images when something has changed since last build. +- Supports dependencies between images, builds will be queued until all parent images are built. +- Run test suites on images, and ensure the tests pass before promoting images. +- Multiple build backends supported (Docker/BuildKit, Kaniko) +- Multiple executors supported (Shell, Docker, Kubernetes) + +## How it works + +DIB recursively parses all Dockerfiles found within a directory, and builds a dependency graph. It computes a unique +hash from each image build context and Dockerfile (plus the hashes from images dependencies if any). This hash is then +converted to human-readable tag, which will make the final image tag. + +When an image build context, Dockerfile, or any parent image changes, the hash changes (as well as the human-readable +tag) and DIB knows the image needs to be rebuilt. If the tag is already present on the registry, DIB considers there is +nothing to do as the image has already been built and pushed. This mechanism allows to only build what is necessary. + +Example with a simple directory structure: + +``` +debian +├── Dockerfile # Image: debian-bullseye +└── nginx + └── Dockerfile # Image: nginx ``` -└── bullseye - ├── Dockerfile - ├── external-parent - │ └── Dockerfile - ├── multistage - │ └── Dockerfile - ├── skipbuild - │ └── Dockerfile - └── sub-image - └── Dockerfile + +The parent `debian-bullseye` image depends on an external image, not managed by DIB : + +```dockerfile +# debian/Dockerfile +FROM debian:bullseye +LABEL name="debian-bullseye" ``` -Then, DIB will check the git diff between the current version of your repository and the previous one. +To figure out the name of the image to build, DIB uses the `name` label (`debian-bullseye` here). The image name is then +appended to the configured registry URL (we'll use `gcr.io/project` in examples). The target image DIB will build here +is `gcr.io/project/debian-bullseye`. + +For the `nginx` image, we need to extend the `gcr.io/project/debian-bullseye` image : -To calculate this "previous" version, DIB use a file called `.docker-version` stored in the docker build directory. -The content of this file is a unique hash of the docker directoy, generated by `dib hash`. +```dockerfile +# debian/nginx/Dockerfile +FROM gcr.io/project/debian-bullseye:DIB_MANAGED_VERSION +LABEL name="nginx" +``` -Then, for each dockerfile, if its was changed since the last update, it is taggued for rebuild. The tag of this newly -built image is the result of `dib hash`. If the image is unchanged, a new tag is created from the previous tag. +The `DIB_MANAGED_VERSION` placeholder tells DIB it should use the latest `debian-bullseye` image built by DIB itself. +DIB will always use the latest built image, based on the current filesystem state. If the `debian-bullseye` +image changed, it will be rebuilt first, then `nginx` will also be rebuilt because it depends on it. -# Installation +## Installation -First, install the cli +### With Go install: ``` go install github.com/radiofrance/dib@latest ``` -Dib uses a "referential" image to store metadata about previous runs. This image needs to be present on the remote -registry. A bootstrap dockerfile is present in the `init` directory with the instructions to build it. +### Download binaries: + +Binaries are available to download from the [GitHub releases](https://github.com/radiofrance/dib/releases) page. + +## Usage -# Initialisation +Check `dib --help` for command usage. -DIB requires a "meta" docker image to store metadata on previous dib builds. By default, dib expects an image named -`dib-referential` to be present on the registry. If you choose to use another name than `dib-referential`, you will need -to set it in your command-line, with `--referential-image`, or in the dib config file. +## Configuration -The `init` folder helps setup this image. It contains a simple minimalist Dockerfile. +DIB can be configured either by command-line flags, environment variables or configuration file. -```bash -docker build -t /dib-referential init -docker push /dib-referential +The command-line flags have the highest priority, then environment variables, then config file. This means you can set +default values in the configuration file, and then override with environment variables of command-line flags. + +### Command-line flags + +Example: + +```shell +dib build --registry-url=gcr.io/project path/to/images/dir +``` + +### Environment variables + +Environment variables must be prefixed by `DIB_` followed by the capitalized, snake_cased flag name. + +Example: + +```shell +export DIB_REGISTRY_URL=gcr.io/project +dib build path/to/images/dir ``` -# Usage +### Configuration file + +A `.dib.yaml` config file is expected in the current working directory. You can change the file location with +the `--config` (`-c`) flag. -Check `dib --help` for command usage +The YAML keys must be camelCased flag names. + +Example + +```yaml +# .dib.yaml +registryUrl: gcr.io/project +``` -# License +## License dib is released under the [CeCILL V2.1 License](https://cecill.info/licences/Licence_CeCILL_V2.1-en.txt) diff --git a/cmd/build.go b/cmd/build.go index cfdc5197..3f3942fe 100644 --- a/cmd/build.go +++ b/cmd/build.go @@ -2,11 +2,8 @@ package cmd import ( "context" - "errors" "fmt" - "os" "path" - "strings" "github.com/aws/aws-sdk-go-v2/config" "github.com/radiofrance/dib/dib" @@ -19,7 +16,6 @@ import ( "github.com/radiofrance/dib/ratelimit" "github.com/radiofrance/dib/registry" "github.com/radiofrance/dib/types" - versn "github.com/radiofrance/dib/version" "github.com/sirupsen/logrus" "github.com/spf13/cobra" kube "gitlab.com/radiofrance/kubecli" @@ -29,15 +25,13 @@ const ( backendDocker = "docker" backendKaniko = "kaniko" - placeholderNonExistent = "non-existent" - junitReportsDirectory = "dist/testresults/goss" + junitReportsDirectory = "dist/testresults/goss" ) type buildOpts struct { // Root options - BuildPath string `mapstructure:"build_path"` - RegistryURL string `mapstructure:"registry_url"` - ReferentialImage string `mapstructure:"referential_image"` + BuildPath string `mapstructure:"build_path"` + RegistryURL string `mapstructure:"registry_url"` // Build specific options DisableGenerateGraph bool `mapstructure:"no_graph"` @@ -105,6 +99,8 @@ var buildCmd = &cobra.Command{ For each image, if any file part of its docker context has changed, the image will be rebuilt. Otherwise, dib will create a new tag based on the previous tag`, Run: func(cmd *cobra.Command, args []string) { + bindPFlagsSnakeCase(cmd.Flags()) + opts := buildOpts{} hydrateOptsFromViper(&opts) @@ -123,23 +119,27 @@ Otherwise, dib will create a new tag based on the previous tag`, }, } -//nolint:lll func init() { rootCmd.AddCommand(buildCmd) - buildCmd.Flags().Bool("dry-run", false, "Simulate what would happen without actually doing anything dangerous.") - buildCmd.Flags().Bool("force-rebuild", false, "Forces rebuilding the entire image graph, without regarding if the target version already exists.") - buildCmd.Flags().Bool("no-graph", false, "Disable generation of graph during the build process.") - buildCmd.Flags().Bool("no-tests", false, "Disable execution of tests during the build process.") - buildCmd.Flags().Bool("no-junit", false, "Disable generation of junit reports when running tests") - buildCmd.Flags().Bool("release", false, "This flag will cause all images to be "+ - "retagged with the tags defined by the 'dib.extra-tags' Dockerfile's label, the referential "+ - "image will also be tagged ") - buildCmd.Flags().Bool("local-only", false, "Build docker images locally, do not push on remote registry") - buildCmd.Flags().StringP("backend", "b", backendDocker, fmt.Sprintf("Build Backend used to run image builds. Supported backends: %v", supportedBackends)) - buildCmd.Flags().Int("rate-limit", 1, "Concurrent number of build that can run simultaneously") - - bindPFlagsSnakeCase(buildCmd.Flags()) + buildCmd.Flags().Bool("dry-run", false, + "Simulate what would happen without actually doing anything dangerous.") + buildCmd.Flags().Bool("force-rebuild", false, + "Forces rebuilding the entire image graph, without regarding if the target version already exists.") + buildCmd.Flags().Bool("no-graph", false, + "Disable generation of graph during the build process.") + buildCmd.Flags().Bool("no-tests", false, + "Disable execution of tests during the build process.") + buildCmd.Flags().Bool("no-junit", false, + "Disable generation of junit reports when running tests") + buildCmd.Flags().Bool("release", false, + "Enable release mode to tag all images with extra tags found in the `dib.extra-tags` Dockerfile labels.") + buildCmd.Flags().Bool("local-only", false, + "Build docker images locally, do not push on remote registry") + buildCmd.Flags().StringP("backend", "b", backendDocker, + fmt.Sprintf("Build Backend used to run image builds. Supported backends: %v", supportedBackends)) + buildCmd.Flags().Int("rate-limit", 1, ""+ + "Concurrent number of builds that can run simultaneously") } func doBuild(opts buildOpts) error { @@ -147,10 +147,6 @@ func doBuild(opts buildOpts) error { if err != nil { return fmt.Errorf("failed to get current working directory: %w", err) } - dockerDir, err := findDockerRootDir(workingDir, opts.BuildPath) - if err != nil { - return err - } gcrRegistry, err := registry.NewRegistry(opts.RegistryURL, opts.DryRun) if err != nil { @@ -178,7 +174,7 @@ func doBuild(opts buildOpts) error { case backendDocker: builder = dockerBuilderTagger case backendKaniko: - builder = createKanikoBuilder(opts, shell, workingDir, dockerDir) + builder = createKanikoBuilder(opts, shell, workingDir) default: logrus.Fatalf("Invalid backend \"%s\": not supported", opts.Backend) } @@ -192,28 +188,11 @@ func doBuild(opts buildOpts) error { logrus.Infof("Building images in directory \"%s\"", path.Join(workingDir, opts.BuildPath)) - currentVersion, err := versn.CheckDockerVersionIntegrity(path.Join(workingDir, dockerDir)) - if err != nil { - return fmt.Errorf("cannot find current version: %w", err) - } - - previousVersion, diffs, err := versn.GetDiffSinceLastDockerVersionChange( - workingDir, shell, gcrRegistry, path.Join(dockerDir, versn.DockerVersionFilename), - path.Join(opts.RegistryURL, opts.ReferentialImage)) - if err != nil { - if errors.Is(err, versn.ErrNoPreviousBuild) { - previousVersion = placeholderNonExistent - } else { - return fmt.Errorf("cannot find previous version: %w", err) - } - } - logrus.Debug("Generate DAG") DAG := dib.GenerateDAG(path.Join(workingDir, opts.BuildPath), opts.RegistryURL) logrus.Debug("Generate DAG -- Done") - err = dib.Plan(DAG, gcrRegistry, diffs, previousVersion, currentVersion, - opts.Release, opts.ForceRebuild, !opts.DisableRunTests) + err = dib.Plan(DAG, gcrRegistry, opts.ForceRebuild, !opts.DisableRunTests) if err != nil { return err } @@ -223,19 +202,9 @@ func doBuild(opts buildOpts) error { return err } - if opts.Release { - err = dib.Retag(DAG, tagger) - if err != nil { - return err - } - - // We retag the referential image to explicit this commit was build using dib - if err := tagger.Tag( - fmt.Sprintf("%s:%s", path.Join(opts.RegistryURL, opts.ReferentialImage), "latest"), - fmt.Sprintf("%s:%s", path.Join(opts.RegistryURL, opts.ReferentialImage), currentVersion), - ); err != nil { - return err - } + err = dib.Retag(DAG, tagger, opts.Release) + if err != nil { + return err } if !opts.DisableGenerateGraph { @@ -248,30 +217,7 @@ func doBuild(opts buildOpts) error { return nil } -// findDockerRootDir iterates over the BuildPath to find the first matching directory containing -// a .docker-version file. We consider this directory as the root docker directory containing all the dockerfiles. -func findDockerRootDir(workingDir, buildPath string) (string, error) { - searchPath := buildPath - for { - _, err := os.Stat(path.Join(workingDir, searchPath, versn.DockerVersionFilename)) - if err == nil { - return searchPath, nil - } - if !errors.Is(err, os.ErrNotExist) { - return "", err - } - - dir, _ := path.Split(searchPath) - dir = strings.TrimSuffix(dir, "/") - if dir == "" { - return "", fmt.Errorf("searching for docker root dir failed, no directory in %s "+ - "contains a %s file", buildPath, versn.DockerVersionFilename) - } - searchPath = dir - } -} - -func createKanikoBuilder(opts buildOpts, shell exec.Executor, workingDir, dockerDir string) *kaniko.Builder { +func createKanikoBuilder(opts buildOpts, shell exec.Executor, workingDir string) *kaniko.Builder { var ( err error executor kaniko.Executor @@ -279,7 +225,7 @@ func createKanikoBuilder(opts buildOpts, shell exec.Executor, workingDir, docker ) if opts.LocalOnly { - executor = createKanikoDockerExecutor(shell, path.Join(workingDir, dockerDir), opts.Kaniko) + executor = createKanikoDockerExecutor(shell, workingDir, opts.Kaniko) contextProvider = kaniko.NewLocalContextProvider() } else { executor, err = createKanikoKubernetesExecutor(opts.Kaniko) diff --git a/cmd/graph.go b/cmd/graph.go index c3eb6d4a..dc102adc 100644 --- a/cmd/graph.go +++ b/cmd/graph.go @@ -1,19 +1,16 @@ package cmd import ( - "errors" "fmt" "log" "path" "github.com/radiofrance/dib/dib" - "github.com/radiofrance/dib/exec" "github.com/radiofrance/dib/graphviz" "github.com/radiofrance/dib/preflight" "github.com/radiofrance/dib/registry" - versn "github.com/radiofrance/dib/version" - "github.com/sirupsen/logrus" + "github.com/sirupsen/logrus" "github.com/spf13/cobra" ) @@ -21,12 +18,10 @@ import ( var graphCmd = &cobra.Command{ Use: "graph", Short: "Create a visual representation of the build graph", - Long: `Create a visual representation of the build graph using graphviz + Long: `Create a visual representation of the build graph using graphviz. -In the generated graph, images are represented with color status -Red means the image will be rebuilt -Yellow means the image will be re-taged from its previous built version -Transparent means no action on the image`, +In the generated graph, images are represented with red color when it needs to be rebuilt. +It remains colorless when no action is required for the image.`, Run: func(cmd *cobra.Command, args []string) { preflight.RunPreflightChecks([]string{"dot"}) @@ -48,41 +43,17 @@ func doGraph(opts rootOpts) error { if err != nil { return fmt.Errorf("failed to get current working directory: %w", err) } - dockerDir, err := findDockerRootDir(workingDir, opts.BuildPath) - if err != nil { - return err - } gcrRegistry, err := registry.NewRegistry(opts.RegistryURL, true) if err != nil { return err } - shell := &exec.ShellExecutor{ - Dir: workingDir, - } - - currentVersion, err := versn.CheckDockerVersionIntegrity(path.Join(workingDir, dockerDir)) - if err != nil { - return fmt.Errorf("cannot find current version: %w", err) - } - - previousVersion, diffs, err := versn.GetDiffSinceLastDockerVersionChange( - workingDir, shell, gcrRegistry, path.Join(dockerDir, versn.DockerVersionFilename), - path.Join(opts.RegistryURL, opts.ReferentialImage)) - if err != nil { - if errors.Is(err, versn.ErrNoPreviousBuild) { - previousVersion = placeholderNonExistent - } else { - return fmt.Errorf("cannot find previous version: %w", err) - } - } - logrus.Debug("Generate DAG") DAG := dib.GenerateDAG(path.Join(workingDir, opts.BuildPath), opts.RegistryURL) logrus.Debug("Generate DAG -- Done") - err = dib.Plan(DAG, gcrRegistry, diffs, previousVersion, currentVersion, true, false, false) + err = dib.Plan(DAG, gcrRegistry, false, false) if err != nil { return err } diff --git a/cmd/hash.go b/cmd/hash.go deleted file mode 100644 index a3949888..00000000 --- a/cmd/hash.go +++ /dev/null @@ -1,31 +0,0 @@ -package cmd - -import ( - "fmt" - "log" - - versionpkg "github.com/radiofrance/dib/version" - "github.com/spf13/cobra" -) - -// hashCmd represents the hash command. -var hashCmd = &cobra.Command{ - Use: "hash [build path]", - Short: "Generates a version hash of the docker directory", - Long: `dib hash will calculate a unique human readable hash of the "docker" directory, which -contains all Dockerfiles. If no argument is passed to dib hash, it will use 'docker' as the directory name`, - Args: cobra.MaximumNArgs(1), - Run: func(cmd *cobra.Command, args []string) { - opts := rootOpts{} - hydrateOptsFromViper(&opts) - hash, err := versionpkg.GetDockerVersionHash(opts.BuildPath) - if err != nil { - log.Fatal(err) - } - fmt.Print(hash) // nolint: forbidigo - }, -} - -func init() { - rootCmd.AddCommand(hashCmd) -} diff --git a/cmd/migrate.go b/cmd/migrate.go new file mode 100644 index 00000000..f47ee5fc --- /dev/null +++ b/cmd/migrate.go @@ -0,0 +1,75 @@ +package cmd + +import ( + "fmt" + "path" + + "github.com/radiofrance/dib/dag" + "github.com/radiofrance/dib/dib" + "github.com/radiofrance/dib/registry" + "github.com/sirupsen/logrus" + "github.com/spf13/cobra" +) + +type migrateOpts struct { + // Root options + BuildPath string `mapstructure:"build_path"` + RegistryURL string `mapstructure:"registry_url"` + + // Migrate specific options + DryRun bool `mapstructure:"dry_run"` + DockerVersion string `mapstructure:"docker_version"` +} + +// migrateCmd represents the migrate command. +var migrateCmd = &cobra.Command{ + Use: "migrate", + Short: "Migrate from old version system (<0.9.0)", + Long: "This command will tag every image from the given docker-version tag with their new tag computed from the current filesystem state.", //nolint:lll + Run: func(cmd *cobra.Command, args []string) { + bindPFlagsSnakeCase(cmd.Flags()) + + opts := migrateOpts{} + hydrateOptsFromViper(&opts) + + err := doMigrate(opts) + if err != nil { + logrus.Fatalf("Migration failed: %v", err) + } + }, +} + +func init() { + rootCmd.AddCommand(migrateCmd) + + migrateCmd.Flags().String("docker-version", "", + "The tag present in the .docker-version file") + migrateCmd.Flags().Bool("dry-run", false, + "Dry run mode, see what would happen without actually changing anything") +} + +func doMigrate(opts migrateOpts) error { + workingDir, err := getWorkingDir() + if err != nil { + return fmt.Errorf("failed to get current working directory: %w", err) + } + + gcrRegistry, err := registry.NewRegistry(opts.RegistryURL, opts.DryRun) + if err != nil { + return err + } + logrus.Debug("Generate DAG") + DAG := dib.GenerateDAG(path.Join(workingDir, opts.BuildPath), opts.RegistryURL) + logrus.Debug("Generate DAG -- Done") + + DAG.Walk(func(node *dag.Node) { + oldTag := node.Image.DockerRef(opts.DockerVersion) + newTag := node.Image.DockerRef(node.Image.Hash) + if err := gcrRegistry.Tag(oldTag, newTag); err != nil { + logrus.Errorf("Failed to tag \"%s\" from \"%s\"", oldTag, newTag) + } + }) + + logrus.Info("Migration process completed") + return nil +} diff --git a/cmd/root.go b/cmd/root.go index a0cd6ebb..f4ad4410 100644 --- a/cmd/root.go +++ b/cmd/root.go @@ -15,7 +15,6 @@ import ( const ( defaultRegistryURL = "eu.gcr.io/my-test-repository" - defaultReferentialImage = "dib-referential" defaultLogLevel = "info" defaultBuildPath = "docker" defaultGossImage = "aelsabbahy/goss:latest" @@ -38,7 +37,7 @@ var rootCmd = &cobra.Command{ Short: "An Opinionated Docker Image Builder", Long: `Docker Image Builder helps building a complex image dependency graph -Run dib --help for mor information`, +Run dib --help for more information`, } // Execute runs the root cobra command. @@ -50,23 +49,16 @@ func Execute() { func init() { cobra.OnInitialize(initConfig, initLogLevel, cleanupDist) - desc := `Path to the directory you want to build All Dockerfiles within this directory will be recursively + rootCmd.PersistentFlags().StringVar(&cfgFile, "config", "", + "config file (default is $HOME/.config/.dib.yaml)") + rootCmd.PersistentFlags().String("build-path", defaultBuildPath, + `Path to the directory containing all Dockerfiles to be built by dib. Every Dockerfile will be recursively found and added to the build graph. You can provide any subdirectory if you want to focus on a reduced set of images, -as long as it has at least one Dockerfile in it. - -It is also required that one of the directories in this path contains a .docker-version file. This directory will -be considered as the root directory for the hash generation and comparison` - - rootCmd.PersistentFlags().StringVar(&cfgFile, "config", "", "config file (default is $HOME/.config/.dib.yaml)") - rootCmd.PersistentFlags().String("build-path", defaultBuildPath, desc) - rootCmd.PersistentFlags().String("registry-url", defaultRegistryURL, "Docker registry URL where images are stored.") - rootCmd.PersistentFlags().String("referential-image", defaultReferentialImage, "Name of an image on "+ - "the registry. This image will be used as a reference for checking build completion of previous dib runs. "+ - "Tags will be added to this image but it has no other purpose.") - rootCmd.PersistentFlags().StringP("log-level", "l", defaultLogLevel, "Log level. Can be any level "+ - "supported by logrus (\"info\", \"debug\", etc...)") - - bindPFlagsSnakeCase(rootCmd.PersistentFlags()) +as long as it has at least one Dockerfile in it.`) + rootCmd.PersistentFlags().String("registry-url", defaultRegistryURL, + "Docker registry URL where images are stored.") + rootCmd.PersistentFlags().StringP("log-level", "l", defaultLogLevel, + "Log level. Can be any level supported by logrus (\"info\", \"debug\", etc...)") } func cleanupDist() { diff --git a/dag/image.go b/dag/image.go index f124a112..77c642a0 100644 --- a/dag/image.go +++ b/dag/image.go @@ -10,19 +10,29 @@ import ( type Image struct { Name string ShortName string - CurrentTag string // Current tag expected to be present on the registry before the build. - TargetTag string // New tag, not present in registry until the image is built and pushed. - ExtraTags []string // A list of tags to make in addition to TargetTag. + ExtraTags []string // A list of tags to make in addition to image hash. + Hash string // Hash of the build context "At the moment" Dockerfile *dockerfile.Dockerfile IgnorePatterns []string NeedsRebuild bool NeedsTests bool - NeedsRetag bool RetagDone bool RebuildDone bool RebuildFailed bool } +// CurrentRef returns the fully-qualified docker ref for the current version. +// If the image needs to be rebuilt, a temporary `dev-` prefix is added to the tag. +func (img Image) CurrentRef() string { + tag := img.Hash + + if img.NeedsRebuild { + tag = "dev-" + img.Hash + } + + return img.DockerRef(tag) +} + // DockerRef returns the fully-qualified docker ref for a given version. func (img Image) DockerRef(version string) string { return fmt.Sprintf("%s:%s", img.Name, version) diff --git a/dag/image_test.go b/dag/image_test.go index 8ef61969..cba57ad6 100644 --- a/dag/image_test.go +++ b/dag/image_test.go @@ -7,6 +7,29 @@ import ( "github.com/stretchr/testify/assert" ) +func Test_CurrentRef_EqualsHashWhenNoRebuildNeeded(t *testing.T) { + t.Parallel() + + image := dag.Image{ + Name: "gcr.io/project-id/nginx", + Hash: "version", + } + + assert.Equal(t, "gcr.io/project-id/nginx:version", image.CurrentRef()) +} + +func Test_CurrentRef_HasDevPrevixWhenNeedsRebuild(t *testing.T) { + t.Parallel() + + image := dag.Image{ + Name: "gcr.io/project-id/nginx", + Hash: "version", + NeedsRebuild: true, + } + + assert.Equal(t, "gcr.io/project-id/nginx:dev-version", image.CurrentRef()) +} + func Test_DockerRef(t *testing.T) { t.Parallel() diff --git a/dib/build.go b/dib/build.go index 613099cc..97f7780a 100644 --- a/dib/build.go +++ b/dib/build.go @@ -96,7 +96,7 @@ func RebuildNode(node *dag.Node, builder types.ImageBuilder, testRunners []types if img.NeedsTests { report.TestsStatus = TestsStatusPassed - if err := testImage(img, testRunners, img.TargetTag); err != nil { + if err := testImage(img, testRunners); err != nil { report.TestsStatus = TestsStatusFailed report.FailureMessage = err.Error() } @@ -117,14 +117,7 @@ func doRebuild(node *dag.Node, builder types.ImageBuilder, rateLimiter ratelimit // of any dib-managed images used as dependencies in the Dockerfile. tagsToReplace := make(map[string]string) for _, parent := range node.Parents() { - if parent.Image.NeedsRebuild { - // The parent image was rebuilt, we have to use its new tag. - tagsToReplace[parent.Image.Name] = parent.Image.DockerRef(parent.Image.TargetTag) - continue - } - - // The parent image has not changed, we can use the existing tag. - tagsToReplace[parent.Image.Name] = parent.Image.DockerRef(parent.Image.CurrentTag) + tagsToReplace[parent.Image.Name] = parent.Image.CurrentRef() } if err := dockerfile.ReplaceTags(*img.Dockerfile, tagsToReplace); err != nil { return fmt.Errorf("failed to replace tag in dockerfile %s: %w", img.Dockerfile.ContextPath, err) @@ -160,13 +153,13 @@ func doRebuild(node *dag.Node, builder types.ImageBuilder, rateLimiter ratelimit opts := types.ImageBuilderOpts{ Context: img.Dockerfile.ContextPath, - Tag: fmt.Sprintf("%s:%s", img.Name, img.TargetTag), + Tag: img.CurrentRef(), Labels: labels, Push: !localOnly, LogOutput: fileOutput, } - logrus.Infof("Building \"%s:%s\" in context \"%s\"", img.Name, img.TargetTag, img.Dockerfile.ContextPath) + logrus.Infof("Building \"%s\" in context \"%s\"", img.CurrentRef(), img.Dockerfile.ContextPath) if err := builder.Build(opts); err != nil { return fmt.Errorf("building image %s failed: %w", img.ShortName, err) diff --git a/dib/generate_dag.go b/dib/generate_dag.go index 3a371ecd..7b0150fc 100644 --- a/dib/generate_dag.go +++ b/dib/generate_dag.go @@ -1,27 +1,40 @@ package dib import ( + "crypto/sha256" + "errors" "fmt" + "io" "os" "path" "path/filepath" - - "github.com/docker/cli/cli/command/image/build" + "sort" + "strings" "github.com/radiofrance/dib/dag" "github.com/radiofrance/dib/dockerfile" + + "github.com/docker/cli/cli/command/image/build" + "github.com/docker/docker/pkg/fileutils" "github.com/sirupsen/logrus" + "github.com/wolfeidau/humanhash" ) +const dockerignore = ".dockerignore" + // GenerateDAG discovers and parses all Dockerfiles at a given path, // and generates the DAG representing the relationships between images. func GenerateDAG(buildPath string, registryPrefix string) *dag.DAG { + var allFiles []string cache := make(map[string]*dag.Node) allParents := make(map[string][]string) err := filepath.Walk(buildPath, func(filePath string, info os.FileInfo, err error) error { if err != nil { return err } + if !info.IsDir() { + allFiles = append(allFiles, filePath) + } if dockerfile.IsDockerfile(filePath) { dckfile, err := dockerfile.ParseDockerfile(filePath) if err != nil { @@ -32,9 +45,9 @@ func GenerateDAG(buildPath string, registryPrefix string) *dag.DAG { if hasSkipLabel && skipBuild == "true" { return nil } - imageShortName, hasSkipLabel := dckfile.Labels["name"] - if !hasSkipLabel { - return fmt.Errorf("missing label \"image\" in Dockerfile at path \"%s\"", filePath) + imageShortName, hasNameLabel := dckfile.Labels["name"] + if !hasNameLabel { + return fmt.Errorf("missing label \"name\" in Dockerfile at path \"%s\"", filePath) } img := &dag.Image{ Name: fmt.Sprintf("%s/%s", registryPrefix, imageShortName), @@ -42,6 +55,11 @@ func GenerateDAG(buildPath string, registryPrefix string) *dag.DAG { Dockerfile: dckfile, } + extraTagsLabel, hasLabel := img.Dockerfile.Labels["dib.extra-tags"] + if hasLabel { + img.ExtraTags = append(img.ExtraTags, strings.Split(extraTagsLabel, ",")...) + } + ignorePatterns, err := build.ReadDockerignore(path.Dir(filePath)) if err != nil { return fmt.Errorf("could not read ignore patterns: %w", err) @@ -66,12 +84,130 @@ func GenerateDAG(buildPath string, registryPrefix string) *dag.DAG { } } - DAG := dag.DAG{} + DAG := &dag.DAG{} // If an image has no parents in the DAG, we consider it a root image for name, img := range cache { if len(img.Parents()) == 0 { DAG.AddNode(cache[name]) } } - return &DAG + + if err := generateHashes(DAG, allFiles); err != nil { + logrus.Fatal(err) + } + + return DAG +} + +func generateHashes(graph *dag.DAG, allFiles []string) error { + nodeFiles := map[*dag.Node][]string{} + + fileBelongsTo := map[string]*dag.Node{} + for _, file := range allFiles { + fileBelongsTo[file] = nil + } + + // First, we do a depth-first search in the image graph to map every file the image they belong to. + // We start from the most specific image paths (children of children of children...), and we get back up + // to parent images, to avoid false-positive and false-negative matches. + // Files matching any pattern in the .dockerignore file are ignored. + graph.WalkInDepth(func(node *dag.Node) { + nodeFiles[node] = []string{} + for _, file := range allFiles { + if !strings.HasPrefix(file, node.Image.Dockerfile.ContextPath) { + // The current file is not lying in the current image build context, nor in a subdirectory. + continue + } + + if fileBelongsTo[file] != nil { + // The current file has already been assigned to an image, most likely to a child image. + continue + } + + if path.Base(file) == dockerignore { + // We ignore dockerignore file itself for simplicity + // In the real world, this file should not be ignored but it + // helps us in managing refactoring + continue + } + + if node.Image.IgnorePatterns != nil { + if matchPattern(node, file) { + // The current file matches a pattern in the dockerignore file + continue + } + } + + // If we reach here, the file is part of the current image's context, we mark it as so. + fileBelongsTo[file] = node + nodeFiles[node] = append(nodeFiles[node], file) + } + }) + + return graph.WalkErr(func(node *dag.Node) error { + var parentHashes []string + for _, parent := range node.Parents() { + parentHashes = append(parentHashes, parent.Image.Hash) + } + + hash, err := hashFiles(nodeFiles[node], parentHashes) + if err != nil { + return fmt.Errorf("could not hash files for node %s: %w", node.Image.Name, err) + } + node.Image.Hash = hash + return nil + }) +} + +// matchPattern checks whether a file matches the images ignore patterns. +// It returns true if the file matches at least one pattern (meaning it should be ignored). +func matchPattern(node *dag.Node, file string) bool { + ignorePatternMatcher, err := fileutils.NewPatternMatcher(node.Image.IgnorePatterns) + if err != nil { + logrus.Errorf("Could not create pattern matcher for %s, ignoring", node.Image.ShortName) + return false + } + + prefix := strings.TrimPrefix(strings.TrimPrefix(file, node.Image.Dockerfile.ContextPath), "/") + match, err := ignorePatternMatcher.Matches(prefix) + if err != nil { + logrus.Errorf("Could not match pattern for %s, ignoring", node.Image.ShortName) + return false + } + return match +} + +// hashFiles computes the sha256 from the contents of the files passed as argument. +// The files are alphabetically sorted so the returned hash is always the same. +// This also means the hash will change if the file names change but the contents don't. +func hashFiles(files []string, parentHashes []string) (string, error) { + hash := sha256.New() + files = append([]string(nil), files...) + sort.Strings(files) + for _, file := range files { + if strings.Contains(file, "\n") { + return "", errors.New("filenames with newlines are not supported") + } + readCloser, err := os.Open(file) + if err != nil { + return "", err + } + hashFile := sha256.New() + _, err = io.Copy(hashFile, readCloser) + readCloser.Close() + if err != nil { + return "", err + } + fmt.Fprintf(hash, "%x %s\n", hashFile.Sum(nil), file) + } + + for _, parentHash := range parentHashes { + hash.Write([]byte(parentHash)) + } + + humanReadableHash, err := humanhash.Humanize(hash.Sum(nil), 4) + if err != nil { + return "", fmt.Errorf("could not humanize hash: %w", err) + } + return humanReadableHash, nil } diff --git a/dib/generate_dag_test.go b/dib/generate_dag_test.go index 42469aaf..0826aa66 100644 --- a/dib/generate_dag_test.go +++ b/dib/generate_dag_test.go @@ -1,26 +1,22 @@ package dib_test import ( + "io/ioutil" "os" "path" "testing" + "github.com/radiofrance/dib/dag" "github.com/radiofrance/dib/dib" "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" ) func Test_GenerateDAG(t *testing.T) { t.Parallel() - cwd, err := os.Getwd() - if err != nil { - t.Fatal("Failed to get current working directory.") - } - - DAG := dib.GenerateDAG( - path.Join(cwd, "../test/fixtures/docker"), - "eu.gcr.io/my-test-repository", - ) + dockerDir := setupFixtures(t) + DAG := dib.GenerateDAG(dockerDir, "eu.gcr.io/my-test-repository") assert.Len(t, DAG.Nodes(), 1) @@ -29,5 +25,165 @@ func Test_GenerateDAG(t *testing.T) { assert.Equal(t, "eu.gcr.io/my-test-repository/bullseye", rootImage.Name) assert.Equal(t, "bullseye", rootImage.ShortName) assert.Len(t, rootNode.Parents(), 0) - assert.Len(t, rootNode.Children(), 3) + assert.Len(t, rootNode.Children(), 2) + + nodes := flattenNodes(DAG) + + multistageNode, exists := nodes["multistage"] + require.True(t, exists) + assert.Equal(t, []string{"latest"}, multistageNode.Image.ExtraTags) +} + +func Test_GenerateDAG_HashesChangeWhenImageContextChanges(t *testing.T) { + t.Parallel() + + testcases := map[string]struct { + AddFileAtPath string + ExpectRootImageHashesToBeEqual bool + ExpectSubImageHashesToBeEqual bool + ExpectMultistageImageHashesToBeEqual bool + }{ + "Child image hash changes when child context changes": { + AddFileAtPath: "bullseye/multistage/newfile", + ExpectRootImageHashesToBeEqual: true, + ExpectSubImageHashesToBeEqual: true, + ExpectMultistageImageHashesToBeEqual: false, + }, + "Child hash changes when parent context changes": { + AddFileAtPath: "bullseye/newfile", + ExpectRootImageHashesToBeEqual: false, + ExpectSubImageHashesToBeEqual: false, + ExpectMultistageImageHashesToBeEqual: false, + }, + } + + //nolint:paralleltest + for name, testcase := range testcases { + t.Run(name, func(t *testing.T) { + t.Parallel() + + // Given I have a docker directory with some Dockerfiles inside + dockerDir := setupFixtures(t) + initialDAG := dib.GenerateDAG(dockerDir, "eu.gcr.io/my-test-repository") + initialNodes := flattenNodes(initialDAG) + + initialRootNode, exists := initialNodes["bullseye"] + require.True(t, exists) + + initialSubNode, exists := initialNodes["sub-image"] + require.True(t, exists) + + initialMultistageNode, exists := initialNodes["multistage"] + require.True(t, exists) + + // When I add a new file in bullseye/multistage/ (child node) + err := ioutil.WriteFile( + path.Join(dockerDir, testcase.AddFileAtPath), + []byte("file contents"), + os.ModePerm, + ) + require.NoError(t, err) + + // Then ONLY the hash of the child node bullseye/multistage should have changed + newDAG := dib.GenerateDAG(dockerDir, "eu.gcr.io/my-test-repository") + newNodes := flattenNodes(newDAG) + + newRootNode, exists := newNodes["bullseye"] + require.True(t, exists) + + newSubNode, exists := newNodes["sub-image"] + require.True(t, exists) + + newMultistageNode, exists := newNodes["multistage"] + require.True(t, exists) + + if testcase.ExpectRootImageHashesToBeEqual { + assert.Equal(t, initialRootNode.Image.Hash, newRootNode.Image.Hash) + } else { + assert.NotEqual(t, initialRootNode.Image.Hash, newRootNode.Image.Hash) + } + + if testcase.ExpectSubImageHashesToBeEqual { + assert.Equal(t, initialSubNode.Image.Hash, newSubNode.Image.Hash) + } else { + assert.NotEqual(t, initialSubNode.Image.Hash, newSubNode.Image.Hash) + } + + if testcase.ExpectMultistageImageHashesToBeEqual { + assert.Equal(t, initialMultistageNode.Image.Hash, newMultistageNode.Image.Hash) + } else { + assert.NotEqual(t, initialMultistageNode.Image.Hash, newMultistageNode.Image.Hash) + } + }) + } +} + +// setupFixtures create a tmp directory with some Dockerfiles inside. +func setupFixtures(t *testing.T) string { + t.Helper() + + tmpDockerDir := t.TempDir() + err := os.MkdirAll(path.Join(tmpDockerDir, "bullseye/multistage"), os.ModePerm) + require.NoError(t, err) + err = os.MkdirAll(path.Join(tmpDockerDir, "bullseye/sub-image"), os.ModePerm) + require.NoError(t, err) + err = os.MkdirAll(path.Join(tmpDockerDir, "bullseye/skipbuild"), os.ModePerm) + require.NoError(t, err) + err = ioutil.WriteFile( + path.Join(tmpDockerDir, "bullseye/Dockerfile"), + []byte(` +FROM debian:bullseye + +LABEL name="bullseye" +LABEL version="v1" + `), + os.ModePerm, + ) + require.NoError(t, err) + err = ioutil.WriteFile( + path.Join(tmpDockerDir, "bullseye/sub-image/Dockerfile"), + []byte(` +FROM eu.gcr.io/my-test-repository/bullseye:v1 + +LABEL name="sub-image" +LABEL version="v1" + `), + os.ModePerm, + ) + require.NoError(t, err) + err = ioutil.WriteFile( + path.Join(tmpDockerDir, "bullseye/multistage/Dockerfile"), + []byte(` +FROM eu.gcr.io/my-test-repository/bullseye:v1 as builder +FROM eu.gcr.io/my-test-repository/node:v1 + +LABEL name="multistage" +LABEL dib.extra-tags="latest" + `), + os.ModePerm, + ) + require.NoError(t, err) + err = ioutil.WriteFile( + path.Join(tmpDockerDir, "bullseye/skipbuild/Dockerfile"), + []byte(` +FROM eu.gcr.io/my-test-repository/bullseye:v1 + +LABEL name="skipbuild" +LABEL skipbuild="true" + `), + os.ModePerm, + ) + require.NoError(t, err) + + return tmpDockerDir +} + +func flattenNodes(graph *dag.DAG) map[string]*dag.Node { + flatNodes := map[string]*dag.Node{} + + graph.Walk(func(node *dag.Node) { + flatNodes[node.Image.ShortName] = node + }) + + return flatNodes } diff --git a/dib/plan.go b/dib/plan.go index 56a0bbd1..634595d5 100644 --- a/dib/plan.go +++ b/dib/plan.go @@ -2,41 +2,15 @@ package dib import ( "fmt" - "path" - "strings" "sync" - "github.com/docker/docker/pkg/fileutils" - "github.com/radiofrance/dib/dag" "github.com/radiofrance/dib/types" "github.com/sirupsen/logrus" ) -const dockerignore = ".dockerignore" - // Plan decides which actions need to be performed on each image. -func Plan(graph *dag.DAG, registry types.DockerRegistry, diffs []string, oldTag, newTag string, - releaseMode, forceRebuild, testsEnabled bool, -) error { - // Populate CurrentTag, TargetTag and ExtraTags for all images in the graph. - graph.Walk(func(node *dag.Node) { - img := node.Image - img.CurrentTag = oldTag - img.TargetTag = newTag - - if img.Dockerfile == nil || img.Dockerfile.Labels == nil { - return - } - - extraTagsLabel, hasLabel := img.Dockerfile.Labels["dib.extra-tags"] - if !hasLabel { - return - } - - node.Image.ExtraTags = append(node.Image.ExtraTags, strings.Split(extraTagsLabel, ",")...) - }) - +func Plan(graph *dag.DAG, registry types.DockerRegistry, forceRebuild, testsEnabled bool) error { if forceRebuild { logrus.Info("force rebuild mode enabled, all images will be rebuild regardless of their changes") graph.Walk(func(node *dag.Node) { @@ -45,34 +19,17 @@ func Plan(graph *dag.DAG, registry types.DockerRegistry, diffs []string, oldTag, }) return nil } - checkDiff(graph, diffs) - - currentTagExistsMap, err := refExistsMapForTag(graph, registry, newTag) - if err != nil { - return err - } - previousTagExistsMap, err := refExistsMapForTag(graph, registry, oldTag) - if err != nil { - return err - } - err = checkNeedsRebuild(graph, previousTagExistsMap, oldTag) + tagExistsMap, err := refExistsMapForTag(graph, registry) if err != nil { return err } - err = checkAlreadyBuilt(graph, currentTagExistsMap, newTag) + err = checkNeedsRebuild(graph, tagExistsMap) if err != nil { return err } - if releaseMode { // In release mode, we retag all images. - err = checkNeedsRetag(graph, currentTagExistsMap, newTag) - if err != nil { - return err - } - } - if !testsEnabled { return nil } @@ -86,159 +43,37 @@ func Plan(graph *dag.DAG, registry types.DockerRegistry, diffs []string, oldTag, return nil } -// checkDiff checks the diffs and marks images to be rebuilt if files in their context have changed. -func checkDiff(graph *dag.DAG, diffs []string) { - diffBelongsTo := map[string]*dag.Node{} - for _, file := range diffs { - diffBelongsTo[file] = nil - } - - // First, we do a depth-first search in the image graph to check if the files in diff belong to an image, - // or is dockerignored - // We start from the most specific image paths (children of children of children...), and we get back up - // to parent images, to avoid false-positive and false-negative matches. - graph.WalkInDepth(func(node *dag.Node) { - for _, file := range diffs { - if !strings.HasPrefix(file, node.Image.Dockerfile.ContextPath) { - // The current file is not lying in the current image build context, nor in a subdirectory. - continue - } - - if diffBelongsTo[file] != nil { - // The current file has already been assigned to an image, most likely to a child image. - continue - } - - if path.Base(file) == dockerignore { - // We ignore dockerignore file itself for simplicity - // In the real world, this file should not be ignored but it - // helps us in managing refactoring - continue - } - - if node.Image.IgnorePatterns != nil { - if matchPattern(node, file) { - // The current file matches a pattern in the dockerignore file - continue - } - } - - // If we reach here, the diff file is part of the current image's context, we mark it as so. - diffBelongsTo[file] = node - } - }) - - for file, node := range diffBelongsTo { - if node != nil { - logrus.Debugf("Image \"%s\" needs a rebuild because file \"%s\" has changed", node.Image.Name, file) - // Mark image and all its children for rebuild. - node.Walk(tagForRebuildFunc) - } - } -} - -func matchPattern(node *dag.Node, file string) bool { - ignorePatternMatcher, err := fileutils.NewPatternMatcher(node.Image.IgnorePatterns) - if err != nil { - logrus.Errorf("Could not create pattern matcher for %s, ignoring", node.Image.ShortName) - return false - } - - prefix := strings.TrimPrefix(strings.TrimPrefix(file, node.Image.Dockerfile.ContextPath), "/") - match, err := ignorePatternMatcher.Matches(prefix) - if err != nil { - logrus.Errorf("Could not match pattern for %s, ignoring", node.Image.ShortName) - return false - } - return match -} - -func tagForRebuildFunc(node *dag.Node) { - node.Image.NeedsRebuild = true -} - // checkNeedsRebuild iterates over the graph to find out which images -// can't be found with the former tag, and must be rebuilt. -func checkNeedsRebuild(graph *dag.DAG, previousTagExistsMap *sync.Map, oldTag string) error { - return graph.WalkErr(func(node *dag.Node) error { - img := node.Image - previousTagExists, present := previousTagExistsMap.Load(img.DockerRef(oldTag)) - if !present { - return fmt.Errorf("could not find ref %s in previousTagExists map", img.DockerRef(oldTag)) - } - if previousTagExists.(bool) { //nolint:forcetypeassert - logrus.Debugf("Previous tag \"%s:%s\" exists, no rebuild required", img.Name, oldTag) - return nil - } - - logrus.Warnf("Previous tag \"%s:%s\" missing, image must be rebuilt", img.Name, oldTag) - node.Walk(tagForRebuildFunc) - return nil - }) -} - -// checkAlreadyBuilt iterates over the graph to find out which images -// already exist in the new version, so they don't need to be built again. -func checkAlreadyBuilt(graph *dag.DAG, currentTagExistsMap *sync.Map, newTag string) error { - return graph.WalkErr(func(node *dag.Node) error { - img := node.Image - if !img.NeedsRebuild { - // Don't rebuild images that didn't change since last built revision. - return nil - } - - refAlreadyExists, present := currentTagExistsMap.Load(img.DockerRef(newTag)) - if !present { - return fmt.Errorf("could not find ref %s in map", img.DockerRef(newTag)) - } - if refAlreadyExists.(bool) { //nolint:forcetypeassert - // Looks like dib has already built this image in a previous run, - // we can skip the build, but we don't want to disable the tests. - // This is to avoid the situation where the tests failed on previous dib run, - // then they can no longer be triggered because the image was built and push. - img.RebuildDone = true - - logrus.Debugf("Image \"%s\" is tagged for rebuild but ref is already present on the registry, skipping."+ - " if you want to rebuild anyway, use --force-rebuild", img.Name) - } - return nil - }) -} - -// checkNeedsRetag iterates over the graph to find out which images need -// to be tagged with the new tag from the latest version. -func checkNeedsRetag(graph *dag.DAG, currentTagExistsMap *sync.Map, newTag string) error { +// can't be found with their current tag, and must be rebuilt. +func checkNeedsRebuild(graph *dag.DAG, tagExistsMap *sync.Map) error { return graph.WalkErr(func(node *dag.Node) error { img := node.Image - if img.NeedsRebuild { - // If this image needs rebuild, then its children too, no need to go deeper - return nil - } - - currentTagExists, present := currentTagExistsMap.Load(img.DockerRef(newTag)) + ref := img.DockerRef(img.Hash) + tagExists, present := tagExistsMap.Load(ref) if !present { - return fmt.Errorf("could not find ref %s in currentTagExists map", img.DockerRef(newTag)) + return fmt.Errorf("could not check if %s exists", ref) } - if currentTagExists.(bool) { //nolint:forcetypeassert - logrus.Debugf("Current tag \"%s:%s\" already exists, nothing to do", img.Name, newTag) + if tagExists.(bool) { //nolint:forcetypeassert + logrus.Debugf("Ref \"%s\" already exists, no rebuild required", ref) return nil } - logrus.Debugf("Current tag \"%s:%s\" does not exist, image will be tagged", img.Name, newTag) - img.NeedsRetag = true + logrus.Warnf("Ref \"%s\" is missing, image must be rebuilt", ref) + img.NeedsRebuild = true return nil }) } -func refExistsMapForTag(graph *dag.DAG, registry types.DockerRegistry, tag string) (*sync.Map, error) { +func refExistsMapForTag(graph *dag.DAG, registry types.DockerRegistry) (*sync.Map, error) { refExistsMap := &sync.Map{} err := graph.WalkAsyncErr(func(node *dag.Node) error { img := node.Image - refAlreadyExists, err := registry.RefExists(img.DockerRef(tag)) + ref := img.DockerRef(img.Hash) + refAlreadyExists, err := registry.RefExists(ref) if err != nil { return err } - refExistsMap.Store(img.DockerRef(tag), refAlreadyExists) + refExistsMap.Store(ref, refAlreadyExists) return nil }) if err != nil { diff --git a/dib/plan_test.go b/dib/plan_test.go index 18e5c0f9..f905a7fd 100644 --- a/dib/plan_test.go +++ b/dib/plan_test.go @@ -12,9 +12,10 @@ import ( "github.com/stretchr/testify/assert" ) -func newNode(name string, contextPath string) *dag.Node { +func newNode(name, hash, contextPath string) *dag.Node { return dag.NewNode(&dag.Image{ Name: name, + Hash: hash, ShortName: path.Base(contextPath), Dockerfile: &dockerfile.Dockerfile{ ContextPath: contextPath, @@ -28,15 +29,15 @@ func newNode(name string, contextPath string) *dag.Node { }) } -//nolint:lll +//nolint:lll,dupl func Test_Plan_RebuildAll(t *testing.T) { t.Parallel() - rootNode := newNode("bullseye", "/root/docker/bullseye") + rootNode := newNode("bullseye", "notexists0", "/root/docker/bullseye") - firstChildNode := newNode("eu.gcr.io/my-test-repository/first", "/root/docker/bullseye/first") - secondChildNode := newNode("eu.gcr.io/my-test-repository/second", "/root/docker/bullseye/second") - subChildNode := newNode("eu.gcr.io/my-test-repository/third", "/root/docker/bullseye/second/third") + firstChildNode := newNode("eu.gcr.io/my-test-repository/first", "exists1", "/root/docker/bullseye/first") + secondChildNode := newNode("eu.gcr.io/my-test-repository/second", "exists2", "/root/docker/bullseye/second") + subChildNode := newNode("eu.gcr.io/my-test-repository/third", "exists3", "/root/docker/bullseye/second/third") secondChildNode.AddChild(subChildNode) @@ -46,20 +47,15 @@ func Test_Plan_RebuildAll(t *testing.T) { DAG := &dag.DAG{} DAG.AddNode(rootNode) - diff := []string{ - "/root/docker/bullseye/Dockerfile", - } - registry := &mock.Registry{Lock: &sync.Mutex{}} registry.ExistingRefs = []string{ - // Old tag from previous version - "bullseye:old", - "eu.gcr.io/my-test-repository/first:old", - "eu.gcr.io/my-test-repository/second:old", - "eu.gcr.io/my-test-repository/third:old", + "bullseye:notexists0", + "eu.gcr.io/my-test-repository/first:exists1", + "eu.gcr.io/my-test-repository/second:exists2", + "eu.gcr.io/my-test-repository/third:exists3", } - err := dib.Plan(DAG, registry, diff, "old", "new", true, true, true) + err := dib.Plan(DAG, registry, true, true) assert.NoError(t, err) assert.True(t, rootNode.Image.NeedsRebuild) // Root image was modified. @@ -67,12 +63,6 @@ func Test_Plan_RebuildAll(t *testing.T) { assert.True(t, secondChildNode.Image.NeedsRebuild) // Second image was NOT modified, but its parent was. assert.True(t, subChildNode.Image.NeedsRebuild) // Second's child image was NOT modified but its parent's parent was. - // All images will be rebuilt, no need to retag anything. - assert.False(t, rootNode.Image.NeedsRetag) - assert.False(t, firstChildNode.Image.NeedsRetag) - assert.False(t, secondChildNode.Image.NeedsRetag) - assert.False(t, subChildNode.Image.NeedsRetag) - // All images will be rebuilt, they all need to be tested because tests are enabled. assert.True(t, rootNode.Image.NeedsTests) assert.True(t, firstChildNode.Image.NeedsTests) @@ -86,14 +76,15 @@ func Test_Plan_RebuildAll(t *testing.T) { assert.False(t, subChildNode.Image.RebuildDone) } -func Test_Plan_RebuildOnlyDiff(t *testing.T) { +//nolint:dupl +func Test_Plan_RebuildOnlyModifiedImages(t *testing.T) { t.Parallel() - rootNode := newNode("bullseye", "/root/docker/bullseye") + rootNode := newNode("bullseye", "exists0", "/root/docker/bullseye") - firstChildNode := newNode("eu.gcr.io/my-test-repository/first", "/root/docker/bullseye/first") - secondChildNode := newNode("eu.gcr.io/my-test-repository/second", "/root/docker/bullseye/second") - subChildNode := newNode("eu.gcr.io/my-test-repository/third", "/root/docker/bullseye/second/third") + firstChildNode := newNode("eu.gcr.io/my-test-repository/first", "notexists1", "/root/docker/bullseye/first") + secondChildNode := newNode("eu.gcr.io/my-test-repository/second", "exists2", "/root/docker/bullseye/second") + subChildNode := newNode("eu.gcr.io/my-test-repository/third", "notexists3", "/root/docker/bullseye/second/third") secondChildNode.AddChild(subChildNode) @@ -103,21 +94,15 @@ func Test_Plan_RebuildOnlyDiff(t *testing.T) { DAG := &dag.DAG{} DAG.AddNode(rootNode) - diff := []string{ - "/root/docker/bullseye/first/nginx.conf", - "/root/docker/bullseye/second/third/Dockerfile", - } - registry := &mock.Registry{Lock: &sync.Mutex{}} registry.ExistingRefs = []string{ - // Old tag from previous version - "bullseye:old", - "eu.gcr.io/my-test-repository/first:old", - "eu.gcr.io/my-test-repository/second:old", - "eu.gcr.io/my-test-repository/third:old", + "bullseye:exists0", + "eu.gcr.io/my-test-repository/first:exists1", + "eu.gcr.io/my-test-repository/second:exists2", + "eu.gcr.io/my-test-repository/third:exists3", } - err := dib.Plan(DAG, registry, diff, "old", "new", true, false, true) + err := dib.Plan(DAG, registry, false, true) assert.NoError(t, err) assert.False(t, rootNode.Image.NeedsRebuild) // Root image was NOT modified. @@ -125,12 +110,6 @@ func Test_Plan_RebuildOnlyDiff(t *testing.T) { assert.False(t, secondChildNode.Image.NeedsRebuild) // Second image was NOT modified, nor its parent. assert.True(t, subChildNode.Image.NeedsRebuild) // Second's child image was modified. - // Images that won't be rebuilt need a new tag - assert.True(t, rootNode.Image.NeedsRetag) - assert.False(t, firstChildNode.Image.NeedsRetag) - assert.True(t, secondChildNode.Image.NeedsRetag) - assert.False(t, subChildNode.Image.NeedsRetag) - // Images that need rebuild need to be tested as tests are enabled assert.False(t, rootNode.Image.NeedsTests) assert.True(t, firstChildNode.Image.NeedsTests) @@ -144,154 +123,14 @@ func Test_Plan_RebuildOnlyDiff(t *testing.T) { assert.False(t, subChildNode.Image.RebuildDone) } -func Test_Plan_ImagesAlreadyBuilt(t *testing.T) { - t.Parallel() - - rootNode := newNode("bullseye", "/root/docker/bullseye") - - firstChildNode := newNode("eu.gcr.io/my-test-repository/first", "/root/docker/bullseye/first") - secondChildNode := newNode("eu.gcr.io/my-test-repository/second", "/root/docker/bullseye/second") - subChildNode := newNode("eu.gcr.io/my-test-repository/third", "/root/docker/bullseye/second/third") - - secondChildNode.AddChild(subChildNode) - - rootNode.AddChild(firstChildNode) - rootNode.AddChild(secondChildNode) - - DAG := &dag.DAG{} - DAG.AddNode(rootNode) - - diff := []string{ - "/root/docker/bullseye/first/nginx.conf", - "/root/docker/bullseye/second/third/Dockerfile", - } - - registry := &mock.Registry{Lock: &sync.Mutex{}} - registry.ExistingRefs = []string{ - // Old tag from previous version - "bullseye:old", - "eu.gcr.io/my-test-repository/first:old", - "eu.gcr.io/my-test-repository/second:old", - "eu.gcr.io/my-test-repository/third:old", - // New tag from current version - "eu.gcr.io/my-test-repository/first:new", - "eu.gcr.io/my-test-repository/third:new", - } - err := dib.Plan(DAG, registry, diff, "old", "new", true, false, true) - assert.NoError(t, err) - - assert.False(t, rootNode.Image.NeedsRebuild) // Root image was NOT modified. - assert.True(t, firstChildNode.Image.NeedsRebuild) // First image was modified. - assert.False(t, secondChildNode.Image.NeedsRebuild) // Second image was NOT modified, nor its parent. - assert.True(t, subChildNode.Image.NeedsRebuild) // Second's child image was modified. - - // Image that already exist in registry are flagged. - assert.False(t, rootNode.Image.RebuildDone) - assert.True(t, firstChildNode.Image.RebuildDone) - assert.False(t, secondChildNode.Image.RebuildDone) - assert.True(t, subChildNode.Image.RebuildDone) - - // Only non-modified images need a new tag - assert.True(t, rootNode.Image.NeedsRetag) - assert.False(t, firstChildNode.Image.NeedsRetag) - assert.True(t, secondChildNode.Image.NeedsRetag) - assert.False(t, subChildNode.Image.NeedsRetag) - - // Images that need rebuild need to be tested as tests are enabled - assert.False(t, rootNode.Image.NeedsTests) - assert.True(t, firstChildNode.Image.NeedsTests) - assert.False(t, secondChildNode.Image.NeedsTests) - assert.True(t, subChildNode.Image.NeedsTests) -} - -func Test_Plan_ImagesAlreadyTagged(t *testing.T) { - t.Parallel() - - rootNode := newNode("bullseye", "/root/docker/bullseye") - - firstChildNode := newNode("eu.gcr.io/my-test-repository/first", "/root/docker/bullseye/first") - secondChildNode := newNode("eu.gcr.io/my-test-repository/second", "/root/docker/bullseye/second") - subChildNode := newNode("eu.gcr.io/my-test-repository/third", "/root/docker/bullseye/second/third") - - secondChildNode.AddChild(subChildNode) - - rootNode.AddChild(firstChildNode) - rootNode.AddChild(secondChildNode) - - DAG := &dag.DAG{} - DAG.AddNode(rootNode) - - diff := []string{""} - - registry := &mock.Registry{Lock: &sync.Mutex{}} - registry.ExistingRefs = []string{ - // Old tag from previous version - "bullseye:old", - "eu.gcr.io/my-test-repository/first:old", - "eu.gcr.io/my-test-repository/second:old", - "eu.gcr.io/my-test-repository/third:old", - // New tag from current version - "bullseye:new", - "eu.gcr.io/my-test-repository/first:new", - "eu.gcr.io/my-test-repository/second:new", - "eu.gcr.io/my-test-repository/third:new", - } - err := dib.Plan(DAG, registry, diff, "old", "new", true, false, true) - assert.NoError(t, err) - - assert.False(t, rootNode.Image.NeedsRebuild) - assert.False(t, firstChildNode.Image.NeedsRebuild) - assert.False(t, secondChildNode.Image.NeedsRebuild) - assert.False(t, subChildNode.Image.NeedsRebuild) - - // No need for a neg tag as it already exist in registry - assert.False(t, rootNode.Image.NeedsRetag) - assert.False(t, firstChildNode.Image.NeedsRetag) - assert.False(t, secondChildNode.Image.NeedsRetag) - assert.False(t, subChildNode.Image.NeedsRetag) -} - -func Test_Plan_OldTagNotFoundInRegistry(t *testing.T) { - t.Parallel() - - rootNode := newNode("bullseye", "/root/docker/bullseye") - - firstChildNode := newNode("eu.gcr.io/my-test-repository/first", "/root/docker/bullseye/first") - secondChildNode := newNode("eu.gcr.io/my-test-repository/second", "/root/docker/bullseye/second") - subChildNode := newNode("eu.gcr.io/my-test-repository/third", "/root/docker/bullseye/second/third") - - secondChildNode.AddChild(subChildNode) - - rootNode.AddChild(firstChildNode) - rootNode.AddChild(secondChildNode) - - DAG := &dag.DAG{} - DAG.AddNode(rootNode) - - diff := []string{ - "/root/docker/bullseye/first/nginx.conf", - "/root/docker/bullseye/second/third/Dockerfile", - } - - registry := &mock.Registry{Lock: &sync.Mutex{}} - registry.ExistingRefs = []string{} - err := dib.Plan(DAG, registry, diff, "old", "new", true, false, true) - assert.NoError(t, err) - - assert.True(t, rootNode.Image.NeedsRebuild) - assert.True(t, firstChildNode.Image.NeedsRebuild) - assert.True(t, secondChildNode.Image.NeedsRebuild) - assert.True(t, subChildNode.Image.NeedsRebuild) -} - func Test_Plan_TestsDisabled(t *testing.T) { t.Parallel() - rootNode := newNode("bullseye", "/root/docker/bullseye") + rootNode := newNode("bullseye", "exists0", "/root/docker/bullseye") - firstChildNode := newNode("eu.gcr.io/my-test-repository/first", "/root/docker/bullseye/first") - secondChildNode := newNode("eu.gcr.io/my-test-repository/second", "/root/docker/bullseye/second") - subChildNode := newNode("eu.gcr.io/my-test-repository/third", "/root/docker/bullseye/second/third") + firstChildNode := newNode("eu.gcr.io/my-test-repository/first", "notexists1", "/root/docker/bullseye/first") + secondChildNode := newNode("eu.gcr.io/my-test-repository/second", "notexists2", "/root/docker/bullseye/second") + subChildNode := newNode("eu.gcr.io/my-test-repository/third", "exists3", "/root/docker/bullseye/second/third") secondChildNode.AddChild(subChildNode) @@ -301,14 +140,9 @@ func Test_Plan_TestsDisabled(t *testing.T) { DAG := &dag.DAG{} DAG.AddNode(rootNode) - diff := []string{ - "/root/docker/bullseye/first/nginx.conf", - "/root/docker/bullseye/second/third/Dockerfile", - } - registry := &mock.Registry{Lock: &sync.Mutex{}} registry.ExistingRefs = []string{} - err := dib.Plan(DAG, registry, diff, "old", "new", true, true, false) + err := dib.Plan(DAG, registry, true, false) assert.NoError(t, err) assert.True(t, rootNode.Image.NeedsRebuild) @@ -321,51 +155,3 @@ func Test_Plan_TestsDisabled(t *testing.T) { assert.False(t, secondChildNode.Image.NeedsTests) assert.False(t, subChildNode.Image.NeedsTests) } - -func Test_Plan_ReleaseModeDisabled(t *testing.T) { - t.Parallel() - - rootNode := newNode("bullseye", "/root/docker/bullseye") - - firstChildNode := newNode("eu.gcr.io/my-test-repository/first", "/root/docker/bullseye/first") - secondChildNode := newNode("eu.gcr.io/my-test-repository/second", "/root/docker/bullseye/second") - subChildNode := newNode("eu.gcr.io/my-test-repository/third", "/root/docker/bullseye/second/third") - - secondChildNode.AddChild(subChildNode) - - rootNode.AddChild(firstChildNode) - rootNode.AddChild(secondChildNode) - - DAG := &dag.DAG{} - DAG.AddNode(rootNode) - - diff := []string{ - "/root/docker/bullseye/first/nginx.conf", - "/root/docker/bullseye/second/third/Dockerfile", - } - - registry := &mock.Registry{Lock: &sync.Mutex{}} - registry.ExistingRefs = []string{ - // Old tag from previous version - "bullseye:old", - "eu.gcr.io/my-test-repository/first:old", - "eu.gcr.io/my-test-repository/second:old", - "eu.gcr.io/my-test-repository/third:old", - } - err := dib.Plan(DAG, registry, diff, "old", "new", false, false, true) - assert.NoError(t, err) - - assert.False(t, rootNode.Image.NeedsRebuild) - assert.True(t, firstChildNode.Image.NeedsRebuild) - assert.False(t, secondChildNode.Image.NeedsRebuild) - assert.True(t, subChildNode.Image.NeedsRebuild) - - // Nothing need to be tagged in dev mode - assert.False(t, rootNode.Image.NeedsRetag) - assert.False(t, firstChildNode.Image.NeedsRetag) - assert.False(t, secondChildNode.Image.NeedsRetag) - assert.False(t, subChildNode.Image.NeedsRetag) - - assert.Equal(t, "new", firstChildNode.Image.TargetTag) - assert.Equal(t, "new", firstChildNode.Image.TargetTag) -} diff --git a/dib/tag.go b/dib/tag.go index 97ecf5ed..94a49038 100644 --- a/dib/tag.go +++ b/dib/tag.go @@ -6,29 +6,30 @@ import ( "github.com/sirupsen/logrus" ) -// Retag iterates over the graph to retag each image with the given tag. -func Retag(graph *dag.DAG, tagger types.ImageTagger) error { +// Retag iterates over the graph to tag all images. +func Retag(graph *dag.DAG, tagger types.ImageTagger, release bool) error { return graph.WalkAsyncErr(func(node *dag.Node) error { img := node.Image if img.RetagDone { return nil } - if img.NeedsRetag { - src := img.DockerRef(img.CurrentTag) - dest := img.DockerRef(img.TargetTag) - logrus.Debugf("Tagging \"%s\" from \"%s\"", dest, src) - if err := tagger.Tag(src, dest); err != nil { + current := img.CurrentRef() + final := img.DockerRef(img.Hash) + if current != final { + logrus.Debugf("Tagging \"%s\" from \"%s\"", final, current) + if err := tagger.Tag(current, final); err != nil { return err } } - src := img.DockerRef(img.TargetTag) - for _, tag := range img.ExtraTags { - dest := img.DockerRef(tag) - logrus.Debugf("Tagging \"%s\" from \"%s\"", dest, src) - if err := tagger.Tag(src, dest); err != nil { - return err + if release { + for _, tag := range img.ExtraTags { + extra := img.DockerRef(tag) + logrus.Debugf("Tagging \"%s\" from \"%s\"", extra, final) + if err := tagger.Tag(final, extra); err != nil { + return err + } } } diff --git a/dib/tag_test.go b/dib/tag_test.go index 52dbac6c..7e6a6b9d 100644 --- a/dib/tag_test.go +++ b/dib/tag_test.go @@ -15,124 +15,63 @@ func Test_Retag_DoesNotRetagIfNoRetagNeeded(t *testing.T) { DAG := &dag.DAG{} DAG.AddNode(dag.NewNode(&dag.Image{ - Name: "registry.example.org/image", - ShortName: "image", - NeedsRetag: false, - RetagDone: false, + Name: "registry.example.org/image", + ShortName: "image", + RetagDone: false, })) tagger := &mock.Tagger{} - err := dib.Retag(DAG, tagger) + err := dib.Retag(DAG, tagger, false) assert.NoError(t, err) assert.Empty(t, tagger.RecordedCallsArgs) } -func Test_Retag_DoesNotRetagIfAlreadyDone(t *testing.T) { +func Test_Retag_RetagWhenRebuild(t *testing.T) { t.Parallel() DAG := &dag.DAG{} DAG.AddNode(dag.NewNode(&dag.Image{ - Name: "registry.example.org/image", - ShortName: "image", - CurrentTag: "old", - TargetTag: "new", - NeedsRetag: true, - RetagDone: true, + Name: "registry.example.org/image", + ShortName: "image", + Hash: "myhash", + RetagDone: false, + NeedsRebuild: true, })) tagger := &mock.Tagger{} - err := dib.Retag(DAG, tagger) + err := dib.Retag(DAG, tagger, false) - assert.NoError(t, err) - assert.Empty(t, tagger.RecordedCallsArgs) -} - -func Test_Retag_NoExtraLabels(t *testing.T) { - t.Parallel() - - img := &dag.Image{ - Name: "registry.example.org/image", - ShortName: "image", - CurrentTag: "old", - TargetTag: "new", - NeedsRetag: true, - RetagDone: false, - } - DAG := &dag.DAG{} - DAG.AddNode(dag.NewNode(img)) - - tagger := &mock.Tagger{} - err := dib.Retag(DAG, tagger) - - assert.NoError(t, err) - - assert.Len(t, tagger.RecordedCallsArgs, 1) + require.NoError(t, err) + require.Len(t, tagger.RecordedCallsArgs, 1) args := tagger.RecordedCallsArgs[0] - assert.Equal(t, "registry.example.org/image:old", args.Src) - assert.Equal(t, "registry.example.org/image:new", args.Dest) - - assert.True(t, img.RetagDone) -} - -func Test_Retag_WithExtraLabels(t *testing.T) { - t.Parallel() - - img := &dag.Image{ - Name: "registry.example.org/image", - ShortName: "image", - CurrentTag: "old", - TargetTag: "new", - ExtraTags: []string{"latest1", "latest2"}, - NeedsRetag: true, - RetagDone: false, - } - DAG := &dag.DAG{} - DAG.AddNode(dag.NewNode(img)) - - tagger := &mock.Tagger{} - err := dib.Retag(DAG, tagger) - - assert.NoError(t, err) - - require.Len(t, tagger.RecordedCallsArgs, 3) - - assert.Equal(t, "registry.example.org/image:old", tagger.RecordedCallsArgs[0].Src) - assert.Equal(t, "registry.example.org/image:new", tagger.RecordedCallsArgs[0].Dest) - - assert.Equal(t, "registry.example.org/image:new", tagger.RecordedCallsArgs[1].Src) - assert.Equal(t, "registry.example.org/image:latest1", tagger.RecordedCallsArgs[1].Dest) - assert.Equal(t, "registry.example.org/image:new", tagger.RecordedCallsArgs[2].Src) - assert.Equal(t, "registry.example.org/image:latest2", tagger.RecordedCallsArgs[2].Dest) - - assert.True(t, img.RetagDone) + assert.Equal(t, "registry.example.org/image:dev-myhash", args.Src) + assert.Equal(t, "registry.example.org/image:myhash", args.Dest) } -func Test_Retag_NoRetagNeededWithExtraLabels(t *testing.T) { +func Test_Retag_ReleaseWithExtraTags(t *testing.T) { t.Parallel() img := &dag.Image{ - Name: "registry.example.org/image", - ShortName: "image", - CurrentTag: "old", - TargetTag: "new", - ExtraTags: []string{"latest1", "latest2"}, - NeedsRetag: false, - RetagDone: false, + Name: "registry.example.org/image", + ShortName: "image", + Hash: "myhash", + ExtraTags: []string{"latest1", "latest2"}, + RetagDone: false, } DAG := &dag.DAG{} DAG.AddNode(dag.NewNode(img)) tagger := &mock.Tagger{} - err := dib.Retag(DAG, tagger) + err := dib.Retag(DAG, tagger, true) assert.NoError(t, err) require.Len(t, tagger.RecordedCallsArgs, 2) - assert.Equal(t, "registry.example.org/image:new", tagger.RecordedCallsArgs[0].Src) + assert.Equal(t, "registry.example.org/image:myhash", tagger.RecordedCallsArgs[0].Src) assert.Equal(t, "registry.example.org/image:latest1", tagger.RecordedCallsArgs[0].Dest) - assert.Equal(t, "registry.example.org/image:new", tagger.RecordedCallsArgs[1].Src) + assert.Equal(t, "registry.example.org/image:myhash", tagger.RecordedCallsArgs[1].Src) assert.Equal(t, "registry.example.org/image:latest2", tagger.RecordedCallsArgs[1].Dest) assert.True(t, img.RetagDone) diff --git a/dib/test.go b/dib/test.go index 27e1a09e..54e353d9 100644 --- a/dib/test.go +++ b/dib/test.go @@ -1,20 +1,19 @@ package dib import ( - "fmt" - "github.com/radiofrance/dib/dag" "github.com/radiofrance/dib/types" "github.com/sirupsen/logrus" ) // testImage runs the tests on an image. -func testImage(img *dag.Image, testRunners []types.TestRunner, newTag string) error { - logrus.Infof("Running tests for \"%s:%s\"", img.Name, newTag) +func testImage(img *dag.Image, testRunners []types.TestRunner) error { + ref := img.CurrentRef() + logrus.Infof("Running tests for \"%s\"", ref) opts := types.RunTestOptions{ ImageName: img.ShortName, - ImageReference: fmt.Sprintf("%s:%s", img.Name, newTag), + ImageReference: ref, DockerContextPath: img.Dockerfile.ContextPath, } for _, runner := range testRunners { diff --git a/docker/builder.go b/docker/builder.go index ff99fce2..72c1d5f3 100644 --- a/docker/builder.go +++ b/docker/builder.go @@ -61,14 +61,9 @@ func (b ImageBuilderTagger) Build(opts types.ImageBuilderOpts) error { // Tag runs a `docker tag`command to retag the source tag with the destination tag. func (b ImageBuilderTagger) Tag(src, dest string) error { if b.dryRun { - logrus.Infof("[DRY-RUN] docker pull %s", src) logrus.Infof("[DRY-RUN] docker tag %s %s", src, dest) return nil } - logrus.Debugf("Running `docker pull %s`", src) - if err := b.exec.ExecuteStdout("docker", "pull", src); err != nil { - return err - } logrus.Debugf("Running `docker tag %s %s`", src, dest) return b.exec.ExecuteStdout("docker", "tag", src, dest) } diff --git a/go.mod b/go.mod index bd06d49f..d2bc6160 100644 --- a/go.mod +++ b/go.mod @@ -8,7 +8,6 @@ require ( github.com/aws/aws-sdk-go-v2/service/s3 v1.13.0 github.com/docker/cli v20.10.16+incompatible github.com/docker/docker v20.10.16+incompatible - github.com/go-git/go-git/v5 v5.4.2 github.com/mholt/archiver/v3 v3.5.1 github.com/sirupsen/logrus v1.8.1 github.com/spf13/cobra v1.4.0 @@ -18,7 +17,6 @@ require ( github.com/wolfeidau/humanhash v1.1.0 gitlab.com/radiofrance/go-container-registry v0.1.2 gitlab.com/radiofrance/kubecli v0.2.3 - golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 golang.org/x/sync v0.0.0-20220513210516-0976fa681c29 golang.org/x/term v0.0.0-20220411215600-e5f449aeb171 k8s.io/api v0.24.0 @@ -34,10 +32,8 @@ require ( github.com/MakeNowJust/heredoc v0.0.0-20170808103936-bb23615498cd // indirect github.com/Microsoft/go-winio v0.5.2 // indirect github.com/Microsoft/hcsshim v0.8.23 // indirect - github.com/ProtonMail/go-crypto v0.0.0-20210428141323-04723f9f07d7 // indirect github.com/PuerkitoBio/purell v1.1.1 // indirect github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect - github.com/acomagu/bufpipe v1.0.3 // indirect github.com/andybalholm/brotli v1.0.3 // indirect github.com/aws/aws-sdk-go-v2/credentials v1.8.0 // indirect github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.10.0 // indirect @@ -59,15 +55,12 @@ require ( github.com/docker/go-units v0.4.0 // indirect github.com/dsnet/compress v0.0.2-0.20210315054119-f66993602bf5 // indirect github.com/emicklei/go-restful v2.9.5+incompatible // indirect - github.com/emirpasic/gods v1.12.0 // indirect github.com/evanphx/json-patch v4.12.0+incompatible // indirect github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d // indirect github.com/fatih/camelcase v1.0.0 // indirect github.com/fsnotify/fsnotify v1.5.1 // indirect github.com/fvbommel/sortorder v1.0.1 // indirect github.com/go-errors/errors v1.0.1 // indirect - github.com/go-git/gcfg v1.5.0 // indirect - github.com/go-git/go-billy/v5 v5.3.1 // indirect github.com/go-logr/logr v1.2.2 // indirect github.com/go-openapi/jsonpointer v0.19.5 // indirect github.com/go-openapi/jsonreference v0.19.5 // indirect @@ -87,10 +80,8 @@ require ( github.com/hashicorp/hcl v1.0.0 // indirect github.com/imdario/mergo v0.3.12 // indirect github.com/inconshreveable/mousetrap v1.0.0 // indirect - github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect github.com/josharian/intern v1.0.0 // indirect github.com/json-iterator/go v1.1.12 // indirect - github.com/kevinburke/ssh_config v0.0.0-20201106050909-4977a11b4351 // indirect github.com/klauspost/compress v1.15.4 // indirect github.com/klauspost/pgzip v1.2.5 // indirect github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect @@ -121,18 +112,15 @@ require ( github.com/pkg/errors v0.9.1 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect github.com/russross/blackfriday v1.6.0 // indirect - github.com/sergi/go-diff v1.1.0 // indirect github.com/spf13/afero v1.8.2 // indirect github.com/spf13/cast v1.4.1 // indirect github.com/spf13/jwalterweatherman v1.1.0 // indirect github.com/subosito/gotenv v1.2.0 // indirect github.com/ulikunitz/xz v0.5.9 // indirect - github.com/xanzy/ssh-agent v0.3.0 // indirect github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca // indirect go.opencensus.io v0.23.0 // indirect go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5 // indirect - golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4 // indirect golang.org/x/net v0.0.0-20220516155154-20f960328961 // indirect golang.org/x/oauth2 v0.0.0-20220411215720-9780585627b5 // indirect golang.org/x/sys v0.0.0-20220513210249-45d2b4557a2a // indirect @@ -142,7 +130,6 @@ require ( google.golang.org/protobuf v1.28.0 // indirect gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/ini.v1 v1.66.4 // indirect - gopkg.in/warnings.v0 v0.1.2 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect k8s.io/component-base v0.24.0 // indirect diff --git a/go.sum b/go.sum index df4b328c..07635947 100644 --- a/go.sum +++ b/go.sum @@ -168,8 +168,6 @@ github.com/Microsoft/hcsshim/test v0.0.0-20210227013316-43a75bb4edd3/go.mod h1:m github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ= github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= github.com/OpenPeeDeeP/depguard v1.0.1/go.mod h1:xsIw86fROiiwelg+jB2uM9PiKihMMmUx/1V+TNhjQvM= -github.com/ProtonMail/go-crypto v0.0.0-20210428141323-04723f9f07d7 h1:YoJbenK9C67SkzkDfmQuVln04ygHj3vjZfd9FL+GmQQ= -github.com/ProtonMail/go-crypto v0.0.0-20210428141323-04723f9f07d7/go.mod h1:z4/9nQmJSSwwds7ejkxaJwO37dru3geImFUdJlaLzQo= github.com/PuerkitoBio/purell v1.0.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= github.com/PuerkitoBio/purell v1.1.1 h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tNFfI= github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= @@ -181,8 +179,6 @@ github.com/Shopify/sarama v1.19.0/go.mod h1:FVkBWblsNy7DGZRfXLU0O9RCGt5g3g3yEuWX github.com/Shopify/toxiproxy v2.1.4+incompatible/go.mod h1:OXgGpZ6Cli1/URJOF1DMxUHB2q5Ap20/P/eIdh4G0pI= github.com/StackExchange/wmi v0.0.0-20180116203802-5d049714c4a6/go.mod h1:3eOhrUMpNV+6aFIbp5/iudMxNCF27Vw2OZgy4xEx0Fg= github.com/StackExchange/wmi v1.2.1/go.mod h1:rcmrprowKIVzvc+NUiLncP2uuArMWLCbu9SBzvHz7e8= -github.com/acomagu/bufpipe v1.0.3 h1:fxAGrHZTgQ9w5QqVItgzwj235/uYZYgbXitB+dLupOk= -github.com/acomagu/bufpipe v1.0.3/go.mod h1:mxdxdup/WdsKVreO5GpW4+M/1CE2sMG4jeGJ2sYmHc4= github.com/agext/levenshtein v1.2.3/go.mod h1:JEDfjyjHDjOF/1e4FlBE/PkbqA9OfWu2ki2W0IB5558= github.com/alecthomas/kingpin v2.2.6+incompatible/go.mod h1:59OFYbFVLKQKq+mqrL6Rw5bR0c3ACQaawgXx0QYndlE= github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= @@ -196,7 +192,6 @@ github.com/andybalholm/brotli v1.0.1/go.mod h1:loMXtMfwqflxFJPmdbJO0a3KNoPuLBgiu github.com/andybalholm/brotli v1.0.2/go.mod h1:loMXtMfwqflxFJPmdbJO0a3KNoPuLBgiu3qAvBg8x/Y= github.com/andybalholm/brotli v1.0.3 h1:fpcw+r1N1h0Poc1F/pHbW40cUm/lMEQslZtCkBQ0UnM= github.com/andybalholm/brotli v1.0.3/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig= -github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239 h1:kFOfPq6dUM1hTo4JG6LR5AXSUEsOjtdm0kw0FtQtMJA= github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239/go.mod h1:2FmKhYUyUczH0OGQWaF5ceTx0UBShxjsH6f8oGKYe2c= github.com/antihax/optional v0.0.0-20180407024304-ca021399b1a6/go.mod h1:V8iCPQYkqmusNa815XgQio277wI47sdRh1dUOLdyC6Q= github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY= @@ -525,8 +520,6 @@ github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkg github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= github.com/emicklei/go-restful v2.9.5+incompatible h1:spTtZBk5DYEvbxMVutUuTyh1Ao2r4iyvLdACqsl/Ljk= github.com/emicklei/go-restful v2.9.5+incompatible/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= -github.com/emirpasic/gods v1.12.0 h1:QAUIPSaCu4G+POclxeqb3F+WPpdKqFGlw36+yOzGlrg= -github.com/emirpasic/gods v1.12.0/go.mod h1:YfzfFFoVP/catgzJb4IKIqXjX78Ha8FMSDh3ymbK86o= github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98= @@ -574,22 +567,11 @@ github.com/getkin/kin-openapi v0.76.0/go.mod h1:660oXbgy5JFMKreazJaQTw7o+X00qeSy github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= github.com/gliderlabs/ssh v0.1.1/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0= -github.com/gliderlabs/ssh v0.2.2 h1:6zsha5zo/TWhRhwqCD3+EarCAgZ2yN28ipRnGPnwkI0= -github.com/gliderlabs/ssh v0.2.2/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0= github.com/go-critic/go-critic v0.4.1/go.mod h1:7/14rZGnZbY6E38VEGk2kVhoq6itzc1E68facVDK23g= github.com/go-critic/go-critic v0.4.3/go.mod h1:j4O3D4RoIwRqlZw5jJpx0BNfXWWbpcJoKu5cYSe4YmQ= github.com/go-critic/go-critic v0.6.1/go.mod h1:SdNCfU0yF3UBjtaZGw6586/WocupMOJuiqgom5DsQxM= github.com/go-errors/errors v1.0.1 h1:LUHzmkK3GUKUrL/1gfBUxAHzcev3apQlezX/+O7ma6w= github.com/go-errors/errors v1.0.1/go.mod h1:f4zRHt4oKfwPJE5k8C9vpYG+aDHdBFUsgrm6/TyX73Q= -github.com/go-git/gcfg v1.5.0 h1:Q5ViNfGF8zFgyJWPqYwA7qGFoMTEiBmdlkcfRmpIMa4= -github.com/go-git/gcfg v1.5.0/go.mod h1:5m20vg6GwYabIxaOonVkTdrILxQMpEShl1xiMF4ua+E= -github.com/go-git/go-billy/v5 v5.2.0/go.mod h1:pmpqyWchKfYfrkb/UVH4otLvyi/5gJlGI4Hb3ZqZ3W0= -github.com/go-git/go-billy/v5 v5.3.1 h1:CPiOUAzKtMRvolEKw+bG1PLRpT7D3LIs3/3ey4Aiu34= -github.com/go-git/go-billy/v5 v5.3.1/go.mod h1:pmpqyWchKfYfrkb/UVH4otLvyi/5gJlGI4Hb3ZqZ3W0= -github.com/go-git/go-git-fixtures/v4 v4.2.1 h1:n9gGL1Ct/yIw+nfsfr8s4+sbhT+Ncu2SubfXjIWgci8= -github.com/go-git/go-git-fixtures/v4 v4.2.1/go.mod h1:K8zd3kDUAykwTdDCr+I0per6Y6vMiRR/nnVTBtavnB0= -github.com/go-git/go-git/v5 v5.4.2 h1:BXyZu9t0VkbiHtqrsvdq39UDhGJTl1h55VW6CSC4aY4= -github.com/go-git/go-git/v5 v5.4.2/go.mod h1:gQ1kArt6d+n+BGd+/B/I74HwRTLhth2+zti4ihgckDc= github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= @@ -935,10 +917,7 @@ github.com/ishidawataru/sctp v0.0.0-20210226210310-f2269e66cdee/go.mod h1:co9pwD github.com/j-keck/arping v0.0.0-20160618110441-2cf9dc699c56/go.mod h1:ymszkNOg6tORTn+6F6j+Jc8TOr5osrynvN6ivFWZ2GA= github.com/jaguilar/vt100 v0.0.0-20150826170717-2703a27b14ea/go.mod h1:QMdK4dGB3YhEW2BmA1wgGpPYI3HZy/5gD705PXKUVSg= github.com/jarcoal/httpmock v1.0.5/go.mod h1:ATjnClrvW/3tijVmpL/va5Z3aAyGvqU3gCT8nX0Txik= -github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOlocH6Fxy8MmwDt+yVQYULKfN0RoTN8A= -github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo= github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1/go.mod h1:E0B/fFc00Y+Rasa88328GlI/XbtyysCtTHZS8h7IrBU= -github.com/jessevdk/go-flags v1.5.0/go.mod h1:Fw0T6WPc1dYxT4mKEZRfG5kJhaTDP9pj1c2EWnYs/m4= github.com/jgautheron/goconst v1.5.1/go.mod h1:aAosetZ5zaeC/2EfMeRswtxUFBpe2Hr7HzkgX4fanO4= github.com/jhump/protoreflect v1.6.1/go.mod h1:RZQ/lnuN+zqeRVpQigTwO6o0AJUkxbnSnpuG7toUTG4= github.com/jingyugao/rowserrcheck v0.0.0-20191204022205-72ab7603b68a/go.mod h1:xRskid8CManxVta/ALEhJha/pweKBaVG6fWgc0yH25s= @@ -980,8 +959,6 @@ github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7V github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM= github.com/julz/importas v0.0.0-20210419104244-841f0c0fe66d/go.mod h1:oSFU2R4XK/P7kNBrnL/FEQlDGN1/6WoxXEjSSXO0DV0= github.com/k0kubun/colorstring v0.0.0-20150214042306-9440f1994b88/go.mod h1:3w7q1U84EfirKl04SVQ/s7nPm1ZPhiXd34z40TNz36k= -github.com/kevinburke/ssh_config v0.0.0-20201106050909-4977a11b4351 h1:DowS9hvgyYSX4TO5NpyC606/Z4SxnNYbT+WX27or6Ck= -github.com/kevinburke/ssh_config v0.0.0-20201106050909-4977a11b4351/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF4nAY/ojJ6r6mM= github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00= github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= @@ -1051,8 +1028,6 @@ github.com/maratori/testpackage v1.0.1/go.mod h1:ddKdw+XG0Phzhx8BFDTKgpWP4i7MpAp github.com/marstr/guid v1.1.0/go.mod h1:74gB1z2wpxxInTG6yaqA7KrtM0NZ+RbrcqDvYHefzho= github.com/matoous/godox v0.0.0-20190911065817-5d6d842e92eb/go.mod h1:1BELzlh859Sh1c6+90blK8lbYy0kwQf1bYlBhBysy1s= github.com/matoous/godox v0.0.0-20210227103229-6504466cf951/go.mod h1:1BELzlh859Sh1c6+90blK8lbYy0kwQf1bYlBhBysy1s= -github.com/matryer/is v1.2.0/go.mod h1:2fLPjFQM9rhQ15aVEtbuwhJinnOqrmgXPNdZsdwlWXA= -github.com/matryer/is v1.4.0 h1:sosSmIWwkYITGrxZ25ULNDeKiMNzFSr4V/eqBQP0PeE= github.com/matryer/is v1.4.0/go.mod h1:8I/i5uYgLzgsgEloJE1U6xx5HkBQpAZvepWuujKwMRU= github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= @@ -1539,8 +1514,6 @@ github.com/wolfeidau/humanhash v1.1.0 h1:06KgtyyABJGBbrfMONrW7S+b5TTYVyrNB/jss5n github.com/wolfeidau/humanhash v1.1.0/go.mod h1:jkpynR1bfyfkmKEQudIC0osWKynFAoayRjzH9OJdVIg= github.com/xanzy/go-gitlab v0.31.0/go.mod h1:sPLojNBn68fMUWSxIJtdVVIP8uSBYqesTfDUseX11Ug= github.com/xanzy/go-gitlab v0.32.0/go.mod h1:sPLojNBn68fMUWSxIJtdVVIP8uSBYqesTfDUseX11Ug= -github.com/xanzy/ssh-agent v0.3.0 h1:wUMzuKtKilRgBAD1sUb8gOwwRr2FGoBVumcjoOACClI= -github.com/xanzy/ssh-agent v0.3.0/go.mod h1:3s9xbODqPuuhK9JV1R321M/FlMZSBvE5aY6eAcqrDh0= github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU= github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415/go.mod h1:GwrjFmJcFw6At/Gs6z4yjiIwzuJ1/+UwLxMQDVQXShQ= github.com/xeipuuv/gojsonschema v0.0.0-20180618132009-1d523034197f/go.mod h1:5yf86TLmAcydyeJq5YvxkGPE2fm/u4myDekKRoLuqhs= @@ -1647,7 +1620,6 @@ golang.org/x/crypto v0.0.0-20181029021203-45a5f77698d3/go.mod h1:6SG95UA2DQfeDnf golang.org/x/crypto v0.0.0-20181030102418-4d3f4d9ffa16/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= -golang.org/x/crypto v0.0.0-20190219172222-a4c6cb3142f2/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= @@ -1672,8 +1644,6 @@ golang.org/x/crypto v0.0.0-20210817164053-32db794688a5/go.mod h1:GvvjBRRGRdwPK5y golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20220214200702-86341886e292/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= -golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4 h1:kUhD7nTDoI3fVd9G4ORWrbV5NY0liEs/Jg2pv5f+bBA= -golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190125153040-c74c464bbbf2/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= @@ -1715,7 +1685,6 @@ golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.5.0/go.mod h1:5OXOZSfqPIIbmVBIIKWRFfZjPR0E5r58TLhUjH0a2Ro= -golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 h1:kQgndtyPBW/JIYERgdxfwMYh3AVStj88WQTlNDi2a+o= golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3/go.mod h1:3p9vT2HGsQu2K1YbXdKPJLVgG5VJdoTa1poYQBtP1AY= golang.org/x/net v0.0.0-20170114055629-f2499483f923/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -1775,7 +1744,6 @@ golang.org/x/net v0.0.0-20201224014010-6772e930b67b/go.mod h1:m0MpNAwzfU5UDzcl9v golang.org/x/net v0.0.0-20210119194325-5f4716e94777/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20210316092652-d523dce5a7f4/go.mod h1:RBQZq4jEuRlivfhVLdyRGr576XBO4/greRjx4P4O3yc= -golang.org/x/net v0.0.0-20210326060303-6b1517762897/go.mod h1:uSPa2vr4CLtc/ILN5odXGNXS6mhrKVzTaCXzk9m6W3k= golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk= golang.org/x/net v0.0.0-20210503060351-7fd8e65b6420/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= @@ -1929,7 +1897,6 @@ golang.org/x/sys v0.0.0-20210403161142-5e06dd20ab57/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210426230700-d19ff857e887/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210502180810-71e4cd670f79/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= @@ -2285,7 +2252,6 @@ gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= -gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= gopkg.in/cheggaaa/pb.v1 v1.0.25/go.mod h1:V/YB90LKu/1FcN3WVnfiiE5oMCibMjukxqG/qStrOgw= gopkg.in/cheggaaa/pb.v1 v1.0.28/go.mod h1:V/YB90LKu/1FcN3WVnfiiE5oMCibMjukxqG/qStrOgw= gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= @@ -2309,7 +2275,6 @@ gopkg.in/square/go-jose.v2 v2.5.1/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76 gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= gopkg.in/warnings.v0 v0.1.1/go.mod h1:jksf8JmL6Qr/oQM2OXTHunEvvTAsrWBLb6OOjuVWRNI= -gopkg.in/warnings.v0 v0.1.2 h1:wFXVbFY8DY5/xOe1ECiWdKCzZlxgshcYVNkBHstARME= gopkg.in/warnings.v0 v0.1.2/go.mod h1:jksf8JmL6Qr/oQM2OXTHunEvvTAsrWBLb6OOjuVWRNI= gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74= gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= diff --git a/graphviz/graphviz.go b/graphviz/graphviz.go index 10ff4431..88b9e649 100644 --- a/graphviz/graphviz.go +++ b/graphviz/graphviz.go @@ -63,13 +63,8 @@ func GenerateDotviz(graph *dag.DAG, output string) error { func generateDotvizImg(node *dag.Node, writer *bufio.Writer) error { img := node.Image color := "white" - switch { - case img.NeedsRebuild && img.NeedsRetag: - return fmt.Errorf("image %s has both RebuildDone and RetagDone", img.Name) - case img.NeedsRebuild: + if img.NeedsRebuild { color = "red" - case img.NeedsRetag: - color = "yellow" } if _, err := writer.WriteString(fmt.Sprintf("\"%s\" [fillcolor=%s style=filled];\n", img.Name, color)); err != nil { diff --git a/init/Dockerfile b/init/Dockerfile deleted file mode 100644 index cc64fe81..00000000 --- a/init/Dockerfile +++ /dev/null @@ -1,11 +0,0 @@ -# This Dockerfile is only used to init your dib managed repository -# -# Within this repository, run the following commands -# -# $ docker build -t /dib-referential . -# $ docker push /dib-referential -# -# If you choose to use another name than `dib-referential`, you will need -# to set it in your command-line, with `--referential-image` -FROM alpine -LABEL name="dib-referential" diff --git a/version/git.go b/version/git.go deleted file mode 100644 index 9900d508..00000000 --- a/version/git.go +++ /dev/null @@ -1,132 +0,0 @@ -package version - -import ( - "errors" - "fmt" - "path" - "strings" - - "github.com/go-git/go-git/v5/plumbing/object" - - "github.com/radiofrance/dib/types" - - "github.com/go-git/go-git/v5/plumbing" - - "github.com/go-git/go-git/v5" - "github.com/radiofrance/dib/exec" -) - -var ErrNoPreviousBuild = errors.New("no previous build was found in git history") - -// GetDiffSinceLastDockerVersionChange computes the diff since last version. -// The last version is the git revision hash of the .docker-version file. -// It returns the hash of the compared revision, and the list of modified files. -func GetDiffSinceLastDockerVersionChange(repositoryPath string, exec exec.Executor, - registry types.DockerRegistry, dockerVersionFile, referentialImage string, -) (string, []string, error) { - repo, err := git.PlainOpen(repositoryPath) - if err != nil { - return "", nil, err - } - - lastChangedDockerVersionHash, err := getLastChangedDockerVersion(repo, registry, dockerVersionFile, referentialImage) - if err != nil { - return "", nil, err - } - - head, err := repo.Head() - if err != nil { - return "", nil, err - } - - if lastChangedDockerVersionHash == plumbing.ZeroHash { - // No previous build was found - return "", nil, nil - } - - // We need to run git diff in both direction (CommitA..CommitB & CommitB..CommitA) to identify - // files that have been renamed, and to keep both old name and new name - versions := fmt.Sprintf("%s..%s", head.Hash().String(), lastChangedDockerVersionHash.String()) - out, err := exec.Execute("git", "diff", versions, "--name-only") - if err != nil { - return "", nil, err - } - diffs := strings.Split(strings.TrimSuffix(out, "\n"), "\n") - - versions = fmt.Sprintf("%s..%s", lastChangedDockerVersionHash.String(), head.Hash().String()) - out, err = exec.Execute("git", "diff", versions, "--name-only") - if err != nil { - return "", nil, err - } - diffs = append(diffs, strings.Split(strings.TrimSuffix(out, "\n"), "\n")...) - - var fullPathDiffs []string - for _, filename := range diffs { - fullPathDiffs = append(fullPathDiffs, path.Join(repositoryPath, filename)) - } - - dockerVersionContent, err := getDockerVersionContentForHash(repo, lastChangedDockerVersionHash, dockerVersionFile) - if err != nil { - return "", nil, fmt.Errorf("failed to get %s content for hash %s", - dockerVersionFile, lastChangedDockerVersionHash.String()) - } - - return strings.TrimSuffix(dockerVersionContent, "\n"), fullPathDiffs, nil -} - -func getDockerVersionContentForHash(repo *git.Repository, lastChangedHash plumbing.Hash, dockerVersionFile string, -) (string, error) { - commitObject, err := repo.CommitObject(lastChangedHash) - if err != nil { - return "", err - } - tree, err := commitObject.Tree() - if err != nil { - return "", err - } - file, err := tree.File(dockerVersionFile) - if err != nil { - return "", err - } - content, err := file.Contents() - if err != nil { - return "", err - } - return content, nil -} - -func getLastChangedDockerVersion(repository *git.Repository, registry types.DockerRegistry, - dockerVersionFile, referentialImage string, -) (plumbing.Hash, error) { - commitLog, err := repository.Log(&git.LogOptions{ - PathFilter: func(p string) bool { - return p == dockerVersionFile - }, - }) - if err != nil { - return plumbing.Hash{}, err - } - - for { - commit, err := commitLog.Next() - if err != nil { - return plumbing.Hash{}, err - } - commitID := commit.ID() - dockerVersionContent, err := getDockerVersionContentForHash(repository, commitID, dockerVersionFile) - if err != nil { - if errors.Is(err, object.ErrFileNotFound) { - // We consider we went as far as we could to find a matching commit that was already built - return plumbing.Hash{}, ErrNoPreviousBuild - } - return plumbing.Hash{}, fmt.Errorf("failed to get %s content for hash %s", dockerVersionFile, commitID) - } - refExists, err := registry.RefExists(fmt.Sprintf("%s:%s", referentialImage, dockerVersionContent)) - if err != nil { - return plumbing.Hash{}, err - } - if refExists { - return commitID, nil - } - } -} diff --git a/version/hash.go b/version/hash.go deleted file mode 100644 index 1e5e7092..00000000 --- a/version/hash.go +++ /dev/null @@ -1,54 +0,0 @@ -package version - -import ( - "crypto/sha256" - "errors" - "fmt" - "io" - "path" - "sort" - "strings" - - "github.com/wolfeidau/humanhash" - "golang.org/x/mod/sumdb/dirhash" -) - -const dockerIgnoreFileName = ".dockerignore" - -// GetDockerVersionHash returns the revision hash of the build directory. -func GetDockerVersionHash(buildPath string) (string, error) { - return dirhash.HashDir(buildPath, "", humanReadableHashFn) -} - -func humanReadableHashFn(files []string, open func(string) (io.ReadCloser, error)) (string, error) { - hash := sha256.New() - files = append([]string(nil), files...) - sort.Strings(files) - for _, file := range files { - if strings.Contains(file, "\n") { - return "", errors.New("dirhash: filenames with newlines are not supported") - } - if file == DockerVersionFilename || path.Base(file) == dockerIgnoreFileName { - // During the hash process, we ignore - // - the hash file itself - // .dockerignore files - continue - } - readCloser, err := open(file) - if err != nil { - return "", err - } - hashFile := sha256.New() - _, err = io.Copy(hashFile, readCloser) - readCloser.Close() - if err != nil { - return "", err - } - fmt.Fprintf(hash, "%x %s\n", hashFile.Sum(nil), file) - } - humanReadableHash, err := humanhash.Humanize(hash.Sum(nil), 4) - if err != nil { - return "", fmt.Errorf("could not humanize hash: %w", err) - } - return humanReadableHash, nil -} diff --git a/version/version.go b/version/version.go deleted file mode 100644 index 4c5bd7ff..00000000 --- a/version/version.go +++ /dev/null @@ -1,42 +0,0 @@ -package version - -import ( - "fmt" - "io/ioutil" - "path" - "strings" -) - -const DockerVersionFilename = ".docker-version" - -// CheckDockerVersionIntegrity verifies the consistency of the hash contained in the -// .docker-version file against the hash computed from the current filesystem state. -// It returns the version if the verification is successful. -func CheckDockerVersionIntegrity(buildPath string) (string, error) { - fileVersion, err := getDockerVersionFromFile(path.Join(buildPath, DockerVersionFilename)) - if err != nil { - return "", err - } - - dockerVersionHash, err := GetDockerVersionHash(buildPath) - if err != nil { - return "", fmt.Errorf("could not obtain docker-version hash: %w", err) - } - - if fileVersion != dockerVersionHash { - return "", fmt.Errorf( - "inconsistency between the content of .docker-version and the result of the hash command", - ) - } - - return dockerVersionHash, nil -} - -// getDockerVersionFromFile reads the current docker version contained in a file. -func getDockerVersionFromFile(filename string) (string, error) { - data, err := ioutil.ReadFile(filename) - if err != nil { - return "", err - } - return strings.TrimSuffix(string(data), "\n"), nil -}