Enable more and less error prone caching

[DaanDeMeyer]

Currently, --incremental has some very basic cache invalidation based on whether the list of configured packages changed or not. This is not sufficient for multiple reasons:

• There's a lot more options that affect the cached images outside of which packages we install
• For BaseTrees=, SkeletonTrees= and PackageManagerTrees=, it's not sufficient to just check whether the option changed, we need to check whether the directory tree or file pointed to by the option changed at all. For directory trees, we have to check every file in the tree for changes
• We currently check whether we can use the cached image or not, we should also be able to check whether we need to rebuild the image at all, this means checking all config options, all configured source files and all input trees for changes and making a cache manifest out of those.

For checking for tree changes, we can probably use systemd-dissect --mtree to and diff the mtree output to see what changed. We should also make sure the diffs between caches are displayed in --debug mode to allow debugging why we don't reuse the cache and rebuild the image.

We should also support appending existing cache images where we install more packages into the existing cached image instead of rebuilding it from scratch.

[septatrix]

• There's a lot more options that affect the cached images outside of which packages we install

Do you already have a list of those options or should we collect them here? One I am currently stumbling over is changes in RootSize= (yes, I know, not present in v15 anymore)

‣  Refreshing partition table…
Partition #1 contains a vfat signature.
The last usable GPT sector is 6815790, but 8912935 is requested.
Failed to add #2 partition: Invalid argument

[malt3]

I noticed that presets in mkosi are roughly analogous to layers in a container image.
This makes me think that mkosi could have incremental builds on a preset level.
Let me explain the idea (in vague terms) before talking about how this could be implemented:

Idea

Let's say I have an multi-preset build like the one used by systemd:

• base holds packages, builds software and outputs a sysroot as a directory
• initrd generates a cpio used as initrd
• system depends on base, initrd and some extra trees (let's say a few tar files with software built elsewhere that can be used as extra trees) and generates a bootable uefi image with a uki

Now I perform one build.
If only one of the tar files for the system preset has changed, I want to be able to rebuild incrementally (keep base and initrd from the last build and only rebuild system).

Possible implementations

Let the user choose what needs to be rebuilt via command line parameters

This requires only a tiny code change but probably works well if used with existing build systems. The build system invoking mkosi would know what external dependencies a preset depends on and can tell mkosi what needs to be rebuilt.
All that is needed would be flags to either keep existing presets or to explicitly rebuild a list of presets. Mkosi should then rebuild all chosen presets and all of the presets that depend on the chosen presets.

Actually check all inputs for a preset and understand what has changed

This could be a lot of work in practice. Mkosi would essentially need to record all inputs for a preset (Trees, resolved packages, the exact configuration, probably more) and decide itself if a preset needs to be rebuilt. It would the need to rebuild the changed presets and all of the presets that depend on changed presets transitively.

Possible alternatives

You can define layers by having separate folders with their own mkosi config and let them depend on each other (by referencing the outputs from one directory in the basetree of another directory).
In this scenario, mkosi cannot by itself orchestrate the whole build and another build system needs to know that layers depend on each other and build layers in the correct order.

---

I think this would make mkosi an optimal build tool for systemd sysext and full os images.
It would provide a similar level of convenience as a Dockerfiles do for building containers. I also think implementing only the first option for now would already provide a lot of value.

[DaanDeMeyer]

I completely agree, but we should do this properly by checking all the inputs. We should also make this easy to debug by providing an informative diff if requested of what's causing a rebuild to happen. We can either use systemd-dissect --mtree or diffoscope to figure out differences. Would love to review PRs for this, but we shouldn't half bake this but do it properly.