Support a caching mode where we always append to the existing caches

[mcassaniti]

I'd like to see yet another way to cache builds using mkosi. Let's assume we're building weekly using the same configuration so package updates can be included in the final image. Most of the work the package manager will do for fetching files or performing package installation will be redone over and over without any difference between runs. Any skeleton or extra directories will remain unchanged too.

I'm proposing that a new caching mechanism be available that caches the working directory. This does come with the caveat that the user is responsible for dealing with any non-idempotent actions. For example, if a package is no longer specified for install they will need to either destroy the cache or add a post-install script to check for and remove the package. The same goes for mkosi.extra and mkosi.skeleton directories.

The benefit is a significant speed-up in image rebuild times, despite a user needing to make sure they manage non-idempotent actions.

As an aside, should discussions be enabled for this repository?

[DaanDeMeyer]

Issues are fine for the scale we're at.

I have had the same idea and I agree, we should have an incremental mode that behaves like you describe.

[mcassaniti]

If I take a look at the code now it will cache until about where the postinst script runs in the pipeline for the final image. Maybe this is enough?

[DaanDeMeyer]

What's missing is that when we invalidate the cache because we add a package or such, we throw away the entire cache and rebuild it from scratch. I figured this was about not throwing away the cache in that case and just installing the extra packages on top of the existing cache.

[mcassaniti]

That would be good, along with some other pieces I'll write up later.

[mcassaniti]

I've raised #1808 to get this started. Once that is merged in whatever form it takes I'll add another caching option.

[DaanDeMeyer]

Let's merge this into #1756