Add support for advanced partitioning customizations (COMPOSER-2399) #4535

achilleas-k · 2024-12-18T13:06:50Z

This PR enables the disk customizations in osbuild-composer.

Tests for the new blueprint configurations have been added to filesystem.sh.

This pull request includes:

adequate testing for the new functionality or fixed issue
adequate documentation informing people about the change such as
- submit a PR for the READMEs listed here
- submit a PR for the osbuild.org website repository if this PR changed any behavior not covered by the automatically updated READMEs

mvo5

Thanks for this PR, I have a very silly question inline, it comes down to if we can find a way to avoid some of the duplication with the images code that are currently present in this PR.

internal/blueprint/customizations.go

internal/blueprint/disk_customizations.go

achilleas-k · 2024-12-19T13:33:47Z

I need to update image-info to handle btrfs subvolumes and ignore swap partitions when mounting filesystems.

thozza · 2025-01-02T05:46:14Z

I need to update image-info to handle btrfs subvolumes and ignore swap partitions when mounting filesystems.

Let's try to standardize on the osbuild-image-info and remove the image-info from this repository.

test/cases/filesystem.sh

.gitlab-ci.yml

achilleas-k · 2025-01-02T18:04:12Z

I need to update image-info to handle btrfs subvolumes and ignore swap partitions when mounting filesystems.

Let's try to standardize on the osbuild-image-info and remove the image-info from this repository.

#4543

achilleas-k · 2025-01-29T17:28:51Z

With #4568 merged, this should be good to go now.

I'll open a new PR later this week for the blueprint (de)duplication and we can discuss the approach there.

EDIT: I forgot, we also need #4569 for the newer snapshots.

achilleas-k · 2025-01-30T10:56:22Z

Rebased (onto #4543) and fixed conflicts.

achilleas-k · 2025-02-03T17:38:46Z

Rebased on @thozza's #4592, which updates the osbuild dependency but also fixes the spec file. Thanks, I missed that.

achilleas-k · 2025-02-04T10:33:30Z

https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/pipelines/1653730001
Three failures:

2x repository issues:
- RHEL 9.6 nightly: https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/jobs/9034350641#L2125
- RHEL 10.0 nightly: https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/jobs/9034350701#L2128
1x stupid typo: https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/jobs/9034350685#L2736

Add DiskCustomization and all its children to the internal blueprint. Add the conversion to the images counterpart to the Convert() function.

Move the creation of the blueprint to a function. We'll then add a separate blueprint function for the new, alternative partitioning customization.

Version 94 of osbuild-composer is over a year old now. It should be fine to test these unconditionally.

Add separate blueprint creation functions, one for each partitioning layout: - disk-plain - disk-lvm - disk-btrfs (Fedora only) The existing 'filesystem' blueprint is also kept. Each function also sets the $EXPECTED_MOUNTPOINTS variable for the mountpoint check that happens after the build.

Remove incorrect comment (not an ostree image). Add a greenprint for depsolving since it can take some time for that to run and it makes more sense than pausing output on "Preparing blueprint".

thozza

I've added one comment, but won't block on it...

internal/blueprint/disk_customizations.go

Run filesystem.sh with all the valid arguments for the customization type.

Can be useful for troubleshooting and tracing test logs.

achilleas-k · 2025-02-04T13:12:35Z

The repository issues were probably because I didn't mark any runners with the internal network variable in .gitlab-ci.yml. I also noticed that not all runner configurations were showing up in the actual pipeline run, but I think I got that sorted now too.

thozza

🤞

achilleas-k · 2025-02-04T14:29:38Z

Filesystem tests are now passing. But I'm still seeing fewer jobs than the ones I listed in the CI config:

filesystem.sh:
  extends: .integration
  parallel:
    matrix:
      - CUSTOMIZATION_TYPE: "disk-btrfs"
        RUNNER:
          - aws/fedora-40-x86_64
      - CUSTOMIZATION_TYPE: "filesystem"
        RUNNER:
          - aws/rhel-8.10-ga-x86_64
        INTERNAL_NETWORK: ["true"]
      - CUSTOMIZATION_TYPE: "disk-plain"
        RUNNER:
          - aws/rhel-9.4-ga-x86_64
          - aws/rhel-10.0-nightly-x86_64
        INTERNAL_NETWORK: ["true"]
      - CUSTOMIZATION_TYPE: "disk-lvm"
        RUNNER:
          - aws/rhel-9.6-nightly-x86_64
        INTERNAL_NETWORK: ["true"]
      - CUSTOMIZATION_TYPE: "disk-lvm"
        RUNNER:
          - aws/centos-stream-9-x86_64
      - CUSTOMIZATION_TYPE: "filesystem"
        RUNNER:
          - aws/centos-stream-10-x86_64
  variables:
    SCRIPT: filesystem.sh ${CUSTOMIZATION_TYPE}

but in the pipeline I'm only seeing:

Fedora 40 with btrfs
CentOS 9 with lvm
RHEL 9.6 with lvm
RHEL 10.0 with plain
CentOS 10 with filesystem (old configs)

EDIT:
Seems the issue is that the step extends .integration, which extends .integration_base, which includes the .ga_rules_all rule:

.ga_rules_all:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/ && $NIGHTLY == "false"'

achilleas-k · 2025-02-04T16:13:32Z

I spoke with @ondrejbudai and we decided that the gitlab CI config could use a lot of simplification and also that we should merge this as is and sort out any other config combinations separately.

achilleas-k requested review from ondrejbudai, mvo5 and thozza December 18, 2024 13:06

achilleas-k changed the title ~~Add support for advanced partitioning customizations~~ Add support for advanced partitioning customizations (COMPOSER-2399) Dec 18, 2024

achilleas-k force-pushed the all-the-partitioning-all-the-time branch 2 times, most recently from df59325 to 423fb63 Compare December 18, 2024 18:08

mvo5 reviewed Dec 18, 2024

View reviewed changes

internal/blueprint/customizations.go Show resolved Hide resolved

internal/blueprint/disk_customizations.go Show resolved Hide resolved

thozza reviewed Jan 2, 2025

View reviewed changes

test/cases/filesystem.sh Show resolved Hide resolved

.gitlab-ci.yml Outdated Show resolved Hide resolved

achilleas-k force-pushed the all-the-partitioning-all-the-time branch 2 times, most recently from 083694d to 4527f9d Compare January 29, 2025 17:25

achilleas-k force-pushed the all-the-partitioning-all-the-time branch from 4527f9d to cbf8577 Compare January 30, 2025 10:55

achilleas-k force-pushed the all-the-partitioning-all-the-time branch 4 times, most recently from eb2e697 to d112ccb Compare February 3, 2025 17:38

achilleas-k added 4 commits February 4, 2025 11:34

blueprint: add local DiskCustomization

0bc8b47

Add DiskCustomization and all its children to the internal blueprint. Add the conversion to the images counterpart to the Convert() function.

blueprint: test the conversion

4cc313c

test/filesystem: move blueprint writing to function

7898477

Move the creation of the blueprint to a function. We'll then add a separate blueprint function for the new, alternative partitioning customization.

test/filesystem: remove conditions on v94

cc2e85b

Version 94 of osbuild-composer is over a year old now. It should be fine to test these unconditionally.

achilleas-k force-pushed the all-the-partitioning-all-the-time branch from d112ccb to 9e07001 Compare February 4, 2025 10:34

achilleas-k added 2 commits February 4, 2025 11:34

test/filesystem: minor cleanups

c59a019

Remove incorrect comment (not an ostree image). Add a greenprint for depsolving since it can take some time for that to run and it makes more sense than pausing output on "Preparing blueprint".

achilleas-k force-pushed the all-the-partitioning-all-the-time branch from 9e07001 to 57e4008 Compare February 4, 2025 10:34

thozza previously approved these changes Feb 4, 2025

View reviewed changes

internal/blueprint/disk_customizations.go Show resolved Hide resolved

achilleas-k added 2 commits February 4, 2025 14:10

gitlab: run all versions of filesystem.sh

5b5379e

Run filesystem.sh with all the valid arguments for the customization type.

test/filesystem: print blueprint contents when creating file

52e8d60

Can be useful for troubleshooting and tracing test logs.

achilleas-k dismissed thozza’s stale review via 52e8d60 February 4, 2025 13:10

achilleas-k force-pushed the all-the-partitioning-all-the-time branch from 57e4008 to 52e8d60 Compare February 4, 2025 13:10

thozza approved these changes Feb 4, 2025

View reviewed changes

achilleas-k enabled auto-merge (rebase) February 4, 2025 17:42

achilleas-k disabled auto-merge February 4, 2025 17:42

supakeen merged commit 448b1e2 into osbuild:main Feb 5, 2025
42 of 46 checks passed

achilleas-k deleted the all-the-partitioning-all-the-time branch February 5, 2025 10:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for advanced partitioning customizations (COMPOSER-2399) #4535

Add support for advanced partitioning customizations (COMPOSER-2399) #4535

achilleas-k commented Dec 18, 2024

mvo5 left a comment

achilleas-k commented Dec 19, 2024

thozza commented Jan 2, 2025

achilleas-k commented Jan 2, 2025

achilleas-k commented Jan 29, 2025 •

edited

Loading

achilleas-k commented Jan 30, 2025

achilleas-k commented Feb 3, 2025

achilleas-k commented Feb 4, 2025

thozza left a comment

achilleas-k commented Feb 4, 2025

thozza left a comment

achilleas-k commented Feb 4, 2025 •

edited

Loading

achilleas-k commented Feb 4, 2025

Add support for advanced partitioning customizations (COMPOSER-2399) #4535

Add support for advanced partitioning customizations (COMPOSER-2399) #4535

Conversation

achilleas-k commented Dec 18, 2024

mvo5 left a comment

Choose a reason for hiding this comment

achilleas-k commented Dec 19, 2024

thozza commented Jan 2, 2025

achilleas-k commented Jan 2, 2025

achilleas-k commented Jan 29, 2025 • edited Loading

achilleas-k commented Jan 30, 2025

achilleas-k commented Feb 3, 2025

achilleas-k commented Feb 4, 2025

thozza left a comment

Choose a reason for hiding this comment

achilleas-k commented Feb 4, 2025

thozza left a comment

Choose a reason for hiding this comment

achilleas-k commented Feb 4, 2025 • edited Loading

achilleas-k commented Feb 4, 2025

achilleas-k commented Jan 29, 2025 •

edited

Loading

achilleas-k commented Feb 4, 2025 •

edited

Loading