Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Openstack image manager doc update #450

Merged
merged 3 commits into from
Apr 26, 2024
Merged

Openstack image manager doc update #450

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6179730
updates flavor-manager.md and added update.md
Apr 23, 2024
210274e
update.md - add usage and output, example yaml file
Apr 24, 2024
66618df
update.md - changed some text and added links
Apr 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 18 additions & 16 deletions ...penstack/day2-operations/image-manager.md → ...ck/day2-operations/image-manager/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,10 +126,11 @@ the help of the OpenStack Image Manager.
provided_until: none
tags: []
versions:
- version: '0.6.0'
url: https://github.com/cirros-dev/cirros/releases/download/0.6.0/cirros-0.6.0-x86_64-disk.img
checksum: "sha256:94e1e2c94dbbae7d4bdc38e68590a1daf73c9de2d03dd693857b4b0a042548e8"
build_date: 2022-09-28
- version: '0.6.2'
url: https://github.com/cirros-dev/cirros/releases/download/0.6.2/cirros-0.6.2-x86_64-disk.img
checksum: "sha256:07e44a73e54c94d988028515403c1ed762055e01b83a767edf3c2b387f78ce00"
build_date: 2023-05-30

```

3. Run the OpenStack Image Manager. It is assumed that a profile with the name `openstack` exists in the
Expand Down Expand Up @@ -169,7 +170,7 @@ in parentheses upon each rotation (except for the latest entry).

```yaml
images:
- name: Ubuntu 16.04
- name: Ubuntu 24.04
format: qcow2
login: ubuntu
min_disk: 8
Expand All @@ -183,25 +184,26 @@ images:
hw_scsi_model: virtio-scsi
hw_watchdog_action: reset
os_distro: ubuntu
os_version: '16.04'
os_version: '24.04'
tags: []
versions:
- version: '20180928'
url: https://cloud-images.ubuntu.com/xenial/20180928/xenial-server-cloudimg-amd64-disk1.img
- version: '20181004'
url: https://cloud-images.ubuntu.com/xenial/20181004/xenial-server-cloudimg-amd64-disk1.img
- version: '20240416'
url: https://cloud-images.ubuntu.com/noble/20240416/noble-server-cloudimg-amd64.img
- version: '20240422'
url: https://cloud-images.ubuntu.com/noble/20240422/noble-server-cloudimg-amd64.img
```


This configuration creates the following images:

* **Ubuntu 16.04 (20180928)**
* **Ubuntu 16.04**
* **Ubuntu 24.04 (20240416)**
* **Ubuntu 24.04**

If a newer build is added, the following rotation takes place:

* **Ubuntu 16.04 (20180928)** does not change
* **Ubuntu 16.04** becomes **Ubuntu 16.04 (20181004)**
* the new image becomes **Ubuntu 16.04**
* **Ubuntu 24.04 (20240416)** does not change
* **Ubuntu 24.04** becomes **Ubuntu 24.04 (20240422)**
* the new image becomes **Ubuntu 24.04**

By default the last three images will be visible. When a fourth image is added, the visibility of
the last image in the list is changed to `community` and the image can be deleted in the future.
Expand Down Expand Up @@ -289,7 +291,7 @@ This makes us independent of the availability of the images in the individual up

### Updating images

Some of the images are automatically updated by a CI job. The latest available build at the time of the CI job execution is mirrored and
Some of the images are automatically updated by a [CI job](update). The latest available build at the time of the CI job execution is mirrored and
made available as the current version.

Currently, the following images are updated once a week (every Sunday at 0 am):
Expand Down
156 changes: 156 additions & 0 deletions docs/guides/operations-guide/openstack/day2-operations/image-manager/update.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
---
sidebar_label: Automated updates
sidebar_position: 1
---

# Image Manager update.py

## Overview

The OpenStack Image Manager `update.py` Script updates the `/etc/images/*.yaml` files to the always latest release of the Distributions, set S3 Mirror Urls and uploads the Images to the mirror.

These updated yaml files are later processed by the Image Manger itself.


## Installation

Prepare to use the `update.py` script.

```
git clone https://github.com/osism/openstack-image-manager/
cd openstack-image-manager
pipenv install
pipenv shell
```

## Usage

```
python contrib/update.py --help

Usage: update.py [OPTIONS]

╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --debug Enable debug logging │
│ --dry-run Do not perform any changes │
│ --minio-access-key TEXT Minio access key [env var: MINIO_ACCESS_KEY] [default: None] │
│ --minio-secret-key TEXT Minio secret key [env var: MINIO_SECRET_KEY] [default: None] │
│ --minio-server TEXT Minio server [env var: MINIO_SERVER] [default: swift.services.a.regiocloud.tech] │
│ --minio-bucket TEXT Minio bucket [env var: MINIO_BUCKET] [default: openstack-images] │
│ --swift-prefix TEXT Swift prefix [env var: SWIFT_PREFIX] [default: swift/v1/AUTH_b182637428444b9aa302bb8d5a5a418c/] │
│ --install-completion Install completion for the current shell. │
│ --show-completion Show completion for the current shell, to copy it or customize the installation. │
│ --help Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```

:::note

* At this time the update.py expects all yamls Files at /etc/images/, which can't be configured at the moment.
* Mirroring can't be disabled at the moment.

:::


Best is to run this Script by cron or a CI job, to update all Distribution Files periodically to the latest release and afterwards run [Openstack Image Manager](../image-manager/).
The Distribution Image yaml files must exists before running the script, you can use the files from Github repo at `etc/images/` as template for your first run.


```
$ python contrib/update.py
2024-04-24 09:29:44 | INFO | main:300 - Processing file /etc/images/centos.yml
2024-04-24 09:29:44 | INFO | update_image:179 - Checking image CentOS Stream 9
2024-04-24 09:29:44 | INFO | update_image:182 - Latest download URL is https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-HEREBE\d+\.\dDRAGONS.x86_64.qcow2
2024-04-24 09:29:44 | INFO | update_image:185 - Getting checksums from https://cloud.centos.org/centos/9-stream/x86_64/images/CHECKSUM
2024-04-24 09:29:44 | INFO | get_latest_default:62 - Latest URL is now https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-20240422.0.x86_64.qcow2
2024-04-24 09:29:44 | INFO | get_latest_default:63 - Latest filename is now CentOS-Stream-GenericCloud-9-20240422.0.x86_64.qcow2
2024-04-24 09:29:44 | INFO | update_image:192 - Checksum of current CentOS-Stream-GenericCloud-9-20240422.0.x86_64.qcow2 is sha256:47dd9ad7048afe96bc6cc0b3fd8922f290e99c29d251affcd22d0afecfe0e337
2024-04-24 09:29:44 | INFO | update_image:208 - Our checksum is sha256:47dd9ad7048afe96bc6cc0b3fd8922f290e99c29d251affcd22d0afecfe0e337
2024-04-24 09:29:44 | INFO | update_image:211 - Image CentOS Stream 9 is up-to-date, nothing to do
2024-04-24 09:29:44 | INFO | main:300 - Processing file /etc/images/debian.yml
2024-04-24 09:29:44 | INFO | update_image:179 - Checking image Debian 11
2024-04-24 09:29:44 | INFO | update_image:182 - Latest download URL is https://cdimage.debian.org/cdimage/cloud/bullseye/latest/debian-11-genericcloud-amd64.raw
2024-04-24 09:29:44 | INFO | update_image:185 - Getting checksums from https://cdimage.debian.org/cdimage/cloud/bullseye/latest/SHA512SUMS
2024-04-24 09:29:45 | INFO | update_image:192 - Checksum of current debian-11-genericcloud-amd64-20240211-1654.raw is sha512:bdccf01b778a602024918e27bb8cfd84be32104609651f457ac1db10ee5d2a490d0c60e21ce3c0a7704e7ca439281724d0d7e48d279c9fc3a5133a7283e321e4
2024-04-24 09:29:45 | INFO | update_image:208 - Our checksum is sha512:bdccf01b778a602024918e27bb8cfd84be32104609651f457ac1db10ee5d2a490d0c60e21ce3c0a7704e7ca439281724d0d7e48d279c9fc3a5133a7283e321e4
2024-04-24 09:29:45 | INFO | update_image:211 - Image Debian 11 is up-to-date, nothing to do
2024-04-24 09:29:45 | INFO | update_image:179 - Checking image Debian 12
2024-04-24 09:29:45 | INFO | update_image:182 - Latest download URL is https://cdimage.debian.org/cdimage/cloud/bookworm/daily/latest/debian-12-genericcloud-amd64-daily.raw
2024-04-24 09:29:45 | INFO | update_image:185 - Getting checksums from https://cdimage.debian.org/cdimage/cloud/bookworm/daily/latest/SHA512SUMS
2024-04-24 09:29:46 | INFO | update_image:192 - Checksum of current debian-12-genericcloud-amd64-daily-20240424-1727.raw is sha512:f4850b3910adb80801649399d4f89be08974a05a198aba7093f6e72d38d82183bc5b36183fb8dd34cd48a3e226d46802d8a8d85e8b5714b67c52e7ea642f085e
2024-04-24 09:29:46 | INFO | update_image:208 - Our checksum is sha512:5401f8c6361bb2a82c2c24b4b4606d95e77229152a80e61f9c613bc88e25de9257057d0ed68b0256b745c4059162a54970fe4a8daf456b2eb67b4f5db5c97fcc
2024-04-24 09:29:46 | INFO | update_image:229 - New values are {'version': '20240424', 'build_date': datetime.date(2024, 4, 24), 'checksum': 'sha512:f4850b3910adb80801649399d4f89be08974a05a198aba7093f6e72d38d82183bc5b36183fb8dd34cd48a3e226d46802d8a8d85e8b5714b67c52e7ea642f085e', 'url': 'https://cdimage.debian.org/cdimage/cloud/bookworm/daily/20240424-1727/debian-12-genericcloud-amd64-daily-20240424-1727.raw'}
2024-04-24 09:29:46 | INFO | main:300 - Processing file /etc/images/rockylinux.yml
2024-04-24 09:29:46 | INFO | update_image:179 - Checking image Rocky 9
2024-04-24 09:29:46 | INFO | update_image:182 - Latest download URL is https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud.latest.x86_64.qcow2
2024-04-24 09:29:46 | INFO | update_image:185 - Getting checksums from https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud.latest.x86_64.qcow2.CHECKSUM
2024-04-24 09:29:47 | INFO | update_image:192 - Checksum of current Rocky-9-GenericCloud.latest.x86_64.qcow2 is sha256:7713278c37f29b0341b0a841ca3ec5c3724df86b4d97e7ee4a2a85def9b2e651
2024-04-24 09:29:47 | INFO | update_image:208 - Our checksum is sha256:7713278c37f29b0341b0a841ca3ec5c3724df86b4d97e7ee4a2a85def9b2e651
2024-04-24 09:29:47 | INFO | update_image:211 - Image Rocky_9 is up-to-date, nothing to do
2024-04-24 09:29:47 | INFO | main:300 - Processing file /etc/images/ubuntu.yml
2024-04-24 09:29:47 | INFO | update_image:179 - Checking image Ubuntu 22.04
2024-04-24 09:29:47 | INFO | update_image:182 - Latest download URL is https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
2024-04-24 09:29:47 | INFO | update_image:185 - Getting checksums from https://cloud-images.ubuntu.com/jammy/current/SHA256SUMS
2024-04-24 09:29:47 | INFO | update_image:192 - Checksum of current jammy-server-cloudimg-amd64.img is sha256:62af6445fd2c31f68a069151938a7dcb49158644cae531dd22efc36c1c15a710
2024-04-24 09:29:47 | INFO | update_image:208 - Our checksum is sha256:62af6445fd2c31f68a069151938a7dcb49158644cae531dd22efc36c1c15a710
2024-04-24 09:29:47 | INFO | update_image:211 - Image Ubuntu_22.04 is up-to-date, nothing to do
2024-04-24 09:29:47 | INFO | update_image:179 - Checking image Ubuntu 22.04 Minimal
2024-04-24 09:29:47 | INFO | update_image:182 - Latest download URL is https://cloud-images.ubuntu.com/minimal/releases/jammy/release/ubuntu-22.04-minimal-cloudimg-amd64.img
2024-04-24 09:29:47 | INFO | update_image:185 - Getting checksums from https://cloud-images.ubuntu.com/minimal/releases/jammy/release/SHA256SUMS
2024-04-24 09:29:48 | INFO | update_image:192 - Checksum of current ubuntu-22.04-minimal-cloudimg-amd64.img is sha256:bd99c64ad9d926eb5769f9f2cfd96ae4989a029bd64bd3e7e7deb8cff4251c65
2024-04-24 09:29:48 | INFO | update_image:208 - Our checksum is sha256:bd99c64ad9d926eb5769f9f2cfd96ae4989a029bd64bd3e7e7deb8cff4251c65
2024-04-24 09:29:48 | INFO | update_image:211 - Image Ubuntu 22.04 Minimal is up-to-date, nothing to do
2024-04-24 09:29:48 | INFO | update_image:179 - Checking image Ubuntu 24.04
2024-04-24 09:29:48 | INFO | update_image:182 - Latest download URL is https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
2024-04-24 09:29:48 | INFO | update_image:185 - Getting checksums from https://cloud-images.ubuntu.com/noble/current/SHA256SUMS
2024-04-24 09:29:48 | INFO | update_image:192 - Checksum of current noble-server-cloudimg-amd64.img is sha256:32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3
2024-04-24 09:29:48 | INFO | update_image:208 - Our checksum is sha256:d7ba8d5d1d073f2dc8351973bf4f35157c846a0ea6ee16fb2a9f45a78953e4a7
2024-04-24 09:29:48 | INFO | update_image:229 - New values are {'version': '20240423', 'build_date': datetime.date(2024, 4, 23), 'checksum': 'sha256:32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3', 'url': 'https://cloud-images.ubuntu.com/noble/20240423/noble-server-cloudimg-amd64.img'}
```

These yaml files are now extended with additional fields and the update.py will take care of the versions, checksum, url and build date to the latest release in the yaml file on every run.

* latest_checksum_url - URL of the distros checksum file
* latest_url - URL of the distros latest image
* mirror_url - URL of the Image File at the local S3 Mirror

```yaml title="someexample.yaml"
---
images:
- name: Debian 12
enable: true
shortname: debian-12
format: qcow2
login: debian
min_disk: 8
min_ram: 512
status: active
visibility: public
multi: true
meta:
architecture: x86_64
hw_disk_bus: scsi
hw_rng_model: virtio
hw_scsi_model: virtio-scsi
hw_watchdog_action: reset
hypervisor_type: qemu
os_distro: debian
os_version: '12'
replace_frequency: quarterly
uuid_validity: last-3
provided_until: none
tags: []
latest_checksum_url: https://cdimage.debian.org/cdimage/cloud/bookworm/daily/latest/SHA512SUMS
latest_url:
https://cdimage.debian.org/cdimage/cloud/bookworm/daily/latest/debian-12-genericcloud-amd64-daily.qcow2
versions:
- build_date: 2024-04-11
checksum:
sha512:3d6f26616e2c8b705993ddef874232887cebe42f1e70fcc020827ac88e8990177d537d34538c71ae2afd3b8baca953fff71eaa7ef71e752e82532c93dcdca436
url:
https://cdimage.debian.org/cdimage/cloud/bookworm/daily/20240411-1714/debian-12-genericcloud-amd64-daily-20240411-1714.qcow2
mirror_url:
https://swift.services.a.regiocloud.tech/swift/v1/AUTH_b182637428444b9aa302bb8d5a5a418c/openstack-images/debian-12/20240411-debian-12.qcow2
version: '20240411'

```
Loading