Automation and optimization work #5

sheepman4267 · 2024-10-29T15:47:14Z

This is the (long-awaited) appstream automation PR!

Here's what it does:

Adds an Ansible playbook which automates steps 1-5 of the current "Regenerating from teaparty server directly" workflow
Optimizes the creation of appstream metadata by symlinking every package to be analyzed into one directory. This script:
- Avoids analyzing packages we don't need to (-currently the script ignores -dbginfo packages, but should probably also ignore -devel ones).
- Only symlinks packages which are actually in the index, which works around Doesn't always cleanup stale eopkgs from deprecated packages ferryd#17.

Here's what it doesn't include yet, which it needs to before this can be marked ready for review:

~~A Taskfile with further automation, including step 7 of the current instructions.~~
~~Updates to README.md documenting the new process.~~
~~A test plan in the description :)~~

Here's what it doesn't include, and I have no plans to:

I don't plan to replicate step 6 of the current instructions in this new workflow. I have thoughts on more automated testing, but 1: The playbook already checks that the generated metadata file's packages are all in the eopkg index, and 2: testing of the new appstream data should happen as part of the package update test plan anyway. IMO, testing it here makes it more likely that someone will become lax when testing the package update, which it probably where it's more likely to go wrong anyway.
Automation of the package update and PR process for appstream-data in the packages repo. On further reflection, I think any approach I can come up with there will be overly brittle and out of scope for this repo. That work will be done, but in a separate Taskfile in that package's files.

Test Plan
Follow the directions in the updated version of the README. That documentation is under test along with the actual process changes.

For reference, the current workflow is under the heading "Regenerating from teaparty server directly" in this repo's README.

…inventory.py

silkeh · 2025-01-08T17:15:06Z

inventory/hosts.yml.example

+      ansible_user: <your_user_on_teaparty>
+      # Set this to your username


Maybe it's better to recommend people to configure their username correctly in their SSH config so this file can be used as is?

For example:

Host *.getsol.us User <username here>

Good idea, but I think it's out of scope here. I think this should be part of our staff onboarding processes, which should be written down (what people need access to, why, how to get there, etc.). If that exists, I've never seen it (at least, not in the "set up your SSH config correctly" level of detail). If we can get that documentation into shape in an appropriate place, I'd love to remove this hacky inventory creation script nonsense. I'm just not convinced that the correct place is here in this repo.

Feel free to convince me otherwise, though!

I didn't mean to suggest to document that here, I just don't think that this project should carry the complexity burden of people not having configured their SSH correctly.

Edit: I've created an issue to document this https://github.com/getsolus/solus-team-docs/issues/60

archive_screenshots.sh

silkeh · 2025-01-08T17:19:50Z

setup_inventory.py

@@ -0,0 +1,27 @@
+#!/bin/python3


If we go the ~/.ssh/config route this should not be needed :S

create_symlinks.py

Co-authored-by: Silke Hofstra <[email protected]>

…laybook

TraceyC77

The README needs a step before generating mainstream data about copying hosts.yml.example to the real file and adding the username. Otherwise go-task full-process errors out. Once that's done this PR is good to go.

In my testing, once I set that up and ran go-task full-process, appstream data generation was successful

TraceyC77 · 2025-01-10T23:22:18Z

README.md

+> Additionally, your GitHub account must have push access to this repository (should be true for all staff).
+1. Ensure that your local clone of this repository is set up to be able to push (Cloning via GitHub CLI recommended).
+2. Ensure you have `go-task` installed. All interaction with this tooling should be possible through the `Taskfile.yml` in this repository.
+3. Run `go-task appstream-init`. This task will install `pyyaml` and `ansible` on your system, and then install the necessary ansible collection.


There needs to be a step here to copy and adapt inventory.yml.example and that the file needs to be named hosts.yml (due to the rule in .gitignore)

I've removed the rule in .gitignore and made the inventory file generic. A username does not need to be listed there if your SSH is configured correctly, per the conversation with @silkeh.

We don't need this to be configured per user if folks set up SSH correctly.

…inventory.py

Co-authored-by: Silke Hofstra <[email protected]>

We don't need this to be configured per user if folks set up SSH correctly.

…laybook

TraceyC77

I re-tested after doing a hard reset on my local copy of the branch so I had the new generic hosts.yml. I already had an entry for teaparty in ~/.ssh.config.
Verified that appstream data was able to be kicked off an a commit with the data was created as expected.

I Silke's concerns have also been addressed.

Good to go!

Please clean up the git commit message and this can be squashed and merged

sheepman4267 added 8 commits October 29, 2024 11:26

.gitignore: Ignore ansible inventory

b7346d6

Break out mirror.tar creation into a separate script

dc39dfe

Add an ansible playbook which automates appstream generation

fa50edf

Add a python script which prints the next git tag

c6e5c0f

Add Taskfile.yml

a533324

Follow consistent script naming practice

3dd44b3

Add script to automate creating ansible inventory

ea8938a

Document and somewhat automate the setup process

408e252

sheepman4267 marked this pull request as ready for review November 21, 2024 21:14

joebonrichie and others added 4 commits November 27, 2024 08:21

Refresh Appstream metainfo

cd39571

setup_inventory: Add shebang line and execute bit

58e7e2b

appstream-init: Install community.general and pass filename to setup_…

d24a545

…inventory.py

Add executable bit to archive_screenshots.sh

d18a342

silkeh reviewed Jan 8, 2025

View reviewed changes

sheepman4267 and others added 7 commits January 8, 2025 16:47

archive_screenshots.sh: set -euo pipefail is indeed a good idea

377acb2

Co-authored-by: Silke Hofstra <[email protected]>

create_symlinks.py: Remove print debugging nonsense

06bff8c

Refresh Appstream metainfo

8dae195

Remove historical README

19efba9

Remove inventory setup; we don't need it

9d64fb3

Merge remote-tracking branch 'origin/ansible-playbook' into ansible-p…

155536e

…laybook

Correct README.md to add BECOME password instructions

2f62411

TraceyC77 requested changes Jan 11, 2025

View reviewed changes

sheepman4267 added 8 commits January 10, 2025 21:39

Commit the actual host.yml file in generic format

99a744d

We don't need this to be configured per user if folks set up SSH correctly.

Update README.md to include info about SSH setup

7fdeec3

Push tags properly

0b562dd

.gitignore: Ignore ansible inventory

bb8bd1f

Break out mirror.tar creation into a separate script

e2fc54e

Add an ansible playbook which automates appstream generation

9709712

Add a python script which prints the next git tag

888c0ca

Add Taskfile.yml

d00219c

sheepman4267 and others added 16 commits January 10, 2025 21:50

Follow consistent script naming practice

ecd2c48

Add script to automate creating ansible inventory

4b496ec

Document and somewhat automate the setup process

0c44cbd

setup_inventory: Add shebang line and execute bit

5b61e0e

appstream-init: Install community.general and pass filename to setup_…

f3e0d70

…inventory.py

Add executable bit to archive_screenshots.sh

b0f91ff

Remove historical README

edc3bd7

Remove inventory setup; we don't need it

a7dbd01

archive_screenshots.sh: set -euo pipefail is indeed a good idea

b4ecb15

Co-authored-by: Silke Hofstra <[email protected]>

create_symlinks.py: Remove print debugging nonsense

4c23487

Correct README.md to add BECOME password instructions

775cf7c

Commit the actual host.yml file in generic format

8ff72d1

We don't need this to be configured per user if folks set up SSH correctly.

Update README.md to include info about SSH setup

7f658ff

Push tags properly

a936283

Merge remote-tracking branch 'origin/ansible-playbook' into ansible-p…

99f1550

…laybook

Refresh Appstream metainfo

8fce07d

TraceyC77 approved these changes Jan 11, 2025

View reviewed changes

sheepman4267 merged commit 9f70d79 into master Jan 11, 2025

sheepman4267 deleted the ansible-playbook branch January 11, 2025 03:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automation and optimization work #5

Automation and optimization work #5

sheepman4267 commented Oct 29, 2024 •

edited

Loading

silkeh Jan 8, 2025

sheepman4267 Jan 8, 2025

silkeh Jan 9, 2025 •

edited

Loading

silkeh Jan 8, 2025

TraceyC77 left a comment •

edited

Loading

TraceyC77 Jan 10, 2025

sheepman4267 Jan 11, 2025

TraceyC77 left a comment •

edited

Loading

		ansible_user: <your_user_on_teaparty>
		# Set this to your username

Automation and optimization work #5

Automation and optimization work #5

Conversation

sheepman4267 commented Oct 29, 2024 • edited Loading

silkeh Jan 8, 2025

Choose a reason for hiding this comment

sheepman4267 Jan 8, 2025

Choose a reason for hiding this comment

silkeh Jan 9, 2025 • edited Loading

Choose a reason for hiding this comment

silkeh Jan 8, 2025

Choose a reason for hiding this comment

TraceyC77 left a comment • edited Loading

Choose a reason for hiding this comment

TraceyC77 Jan 10, 2025

Choose a reason for hiding this comment

sheepman4267 Jan 11, 2025

Choose a reason for hiding this comment

TraceyC77 left a comment • edited Loading

Choose a reason for hiding this comment

sheepman4267 commented Oct 29, 2024 •

edited

Loading

silkeh Jan 9, 2025 •

edited

Loading

TraceyC77 left a comment •

edited

Loading

TraceyC77 left a comment •

edited

Loading