Add "list-testplan" subcommand (New) #1047
Conversation
The new subcommand works similarly to "list-bootstrapped" since it also bootstraps a given test plan. However, it filters out the instantiated jobs and replaces them with their original template. In the end, a list of jobs and templates used in this test plan are returned as JSON-formatted data. This is to be used by external tools, for instance to create a document listing information (summary, description...) about executed jobs and templates. Fix CHECKBOX-1263
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1047 +/- ##
==========================================
+ Coverage 40.76% 40.80% +0.04%
==========================================
Files 336 336
Lines 37462 37490 +28
Branches 6371 6376 +5
==========================================
+ Hits 15271 15298 +27
Misses 21550 21550
- Partials 641 642 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
| raise SystemExit("Test plan not found") | ||
| self.sa.select_test_plan(ctx.args.TEST_PLAN) | ||
| self.sa.bootstrap() | ||
| job_and_template_list = [] |
There was a problem hiding this comment.
Minor small fix, use a set here, way faster to check if something is in there or not
There was a problem hiding this comment.
If I use a set() instead of a list [], I get the following error:
Traceback (most recent call last):
File "/home/pieq/.venvs/checkbox/bin/checkbox-cli", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/pieq/dev/work/checkbox/checkbox-ng/checkbox_ng/launcher/checkbox_cli.py", line 165, in main
return subcmd.invoked(ctx)
^^^^^^^^^^^^^^^^^^^
File "/home/pieq/dev/work/checkbox/checkbox-ng/checkbox_ng/launcher/subcommands.py", line 1375, in invoked
job_and_template_set.add(attrs)
TypeError: unhashable type: 'dict'
|
Works like a charm, thanks! Already using it for a refreshed doc version |
|
This implementation depends on bootstrapping, which is not good, because if ran on a device without the proper resources, some jobs may be ignored and would not show up in the generated output. I need to revisit how to get all the information we need. |
|
This PR is replaced by #1065 |
Description
The new subcommand works similarly to "list-bootstrapped" since it also
bootstraps a given test plan.
However, it filters out the instantiated jobs and replaces them with
their original template.
In the end, a list of jobs and templates used in this test plan are
returned as JSON-formatted data.
This is to be used by external tools, for instance to create a document
listing information (summary, description...) about executed jobs and
templates.
Resolved issues
CHECKBOX-1263
Documentation
TBD
Tests
Given the following units:
The command
$ checkbox-cli list-testplan com.canonical.certification::pieq-job-test-plan | jqwill return:[ { "id": "pieq-job-fail", "plugin": "shell", "command": "false", "flags": "also-after-suspend", "unit": "job", "certification_status": "blocker" }, { "id": "pieq-job-sibling", "plugin": "shell", "command": "echo \"This is a job\"", "siblings": "[\n {\n \"id\": \"after-suspend-pieq-job-sibling\",\n \"depends\": \"com.canonical.certification::suspend/suspend_advanced_auto pieq-job-sibling\"\n }\n ]", "unit": "job", "certification_status": "unspecified" }, { "unit": "template", "template-resource": "demoresource", "template-id": "demo-template", "template-summary": "Display the path of all the disks on the system", "template-description": " Walking sysfs devices, display information about disk while retaining\n other information that could be useful, but we will not get into too much\n details here because this is not the point.", "id": "demo-test-storage-{name}", "plugin": "shell", "command": "echo \"This is the path: {path}\"", "flags": "also-after-suspend", "certification_status": "unspecified" }, { "id": "rtc", "estimated_duration": "0.02", "plugin": "resource", "category_id": "information_gathering", "command": " if [ -e /proc/driver/rtc ]\n then\n echo \"state: supported\"\n else\n echo \"state: unsupported\"\n fi", "_summary": "Creates resource info for RTC", "unit": "job", "certification_status": "unspecified" }, { "id": "sleep", "estimated_duration": "0.03", "plugin": "resource", "category_id": "information_gathering", "command": " tr ' ' '\\n' < /sys/power/state | while IFS= read -r state; do echo \"$state: supported\"; done\n if [ -e /sys/power/mem_sleep ]; then\n awk -F\"[][]\" '{ print \"mem_sleep: \" $2 }' < /sys/power/mem_sleep\n else\n echo \"mem_sleep: unsupported\"\n fi", "_summary": "Create resource info for supported sleep states", "unit": "job", "certification_status": "unspecified" }, { "plugin": "shell", "category_id": "com.canonical.plainbox::suspend", "id": "suspend/suspend_advanced_auto", "requires": " sleep.mem == 'supported'\n rtc.state == 'supported'", "_summary": "Automated test of suspend function", "_description": "This is the automated version of suspend/suspend_advanced.", "user": "root", "environ": "PLAINBOX_SESSION_SHARE RTC_DEVICE_FILE", "command": "if [[ -v SNAP ]]; then\n export LD_LIBRARY_PATH=\"$LD_LIBRARY_PATH:$SNAP/usr/lib/fwts\"\nfi\n# fwts s3 is not available on all architectures (i.e ARM)\nif fwts --show-tests-categories | grep -q 's3 '; then\n echo \"Calling fwts\"\n set -o pipefail; checkbox-support-fwts_test -f none -l \"$PLAINBOX_SESSION_SHARE\"/suspend_single.log -s s3 --s3-sleep-delay=30 --s3-device-check --s3-device-check-delay=45 | tee \"$PLAINBOX_SESSION_SHARE\"/suspend_single_times.log\nelse\n if [ -z \"$RTC_DEVICE_FILE\" ]; then\n echo \"Calling rtcwake\"\n rtcwake -m no -s 30 && systemctl suspend || exit 1\n else\n echo \"Calling rtcwake with -d $RTC_DEVICE_FILE\"\n rtcwake -d \"$RTC_DEVICE_FILE\" -m no -s 30 && systemctl suspend || exit 1\n fi\nfi", "estimated_duration": "90.000", "unit": "job", "certification_status": "unspecified" }, { "id": "after-suspend-pieq-job-fail", "plugin": "shell", "command": "false", "flags": "", "_summary": "pieq-job-fail after suspend (S3)", "depends": "com.canonical.certification::pieq-job-fail com.canonical.certification::suspend/suspend_advanced_auto", "unit": "job", "certification_status": "unspecified" }, { "id": "after-suspend-pieq-job-sibling", "plugin": "shell", "command": "echo \"This is a job\"", "depends": "com.canonical.certification::suspend/suspend_advanced_auto pieq-job-sibling", "unit": "job", "certification_status": "unspecified" } ]