Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZWE Integration Test Framework #3844

Open
wants to merge 55 commits into
base: user/markackert/feat-v3-jcl-rebase
Choose a base branch
from

Conversation

MarkAckert
Copy link
Member

@MarkAckert MarkAckert commented May 31, 2024

Address #3922

This PR creates a new ZWE Integration Test Framework which focuses on delivering clear, easy-to-write tests for ZWE commands and uses Jest snapshots to capture and test command output. Higher fidelity tests could be written later with more intelligent analysis of zwe command output. More will be added to this description and the test README soon.

  • Batteries Included - setup, teardown, smart runner, cfg-yaml autocomplete, test logs, clean snapshots
  • zwe built from local repo, dependencies downloaded from manifest.json.template
  • Minimum System Requirements (test user account must be able to run zwe commands)
  • Jest Framework
  • Configuration options to help speed up dev test creation and iteration
  • CI Build and Nightly Build
  • SHORT vs LONG test designation, as defined by requiring new runs of init generate vs sharing a single run. SHORT tests complete on average in ~100s per command group, of which up to ~50s is the single shared init generate command.

Limitations:

  • Time: Each test which requires a modification to the zowe.yaml takes roughly 30s to run, bulk of which is backend job execution.
    • Mitigate via SHORT vs LONG designation and careful test design.
  • Some ZWE commands cause the shell to hang. Still investigating.

To be completed before merge:

  • A stable set of tests with committed snapshots. These are still changing frequently.
  • Investigations into USS / Shell hangs.
  • Verification of portability - run this on both internal systems and Marist environments.
  • Github Action
  • Code review feedback

@Martin-Zeithaml
Copy link
Contributor

Recently the behavior of zwe init command was changed, basically useConfigmgr does not matter.
There was a decision made not to rewrite zwe init certificate due to complexity of the code, in other words keep it in shell.
But this would be a huge limitation to have zowe.yaml as one uss file only and without ECMAS2020 scripting.
There is an internal command zwe internal config output to prepare merged config for zwe init certificate.

It does not matter now, if config is zowe.yaml or FILE(zowe.yaml):PARMLIB(TEST(ZWEYAML)), there is no message ZWEL0316E anymore.

This test (and for other commands) is kind of obsolete.

@MarkAckert MarkAckert changed the base branch from feature/v3/jcl to user/markackert/feat-v3-jcl-rebase December 11, 2024 21:28
Signed-off-by: MarkAckert <[email protected]>
Copy link

github-actions bot commented Jan 24, 2025

build 6842 FAILED.
Link to workflow run: https://github.com/zowe/zowe-install-packaging/actions/runs/12954089245

Signed-off-by: MarkAckert <[email protected]>
Copy link

github-actions bot commented Jan 24, 2025

build 6844 SUCCEEDED.
Link to workflow run: https://github.com/zowe/zowe-install-packaging/actions/runs/12954571425

Copy link

github-actions bot commented Jan 24, 2025

Test workflow 6019 is started.
Running install test: Convenience Pax
The zowe artifact being used by this test workflow: libs-snapshot-local/org/zowe/3.1.0-PR-3844/zowe-3.1.0-pr-3844-6844-20250124173612.pax
Running on machine: zzow09
Result: FAILURE
Link to workflow run: https://github.com/zowe/zowe-install-packaging/actions/runs/12954648307

Signed-off-by: MarkAckert <[email protected]>
Copy link

github-actions bot commented Jan 27, 2025

build 6859 SUCCEEDED.
Link to workflow run: https://github.com/zowe/zowe-install-packaging/actions/runs/12998019085

Copy link

github-actions bot commented Jan 27, 2025

Test workflow 6026 is started.
Running install test: Convenience Pax
The zowe artifact being used by this test workflow: libs-snapshot-local/org/zowe/3.1.0-PR-3844/zowe-3.1.0-pr-3844-6859-20250127205128.pax
Running on machine: zzow09
Result: FAILURE
Link to workflow run: https://github.com/zowe/zowe-install-packaging/actions/runs/12998095136

Signed-off-by: MarkAckert <[email protected]>
Copy link

github-actions bot commented Feb 6, 2025

build 6946 SUCCEEDED.
Link to workflow run: https://github.com/zowe/zowe-install-packaging/actions/runs/13186256839

Copy link

github-actions bot commented Feb 7, 2025

build 6956 SUCCEEDED.
Link to workflow run: https://github.com/zowe/zowe-install-packaging/actions/runs/13205186648

Signed-off-by: MarkAckert <[email protected]>
Copy link

github-actions bot commented Feb 7, 2025

build 6959 SUCCEEDED.
Link to workflow run: https://github.com/zowe/zowe-install-packaging/actions/runs/13206309082

Copy link

github-actions bot commented Feb 7, 2025

build 6960 SUCCEEDED.
Link to workflow run: https://github.com/zowe/zowe-install-packaging/actions/runs/13206600965

Copy link

github-actions bot commented Feb 7, 2025

build 6965 SUCCEEDED.
Link to workflow run: https://github.com/zowe/zowe-install-packaging/actions/runs/13207577947

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

3 participants