Brainstorm ideas for implementing a WC testing pipeline

[yulianedyalkova]

Can we find a solution that doesn't include implementing the whole end to end solution in Golang that also covers the following criteria:

• Teams can reuse the same pipelines to create clusters with different configurations
• There is a way to couple configuration to tests in order to filter out which tests are being ran
• There is a way to run and debug tests from a local machine
• Solution should be easily expandable for cluster upgrades

Acceptance criteria:

• We have one or more potential solutions documented in a way that the team can comment on

[AverageMarcus]

Dumping links to various things we've discussed in meetings around this topic:

• Hackathon attempt at a test suite - https://github.com/giantswarm/giantswarm-e2e-tests/tree/wip-cluster-suite
• Upstream e2e testing framework - https://github.com/kubernetes-sigs/e2e-framework
• Blog post introducing the above framework - https://medium.com/programming-kubernetes/end-to-end-testing-of-kubernetes-resources-with-the-e2e-framework-ac52e7e58db8
• CAPO contributor looking to do similar with Ginkgo - How to reuse expensive test initialisation? onsi/ginkgo#853
• Similar issue looking at how to handle expensive setup and tear down - Patterns for managing expensive spec setup onsi/ginkgo#1108

[AverageMarcus]

My general / unorganised thoughts:

• Triggering of the test app needs a standard interface so it can be triggered from CI/CD pipelines without being bespoke for each use-case / provider / team.
• The framework would need some way of getting the MC's kubeconfig from the environment its running in (either as env var or passed in as a run argument flag)
• The MVP of the framework would need to have at least the following: create cluster, wait for cluster ready, delete cluster
• Features of the testing framework (long-term):
  • validation of MC access
  • validation of values yaml (schema?)
  • fetching latest version of various apps
  • creation of cluster
  • app installation
  • wait for cluster to be ready
  • deletion of cluster
  • validation/assertion helpers (e.g. is app running, does config equal, etc)
  • upgrade app (which can also be used for upgrading cluster)
  • pre/post hooks
• Tekton Pipelines to go along with the framework:
  • PR trigger from cluster-xxx or default-apps app repos. These will be standard set of tests but with the app versions possibly replaced to test with the changes in the PR.
  • PR trigger from another app repo. These will use the latest version of cluster app but would need to install the current repos app as part of the cluster creation
  • PR trigger from the tests own repo. These should run all tests with the standard inputs and exists to test the changes being made to the tests themselves.
  • Nightly cron trigger. These run all tests with the standard inputs. Failures need to be raised somehow (issue creation or slack notification). These would also catch changes made to the MC that effects the workloads cluster creation.
• Ownership is unclear. The Tekton pipelines mentioned above make sense to belong to Team Tinkerers but I'm not sure if it makes sense for them to also own the testing framework long-term. Some concerns around the maintainability of it when priorities of the team move on to other topics that aren't testing specific (e.g. fork management).
• We don't have the resources or skillsets in the team right now to do this in an ideal way. Would require help from other teams. This feeds back in to the concerns about long-term maintenance.

Questions:

• Where will the tests live? Will they go alongside the cluster-xxx apps or in dedicated repos?
• Testing style? - Standard Go tests? BDD style (e.g. ginkgo)?
• Do we want to make use of the upstream e2e-framework project? (This would also depend on the answer to the previous question)

[yulianedyalkova]

The MVP of the framework would need to have at least the following: create cluster, wait for cluster ready, delete cluster

I reached the same conclusion, I feel like doing the cluster operations in tekton tasks will result in a really awkward and unusable interface.

Ownership is unclear

Theoretically this is part of the testing building blocks and framework so I do see us having the ownership, at least of the basic cluster operations in the MVP. Up to discussion if that remains the case if teams start adding a lot more complexity to it. But we should also define the architecture of the tool in a way that we provide (golang) interfaces that can be further implemented and owned by the teams where possible.

We don't have the resources or skillsets in the team right now to do this in an ideal way.

I can definitely assist in architecture and reviewing PRs, can also join active coding if there is no other way. If we don't scale, I will make sure to address this.

Testing style? - Standard Go tests? BDD style (e.g. ginkgo)?

I personally find ginkgo/gomega a lot more readable and flexible that plain go tests, it also makes it super easy for people that don't understand the whole framework to add tests as it almost feels like documentation (which was one of our requirements). The wip e2e framework from Hydra is also using them.

Do we want to make use of the upstream e2e-framework project? (This would also depend on the answer to the previous question)

That strongly depends on what the framework provides and if using it will make our lives easier or harder (as we perform a lot of GS-specific operations).

[AverageMarcus]

That strongly depends on what the framework provides and if using it will make our lives easier or harder (as we perform a lot of GS-specific operations).

At the current stage it's really in its early days so the amount of things it offers is pretty limited.

It also introduces some ways of writing tests I've never seen myself before (features with labels and asses, etc.)

I have no strong feelings one way or the other. My general thoughts are:

• If we chose to use it, we can also contribute upstream and improve it for everyone (yay open source!)
• If we chose not to use it, I have a feeling we'll be replicating a bunch of its features into our framework but in a way that then works with Ginkgo
• I don't currently know how easy it is to build a framework that extends or builds on top of Ginkgo. I need to investigate this unless someone else already knows. My concern here is we'll either replace Ginkgo completely and proxy through to the ginkgo functions or we'll act as a peer dependancy and then have to keep up to date with changes of ginkgo.

[yulianedyalkova]

We've reached a decision - we will implement our own framework using go/ginkgo.