Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kie-issues#1114: Containerize Playwright end-to-end tests to resolve screenshot comparison issues caused by OS differences #2866

Open
wants to merge 24 commits into
base: main
Choose a base branch
from
Open

kie-issues#1114: Containerize Playwright end-to-end tests to resolve screenshot comparison issues caused by OS differences #2866

wants to merge 24 commits into from

Conversation

ljmotta
Copy link
Contributor

@ljmotta ljmotta commented Jan 21, 2025 •
edited
Loading

Closes apache/incubator-kie-issues#1114
Closes #2839

Description

This PR introduces containerized end-to-end (E2E) Playwright tests to ensure consistent and reliable test execution across environments.

Tests were executed in the following environments:

  • macOS
    • arm64
    • Sonoma 14.6.1
    • Podman Desktop v1.15
    • Podman Engine 5.3.1
    • Docker CLI 27.2.1
  • Linux WSL
    • amd64
    • Ubuntu 22.04.3 LTS
    • Docker CE CLI 25.0.2
    • Docker CE Engine 25.0.2
  • Windows (To be tested)

Known Issues

  • Running the online-editor containerized test-e2e script on macOS requires starting the services in a separate terminal.
  • Performance: Tests may run slower on macOS due to architecture differences.
  • Flakiness: Retries are enabled because tests can occasionally fail without valid reasons when running in the restricted containerized environment.
  • Debugging tests is not currently supported but could be added in the future.
  • The Docker CLI is required to run the containerized tests.

Trying it out

  • Install the dependencies.
  • Build the @kie-tools/playwright-base image with the following command:
KIE_TOOLS_BUILD__buildContainerImages=true pnpm -F @kie-tools/playwright-base image:docker:build
  • In a package that contains Playwright e2e tests, run the following:
KIE_TOOLS_BUILD__runContainerizedEndToEndTests=true pnpm test-e2e

This command will start the required services and run the tests inside the container.

Updating screenshots

To update screenshots, use the pnpm test-e2e:container:shell script to open a shell inside the container:

pnpm test-e2e:container:shell
# In the container shell, manually call the `test-e2e:run` passing the `-u` or `--update-snapshots` flag
pnpm test-e2e:run -u
# or
pnpm test-e2e:run --update-snapshots

To improve performance, you can filter which tests will run by adding the -g flag:

pnpm test-e2e:run -u -g "test name"

ljmotta added 21 commits April 11, 2024 21:49
@ljmotta
initial steps to create a playwright environment with docker
c432d4b
@ljmotta
Merge remote-tracking branch 'origin/main' into playwright-docker
6c1ac33
@ljmotta
Add container solution to generate playwright screenshots
1d47181
@ljmotta
run bash script after up
1a44663
@ljmotta
Update container image and scripts
7fad5fa
@ljmotta
Use run script os in check script, and use container on all runs
227aac0
@ljmotta
Add containerizedEndToEndTests
1fe89ff
@ljmotta
Improve scripts and use dynamic host
09eb7c0
@ljmotta
Add missing cross-env
f53f4b3
@ljmotta
Rollback operator.yml
dd1b1e4
@ljmotta
Simplify CI
69fd7c3
@ljmotta
package.json adjustments
1a81a4b
@ljmotta
Simplify container file
ce08de1
@ljmotta
Make it compatible with macOS
5963a30
@ljmotta
Use nginx to forward host.docker.internal to localhost - missing onli…
c42ba1e 
…ne-editor
@ljmotta
Add reverser proxies for other services
fad1d53
@ljmotta
Rollback CI
9ff8897
@ljmotta
Writting playwright-base README
1f9945a
@ljmotta
Oops
5cfabe0
@ljmotta
Clean up
3b89a5b
@ljmotta
Update README and enable retries
f4cc454
@ljmotta ljmotta added area:dependencies Pull requests that update a dependency file area:tests labels Jan 21, 2025
@ljmotta ljmotta self-assigned this Jan 21, 2025
@ljmotta ljmotta requested a review from tiagobento as a code owner January 21, 2025 22:08
@ljmotta ljmotta added the pr: wip PR is still under development label Jan 21, 2025
@jomarko
Copy link
Contributor

jomarko commented Jan 22, 2025

Thank you @ljmotta . Please, let us know this is ready for a verification. I will try on non-MAC environment.

@ljmotta
Copy link
Contributor Author

ljmotta commented Jan 23, 2025

@jomarko This PR can be tested for macOS and Linux. I still need to make some adjustments for native Windows (non WSL).

thiagoelg
thiagoelg reviewed Jan 23, 2025
View reviewed changes
Copy link
Member

@thiagoelg thiagoelg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome PR @ljmotta! This will make our e2e tests more stable and concise!

I've made some comments below about the package.json scripts; let me know what you think.

packages/playwright-base/package.json Outdated Show resolved Hide resolved
packages/online-editor/package.json Outdated
"test-e2e": "run-script-if --ignore-errors \"$(build-env endToEndTests.ignoreFailures)\" --bool \"$(build-env endToEndTests.run)\" --then \"pnpm rimraf ./dist-tests-e2e\" \"pnpm test-e2e:run\" --else \"pnpm test-e2e:container\"",
"test-e2e:container": "run-script-if --bool \"$(build-env containerizedEndToEndTests.run)\" --then \"pnpm rimraf ./dist-tests-e2e\" \"pnpm test-e2e:container:check\"",
"test-e2e:container:bash": "docker exec -it kie-tools-playwright-containerization-online-editor /bin/bash -c \"cd kie-tools/packages/online-editor && exec /bin/bash\"",
"test-e2e:container:build": "cross-env KIE_TOOLS_BUILD__buildContainerImages=true pnpm -F playwright-base image:docker:build",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should handle the playwright-base image as a dependency, and as long as the dependencies are built, this image should exist.

It seems wrong to call the build script of a different package here, wdyt?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would simplify a lot!

@ljmotta
Copy link
Contributor Author

ljmotta commented Jan 23, 2025

Status update regarding Windows. My current setup with Docker CLI + Docker WSL didn't work, as WSL doesn't handle Windows symlinks [1][2], which is heavily used by pnpm, so mounting the Windows volume in the container doesn't work. I'll try a different approach with Docker CLI + Podman Desktop to check. Podman works a little different, but it also uses WSL behind the scenes so I'm not too confident, but I'll give it a shot.

[1] microsoft/WSL#5118
[2] microsoft/WSL#12250

@tiagobento
Copy link
Contributor

Maybe try hard links @ljmotta? I think we had a similar issue with maven-m2-repo-via-http-inage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thiagoelg thiagoelg thiagoelg left review comments

@tiagobento tiagobento Awaiting requested review from tiagobento tiagobento is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@ljmotta ljmotta

Labels
area:dependencies Pull requests that update a dependency file area:tests pr: wip PR is still under development
Projects
None yet
Milestone
No milestone
4 participants
@ljmotta @jomarko @tiagobento @thiagoelg