Convert ci-shark-ai.yml to pkgci_shark_ai.yml.
#625
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Marius Brehler <[email protected]>
This was referenced Dec 4, 2024
renxida
added a commit
that referenced
this pull request
Jan 10, 2025
… build packages once (#780) This builds on #625, #589 to make progress on issue #584. This adds a pkgci.yml to run multiple package-based CI tasks after building package using Scott's changes in #667. This gives us the following benefits: * Integration test workflows are faster because they now use dev packages, without needing to build them from source or use editable installs. Also, if more integration tests are added, they can reuse the built packages. * Users and developers can access the same dev packages to reproduce CI results * Only one runner needs the build requirements (potentially including clang, ninja, CMake, Rust, etc.), other runners only need Python. This also switches to using uv to create venvs, which is faster. This PR brings shortfin CPU LLM CI time to roughly half an hour on the mi250 runner to a few seconds of package build (fast due to caching) and around 5 minutes of testing. --------- Co-authored-by: Scott Todd <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This builds on #589 to make progress on #584.
On the surface, this adds complexity right now. The benefits are:
This also switches to using Python venvs, tightening configuration control in workflows and ensuring that only the expected packages are installed and used for tests, without leaking installed packages across workflow runs. That does increase the workflow time though.
Metrics using
pipLogs before: https://github.com/nod-ai/shark-ai/actions/runs/12058325322
Install pip deps(no venv, reusing already installed packages and getting the wrong IREE versions)Run LLM Integration TestsLogs after (cold cache): https://github.com/nod-ai/shark-ai/actions/runs/12059301876/attempts/1?pr=625
Build shortfin(cold cache)Setup venv(from scratch, no deps reused)Install nightly IREE packagesRun LLM Integration TestsLogs after (warm cache): https://github.com/nod-ai/shark-ai/actions/runs/12059301876/attempts/2?pr=625
Build shortfin(warm cache, 46% cache hit rate)Setup venv(from scratch, no deps reused)Install nightly IREE packagesRun LLM Integration TestsMetrics using
uvuv(https://docs.astral.sh/uv/) can be used as an alternative topipLogs (cold cache): https://github.com/nod-ai/shark-ai/actions/runs/12147344797?pr=625
Build shortfin(cold cache)Setup venv(from scratch, no deps reused)Install nightly IREE packagesRun LLM Integration Tests(failed)Logs (warm cache): https://github.com/nod-ai/shark-ai/actions/runs/12147452119?pr=625
Build shortfin(cold cache)Setup venv(from scratch, no deps reused)Install nightly IREE packagesRun LLM Integration Tests(failed)