Merge Distributed Qualification Tools CLI #1516
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Add arguments for running tools in distributed mode Signed-off-by: Partho Sarthi <[email protected]> * Refactor to use tools config file Signed-off-by: Partho Sarthi <[email protected]> * Update specification Signed-off-by: Partho Sarthi <[email protected]> * Update tools config file Signed-off-by: Partho Sarthi <[email protected]> * Update comment Signed-off-by: Partho Sarthi <[email protected]> * Add pylint exception Signed-off-by: Partho Sarthi <[email protected]> * Include hdfs output dir in tools config file Signed-off-by: Partho Sarthi <[email protected]> * Add comment about assumption of Spark JARs Signed-off-by: Partho Sarthi <[email protected]> * Revert changes in stats report Signed-off-by: Partho Sarthi <[email protected]> * Submission mode Args Signed-off-by: Partho Sarthi <[email protected]> * Modify the arguments structure Signed-off-by: Partho Sarthi <[email protected]> * Bump up the API version for tools config file Signed-off-by: Partho Sarthi <[email protected]> * Update python arg tests Signed-off-by: Partho Sarthi <[email protected]> * Remove pylint disable rule in CSPs Signed-off-by: Partho Sarthi <[email protected]> --------- Signed-off-by: Partho Sarthi <[email protected]>
) * Add implementation for running tools in distributed mode Signed-off-by: Partho Sarthi <[email protected]> * Fix temp directory usage and remove automatic spark properties calculation Signed-off-by: Partho Sarthi <[email protected]> * Refactor Signed-off-by: Partho Sarthi <[email protected]> * Simplify hashing of intermediate output dirs Signed-off-by: Partho Sarthi <[email protected]> * Fix typing hint for python 3.8 Signed-off-by: Partho Sarthi <[email protected]> --------- Signed-off-by: Partho Sarthi <[email protected]>
parthosa
added
feature request
|
This needs licensing errors fixed #1518 |
This PR fixes the license headers for new files created in the distributed tools branch to be `2025` and files modified to be `*-2025` --------- Signed-off-by: Partho Sarthi <[email protected]>
amahussein
previously approved these changes
Feb 4, 2025
amahussein
approved these changes
Feb 5, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1249.
This PR merged the distributed tool branch to dev. We have setup internal CI pipeline to test this feature.
Created a follow-up to support recursive processing of event logs (#1515)