Refactored Snakemake workflow and generate example output #42
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Updated the dockerfile to use a two stage process. Snakemake image is used as the runtime environment. Supports / Partially addresses #17
Updated snakemake workflow to work with Dockerfile. Also fixes module not found error by exposing pythonpath. Partially addresses #17
- Temporary fix for ValueError with multiple context URLs (#34) by returning empty context for URLs that don't match allowed_urls_pattern.
FromAsCasing - FROM and AS should be in the same case.
- Added get_max_id_from_files method. - If --workflow-ids argument is not provided, created_files.json should contain all workflows downloaded. get_max_id_from_files() finds the maximum ID in the sourced workflows. - Part of wider effort to address #17
Removes list_expected_files() method.
- Added progress bar and removed line-by-line logging for easier evaluation of download and processing progress. - Addresses #33
- Added preliminary module create_ro_crate to create an RO crate for a given file. - Addresses #37
- Added a rule in Snakemake workflow to create an RO crate. - Addresses #37
Addresses #37
- Added poetry lock
There was a problem hiding this comment.
Not knowing much about snakemake I could be wrong but should this be (git) ignored?
There was a problem hiding this comment.
Think I misunderstood the purpose of the ro-crate-metadata folder
| @@ -57,6 +53,12 @@ def info(self): | |||
| def geturl(self): | |||
| return url | |||
|
|
|||
| if not allowed_urls_re.match(url): | |||
| return Response(json.dumps({"@context": {}})) | |||
| # raise ValueError( | |||
There was a problem hiding this comment.
Looks like this might be redundant now? (edit: the comment I mean)
| @@ -32,7 +35,7 @@ def merge_all_files( | |||
|
|
|||
| for i, fn in enumerate(filenames): | |||
| with open(fn, "r") as f: | |||
| print(f"Processing {fn}, {i}/{len(filenames)}") | |||
| update_progress_bar(i + 1, len(filenames)) | |||
There was a problem hiding this comment.
Idle comment but could something like https://github.com/tqdm/tqdm work for this?
OliverWoolland
approved these changes
Aug 6, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR primarily addresses: #37, #34, #33, #28, #17
The snakemake workflow consists of 3 main rules:
source_ro_cratessources the RO crates and checks their outputmerge_filescreates the merged.ttlfilecreate_ro_cratecreates thero-crate-metadata.jsonfileThe example
.ttlfile consists of all versions of all publicly accessible RO crates on WorkflowHub.