Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: copy regex #889

Merged
merged 5 commits into from
Dec 3, 2024
Merged

fix: copy regex #889

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
cbadcac
feat: publish capture-dates.geojson file
amfage Dec 3, 2024
38ad1c9
fix: formatting
amfage Dec 3, 2024
e65eeb0
fix: copy regex
amfage Dec 3, 2024
ff07042
fix: escape character
amfage Dec 3, 2024
5d80c28
Merge branch 'master' into fix/copy-regex
amfage Dec 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 14 additions & 14 deletions workflows/raster/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,20 +211,20 @@ Access permissions are controlled by the [Bucket Sharing Config](https://github.

## Workflow Input Parameters

| Parameter | Type | Default | Description |
| -------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| user_group | enum | none | Group of users running the workflow |
| ticket | str | | Ticket ID e.g. 'AIP-55' |
| region | enum | | Region of the dataset |
| source | str | s3://linz-imagery-staging/test/sample/ | The URIs (paths) to the s3 source location. Separate multiple source paths with `;` |
| target | str | s3://linz-imagery-staging/test/sample_target/ | The URIs (paths) to the s3 target location |
| include | regex | \\.tiff?\$\|\\.json\$\|\\.tfw\$\|\\Q\/capture-area.geojson\\E\|\\Q\/capture-dates.geojson\\E | A regular expression to match object path(s) or name(s) from within the source path to include in the copy. |
| exclude | regex | | A regular expression to match object path(s) or name(s) from within the source path to exclude from the copy. |
| copy_option | enum | --no-clobber | <dl><dt>`--no-clobber` </dt><dd> Skip overwriting existing files.</dd><dt> `--force` </dt><dd> Overwrite all files. </dd><dt> `--force-no-clobber` </dt><dd> Overwrite only changed files, skip unchanged files. </dd></dl> |
| flatten | enum | false | Flatten the files in the target location (useful for multiple source locations) |
| group | int | 1000 | The maximum number of files for each pod to copy (will use the value of `group` or `group_size` that is reached first). |
| group_size | str | 100Gi | The maximum group size of files for each pod to copy (will use the value of `group` or `group_size` that is reached first). |
| transform | str | `f` | String to be transformed from source to target to renamed filenames, e.g. `f.replace("text to replace", "new_text_to_use")`. Leave as `f` for no transformation. |
| Parameter | Type | Default | Description |
| -------------------- | ----- |---------------------------------------------------------------------------------------------------------------------------------------------------------------| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| user_group | enum | none | Group of users running the workflow |
| ticket | str | | Ticket ID e.g. 'AIP-55' |
| region | enum | | Region of the dataset |
| source | str | s3://linz-imagery-staging/test/sample/ | The URIs (paths) to the s3 source location. Separate multiple source paths with `;` |
| target | str | s3://linz-imagery-staging/test/sample_target/ | The URIs (paths) to the s3 target location |
| include | regex | \\.tiff?\$\|\\.json\$\|\\.tfw\$\|/capture-area\\.geojson\$\|/capture-area\\.geojson\$ | A regular expression to match object path(s) or name(s) from within the source path to include in the copy. |
| exclude | regex | | A regular expression to match object path(s) or name(s) from within the source path to exclude from the copy. |
| copy_option | enum | --no-clobber | <dl><dt>`--no-clobber` </dt><dd> Skip overwriting existing files.</dd><dt> `--force` </dt><dd> Overwrite all files. </dd><dt> `--force-no-clobber` </dt><dd> Overwrite only changed files, skip unchanged files. </dd></dl> |
| flatten | enum | false | Flatten the files in the target location (useful for multiple source locations) |
| group | int | 1000 | The maximum number of files for each pod to copy (will use the value of `group` or `group_size` that is reached first). |
| group_size | str | 100Gi | The maximum group size of files for each pod to copy (will use the value of `group` or `group_size` that is reached first). |
| transform | str | `f` | String to be transformed from source to target to renamed filenames, e.g. `f.replace("text to replace", "new_text_to_use")`. Leave as `f` for no transformation. |
| aws_role_config_path | str | `s3://linz-bucket-config/config-write.elevation.json,s3://linz-bucket-config/config-write.imagery.json,s3://linz-bucket-config/config-write.topographic.json` | s3 URL or comma-separated list of s3 URLs allowing the workflow to write to a target(s). |

## Examples
Expand Down
2 changes: 1 addition & 1 deletion workflows/raster/copy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ spec:
- name: target
value: 's3://linz-imagery-staging/test/sample_target/'
- name: include
value: '\.tiff?$|\.json$|\.tfw$|\Q/capture-area.geojson\E|\Q/capture-dates.geojson\E'
value: '\.tiff?$|\.json$|\.tfw$|/capture-area\.geojson$|/capture-dates\.geojson$'
- name: exclude
value: ''
- name: copy_option
Expand Down
Loading