Skip to content

Commit

Permalink
feat: shorthand syntax for pipeline intermediate files (#174)
Browse files Browse the repository at this point in the history 
* feat: shorthand syntax for pipeline intermediate files

* update README.md and step.yml

* add e2e test
  • Loading branch information
@morbalint
morbalint authored Mar 23, 2023
1 parent 4cfb69b commit d58ca90
Show file tree
Hide file tree
Showing 5 changed files with 102 additions and 48 deletions.
41 changes: 22 additions & 19 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ Provide a language template description using [https://golang.org/pkg/text/templ

The **Files to share between pipeline stages** input specifies the files meant to be intermediate files shared between the Pipeline Stages. When uploading the Pipeline intermediate files, you must assign environment variable keys to them in the **Files to share between pipeline stages** input.
The inputs `path:env_key` values will be saved together with the file and later automatically reconstructed by the [Pull Pipeline intermediate files Step](https://www.bitrise.io/integrations/steps/pull-intermediate-files).
You can use a shorthand of just `$env_var` for `$env_var:env_var`, when the `$env_var` holds the path to the file(s) you want to share with subsequent stages.
The directories you specify will be archived and uploaded as a single file.

#### Configuring the Debug section of the Step
Expand All @@ -62,11 +63,13 @@ Bundletool generates an APK from an Android App Bundle so that you can test the

- [Deployment on Bitrise](https://devcenter.bitrise.io/deploy/deployment-index/)
- [Watching an app](https://devcenter.bitrise.io/builds/configuring-notifications/#watching-an-app)
- [Using artifacts from different Stages](https://devcenter.bitrise.io/en/builds/build-pipelines/configuring-a-bitrise-pipeline.html#using-artifacts-from-different-stages)

### Related Steps

- [Deploy to Google Play](https://www.bitrise.io/integrations/steps/google-play-deploy)
- [Deploy to iTunesConnect](https://www.bitrise.io/integrations/steps/deploy-to-itunesconnect-deliver)
- [Pull Pipeline intermediate files](https://www.bitrise.io/integrations/steps/pull-intermediate-files)
</details>

## 🧩 Get started
Expand Down Expand Up @@ -101,13 +104,13 @@ The Step can handle multiple file uploads in one go. In this case the **deploy_p
- deploy-to-bitrise-io:
inputs:
- pipeline_intermediate_files: |-
$BITRISE_IPA_PATH:BITRISE_IPA_PATH
$BITRISE_IPA_PATH
$BITRISE_APK_PATH:DEVELOPMENT_APK_PATH
./path/to/test_reports:TEST_REPORTS_DIR
$BITRISE_SOURCE_DIR/deploy_dir:DEPLOY_DIR
```
The Step supports sharing files between pipeline stages. The input needs to be a newline (`\n`) separated list of file path - env key pairs (`{path}:{env_key}`).
The Step supports sharing files between pipeline stages. The input needs to be a newline (`\n`) separated list of file path - env key pairs (`{path}:{env_key}` or `{env_var}`).
This metadata will be saved with the individual files and restored by the [Pull Pipeline intermediate files Step](https://www.bitrise.io/integrations/steps/pull-intermediate-files).


Expand All @@ -116,23 +119,23 @@ This metadata will be saved with the individual files and restored by the [Pull
<details>
<summary>Inputs</summary>

| Key | Description | Flags | Default |
| --- | --- | --- | --- |
| `deploy_path` | Specify the directory or file path which will be deployed. If the specified path is a directory, then every file in the specified directory, excluding sub-directories, will be deployed. To upload the directory's content recursively, you should use the **Compress the artifacts into one file?** option which compresses the whole directory, with every sub-directory included. If you specify a file path, then only the specified file will be deployed. | | `$BITRISE_DEPLOY_DIR` |
| `is_compress` | If this option is set to `true` and a Deploy directory was specified, the artifacts in that directory will be compressed into a single ZIP file. You can specify a custom name for the ZIP using the `zip_name` option. If you do not specify a custom name, the default `Deploy directory` name will be used. If this option is set to `false`, the artifacts found in the Deploy directory folder will be deployed separately. | required | `false` |
| `zip_name` | If you do not specify a custom name, the Deploy directory name will be used. You can specify a custom name for the ZIP using the `zip_name` option. This option only works if you selected *true* for *is_compress*. | | |
| `notify_user_groups` | Your App's user roles you want to notify. Separate the role names with commas. Possible role names: * none * testers * developers * admins * owner * everyone An example to notify your developers and testers: `testers, developers` If you want to notify everyone in the app's team, just specify `everyone`. If you don't want to notify anyone, set this to `none`. | | `everyone` |
| `notify_email_list` | Email addresses to notify. Separate them with commas. You can specify any email address, the recipients don't have to be in your team. Please note that if the email address is associated with a Bitrise account, the user must be [watching](https://devcenter.bitrise.io/builds/configuring-notifications/#watching-an-app) the app. | sensitive | |
| `is_enable_public_page` | If this option is enabled, a public install page will be available with a long and random URL which can be shared with others who are not registered on Bitrise. If you disable this option, the **Notify: Emails** option will be ignored and the **Notify: User Roles** users will receive the build's URL instead of the public page's URL! | required | `true` |
| `bundletool_version` | If you need a specific [bundletool version]((https://github.com/google/bundletool/releases) other than the default version, you can modify the value of the **Bundletool version** required input. | required | `1.8.1` |
| `build_url` | Unique build URL of this build on Bitrise.io | required | `$BITRISE_BUILD_URL` |
| `build_api_token` | The build's API Token for the build on Bitrise.io | required, sensitive | `$BITRISE_BUILD_API_TOKEN` |
| `pipeline_intermediate_files` | A newline (`\n`) separated list of file path - env key pairs (`{path}:{env_key}`). The input uses a `{path}:{env_key}` syntax. The colon character (`:`) is the delimiter between the file path and the environment variable key. The file path can be specified with environment variables or direct paths, and can point to both a local file or directory: ``` $BITRISE_IPA_PATH:BITRISE_IPA_PATH $BITRISE_APK_PATH:DEVELOPMENT_APK_PATH ./path/to/test_reports:TEST_REPORTS_DIR $BITRISE_SOURCE_DIR/deploy_dir:DEPLOY_DIR ``` | | |
| `addon_api_base_url` | The URL where test API is accessible. | required | `https://vdt.bitrise.io/test` |
| `addon_api_token` | The token required to authenticate with the API. | sensitive | `$ADDON_VDTESTING_API_TOKEN` |
| `public_install_page_url_map_format` | Provide a language template description using [Golang templates](https://golang.org/pkg/text/template) so that the **Deploy to Bitrise.io** Step can build the required custom output. | required | `{{range $index, $element := .}}{{if $index}}\|{{end}}{{$element.File}}=>{{$element.URL}}{{end}}` |
| `permanent_download_url_map_format` | Provide a language template description using [Golang templates](https://golang.org/pkg/text/template) so that the **Deploy to Bitrise.io** Step can build the required custom output for the permanent download URL. | required | `{{range $index, $element := .}}{{if $index}}\|{{end}}{{$element.File}}=>{{$element.URL}}{{end}}` |
| `debug_mode` | The Step will print more verbose logs if enabled. | required | `false` |
| Key | Description | Flags | Default |
| --- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| --- | --- |
| `deploy_path` | Specify the directory or file path which will be deployed. If the specified path is a directory, then every file in the specified directory, excluding sub-directories, will be deployed. To upload the directory's content recursively, you should use the **Compress the artifacts into one file?** option which compresses the whole directory, with every sub-directory included. If you specify a file path, then only the specified file will be deployed. | | `$BITRISE_DEPLOY_DIR` |
| `is_compress` | If this option is set to `true` and a Deploy directory was specified, the artifacts in that directory will be compressed into a single ZIP file. You can specify a custom name for the ZIP using the `zip_name` option. If you do not specify a custom name, the default `Deploy directory` name will be used. If this option is set to `false`, the artifacts found in the Deploy directory folder will be deployed separately. | required | `false` |
| `zip_name` | If you do not specify a custom name, the Deploy directory name will be used. You can specify a custom name for the ZIP using the `zip_name` option. This option only works if you selected *true* for *is_compress*. | | |
| `notify_user_groups` | Your App's user roles you want to notify. Separate the role names with commas. Possible role names: * none * testers * developers * admins * owner * everyone An example to notify your developers and testers: `testers, developers` If you want to notify everyone in the app's team, just specify `everyone`. If you don't want to notify anyone, set this to `none`. | | `everyone` |
| `notify_email_list` | Email addresses to notify. Separate them with commas. You can specify any email address, the recipients don't have to be in your team. Please note that if the email address is associated with a Bitrise account, the user must be [watching](https://devcenter.bitrise.io/builds/configuring-notifications/#watching-an-app) the app. | sensitive | |
| `is_enable_public_page` | If this option is enabled, a public install page will be available with a long and random URL which can be shared with others who are not registered on Bitrise. If you disable this option, the **Notify: Emails** option will be ignored and the **Notify: User Roles** users will receive the build's URL instead of the public page's URL! | required | `true` |
| `bundletool_version` | If you need a specific [bundletool version]((https://github.com/google/bundletool/releases) other than the default version, you can modify the value of the **Bundletool version** required input. | required | `1.8.1` |
| `build_url` | Unique build URL of this build on Bitrise.io | required | `$BITRISE_BUILD_URL` |
| `build_api_token` | The build's API Token for the build on Bitrise.io | required, sensitive | `$BITRISE_BUILD_API_TOKEN` |
| `pipeline_intermediate_files` | A newline (`\n`) separated list of file path - env key pairs (`{path}:{env_key}`). The input uses a `{path}:{env_key}` syntax. The colon character (`:`) is the delimiter between the file path and the environment variable key. A shorthand syntax of `{env_var}` is also available. The file path can be specified with environment variables or direct paths, and can point to both a local file or directory: ``` $BITRISE_IPA_PATH $BITRISE_APK_PATH:DEVELOPMENT_APK_PATH ./path/to/test_reports:TEST_REPORTS_DIR $BITRISE_SOURCE_DIR/deploy_dir:DEPLOY_DIR ``` | | |
| `addon_api_base_url` | The URL where test API is accessible. | required | `https://vdt.bitrise.io/test` |
| `addon_api_token` | The token required to authenticate with the API. | sensitive | `$ADDON_VDTESTING_API_TOKEN` |
| `public_install_page_url_map_format` | Provide a language template description using [Golang templates](https://golang.org/pkg/text/template) so that the **Deploy to Bitrise.io** Step can build the required custom output. | required | `{{range $index, $element := .}}{{if $index}}\|{{end}}{{$element.File}}=>{{$element.URL}}{{end}}` |
| `permanent_download_url_map_format` | Provide a language template description using [Golang templates](https://golang.org/pkg/text/template) so that the **Deploy to Bitrise.io** Step can build the required custom output for the permanent download URL. | required | `{{range $index, $element := .}}{{if $index}}\|{{end}}{{$element.File}}=>{{$element.URL}}{{end}}` |
| `debug_mode` | The Step will print more verbose logs if enabled. | required | `false` |
</details>

<details>
Expand Down
15 changes: 10 additions & 5 deletions bitrise.yml
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
format_version: 5
format_version: "5"
default_step_lib_source: https://github.com/bitrise-io/bitrise-steplib.git

app:
Expand All @@ -17,10 +17,10 @@ app:
workflows:
test:
steps:
- go-list:
- golint:
- errcheck:
- go-test:
- go-list: {}
- golint: {}
- errcheck: {}
- go-test: {}

ci:
before_run:
Expand Down Expand Up @@ -159,6 +159,10 @@ workflows:
- notify_email_list: $NOTIFY_EMAIL_LIST
- is_enable_public_page: "true"
- debug_mode: "true"
- set-env-var:
inputs:
- destination_keys: PIPELINE_FILE_3
- value: "./xcresults/xcresult3-flaky-with-rerun.xcresult"
- path::./:
title: Pipeline intermediate file upload
run_if: true
Expand All @@ -170,6 +174,7 @@ workflows:
- pipeline_intermediate_files: |-
./ipas/ios-simple-objc.ipa:PIPELINE_FILE_1
./xcresults/xcresult3-flaky-with-rerun.xcresult:PIPELINE_FILE_2
$PIPELINE_FILE_3
- notify_user_groups: $NOTIFY_USER_GROUPS
- notify_email_list: $NOTIFY_EMAIL_LIST
- is_enable_public_page: "true"
Expand Down
31 changes: 20 additions & 11 deletions deployment/collector.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -118,19 +118,28 @@ func (c Collector) processIntermediateFiles(s string) (map[string]string, error)
continue
}

if strings.Count(item, separator) != 1 {
return nil, fmt.Errorf("invalid item (%s): doesn't contain exactly one '%s' character", item, separator)
}

idx := strings.LastIndex(item, separator)
path := item[:idx]
if path == "" {
return nil, fmt.Errorf("invalid item (%s): doesn't specify file path", item)
if strings.Count(item, separator) > 1 {
return nil, fmt.Errorf("invalid item (%s): contains more than one '%s' character", item, separator)
}
var key, path string
if strings.Count(item, separator) == 0 {
if item[0] == '$' {
key = item[1:]
} else {
key = item
}
path = item
} else {
idx := strings.LastIndex(item, separator)
path = item[:idx]
if path == "" {
return nil, fmt.Errorf("invalid item (%s): doesn't specify file path", item)
}

key := item[idx+1:]
if key == "" {
return nil, fmt.Errorf("invalid item (%s): doesn't specify key", item)
key = item[idx+1:]
if key == "" {
return nil, fmt.Errorf("invalid item (%s): doesn't specify key", item)
}
}

path, err := filepath.Abs(path)
Expand Down
Loading

0 comments on commit d58ca90

Please sign in to comment.