From d58ca9017d839331edc6b66fd2cf0be5fe73e978 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?B=C3=A1lint?= Date: Thu, 23 Mar 2023 08:00:02 +0100 Subject: [PATCH] feat: shorthand syntax for pipeline intermediate files (#174) * feat: shorthand syntax for pipeline intermediate files * update README.md and step.yml * add e2e test --- README.md | 41 +++++++++++++++------------- bitrise.yml | 15 +++++++---- deployment/collector.go | 31 +++++++++++++-------- deployment/collector_test.go | 52 +++++++++++++++++++++++++++++------- step.yml | 11 +++++--- 5 files changed, 102 insertions(+), 48 deletions(-) diff --git a/README.md b/README.md index ef1ed3c9..c11d4be2 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,7 @@ Provide a language template description using [https://golang.org/pkg/text/templ The **Files to share between pipeline stages** input specifies the files meant to be intermediate files shared between the Pipeline Stages. When uploading the Pipeline intermediate files, you must assign environment variable keys to them in the **Files to share between pipeline stages** input. The inputs `path:env_key` values will be saved together with the file and later automatically reconstructed by the [Pull Pipeline intermediate files Step](https://www.bitrise.io/integrations/steps/pull-intermediate-files). +You can use a shorthand of just `$env_var` for `$env_var:env_var`, when the `$env_var` holds the path to the file(s) you want to share with subsequent stages. The directories you specify will be archived and uploaded as a single file. #### Configuring the Debug section of the Step @@ -62,11 +63,13 @@ Bundletool generates an APK from an Android App Bundle so that you can test the - [Deployment on Bitrise](https://devcenter.bitrise.io/deploy/deployment-index/) - [Watching an app](https://devcenter.bitrise.io/builds/configuring-notifications/#watching-an-app) +- [Using artifacts from different Stages](https://devcenter.bitrise.io/en/builds/build-pipelines/configuring-a-bitrise-pipeline.html#using-artifacts-from-different-stages) ### Related Steps - [Deploy to Google Play](https://www.bitrise.io/integrations/steps/google-play-deploy) - [Deploy to iTunesConnect](https://www.bitrise.io/integrations/steps/deploy-to-itunesconnect-deliver) +- [Pull Pipeline intermediate files](https://www.bitrise.io/integrations/steps/pull-intermediate-files) ## 🧩 Get started @@ -101,13 +104,13 @@ The Step can handle multiple file uploads in one go. In this case the **deploy_p - deploy-to-bitrise-io: inputs: - pipeline_intermediate_files: |- - $BITRISE_IPA_PATH:BITRISE_IPA_PATH + $BITRISE_IPA_PATH $BITRISE_APK_PATH:DEVELOPMENT_APK_PATH ./path/to/test_reports:TEST_REPORTS_DIR $BITRISE_SOURCE_DIR/deploy_dir:DEPLOY_DIR ``` -The Step supports sharing files between pipeline stages. The input needs to be a newline (`\n`) separated list of file path - env key pairs (`{path}:{env_key}`). +The Step supports sharing files between pipeline stages. The input needs to be a newline (`\n`) separated list of file path - env key pairs (`{path}:{env_key}` or `{env_var}`). This metadata will be saved with the individual files and restored by the [Pull Pipeline intermediate files Step](https://www.bitrise.io/integrations/steps/pull-intermediate-files). @@ -116,23 +119,23 @@ This metadata will be saved with the individual files and restored by the [Pull
Inputs -| Key | Description | Flags | Default | -| --- | --- | --- | --- | -| `deploy_path` | Specify the directory or file path which will be deployed. If the specified path is a directory, then every file in the specified directory, excluding sub-directories, will be deployed. To upload the directory's content recursively, you should use the **Compress the artifacts into one file?** option which compresses the whole directory, with every sub-directory included. If you specify a file path, then only the specified file will be deployed. | | `$BITRISE_DEPLOY_DIR` | -| `is_compress` | If this option is set to `true` and a Deploy directory was specified, the artifacts in that directory will be compressed into a single ZIP file. You can specify a custom name for the ZIP using the `zip_name` option. If you do not specify a custom name, the default `Deploy directory` name will be used. If this option is set to `false`, the artifacts found in the Deploy directory folder will be deployed separately. | required | `false` | -| `zip_name` | If you do not specify a custom name, the Deploy directory name will be used. You can specify a custom name for the ZIP using the `zip_name` option. This option only works if you selected *true* for *is_compress*. | | | -| `notify_user_groups` | Your App's user roles you want to notify. Separate the role names with commas. Possible role names: * none * testers * developers * admins * owner * everyone An example to notify your developers and testers: `testers, developers` If you want to notify everyone in the app's team, just specify `everyone`. If you don't want to notify anyone, set this to `none`. | | `everyone` | -| `notify_email_list` | Email addresses to notify. Separate them with commas. You can specify any email address, the recipients don't have to be in your team. Please note that if the email address is associated with a Bitrise account, the user must be [watching](https://devcenter.bitrise.io/builds/configuring-notifications/#watching-an-app) the app. | sensitive | | -| `is_enable_public_page` | If this option is enabled, a public install page will be available with a long and random URL which can be shared with others who are not registered on Bitrise. If you disable this option, the **Notify: Emails** option will be ignored and the **Notify: User Roles** users will receive the build's URL instead of the public page's URL! | required | `true` | -| `bundletool_version` | If you need a specific [bundletool version]((https://github.com/google/bundletool/releases) other than the default version, you can modify the value of the **Bundletool version** required input. | required | `1.8.1` | -| `build_url` | Unique build URL of this build on Bitrise.io | required | `$BITRISE_BUILD_URL` | -| `build_api_token` | The build's API Token for the build on Bitrise.io | required, sensitive | `$BITRISE_BUILD_API_TOKEN` | -| `pipeline_intermediate_files` | A newline (`\n`) separated list of file path - env key pairs (`{path}:{env_key}`). The input uses a `{path}:{env_key}` syntax. The colon character (`:`) is the delimiter between the file path and the environment variable key. The file path can be specified with environment variables or direct paths, and can point to both a local file or directory: ``` $BITRISE_IPA_PATH:BITRISE_IPA_PATH $BITRISE_APK_PATH:DEVELOPMENT_APK_PATH ./path/to/test_reports:TEST_REPORTS_DIR $BITRISE_SOURCE_DIR/deploy_dir:DEPLOY_DIR ``` | | | -| `addon_api_base_url` | The URL where test API is accessible. | required | `https://vdt.bitrise.io/test` | -| `addon_api_token` | The token required to authenticate with the API. | sensitive | `$ADDON_VDTESTING_API_TOKEN` | -| `public_install_page_url_map_format` | Provide a language template description using [Golang templates](https://golang.org/pkg/text/template) so that the **Deploy to Bitrise.io** Step can build the required custom output. | required | `{{range $index, $element := .}}{{if $index}}\|{{end}}{{$element.File}}=>{{$element.URL}}{{end}}` | -| `permanent_download_url_map_format` | Provide a language template description using [Golang templates](https://golang.org/pkg/text/template) so that the **Deploy to Bitrise.io** Step can build the required custom output for the permanent download URL. | required | `{{range $index, $element := .}}{{if $index}}\|{{end}}{{$element.File}}=>{{$element.URL}}{{end}}` | -| `debug_mode` | The Step will print more verbose logs if enabled. | required | `false` | +| Key | Description | Flags | Default | +| --- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| --- | --- | +| `deploy_path` | Specify the directory or file path which will be deployed. If the specified path is a directory, then every file in the specified directory, excluding sub-directories, will be deployed. To upload the directory's content recursively, you should use the **Compress the artifacts into one file?** option which compresses the whole directory, with every sub-directory included. If you specify a file path, then only the specified file will be deployed. | | `$BITRISE_DEPLOY_DIR` | +| `is_compress` | If this option is set to `true` and a Deploy directory was specified, the artifacts in that directory will be compressed into a single ZIP file. You can specify a custom name for the ZIP using the `zip_name` option. If you do not specify a custom name, the default `Deploy directory` name will be used. If this option is set to `false`, the artifacts found in the Deploy directory folder will be deployed separately. | required | `false` | +| `zip_name` | If you do not specify a custom name, the Deploy directory name will be used. You can specify a custom name for the ZIP using the `zip_name` option. This option only works if you selected *true* for *is_compress*. | | | +| `notify_user_groups` | Your App's user roles you want to notify. Separate the role names with commas. Possible role names: * none * testers * developers * admins * owner * everyone An example to notify your developers and testers: `testers, developers` If you want to notify everyone in the app's team, just specify `everyone`. If you don't want to notify anyone, set this to `none`. | | `everyone` | +| `notify_email_list` | Email addresses to notify. Separate them with commas. You can specify any email address, the recipients don't have to be in your team. Please note that if the email address is associated with a Bitrise account, the user must be [watching](https://devcenter.bitrise.io/builds/configuring-notifications/#watching-an-app) the app. | sensitive | | +| `is_enable_public_page` | If this option is enabled, a public install page will be available with a long and random URL which can be shared with others who are not registered on Bitrise. If you disable this option, the **Notify: Emails** option will be ignored and the **Notify: User Roles** users will receive the build's URL instead of the public page's URL! | required | `true` | +| `bundletool_version` | If you need a specific [bundletool version]((https://github.com/google/bundletool/releases) other than the default version, you can modify the value of the **Bundletool version** required input. | required | `1.8.1` | +| `build_url` | Unique build URL of this build on Bitrise.io | required | `$BITRISE_BUILD_URL` | +| `build_api_token` | The build's API Token for the build on Bitrise.io | required, sensitive | `$BITRISE_BUILD_API_TOKEN` | +| `pipeline_intermediate_files` | A newline (`\n`) separated list of file path - env key pairs (`{path}:{env_key}`). The input uses a `{path}:{env_key}` syntax. The colon character (`:`) is the delimiter between the file path and the environment variable key. A shorthand syntax of `{env_var}` is also available. The file path can be specified with environment variables or direct paths, and can point to both a local file or directory: ``` $BITRISE_IPA_PATH $BITRISE_APK_PATH:DEVELOPMENT_APK_PATH ./path/to/test_reports:TEST_REPORTS_DIR $BITRISE_SOURCE_DIR/deploy_dir:DEPLOY_DIR ``` | | | +| `addon_api_base_url` | The URL where test API is accessible. | required | `https://vdt.bitrise.io/test` | +| `addon_api_token` | The token required to authenticate with the API. | sensitive | `$ADDON_VDTESTING_API_TOKEN` | +| `public_install_page_url_map_format` | Provide a language template description using [Golang templates](https://golang.org/pkg/text/template) so that the **Deploy to Bitrise.io** Step can build the required custom output. | required | `{{range $index, $element := .}}{{if $index}}\|{{end}}{{$element.File}}=>{{$element.URL}}{{end}}` | +| `permanent_download_url_map_format` | Provide a language template description using [Golang templates](https://golang.org/pkg/text/template) so that the **Deploy to Bitrise.io** Step can build the required custom output for the permanent download URL. | required | `{{range $index, $element := .}}{{if $index}}\|{{end}}{{$element.File}}=>{{$element.URL}}{{end}}` | +| `debug_mode` | The Step will print more verbose logs if enabled. | required | `false` |
diff --git a/bitrise.yml b/bitrise.yml index 37d30854..1297e84d 100644 --- a/bitrise.yml +++ b/bitrise.yml @@ -1,4 +1,4 @@ -format_version: 5 +format_version: "5" default_step_lib_source: https://github.com/bitrise-io/bitrise-steplib.git app: @@ -17,10 +17,10 @@ app: workflows: test: steps: - - go-list: - - golint: - - errcheck: - - go-test: + - go-list: {} + - golint: {} + - errcheck: {} + - go-test: {} ci: before_run: @@ -159,6 +159,10 @@ workflows: - notify_email_list: $NOTIFY_EMAIL_LIST - is_enable_public_page: "true" - debug_mode: "true" + - set-env-var: + inputs: + - destination_keys: PIPELINE_FILE_3 + - value: "./xcresults/xcresult3-flaky-with-rerun.xcresult" - path::./: title: Pipeline intermediate file upload run_if: true @@ -170,6 +174,7 @@ workflows: - pipeline_intermediate_files: |- ./ipas/ios-simple-objc.ipa:PIPELINE_FILE_1 ./xcresults/xcresult3-flaky-with-rerun.xcresult:PIPELINE_FILE_2 + $PIPELINE_FILE_3 - notify_user_groups: $NOTIFY_USER_GROUPS - notify_email_list: $NOTIFY_EMAIL_LIST - is_enable_public_page: "true" diff --git a/deployment/collector.go b/deployment/collector.go index e35836e0..7b3fffc7 100644 --- a/deployment/collector.go +++ b/deployment/collector.go @@ -118,19 +118,28 @@ func (c Collector) processIntermediateFiles(s string) (map[string]string, error) continue } - if strings.Count(item, separator) != 1 { - return nil, fmt.Errorf("invalid item (%s): doesn't contain exactly one '%s' character", item, separator) - } - - idx := strings.LastIndex(item, separator) - path := item[:idx] - if path == "" { - return nil, fmt.Errorf("invalid item (%s): doesn't specify file path", item) + if strings.Count(item, separator) > 1 { + return nil, fmt.Errorf("invalid item (%s): contains more than one '%s' character", item, separator) } + var key, path string + if strings.Count(item, separator) == 0 { + if item[0] == '$' { + key = item[1:] + } else { + key = item + } + path = item + } else { + idx := strings.LastIndex(item, separator) + path = item[:idx] + if path == "" { + return nil, fmt.Errorf("invalid item (%s): doesn't specify file path", item) + } - key := item[idx+1:] - if key == "" { - return nil, fmt.Errorf("invalid item (%s): doesn't specify key", item) + key = item[idx+1:] + if key == "" { + return nil, fmt.Errorf("invalid item (%s): doesn't specify key", item) + } } path, err := filepath.Abs(path) diff --git a/deployment/collector_test.go b/deployment/collector_test.go index b550c4c3..f3c904fd 100644 --- a/deployment/collector_test.go +++ b/deployment/collector_test.go @@ -105,6 +105,34 @@ func Test_GivenIntermediateFiles_WhenProcessing_ThenConvertsCorrectly(t *testing }, wantErr: false, }, + { + name: "shorthand syntax when path value from env var, same as exported env var", + list: "$BITRISE_IPA_PATH", + want: []DeployableItem{ + { + Path: filepath.Join(currentDir, "$BITRISE_IPA_PATH"), + IntermediateFileMeta: &IntermediateFileMetaData{ + EnvKey: "BITRISE_IPA_PATH", + IsDir: false, + }, + }, + }, + wantErr: false, + }, + { + name: "shorthand syntax when path value is the same as exported env var", + list: "my_binary", + want: []DeployableItem{ + { + Path: filepath.Join(currentDir, "my_binary"), + IntermediateFileMeta: &IntermediateFileMetaData{ + EnvKey: "my_binary", + IsDir: false, + }, + }, + }, + wantErr: false, + }, { name: "Item is empty", list: "", @@ -282,18 +310,22 @@ func Test_GivenDeployDirectories_WhenIntermediateDirectoriesSpecified_ThenMerges zips := map[string][]*zip.File{ "/dir1234.zip": { - {FileHeader: zip.FileHeader{ - Name: "test_file.txt", - CRC32: 0xb095e5e3, - UncompressedSize64: 12, - }}, + { + FileHeader: zip.FileHeader{ + Name: "test_file.txt", + CRC32: 0xb095e5e3, + UncompressedSize64: 12, + }, + }, }, filepath.Join(tempDir, "dir.zip"): { - {FileHeader: zip.FileHeader{ - Name: "test_file.txt", - CRC32: 0xb095e5e3, - UncompressedSize64: 12, - }}, + { + FileHeader: zip.FileHeader{ + Name: "test_file.txt", + CRC32: 0xb095e5e3, + UncompressedSize64: 12, + }, + }, }, } diff --git a/step.yml b/step.yml index 1f740562..83b72829 100644 --- a/step.yml +++ b/step.yml @@ -34,7 +34,8 @@ description: |- ### Configuring the Pipeline Intermediate File Sharing section of the Step The **Files to share between pipeline stages** input specifies the files meant to be intermediate files shared between the Pipeline Stages. When uploading the Pipeline intermediate files, you must assign environment variable keys to them in the **Files to share between pipeline stages** input. - The inputs `path:env_key` values will be saved together with the file and later automatically reconstructed by the [Pull Pipeline intermediate files Step](https://www.bitrise.io/integrations/steps/pull-intermediate-files). + The inputs `path:env_key` values will be saved together with the file and later automatically reconstructed by the [Pull Pipeline intermediate files Step](https://www.bitrise.io/integrations/steps/pull-intermediate-files). + You can use a shorthand of just `$env_var` for `$env_var:env_var`, when the `$env_var` holds the path to the file(s) you want to share with subsequent stages. The directories you specify will be archived and uploaded as a single file. #### Configuring the Debug section of the Step @@ -57,11 +58,13 @@ description: |- - [Deployment on Bitrise](https://devcenter.bitrise.io/deploy/deployment-index/) - [Watching an app](https://devcenter.bitrise.io/builds/configuring-notifications/#watching-an-app) - - ### Related Steps + - [Using artifacts from different Stages](https://devcenter.bitrise.io/en/builds/build-pipelines/configuring-a-bitrise-pipeline.html#using-artifacts-from-different-stages) + + ### Related Steps - [Deploy to Google Play](https://www.bitrise.io/integrations/steps/google-play-deploy) - [Deploy to iTunesConnect](https://www.bitrise.io/integrations/steps/deploy-to-itunesconnect-deliver) + - [Pull Pipeline intermediate files](https://www.bitrise.io/integrations/steps/pull-intermediate-files) website: https://github.com/bitrise-steplib/steps-deploy-to-bitrise-io source_code_url: https://github.com/bitrise-steplib/steps-deploy-to-bitrise-io support_url: https://github.com/bitrise-steplib/steps-deploy-to-bitrise-io/issues @@ -222,6 +225,7 @@ inputs: The input uses a `{path}:{env_key}` syntax. The colon character (`:`) is the delimiter between the file path and the environment variable key. + A shorthand syntax of `$ENV_VAR` can be used for `$ENV_VAR:ENV_VAR` when the name of the env var in the current workflow will become the shared env_key. The file path can be specified with environment variables or direct paths, and can point to both a local file or directory: @@ -230,6 +234,7 @@ inputs: $BITRISE_APK_PATH:DEVELOPMENT_APK_PATH ./path/to/test_reports:TEST_REPORTS_DIR $BITRISE_SOURCE_DIR/deploy_dir:DEPLOY_DIR + $BITRISE_IPA_PATH ``` - addon_api_base_url: "https://vdt.bitrise.io/test" opts: