Skip to content

Commit

Permalink
update release scripts to release to Apache DataFusion not Apache Arr…
Browse files Browse the repository at this point in the history 
…ow (#1163)

* update release scripts to release to Apache DataFusion not Apache Arrow

* more updates
  • Loading branch information
@andygrove
andygrove authored Jan 16, 2025
1 parent 9051983 commit 9c9ca0d
Show file tree
Hide file tree
Showing 4 changed files with 67 additions and 98 deletions.
84 changes: 27 additions & 57 deletions dev/release/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,9 @@ Although some tasks can only be performed by a PMC member, many tasks can be per

### Post-Release

| Task | Role Required |
| ------------------------------------------------------- | ------------- |
| Create PR against arrow-site with updated documentation | None |
| Task | Role Required |
|--------------------------------------------------------------| ------------- |
| Create PR against datafusion-site with updated documentation | None |

## Detailed Guide

Expand All @@ -71,7 +71,7 @@ Although some tasks can only be performed by a PMC member, many tasks can be per
- You will need a GitHub Personal Access Token with "repo" access. Follow
[these instructions](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)
to generate one if you do not already have one.
- Have upstream git repo `[email protected]:apache/arrow-ballista.git` add as git remote `apache`.
- Have upstream git repo `[email protected]:apache/datafusion-ballista.git` add as git remote `apache`.

### Preparing the `main` Branch

Expand Down Expand Up @@ -102,7 +102,7 @@ pip3 install PyGitHub
Run the following command to generate the changelog content.

```bash
$ GITHUB_TOKEN=<TOKEN> ./dev/release/generate-changelog.py apache/arrow-ballista 0.11.0 HEAD > 0.12.0.md
$ GITHUB_TOKEN=<TOKEN> ./dev/release/generate-changelog.py apache/datafusion-ballista 0.11.0 HEAD > 0.12.0.md
```

This script creates a changelog from GitHub PRs based on the labels associated with them as well as looking for
Expand All @@ -121,9 +121,9 @@ This process is not fully automated, so there are some additional manual steps:
- Add the following content (copy from the previous version's changelog and update as appropriate:

```
## [0.12.0](https://github.com/apache/arrow-ballista/tree/0.12.0) (2024-01-14)
## [0.12.0](https://github.com/apache/datafusion-ballista/tree/0.12.0) (2024-01-14)
[Full Changelog](https://github.com/apache/arrow-ballista/compare/0.11.0...0.12.0)
[Full Changelog](https://github.com/apache/datafusion-ballista/compare/0.11.0...0.12.0)
```

Send a PR to get these changes merged into the release branch (e.g. `branch-0.12`). If new commits that could change the
Expand Down Expand Up @@ -157,16 +157,16 @@ git push apache <version>
### Create, sign, and upload artifacts

- Make sure your signing key is added to the following files in SVN:
- https://dist.apache.org/repos/dist/dev/arrow/KEYS
- https://dist.apache.org/repos/dist/release/arrow/KEYS
- https://dist.apache.org/repos/dist/dev/datafusion/KEYS
- https://dist.apache.org/repos/dist/release/datafusion/KEYS

See instructions at https://infra.apache.org/release-signing.html#generate for generating keys.

Committers can add signing keys in Subversion client with their ASF account. e.g.:

```bash
$ svn co https://dist.apache.org/repos/dist/dev/arrow
$ cd arrow
$ svn co https://dist.apache.org/repos/dist/dev/datafusion
$ cd datafusion
$ editor KEYS
$ svn ci KEYS
```
Expand All @@ -186,43 +186,17 @@ GH_TOKEN=<TOKEN> ./dev/release/create-tarball.sh 0.11.0 1

The `create-tarball.sh` script

1. creates and uploads all release candidate artifacts to the [arrow
dev](https://dist.apache.org/repos/dist/dev/arrow) location on the
1. creates and uploads all release candidate artifacts to the [datafusion
dev](https://dist.apache.org/repos/dist/dev/datafusion) location on the
apache distribution svn server

2. provide you an email template to
send to dev@arrow.apache.org for release voting.
send to dev@datafusion.apache.org for release voting.

### Vote on Release Candidate artifacts

Send the email output from the script to dev@arrow.apache.org. The email should look like
Send the email output from the script to dev@datafusion.apache.org.

```
To: [email protected]
Subject: [VOTE][Ballista] Release Apache Arrow Ballista 0.8.0 RC0
Hi,
I would like to propose a release of Apache Arrow Ballista Implementation,
version 0.8.0.
This release candidate is based on commit: a5dd428f57e62db20a945e8b1895de91405958c4 [1]
The proposed release artifacts and signatures are hosted at [2].
The changelog is located at [3].
Please download, verify checksums and signatures, run the unit tests,
and vote on the release.
The vote will be open for at least 72 hours.
[ ] +1 Release this as Apache Arrow Ballista 0.8.0
[ ] +0
[ ] -1 Do not release this as Apache Arrow Ballista 0.8.0 because...
[1]: https://github.com/apache/arrow-ballista/tree/a5dd428f57e62db20a945e8b1895de91405958c4
[2]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-ballista-0.8.0
[3]: https://github.com/apache/arrow-ballista/blob/a5dd428f57e62db20a945e8b1895de91405958c4/CHANGELOG.md
```

For the release to become "official" it needs at least three PMC members to vote +1 on it.

Expand All @@ -246,7 +220,7 @@ NOTE: steps in this section can only be done by PMC members.
### After the release is approved

Move artifacts to the release location in SVN, e.g.
https://dist.apache.org/repos/dist/release/arrow/arrow-ballista-0.8.0/, using
https://dist.apache.org/repos/dist/release/datafusion/datafusion-ballista-0.8.0/, using
the `release-tarball.sh` script:

```shell
Expand All @@ -271,7 +245,7 @@ Only approved releases of the tarball should be published to
crates.io, in order to conform to Apache Software Foundation
governance standards.

An Arrow committer can publish this crate after an official project release has
A DataFusion committer can publish this crate after an official project release has
been made to crates.io using the following instructions.

Follow [these
Expand Down Expand Up @@ -311,11 +285,11 @@ dot -Tsvg dev/release/crate-deps.dot > dev/release/crate-deps.svg

Pushing a release tag causes Docker images to be published.

Images can be found at [https://github.com/apache/arrow-ballista/pkgs/container/arrow-ballista-standalone](https://github.com/apache/arrow-ballista/pkgs/container/arrow-ballista-standalone)
Images can be found at [https://github.com/apache/datafusion-ballista/pkgs/container/datafusion-ballista-standalone](https://github.com/apache/datafusion-ballista/pkgs/container/datafusion-ballista-standalone)

### Call the vote

Call the vote on the Arrow dev list by replying to the RC voting thread. The
Call the vote on the DataFusion dev list by replying to the RC voting thread. The
reply should have a new subject constructed by adding `[RESULT]` prefix to the
old subject line.

Expand All @@ -328,8 +302,8 @@ with the release verification.

### Add the release to Apache Reporter

Add the release to https://reporter.apache.org/addrelease.html?arrow with a version name prefixed with `RS-BALLISTA-`,
for example `RS-BALLISTA-0.9.0`.
Add the release to https://reporter.apache.org/addrelease.html?datafusion with a version name prefixed with `BALLISTA-`,
for example `BALLISTA-0.9.0`.

The release information is used to generate a template for a board report (see example
[here](https://github.com/apache/arrow/pull/14357)).
Expand All @@ -346,13 +320,13 @@ Release candidates should be deleted once the release is published.
Get a list of Ballista release candidates:

```bash
svn ls https://dist.apache.org/repos/dist/dev/arrow | grep ballista
svn ls https://dist.apache.org/repos/dist/dev/datafusion | grep ballista
```

Delete a release candidate:

```bash
svn delete -m "delete old Ballista RC" https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-ballista-0.8.0-rc1/
svn delete -m "delete old Ballista RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-ballista-0.8.0-rc1/
```

#### Deleting old releases from `release` svn
Expand All @@ -362,13 +336,13 @@ Only the latest release should be available. Delete old releases after publishin
Get a list of Ballista releases:

```bash
svn ls https://dist.apache.org/repos/dist/release/arrow | grep ballista
svn ls https://dist.apache.org/repos/dist/release/datafusion | grep ballista
```

Delete a release:

```bash
svn delete -m "delete old Ballista release" https://dist.apache.org/repos/dist/release/arrow/arrow-ballista-0.8.0
svn delete -m "delete old Ballista release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-ballista-0.8.0
```

### Optional: Write a blog post announcing the release
Expand All @@ -384,11 +358,7 @@ git shortlog -sn 0.11.0..0.10.0 ballista ballista-cli examples | wc -l
```

Once there is consensus on the contents of the post, create a PR to add a blog post to the
[arrow-site](https://github.com/apache/arrow-site) repository. Note that there is no need for a formal
[datafusion-site](https://github.com/apache/datafusion-site) repository. Note that there is no need for a formal
PMC vote on the blog post contents since this isn't considered to be a "release".

Here is an example blog post PR:

- https://github.com/apache/arrow-site/pull/217

Once the PR is merged, a GitHub action will publish the new blog post to https://arrow.apache.org/blog/.
Once the PR is merged, a GitHub action will publish the new blog post to https://datafusion.apache.org/blog/.
35 changes: 17 additions & 18 deletions dev/release/create-tarball.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@
# Adapted from https://github.com/apache/arrow-rs/tree/master/dev/release/create-tarball.sh

# This script creates a signed tarball in
# dev/dist/apache-arrow-ballista-<version>-<sha>.tar.gz and uploads it to
# the "dev" area of the dist.apache.arrow repository and prepares an
# email for sending to the dev@arrow.apache.org list for a formal
# dev/dist/apache-datafusion-ballista-<version>-<sha>.tar.gz and uploads it to
# the "dev" area of the dist.apache.datafusion repository and prepares an
# email for sending to the dev@datafusion.apache.org list for a formal
# vote.
#
# See release/README.md for full release instructions
Expand Down Expand Up @@ -65,26 +65,25 @@ tag="${version}-rc${rc}"
echo "Attempting to create ${tarball} from tag ${tag}"
release_hash=$(cd "${SOURCE_TOP_DIR}" && git rev-list --max-count=1 ${tag})

release=apache-arrow-ballista-${version}
release=apache-datafusion-ballista-${version}
distdir=${SOURCE_TOP_DIR}/dev/dist/${release}-rc${rc}
tarname=${release}.tar.gz
tarball=${distdir}/${tarname}
url="https://dist.apache.org/repos/dist/dev/arrow/${release}-rc${rc}"
url="https://dist.apache.org/repos/dist/dev/datafusion/${release}-rc${rc}"

if [ -z "$release_hash" ]; then
echo "Cannot continue: unknown git tag: ${tag}"
fi

echo "Draft email for dev@arrow.apache.org mailing list"
echo "Draft email for dev@datafusion.apache.org mailing list"
echo ""
echo "---------------------------------------------------------"
cat <<MAIL
To: dev@arrow.apache.org
Subject: [VOTE][RUST][Ballista] Release Apache Arrow Ballista ${version} RC${rc}
To: dev@datafusion.apache.org
Subject: [VOTE][RUST][Ballista] Release Apache DataFusion Ballista ${version} RC${rc}
Hi,
I would like to propose a release of Apache Arrow Ballista Implementation,
version ${version}.
I would like to propose a release of Apache DataFusion Ballista version ${version}.
This release candidate is based on commit: ${release_hash} [1]
The proposed release tarball and signatures are hosted at [2].
Expand All @@ -96,15 +95,15 @@ on the release. The vote will be open for at least 72 hours.
Only votes from PMC members are binding, but all members of the community are
encouraged to test the release and vote with "(non-binding)".
The standard verification procedure is documented at https://github.com/apache/arrow-ballista/blob/main/dev/release/README.md#verifying-release-candidates.
The standard verification procedure is documented at https://github.com/apache/datafusion-ballista/blob/main/dev/release/README.md#verifying-release-candidates.
[ ] +1 Release this as Apache Arrow Ballista ${version}
[ ] +1 Release this as Apache DataFusion Ballista ${version}
[ ] +0
[ ] -1 Do not release this as Apache Arrow Ballista ${version} because...
[ ] -1 Do not release this as Apache DataFusion Ballista ${version} because...
[1]: https://github.com/apache/arrow-ballista/tree/${release_hash}
[1]: https://github.com/apache/datafusion-ballista/tree/${release_hash}
[2]: ${url}
[3]: https://github.com/apache/arrow-ballista/blob/${release_hash}/CHANGELOG.md
[3]: https://github.com/apache/datafusion-ballista/blob/${release_hash}/CHANGELOG.md
MAIL
echo "---------------------------------------------------------"

Expand All @@ -121,12 +120,12 @@ echo "Signing tarball and creating checksums"
gpg --armor --output ${tarball}.asc --detach-sig ${tarball}
# create signing with relative path of tarball
# so that they can be verified with a command such as
# shasum --check apache-arrow-ballista-4.1.0-rc2.tar.gz.sha512
# shasum --check apache-datafusion-ballista-4.1.0-rc2.tar.gz.sha512
(cd ${distdir} && shasum -a 256 ${tarname}) > ${tarball}.sha256
(cd ${distdir} && shasum -a 512 ${tarname}) > ${tarball}.sha512


echo "Uploading to apache dist/dev to ${url}"
svn co --depth=empty https://dist.apache.org/repos/dist/dev/arrow ${SOURCE_TOP_DIR}/dev/dist
svn co --depth=empty https://dist.apache.org/repos/dist/dev/datafusion ${SOURCE_TOP_DIR}/dev/dist
svn add ${distdir}
svn ci -m "Apache Arrow Ballista ${version} ${rc}" ${distdir}
svn ci -m "Apache DataFusion Ballista ${version} ${rc}" ${distdir}
16 changes: 8 additions & 8 deletions dev/release/release-tarball.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,10 @@
# Adapted from https://github.com/apache/arrow-rs/tree/master/dev/release/release-tarball.sh

# This script copies a tarball from the "dev" area of the
# dist.apache.arrow repository to the "release" area
# dist.apache.datafusion repository to the "release" area
#
# This script should only be run after the release has been approved
# by the arrow PMC committee.
# by the DataFusion PMC committee.
#
# See release/README.md for full release instructions
#
Expand All @@ -43,7 +43,7 @@ fi
version=$1
rc=$2

tmp_dir=tmp-apache-arrow-ballista-dist
tmp_dir=tmp-apache-datafusion-ballista-dist

echo "Recreate temporary directory: ${tmp_dir}"
rm -rf ${tmp_dir}
Expand All @@ -52,23 +52,23 @@ mkdir -p ${tmp_dir}
echo "Clone dev dist repository"
svn \
co \
https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-ballista-${version}-rc${rc} \
https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-ballista-${version}-rc${rc} \
${tmp_dir}/dev

echo "Clone release dist repository"
svn co https://dist.apache.org/repos/dist/release/arrow ${tmp_dir}/release
svn co https://dist.apache.org/repos/dist/release/datafusion ${tmp_dir}/release

echo "Copy ${version}-rc${rc} to release working copy"
release_version=arrow-ballista-${version}
release_version=datafusion-ballista-${version}
mkdir -p ${tmp_dir}/release/${release_version}
cp -r ${tmp_dir}/dev/* ${tmp_dir}/release/${release_version}/
svn add ${tmp_dir}/release/${release_version}

echo "Commit release"
svn ci -m "Apache Arrow Ballista ${version}" ${tmp_dir}/release
svn ci -m "Apache DataFusion Ballista ${version}" ${tmp_dir}/release

echo "Clean up"
rm -rf ${tmp_dir}

echo "Success! The release is available here:"
echo " https://dist.apache.org/repos/dist/release/arrow/${release_version}"
echo " https://dist.apache.org/repos/dist/release/datafusion/${release_version}"
Loading

0 comments on commit 9c9ca0d

Please sign in to comment.