Reproducible Builds

[IzzySoft]

Your app was confirmed to be a Reproducible Build with our rbtlog – until the current version, which we cannot build anymore as you've switched to JDK-22. That version is not supported by any stable/LTS release (like Debian Bookworm or Ubuntu Jammy) – but to our knowledge only in Trixie, which is not a stable target.

Would it be an option for you to use JDK-21 instead? That we could easily support. Using Trixie might break anytime, as it's still "in flux" and not a stable/LTS release.

Thanks a lot in advance!

[Semper-Viventem]

Hey @IzzySoft, nice to see you again. I see there is a build config for each app in the repo. would it be possible to add trixie jdk 22 separately for MetaRadar? I did it for the upstream F-Droid data yesterday.

Also, I have a separate build config for FDroid because it requires the internet connection to be disabled by default, maybe we can create a dedicated build variant for your repo?

[obfusk]

It is possible to use trixie, yes. But trixie is a moving target. It currently has multiple JDK versions (and sid has even more) but it almost certainly will not ship with all of them when it's released. We don't know if OpenJDK 22 will still be there later. And we'd like to avoid having a build recipe that works today but might break any time in the (near) future.

F-Droid only cares that the build works for them, today. It doesn't matter to them whether someone else can reproduce the results later (good luck reproducing any f-droid build from before their switch from bullseye to bookworm, let alone builds on stretch). But we want people to be able to independently verify our results. And not just today, but also next month or even next year. Which means we really want to avoid using a setup that cannot be easily replicated and could break any time.

Do you need JDK 22 specifically? Is JDK 21 not an option for some reason? Because we could easily accommodate that.

[IzzySoft]

I wonder that F-Droid accepted that, as in the past they always insisted that also older builds need to be kept reproducible for potential later rebuilds. Though as Fay just pointed out, that deviated from how they did it as in hindsight you could hardly tell what JDK something was build with or even on what Debian version.

maybe we can create a dedicated build variant for your repo?

If that would make it easier for you, why not? We'd then just need to know what APK to fetch (by file pattern regex, e.g. /iod-release/i).

[obfusk]

FYI, the tag for v0.26.1-beta points to commit 42993e2 but the commit embedded in the APK is b8e58ce so it will fail to be reproducible.

As Izzy says:

§1: Always build the APK from exactly the commit your release-tag will point to!

[IzzySoft]

@Semper-Viventem so what's the status here? Currently I have to reset the recipe at each update as we cannot build it. If it takes longer until we find a solution, I'd disable the RB recipe and wait for your ping here.

[obfusk]

FYI: OpenJDK 21 is an LTS release, 22-24 are not; trixie will likely ship with OpenJDK 21 only. OpenJDK 22-24 have just been proposed to be removed from trixie. F-Droid's recipe will break when that happens (and sid is too unstable to reliably use as ongoing transitions often make packages temporarily uninstallable; moving targets are also a very bad choice for reproducibility). This is exactly why we didn't want to use OpenJDK 22 from trixie. Could you use OpenJDK 21 instead of 22?

[IzzySoft]

@Semper-Viventem so what are your plans here? Until today, I've reset the run for your app each time, hoping and waiting for some progress here. As outlined multiple times, JDK22 is no option here; the latest stable we can offer is JDK21 as that's the latest LTS.

I'll stop resetting the recipe tomorrow, as I see no way in getting the current release RB – so it will be marked "RB failed" then. Still have hopes we find a solution for the next release 🤞

[Semper-Viventem]

Hey @IzzySoft , I'll downgrade JDK to 21 LTS this week. Sorry for the late reply.

[IzzySoft]

Thanks, that's good news! Won't help the current release though I'm afraid (APK is already distributed for a week now, not a good idea to replace it). So may I suggest I let the current version fail tomorrow, and we "heal" it with the next then? It won't be a "hard fail" (aka "RB failed") anyway, just a concrete-hard (:stuck_out_tongue_winking_eye:) real build fail, so RB status will show "we tried but could not build".

[0000FlyBear0000]

а где робототехника?

[Semper-Viventem]

Hey @IzzySoft, I'm sorry for making you wait. I introduced a new way to setup the JDK version for the project. Now it is possible to setup target JDK from environment variable JAVA_CONFIG_VERSION during the configuration build stage. Changes are released since this PR. I think it will require to support that env variable on your side to setup JAVA_CONFIG_VERSION: 21, but I'm not sure about the recipes config syntax

[IzzySoft]

This is the block from the failed JDK-22 release:

  - tag: v0.26.1-beta
    apks:
      - apk_pattern: app-github-release\.apk
        apk_url: https://github.com/Semper-Viventem/MetaRadar/releases/download/$$TAG$$/app-github-release.apk
        build:
          - ./gradlew assembleGithubRelease
          - mv app/build/outputs/apk/github/release/app-github-release-unsigned.apk /outputs/unsigned.apk
        build_home_dir: /build
        build_repo_dir: /build/repo
        build_user: build
        provisioning:
          android_home: /opt/sdk
          build_tools:
          cmake:
          cmdline_tools:
            version: '12.0'
            url: https://dl.google.com/android/repository/commandlinetools-linux-11076708_latest.zip
            sha256: 2d2d50857e4eb553af5a6dc3ad507a17adf43d115264b1afc116f95c92e5e258
          extra_packages: []
          image: debian:bookworm-slim
          jdk: openjdk-17-jdk-headless
          ndk:
          platform:
          platform_tools:
          tools:
          verify_gradle_wrapper: true

So you see we have jdk: for providing the JDK. We can set that to 21 when switching to image: ubuntu:jammy. And if you need a variable set then, it would go into build: as the very first step then:

        build:
          - export JAVA_CONFIG_VERSION=21
          - ./gradlew assembleGithubRelease
          - mv app/build/outputs/apk/github/release/app-github-release-unsigned.apk /outputs/unsigned.apk

So yeah, totally doable.

[obfusk]

I'm not sure how that helps? We could have patched the source to allow OpenJDK 21. But for Reproducible Builds both builds need to use the same JDK. You seem to be using OpenJDK 22 still. Allowing us to use OpenJDK 21 more easily doesn't fix the mismatch.

[obfusk]

In this particular case there seem to be zero differences in the generated DEX files (though that is not something that can be relied on to always be the case). But the build is still not reproducible since kotlin-tooling-metadata.json contains different values.

[obfusk]

It would be a lot easier and unlikely to break in the future if you could simply use OpenJDK 21 for your builds. As I asked before: is that not an option for some reason?

[obfusk]

Also: the embedded commit hash is once again not the commit the tag points to.

[obfusk]

The good news it that it can be made reproducible. At least for this release. But using a different JDK is far from ideal and could easily break if OpenJDK 21 and 22 do generate different code after something changes (some apps have the same DEX files with 11 and 17 but most don't). And if they always produce identical results (apart from the metadata JSON), I don't see why you can't use OpenJDK 21 as well.

          - git reset --soft f143c1d894bab55a62dea9404f5d5c7db89c2a21
          - JAVA_CONFIG_VERSION=21 ./gradlew assembleGithubRelease
          - OUT=app/build/outputs/apk/github/release/app-github-release-unsigned.apk
          - git clone -b v0.2.8 https://github.com/obfusk/reproducible-apk-tools.git
          - reproducible-apk-tools/inplace-fix.py --internal --zipalign fix-files "$OUT" 'sed s/21/22/g' kotlin-tooling-metadata.json
          - mv "$OUT" /outputs/unsigned.apk

[Semper-Viventem]

Hey, it seems that I've chosen a wrong default JDK. Now it should be 21, the fix is in master and will be in the release branch soon. Reproducible builds should work.

I use a particular JDK distribution optimized for my ARM64 hardware, that's why I decided to switch all the project to JDK 22. But it should be fine now, I'll use 22 only for my debug builds, it is better to keep releasing on the LTS version.

About a wrong commit hash. Probably there is some misconfiguration in GitHub Actions. I'll check

[obfusk]

Thanks!

Whilst your code to be able to switch the default between Java 21 and 22 is clever, I don't think it's needed. You should have no issues building with 22 on your machine even if it's set to 21. It's just the target version. You can build with a newer JDK without changing that, though that can easily give you non-identical results it shouldn't complain it's too new :)

[obfusk]

About a wrong commit hash. Probably there is some misconfiguration in GitHub Actions. I'll check

The APK is built from the merge commit. That seems correct. But somehow the tag is using the commit from before the merge from master into release, not the merge commit the github action is run on. Not sure why.

[obfusk]

Oh. ${${REPO_BRANCH}:-master} should be ${REPO_BRANCH:-master}. In .github/actions/publish-release-apk/entrypoint.sh. The hub checkout is never run.

[Semper-Viventem]

Oh. ${${REPO_BRANCH}:-master} should be ${REPO_BRANCH:-master}. In .github/actions/publish-release-apk/entrypoint.sh. The hub checkout is never run.

Oh, that's a great catch, thanks! I'll fix this line and check

BTW Could you please help me to find the place where the commit hash is embedded in the APK? I checked the META-INF and AndroidManifest files but didn't find it.

[obfusk]

META-INF/version-control-info.textproto

I have a shell script to extract it: https://gist.github.com/obfusk/4e3bbad0059be45cb555c6ef67c08588

[obfusk]

The hub checkout command is to check out pull requests though. Also odd that the checkout would be needed. I would expect the action to run on the same commit as the workflow.

[obfusk]

Looks like hub release uses the main branch unless -t is specified.

So I think you need to use hub release create with -t ${REPO_BRANCH:-master} instead of the hub checkout.

[Semper-Viventem]

It looks like the embedded commit is correct, it points to the release branch where the GitHub workflow has been run on. But the problem is with GitHub release action because the hub is run in an isolated environment

[Semper-Viventem]

-t, --commitish TARGET
A commit SHA or branch name to attach the release to, only used if TAG does not already exist (default: main branch).

I suspect the default behavior is not main but the current branch because I don't have main branch in the project.

[obfusk]

Yeah, I assume it means "default branch". So that would be "master".

[obfusk]

The new release still used the wrong commit, even though the hub checkout was correct. So I'm pretty sure it was indeed the -t using the default branch instead of the current one.

[Semper-Viventem]

Yep. I've run a new pipeline with -t fix, let's see

[Semper-Viventem]

@obfusk it looks like the issue is fixed now. Release commit points on the release branch. Thank you for your help!

[obfusk]

Indeed. Now fully reproducible without any workarounds on our end :)

[IzzySoft]

Confirmed:

BUILD SUCCESSFUL in 5m 12s
48 actionable tasks: 48 executed
+ mv app/build/outputs/apk/github/release/app-github-release-unsigned.apk /outputs/unsigned.apk

--- END BUILD LOG ---
Keeping '07eb8a05c7eee10483b17ad38ed8a9e40571651dea775d93ee922c273978887b-f.cking.software-v0.26.5-beta-upstream.apk'...
Keeping '727bff964cbab84789ba113ac2da34e534fb6f70e4c30ca5d55a858c1c4ab6fd-f.cking.software-v0.26.5-beta-unsigned.apk'...
Reproducible: True
Tags built: 1.

(built on ubuntu:jammy with openjdk-21-jdk-headless, just plain ./gradlew assembleGithubRelease and no extras as @obfusk already wrote). Thanks to both of you for your efforts!