How to author a custom buildpack that depends on toolchain introduced by dependent buildpacks?

[pacphi]

First, let me state what I'm trying to do from a design perspective...

I want to be able add an artifact dependency to a pom.xml or build.gradle file before executing build. I also want to be able to add an artifact to a pre-built executable .jar file. (Both are admittedly unorthodox approaches, but let's table that).

Here's how I'm thinking about it, and I wonder if a detect script in the custom buildpack will suffice for where all the logic belongs or if some (other more complicated) approach is necessary:

• Try to detect if this artifact declared as a (transitive) dependency in your Maven pom.xml or Gradle build.gradle file
• Check if the prerequisite toolset is installed
• If either the pom.xml or build.gradle file are found, verify whether or not the app is built-with Spring Boot Actuator
• Try to find a dependency declared on spring-boot-starter-runtime-metadata. If it is not declared, add it.
• Else, if neither file exists, but an executable .jar file is found, verify whether the .jar embedded the Spring Boot Actuator
• Next, verify whether or not spring-boot-starter-runtime-metadata is embedded
• If it is not embedded, unpack the executable .jar, source and add that artifact, then repackage the executable .jar

Here's the breakdown.

The structure of my project:

+- bin
  +- build
  +- detect
+- package.toml
+- builder.toml
+- buildpack.toml

Not including the script. But happy to share if you think it's valuable to.

package.toml

[buildpack]
uri = "."

builder.toml

description = "Attempts to detect if source or artifact is built with a compatible Spring Boot version and then adds a dependency on spring-boot-starter-runtime-metadata if it's not already present."

[stack]
build-image = "paketobuildpacks/build:tiny-cnb"
run-image = "paketobuildpacks/run:tiny-cnb"

[[buildpacks]]
id = "paketo-buildpacks/bellsoft-liberica"
uri = "docker://paketobuildpacks/bellsoft-liberica"

[[buildpacks]]
id = "paketo-buildpacks/maven"
uri = "docker://paketobuildpacks/maven"

[[buildpacks]]
id = "paketo-buildpacks/gradle"
uri = "docker://paketobuildpacks/gradle"

[[buildpacks]]
id = "cf-toolsuite/spring-boot-runtime-metadata-buildpack"
uri = "."

[[order]]
[[order.group]]
id = "paketo-buildpacks/bellsoft-liberica"

[[order.group]]
id = "paketo-buildpacks/maven"

[[order.group]]
id = "paketo-buildpacks/gradle"

[[order.group]]
id = "cf-toolsuite/spring-boot-runtime-metadata-buildpack"

buildpack.toml

api = "0.10"
WithWindowsBuild = false
WithLinuxBuild = false

[buildpack]
id = "cf-toolsuite/spring-boot-runtime-metadata-buildpack"
version = "0.2.0"
name = "Spring Boot Runtime Metadata Buildpack"
homepage = "https://github.com/cf-toolsuite/spring-boot-runtime-metadata-buildpack"

# Targets the buildpack will work with
[[targets]]
os = "linux"

# Stacks (deprecated) the buildpack will work with
[[stacks]]
id = "*"

How I built everything

pack buildpack package cftoolsuite/spring-boot-runtime-metadata-buildpack:latest --config ./package.toml --format image
pack builder create cftoolsuite/spring-boot-runtime-metadata-builder --config builder.toml

What I am seeing when I try to build an app container image using the custom builder:

$ gh clone repo cf-toolsuite/cf-butler
$ cd cf-butler
$ pack build cftoolsuite/cf-butler \
     --path . \
     --env BP_MAVEN_ACTIVE_PROFILES=mysql,expose-runtime-metadata \
     --env BP_JVM_VERSION=21.* \
     --builder cftoolsuite/spring-boot-runtime-metadata-builder \
     --volume $HOME/.m2:/home/cnb/.m2:rw
Warning: Using untrusted builder with volume mounts. If there is sensitive data in the volumes, this may present a security vulnerability.
latest: Pulling from cftoolsuite/spring-boot-runtime-metadata-builder
c6b348cd73bd: Already exists 
bde1b3da34ee: Already exists 
23c88e7c16a6: Already exists 
28fc7d10919c: Already exists 
000fed2157ac: Already exists 
97d38d26b203: Already exists 
0918960ae92b: Already exists 
78d21e40a282: Already exists 
f10bc4ceffe5: Already exists 
6e3cb2b8ca58: Pull complete 
c10f01c00d1f: Pull complete 
5baa57fb8ec4: Pull complete 
a20176c97e6f: Pull complete 
b8aa37325587: Pull complete 
722799e7929f: Pull complete 
d096d533cd4d: Pull complete 
89732bc75041: Pull complete 
Digest: sha256:0e18d3c45ccdb82fcb78c13909bf48fc7447227985c0819566119a41c060fa69
Status: Downloaded newer image for cftoolsuite/spring-boot-runtime-metadata-builder:latest
tiny-cnb: Pulling from paketobuildpacks/run
Digest: sha256:adf913cf28031f2090aeaedac65edb36f2987d81a23a8dffab5ea18ca216c94c
Status: Image is up to date for paketobuildpacks/run:tiny-cnb
0.18.5: Pulling from buildpacksio/lifecycle
Digest: sha256:16a4c9e945381ad182fbb92710f0643e1c434ce11c3c43fb0ce6fcfce0d5efb7
Status: Image is up to date for buildpacksio/lifecycle:0.18.5
===> ANALYZING
[analyzer] Image with name "cftoolsuite/cf-butler" not found
===> DETECTING
[detector] ======== Output: paketo-buildpacks/[email protected] ========
[detector]   Build Configuration:
[detector]     $BP_EXCLUDE_FILES                                                                       colon separated list of glob patterns, matched source files are removed
[detector]     $BP_GRADLE_ADDITIONAL_BUILD_ARGUMENTS                                                   the additionnal arguments (appended to BP_GRADLE_BUILD_ARGUMENTS) to pass to Gradle
[detector]     $BP_GRADLE_BUILD_ARGUMENTS             --no-daemon -Dorg.gradle.welcome=never assemble  the arguments to pass to Gradle
[detector]     $BP_GRADLE_BUILD_FILE                                                                   the location of the main build config file, relative to the application root
[detector]     $BP_GRADLE_BUILT_ARTIFACT              build/libs/*.[jw]ar                              the built application artifact explicitly.  Supersedes $BP_GRADLE_BUILT_MODULE
[detector]     $BP_GRADLE_BUILT_MODULE                                                                 the module to find application artifact in
[detector]     $BP_GRADLE_INIT_SCRIPT_PATH                                                             the path to a Gradle init script file
[detector]     $BP_INCLUDE_FILES                                                                       colon separated list of glob patterns, matched source files are included
[detector]     $BP_JAVA_INSTALL_NODE                  false                                            whether to install Yarn/Node binaries based on the presence of a package.json or yarn.lock file
[detector]     $BP_NODE_PROJECT_PATH                                                                   configure a project subdirectory to look for `package.json` and `yarn.lock` files
[detector] SKIPPED: No plans could be resolved
[detector] ======== Output: cf-toolsuite/[email protected] ========
[detector] Maven is not installed.  Please install Maven and try again.
[detector] jar is not installed.  Please install a distro of JDK >= 17.
[detector] ======== Results ========
[detector] pass: paketo-buildpacks/[email protected]
[detector] pass: paketo-buildpacks/[email protected]
[detector] fail: paketo-buildpacks/[email protected]
[detector] err:  cf-toolsuite/[email protected] (1)
[detector] ERROR: No buildpack groups passed detection.
[detector] ERROR: failed to detect: buildpack(s) failed with err
ERROR: failed to build: executing lifecycle: failed with status code: 21

Any pointers? I'm open to all suggestions.

(I saw an earlier post about how Maven cleans up source. I don't think I'll suffer from that problem, but I do need mvn, gradle, and jar to be available to the custom buildpack's scripts. I want to be able to invoke these before the Maven or Gradle buildpacks get involved in build. That's what I really need advice on. How can I facilitate that?)

[dmikusa]

Let's take the case that you want to have a buildpack that will add Spring Boot Actuator JARS if the project does not include them. There are two approaches I can think of here.

1. Run a command with Maven/Gradle to figure out what dependencies are installed. Search for the target dependency. If it's not present, modify pom.xml/build.gradle such that it's included. You'd need to do this before the Maven/Gradle buildpack run, so that the app is built to include the added dependency or you'd need to run the build a second time.

2. Look at the resulting JAR file from the build process, i.e. the built application JAR, and check if the JAR has the target dependency. If not add it but not to the build tool config, just download and add the JAR directly.

The second approach is simple and has been pretty reliable. It is something that we have used in Paketo buildpacks. We do this in the spring boot buildpack for Spring Cloud Bindings.

The first approach is something that we've tried to avoid doing because, in a nutshell, it's complicated. For example:

• The pom.xml/build.gradle files, especially the latter, are complicated. They're not a single file with everything you need in it. There are parent poms in Maven and Gradle is more complicated. This means to get any info from them reliably, you need to run the tool and have it tell you the information you need. This creates some ordering challenges cause you now need the tool to get this info, but you want to modify the build configuration file before the tool runs.

• Even if you can get the info you need, modifying pom.xml or build.gradle reliably for everyone's different pom.xml/build.gradle is just difficult. You run the risk of breaking some application that has a configuration you didn't anticipate. Maybe if you have a constrained set of applications with very similar pom.xml/build.gradles you could make some assumptions and get this to work, but in a general sense we don't think it's possible to do in a reliable way.

• The present Maven & Gradle buildpacks don't have functionality for you to run any arbitrary pre-build commands. They're set up to run Maven/Gradle once and that command is expected to produce the artifact (JAR/WAR/etc...). This doesn't facilitate running commands to interrogate the app first, which again makes ordering things difficult.

We've not really looked into the second option, because the first option has worked well enough for us. I don't want to say it's impossible, but it would require changes to the way existing buildpacks work or forking and modifying existing buildpack. That's compared to the second option where you can just add a custom buildpack at the end of the list of existing buildpacks and have it conditionally insert the required JAR.

---

Two other things to consider:

1. The typical build of a Spring Boot app with buildpacks will not run against the JAR file. It will run on the exploded JAR file (i.e. the contents of your JAR). It's a slight difference but even when you run Maven/Gradle in the build environment, the build tool's buildpack takes the resulting build artifact and expands it (for Spring Boot apps). The same thing happens if you build with Spring Boot Build Tools.

2. Not every application runs Maven/Gradle inside the build container. If you pack build -p target/app.jar then it skips that entirely. Maybe that's not necessary for your particular case, but it's something to consider as well.

[pacphi]

Thanks @dmikusa! The remarks you made in the third bullet point above related to the first approach is valuable info.

I think the second approach suggested is what I will adapt my own approach to.

So, two scenarios exist:

a) post-build for Maven or Gradle, check if artifact exists, if it doesn't add it
b) if pre-built artifact supplied in path, do the same check as in a), what's not clear to me is, do I need to explode/unpack jar, or will that have been done for me, and I just need to check a well-known relative path (e.g., BOOT-INF/lib)?

And another thing to consider, dependencies typically have transitive dependencies. How do I add the transitives? One approach I was using was to use mvn dependency:get but that was with expectation that mvn was available to be called from the custom buildpack.

Your bullet point 2 in "two other things to consider" is something I had absolutely wanted to account for in my design, see 5th bullet point of my original post.

Footnote:

I'm not a Go-lang expert. I had been using bash.

Here's the naive implementation of a bash script I'd started with and will now have to adapt...

#!/usr/bin/env bash

# Detection script for spring-boot-starter-runtime-metadata artifact

# I try to detect if this artifact declared as a (transitive) dependency in your Maven pom.xml or Gradle build.gradle file
# First, I check if the prerequisite toolset is installed
# Next, if either the pom.xml or build.gradle file are found, I verify whether or not the app is built-with Spring Boot Actuator
# Then, I try to find a dependency declared on spring-boot-starter-runtime-metadata. If it is not declared, I will add it.
# Else, if neither file exists, but I find an executable .jar file, I verify whether the .jar embedded the Spring Boot Actuator
# Next, I will verify whether or not spring-boot-starter-runtime-metadata is embedded
# If it is not embedded, I will unpack the executable .jar, source and add that artifact, then repackage the executable .jar

SPRING_BOOT_GROUP_ID="org.springframework.boot"
SPRING_BOOT_ARTIFACT_ID="spring-boot-actuator"
SPRING_BOOT_VERSION="3.2.0"

SPRING_BOOT_RUNTIME_METADATA_GROUP_ID="org.cftoolsuite.actuator"
SPRING_BOOT_RUNTIME_METADATA_ARTIFACT_ID="spring-boot-starter-runtime-metadata"
SPRING_BOOT_RUNTIME_METADATA_VERSION="0.2.0"

checkPrereqs() {
  if command -v mvn >/dev/null 2>&1; then
    echo "Maven is installed."
    mavenInstalled=0
  else
    echo "Maven is not installed.  Please install Maven and try again."
    mavenInstalled=1
  fi
  if command -v jar >/dev/null 2>&1; then
    echo "jar is installed."
    versionInstalled=$(jar --version | awk -F' ' '{print $2}' | awk -F'.' '{print $1}')
    if [ "$versionInstalled" -ge 17 ]; then
      echo "jar version compatible"
      jarInstalled=0
    else
      echo "jar version incompatible.  Please install a distro of JDK >= 17."
      jarInstalled=1
    fi
  else
    echo "jar is not installed.  Please install a distro of JDK >= 17."
    jarInstalled=1
  fi
  prereqsSatisfied=$((mavenInstalled + jarInstalled))
  if [ "$prereqsSatisfied" -eq 0 ]; then
    return 0
  else
    exit 1
  fi
}

version_ge() {
  test "$(printf '%s\n' "$@" | sort -V | head -n 1)" != "$1";
}

# Function that adds a dependency on spring-boot-starter-runtime-metadata to pom.xml
addMavenDependency() {
  local pomFile="pom.xml"

  if [[ -f "$pomFile" ]]; then
    awk -v groupId="$SPRING_BOOT_RUNTIME_METADATA_GROUP_ID" -v artifactId="$SPRING_BOOT_RUNTIME_METADATA_ARTIFACT_ID" -v version="$SPRING_BOOT_RUNTIME_METADATA_VERSION" '
    /<dependencyManagement>/ { inDependencyManagementBlock=1 }
    /<\/dependencyManagement>/ { inDependencyManagementBlock=0 }
    /<plugins>/ { inPluginsBlock=1 }
    /<\/plugins>/ { inPluginsBlock=0 }
    /<profiles>/,/<\/profiles>/ { inProfilesBlock=1 }
    /<\/profiles>/ { inProfilesBlock=0 }
    !inPluginsBlock && !inDependencyManagementBlock && !inProfilesBlock && /<\/dependencies>/ {
      print "    <dependency>"
      print "      <groupId>"groupId"</groupId>"
      print "      <artifactId>"artifactId"</artifactId>"
      print "      <version>"version"</version>"
      print "    </dependency>"
    }
    { print }
    ' "$pomFile" > tmp && mv tmp "$pomFile"
    echo "Dependency added to $pomFile."
  else
    echo "$pomFile not found."
  fi
}

# Function that adds a dependency on spring-boot-starter-runtime-metadata to build.gradle
addGradleDependency() {
  local dependency="implementation '$SPRING_BOOT_RUNTIME_METADATA_GROUP_ID:$SPRING_BOOT_RUNTIME_METADATA_ARTIFACT_ID:$SPRING_BOOT_RUNTIME_METADATA_VERSION'"
  local gradleFile="build.gradle"

  if [[ -f "$gradleFile" ]]; then
    awk -v dep="$dependency" '
    $0 ~ /^dependencies[[:space:]]*\{/ {
      print
      print "    " dep
      depAdded = 1
      next
    }
    { print }
    END {
      if (!depAdded) {
        print "dependencies {"
        print "    " dep
        print "}"
      }
    }
    ' "$gradleFile" > tmp && mv tmp "$gradleFile"
    echo "Dependency added to $gradleFile."
  else
    echo "$gradleFile not found."
  fi
}

# Function to check if a specific dependency exists in the project using the provided command
checkDependency() {
  local command="$1"
  local dependency="$2"

  echo "Checking for $dependency dependency..."
  # Evaluate the passed command and pipe its output to grep
  eval "$command" | grep "$dependency" > /dev/null

  if [ $? -eq 0 ]; then
    echo "Dependency $dependency found."
    return 0
  else
    echo "Dependency $dependency not found."
    return 1
  fi
}

# Function that checks if executable jar contains spring-boot-starter-runtime-metadata
checkForDependencyInExecJar() {
  local jarFilePath="$1"
  if [[ $jarFilePath =~ \.jar$ ]]; then
    if jar tvf "$jarFilePath" | grep "$SPRING_BOOT_RUNTIME_METADATA_ARTIFACT_ID" > /dev/null; then
      echo "Dependency $SPRING_BOOT_RUNTIME_METADATA_ARTIFACT_ID found in $jarFilePath."
      return 0
    else
      echo "Dependency $SPRING_BOOT_RUNTIME_METADATA_ARTIFACT_ID not found in $jarFilePath."
      return 1
    fi
  else
    echo "$jarFilePath not a valid JAR."
    return 1
  fi
}

# Function to check if the project is a Maven project and if it includes the specified dependency
isMavenBuild() {
  # Check if mvnw and pom.xml exist in the current directory
  if [[ -f "mvnw" && -f "pom.xml" ]]; then
    echo "Maven wrapper and pom.xml detected."
    local mvnCommand="./mvnw dependency:tree $BP_MAVEN_ADDITIONAL_BUILD_ARGUMENTS"
    # Use the generic function to check the dependency
    # First for Spring Boot
    detected_spring_boot_version=$($mvnCommand | grep "$SPRING_BOOT_GROUP_ID:$SPRING_BOOT_ARTIFACT_ID" | head -n 1 | sed -r 's/.*:(.*):.*/\1/')
    if [ -z "$detected_spring_boot_version" ]; then
      echo "Spring Boot version could not be determined."
      exit 1
    fi
    if version_ge "$detected_spring_boot_version" "$SPRING_BOOT_VERSION"; then
      echo "Spring Boot version is compatible."
    else
      echo "Spring Boot version is incompatible."
      exit 1
    fi
    # Second for Spring Boot Runtime Metadata
    checkDependency "$mvnCommand" "$SPRING_BOOT_RUNTIME_METADATA_ARTIFACT_ID"
    if [ $? -eq 1 ]; then
      addMavenDependency
      return 0
    fi
  else
    echo "Maven project not detected."
    return 1
  fi
}

# Function to check if the project is a Gradle project and if it includes the specified dependency
isGradleBuild() {
  # Check if gradlew and build.gradle exist in the current directory
  if [[ -f "gradlew" && -f "build.gradle" ]]; then
    echo "Gradle wrapper and build.gradle detected."
    local gradleCommand="./gradlew dependencies $BP_GRADLE_ADDITIONAL_BUILD_ARGUMENTS"
    # Use the generic function to check the dependency
    # First for Spring Boot
    $gradleCommand > dependencies.txt
    detected_spring_boot_version=$(cat dependencies.txt | grep "$SPRING_BOOT_GROUP_ID:$SPRING_BOOT_ARTIFACT_ID" | tail -n 1 |  awk -F':' '{print $3}')
    if [ -z "$detected_spring_boot_version" ]; then
      echo "No dependencies on Spring Boot."
      exit 1
    fi
    if version_ge "$detected_spring_boot_version" "$SPRING_BOOT_VERSION"; then
      echo "Spring Boot version is compatible."
    else
      echo "Spring Boot version is incompatible."
      exit 1
    fi
    # Second for Spring Boot Runtime Metadata
    checkDependency "$gradleCommand" "$SPRING_BOOT_RUNTIME_METADATA_ARTIFACT_ID"
    if [ $? -eq 1 ]; then
      addGradleDependency
    fi
  else
    echo "Gradle project not detected."
    return 1
  fi
}

# Function to repackage executable .jar file so that it includes spring-boot-starter-runtime-metadata
repackageExecJar() {
  local jarFilePath="$1"
  if [[ $jarFilePath =~ \.jar$ ]]; then
    jar xvf $jarFilePath
    rm -f $jarFilePath
    detected_spring_boot_jar=$(find $PWD | grep "$SPRING_BOOT_ARTIFACT_ID" | head -n 1)
    detected_spring_boot_filename="${detected_spring_boot_jar%.jar}"
    detected_spring_boot_version="${detected_spring_boot_filename##*-}"
  fi
  if [ -z "$detected_spring_boot_version" ]; then
      echo "No dependencies on Spring Boot."
      exit 1
  fi
  if version_ge "$detected_spring_boot_version" "$SPRING_BOOT_VERSION"; then
    echo "Spring Boot version is compatible."
  else
    echo "Spring Boot version is incompatible."
    exit 1
  fi
  mvn dependency:get -Dartifact=$SPRING_BOOT_RUNTIME_METADATA_GROUP_ID:$SPRING_BOOT_RUNTIME_METADATA_ARTIFACT_ID:$SPRING_BOOT_RUNTIME_METADATA_VERSION
  SPRING_BOOT_RUNTIME_METADATA_GROUP_PATH="${SPRING_BOOT_RUNTIME_METADATA_GROUP_ID//.//}"
  cp -f $HOME/.m2/repository/$SPRING_BOOT_RUNTIME_METADATA_GROUP_PATH/$SPRING_BOOT_RUNTIME_METADATA_ARTIFACT_ID/$SPRING_BOOT_RUNTIME_METADATA_VERSION/$SPRING_BOOT_RUNTIME_METADATA_ARTIFACT_ID-$SPRING_BOOT_RUNTIME_METADATA_VERSION.jar BOOT-INF/lib
  # FIXME Need to copy all the transitive dependencies from cyclonedx too
  # TODO How to account for potential collisions, mismatched versions?
  jar cvf $jarFilePath ./
}


# Main block
if checkPrereqs; then
  if isMavenBuild; then
    # Maven build detected and processed
    exit 0
  elif isGradleBuild; then
    # Gradle build detected and processed
    exit 0
  else
    if checkForDependencyInExecJar "${1:-$PWD}"; then
      # Executable .jar file detected
      echo "Dependency found in executable JAR."
      exit 0
    else
      repackageExecJar "${1:-$PWD}"
      exit 0
    fi
  fi
fi

[dmikusa]

Sorry for the delay.

b) if pre-built artifact supplied in path, do the same check as in a), what's not clear to me is, do I need to explode/unpack jar, or will that have been done for me, and I just need to check a well-known relative path (e.g., BOOT-INF/lib)?

The answer is that it depends on what's provided to the buildpack.

If you pack build --path to/some/file.jar then what the buildpacks will see is the exploded contents of the JAR file.

If you pack build --path java-projet-root, then some of the buildpacks those that run before Maven/Gradle/and other build tools will see the source code. Buildpacks that run after Maven/Gradle/and other build tools will see the exploded contents of the JAR file produced by the build.

If you have a Spring Boot app, that's really all you need to be concerned with however there is another possible scenario. If your application build produces multiple JAR files then what you'll see before build is the source cod and if your build pack runs after build, you'll see a directory containing all of the JAR files saved from the build process. For Spring Boot apps, this doesn't generally happen because the output is a single JAR.

[dmikusa]

And another thing to consider, dependencies typically have transitive dependencies. How do I add the transitives? One approach I was using was to use mvn dependency:get but that was with expectation that mvn was available to be called from the custom buildpack.

That would really be on you to calculate in advance. The buildpack you create would need to know everything that is required to be installed, transitive dependencies included. If that's not something you can calculate in advance then it gets a lot trickier.

I would try to avoid running Maven or Gradle because that's going to create some circular dependencies as I previously mentioned. What you might be able to do is to have a CI process that runs and calculates the dependencies, maybe that runs Maven or Gradle against some stock app, and then produces an artifact that your buildpack can then take and install. The artifact can contain all the JARs that need to be installed, so the buildpack becomes pretty simple. Detect what dependency to install, fetch the artifact, then download and extract it.

Another possibility, if you need really tight integration with the build tool, is that you might be better off with a Maven or Gradle plugin. Something that integrates into the build tool and thus has access to all of the information natively.

Hope that helps!