Improved detection for standalone jar files by using pom.properties file if available #724

Nikemare · 2023-11-17T16:32:29Z

Another improvement for standalone jar files

Optimized extract jar to only spawn "META-INF" directory
First checks if pom.properties is available - see: https://maven.apache.org/shared/maven-archiver/index.html
Fallback to old logic if pom.properties is not available

Steps to reproduce:

Download org.jacoco.report-0.8.8.jar, xmlschema-core-2.2.5.jar, jackson-core-2.16.0.jar, junit-4.13.2.jar, wsdl4j-1.6.3.jar and put them into empty folder.
Run cdxgen against this folder

xmlschema-core and org.jacoco.report before my change:

"components": [
    {
      "group": "org.apache",
      "name": "xmlschema-core",
      "version": "2.2.5",
      "purl": "pkg:maven/org.apache/[email protected]?type=jar",
      "type": "framework",
      "bom-ref": "pkg:maven/org.apache/[email protected]?type=jar",
      "evidence": {
        "identity": {
          "field": "purl",
          "confidence": 0.5,
          "methods": [
            {
              "technique": "filename",
              "confidence": 0.5,
              "value": "xmlschema-core-2.2.5.jar"
            }
          ]
        }
      },
      "properties": [
        {
          "name": "SrcFile",
          "value": "xmlschema-core-2.2.5.jar"
        }
      ]
    },
    {
      "group": "org.jacoco.report",
      "name": "org.jacoco.report",
      "version": "0.8.8.202204050719",
      "purl": "pkg:maven/org.jacoco.report/[email protected]?type=jar",
      "type": "library",
      "bom-ref": "pkg:maven/org.jacoco.report/[email protected]?type=jar",
      "evidence": {
        "identity": {
          "field": "purl",
          "confidence": 0.5,
          "methods": [
            {
              "technique": "filename",
              "confidence": 0.5,
              "value": "org.jacoco.report-0.8.8.jar"
            }
          ]
        }
      },
      "properties": [
        {
          "name": "SrcFile",
          "value": "org.jacoco.report-0.8.8.jar"
        }
      ]
    }
]

and after my change with correct group, name and version, which results also in successfully fetched license information:

"components": [
    {
      "publisher": "The Apache Software Foundation",
      "group": "org.apache.ws.xmlschema",
      "name": "xmlschema-core",
      "version": "2.2.5",
      "description": "Commons XMLSchema is a light weight schema object model that can be used to manipulate or\n        generate XML schema.",
      "licenses": [
        {
          "license": {
            "id": "Apache-2.0",
            "url": "https://opensource.org/licenses/Apache-2.0"
          }
        }
      ],
      "purl": "pkg:maven/org.apache.ws.xmlschema/[email protected]?type=jar",
      "externalReferences": [
        {
          "type": "vcs",
          "url": "https://gitbox.apache.org/repos/asf?p=ws-xmlschema.git;a=summary"
        }
      ],
      "type": "framework",
      "bom-ref": "pkg:maven/org.apache.ws.xmlschema/[email protected]?type=jar",
      "evidence": {
        "identity": {
          "field": "purl",
          "confidence": 1,
          "methods": [
            {
              "technique": "manifest-analysis",
              "confidence": 1,
              "value": "xmlschema-core-2.2.5.jar"
            }
          ]
        }
      },
      "properties": [
        {
          "name": "SrcFile",
          "value": "xmlschema-core-2.2.5.jar"
        }
      ]
    },
    {
      "publisher": "Mountainminds GmbH & Co. KG",
      "group": "org.jacoco",
      "name": "org.jacoco.report",
      "version": "0.8.8",
      "description": "JaCoCo Report",
      "licenses": [
        {
          "license": {
            "name": "Eclipse Public License 2.0"
          }
        }
      ],
      "purl": "pkg:maven/org.jacoco/[email protected]?type=jar",
      "externalReferences": [
        {
          "type": "vcs",
          "url": "https://github.com/jacoco/jacoco"
        }
      ],
      "type": "library",
      "bom-ref": "pkg:maven/org.jacoco/[email protected]?type=jar",
      "evidence": {
        "identity": {
          "field": "purl",
          "confidence": 1,
          "methods": [
            {
              "technique": "manifest-analysis",
              "confidence": 1,
              "value": "org.jacoco.report-0.8.8.jar"
            }
          ]
        }
      },
      "properties": [
        {
          "name": "SrcFile",
          "value": "org.jacoco.report-0.8.8.jar"
        }
      ]
    }

In both cases FETCH_LICENSE was set to true. Other components have unchanged results, as they were resolved correctly already before my change.

use pom.properties if available fallback to old logic if pom properties is not available extract only META-INF from jar Signed-off-by: Nikemare <[email protected]>

Signed-off-by: Nikemare <[email protected]>

prabhu · 2023-11-19T15:02:15Z

@Nikemare This is impressive work. Could you kindly add some repotests to test this?

@heubeck could you kindly check how this is looking for your workflows?

Nikemare · 2023-11-19T17:54:37Z

@prabhu Thank you. About repotests I'm a little bit lost in the moment, if you can guide me in the right direction I will add it.

I found repotests.yml

I guess I need to add a checkout step, right?
Should I create a repo in my GitHub space?
Then I guess I need to add another line in step 'repotests' which will generate a json

But then, how do I validate the json? Or is the auto-validation via --validate enough?

prabhu · 2023-11-19T18:28:32Z

@Nikemare, it's a good question.

Can we do something similar to our jenkins tests? Download some jars to a directory and run cdxgen?

https://github.com/CycloneDX/cdxgen/blob/master/.github/workflows/repotests.yml#L215

We can just print the sbom with -p for now. In the future, we can add some snapshot based testing.

Signed-off-by: Nikemare <[email protected]>

prabhu

Thank you so much!

Nikemare · 2023-11-19T19:11:35Z

@prabhu I made a mistake in repotest, will commit fix in a second.

Missed to specify directory in repotest for standalone jar files Signed-off-by: Nikemare <[email protected]>

Nikemare · 2023-11-19T19:37:30Z

Step standalone jar files in repotests takes 2 seconds with following output now:

Run mkdir -p standalone-jar-files
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  125k  100  125k    0     0  1261k      0 --:--:-- --:--:-- --:--:-- 1271k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  169k  100  169k    0     0  2169k      0 --:--:-- --:--:-- --:--:-- 2197k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  565k  100  565k    0     0  7152k      0 --:--:-- --:--:-- --:--:-- 7250k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  375k  100  375k    0     0  5361k      0 --:--:-- --:--:-- --:--:-- 5443k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0  182k    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  182k  100  182k    0     0  1442k      0 --:--:-- --:--:-- --:--:-- 1436k
╔═══════════════════════════╤═════════════════════════════════════╤═══════════════════════════╤═════════════════╗
║ Group                     │ Name                                │                   Version │ Scope           ║
╚═══════════════════════════╧═════════════════════════════════════╧═══════════════════════════╧═════════════════╝
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║ org.apache.ws.xmlschema   │ xmlschema-core                      │                     2.2.5 │                 ║
╚═══════════════════════════╧═════════════════════════════════════╧═══════════════════════════╧═════════════════╝
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║ wsdl4j                    │ wsdl4j                              │                     1.6.3 │                 ║
╚═══════════════════════════╧═════════════════════════════════════╧═══════════════════════════╧═════════════════╝
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║ org.jacoco                │ org.jacoco.report                   │                     0.8.8 │                 ║
╚═══════════════════════════╧═════════════════════════════════════╧═══════════════════════════╧═════════════════╝
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║ junit                     │ junit                               │                    4.13.2 │                 ║
╚═══════════════════════════╧═════════════════════════════════════╧═══════════════════════════╧═════════════════╝
╟───────────────────────────┼─────────────────────────────────────┼───────────────────────────┼─────────────────╢
║ com.fasterxml.jackson.cor │ jackson-core                        │                    2.16.0 │                 ║
║ e                         │                                     │                           │                 ║
╚═══════════════════════════╧═════════════════════════════════════╧═══════════════════════════╧═════════════════╝
BOM includes 5 components and 0 dependencies

prabhu · 2023-11-19T20:00:18Z

.github/workflows/repotests.yml

@@ -228,7 +228,7 @@ jobs:
          curl --output-dir standalone-jar-files -LO https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.16.0/jackson-core-2.16.0.jar
          curl --output-dir standalone-jar-files -LO https://repo1.maven.org/maven2/junit/junit/4.13.2/junit-4.13.2.jar
          curl --output-dir standalone-jar-files -LO https://repo1.maven.org/maven2/wsdl4j/wsdl4j/1.6.3/wsdl4j-1.6.3.jar
-          FETCH_LICENSE=true bin/cdxgen.js -p -o bomresults/bom-standalone-jar-files.json --validate
+          FETCH_LICENSE=true bin/cdxgen.js -p standalone-jar-files -o bomresults/bom-standalone-jar-files.json --validate


Well spotted!

heubeck · 2023-11-20T07:42:30Z

Works fine for me - but I'm analyzing only source code repos so far.

prabhu · 2023-11-20T11:21:51Z

utils.js

+        if (existsSync(mavenDir)) {
+          let groupDir = readdirSync(mavenDir);
+          if (groupDir && groupDir.length) {
+            let artifactDir = readdirSync(join(mavenDir, groupDir[0]));


@Nikemare, This line is failing while creating an SBOM for container images.

Error: ENOTDIR: not a directory, scandir '/tmp/jar-deps-pgCSWJ/META-INF/maven/extension.xml' at readdirSync (node:fs:1516:26) at extractJarArchive (file:///mnt/work/CycloneDX/cdxgen/utils.js:6650:31) at createJarBom (file:///mnt/work/CycloneDX/cdxgen/index.js:1084:21) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async createMultiXBom (file:///mnt/work/CycloneDX/cdxgen/index.js:4868:15) at async createBom (file:///mnt/work/CycloneDX/cdxgen/index.js:5388:21) at async file:///mnt/work/CycloneDX/cdxgen/bin/cdxgen.js:366:20 { errno: -20,

Below change at 6649 helps

if ( groupDir && groupDir.length && lstatSync(join(mavenDir, groupDir[0])).isDirectory() ) {

@prabhu I can provide a fix in the evening (CET), or will you fix it on your own?

The evening sounds good! Thank you so much!

Nikemare added 4 commits November 17, 2023 17:09

parse pom properties files for stanadalone jars

0445b31

use pom.properties if available fallback to old logic if pom properties is not available extract only META-INF from jar Signed-off-by: Nikemare <[email protected]>

Formatted utils.js

932b6f4

Signed-off-by: Nikemare <[email protected]>

Fixed console.log

7d5dc0f

Signed-off-by: Nikemare <[email protected]>

Better confidence info in extractJarArchive

30eeaab

Signed-off-by: Nikemare <[email protected]>

Added repotest for standalone jar files

1b0d4c6

Signed-off-by: Nikemare <[email protected]>

prabhu approved these changes Nov 19, 2023

View reviewed changes

Fix repotest for standalone jar files

c7d633c

Missed to specify directory in repotest for standalone jar files Signed-off-by: Nikemare <[email protected]>

prabhu reviewed Nov 19, 2023

View reviewed changes

prabhu merged commit b0105c9 into CycloneDX:master Nov 19, 2023
17 checks passed

prabhu reviewed Nov 20, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improved detection for standalone jar files by using pom.properties file if available #724

Improved detection for standalone jar files by using pom.properties file if available #724

Nikemare commented Nov 17, 2023 •

edited

Loading

prabhu commented Nov 19, 2023

Nikemare commented Nov 19, 2023 •

edited

Loading

prabhu commented Nov 19, 2023

prabhu left a comment

Nikemare commented Nov 19, 2023

Nikemare commented Nov 19, 2023

prabhu Nov 19, 2023

heubeck commented Nov 20, 2023

prabhu Nov 20, 2023 •

edited

Loading

Nikemare Nov 20, 2023

prabhu Nov 20, 2023

Nikemare Nov 20, 2023

Improved detection for standalone jar files by using pom.properties file if available #724

Improved detection for standalone jar files by using pom.properties file if available #724

Conversation

Nikemare commented Nov 17, 2023 • edited Loading

prabhu commented Nov 19, 2023

Nikemare commented Nov 19, 2023 • edited Loading

prabhu commented Nov 19, 2023

prabhu left a comment

Choose a reason for hiding this comment

Nikemare commented Nov 19, 2023

Nikemare commented Nov 19, 2023

prabhu Nov 19, 2023

Choose a reason for hiding this comment

heubeck commented Nov 20, 2023

prabhu Nov 20, 2023 • edited Loading

Choose a reason for hiding this comment

Nikemare Nov 20, 2023

Choose a reason for hiding this comment

prabhu Nov 20, 2023

Choose a reason for hiding this comment

Nikemare Nov 20, 2023

Choose a reason for hiding this comment

Nikemare commented Nov 17, 2023 •

edited

Loading

Nikemare commented Nov 19, 2023 •

edited

Loading

prabhu Nov 20, 2023 •

edited

Loading