Skip to content

Commit

Permalink
Support a component list report command with column (--where) fil…
Browse files Browse the repository at this point in the history
…ters and `--summary` options (#85)

* Support a Component list/report command with filters and summary options

Signed-off-by: Matt Rutkowski <[email protected]>

* Assure all component display methods use pointers to ComponentMap entries

Signed-off-by: Matt Rutkowski <[email protected]>

* Add component list tests; support copyright column

Signed-off-by: Matt Rutkowski <[email protected]>

* Add tests for component list command using existing test data

Signed-off-by: Matt Rutkowski <[email protected]>

* Support where clause with float64 values; add testcase

Signed-off-by: Matt Rutkowski <[email protected]>

* Add Manufacturer, Publisher, HasPedigree columns to component list

Signed-off-by: Matt Rutkowski <[email protected]>

* Remove uncalled isEmpty() method

Signed-off-by: Matt Rutkowski <[email protected]>

* Add v1.6 testcase that includes varying component fields

Signed-off-by: Matt Rutkowski <[email protected]>

* Add JSON test files from the v1.6 spec. repo.

Signed-off-by: Matt Rutkowski <[email protected]>

* Add JSON test files from the v1.6 spec. repo.

Signed-off-by: Matt Rutkowski <[email protected]>

* Add JSON test files from the v1.6 spec. repo.

Signed-off-by: Matt Rutkowski <[email protected]>

* Add JSON test files from the v1.6 spec. repo.

Signed-off-by: Matt Rutkowski <[email protected]>

* prepare for more columns in component list output

Signed-off-by: Matt Rutkowski <[email protected]>

* Support full set of column data for component list

Signed-off-by: Matt Rutkowski <[email protected]>

* Consolidate report/list formatting flags

Signed-off-by: Matt Rutkowski <[email protected]>

* Improve README top-level abstract

Signed-off-by: Matt Rutkowski <[email protected]>

* Improve README top-level abstract

Signed-off-by: Matt Rutkowski <[email protected]>

* Improve README top-level abstract

Signed-off-by: Matt Rutkowski <[email protected]>

---------

Signed-off-by: Matt Rutkowski <[email protected]>
  • Loading branch information
mrutkows authored May 6, 2024
1 parent edc06ac commit f892978
Show file tree
Hide file tree
Showing 46 changed files with 2,588 additions and 325 deletions.
1 change: 1 addition & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"BOMREF",
"BSDL",
"callstack",
"CBOM",
"CDLA",
"cdxschema",
"CISA",
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@

This utility was designed to be an API platform to validate, analyze and edit **Bills-of-Materials (BOMs)**. Initially, it was created to validate **CycloneDX** or **SPDX-formatted** BOMs against versioned JSON schemas (as published by their respective standards communities) or customized schema variants designed by organizations that may have stricter compliance requirements.

The utility now includes a rich set of commands, listed below, such as **trim**, **patch** (IETF RFC 6902) and **diff** as well as commands used to create filtered reports, in various formats, using the utility's powerful, SQL-like **query** command capability.
Supported report commands can easily extract **component**, **service**, component **license**, **license policy**, **vulnerability** and other BOM information. These reports are designed to enable verification for most [BOM use cases](#cyclonedx-use-cases) as well as custom security and compliance requirements. Specifically, these commands can be used to create customized, filtered reports, in various formats *(e.g., CSV, markdown, JSON)*, using the utility's powerful, SQL-like **query** command capability to only include information *where* (i.e., using the `--where` flag) data values match specified patterns.

Supported report commands can easily extract **license**, **license policy**, **vulnerability**, **component**, **service** and other BOM information enabling verification for most [BOM use cases](#cyclonedx-use-cases) as well as custom security and compliance requirements.
The utility now includes a rich set of commands, listed below, such as **trim**, **patch** (IETF RFC 6902) and **diff**.

*Please note that the utility supports all BOM variants such as **Software** (SBOM), **Hardware** (HBOM), **Manufacturing** (MBOM), **AI/ML** (MLBOM), etc. that adhere to their respective schemas.*

Expand Down
174 changes: 128 additions & 46 deletions cmd/component.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,40 +39,105 @@ const (

var VALID_SUBCOMMANDS_COMPONENT = []string{SUBCOMMAND_COMPONENT_LIST}

var COMPONENT_LIST_ROW_DATA = []ColumnFormatData{
*NewColumnFormatData(COMPONENT_FILTER_KEY_TYPE, DEFAULT_COLUMN_TRUNCATE_LENGTH, REPORT_SUMMARY_DATA_TRUE, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_NAME, DEFAULT_COLUMN_TRUNCATE_LENGTH, REPORT_SUMMARY_DATA_TRUE, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_VERSION, DEFAULT_COLUMN_TRUNCATE_LENGTH, REPORT_SUMMARY_DATA_TRUE, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_BOMREF, DEFAULT_COLUMN_TRUNCATE_LENGTH, REPORT_SUMMARY_DATA_TRUE, REPORT_REPLACE_LINE_FEEDS_TRUE),
}

// filter keys
// Note: these string values MUST match annotations for the ComponentInfo struct fields
// Type string `json:"type"`
// Publisher string `json:"publisher,omitempty"`
// Scope string `json:"scope,omitempty"`
// Copyright string `json:"copyright,omitempty"`
// Cpe string `json:"cpe,omitempty"` // See: https://nvd.nist.gov/products/cpe
// Purl string `json:"purl,omitempty" scvs:"bom:resource:identifiers:purl"` // See: https://github.com/package-url/purl-spec
// Swid *CDXSwid `json:"swid,omitempty"`
const (
COMPONENT_FILTER_KEY_TYPE = "type"
COMPONENT_FILTER_KEY_NAME = "name"
COMPONENT_FILTER_KEY_VERSION = "version"
COMPONENT_FILTER_KEY_BOMREF = "bom-ref"
COMPONENT_FILTER_KEY_BOMREF = "bom-ref"
COMPONENT_FILTER_KEY_GROUP = "group"
COMPONENT_FILTER_KEY_TYPE = "type"
COMPONENT_FILTER_KEY_NAME = "name"
COMPONENT_FILTER_KEY_DESCRIPTION = "description"
COMPONENT_FILTER_KEY_VERSION = "version"
COMPONENT_FILTER_KEY_COPYRIGHT = "copyright"
COMPONENT_FILTER_KEY_PURL = "purl"
COMPONENT_FILTER_KEY_SWID = "swid-tag-id"
COMPONENT_FILTER_KEY_CPE = "cpe"
COMPONENT_FILTER_KEY_SUPPLIER_NAME = "supplier-name"
COMPONENT_FILTER_KEY_SUPPLIER_URL = "supplier-url"
COMPONENT_FILTER_KEY_MANUFACTURER_NAME = "manufacturer-name"
COMPONENT_FILTER_KEY_MANUFACTURER_URL = "manufacturer-url"
COMPONENT_FILTER_KEY_PUBLISHER = "publisher"
COMPONENT_FILTER_KEY_NUM_LICENSES = "number-licenses"
COMPONENT_FILTER_KEY_NUM_HASHES = "number-hashes"
COMPONENT_FILTER_KEY_HAS_PEDIGREE = "has-pedigree"
COMPONENT_FILTER_KEY_HAS_EVIDENCE = "has-evidence"
COMPONENT_FILTER_KEY_MIME_TYPE = "mime-type"
COMPONENT_FILTER_KEY_HAS_SCOPE = "scope"
COMPONENT_FILTER_KEY_HAS_COMPONENTS = "has-components"
COMPONENT_FILTER_KEY_HAS_RELEASE_NOTES = "has-release-notes"
COMPONENT_FILTER_KEY_HAS_MODEL_CARD = "has-model-card"
COMPONENT_FILTER_KEY_HAS_DATA = "has-data"
COMPONENT_FILTER_KEY_HAS_TAGS = "has-tags"
COMPONENT_FILTER_KEY_HAS_SIGNATURE = "has-signature"
)

var VALID_COMPONENT_FILTER_KEYS = []string{
COMPONENT_FILTER_KEY_BOMREF,
COMPONENT_FILTER_KEY_GROUP,
COMPONENT_FILTER_KEY_TYPE,
COMPONENT_FILTER_KEY_NAME,
COMPONENT_FILTER_KEY_DESCRIPTION,
COMPONENT_FILTER_KEY_VERSION,
COMPONENT_FILTER_KEY_BOMREF,
COMPONENT_FILTER_KEY_COPYRIGHT,
COMPONENT_FILTER_KEY_PURL,
COMPONENT_FILTER_KEY_CPE,
COMPONENT_FILTER_KEY_SWID,
COMPONENT_FILTER_KEY_SUPPLIER_NAME,
COMPONENT_FILTER_KEY_SUPPLIER_URL,
COMPONENT_FILTER_KEY_MANUFACTURER_NAME,
COMPONENT_FILTER_KEY_MANUFACTURER_URL,
COMPONENT_FILTER_KEY_PUBLISHER,
COMPONENT_FILTER_KEY_NUM_LICENSES,
COMPONENT_FILTER_KEY_NUM_HASHES,
COMPONENT_FILTER_KEY_HAS_PEDIGREE,
COMPONENT_FILTER_KEY_HAS_EVIDENCE,
COMPONENT_FILTER_KEY_MIME_TYPE,
COMPONENT_FILTER_KEY_HAS_SCOPE,
COMPONENT_FILTER_KEY_HAS_COMPONENTS,
COMPONENT_FILTER_KEY_HAS_RELEASE_NOTES,
COMPONENT_FILTER_KEY_HAS_MODEL_CARD,
COMPONENT_FILTER_KEY_HAS_DATA,
COMPONENT_FILTER_KEY_HAS_TAGS,
COMPONENT_FILTER_KEY_HAS_SIGNATURE,
}

var COMPONENT_LIST_ROW_DATA = []ColumnFormatData{
*NewColumnFormatData(COMPONENT_FILTER_KEY_BOMREF, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_GROUP, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_TYPE, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_NAME, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_VERSION, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_DESCRIPTION, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, REPORT_REPLACE_LINE_FEEDS_TRUE),
*NewColumnFormatData(COMPONENT_FILTER_KEY_COPYRIGHT, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_SUPPLIER_NAME, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_SUPPLIER_URL, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_MANUFACTURER_NAME, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_MANUFACTURER_URL, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_PUBLISHER, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_PURL, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_SWID, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_CPE, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_MIME_TYPE, REPORT_DO_NOT_TRUNCATE, false, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_HAS_SCOPE, REPORT_DO_NOT_TRUNCATE, false, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_NUM_HASHES, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_NUM_LICENSES, REPORT_DO_NOT_TRUNCATE, REPORT_SUMMARY_DATA, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_HAS_PEDIGREE, REPORT_DO_NOT_TRUNCATE, false, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_HAS_EVIDENCE, REPORT_DO_NOT_TRUNCATE, false, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_HAS_COMPONENTS, REPORT_DO_NOT_TRUNCATE, false, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_HAS_RELEASE_NOTES, REPORT_DO_NOT_TRUNCATE, false, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_HAS_MODEL_CARD, REPORT_DO_NOT_TRUNCATE, false, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_HAS_DATA, REPORT_DO_NOT_TRUNCATE, false, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_HAS_TAGS, REPORT_DO_NOT_TRUNCATE, false, false),
*NewColumnFormatData(COMPONENT_FILTER_KEY_HAS_SIGNATURE, REPORT_DO_NOT_TRUNCATE, false, false),
}

// Flags. Reuse query flag values where possible
const (
FLAG_COMPONENT_TYPE = "type"
FLAG_COMPONENT_TYPE_HELP = "filter output by component type(s)"
FLAG_COMPONENT_SUMMARY = "summary"
FLAG_COMPONENT_TYPE = "type"
// FLAG_COMPONENT_TYPE_HELP = "filter output by component type(s)"
FLAG_COMPONENT_SUMMARY_HELP = "summarize component information when listing in supported formats"
)

const (
Expand All @@ -94,7 +159,11 @@ func NewCommandComponent() *cobra.Command {
command.Long = "Report on components found in the BOM input file"
command.Flags().StringVarP(&utils.GlobalFlags.PersistentFlags.OutputFormat, FLAG_FILE_OUTPUT_FORMAT, "", FORMAT_TEXT,
FLAG_COMPONENT_OUTPUT_FORMAT_HELP+COMPONENT_LIST_OUTPUT_SUPPORTED_FORMATS)
command.Flags().StringP(FLAG_COMPONENT_TYPE, "", "", FLAG_COMPONENT_TYPE_HELP)
//command.Flags().StringP(FLAG_COMPONENT_TYPE, "", "", FLAG_COMPONENT_TYPE_HELP)
command.Flags().BoolVarP(
&utils.GlobalFlags.ComponentFlags.Summary,
FLAG_COMPONENT_SUMMARY, "", false,
FLAG_COMPONENT_SUMMARY_HELP)
command.Flags().StringP(FLAG_REPORT_WHERE, "", "", FLAG_REPORT_WHERE_HELP)
command.RunE = componentCmdImpl
command.ValidArgs = VALID_SUBCOMMANDS_COMPONENT
Expand Down Expand Up @@ -145,7 +214,7 @@ func componentCmdImpl(cmd *cobra.Command, args []string) (err error) {
whereFilters, err := processWhereFlag(cmd)

if err == nil {
err = ListComponents(writer, utils.GlobalFlags.PersistentFlags, whereFilters)
err = ListComponents(writer, utils.GlobalFlags.PersistentFlags, utils.GlobalFlags.ComponentFlags, whereFilters)
}

return
Expand All @@ -160,7 +229,7 @@ func processComponentListResults(err error) {
}

// NOTE: resourceType has already been validated
func ListComponents(writer io.Writer, persistentFlags utils.PersistentCommandFlags, whereFilters []common.WhereFilter) (err error) {
func ListComponents(writer io.Writer, persistentFlags utils.PersistentCommandFlags, flags utils.ComponentCommandFlags, whereFilters []common.WhereFilter) (err error) {
getLogger().Enter()
defer getLogger().Exit()

Expand Down Expand Up @@ -191,16 +260,16 @@ func ListComponents(writer io.Writer, persistentFlags utils.PersistentCommandFla
getLogger().Infof("Outputting listing (`%s` format)...", format)
switch format {
case FORMAT_TEXT:
err = DisplayComponentListText(document, writer)
err = DisplayComponentListText(document, writer, flags)
case FORMAT_CSV:
err = DisplayComponentListCSV(document, writer)
err = DisplayComponentListCSV(document, writer, flags)
case FORMAT_MARKDOWN:
err = DisplayComponentListMarkdown(document, writer)
err = DisplayComponentListMarkdown(document, writer, flags)
default:
// Default to Text output for anything else (set as flag default)
getLogger().Warningf("Listing not supported for `%s` format; defaulting to `%s` format...",
format, FORMAT_TEXT)
err = DisplayComponentListText(document, writer)
err = DisplayComponentListText(document, writer, flags)
}
return
}
Expand Down Expand Up @@ -232,21 +301,25 @@ func loadDocumentComponents(document *schema.BOM, whereFilters []common.WhereFil
return
}

// NOTE: component hashmap values are pointers to CDXComponentInfo structs
func sortComponents(entries []multimap.Entry) {
// Sort by Type then Name
sort.Slice(entries, func(i, j int) bool {
resource1 := (entries[i].Value).(schema.CDXComponentInfo)
resource2 := (entries[j].Value).(schema.CDXComponentInfo)
if resource1.ResourceType != resource2.ResourceType {
return resource1.ResourceType < resource2.ResourceType
resource1 := (entries[i].Value).(*schema.CDXComponentInfo)
resource2 := (entries[j].Value).(*schema.CDXComponentInfo)
if resource1.Group != resource2.Group {
return resource1.Group < resource2.Group
}
if resource1.Type != resource2.Type {
return resource1.Type < resource2.Type
}
return resource1.Name < resource2.Name
})
}

// NOTE: This list is NOT de-duplicated
// TODO: Add a --no-title flag to skip title output
func DisplayComponentListText(bom *schema.BOM, writer io.Writer) (err error) {
func DisplayComponentListText(bom *schema.BOM, writer io.Writer, flags utils.ComponentCommandFlags) (err error) {
getLogger().Enter()
defer getLogger().Exit()

Expand All @@ -258,7 +331,7 @@ func DisplayComponentListText(bom *schema.BOM, writer io.Writer) (err error) {
w.Init(writer, 8, 2, 2, ' ', 0)

// create title row and underline row from slices of optional and compulsory titles
titles, underlines := prepareReportTitleData(COMPONENT_LIST_ROW_DATA, true)
titles, underlines := prepareReportTitleData(COMPONENT_LIST_ROW_DATA, flags.Summary)

// Add tabs between column titles for the tabWRiter
fmt.Fprintf(w, "%s\n", strings.Join(titles, "\t"))
Expand All @@ -278,11 +351,14 @@ func DisplayComponentListText(bom *schema.BOM, writer io.Writer) (err error) {

// Emit row data
var line []string
var pComponentInfo *schema.CDXComponentInfo
for _, entry := range entries {
// NOTE: component hashmap values are pointers to CDXComponentInfo structs
pComponentInfo = entry.Value.(*schema.CDXComponentInfo)
line, err = prepareReportLineData(
entry.Value.(schema.CDXComponentInfo),
*pComponentInfo,
COMPONENT_LIST_ROW_DATA,
true,
flags.Summary,
)
// Only emit line if no error
if err != nil {
Expand All @@ -294,7 +370,7 @@ func DisplayComponentListText(bom *schema.BOM, writer io.Writer) (err error) {
}

// TODO: Add a --no-title flag to skip title output
func DisplayComponentListCSV(bom *schema.BOM, writer io.Writer) (err error) {
func DisplayComponentListCSV(bom *schema.BOM, writer io.Writer, flags utils.ComponentCommandFlags) (err error) {
getLogger().Enter()
defer getLogger().Exit()

Expand All @@ -303,14 +379,14 @@ func DisplayComponentListCSV(bom *schema.BOM, writer io.Writer) (err error) {
defer w.Flush()

// Create title row data as []string
titles, _ := prepareReportTitleData(COMPONENT_LIST_ROW_DATA, true)
titles, _ := prepareReportTitleData(COMPONENT_LIST_ROW_DATA, flags.Summary)

if err = w.Write(titles); err != nil {
return getLogger().Errorf("error writing to output (%v): %s", titles, err)
}

// Display a warning "missing" in the actual output and return (short-circuit)
entries := bom.ResourceMap.Entries()
entries := bom.ComponentMap.Entries()

// Emit no resource found warning into output
if len(entries) == 0 {
Expand All @@ -326,11 +402,14 @@ func DisplayComponentListCSV(bom *schema.BOM, writer io.Writer) (err error) {
sortComponents(entries)

var line []string
var pComponentInfo *schema.CDXComponentInfo
for _, entry := range entries {
// NOTE: component hashmap values are pointers to CDXComponentInfo structs
pComponentInfo = entry.Value.(*schema.CDXComponentInfo)
line, err = prepareReportLineData(
entry.Value.(schema.CDXResourceInfo),
*pComponentInfo,
COMPONENT_LIST_ROW_DATA,
true,
flags.Summary,
)
// Only emit line if no error
if err != nil {
Expand All @@ -344,22 +423,22 @@ func DisplayComponentListCSV(bom *schema.BOM, writer io.Writer) (err error) {
}

// TODO: Add a --no-title flag to skip title output
func DisplayComponentListMarkdown(bom *schema.BOM, writer io.Writer) (err error) {
func DisplayComponentListMarkdown(bom *schema.BOM, writer io.Writer, flags utils.ComponentCommandFlags) (err error) {
getLogger().Enter()
defer getLogger().Exit()

// Create title row data as []string, include all columns that are flagged "summary" data
titles, _ := prepareReportTitleData(COMPONENT_LIST_ROW_DATA, true)
titles, _ := prepareReportTitleData(COMPONENT_LIST_ROW_DATA, flags.Summary)
titleRow := createMarkdownRow(titles)
fmt.Fprintf(writer, "%s\n", titleRow)

// create alignment row, include all columns that are flagged "summary" data
alignments := createMarkdownColumnAlignmentRow(COMPONENT_LIST_ROW_DATA, true)
alignments := createMarkdownColumnAlignmentRow(COMPONENT_LIST_ROW_DATA, flags.Summary)
alignmentRow := createMarkdownRow(alignments)
fmt.Fprintf(writer, "%s\n", alignmentRow)

// Display a warning "missing" in the actual output and return (short-circuit)
entries := bom.ResourceMap.Entries()
entries := bom.ComponentMap.Entries()

// Emit no components found warning into output
if len(entries) == 0 {
Expand All @@ -372,11 +451,14 @@ func DisplayComponentListMarkdown(bom *schema.BOM, writer io.Writer) (err error)

var line []string
var lineRow string
var pComponentInfo *schema.CDXComponentInfo
for _, entry := range entries {
// NOTE: component hashmap values are pointers to CDXComponentInfo structs
pComponentInfo = entry.Value.(*schema.CDXComponentInfo)
line, err = prepareReportLineData(
entry.Value.(schema.CDXResourceInfo),
*pComponentInfo,
COMPONENT_LIST_ROW_DATA,
true,
flags.Summary,
)
// Only emit line if no error
if err != nil {
Expand Down
Loading

0 comments on commit f892978

Please sign in to comment.