New command-line interface (options)

[zvr]

This (finally) adds the brand new CLI.
It introduces a number of new command-line options, and essentially re-implements the top-level code of spec-parser.

As can be seen in the updated README, one can now call and only generate what is required -- or nothing at all, if only input validation is needed.

usage: main.py [-h] [-V] [-d] [-v] [-f] [-n]
               [-o OUTPUT]
               [-j] [-J dir]
               [-m] [-M dir]
               [-p] [-P dir]
               [-r] [-R dir]
               [-t] [-T dir]
               [-w] [-W dir]
               input_dir

Generate documentation from an SPDXv3 model.

positional arguments:
  input_dir             Path to the input 'model' directory.

options:
  -h, --help                                Show this help message and exit
  -d, --debug                               Print debug output
  -f, --force                               Force overwrite of existing output directories.
  -j, --generate-jsondump                   Generate a dump of the model in JSON format.
  -J, --output-jsondump OUTPUT_JSONDUMP     Output directory for JSON dump file.
  -m, --generate-mkdocs                     Generate mkdocs output.
  -M, --output-mkdocs OUTPUT_MKDOCS         Output directory for mkdocs files.
  -n, --no-output                           Perform no output generation, only input validation.
  -o, --output OUTPUT                       Single output directory for all output types.
  -p, --generate-plantuml                   Generate PlantUML output.
  -P, --output-plantuml OUTPUT_PLANTUML     Output directory for PlantUML files.
  -r, --generate-rdf                        Generate RDF output.
  -R, --output-rdf OUTPUT_RDF               Output directory for RDF files.
  -t, --generate-tex                        Generate TeX output.
  -T, --output-tex OUTPUT_TEX               Output directory for TeX files.
  -v, --verbose                             Print verbose output
  -V, --version                             Show program version number and exit
  -w, --generate-webpages                   Generate web pages output.
  -W, --output-webpages OUTPUT_WEBPAGES     Output directory for web pages.

Fixes #152
Closes #161
Closes #179

[zvr]

I will wait before merging this, to give a chance to update the workflows in other repos.

[goneall]

@zvr - Should we start tagging releases for the spec parser? That way we can update the CI's to reference the tag and not worry about breaking them when we update the spec parser. The other advantage is that the change to update the version in the CI will trigger the CI and test out the changes before unrelated merges happen.

To implement this suggestion:

• Tag the current main (e.g. 0.1.0)
• Update the CI in spdx-spec repo to reference that tag rather than main
• Update the CI in spdx-3-model to reference the tag rather than main
• Merge this PR
• Create a new tag (e.g. 1.0.0)
• Update the CI in spdx-spec repo to use the new command line and update the reference in the CI
• Update the CI in spdx-3-model repo to use the new command line and update the reference in the CI

[bact]

The new command line looks great. Thank you @zvr

Also agrees with @goneall suggestions on versioning/tagging.

Workflows to be updated (in different branches, may including support/3.0):

• Validation on spdx-3-model repo: https://github.com/spdx/spdx-3-model/blob/main/.github/workflows/validate-pr.yml
  • Relevant lines: https://github.com/spdx/spdx-3-model/blob/a745f63e8643d5ae0f0851fcfa6836085308f80b/.github/workflows/validate-pr.yml#L24
• Publication on spdx-spec repo: https://github.com/spdx/spdx-spec/blob/develop/.github/workflows/publish_v3.yml
  • Relevant lines: https://github.com/spdx/spdx-spec/blob/61a649da8ca27924ac1ca8d2a061cb228839b24c/.github/workflows/publish_v3.yml#L40-L43

[zvr]

@bact On the workflows:

• for the model repo, there is no change needed, the invocation with -n is correct. One might delete the call with -h while setting it up (line 21), which does not serve any purpose
• for the spec repo, from a quick look the changes should be:
  • fix the invocation of spec-parser on line 151
  • no need to install a fake pandoc (lines 145-149)
  • probably no need to do all the copying around (lines 166-187, 200-208, etc.) or creating directories (lines 152-155), if the invocation (first sub-bullet above) specifies where the output for each type (mkdocs and rdf) should go.

[zvr]

I disagree on the tagging, especially if combined with the understanding that "tags are immutable".

There is no plan to support more than one (the latest) version of spec-parser. So, any changes/enhancements/bug fixes will only be done in a sequential linear way, on a single branch. I mean, if we decide that "superclasses" output need to be changed (cf. spdx/spdx-3-model#960) the changes will only be applied to the latest version (for example, the one with the updated CLI options, if it has been merged). Therefore we should really discourage use of older versions.

Giving "version" tags is helpful when the "API" changes in an incompatible way, so that callers can still use the old one (as Gary described). But this is not happening "often" (or ever). This change on this PR is the only such change in the last two years and I have announced it... a year(?) ago. On the other hand, there will be changes in the functionality (for example, generating the web pages to point the RDF endpoints to), without change in the calling API. Having an immutable tag on a version prevents users from getting the latest enhancements/fixes for this.

There will be more API changes coming (e.g. including an option to specify which languages should be handled, when multi-language input is available), but my plan is to have all these changes made in a backwards compatible manner -- that's why I took so long to design and implement the system in this PR.

In the end, if any script/automation really needs to use a specific instance of spec-parser, it can always use the corresponding commit-id.

[bact]

• for the model repo, there is no change needed, the invocation with -n is correct. One might delete the call with -h while setting it up (line 21), which does not serve any purpose

Do we like to have --debug here as well so the PR author have more information ?

• (Update: PR proposed here: Print verbose info during PR validation (target: 'develop' branch) spdx-3-model#963)

[zvr]

Do we like to have --debug here as well so the PR author have more information ?

No, --debug is for debugging the spec-parser software; it should never be used by people not interested in modifying spec-parser.

[bact]

Put in help description to specifically say that --debug is for the debug of spec-parser (not the debug of the Markdown spec file) as described by the author in the comment section.

General stylistic suggestions. No need to change if feel unnecessary.

[bact]

Btw, in the option descriptions, we have two kinds of "output".

• Screen output, as in --debug and --verbose
• File output, as in TeX/MkDocs/HTML/etc generation (--generate-, --output- and --no-output)

I suggest that we change the descriptions of --debug and --verbose to use the word "information" (or "log", etc) instead of "output" to clearly differentiate the two.