Implement improved demo build pipeline

[brownj85]

Context

This PR is the initial phase of a re-factor of the demo build process. The goal is to improve the organization of the QML repo, make it easier to use, improve build times and simplify the process of deploying to pennylane.ai. To that end, this PR introduces a command line tool qml which will allow us to abstract over the sphinx-build process, and acts as an extension point to introduce new tooling. The qml tool can be install by running pip install . in the repository root.

Changes

Introduce a new structure for the demo files that gives each demo its own directory, containing the metadata (metadata.json), runtime requirements (requirements.in), any resources used by the demo, as well as the demo file itself (demo.py). Since this structure is not directly compatible with sphinx or sphinx-gallery, we use the qml tool to orchestrate the build process. (The new demo files are stored in the demonstrations_v2 directory, to avoid interfering with the existing build).

The command qml build [demo names] does the following:

• copies the requested demo(s) into staging directory _build/demonstrations, structured to be compatible with sphinx-gallery
• if execution is requested (--execute), install execution dependencies for requested demo(s). The package versions are constrained by the executable-dependencies dependency group in pyproject.toml
• calls sphinx-build on the staging directory

Testing

Testers should verifying that the qml build command executes correctly using json and html formats, and produces output identical to the make html or make json commands. Testers should also run qml build with and without the --execute flag.

Installation

Create a Python3.10 virtual environment and install the qml tool:

qml $ python3.10 -m venv .venv
qml $ source .venv/bin/activate
qml (.venv) $ pip install -e .

Then qml build can be run anywhere in the repo:

qml build --format {html, json} [--execute]

[anthayes92]

Looks like a great improvement, nice work @brownj85! 🚀

I've reviewed the additions to lib/qml/* here. A few questions and suggestions.

Are the tutorials mainly copy code? Probably too much to be reviewed in a PR, perhaps this can be QA'd interactively?

[anthayes92]

Question

Should we include some error handling for processing the args here?

[brownj85]

No, we can just delegate to the underlying process to raise an appropriate error.

[anthayes92]

Question

For my understanding - is this all being performed on a GitHub runner or are we executing this on some other hardware?

[brownj85]

This will be run in a github runner, for the time being. We may look into moving demo execution into software cloud.

[Alan-eMartin]

Question:
@brownj85 — Do you have any suggestions on how we can test this locally?

Or as @anthayes92 mentioned above, should we do a live QA of these changes?

[brownj85]

Hey @Alan-eMartin, @anthayes92 added some suggestions on testing to the top comment.

[Alan-eMartin]

Thanks for the presentation yesterday @brownj85, looks great.

[rashidnhm]

Would it make sense to use the qml CLI tool in the CI builds now, or would that be introduced in a subsequent PR?

[brownj85]

Would it make sense to use the qml CLI tool in the CI builds now, or would that be introduced in a subsequent PR?

That will be in a a subsequent PR - the diff for this is already massive.

[github-actions]

Thank you for opening this pull request.

You can find the built site at this link.

Deployment Info:

• Pull Request ID: 1286
• Deployment SHA: 00f517ce15815f960c77699147f7ba1c1cb64b9c
  (The Deployment SHA refers to the latest commit hash the docs were built from)

Note: It may take several minutes for updates to this pull request to be reflected on the deployed site.