From 4c18f170d1ec07c69f5e55debef9ff5262158ae3 Mon Sep 17 00:00:00 2001 From: Gil Robalo Rei Date: Sat, 18 Jan 2025 10:41:58 +0100 Subject: [PATCH 1/6] feat: add markers in README and CONTRIBUTING --- CONTRIBUTING.md | 2 ++ README.md | 23 ++++++++++++++++++----- 2 files changed, 20 insertions(+), 5 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index cd9980d9..2baf94ef 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -32,9 +32,11 @@ have to solve them yourself. ### 1. Install QUEENS in developer mode Install QUEENS as described in the [README.md](README.md) and run: + ``` pip install -e .[develop] ``` + ### 2. Configure our git-hooks To help you write style-compliant code, we use the [pre-commit](https://pre-commit.com/) package to manage all our git diff --git a/README.md b/README.md index fb756001..9cbdfec5 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,11 @@ -
+ - - - QUEENS logo + + + QUEENS logo +
@@ -24,8 +25,11 @@
+ QUEENS (**Q**uantification of **U**ncertain **E**ffects in **En**gineering **S**ystems) is a Python framework for solver-independent multi-query analyses of large-scale computational models. + + :chart_with_upwards_trend: **QUEENS** offers a large collection of cutting-edge algorithms for deterministic and probabilistic analyses such as: * parameter studies and identification * sensitivity analysis @@ -38,17 +42,22 @@ QUEENS (**Q**uantification of **U**ncertain **E**ffects in **En**gineering **S** * robust data, resource, and error management * easy switching between analysis types * smooth scaling from laptop to HPC cluster + ## :rocket: Getting started + >**Prerequisites**: Unix system and environment management system (we recommend [miniforge](https://conda-forge.org/download/)) + + Clone the QUEENS repository to your local machine. Navigate to its base directory, then: ``` conda env create conda activate queens pip install -e . ``` + ## :crown: Workflow example @@ -97,7 +106,7 @@ if __name__ == "__main__":
-QUEENS logo +QUEENS example
## :busts_in_silhouette: Contributing @@ -107,6 +116,7 @@ Your contributions are welcome! Please follow our [contributing guidelines](http ## :page_with_curl: How to cite If you use QUEENS in your work, please cite the relevant method papers and + ```bib @misc{queens, author = {QUEENS}, @@ -115,6 +125,9 @@ If you use QUEENS in your work, please cite the relevant method papers and howpublished = {\url{https://www.queens-py.org}} } ``` + ## :woman_judge: License + Licensed under GNU LGPL-3.0 (or later). See [LICENSE](LICENSE). + From 0f9b93783f2c0c87f3641571640e5ecbb0d99603 Mon Sep 17 00:00:00 2001 From: Gil Robalo Rei Date: Sat, 18 Jan 2025 10:42:39 +0100 Subject: [PATCH 2/6] refactor(tests/README.md): format table --- tests/README.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/tests/README.md b/tests/README.md index 1e2f1fdf..7134fe90 100644 --- a/tests/README.md +++ b/tests/README.md @@ -16,23 +16,23 @@ Therefore, we test the QUEENS code base ## :running_woman: Running tests QUEENS is tested using [pytest](https://docs.pytest.org/en/stable/index.html). For a comprehensive list of pytest commands, see [here](https://docs.pytest.org/en/stable/how-to/usage.html). Some additional useful commands to test QUEENS are listed in the following: -| Description | Command | -| ----------- | ----------- | -| Test parallelly using pytest-xdist | `pytest -n ` | -| Test with verbose output | `pytest -ra -v` | -| Test with logging output | `pytest -o log_cli=true --log-cli-level=INFO` | -| Generate a coverage report | `pytest --cov-report=html --cov` | -| Test only the last failed | `pytest --lf` | +| Test | Command | +| ----------------------------- | --------------------------------------------- | +| In parallel with pytest-xdist | `pytest -n ` | +| With verbose output | `pytest -ra -v` | +| With logging output | `pytest -o log_cli=true --log-cli-level=INFO` | +| With coverage report | `pytest --cov-report=html --cov` | +| Only the last failed | `pytest --lf` | ### :bookmark: Pytest markers In QUEENS, tests are organized using pytest markers. This allows you to run all tests in a group with a single command: -| Description| Command | -| ----------- | ----------- | -| Unit tests | `pytest -m unit_tests` | -| Integration tests without 4C | `pytest -m integration_tests` | -| Integration tests with 4C (see below) | `pytest -m integration_tests_fourc` | -|Get a list of all available markers| `pytest --markers` | +| Description | Command | +| ------------------------------- | ----------------------------------- | +| Unit tests | `pytest -m unit_tests` | +| Integration tests | `pytest -m integration_tests` | +| 4C integration test (see below) | `pytest -m integration_tests_fourc` | +| List markers | `pytest --markers` | ### :four_leaf_clover: Integration tests with 4C For the integration tests in QUEENS that require the multiphysics simulation framework [4C](https://github.com/4C-multiphysics/4C), the user needs to create a **symbolic link** to the 4C-executable and store it under `/config`: From 42cc12c21ed93513290cca241bf3c7c4d4ef39df Mon Sep 17 00:00:00 2001 From: Gil Robalo Rei Date: Sat, 18 Jan 2025 10:45:19 +0100 Subject: [PATCH 3/6] feat(images): add images for README these images are downloaded to ensure that everything works offline --- .../source/images}/monte_carlo_uq.png | Bin doc/source/images/queens_logo_day.svg | 327 ++++++++++++++++++ doc/source/images/queens_logo_night.svg | 327 ++++++++++++++++++ 3 files changed, 654 insertions(+) rename {readme_images => doc/source/images}/monte_carlo_uq.png (100%) create mode 100644 doc/source/images/queens_logo_day.svg create mode 100644 doc/source/images/queens_logo_night.svg diff --git a/readme_images/monte_carlo_uq.png b/doc/source/images/monte_carlo_uq.png similarity index 100% rename from readme_images/monte_carlo_uq.png rename to doc/source/images/monte_carlo_uq.png diff --git a/doc/source/images/queens_logo_day.svg b/doc/source/images/queens_logo_day.svg new file mode 100644 index 00000000..96d520b5 --- /dev/null +++ b/doc/source/images/queens_logo_day.svg @@ -0,0 +1,327 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + QUEENS + + diff --git a/doc/source/images/queens_logo_night.svg b/doc/source/images/queens_logo_night.svg new file mode 100644 index 00000000..e9e0ea15 --- /dev/null +++ b/doc/source/images/queens_logo_night.svg @@ -0,0 +1,327 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + QUEENS + + From 6f6256f62a7562af5dfcea33874122e0ae43d568 Mon Sep 17 00:00:00 2001 From: Gil Robalo Rei Date: Sat, 18 Jan 2025 10:45:47 +0100 Subject: [PATCH 4/6] feat(doc): introduce custom sphinx extension --- .gitignore | 4 +- CONTRIBUTING.md | 2 +- dev-requirements.in | 1 + dev-requirements.txt | 17 ++ doc/source/_ext/create_documentation_files.py | 250 ++++++++++++++++++ doc/source/_ext/queens_sphinx_extension.py | 38 +++ doc/source/_ext/templates/development.rst.j2 | 11 + doc/source/_ext/templates/intro.md.j2 | 37 +++ doc/source/_ext/templates/overview.rst.j2 | 16 ++ doc/source/_ext/templates/tutorials.rst.j2 | 19 ++ doc/source/architecture.rst | 115 -------- doc/source/conf.py | 14 +- doc/source/index.rst | 22 +- doc/source/intro.rst | 10 - doc/source/tutorials.rst | 6 - test_utils/get_queens_example_from_readme.py | 34 ++- 16 files changed, 435 insertions(+), 161 deletions(-) create mode 100644 doc/source/_ext/create_documentation_files.py create mode 100644 doc/source/_ext/queens_sphinx_extension.py create mode 100644 doc/source/_ext/templates/development.rst.j2 create mode 100644 doc/source/_ext/templates/intro.md.j2 create mode 100644 doc/source/_ext/templates/overview.rst.j2 create mode 100644 doc/source/_ext/templates/tutorials.rst.j2 delete mode 100644 doc/source/architecture.rst delete mode 100644 doc/source/intro.rst delete mode 100644 doc/source/tutorials.rst diff --git a/.gitignore b/.gitignore index 29e50e65..7518a3d7 100644 --- a/.gitignore +++ b/.gitignore @@ -2,12 +2,10 @@ __pycache__ # do not track doc files that are automatically build doc/build doc/source/*.rst +doc/source/*.md # but keep index.rst !doc/source/index.rst -!doc/source/intro.rst -!doc/source/tutorials.rst -!doc/source/architecture.rst # C extensions *.so diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2baf94ef..46a80f38 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -56,7 +56,7 @@ The code checks are conducted with [Pylint](https://pylint.org/), Compliance with [Google style docstrings](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings) is checked with [ruff](https://github.com/astral-sh/ruff). Complete and meaningful docstrings are required as they are used to generate the -[documentation](#reading-and-writing-documentation). +[documentation](#book-documentation). #### Commit messages Please provide meaningful commit messages based on the diff --git a/dev-requirements.in b/dev-requirements.in index 3127e5dc..2ee9356c 100644 --- a/dev-requirements.in +++ b/dev-requirements.in @@ -20,3 +20,4 @@ commitizen>=3.12.0 docformatter>=1.5.1 yamllint>=1.19.0 ruff +myst-parser diff --git a/dev-requirements.txt b/dev-requirements.txt index e46df750..17e122b7 100644 --- a/dev-requirements.txt +++ b/dev-requirements.txt @@ -62,6 +62,7 @@ docformatter==1.7.5 # via -r dev-requirements.in docutils==0.21.2 # via + # myst-parser # nbsphinx # pydata-sphinx-theme # sphinx @@ -87,6 +88,7 @@ jinja2==3.1.4 # via # -c requirements.txt # commitizen + # myst-parser # nbconvert # nbsphinx # sphinx @@ -108,6 +110,11 @@ liccheck==0.9.2 # via -r dev-requirements.in licenseheaders==0.8.8 # via -r dev-requirements.in +markdown-it-py==3.0.0 + # via + # -c requirements.txt + # mdit-py-plugins + # myst-parser markupsafe==3.0.2 # via # -c requirements.txt @@ -115,8 +122,16 @@ markupsafe==3.0.2 # nbconvert mccabe==0.7.0 # via pylint +mdit-py-plugins==0.4.2 + # via myst-parser +mdurl==0.1.2 + # via + # -c requirements.txt + # markdown-it-py mistune==3.0.2 # via nbconvert +myst-parser==4.0.0 + # via -r dev-requirements.in nbclient==0.10.0 # via nbconvert nbconvert==7.16.4 @@ -188,6 +203,7 @@ pyyaml==6.0.2 # via # -c requirements.txt # commitizen + # myst-parser # pre-commit # yamllint pyzmq==26.2.0 @@ -228,6 +244,7 @@ soupsieve==2.6 sphinx==8.1.3 # via # -r dev-requirements.in + # myst-parser # nbsphinx # pydata-sphinx-theme sphinxcontrib-applehelp==2.0.0 diff --git a/doc/source/_ext/create_documentation_files.py b/doc/source/_ext/create_documentation_files.py new file mode 100644 index 00000000..64da2327 --- /dev/null +++ b/doc/source/_ext/create_documentation_files.py @@ -0,0 +1,250 @@ +#!/usr/bin/env python3 +# +# SPDX-License-Identifier: LGPL-3.0-or-later +# Copyright (c) 2025, QUEENS contributors. +# +# This file is part of QUEENS. +# +# QUEENS is free software: you can redistribute it and/or modify it under the terms of the GNU +# Lesser General Public License as published by the Free Software Foundation, either version 3 of +# the License, or (at your option) any later version. QUEENS is distributed in the hope that it will +# be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You +# should have received a copy of the GNU Lesser General Public License along with QUEENS. If not, +# see . +# +"""Documentation creation utils.""" + +import pydoc +import re +import sys +from pathlib import Path + +import requests + +from queens.utils.injector import inject +from queens.utils.path_utils import relative_path_from_queens + +sys.path.insert(1, str(relative_path_from_queens("test_utils").resolve())) +from get_queens_example_from_readme import ( # pylint: disable=import-error, wrong-import-position,wrong-import-order + extract_from_markdown_file_by_marker, + get_queens_example_from_readme, +) + + +def get_template_path_by_name(template_name): + """Get template path by name from the template folder. + + Args: + template_name (str): Temple name in the template folder. + + Returns: + pathlib.Path: Path to template + """ + template_path = (Path(__file__).parent / "templates") / template_name + return template_path + + +def relative_to_doc_source(relative_path): + """Relative path from documentation source. + + Args: + relative_path (str): path relative to `doc/source/` + + Returns: + pathlib.Path: Path relative from documentation + """ + return relative_path_from_queens("doc/source/" + relative_path) + + +def create_tutorial_from_readme(): + """Create tutorial from readme.""" + example = get_queens_example_from_readme(".") + tutorial_template = get_template_path_by_name("tutorials.rst.j2") + tutorial = relative_to_doc_source("tutorials.rst") + + inject({"example_text": example.replace("\n", "\n ")}, tutorial_template, tutorial) + + +def remove_markdown_emojis(md_text): + """Remove emojis from markdown text. + + Args: + md_text (str): Markdown text + + Returns: + str: Cleaned text + """ + # Define a regex pattern for matching markdown-style emojis + emoji_pattern = re.compile(r":\w+:") + + for emoji in emoji_pattern.findall(md_text): + + # Remove markdown emojis from the text + md_text = md_text.replace(emoji, "") + + # Replace emojis in reference to other sections + md_text = md_text.replace(emoji[1:-1], "") + + return md_text + + +def prepend_relative_links(md_text, base_url): + """Prepend url for relative links. + + Args: + md_text (str): Text to check + base_url (str): Base URL to add + + Returns: + str: Prepended markdown text + """ + md_link_regex = "\\[([^]]+)]\\(\\s*(.*)\\s*\\)" + for match in re.findall(md_link_regex, md_text): + _, link = match + + # For local reference that started with an emoji + if link.strip().startswith("#-"): + new_link = "#" + link[2:].strip().lower() + md_text = md_text.replace(f"({link})", f"({new_link})") + + # No http links or proper references within the file references + if not link.strip().startswith("http") and not link.strip().startswith("#"): + md_text = md_text.replace(f"({link})", f"({base_url}/{link.strip()})") + + return md_text + + +def clean_markdown(md_text): + """Clean markdown. + + Removes emojis and prepends links. + + Args: + md_text (str): Original markdown text. + + Returns: + str: Markdown text + """ + md_text = remove_markdown_emojis(md_text) + md_text = prepend_relative_links(md_text, "https://www.github.com/queens-py/queens/blob/main") + return md_text + + +def clean_markdown_file(md_path, new_path): + """Load markdown and escape relative links and emojis. + + Args: + md_path (pathlib.Path, str): Path to an existing markdown file + new_path (pathlib.Path, str): Path for the cleaned file + + Returns: + str: file name of the new markdown file + """ + md_text = clean_markdown(relative_path_from_queens(md_path).read_text()) + new_path = Path(new_path) + new_path.write_text(md_text, encoding="utf-8") + return new_path.name + + +def create_development(): + """Create development page.""" + development_template = get_template_path_by_name("development.rst.j2") + development_path = relative_to_doc_source("development.rst") + + md_paths = [] + md_paths.append( + clean_markdown_file( + relative_path_from_queens("CONTRIBUTING.md"), relative_to_doc_source("contributing.md") + ) + ) + md_paths.append( + clean_markdown_file( + relative_path_from_queens("tests/README.md"), relative_to_doc_source("testing.md") + ) + ) + inject({"md_paths": md_paths}, development_template, development_path) + + +def create_intro(): + """Generate landing page.""" + intro_template = get_template_path_by_name("intro.md.j2") + into_path = relative_to_doc_source("intro.md") + + def extract_from_markdown_by_marker(marker_name, md_path): + return clean_markdown(extract_from_markdown_file_by_marker(marker_name, md_path)) + + inject( + { + "readme_path": relative_path_from_queens("README.md"), + "contributing_path": relative_path_from_queens("CONTRIBUTING.md"), + "extract_from_markdown_by_marker": extract_from_markdown_by_marker, + }, + intro_template, + into_path, + ) + + +def create_overview(): + """Create overview of the QUEENS package.""" + overview_template = get_template_path_by_name("overview.rst.j2") + overview_path = relative_to_doc_source("overview.rst") + + queens_base_path = relative_path_from_queens("queens") + + def get_module_description(python_file): + """Get module description. + + Args: + python_file (pathlib.Path): Path to python file. + + Returns: + str: Module description. + """ + module_documentation = pydoc.importfile(str(python_file)).__doc__.split("\n\n") + return "\n\n".join([m.replace("\n", " ") for m in module_documentation[1:]]) + + modules = [] + for path in sorted(queens_base_path.iterdir()): + if path.name.startswith("__") or not path.is_dir(): + continue + + description = get_module_description(path / "__init__.py") + name = path.stem + + modules.append( + { + "name": name.replace("_", " ").title(), + "module": "queens." + name, + "description": description, + } + ) + + inject({"modules": modules, "len": len}, overview_template, overview_path) + + +def download_images(): + """Download images.""" + + def download_file_from_url(url, file_name): + """Download file from an url.""" + url_request = requests.get(url, timeout=10) + with open(file_name, "wb") as f: + f.write(url_request.content) + + download_file_from_url( + "https://raw.githubusercontent.com/queens-py/queens-design/main/logo/queens_logo_night.svg", + relative_to_doc_source("images/queens_logo_night.svg"), + ) + download_file_from_url( + "https://raw.githubusercontent.com/queens-py/queens-design/main/logo/queens_logo_day.svg", + relative_to_doc_source("images/queens_logo_day.svg"), + ) + + +def main(): + """Create all the rst files.""" + create_intro() + create_tutorial_from_readme() + create_development() + create_overview() diff --git a/doc/source/_ext/queens_sphinx_extension.py b/doc/source/_ext/queens_sphinx_extension.py new file mode 100644 index 00000000..0946606d --- /dev/null +++ b/doc/source/_ext/queens_sphinx_extension.py @@ -0,0 +1,38 @@ +# +# SPDX-License-Identifier: LGPL-3.0-or-later +# Copyright (c) 2025, QUEENS contributors. +# +# This file is part of QUEENS. +# +# QUEENS is free software: you can redistribute it and/or modify it under the terms of the GNU +# Lesser General Public License as published by the Free Software Foundation, either version 3 of +# the License, or (at your option) any later version. QUEENS is distributed in the hope that it will +# be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You +# should have received a copy of the GNU Lesser General Public License along with QUEENS. If not, +# see . +# +"""This is a custom extension to create files for sphinx using python.""" + +import os +import sys + +from sphinx.application import Sphinx + +sys.path.insert(0, os.path.abspath(".")) + +import create_documentation_files # pylint: disable=wrong-import-position + + +def run_custom_code(app: Sphinx): # pylint: disable=unused-argument + """Run the custom code.""" + create_documentation_files.main() + + +def setup(app: Sphinx): # pylint: disable=unused-argument + """Setup up sphinx app.""" + app.connect("builder-inited", run_custom_code) + return { + "version": "0.1", + "parallel_read_safe": True, + } diff --git a/doc/source/_ext/templates/development.rst.j2 b/doc/source/_ext/templates/development.rst.j2 new file mode 100644 index 00000000..7e49d1d4 --- /dev/null +++ b/doc/source/_ext/templates/development.rst.j2 @@ -0,0 +1,11 @@ +Development +=========== + +Thanks for your interest in developing QUEENS! + +.. toctree:: + :maxdepth: 1 + + +{% for md_path in md_paths %} + {{ md_path }}{% endfor %} diff --git a/doc/source/_ext/templates/intro.md.j2 b/doc/source/_ext/templates/intro.md.j2 new file mode 100644 index 00000000..3b159c4b --- /dev/null +++ b/doc/source/_ext/templates/intro.md.j2 @@ -0,0 +1,37 @@ +# Introduction + +```{image} images/queens_logo_day.svg +:class: only-light +:width: 500px +:align: center +``` + +```{image} images/queens_logo_night.svg +:class: only-dark +:width: 500px +:align: center +``` + +{{ extract_from_markdown_by_marker("description", readme_path) }} + +## Capabilities +{{ extract_from_markdown_by_marker("capabilities", readme_path) }} + + +## Installation +{{ extract_from_markdown_by_marker("prerequisites", readme_path) }} + +{{ extract_from_markdown_by_marker("installation", readme_path) }} + +For development, install the additional required packages via: +{{ extract_from_markdown_by_marker("installation_develop", contributing_path) }} + +> Note: We recommend using conda/mamba environments and installing performance-critical packages (e.g., numpy, scipy, ...) using `conda install .` The reason for this is the choice of BLAS library (linear algebra packages). Conda (depending on the channel) installs numpy and the [mkl](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-mkl-and-third-party-applications-how-to-use-them-together.html) library from Intel, in contrast to pip which defaults back to the linear algebra package installed on the system. According to certain benchmarks ([here](http://markus-beuckelmann.de/blog/boosting-numpy-blas.html) or [here](https://medium.com/analytics-vidhya/why-conda-install-instead-of-pip-install-ba4c6826a0ae)), the mkl library is able to outperform other linear algebra libraries, especially on Intel devices. Particularly for use cases where linear algebra operations dominate the computational costs, the benefit can be huge. + +## Citing QUEENS +You used QUEENS for a publication? Don't forget to cite +{{ extract_from_markdown_by_marker("citation", readme_path) }} +and the respective methods papers. + +## License +{{ extract_from_markdown_by_marker("license", readme_path) }} diff --git a/doc/source/_ext/templates/overview.rst.j2 b/doc/source/_ext/templates/overview.rst.j2 new file mode 100644 index 00000000..253cc88c --- /dev/null +++ b/doc/source/_ext/templates/overview.rst.j2 @@ -0,0 +1,16 @@ +Modules Overview +================ + +Here, we present a brief overview of the fundamental modules of QUEENS. + +{% for module in modules %} +{{ module["name"] }} +{{ "-"*len(module["name"]) }} + +.. toctree:: + :maxdepth: 1 + + {{ module["module"] }} + +{{ module["description"] }} +{% endfor %} diff --git a/doc/source/_ext/templates/tutorials.rst.j2 b/doc/source/_ext/templates/tutorials.rst.j2 new file mode 100644 index 00000000..11a9af55 --- /dev/null +++ b/doc/source/_ext/templates/tutorials.rst.j2 @@ -0,0 +1,19 @@ +.. _tutorials: + +Tutorials and examples +================================== + +Work in progress! + +But here a simple example from our `README.md `_: + +.. code-block:: python + +{{ example_text }} + +Resulting in the histogram: + +.. image:: images/monte_carlo_uq.png + :width: 500 + :align: center + :alt: Monte Carlo Histogram diff --git a/doc/source/architecture.rst b/doc/source/architecture.rst deleted file mode 100644 index 5222c1c1..00000000 --- a/doc/source/architecture.rst +++ /dev/null @@ -1,115 +0,0 @@ ---------------------------- -Architecture Overview ---------------------------- -Here, we present a brief overview of the fundamental architecture of QUEENS. - -Data processor ---------------------------- -.. toctree:: - :maxdepth: 4 - - queens.data_processor - - -Distributions ---------------------------- -.. toctree:: - :maxdepth: 4 - - queens.distributions - - -Drivers ---------------------------- -.. toctree:: - :maxdepth: 4 - - queens.drivers - - -Example simulator functions ---------------------------- -.. toctree:: - :maxdepth: 4 - - queens.example_simulator_functions - - -External geometry ---------------------------- -.. toctree:: - :maxdepth: 4 - - queens.external_geometry - - -Interfaces ---------------------------- -.. toctree:: - :maxdepth: 4 - - queens.interfaces - - -Iterators ---------------------------- -.. toctree:: - :maxdepth: 4 - - queens.iterators - - -Models ---------------------------- -.. toctree:: - :maxdepth: 4 - - queens.models - - -Parameters ---------------------------- -.. toctree:: - :maxdepth: 4 - - queens.parameters - - -Schedulers ---------------------------- -.. toctree:: - :maxdepth: 4 - - queens.schedulers - - -Stochastic optimizers ---------------------------- -.. toctree:: - :maxdepth: 4 - - queens.stochastic_optimizers - - -Utils ---------------------------- -.. toctree:: - :maxdepth: 4 - - queens.utils - - -Variational distributions ---------------------------- -.. toctree:: - :maxdepth: 4 - - queens.variational_distributions - - -Visualization ---------------------------- -.. toctree:: - :maxdepth: 4 - - queens.visualization diff --git a/doc/source/conf.py b/doc/source/conf.py index de532a85..5015c3ea 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # @@ -26,8 +26,8 @@ import sys from pathlib import Path -# sys.path.insert(0, str(Path('.')resolve)) -sys.path.insert(0, str(Path("../../").resolve)) +sys.path.insert(0, str(Path("../../").resolve())) +sys.path.append(str(Path("_ext").resolve())) # -- General configuration ------------------------------------------------ @@ -48,6 +48,8 @@ "sphinx.ext.inheritance_diagram", "sphinx.ext.napoleon", "nbsphinx", + "queens_sphinx_extension", + "myst_parser", ] # Custom command to make the returns in the docstring behave like the parameter/argument section. @@ -151,8 +153,8 @@ # html_theme_options = { "logo": { - "image_light": "https://raw.githubusercontent.com/queens-py/queens-design/main/logo/queens_logo_day.svg", - "image_dark": "https://raw.githubusercontent.com/queens-py/queens-design/main/logo/queens_logo_night.svg", + "image_light": "images/queens_logo_day.svg", + "image_dark": "images/queens_logo_night.svg", } } @@ -182,7 +184,7 @@ # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ["_static"] +# html_static_path = ["_static"] # Add any extra paths that contain custom files (such as robots.txt or # .htaccess) here, relative to this directory. These files are copied diff --git a/doc/source/index.rst b/doc/source/index.rst index 4a59291c..4a703ba7 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -1,19 +1,22 @@ -.. pqueens documentation master file, created by - sphinx-quickstart on Tue Apr 4 13:44:16 2017. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - Welcome to the QUEENS' documentation! -=================================== +===================================== + +**Some useful links**: + +* `Repository on GitHub `_ +* `Website `_ + .. toctree:: - :maxdepth: 2 + :maxdepth: 1 :caption: Contents: intro - architecture + overview tutorials - modules.rst + development + queens + Indices and tables @@ -21,4 +24,3 @@ Indices and tables * :ref:`genindex` * :ref:`modindex` -* :ref:`search` diff --git a/doc/source/intro.rst b/doc/source/intro.rst deleted file mode 100644 index bd950c1e..00000000 --- a/doc/source/intro.rst +++ /dev/null @@ -1,10 +0,0 @@ -Introduction -=================================== -QUEENS (Quantification of Uncertain Effects in Engineering Systems) is a Python framework for solver-independent multi-query analyses of large-scale computational models. - - -Installation ------------- -For a quick installation guide, QUEENS users should refer to our `README.md `_. - -QUEENS contributors should follow the installation guide and the instructions in our `CONTRIBUTING.md `_. diff --git a/doc/source/tutorials.rst b/doc/source/tutorials.rst deleted file mode 100644 index d9baef81..00000000 --- a/doc/source/tutorials.rst +++ /dev/null @@ -1,6 +0,0 @@ -.. _tutorials: - -Tutorials and examples -================================== - -Work in progress... diff --git a/test_utils/get_queens_example_from_readme.py b/test_utils/get_queens_example_from_readme.py index 504135fe..8b87a1c2 100644 --- a/test_utils/get_queens_example_from_readme.py +++ b/test_utils/get_queens_example_from_readme.py @@ -15,29 +15,43 @@ """Extract QUEENS example from the readme.""" +from pathlib import Path + from queens.utils.path_utils import relative_path_from_queens -EXAMPLE_MARKER = "" - -def get_queens_example_from_readme(output_dir): - """Extract the example from the readme. +def extract_from_markdown_file_by_marker(marker_name, md_file): + """Extract section from an markdown file. Args: - output_dir (str): Output directory for the QUEENS run. + marker_name (str): Name of the section to extract. + md_file (str, pathlib.Path): Path to the markdown file + + Returns: + str: section as string """ - readme_path = relative_path_from_queens("README.md") + marker = f"" # Split the example in the readme using the marker - text = readme_path.read_text().split(EXAMPLE_MARKER) + text = Path(md_file).read_text(encoding="utf-8").split(marker) # Only one example should appear if len(text) != 3: - raise ValueError("Could not extract the example from the readme!") + raise ValueError(f"Could not extract the section marked with '{marker}' from {md_file}!") + + return text[1] + + +def get_queens_example_from_readme(output_dir): + """Extract the example from the readme. + + Args: + output_dir (str): Output directory for the QUEENS run. + """ + readme_path = relative_path_from_queens("README.md") - # Extract the source example_source = ( - text[1] + extract_from_markdown_file_by_marker("example", readme_path) .replace("```python", "") .replace("```", "") .replace('output_dir="."', f'output_dir="{output_dir}"') From 686bee6074ef63c8914d041cb33c476d7dc8cd5b Mon Sep 17 00:00:00 2001 From: Gil Robalo Rei Date: Sat, 18 Jan 2025 05:11:08 +0100 Subject: [PATCH 5/6] docs: add provisional description for each subpackage --- queens/data_processor/__init__.py | 7 ++++--- queens/distributions/__init__.py | 7 +++++-- queens/drivers/__init__.py | 2 +- queens/example_simulator_functions/__init__.py | 4 ++-- queens/external_geometry/__init__.py | 7 ++++--- queens/global_settings.py | 8 ++++++-- queens/interfaces/__init__.py | 6 +----- queens/iterators/__init__.py | 12 +++++------- queens/main.py | 8 ++++++-- queens/models/__init__.py | 6 +++--- queens/parameters/__init__.py | 9 +++++++-- queens/parameters/parameters.py | 4 ++-- queens/schedulers/__init__.py | 2 +- queens/stochastic_optimizers/__init__.py | 7 +++++-- queens/utils/__init__.py | 7 +++++-- queens/variational_distributions/__init__.py | 7 +++++-- queens/visualization/__init__.py | 6 +++++- 17 files changed, 67 insertions(+), 42 deletions(-) diff --git a/queens/data_processor/__init__.py b/queens/data_processor/__init__.py index 1e5e9f70..592712d3 100644 --- a/queens/data_processor/__init__.py +++ b/queens/data_processor/__init__.py @@ -1,6 +1,6 @@ # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # @@ -12,9 +12,10 @@ # should have received a copy of the GNU Lesser General Public License along with QUEENS. If not, # see . # -"""Data processor. +"""Data Processor. -Extract data from simulation output. +The data processor modules are used to extract data from simulation +model outputs. """ from queens.data_processor.data_processor_csv import DataProcessorCsv diff --git a/queens/distributions/__init__.py b/queens/distributions/__init__.py index f9cbee30..069deb34 100644 --- a/queens/distributions/__init__.py +++ b/queens/distributions/__init__.py @@ -1,6 +1,6 @@ # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # @@ -12,7 +12,10 @@ # should have received a copy of the GNU Lesser General Public License along with QUEENS. If not, # see . # -"""Distributions.""" +"""Distributions. + +Probability distribution modules of QUEENS. +""" from queens.distributions.bernoulli import BernoulliDistribution from queens.distributions.beta import BetaDistribution diff --git a/queens/drivers/__init__.py b/queens/drivers/__init__.py index a99fe9b1..0d108375 100644 --- a/queens/drivers/__init__.py +++ b/queens/drivers/__init__.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # diff --git a/queens/example_simulator_functions/__init__.py b/queens/example_simulator_functions/__init__.py index a863499d..228e7405 100644 --- a/queens/example_simulator_functions/__init__.py +++ b/queens/example_simulator_functions/__init__.py @@ -1,6 +1,6 @@ # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # @@ -15,7 +15,7 @@ """Example simulator functions. This package contains a set of example simulator functions, which can be -used as benchmark. +used for benchmarking. """ from queens.example_simulator_functions.agawal09 import agawal09a diff --git a/queens/external_geometry/__init__.py b/queens/external_geometry/__init__.py index 6c5411e3..dc0b8d78 100644 --- a/queens/external_geometry/__init__.py +++ b/queens/external_geometry/__init__.py @@ -1,6 +1,6 @@ # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # @@ -12,9 +12,10 @@ # should have received a copy of the GNU Lesser General Public License along with QUEENS. If not, # see . # -"""External geometry module. +"""External geometry. -Read in external geometry to QUEENS. +Read in external geometry from simulation software to QUEENS for +preprocessing. """ from queens.external_geometry.fourc_dat_geometry import FourcDatExternalGeometry diff --git a/queens/global_settings.py b/queens/global_settings.py index 293bba8c..917d2ea9 100644 --- a/queens/global_settings.py +++ b/queens/global_settings.py @@ -1,6 +1,6 @@ # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # @@ -12,7 +12,11 @@ # should have received a copy of the GNU Lesser General Public License along with QUEENS. If not, # see . # -"""Global Settings module.""" +"""Global Settings. + +This module provides a context for QUEENS runs with exit functionality +for logging and working with remote resources. +""" import logging from pathlib import Path diff --git a/queens/interfaces/__init__.py b/queens/interfaces/__init__.py index a110820a..8d8aa033 100644 --- a/queens/interfaces/__init__.py +++ b/queens/interfaces/__init__.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # @@ -17,10 +17,6 @@ This package contains a set of so-called interfaces. The purpose of an interface is essentially the mapping of inputs to outputs. - -The mapping is made by passing the inputs further down to a -*regression_approximation* or a *mf_regression_approximation*, both of -which essentially then evaluate a regression model themselves. """ from queens.interfaces.bmfia_interface import BmfiaInterface diff --git a/queens/iterators/__init__.py b/queens/iterators/__init__.py index dfc999dd..e2d0aa83 100644 --- a/queens/iterators/__init__.py +++ b/queens/iterators/__init__.py @@ -1,6 +1,6 @@ # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # @@ -14,12 +14,10 @@ # """Iterators. -The iterator package contains the implementation of several UQ and -optimization methods, each of which is implemented in their own iterator -class. The iterator is therefor one of the central building blocks, as -the iterators orchestrate the evaluations on one or multiple models. -QUEENS also permits nesting of iterators to enable hierarchical methods -or surrogate based UQ approaches. +The iterator package contains methods for parameters studies, +uncertainty quantification, sensitivity analysis, Bayesian inverse +analysis, and optimization. These methods iterate on QUEENS models in a +multi-query fashion. """ from queens.iterators.black_box_variational_bayes import BBVIIterator diff --git a/queens/main.py b/queens/main.py index 139bc417..8a6ed8a8 100644 --- a/queens/main.py +++ b/queens/main.py @@ -1,6 +1,6 @@ # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # @@ -12,7 +12,11 @@ # should have received a copy of the GNU Lesser General Public License along with QUEENS. If not, # see . # -"""Main module of QUEENS containing the high-level control routine.""" +"""QUEENS main. + +Main module of QUEENS containing the high-level control routine for +input file workflow. +""" import logging import sys diff --git a/queens/models/__init__.py b/queens/models/__init__.py index f6c9b406..47978ce8 100644 --- a/queens/models/__init__.py +++ b/queens/models/__init__.py @@ -1,6 +1,6 @@ # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # @@ -15,8 +15,8 @@ """Models. The model package contains several types of models to be used in the -context of UQ. Within QUEENS, the model class of object holds and stores -the input and output data, and can evaluate itself to produce data. +context of multi-query analysis. Within QUEENS, the model returns its +evaluations and possibly gradients. """ from queens.models.bmfmc_model import BMFMCModel diff --git a/queens/parameters/__init__.py b/queens/parameters/__init__.py index 9eb6b3a6..4d1542c2 100644 --- a/queens/parameters/__init__.py +++ b/queens/parameters/__init__.py @@ -1,6 +1,6 @@ # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # @@ -12,6 +12,11 @@ # should have received a copy of the GNU Lesser General Public License along with QUEENS. If not, # see . # -"""Parameters.""" +"""Parameters. + +In the context of QUEENS, parameters are altered for the multi-query +analysis. They can either be deterministic, random variables or random +fields. +""" from queens.parameters.parameters import Parameters diff --git a/queens/parameters/parameters.py b/queens/parameters/parameters.py index d5809727..7f78dfa5 100644 --- a/queens/parameters/parameters.py +++ b/queens/parameters/parameters.py @@ -1,6 +1,6 @@ # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # @@ -12,7 +12,7 @@ # should have received a copy of the GNU Lesser General Public License along with QUEENS. If not, # see . # -"""Parameters module.""" +"""Parameters.""" import logging diff --git a/queens/schedulers/__init__.py b/queens/schedulers/__init__.py index c23f1ca8..e2b7521c 100644 --- a/queens/schedulers/__init__.py +++ b/queens/schedulers/__init__.py @@ -1,6 +1,6 @@ # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # diff --git a/queens/stochastic_optimizers/__init__.py b/queens/stochastic_optimizers/__init__.py index bed03e84..f537a79f 100644 --- a/queens/stochastic_optimizers/__init__.py +++ b/queens/stochastic_optimizers/__init__.py @@ -1,6 +1,6 @@ # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # @@ -12,7 +12,10 @@ # should have received a copy of the GNU Lesser General Public License along with QUEENS. If not, # see . # -"""Stochastic optimizers init.""" +"""Stochastic optimizers. + +This package consists of a set of stochastic optimizers. +""" from queens.stochastic_optimizers.adam import Adam from queens.stochastic_optimizers.adamax import Adamax diff --git a/queens/utils/__init__.py b/queens/utils/__init__.py index b0d0e4a2..5e0ea865 100644 --- a/queens/utils/__init__.py +++ b/queens/utils/__init__.py @@ -1,6 +1,6 @@ # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # @@ -12,4 +12,7 @@ # should have received a copy of the GNU Lesser General Public License along with QUEENS. If not, # see . # -"""Utils.""" +"""Utils. + +A large collections of utilities used in all of QUEENS. +""" diff --git a/queens/variational_distributions/__init__.py b/queens/variational_distributions/__init__.py index f33b172c..4cf603ed 100644 --- a/queens/variational_distributions/__init__.py +++ b/queens/variational_distributions/__init__.py @@ -1,6 +1,6 @@ # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # @@ -12,7 +12,10 @@ # should have received a copy of the GNU Lesser General Public License along with QUEENS. If not, # see . # -"""Variational distributions init.""" +"""Variational distributions. + +A probability distributions package for variational inference. +""" from queens.variational_distributions.full_rank_normal import FullRankNormalVariational from queens.variational_distributions.joint import JointVariational diff --git a/queens/visualization/__init__.py b/queens/visualization/__init__.py index 6ec608e5..4312c98a 100644 --- a/queens/visualization/__init__.py +++ b/queens/visualization/__init__.py @@ -1,6 +1,6 @@ # # SPDX-License-Identifier: LGPL-3.0-or-later -# Copyright (c) 2024, QUEENS contributors. +# Copyright (c) 2025, QUEENS contributors. # # This file is part of QUEENS. # @@ -12,3 +12,7 @@ # should have received a copy of the GNU Lesser General Public License along with QUEENS. If not, # see . # +"""Visualization. + +A set of modules to visualize results of different iterators. +""" From 943b829b1c231b0e1d93ecf44a3c4f35fcbd9660 Mon Sep 17 00:00:00 2001 From: Gil Robalo Rei Date: Thu, 23 Jan 2025 17:36:32 +0100 Subject: [PATCH 6/6] build: update dev-requirments.txt --- dev-requirements.txt | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/dev-requirements.txt b/dev-requirements.txt index 17e122b7..0f569ee6 100644 --- a/dev-requirements.txt +++ b/dev-requirements.txt @@ -1,5 +1,5 @@ # -# This file is autogenerated by pip-compile with Python 3.11 +# This file is autogenerated by pip-compile with Python 3.10 # by the following command: # # pip-compile --output-file=dev-requirements.txt dev-requirements.in @@ -80,6 +80,10 @@ idna==3.10 # requests imagesize==1.4.1 # via sphinx +importlib-metadata==6.11.0 + # via + # -c requirements.txt + # build isort==5.13.2 # via # -r dev-requirements.in @@ -267,6 +271,14 @@ tinycss2==1.4.0 # via nbconvert toml==0.10.2 # via liccheck +tomli==2.0.2 + # via + # -c requirements.txt + # build + # pip-tools + # pre-commit-hooks + # pylint + # sphinx tomlkit==0.13.2 # via # commitizen @@ -286,6 +298,7 @@ traitlets==5.14.3 typing-extensions==4.12.2 # via # -c requirements.txt + # astroid # pydata-sphinx-theme untokenize==0.1.1 # via docformatter @@ -307,6 +320,10 @@ wheel==0.44.0 # pip-tools yamllint==1.35.1 # via -r dev-requirements.in +zipp==3.20.2 + # via + # -c requirements.txt + # importlib-metadata # The following packages are considered to be unsafe in a requirements file: # pip