#506: Add capability to create and export a dataset containing shared…

… blocks (#519)

* #506: upgrade vscode editor settings

* #506: fix pylintrc file waring codes mispelling

* #506: create skeleton for new utility to generate JSON output from cli

* #506: add user prompt for tasks to add

* #506: start implementing new Phase populate_from_specification method

* #506: start implementing new specification for a Dataset and some refactoring

* #506: rework Phase specification dictionaries

* #506: implement populate_from_specification missing code

* #506: improve JSONDatasetMaker class

* #506: add vscode task to debug or run the dataset maker

* #506: create communicators and add communications to the popultaed phase

* #506: fix double quotes

* #506: change output directory for data and config file generated by the JSONDatasetMaker

* #506: refactor - improve code quality - add dump and load from file functions for specifications

* #506: code formatting

* #506: update doc strings

* #506: fix ci warnings

* #506: improve util script arguments handling and display and add more comments

* #506: fix ci warnings

* #506: remove trailing space

* #506: fix messages

* #506: improve PhaseSpecification file format with intriduction of  PhaseSpecificationNormalizer class

* #506: fix trailing whitespace

* #506: some refactoring

* #506: wip interactive inputs

* #506: adding the CCM example data without blocks for visualization purposes

* #506: the configuration file to run the CCM example illustration problem for this case

* #506: feat to add objects unitary in interactive mode

* #506: remove undesired communications set from the rank specification

* #506: fix bad output stem in ccm example config file

* #506: code formatting and update sample config build

* #506: fix communications in phase

* #506: rename ccm_example_data directory to ccm_example

* #506: reorganize interactive commands in script

* #506: remove trailing whitespace

* #506: fix Object class types and get_shared_block_id error if no shared block

* #506: refactor and add comments

* #506: rename console script for consistency

* #506: update bad quotes

* #506: fixed entity ID

* #506: fix error if task_footprint_bytes or task_working_bytes not defined in user defined data

* #506: improve specification validation add add optional multiple_sharing support

* #506: modified CCM example to simple "phase step" (no LB) and call vt-tv

* #506: update user defined home dict key for shared block

* #506: update checks of specification in populate_from_specification logic and add warning for orphan shared blocks

* #506: add unit tests for data files maker

* #506: remove trailing whitespace

* #506: fix specific problem in case of a shared block for tasks on different ranks

* #506: add support of home specification for shared blocks user defined data

* #506: remove completely links between ranks and shared blocks in populate_from_specification

* #506: rename test class

* #506: update typing because CI error

* #506: remove unused import

* #506: try fix typing for ci

* #506: remove typing causing CI error

* #506: fix obsolete documentation

* #506: fix wrong import position ausing error if lbaf package not installed

* #506: update docs related to arguments descriptions and to the maker script

* #506: checking in the generated datasets that correspond to CCM with blocks as well

* #506: update doc comments and formatting

* #506: improve comments

* #506: fix trailing space

* #506: update error log message and comment

* #506: remove sample run configuration generation from script

* #506: fix 2 problems in interactive mode

* #506: update home to home_rank in phase specification

* #506: fix unit tests

* #506: add additional interactive input validation

* #506: refactor and fix errors and improve rank input in interactve mode

* #506: catch any error in interactive mode loop

* #506: fix trailing comma generating unexpected tuple

* #506: fix remaining extra comma generating tuple

* #506: fix conversion error between int and string in interactive mode for tasks lists inputs

* #506: fix typo adn update error message

* #506: fix extra util input issue in interactive mode

* #506: input fixes

* #506: add new unit test with specified element keys

* #506: WIP remove element and refactoring

* #506: enable remove rank or task in interactive mode

* #506: fix syntax

* #506: fixes

* #506: promote local variable spec to class variable in maker and implement 2 remaining remove* functions

* #506: fix bad method call on error

* #506: fix trailing whitespace

* #506: fix another trailing whitespace in unit tests

* #506: fix ccm example json files wrong key

* #506: fix old variable

* #506: fix another variable name

* #506: fix typo

* #506: fix typo

---------

Co-authored-by: Philippe P. Pébaÿ <[email protected]>

.github/workflows/action-git-diff-check.yml

@@ -7,7 +7,7 @@ jobs:
     name: Run git check
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
           ref: ${{ github.event.pull_request.head.sha }}

.github/workflows/build-and-publish-to-pypi.yml

@@ -7,7 +7,7 @@ jobs:
     steps:
       - uses: actions/checkout@v4
       - name: Set up Python 3.8
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: 3.8
       - name: Install build

.github/workflows/check-commit-format.yml

@@ -7,7 +7,7 @@ jobs:
     name: Check commit message format
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
           ref: ${{ github.event.pull_request.head.sha }}

.github/workflows/code-quality.yml

@@ -18,15 +18,15 @@ jobs:
       deploy_badges_src_branch: develop
       deploy_badges_dst_branch: deploy-badges
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Install packages
         run: |
           sudo apt-get update -y
           sudo apt-get install -y git xvfb
 
       - name: Python Setup
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
 
@@ -50,7 +50,7 @@ jobs:
           echo "coverage_report_exists=$COVERAGE_REPORT_EXISTS" >> $GITHUB_OUTPUT
 
       - name: Report Test results
-        uses: phoenix-actions/test-reporting@v12
+        uses: phoenix-actions/test-reporting@v15
         if: success() || failure()
         with:
           name: Tests report (${{ matrix.os }}, ${{ matrix.python-version }})

.github/workflows/deploy_docs.yml

@@ -15,9 +15,9 @@ jobs:
       python-version: '3.8'
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - name: Set up Python ${{ env.python-version }}
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ env.python-version }}
           architecture: 'x64'

.github/workflows/find-trailing-whitespace.yml

@@ -7,7 +7,7 @@ jobs:
     name: Find Trailing Whitespace
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: DARMA-tasking/find-trailing-whitespace@master
         with:
           exclude: "doc" # ; separated path to exclude

.pylintrc

@@ -90,8 +90,8 @@ disable=raw-checker-failed,
         deprecated-pragma,
         use-symbolic-message-instead,
         logging-fstring-interpolation,
-        missing-module-doctring,
-        missing-class-doctring,
+        missing-module-docstring,
+        missing-class-docstring,
         protected-access
 
 # Enable the message, report, category or checker with the given id(s). You can

.vscode/launch.json

@@ -42,6 +42,16 @@
             "console": "integratedTerminal",
             "justMyCode": true,
             "args": [ ]
-        }
+        },
+        {
+            "name": "LBAF: Data Files Maker",
+            "python": "${workspaceFolder}/venv311/bin/python",
+            "type": "python",
+            "request": "launch",
+            "program": "${workspaceFolder}/src/lbaf/Utils/lbsJSONDataFilesMaker.py",
+            "console": "integratedTerminal",
+            "justMyCode": true,
+            "args": ["--interactive"]
+        },
     ]
 }

.vscode/settings.json

@@ -1,31 +1,31 @@
 {
-    "python.linting.enabled": true,
-    "python.linting.pylintEnabled": true,
-    "python.linting.pylintPath": "${workspaceFolder}/venv/bin/pylint",
-    "python.linting.ignorePatterns": [
+    // Linting (with PyLint) configuration for VSCode
+    "pylint.args": [
+        "--rcfile=${workspaceFolder}/.pylintrc"
+    ],
+    "pylint.ignorePatterns": [
         "**/site-packages/**/*.py",
         ".vscode/*.py"
     ],
-    "pylint.args": [
-        "--rcfile=${workspaceFolder}/.pylintrc"
+    "pylint.path": [
+        "${workspaceFolder}/venv/bin/pylint"
     ],
     "pylint.importStrategy": "fromEnvironment",
     "python.testing.unittestArgs": [
-        "-v",
+        "discover",
         "-s",
         "./tests",
         "-p",
-        "test_*.py"
+        "test_*.py",
+        "-t",
+        "./",
+        "-v",
     ],
-
-    "python.testing.unittestEnabled": true,
-    "python.testing.pytestEnabled": false,
+    // Unit Tests (with unittest) configuration for VSCode
+    "python.testing.pytestEnabled": false, // disable pytest lib
+    "python.testing.unittestEnabled": true, // enable unitest lib
     "[python]": {
         "editor.defaultFormatter": "ms-python.autopep8"
     },
-    "python.formatting.provider": "none",
-    "python.formatting.autopep8Args": [
-        "--max-line-length",
-        "120"
-    ],
+    "autopep8.args": ["--max-line-length","120"]
 }

config/ccm-example.yaml

@@ -0,0 +1,40 @@
+# Specify input
+from_data:
+  data_stem: ../data/ccm_example/data
+  phase_ids:
+  - 0
+check_schema: false
+
+# Specify work model
+work_model:
+  name: AffineCombination
+  parameters:
+    alpha: 1.0
+    beta: 0.0
+    gamma: 0.0
+    upper_bounds:
+      max_memory_usage: 8000000000.0
+
+# Specify algorithm
+algorithm:
+  name: PhaseStepper
+
+# Specify output
+output_dir: ../output
+output_file_stem: output_file
+write_JSON:
+  compressed: true
+  suffix: json
+  communications: true
+  offline_LB_compatible: true
+visualization:
+  x_ranks: 2
+  y_ranks: 1
+  z_ranks: 1
+  object_jitter: 0.5
+  rank_qoi: load
+  object_qoi: shared_block_id
+  save_meshes: true
+  force_continuous_object_qoi: true
+  output_visualization_dir: ../output
+  output_visualization_file_stem: output_file

data/ccm_example/data.0.json

@@ -0,0 +1 @@
+{"metadata":{"type":"LBDatafile","rank":0},"phases":[{"id":0,"tasks":[{"entity":{"home":0,"id":0,"migratable":true,"type":"object"},"node":0,"resource":"cpu","time":2.0,"user_defined":{"shared_id":0,"shared_bytes":10000.0,"home_rank":0}},{"entity":{"home":0,"id":1,"migratable":true,"type":"object"},"node":0,"resource":"cpu","time":3.5,"user_defined":{"shared_id":0,"shared_bytes":10000.0,"home_rank":0}}],"communications":[{"type":"SendRecv","to":{"type":"object","id":2},"messages":1,"from":{"type":"object","id":0},"bytes":10000.0},{"type":"SendRecv","to":{"type":"object","id":2},"messages":1,"from":{"type":"object","id":1},"bytes":15000.0},{"type":"SendRecv","to":{"type":"object","id":1},"messages":1,"from":{"type":"object","id":0},"bytes":25000.0}]}]}

data/ccm_example/data.1.json

@@ -0,0 +1 @@
+{"metadata":{"type":"LBDatafile","rank":1},"phases":[{"id":0,"tasks":[{"entity":{"home":1,"id":2,"migratable":true,"type":"object"},"node":1,"resource":"cpu","time":5.0,"user_defined":{"shared_id":1,"shared_bytes":15000.0,"home_rank":1}}],"communications":[{"type":"SendRecv","to":{"type":"object","id":1},"messages":1,"from":{"type":"object","id":2},"bytes":20000.0}]}]}

setup.cfg

@@ -48,4 +48,5 @@ console_scripts =
     lbaf-vt-data-extractor = lbaf:vt_data_extractor
     lbaf-vt-data-files-validator-loader = lbaf:vt_data_files_validator_loader
     lbaf-vt-data-files-validator = lbaf:vt_data_files_validator
+    lbaf-vt-data-files-maker = lbaf:vt_data_files_maker
     lbaf-vt-data-stat-files-updater = lbaf:vt_data_stat_files_updater

src/lbaf/Applications/LBAF_app.py

@@ -31,7 +31,7 @@
 # pylint:enable=C0413:wrong-import-position
 
 class InternalParameters:
-    """Represent the parameters used internally by a a LBAF Application"""
+    """Represent the parameters used internally by a LBAF Application"""
 
     __logger: Logger
 

src/lbaf/Execution/lbsPhaseSpecification.py

@@ -0,0 +1,128 @@
+from typing import List, Dict, TypedDict, Union, Set, Callable, Optional, cast
+
+class SharedBlockSpecification(TypedDict):
+    # The shared block size
+    size: float
+    # The ID of the unique rank to which a shared block ultimately belong
+    home: int
+    # The set of tasks accessing this shared block
+    tasks: Set[int]
+
+CommunicationSpecification = TypedDict('CommunicationSpecification', {
+    'size': float,
+    'from': int,
+    'to': int
+})
+
+class RankSpecification(TypedDict):
+    # The task ids
+    tasks: Set[int]
+
+class PhaseSpecification(TypedDict):
+    """Dictionary representing a phase specification"""
+
+    # Tasks specifications
+    tasks: Union[
+        List[float], # Tasks volumes as a list (element index=task id)
+        Dict[int,float] # Tasks volumes as a dictionary (dictionary key=task id)
+    ]
+
+    # Shared blocks specifications
+    shared_blocks: Union[
+        List[SharedBlockSpecification], # where index = shared block id
+        Dict[int,SharedBlockSpecification] # where dictionary key = shared block id
+    ]
+
+    # Communications specifications
+    communications: Union[
+        List[CommunicationSpecification],  # where index = communication id
+        Dict[int,CommunicationSpecification] # where dictionary key = communication id
+    ]
+
+    # Rank distributions
+    ranks: Dict[int,RankSpecification] # where index = rank id
+
+class PhaseSpecificationNormalizer:
+    """Provides normalization and denormalization for PhaseSpecification
+    where inner sets are represented as lists to improve readability in JSON or YAML
+    """
+
+    def __normalize_member(self, data: Union[dict,list], transform: Optional[Callable] = None) -> Union[dict,list]:
+        """Normalize a member that can be represented as a dict where key is the item key or as a list
+        where id is the index in the list
+        """
+
+        if isinstance(data, list):
+            return [ transform(o) for o in data ] if transform is not None else data
+        elif isinstance(data, dict):
+            return { o_id:transform(o) for o_id, o in data.items() } if transform is not None else data # pylint: disable=E1101 (no-member)
+        else:
+            raise RuntimeError("data must be list or dict")
+
+    def normalize(self, spec: PhaseSpecification)-> dict:
+        """Normalize a phase specification to represent inner sets as lists
+
+        Note: the sets converted to lists are
+        - `self.shared_blocks.tasks`
+        - `self.ranks.tasks`
+        - `self.ranks.communications`
+
+        This method should be called before json or yaml serialization.
+        Denormalization should be executed using the method denormalize
+
+        Note: the normalized specification data is easier to read and edit in json or yaml.
+        """
+
+        return {
+            "tasks": self.__normalize_member(spec.get("tasks", {})),
+            "shared_blocks": self.__normalize_member(
+                spec.get("shared_blocks", []),
+                lambda b: {
+                    "size": b.get("size"),
+                    "tasks": list(b.get("tasks", {})),
+                    "home_rank": b.get("home_rank"),
+                }
+            ),
+            "communications": self.__normalize_member(spec.get("communications", [])),
+            "ranks": self.__normalize_member(
+                spec.get("ranks", {}),
+                lambda r: {
+                    "tasks": list(r.get("tasks", {}))
+                }
+            )
+        }
+
+    def denormalize(self, data: dict)-> PhaseSpecification:
+        """Create a phase specification from a normalized specification where some lists must
+        be converted to sets.
+
+        Detail: the following lists will be converted to sets to ensure each element is unique
+        - `data.shared_blocks.tasks`
+        - `data.ranks.tasks`
+        - `data.ranks.communications`
+
+        This method should be called after json or yaml deserialization.
+        This is the reverse implementation of the normalize method.
+        """
+
+        return PhaseSpecification({
+            "tasks": self.__normalize_member(data.get("tasks", [])),
+            "shared_blocks": self.__normalize_member(
+                data.get("shared_blocks", []),
+                lambda b: SharedBlockSpecification({
+                    "size": b.get("size", 0.0),
+                    "tasks": set(b.get("tasks", {})),
+                    "home_rank": b.get("home_rank")
+                })
+            ),
+            "communications": self.__normalize_member(
+                data.get("communications", []),
+                lambda c: cast(CommunicationSpecification, c)
+            ),
+            "ranks": self.__normalize_member(
+                data.get("ranks", []),
+                lambda r: RankSpecification({
+                    "tasks": set(r.get("tasks", []))
+                })
+            )
+        })

src/lbaf/IO/lbsVTDataWriter.py

@@ -3,6 +3,7 @@
 import os
 import sys
 from logging import Logger
+from typing import Optional
 
 import brotli
 
@@ -19,7 +20,7 @@ class VTDataWriter:
     def __init__(
         self,
         logger: Logger,
-        output_dir: str,
+        output_dir: Optional[str],
         stem: str,
         parameters: dict):
         """Class constructor