Pdstools V4 (#260)

* Make regex strings raw strings * Remove True from string cache call * Utilise non-strict dataframe init to make tests more meaningful * Use non-strict in dataframe init also * Names in group_by iteration now returns a tuple * Update named gropuby keys, non-strict inits * partition_by now also returns tuple as keys * Add basic sanity test for model_summary (it failed) * suffix is now in `name` namespace * Remove assert * Add pandoc version check, so we get a warning when not installed * Add testbook as explicit test optional dependency * Set min polars version to 1 * bump version * Remove True from string cache call * Utilise non-strict dataframe init to make tests more meaningful * Use non-strict in dataframe init also * Names in group_by iteration now returns a tuple * Update named gropuby keys, non-strict inits * partition_by now also returns tuple as keys * Add basic sanity test for model_summary (it failed) * suffix is now in `name` namespace * Remove assert * Add pandoc version check, so we get a warning when not installed * Add testbook as explicit test optional dependency * Set min polars version to 1 * expr.cut update * add version info to logs and fix quarto process * Revert "add version info to logs and fix quarto process" This reverts commit 2c153db. * wip commit to merge with polars 1.0 * Initial commit for V4 changes * Add IH namespace * lots of things. * cleanup * add pydantic as api requirement * Add cached properties for consistent ordering * small fry * Remove redundant except path * Improve file caching * Add back some init modules * Bump minimum python version to 3.9 * Drop 3.8 tests * Back to _typing * Updated ADMExplained * Updated ADMExplained to V4 * Removed stale/outdated notebooks * Slowly expose some more modules in __init__ * Dropped R version of PDS tools * Dropped R version of PDS tools * Health Check works till tree_map * uniques should be sets * Starting to improve coverage * Rename * removed legacy ih * made `show_versions` really cool. * typing & empty requirements * Casting to proper types * Use literal for file column name * Applying schema types again * Fixed dataset tests * Fixed end-to-end tests * Expose show_versions again * Make linter happy * Ignore API functions - might mock later * Rename missing dependency exception * ignore unnecessary lines * test, test, test, test, test, test * Fixing issue with inconsitent coloring and ordering in performance volume distribution plot * Make regex strings raw strings * Revert "add version info to logs and fix quarto process" This reverts commit 2c153db. * lots of things. * Fixed some rebasing woes * Minor cleanup * Cleaned up / overloaded show_versions * PathLike didn't work well * Updated value finder to V4 * Re-expose anonymization * patch * make HC work with CDHSample * Fixed missing import * covcovcovcovcov * Model reports aligned with v4. Need more work still. * Fixed some polars warnings * Using new Philip Mann plot from plots * apply schema to predictorsnapshot * generate_hc func work * Supporting cmd-line argument again * fıx generate model report * add generate_excel and fix mypy warnings * add tests for plots * Attempt to fix predictions. * fix health_check unit tests * rename excel function to excel_report * Trying to get prediction class going again * Fixed circular imports * Predictions work again with new CDH guidelines * Added ADM channel summary tests * Fixed some aggregates, added back overall ADM summary * Aligned binaggregator and improved coverage * Improved efficiency of JSON context key extraction * Fixed some AGB tests, adjusted to new syntax * Fitted to new naming * reorder imports * Bye r * will fix later * ignore cache dirs * ignore cache dirs * I <3 uv * Updated GH actions a bit HealthCheck now also works with UV, tests don't run for draft PRs * main -> master branch * HC up again, needs more refactoring still * fix fstring issue * hide plots * import relatively * HC up again, needs more refactoring still * Fixed tests * updated dependencies * Ignoring a lot of AGB tests for now. To be revisited later. * so much coverage. * remove redundant model_id check * add annotations * add pytest-httpx to test deps * Made test data smaller so faster * Now also uses shared report utils * fix app - Add query argument to hc and model reports - make save_data function of Datamart return abs paths - Update health_check app code, compatible with v_4 - Update read_ds_export docstring - make _apply_query more robust - commented out freezing AGBTree tests. * Added issue/group overview to HC * fix propensity distribution plot * Prediction Studio DX API Library Co-authored-by: patis9 <[email protected]> Co-authored-by: kass1 <[email protected]> Co-authored-by: Stijn Kas <[email protected]> * predictor_categorization applies in-place * remove R reference * removed trailing comma * Bounded the character regex to avoid infinite match * fix * re-assign now * re-expose prediction_studio * fall back to raw response if json parsing fails (todo; make consistent) * give get_metric default start_date * add pytest-mock to test dependencies * fix tqdm woes * moved pred studio tests to top-level tests folder * remove some warnings * Resolve Rename PerformanceBin to PredictorPerformance #251 * Remove sys.path.appends in tests (we now have `uv pip install -e` technology) * Fixing ADM summaries for value analysis * small fry * fix treatment missing error * Added Ruff and others as pre-commit hook, fixed resulting exceptions * Added verbose mode. * fix no context_key bug * More optional deps * Allow timeout customization * Docstring for ADMDatamart * Bump version to alpha 1 * limit x axis of heatmap in HC. * deleted 'new' file * Rebase with master * Temp remove 3.13 for CI because of onnx * wrong copy-paste * Fix tests ? * add openpyxl to test deps * commented out flakey test * d --------- Co-authored-by: Uyanik, Yusuf <[email protected]> Co-authored-by: Otto Perdeck <[email protected]> Co-authored-by: yusufuyanik1 <[email protected]> Co-authored-by: patil-sharanu <[email protected]> Co-authored-by: patis9 <[email protected]>
pegasystems · Oct 30, 2024 · 5a42b5c · 5a42b5c
1 parent a76a11e
commit 5a42b5c
Show file tree

Hide file tree

Showing 431 changed files with 14,900 additions and 166,245 deletions.
diff --git a/.github/ISSUE_TEMPLATE/bug_report_python.yml b/.github/ISSUE_TEMPLATE/bug_report_python.yml
@@ -64,15 +64,15 @@ body:
     attributes:
       label: Installed versions
       description: >
-        Please paste the output of ``pdstools.show_versions()``, 
+        Please paste the output of ``pdstools.show_versions()``,
         see [here](https://pegasystems.github.io/pega-datascientist-tools/Python/autoapi/pdstools/utils/show_versions/index.html#pdstools.utils.show_versions.show_versions) for an example.
       value: |
         <details>
 
         ```
         Replace this line with the output of pdstools.show_versions(), leave the backticks in place
         ```
-        
+
         </details>
     validations:
       required: true
diff --git a/.github/ISSUE_TEMPLATE/bug_report_r.yml b/.github/ISSUE_TEMPLATE/bug_report_r.yml
diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml
@@ -35,4 +35,4 @@ body:
     attributes:
       label: Additional context
       placeholder: >
-        Add any other context or screenshots about the feature request here.
+        Add any other context or screenshots about the feature request here.
diff --git a/.github/workflows/Docs check.yml b/.github/workflows/Docs check.yml
@@ -1,7 +1,7 @@
 name: Check Python docs
 
 on:
-  pull_request: 
+  pull_request:
     paths:
       - python/**
 
@@ -34,4 +34,4 @@ jobs:
 
       - name: Build Python documentation
         working-directory: python/docs
-        run: make html
+        run: make html
diff --git a/.github/workflows/Docs deploy.yml b/.github/workflows/Docs deploy.yml
@@ -44,7 +44,6 @@ jobs:
         run: |
           git clone https://github.com/${{ github.repository }}.git --branch gh-pages --single-branch gh-pages
           cp -r python/docs/build/html/* gh-pages/Python/
-          cp -r r/docs/* gh-pages/R/
           cd gh-pages
           git config --local user.email "[email protected]"
           git config --local user.name "GitHub Action"

diff --git a/.github/workflows/Healthcheck tests.yml b/.github/workflows/Healthcheck tests.yml
@@ -8,43 +8,46 @@ on:
       - python/**
       - reports/*.qmd
       - .github/workflows/**
-  pull_request: 
+  pull_request:
+    types: [opened, synchronize, reopened, ready_for_review]
     paths:
       - python/**
       - reports/*.qmd
+      - .github/workflows/**
+
 
 jobs:
   HealthCheck-Test:
     runs-on: ubuntu-latest
     name: Test Health Check
+    if: github.event_name == 'push' || (github.event_name == 'pull_request' && github.event.pull_request.draft == false)
+
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Set up Quarto
         uses: quarto-dev/quarto-actions/setup@v2
         with:
           version: 1.3.450
-
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v3
+
       - name: Set up Python
-        uses: actions/setup-python@v4
+        run: uv python install 3.12
+
+      - name: Enable caching
+        uses: astral-sh/setup-uv@v3
         with:
-          python-version: '3.11'
-          cache: pip
-
-      - name: Install dependencies
-        working-directory: python
-        run: |
-          pip install --upgrade pip
-          pip install -r requirements.txt
-          pip install -r tests/test-requirements.txt
-          pip install papermill
-          pip install itables
-          pip install jinja2
-          ipython kernel install --name "python3" --user
-          sudo apt-get install -y pandoc
-          
-      - name: Install pdstools
-        run: pip install .
+          enable-cache: true
+          cache-dependency-glob: |
+            **/pyproject.toml
+
+      - name: Install the project
+        run: uv sync --extra healthcheck --extra tests
+
+      - name: Set up Pandoc
+        run: sudo apt-get install -y pandoc
 
       - name: Run Health Check
-        run: pytest python/tests/test_healthcheck.py
+        run: uv run pytest python/tests/test_healthcheck.py
diff --git a/.github/workflows/Python release.yml b/.github/workflows/Python release.yml
@@ -1,6 +1,6 @@
 name: Publish Python 🐍 distributions 📦 to PyPI
 
-on: 
+on:
   push:
     tags:
       - '*'
@@ -9,7 +9,7 @@ jobs:
   build-n-publish:
     name: Build and publish Python 🐍 distributions 📦 to PyPI
     runs-on: ubuntu-latest
-    
+
     steps:
     - uses: actions/checkout@v3
     - name: Set up Python

diff --git a/.github/workflows/Python tests.yml b/.github/workflows/Python tests.yml
@@ -7,17 +7,19 @@ on:
     paths:
       - python/**
       - .github/workflows/**
-  pull_request: 
+  pull_request:
+    types: [opened, synchronize, reopened, ready_for_review]
     paths:
       - python/**
+      - reports/*.qmd
       - .github/workflows/**
 
 jobs:
   Tests:
     runs-on: ${{matrix.os}}
+    if: github.event_name == 'push' || (github.event_name == 'pull_request' && github.event.pull_request.draft == false)
 
-    name: pdstools (Python ${{ matrix.python-version }}) on ${{ matrix.os }} 
-
+    name: pdstools (Python ${{ matrix.python-version }}) on ${{ matrix.os }}
 
     strategy:
       fail-fast: true
@@ -32,25 +34,36 @@ jobs:
           - os: windows-latest
             python-version: "3.11"
           # - os: windows-latest
-          #   python-version: 3.12.x
+          #   python-version: "3.12"
           - os: macOS-latest
             python-version: "3.9"
           - os: macOS-latest
             python-version: "3.10"
           - os: macOS-latest
             python-version: "3.11"
           # - os: macOS-latest
-          #   python-version: 3.12.x
+            # python-version: "3.12"
     env:
       GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
 
+
     steps:
+
       - uses: actions/checkout@v4
 
+      - name: Check for draft PR
+        if: github.event_name == 'pull_request'
+        run: |
+          if [[ "${{ github.event.pull_request.draft }}" == "true" ]]; then
+            echo "This is a draft PR, exiting."
+            exit 1
+          fi
+        continue-on-error: true
+
       - name: Install uv
         uses: astral-sh/setup-uv@v3
-        
-      
+
+
       - name: Set up Python ${{ matrix.python-version }}
         run: uv python install ${{ matrix.python-version }}
 
@@ -59,15 +72,14 @@ jobs:
         with:
           enable-cache: true
           cache-dependency-glob: |
-            **/requirements*.txt
             **/pyproject.toml
-    
+
       - name: Install the project
-        run: uv sync --extra tests 
-    
+        run: uv sync --extra tests
+
       - name: Run tests
         run: uv run pytest --cov=./python/pdstools --cov-report=xml --cov-config=./python/tests/.coveragerc --ignore=python/tests/test_healthcheck.py --ignore=python/tests/test_ADMTrees.py
-        
+
       - name: Upload coverage reports to Codecov
         uses: codecov/[email protected]
         with:

diff --git a/.github/workflows/R docs.yml b/.github/workflows/R docs.yml
diff --git a/.github/workflows/R tests.yml b/.github/workflows/R tests.yml
diff --git a/.github/workflows/mend-scan.yml b/.github/workflows/mend-scan.yml
@@ -1,7 +1,7 @@
 # Reference: https://github.com/mend-toolkit/mend-examples/blob/main/Unified%20Agent/CI-CD/GitHub.yml
 
 name: Mend Unified Agent Release Scan
-on: 
+on:
   workflow_dispatch:
   push:
    branches:

diff --git a/.gitignore b/.gitignore
@@ -33,3 +33,4 @@ python/*.ipynb_checkpoints/*
 **/*-checkpoint.ipynb
 **/META-INF/*
 r/tests/testthat/d/tmp2
+**/cache