Data structure for filters #2168

ychiucco · 2025-01-07T15:04:38Z

closes #2170
closes #2155
closes #1401
closes #2153

Checklist before merging

I added an appropriate entry to CHANGELOG.md
I added logging to new code - if appropriate.
I merged main into the current branch.

github-actions · 2025-01-07T15:09:00Z

Benchmark comparison

GET /api/alive/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`
vanilla	0.9	1.0	0.90
power	0.8	0.9	0.89
dataset	0.8	1.0	0.80
project	0.8	1.0	0.80
job	0.8	0.9	0.89

GET /api/v2/dataset/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	10.3	9.8	1.05	0.6	0.6
power	401.1	434.1	0.92	345.4	366.2
dataset	140.1	162.4	0.86	123.2	134.0
project	41.0	47.5	0.86	43.9	47.2
job	28.2	31.0	0.91	29.7	32.4

GET /api/v2/job/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	6.4	6.2	1.03	2.2	2.2
power	141.2	143.2	0.99	662.6	652.2
dataset	144.7	152.9	0.95	916.0	905.2
project	48.3	49.6	0.97	287.2	283.8
job	43.8	41.0	1.07	227.6	224.9

GET /api/v2/project/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	8.0	8.1	0.99	0.1	0.1
power	7.7	6.9	1.12	0.1	0.1
dataset	7.8	7.1	1.10	0.1	0.1
project	10.0	9.6	1.04	2.3	2.3
job	8.7	7.8	1.12	0.1	0.1

GET /api/v2/task/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	6.7	5.9	1.14	0.8	0.8
power	6.5	6.0	1.08	0.8	0.8
dataset	6.4	6.1	1.05	0.8	0.8
project	6.2	6.5	0.95	0.8	0.8
job	6.1	6.1	1.00	0.8	0.8

GET /api/v2/task-group/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	7.7	7.9	0.97	1.6	1.6
power	7.4	6.8	1.09	1.6	1.6
dataset	7.5	7.0	1.07	1.6	1.6
project	7.8	7.5	1.04	1.6	1.6
job	7.7	7.8	0.99	1.6	1.6

GET /api/v2/workflow/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	12.0	15.5	0.77	0.8	0.8
power	25.6	26.6	0.96	28.0	29.1
dataset	19.8	20.6	0.96	16.2	16.7
project	23.6	22.6	1.04	20.4	21.0
job	11.9	11.1	1.07	0.8	0.8

POST /api/v2/project/3/dataset/406/images/query/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
dataset	161.8	163.0	0.99	275.1	275.1

GET /auth/current-user/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	4.8	4.9	0.98	0.2	0.2
power	5.2	4.7	1.11	0.2	0.2
dataset	5.1	4.7	1.09	0.2	0.2
project	4.5	4.7	0.96	0.2	0.2
job	4.6	4.7	0.98	0.1	0.1

POST /auth/token/login/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
dataset	274.1	274.2	1.00	0.2	0.2

github-actions · 2025-01-07T15:20:03Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)
fractal_server/app/models/v2
dataset.py	$This PR adds 29 statements to fractal_server/app/models/v2/dataset.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/app/models/v2/dataset.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/models/v2/dataset.py is 100% (29/29). The file did not seem to exist on the base branch.$	$In this PR, 4 new statements are added to fractal_server/app/models/v2/dataset.py, 4 of which are covered (100%).$
job.py	$This PR adds 33 statements to fractal_server/app/models/v2/job.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/app/models/v2/job.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/models/v2/job.py is 100% (33/33). The file did not seem to exist on the base branch.$	$In this PR, 2 new statements are added to fractal_server/app/models/v2/job.py, 2 of which are covered (100%).$
workflowtask.py	$This PR adds 25 statements to fractal_server/app/models/v2/workflowtask.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/app/models/v2/workflowtask.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/models/v2/workflowtask.py is 100% (25/25). The file did not seem to exist on the base branch.$	$In this PR, 2 new statements are added to fractal_server/app/models/v2/workflowtask.py, 2 of which are covered (100%).$
fractal_server/app/routes/api/v2
images.py	$This PR adds 116 statements to fractal_server/app/routes/api/v2/images.py. The file did not seem to exist on the base branch.$	$This PR adds 3 statements missing coverage to fractal_server/app/routes/api/v2/images.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/routes/api/v2/images.py is 96.1% (113/116). The file did not seem to exist on the base branch.$	$In this PR, 12 new statements are added to fractal_server/app/routes/api/v2/images.py, 12 of which are covered (100%).$
workflowtask.py	$This PR adds 102 statements to fractal_server/app/routes/api/v2/workflowtask.py. The file did not seem to exist on the base branch.$	$This PR adds 2 statements missing coverage to fractal_server/app/routes/api/v2/workflowtask.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/routes/api/v2/workflowtask.py is 96.47% (100/102). The file did not seem to exist on the base branch.$	$In this PR, 1 new statements are added to fractal_server/app/routes/api/v2/workflowtask.py, 1 of which are covered (100%).$
fractal_server/app/runner/v2
merge_outputs.py	$This PR adds 16 statements to fractal_server/app/runner/v2/merge_outputs.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/app/runner/v2/merge_outputs.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/runner/v2/merge_outputs.py is 100% (16/16). The file did not seem to exist on the base branch.$	$In this PR, 6 new statements are added to fractal_server/app/runner/v2/merge_outputs.py, 6 of which are covered (100%).$
runner.py	$This PR adds 122 statements to fractal_server/app/runner/v2/runner.py. The file did not seem to exist on the base branch.$	$This PR adds 4 statements missing coverage to fractal_server/app/runner/v2/runner.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/runner/v2/runner.py is 95.06% (118/122). The file did not seem to exist on the base branch.$	$In this PR, 15 new statements are added to fractal_server/app/runner/v2/runner.py, 15 of which are covered (100%).$
task_interface.py	$This PR adds 49 statements to fractal_server/app/runner/v2/task_interface.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/app/runner/v2/task_interface.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/runner/v2/task_interface.py is 100% (49/49). The file did not seem to exist on the base branch.$	$In this PR, 20 new statements are added to fractal_server/app/runner/v2/task_interface.py, 20 of which are covered (100%).$
fractal_server/app/runner/v2/_local
__init__.py	$This PR adds 19 statements to fractal_server/app/runner/v2/_local/__init__.py. The file did not seem to exist on the base branch.$	$This PR adds 1 statements missing coverage to fractal_server/app/runner/v2/_local/__init__.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/runner/v2/_local/__init__.py is 90.47% (18/19). The file did not seem to exist on the base branch.$	$In this PR, 1 new statements are added to fractal_server/app/runner/v2/_local/__init__.py, 1 of which are covered (100%).$
fractal_server/app/runner/v2/_local_experimental
__init__.py	$This PR adds 25 statements to fractal_server/app/runner/v2/_local_experimental/__init__.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/app/runner/v2/_local_experimental/__init__.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/runner/v2/_local_experimental/__init__.py is 100% (25/25). The file did not seem to exist on the base branch.$	$In this PR, 1 new statements are added to fractal_server/app/runner/v2/_local_experimental/__init__.py, 1 of which are covered (100%).$
fractal_server/app/runner/v2/_slurm_ssh
__init__.py	$This PR adds 31 statements to fractal_server/app/runner/v2/_slurm_ssh/__init__.py. The file did not seem to exist on the base branch.$	$This PR adds 1 statements missing coverage to fractal_server/app/runner/v2/_slurm_ssh/__init__.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/runner/v2/_slurm_ssh/__init__.py is 93.93% (30/31). The file did not seem to exist on the base branch.$	$In this PR, 1 new statements are added to fractal_server/app/runner/v2/_slurm_ssh/__init__.py, 1 of which are covered (100%).$
fractal_server/app/runner/v2/_slurm_sudo
__init__.py	$This PR adds 22 statements to fractal_server/app/runner/v2/_slurm_sudo/__init__.py. The file did not seem to exist on the base branch.$	$This PR adds 2 statements missing coverage to fractal_server/app/runner/v2/_slurm_sudo/__init__.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/runner/v2/_slurm_sudo/__init__.py is 84.61% (20/22). The file did not seem to exist on the base branch.$	$In this PR, 1 new statements are added to fractal_server/app/runner/v2/_slurm_sudo/__init__.py, 1 of which are covered (100%).$
fractal_server/app/schemas
_filter_validators.py	$This PR adds 23 statements to fractal_server/app/schemas/_filter_validators.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/app/schemas/_filter_validators.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/schemas/_filter_validators.py is 100% (23/23). The file did not seem to exist on the base branch.$	$In this PR, 23 new statements are added to fractal_server/app/schemas/_filter_validators.py, 23 of which are covered (100%).$
_validators.py	$This PR adds 59 statements to fractal_server/app/schemas/_validators.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/app/schemas/_validators.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/schemas/_validators.py is 98.92% (59/59). The file did not seem to exist on the base branch.$	$In this PR, 6 new statements are added to fractal_server/app/schemas/_validators.py, 6 of which are covered (100%).$
fractal_server/app/schemas/v2
dataset.py	$This PR adds 78 statements to fractal_server/app/schemas/v2/dataset.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/app/schemas/v2/dataset.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/schemas/v2/dataset.py is 100% (78/78). The file did not seem to exist on the base branch.$	$In this PR, 27 new statements are added to fractal_server/app/schemas/v2/dataset.py, 27 of which are covered (100%).$
dumps.py	$This PR adds 38 statements to fractal_server/app/schemas/v2/dumps.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/app/schemas/v2/dumps.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/schemas/v2/dumps.py is 100% (38/38). The file did not seem to exist on the base branch.$	$In this PR, 4 new statements are added to fractal_server/app/schemas/v2/dumps.py, 4 of which are covered (100%).$
job.py	$This PR adds 66 statements to fractal_server/app/schemas/v2/job.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/app/schemas/v2/job.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/schemas/v2/job.py is 100% (66/66). The file did not seem to exist on the base branch.$	$In this PR, 9 new statements are added to fractal_server/app/schemas/v2/job.py, 9 of which are covered (100%).$
task.py	$This PR adds 126 statements to fractal_server/app/schemas/v2/task.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/app/schemas/v2/task.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/schemas/v2/task.py is 100% (126/126). The file did not seem to exist on the base branch.$	$In this PR, 1 new statements are added to fractal_server/app/schemas/v2/task.py, 1 of which are covered (100%).$
task_group.py	$This PR adds 82 statements to fractal_server/app/schemas/v2/task_group.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/app/schemas/v2/task_group.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/schemas/v2/task_group.py is 100% (82/82). The file did not seem to exist on the base branch.$	$In this PR, 1 new statements are added to fractal_server/app/schemas/v2/task_group.py, 1 of which are covered (100%).$
workflowtask.py	$This PR adds 118 statements to fractal_server/app/schemas/v2/workflowtask.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/app/schemas/v2/workflowtask.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/app/schemas/v2/workflowtask.py is 100% (118/118). The file did not seem to exist on the base branch.$	$In this PR, 19 new statements are added to fractal_server/app/schemas/v2/workflowtask.py, 19 of which are covered (100%).$
fractal_server/images
models.py	$This PR adds 53 statements to fractal_server/images/models.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/images/models.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/images/models.py is 98.55% (53/53). The file did not seem to exist on the base branch.$	$In this PR, 10 new statements are added to fractal_server/images/models.py, 10 of which are covered (100%).$
tools.py	$This PR adds 31 statements to fractal_server/images/tools.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/images/tools.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/images/tools.py is 100% (31/31). The file did not seem to exist on the base branch.$	$In this PR, 9 new statements are added to fractal_server/images/tools.py, 9 of which are covered (100%).$
Project Total

_{This report was generated by python-coverage-comment-action}

fractal_server/app/models/v2/dataset.py

fractal_server/app/models/v2/workflowtask.py

fractal_server/app/routes/api/v2/images.py

fractal_server/app/schemas/_validators.py

[skip ci]

fractal_server/app/runner/v2/merge_outputs.py

tests/v2/05_images/test_image_models.py

tests/v2/05_images/test_unit_image_tools.py

rename validator

9ee7b29

ychiucco linked an issue Jan 7, 2025 that may be closed by this pull request

Data structure for filters #2153

Open

add accept_none=True to valdict_scalarvalues

c64b3ed

ychiucco added 20 commits January 7, 2025 16:22

change dataset model and add migration

a3b1f91

rename variable

a6a0ce1

modify validator

6977433

update dataset schemas

bee422a

fix type hint

42c6bc7

remaining dataset schemas

22949ad

split filters in workflow task model

1bd9a8f

split filters in workflowtask schemas

50713fb

remove default factory from dataset read

10dd941

attribute_filters in jobv2

5016b3f

attribute_filters in jobv2 schemas

32bdef0

fix test_schemas_dataset_v2

3d89275

fix test_workflow_task_dump

1d33e67

remove old columns

d306002

new filters validators

d0faae0

fix test api dataset

2b77e1b

remove input_filters

6d2b5c7

remove Filters schema

f2a4af8

remove Filters

6527b8a

remove Filters from runner

643be06

tcompa reviewed Jan 8, 2025

View reviewed changes

fractal_server/app/models/v2/dataset.py Show resolved Hide resolved

tcompa reviewed Jan 8, 2025

View reviewed changes

fractal_server/app/models/v2/workflowtask.py Show resolved Hide resolved

tcompa reviewed Jan 8, 2025

View reviewed changes

fractal_server/app/routes/api/v2/images.py Outdated Show resolved Hide resolved

tcompa reviewed Jan 8, 2025

View reviewed changes

fractal_server/app/schemas/_validators.py Show resolved Hide resolved

tcompa reviewed Jan 8, 2025

View reviewed changes

fractal_server/app/schemas/_validators.py Outdated Show resolved Hide resolved

tcompa marked this pull request as ready for review January 15, 2025 08:35

tcompa and others added 7 commits January 15, 2025 09:46

Add some validation tests

2e74813

Extend some validation tests

27d03b9

remove outdated comment

337be2c

[skip ci]

unit test merge_outputs

4cda1f1

refactor merge_outputs

c18194c

improve unit test

f1d067f

test update_legacy_filters

6a67061

tcompa reviewed Jan 15, 2025

View reviewed changes

fractal_server/app/runner/v2/merge_outputs.py Outdated Show resolved Hide resolved

tcompa reviewed Jan 15, 2025

View reviewed changes

fractal_server/app/runner/v2/merge_outputs.py Show resolved Hide resolved

tcompa reviewed Jan 15, 2025

View reviewed changes

fractal_server/app/runner/v2/merge_outputs.py Outdated Show resolved Hide resolved

simplify check

ca2aba1

tcompa reviewed Jan 15, 2025

View reviewed changes

fractal_server/app/runner/v2/merge_outputs.py Outdated Show resolved Hide resolved

ychiucco and others added 6 commits January 15, 2025 14:41

deduplicate image_list_removals

ef05293

assert correct order

e77d32e

Comment [skip ci]

d23efb0

Add case to unit test

80801f9

Only use job_attribute_filters in runner - ref #2155

87f4fc6

Extend unit test

837ed30

tcompa reviewed Jan 15, 2025

View reviewed changes

tests/v2/05_images/test_image_models.py Outdated Show resolved Hide resolved

Rename schema

898c561

tcompa reviewed Jan 15, 2025

View reviewed changes

tests/v2/05_images/test_unit_image_tools.py Show resolved Hide resolved

ychiucco and others added 3 commits January 15, 2025 15:32

remove default from image_ok and image_fail

02d5526

look fok unexisting image

45d573b

CHANGELOG [skip ci]

0df4d4a

tcompa merged commit 5a48a82 into dev-2.11 Jan 15, 2025

This was linked to issues Jan 15, 2025

Prevent tasks from setting an output attribute filter #2170

Open

Replace attribute filters rather than merging them #2155

Open

Allow complex filters being set #1401

Open

tcompa deleted the 2153-data-structure-for-filters branch January 15, 2025 14:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data structure for filters #2168

Data structure for filters #2168

ychiucco commented Jan 7, 2025 •

edited by tcompa

Loading

github-actions bot commented Jan 7, 2025 •

edited

Loading

github-actions bot commented Jan 7, 2025 •

edited

Loading

Data structure for filters #2168

Data structure for filters #2168

Conversation

ychiucco commented Jan 7, 2025 • edited by tcompa Loading

Checklist before merging

github-actions bot commented Jan 7, 2025 • edited Loading

Benchmark comparison

GET /api/alive/

GET /api/v2/dataset/

GET /api/v2/job/

GET /api/v2/project/

GET /api/v2/task/

GET /api/v2/task-group/

GET /api/v2/workflow/

POST /api/v2/project/3/dataset/406/images/query/

GET /auth/current-user/

POST /auth/token/login/

github-actions bot commented Jan 7, 2025 • edited Loading

Coverage report

ychiucco commented Jan 7, 2025 •

edited by tcompa

Loading

github-actions bot commented Jan 7, 2025 •

edited

Loading

github-actions bot commented Jan 7, 2025 •

edited

Loading