Replace temporary files with database access in highest level runner module #2169

mfranzon · 2025-01-08T08:19:25Z

close #2135

Checklist before merging

I added an appropriate entry to CHANGELOG.md
I added logging to new code - if appropriate.
I merged main into the current branch.

… history, filters, images objects

…ccess-in-highest-level-runner-module

github-actions · 2025-01-08T09:31:24Z

Benchmark comparison

GET /api/alive/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`
vanilla	0.8	1.0	0.80
power	0.8	0.9	0.89
dataset	0.8	1.0	0.80
project	0.8	1.0	0.80
job	0.8	0.9	0.89

GET /api/v2/dataset/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	10.8	9.8	1.10	0.6	0.6
power	425.5	434.1	0.98	366.2	366.2
dataset	160.3	162.4	0.99	134.0	134.0
project	47.4	47.5	1.00	47.2	47.2
job	30.9	31.0	1.00	32.4	32.4

GET /api/v2/job/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	7.1	6.2	1.15	2.2	2.2
power	142.9	143.2	1.00	652.2	652.2
dataset	150.6	152.9	0.98	905.2	905.2
project	48.8	49.6	0.98	283.8	283.8
job	43.4	41.0	1.06	224.9	224.9

GET /api/v2/project/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	7.5	8.1	0.93	0.1	0.1
power	7.2	6.9	1.04	0.1	0.1
dataset	7.2	7.1	1.01	0.1	0.1
project	9.4	9.6	0.98	2.3	2.3
job	7.3	7.8	0.94	0.1	0.1

GET /api/v2/task/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	6.0	5.9	1.02	0.8	0.8
power	6.2	6.0	1.03	0.8	0.8
dataset	6.3	6.1	1.03	0.8	0.8
project	6.2	6.5	0.95	0.8	0.8
job	6.4	6.1	1.05	0.8	0.8

GET /api/v2/task-group/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	8.4	7.9	1.06	1.6	1.6
power	7.2	6.8	1.06	1.6	1.6
dataset	7.2	7.0	1.03	1.6	1.6
project	7.4	7.5	0.99	1.6	1.6
job	7.2	7.8	0.92	1.6	1.6

GET /api/v2/workflow/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	11.8	15.5	0.76	0.8	0.8
power	26.1	26.6	0.98	29.1	29.1
dataset	22.0	20.6	1.07	16.7	16.7
project	24.8	22.6	1.10	21.0	21.0
job	12.7	11.1	1.14	0.8	0.8

POST /api/v2/project/3/dataset/406/images/query/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
dataset	156.1	163.0	0.96	275.1	275.1

GET /auth/current-user/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	4.8	4.9	0.98	0.2	0.2
power	4.5	4.7	0.96	0.2	0.2
dataset	4.5	4.7	0.96	0.2	0.2
project	4.6	4.7	0.98	0.2	0.2
job	4.4	4.7	0.94	0.1	0.1

POST /auth/token/login/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
dataset	274.2	274.2	1.00	0.2	0.2

fractal_server/app/routes/api/v2/status.py

github-actions · 2025-01-09T09:21:31Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)
fractal_server
__init__.py	$This PR adds 1 statement to fractal_server/__init__.py. The file did not seem to exist on the base branch.$	$This PR adds 0 statements missing coverage to fractal_server/__init__.py. The file did not seem to exist on the base branch.$	$The coverage rate of fractal_server/__init__.py is 100% (1/1). The file did not seem to exist on the base branch.$	$In this PR, 1 new statements are added to fractal_server/__init__.py, 1 of which are covered (100%).$
Project Total

_{This report was generated by python-coverage-comment-action}

…ccess-in-highest-level-runner-module

…sage

…ghest-level-runner-module' of github.com:fractal-analytics-platform/fractal-server into 2135-replace-temporary-files-with-database-access-in-highest-level-runner-module

…s submitted

fractal_server/app/routes/api/v2/status.py

…orkflow) functions

…ghest-level-runner-module' of github.com:fractal-analytics-platform/fractal-server into 2135-replace-temporary-files-with-database-access-in-highest-level-runner-module

…se-access-in-highest-level-runner-module

…ghest-level-runner-module' of github.com:fractal-analytics-platform/fractal-server into 2135-replace-temporary-files-with-database-access-in-highest-level-runner-module

tests/v2/03_api/test_api_status.py

fractal_server/app/runner/v2/handle_failed_job.py

fractal_server/app/runner/v2/runner.py

mfranzon added 2 commits January 7, 2025 15:50

update runner introducing db connections and removing local files for…

30d82f7

… history, filters, images objects

Merge branch 'main' into 2135-replace-temporary-files-with-database-a…

750bfac

…ccess-in-highest-level-runner-module

mfranzon linked an issue Jan 8, 2025 that may be closed by this pull request

Replace temporary files with database access, in highest-level runner module #2135

Open

mfranzon added 3 commits January 8, 2025 09:24

fix get call

0bafd41

remove populate_existing attribute

096aad9

get dataset from db in runner

ff664d3

mfranzon added 5 commits January 8, 2025 14:24

update tests, use db instead of python objects

0074c2b

remove DatasetV2Mock, use DatasetV2 model

813a778

add database logic for history update

a97c086

add database logic for attrs update, move tmp_history, inside loop

9e360b6

remove history,filters,image local files

42b03c9

tcompa reviewed Jan 9, 2025

View reviewed changes

fractal_server/app/routes/api/v2/status.py Outdated Show resolved Hide resolved

tcompa and others added 8 commits January 9, 2025 11:11

Merge branch 'main' into 2135-replace-temporary-files-with-database-a…

fae9350

…ccess-in-highest-level-runner-module

[skip ci] BROKEN - add submitted status in history, improve history u…

d41516c

…sage

Merge branch '2135-replace-temporary-files-with-database-access-in-hi…

177ada2

…ghest-level-runner-module' of github.com:fractal-analytics-platform/fractal-server into 2135-replace-temporary-files-with-database-access-in-highest-level-runner-module

add submitted status in history, improve history usage

de29ce9

add logic if failed_wftask is None, set last wftask to failed if it i…

15d485e

…s submitted

add condition if wf has no task

cd4351f

use db data for runner response, adapt tests

69ac3ff

fix tests with new status endpoint logic

64dd991

tcompa reviewed Jan 13, 2025

View reviewed changes

fractal_server/app/routes/api/v2/status.py Outdated Show resolved Hide resolved

mfranzon and others added 7 commits January 13, 2025 10:53

review status endpoint logic, add arg to flake8

81f7267

update test for coverage

e2cbfdc

Possible other way of finding SUBMITTED wftasks

4da4d1c

Remove spurious enumerate

3b0c1ac

Style

45f00a8

Fix test

e3c5460

Add log

cf26693

tcompa and others added 14 commits January 14, 2025 09:17

Do not return dataset attributes from execute_tasks_v2 (and process-w…

e826ac9

…orkflow) functions

BROKEN - start updating tests/v2/04_runner/test_fractal_examples.py

9201e7d

refactor tests

ee978f8

BROKEN - start updating tests/v2/04_runner/test_fractal_examples.py

8e16efd

Merge branch '2135-replace-temporary-files-with-database-access-in-hi…

e218298

…ghest-level-runner-module' of github.com:fractal-analytics-platform/fractal-server into 2135-replace-temporary-files-with-database-access-in-highest-level-runner-module

Fix tests/v2/04_runner/test_fractal_examples.py

ac2e956

Style/docstrings/comments

6425917

refactor test_dummy_example

57fabf3

Merge branch '2135-replace-temporary-files-with-database-access-in-hi…

0b5974b

…ghest-level-runner-module' of github.com:fractal-analytics-platform/fractal-server into 2135-replace-temporary-files-with-database-access-in-highest-level-runner-module

Remove some output of execute_tasks_v2

ef76daf

refactor test_dummy_example

60f4edd

Merge branch '2135-replace-temporary-files-with-database-access-in-hi…

99d8e85

…ghest-level-runner-module' of github.com:fractal-analytics-platform/fractal-server into 2135-replace-temporary-files-with-database-access-in-highest-level-runner-module

Review how to set task status to failed

ce8b73f

Style [skip ci]

23e5151

tcompa changed the base branch from main to dev-2.11 January 14, 2025 10:11

tcompa and others added 5 commits January 14, 2025 11:14

Merge branch 'dev-2.11' into 2135-replace-temporary-files-with-databa…

6de6757

…se-access-in-highest-level-runner-module

fix arg, update test

1a6e86a

Merge branch '2135-replace-temporary-files-with-database-access-in-hi…

b8099a8

…ghest-level-runner-module' of github.com:fractal-analytics-platform/fractal-server into 2135-replace-temporary-files-with-database-access-in-highest-level-runner-module

CHANGELOG [skip ci]

b7ef0ec

Trigger GHA workflows also for dev-2.11 branch

d2ebb5a

tcompa reviewed Jan 14, 2025

View reviewed changes

tests/v2/03_api/test_api_status.py Outdated Show resolved Hide resolved

tcompa reviewed Jan 14, 2025

View reviewed changes

fractal_server/app/runner/v2/handle_failed_job.py Outdated Show resolved Hide resolved

tcompa reviewed Jan 14, 2025

View reviewed changes

fractal_server/app/runner/v2/runner.py Outdated Show resolved Hide resolved

tcompa added 5 commits January 14, 2025 11:48

Handle empty-history case in mark_last_wftask_as_failed

95f6bf7

Fix comment

30ab975

Fix f string

34879b7

Remove db.refresh

3d56db2

Add test_unit_mark_last_wftask_as_failed

74dfe65

tcompa merged commit 803c147 into dev-2.11 Jan 14, 2025
17 checks passed

tcompa deleted the 2135-replace-temporary-files-with-database-access-in-highest-level-runner-module branch January 14, 2025 11:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace temporary files with database access in highest level runner module #2169

Replace temporary files with database access in highest level runner module #2169

mfranzon commented Jan 8, 2025 •

edited by tcompa

Loading

github-actions bot commented Jan 8, 2025 •

edited

Loading

github-actions bot commented Jan 9, 2025 •

edited

Loading

Replace temporary files with database access in highest level runner module #2169

Replace temporary files with database access in highest level runner module #2169

Conversation

mfranzon commented Jan 8, 2025 • edited by tcompa Loading

Checklist before merging

github-actions bot commented Jan 8, 2025 • edited Loading

Benchmark comparison

GET /api/alive/

GET /api/v2/dataset/

GET /api/v2/job/

GET /api/v2/project/

GET /api/v2/task/

GET /api/v2/task-group/

GET /api/v2/workflow/

POST /api/v2/project/3/dataset/406/images/query/

GET /auth/current-user/

POST /auth/token/login/

github-actions bot commented Jan 9, 2025 • edited Loading

Coverage report

mfranzon commented Jan 8, 2025 •

edited by tcompa

Loading

github-actions bot commented Jan 8, 2025 •

edited

Loading

github-actions bot commented Jan 9, 2025 •

edited

Loading