Calling add-event for an empty project fails with "Attempting to overwrite existing event. "

[lcocault-cap]

Description

While testing the RAMP framework with the "Titanic" challenge from the "ramp kits", in order to check the proper setup of my environment, my "add-event" command is rejected with a message "Attempting to overwrite existing event. " whereas my database is actually empty.
Note that, in my context, the PostgreSQL database is deployed from a Docker image. I do not see any for that deployment detail to cause the issue, but it is probably the only "specific" deployment decision I have made.

Steps/Code to Reproduce

RAMP installation is based on the following steps:

1. python3.7 -m venv ramp-board
2. ramp-board/bin/activate
3. pip install --trusted-host artifactory.cnes.fr -r requirements.txt
4. make install
6. createuser -P ramp
7. createdb -O ramp rampdb
8. pip install --trusted-host artifactory.cnes.fr -r requirements.txt
9. ramp_test_submission --submission starting_kit --save-y-preds
10. ramp-test --submission random_forest_20_5 --save-y-preds
11. ramp_leaderboard --metric=acc
12. mkdir ramp-challenge
13. cd ramp-challenge/
14. ramp-setup init
15. ramp-setup init-event --name=titanic_bateau_coule
16. Edit the events/titanic_bateau_coule/config.yml (to point to the database)
17. cd events/titanic_bateau_coule
18. cp -R /work/RAMP/titanic/* .
19. ramp-database add-event

Expected Results

I expected the add-event to succeed

Actual Results

Instead, I have the following stacktrace:
Traceback (most recent call last):
File "/work/RAMP/venv/ramp-board/bin/ramp-database", line 11, in
load_entry_point('ramp-database==0.9.0.dev0', 'console_scripts', 'ramp-database')()
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_database/cli.py", line 540, in start
main()
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_database/cli.py", line 261, in add_event
force,
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_database/tools/event.py", line 261, in add_event
"Attempting to overwrite existing event. "
ValueError: Attempting to overwrite existing event. Use force=True to overwrite.

When I check my database, I can confirm there is no existing event in the "events" table.

Anyway, I tried running the command with the force argument as suggested:
ramp-database add-event --force=True
Traceback (most recent call last):
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2634, in delete
state = attributes.instance_state(instance)
AttributeError: 'list' object has no attribute '_sa_instance_state'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/work/RAMP/venv/ramp-board/bin/ramp-database", line 11, in
load_entry_point('ramp-database==0.9.0.dev0', 'console_scripts', 'ramp-database')()
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_database/cli.py", line 540, in start
main()
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 754, in invoke
return _callback(*args, **kwargs)
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_database/cli.py", line 261, in add_event
force,
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_database/tools/event.py", line 264, in add_event
delete_event(session, event_name)
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_database/tools/event.py", line 72, in delete_event
session.delete(event)
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2638, in delete
replace_context=err,
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise
raise exception
sqlalchemy.orm.exc.UnmappedInstanceError: Class 'builtins.list' is not mapped

Versions

Python 3.7.12 (default, Nov 17 2021, 09:09:30)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
NumPy 1.21.4
SciPy 1.7.3
Scikit-Learn 1.0.1

[agramfort]

thx @lcocault-cap for the detailed bug report.

@frcaud can you try to replicate?

@lcocault-cap can you share the docker file you used to setup the container?

[lcocault-cap]

@agramfort , since I test within the CNES environment I depend on the images available in the CNES artifactory and I am not sure it is relevant to extract and send the whole image file. However, based on the image metadata, I think the version I use corresponds to this commit: docker-library/postgres@7f5f6da

[frcaud]

@lcocault-cap did you first try to deploy the event with the 'deploy-event' command ?

Deploy event (add problem and event to the database, optionally clone kit and data)

ramp-setup deploy-event [OPTIONS]

It seems to me that add-event should be used to add an event to an existing problem (+event) previously deployed with 'deploy-event'

[lcocault-cap]

I tried the deploy-event and I have another error.

Traceback (most recent call last):
File "/work/RAMP/venv/ramp-board/bin/ramp-setup", line 11, in
load_entry_point('ramp-utils==0.9.0.dev0', 'console_scripts', 'ramp-setup')()
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_utils/cli.py", line 234, in start
main()
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_utils/cli.py", line 131, in deploy_event
deploy.deploy_ramp_event(config, event_config, cloning, force)
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_utils/deploy.py", line 36, in deploy_ramp_event
ramp_config = generate_ramp_config(event_config, config)
File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_utils/ramp.py", line 62, in generate_ramp_config
config = event_config["ramp"]
KeyError: 'ramp'

Maybe something is missing in my configuration file?

[rth]

ramp-database add-event

@lcocault-cap Can you try:

ramp setup deploy-event --event-config events/iris_test/config.yml --no-cloning

instead as indicated in https://paris-saclay-cds.github.io/ramp-docs/ramp-board/stable/create_ramp_event.html#deploy-a-specific-ramp-event

I don't know if anyone used ramp-database add-event recently, but personally, I never did to create new events.

[lcocault-cap]

Hey, this is far better. After a few configuration actions, it finally worked. Now, my events table is initialized:

rampdb=# select * from events;
3 | titanic_bateau_coule | Human readable event name to display on website | 3 | 1 |
80 | f | f | t | t | f |
900 | 2000-01-01 00:00:00 | 2100-01-01 00:00:00 | 2100-01-01 00:00:00 | auc |
| | |
| 0 | | | |
| | |
| starting_kit | /work/RAMP/venv/ramp-board/ramp-challenge/events/titanic_bateau_coule/submissions

As a conclusion, what do you advice for this issue? Should I close it as irrelevant? Or is there a problem with the add-event that I am the first to declare because this feature would never be used? Maybe, the add-event action should be disposed?

[agramfort]

+1 to deprecate "add-event action" Message ID: ***@***.***>

…

[frcaud]

I tested 'add-event' further. It creates an event in database when the options are filled and provided that problem (+event) was previously created with deploy-event:
Example:
ramp setup init-event --name titanic_test
(fill config.yml with relevant infos)
ramp setup deploy-event --event-config events/titanic_test/config.yml
Now titanic_test is deployed and the event appears in the DB as well as on frontend

If we now use add-event for titanic_test2:
ramp-database add-event --problem titanic --event titanic_test2 --title <title> --submissions-dir <subdir>
this new event will appear in the DB but not on frontend.

Best way to create another event for an already deployed problem is to use deploy-event though:
first init-event: ramp setup init-event --name titanic_test3
(fill config.yml with relevant infos)
ramp setup deploy-event --event-config events/titanic_test3/config.yml

titanic_test and titanic_test3 are correctly deployed on frontend and appear in DB
titanic_test2 is on DB but not on frontend

[agramfort]

thanks for the investigation. Do not hesitate to update the doc to clarify this. Message ID: ***@***.***>

…