Stuck in 3d_fullres for nnUNetv2_plan_experiment

[rafiibnsultan]

After running the command for nnUNetv2_plan_experiment -d 420 -tf 1 -tl 1 it gets stuck after a certain point.

In Configuration: 3d_fullres... it gets stuck for infinity after 96% of progress. Nothing happens, no errors are shown either.

Output stack:
nnUNetv2_plan_experiment -d 420 -tf 1 -tl 1
Preprocessing dataset Dataset420_CTORGSwinAttUNet
Configuration: 2d...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 119/119 [1:56:36<00:00, 58.79s/it]
Configuration: 3d_fullres...
96%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 114/119 [2:52:42<04:13, 50.76s/it]

[FabianIsensee]

Hey, can you please verify that your system did not run out of memory? Please try rerunning the 3d_fullres config with fewer workers. Not sure if you already did that because tl and tf are not valid for v2!?

[rafiibnsultan]

I ran it with and without the tl and tf, how to run it with fewer workers?

I don't think it ran out of memory, because I tried it 3 times each time it got stuck at 96%, it could also be a dataset issue

[FabianIsensee]

Are you using nnU-Net v2 or v1?