Filenotfound error when attempting single animal model training

[viralmysteries]

Note: I am a very new user to SLEAP and to Python more generally. I want to use SLEAP to analyze some video files I take of sea slug locomotion. I was running through the tutorial and everything was going well until I tried to run training on my video. I wanted to use the single animal mode, as I am only looking at the activity of 1 animal, but every time I ran the model, it returned the following error:

FileNotFoundError: [WinError 2] The system cannot find the file specified
Run Path: E:/Viral's Doctoral Education/Doctoral Research/Aged Aplysia Videos/Juveniles/Aug2222\models\220912_133904.single_instance.n=22

I didn't see anything about initializing the file for the model in the tutorial, so I was very confused. Is there something really simple that I've simply missed? I'd appreciate any help folks can provide.

[talmo]

Hi @viralmysteries,

Can you try moving your project + data folder over to a path that doesn't contain an apostrophe ("Viral**'**s"), re-running it and letting us know if it works? This might be a bug in how we handle paths in particular places.

Talmo

[viralmysteries]

Hey thanks for the quick reply! Unfortunately I did that and nothing changed, I had the same error.

[talmo]

Hi @viralmysteries,

Thanks for trying that!

Do you mind pasting the full console log? I feel like this error might be a symptom of an earlier issue.

Did the training complete properly before you got this error?

Cheers,

Talmo

[viralmysteries]

Here you go! And no, the training did not complete. The error happened maybe 10 seconds after I hit RUN. It generated the GUI box for displaying the training, but it didn't show anything before it gave me the error.

Start training single_instance...
['sleap-train', 'C:\Users\18622\AppData\Local\Temp\tmp83qdknqt\220912_182628_training_job.json', 'C:/Users/18622/Documents/SLEAP/Aug2222_Stim1.slp', '--zmq', '--save_viz']
INFO:sleap.nn.training:Versions:
SLEAP: 1.2.7
TensorFlow: 2.6.3
Numpy: 1.19.5
Python: 3.7.12
OS: Windows-10-10.0.19041-SP0
INFO:sleap.nn.training:Training labels file: C:/Users/18622/Documents/SLEAP/Aug2222_Stim1.slp
INFO:sleap.nn.training:Training profile: C:\Users\18622\AppData\Local\Temp\tmp83qdknqt\220912_182628_training_job.json
INFO:sleap.nn.training:
INFO:sleap.nn.training:Arguments:
INFO:sleap.nn.training:{
"training_job_path": "C:\Users\18622\AppData\Local\Temp\tmp83qdknqt\220912_182628_training_job.json",
"labels_path": "C:/Users/18622/Documents/SLEAP/Aug2222_Stim1.slp",
"video_paths": [
""
],
"val_labels": null,
"test_labels": null,
"tensorboard": false,
"save_viz": true,
"zmq": true,
"run_name": "",
"prefix": "",
"suffix": "",
"cpu": false,
"first_gpu": false,
"last_gpu": false,
"gpu": "auto"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Training job:
INFO:sleap.nn.training:{
"data": {
"labels": {
"training_labels": null,
"validation_labels": null,
"validation_fraction": 0.1,
"test_labels": null,
"split_by_inds": false,
"training_inds": null,
"validation_inds": null,
"test_inds": null,
"search_path_hints": [],
"skeletons": []
},
"preprocessing": {
"ensure_rgb": false,
"ensure_grayscale": true,
"imagenet_mode": null,
"input_scaling": 1.0,
"pad_to_stride": null,
"resize_and_pad_to_target": true,
"target_height": null,
"target_width": null
},
"instance_cropping": {
"center_on_part": null,
"crop_size": null,
"crop_size_detection_padding": 16
}
},
"model": {
"backbone": {
"leap": null,
"unet": {
"stem_stride": null,
"max_stride": 16,
"output_stride": 2,
"filters": 16,
"filters_rate": 2.0,
"middle_block": true,
"up_interpolate": true,
"stacks": 1
},
"hourglass": null,
"resnet": null,
"pretrained_encoder": null
},
"heads": {
"single_instance": {
"part_names": null,
"sigma": 2.5,
"output_stride": 2,
"loss_weight": 1.0,
"offset_refinement": false
},
"centroid": null,
"centered_instance": null,
"multi_instance": null,
"multi_class_bottomup": null,
"multi_class_topdown": null
}
},
"optimization": {
"preload_data": true,
"augmentation_config": {
"rotate": true,
"rotation_min_angle": -15.0,
"rotation_max_angle": 15.0,
"translate": false,
"translate_min": -5,
"translate_max": 5,
"scale": false,
"scale_min": 0.9,
"scale_max": 1.1,
"uniform_noise": false,
"uniform_noise_min_val": 0.0,
"uniform_noise_max_val": 10.0,
"gaussian_noise": false,
"gaussian_noise_mean": 5.0,
"gaussian_noise_stddev": 1.0,
"contrast": false,
"contrast_min_gamma": 0.5,
"contrast_max_gamma": 2.0,
"brightness": false,
"brightness_min_val": 0.0,
"brightness_max_val": 10.0,
"random_crop": false,
"random_crop_height": 256,
"random_crop_width": 256,
"random_flip": false,
"flip_horizontal": true
},
"online_shuffling": true,
"shuffle_buffer_size": 128,
"prefetch": true,
"batch_size": 4,
"batches_per_epoch": null,
"min_batches_per_epoch": 200,
"val_batches_per_epoch": null,
"min_val_batches_per_epoch": 10,
"epochs": 200,
"optimizer": "adam",
"initial_learning_rate": 0.0001,
"learning_rate_schedule": {
"reduce_on_plateau": true,
"reduction_factor": 0.5,
"plateau_min_delta": 1e-06,
"plateau_patience": 5,
"plateau_cooldown": 3,
"min_learning_rate": 1e-08
},
"hard_keypoint_mining": {
"online_mining": false,
"hard_to_easy_ratio": 2.0,
"min_hard_keypoints": 2,
"max_hard_keypoints": null,
"loss_scale": 5.0
},
"early_stopping": {
"stop_training_on_plateau": true,
"plateau_min_delta": 1e-08,
"plateau_patience": 10
}
},
"outputs": {
"save_outputs": true,
"run_name": "220912_182627.single_instance.n=22",
"run_name_prefix": "",
"run_name_suffix": "",
"runs_folder": "C:/Users/18622/Documents/SLEAP\models",
"tags": [
""
],
"save_visualizations": true,
"delete_viz_images": true,
"zip_outputs": false,
"log_to_csv": true,
"checkpointing": {
"initial_model": false,
"best_model": true,
"every_epoch": false,
"latest_model": false,
"final_model": false
},
"tensorboard": {
"write_logs": false,
"loss_frequency": "epoch",
"architecture_graph": false,
"profile_graph": false,
"visualizations": true
},
"zmq": {
"subscribe_to_controller": true,
"controller_address": "tcp://127.0.0.1:9000",
"controller_polling_timeout": 10,
"publish_updates": true,
"publish_address": "tcp://127.0.0.1:9001"
}
},
"name": "",
"description": "",
"sleap_version": "1.2.7",
"filename": "C:\Users\18622\AppData\Local\Temp\tmp83qdknqt\220912_182628_training_job.json"
}
INFO:sleap.nn.training:
Traceback (most recent call last):
File "C:\Users\18622\miniconda3\envs\sleap\Scripts\sleap-train-script.py", line 33, in
sys.exit(load_entry_point('sleap==1.2.7', 'console_scripts', 'sleap-train')())
File "C:\Users\18622\miniconda3\envs\sleap\lib\site-packages\sleap\nn\training.py", line 1947, in main
gpu_ind = np.argmax(sleap.nn.system.get_gpu_memory())
File "C:\Users\18622\miniconda3\envs\sleap\lib\site-packages\sleap\nn\system.py", line 201, in get_gpu_memory
memory_poll = subprocess.run(command, capture_output=True)
File "C:\Users\18622\miniconda3\envs\sleap\lib\subprocess.py", line 488, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\18622\miniconda3\envs\sleap\lib\subprocess.py", line 800, in init
restore_signals, start_new_session)
File "C:\Users\18622\miniconda3\envs\sleap\lib\subprocess.py", line 1207, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
Run Path: C:/Users/18622/Documents/SLEAP\models\220912_182627.single_instance.n=22

[roomrys]

Hi @viralmysteries,

The error seems to be caused from calling get_gpu_memory as seen in this line of your traceback:

File "C:\Users\18622\miniconda3\envs\sleap\lib\site-packages\sleap\nn\system.py", line 201, in get_gpu_memory
memory_poll = subprocess.run(command, capture_output=True)

Can you try installing SLEAP v1.2.8 - we realize that there was an issue with how we were allocating which GPU to use.

Let us know how that works!

Thanks,
Liezl