Saving a model check point when it reaches a certain loss

[blueyred]

Is there anyway to automatically save the "best" model during training rather than on fixed steps?
Often the checkpoints saved sit between the results with the best statistics (eg. on variance models - pitch_loss).

There looks like there's some form of monitor callback in the DsModelCheckpoint, I was wondering if it could be set to use the "_update_best_and_save" to monitor a parameter (like pitch loss or dur_loss on variance training) and automatically save a checkpoint when it hits a new "best value"?

This would be really useful in later stage training. Is it as simple as setting up a variable to change the monitor on DsModelCheckpoint from

monitor='step'

to

monitor='pitch_loss'

?

[blueyred]

After a bit of digging, I think I understand the situation, the answer is in here I think
https://lightning.ai/docs/pytorch/stable/common/checkpointing_intermediate.html

[yqzhishen]

We are using generative models, so loss or any objective metrics do not tell everything. The best judge is still the human brain.