Output checkpoint/restart files during div.B solve?

[gnwong]

It might be useful to let the user request restart files during the b_cleanup step of the resize_restart hullabaloo. That process takes long enough that I've been killed by the scheduler before, and I figure it wouldn't be that hard to make it output every N iterations.

This would be a massive qol improvement and remove some of the anxiety associated with hitting enter on the sbatch command...

[bprather]

This would be huge to have, yeah. It's something we discussed some back when the solver was much slower.

It should be simple to call the output functions in the right places and add Metadata::Restart to the necessary fields, probably just phi and possibly divB_RHS (the solve doesn't actually modify the B field until it's finished, so we'd need to save those temporary solution variables). That would be a first thing to get working and could be pretty simple.

However, there might be some non-trivial logic in distinguishing restarts from normal startup in CleanupDivergence, and treating the fields correctly when restarting. Currently it always replaces the contents of phiand divB_RHS, and that would need to be fixed in the case they're loaded. This still might not be too hard, actually, but it would require some testing I don't have much time for.