Feature request: Option to enable timing for each iteration

[EmilHvitfeldt]

Pretty much what it says in the title.

Sometimes I have a suspicion that the computational time is related to one of the hyperparameter and it would be neat if you were able to test that hypothesis.

[juliasilge]

Related to #23

[simonpcouch]

Reiterated in a SO post here: https://stackoverflow.com/questions/74815668/is-there-a-metric-for-run-times-or-some-other-recommended-way-to-collect-runtim

[bokov]

I understand and respect that the devs are not going in that direction...

#450

Do the devs or any users reading this have any workaround suggestions for collecting timings as cleanly as possible without a built-in feature for now?

[bokov]

Pretty much what it says in the title.

Sometimes I have a suspicion that the computational time is related to one of the hyperparameter and it would be neat if you were able to test that hypothesis.

@EmilHvitfeldt , in your PR you mention ...

The timing is done by using system.time() (like we do in parsnip) around the tune_grid_loop_iter() call in tune_grid_loop_iter_safely().

Is there a command or option setting in parsnip that records run-times? Could you elaborate on where to find it?
Thanks.

[EmilHvitfeldt]

If you set verbosity = 2 then parsnip records run time. But there isn't a easy way to handle this in tune, and it wouldn't record the preprocessing times which in some cases takes longer than the model-fit

library(parsnip)

lm_spec <- linear_reg()

lm_fit <- fit(lm_spec, mpg ~ ., data = mtcars,
              control = control_parsnip(verbosity = 2))

lm_fit$elapsed
#>    user  system elapsed 
#>   0.002   0.000   0.002

^{Created on 2022-12-15 with reprex v2.0.2}

[simonpcouch]

Ah, related to #880, tidymodels/workflows#191, and friends. :)