feat: add callbacks for resample and benchmark

[be-marc]

Additional overhead is less than 3% when using 10ms models

[sebffischer]

Looks good! I think we should also provide an example for how to implement a custom callback.

[be-marc]

Looks good! I think we should also provide an example for how to implement a custom callback.

callback_evaluation() has an example now

[mb706]

are there any interesting cases where workhorse() is executed in a different (parallelization cluster) process that might be interesting here that should be tested?

[be-marc]

Callbacks will behave differently depending on the parallelization.

  task = tsk("pima")
  learner = lrn("classif.rpart")
  resampling = rsmp("cv", folds = 3)

  callback = callback_evaluation("test",
    on_evaluation_begin = function(callback, context) {
      if (context$iteration == 1) context$task$select(c("age", "insulin"))
    }
  )

  rr = resample(task, learner, resampling, callbacks = callback)

  rr$learners[[1]]$state$feature_names
  # > [1] "age"     "insulin"
  rr$learners[[2]]$state$feature_names
  # > [1] "age"     "insulin"

  with_future("multisession", {
    rr = resample(task, learner, resampling, callbacks = callback)
  })

  rr$learners[[1]]$state$feature_names
  # > [1] "age"     "insulin"
  rr$learners[[2]]$state$feature_names
  # > [1] "age"      "glucose"  "insulin"  "mass"     "pedigree" "pregnant" "pressure" "triceps"

I think we should not clone these objects to get a consistent behavior. This will add unnecessary overhead. This is now documented in callback_evaluation().

[be-marc]

Use cases:

• Extract information from the model (without store_models = TRUE)
• Score learner on additional set
• Score HPCs on outer test set in nested resampling