GeneralizationOpt: diagnostics for the generalization gap of ML hyperparameters

Source: R/GeneralizationOpt.R

R interface to Python's GPopt.GeneralizationOpt. Explores a hyperparameter space with Sobol sequences, fits a surrogate model to the generalization gap (train vs. test performance), and lets you compute Sobol sensitivity indices and run diagnostics – useful to understand *which* hyperparameters drive over/under-fitting, beyond just tuning for best validation score.

GeneralizationOpt(hyperparams, random_state = 42L, venv_path = "./venv")

Arguments

hyperparams

a named list describing the hyperparameter space, in the form `list(param_name = c(min_val, max_val, log_scale))`, e.g. `list(nu = c(0.001, 10, TRUE), lam = c(1e-4, 1e4, TRUE))`

random_state

integer random seed

venv_path

path to the Python virtual environment created with `uv` (see the package README for setup instructions)

Value

A Python `GPopt.GeneralizationOpt` object. Useful members (accessed with `$`):

  • `$generate_sobol_samples(n = ...)`

  • `$evaluate_configurations(...)`

  • `$fit_surrogate(surrogate = ..., target_gap = "gap_abs")`

  • `$compute_sobol_indices(...)`

  • `$optimize_acquisition(...)`

  • `$run_diagnostics(...)`

Examples

if (FALSE) { # \dontrun{
gopt <- GeneralizationOpt(
  hyperparams = list(
    nu  = c(0.001, 10, TRUE),
    lam = c(1e-4, 1e4, TRUE)
  ),
  venv_path = "./venv"
)
samples <- gopt$generate_sobol_samples(n = 64L)
} # }