botorch.optim¶

Core¶

Core abstractions and generic optimizers.

class botorch.optim.core.OptimizationStatus(value)[source]¶

Bases: int, Enum

An enumeration.

RUNNING = 1¶

SUCCESS = 2¶

FAILURE = 3¶

STOPPED = 4¶

class botorch.optim.core.OptimizationResult(step: 'int', fval: 'Union[float, int]', status: 'OptimizationStatus', runtime: 'Optional[float]' = None, message: 'Optional[str]' = None)[source]¶

Bases: object

Parameters:

• step (int) –

• fval (float | int) –

• status (OptimizationStatus) –

• runtime (float | None) –

• message (str | None) –

step: int¶

fval: float | int¶

status: OptimizationStatus¶

runtime: float | None = None¶

message: str | None = None¶

botorch.optim.core.scipy_minimize(closure, parameters, bounds=None, callback=None, x0=None, method='L-BFGS-B', options=None, timeout_sec=None)[source]¶

Generic scipy.optimize.minimize-based optimization routine.

Parameters:

• closure (Callable[[], Tuple[Tensor, Sequence[Tensor | None]]] | NdarrayOptimizationClosure) – Callable that returns a tensor and an iterable of gradient tensors or NdarrayOptimizationClosure instance.

• parameters (Dict[str, Tensor]) – A dictionary of tensors to be optimized.

• bounds (Dict[str, Tuple[float | None, float | None]] | None) – A dictionary mapping parameter names to lower and upper bounds.

• callback (Callable[[Dict[str, Tensor], OptimizationResult], None] | None) – A callable taking parameters and an OptimizationResult as arguments.

• x0 (ndarray | None) – An optional initialization vector passed to scipy.optimize.minimize.

• method (str) – Solver type, passed along to scipy.minimize.

• options (Dict[str, Any] | None) – Dictionary of solver options, passed along to scipy.minimize.

• timeout_sec (float | None) – Timeout in seconds to wait before aborting the optimization loop if not converged (will return the best found solution thus far).

Returns:

An OptimizationResult summarizing the final state of the run.

Return type:

OptimizationResult

botorch.optim.core.torch_minimize(closure, parameters, bounds=None, callback=None, optimizer=<class 'torch.optim.adam.Adam'>, scheduler=None, step_limit=None, timeout_sec=None, stopping_criterion=None)[source]¶

Generic torch.optim-based optimization routine.

Parameters:

• closure (Callable[[], Tuple[Tensor, Sequence[Tensor | None]]]) – Callable that returns a tensor and an iterable of gradient tensors. Responsible for setting relevant parameters’ grad attributes.

• parameters (Dict[str, Tensor]) – A dictionary of tensors to be optimized.

• bounds (Dict[str, Tuple[float | None, float | None]] | None) – An optional dictionary of bounds for elements of parameters.

• callback (Callable[[Dict[str, Tensor], OptimizationResult], None] | None) – A callable taking parameters and an OptimizationResult as arguments.

• optimizer (Optimizer | Callable[[List[Tensor]], Optimizer]) – A torch.optim.Optimizer instance or a factory that takes a list of parameters and returns an Optimizer instance.

• scheduler (LRScheduler | Callable[[Optimizer], LRScheduler] | None) – A torch.optim.lr_scheduler._LRScheduler instance or a factory that takes a Optimizer instance and returns a _LRSchedule instance.

• step_limit (int | None) – Integer specifying a maximum number of optimization steps. One of step_limit, stopping_criterion, or timeout_sec must be passed.

• timeout_sec (float | None) – Timeout in seconds before terminating the optimization loop. One of step_limit, stopping_criterion, or timeout_sec must be passed.

• stopping_criterion (Callable[[Tensor], bool] | None) – A StoppingCriterion for the optimization loop.

Returns:

An OptimizationResult summarizing the final state of the run.

Return type:

OptimizationResult

Acquisition Function Optimization¶

Methods for optimizing acquisition functions.

class botorch.optim.optimize.OptimizeAcqfInputs(acq_function, bounds, q, num_restarts, raw_samples, options, inequality_constraints, equality_constraints, nonlinear_inequality_constraints, fixed_features, post_processing_func, batch_initial_conditions, return_best_only, gen_candidates, sequential, ic_generator=None, timeout_sec=None, return_full_tree=False, retry_on_optimization_warning=True, ic_gen_kwargs=<factory>)[source]¶

Bases: object

Container for inputs to optimize_acqf.

See docstring for optimize_acqf for explanation of parameters.

Parameters:

• acq_function (AcquisitionFunction) –

• bounds (Tensor) –

• q (int) –

• num_restarts (int) –

• raw_samples (int | None) –

• options (Dict[str, bool | float | int | str] | None) –

• inequality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

• equality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

• nonlinear_inequality_constraints (List[Callable] | None) –

• fixed_features (Dict[int, float] | None) –

• post_processing_func (Callable[[Tensor], Tensor] | None) –

• batch_initial_conditions (Tensor | None) –

• return_best_only (bool) –

• gen_candidates (Callable[[Tensor, AcquisitionFunction, Any], Tuple[Tensor, Tensor]]) –

• sequential (bool) –

• ic_generator (Callable[[qKnowledgeGradient, Tensor, int, int, int, Dict[int, float] | None, Dict[str, bool | float | int] | None, List[Tuple[Tensor, Tensor, float]] | None, List[Tuple[Tensor, Tensor, float]] | None], Tensor | None] | None) –

• timeout_sec (float | None) –

• return_full_tree (bool) –

• retry_on_optimization_warning (bool) –

• ic_gen_kwargs (Dict) –

acq_function: AcquisitionFunction¶

bounds: Tensor¶

q: int¶

num_restarts: int¶

raw_samples: int | None¶

options: Dict[str, bool | float | int | str] | None¶

inequality_constraints: List[Tuple[Tensor, Tensor, float]] | None¶

equality_constraints: List[Tuple[Tensor, Tensor, float]] | None¶

nonlinear_inequality_constraints: List[Callable] | None¶

fixed_features: Dict[int, float] | None¶

post_processing_func: Callable[[Tensor], Tensor] | None¶

batch_initial_conditions: Tensor | None¶

return_best_only: bool¶

gen_candidates: Callable[[Tensor, AcquisitionFunction, Any], Tuple[Tensor, Tensor]]¶

sequential: bool¶

ic_generator: Callable[[qKnowledgeGradient, Tensor, int, int, int, Dict[int, float] | None, Dict[str, bool | float | int] | None, List[Tuple[Tensor, Tensor, float]] | None, List[Tuple[Tensor, Tensor, float]] | None], Tensor | None] | None = None¶

timeout_sec: float | None = None¶

return_full_tree: bool = False¶

retry_on_optimization_warning: bool = True¶

ic_gen_kwargs: Dict¶

property full_tree: bool¶

get_ic_generator()[source]¶

Return type:

Callable[[qKnowledgeGradient, Tensor, int, int, int, Dict[int, float] | None, Dict[str, bool | float | int] | None, List[Tuple[Tensor, Tensor, float]] | None, List[Tuple[Tensor, Tensor, float]] | None], Tensor | None]

botorch.optim.optimize.optimize_acqf(acq_function, bounds, q, num_restarts, raw_samples=None, options=None, inequality_constraints=None, equality_constraints=None, nonlinear_inequality_constraints=None, fixed_features=None, post_processing_func=None, batch_initial_conditions=None, return_best_only=True, gen_candidates=None, sequential=False, *, ic_generator=None, timeout_sec=None, return_full_tree=False, retry_on_optimization_warning=True, **ic_gen_kwargs)[source]¶

Generate a set of candidates via multi-start optimization.

Parameters:

• acq_function (AcquisitionFunction) – An AcquisitionFunction.

• bounds (Tensor) – A 2 x d tensor of lower and upper bounds for each column of X (if inequality_constraints is provided, these bounds can be -inf and +inf, respectively).

• q (int) – The number of candidates.

• num_restarts (int) – The number of starting points for multistart acquisition function optimization.

• raw_samples (int | None) – The number of samples for initialization. This is required if batch_initial_conditions is not specified.

• options (Dict[str, bool | float | int | str] | None) – Options for candidate generation.

• inequality_constraints (List[Tuple[Tensor, Tensor, float]] | None) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs

• equality_constraints (List[Tuple[Tensor, Tensor, float]] | None) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an equality constraint of the form sum_i (X[indices[i]] * coefficients[i]) = rhs

• nonlinear_inequality_constraints (List[Callable] | None) – A list of callables with that represent non-linear inequality constraints of the form callable(x) >= 0. Each callable is expected to take a (num_restarts) x q x d-dim tensor as an input and return a (num_restarts) x q-dim tensor with the constraint values. The constraints will later be passed to SLSQP. You need to pass in batch_initial_conditions in this case. Using non-linear inequality constraints also requires that batch_limit is set to 1, which will be done automatically if not specified in options.

• fixed_features (Dict[int, float] | None) – A map {feature_index: value} for features that should be fixed to a particular value during generation.

• post_processing_func (Callable[[Tensor], Tensor] | None) – A function that post-processes an optimization result appropriately (i.e., according to round-trip transformations).

• batch_initial_conditions (Tensor | None) – A tensor to specify the initial conditions. Set this if you do not want to use default initialization strategy.

• return_best_only (bool) – If False, outputs the solutions corresponding to all random restart initializations of the optimization.

• gen_candidates (Callable[[Tensor, AcquisitionFunction, Any], Tuple[Tensor, Tensor]] | None) – A callable for generating candidates (and their associated acquisition values) given a tensor of initial conditions and an acquisition function. Other common inputs include lower and upper bounds and a dictionary of options, but refer to the documentation of specific generation functions (e.g gen_candidates_scipy and gen_candidates_torch) for method-specific inputs. Default: gen_candidates_scipy

• sequential (bool) – If False, uses joint optimization, otherwise uses sequential optimization.

• ic_generator (Callable[[qKnowledgeGradient, Tensor, int, int, int, Dict[int, float] | None, Dict[str, bool | float | int] | None, List[Tuple[Tensor, Tensor, float]] | None, List[Tuple[Tensor, Tensor, float]] | None], Tensor | None] | None) – Function for generating initial conditions. Not needed when batch_initial_conditions are provided. Defaults to gen_one_shot_kg_initial_conditions for qKnowledgeGradient acquisition functions and gen_batch_initial_conditions otherwise. Must be specified for nonlinear inequality constraints.

• timeout_sec (float | None) – Max amount of time optimization can run for.

• return_full_tree (bool) –

• retry_on_optimization_warning (bool) – Whether to retry candidate generation with a new set of initial conditions when it fails with an OptimizationWarning.

• ic_gen_kwargs (Any) – Additional keyword arguments passed to function specified by ic_generator

Returns:

A two-element tuple containing

• A tensor of generated candidates. The shape is

  – q x d if return_best_only is True (default) – num_restarts x q x d if return_best_only is False

• a tensor of associated acquisition values. If sequential=False,

  this is a (num_restarts)-dim tensor of joint acquisition values (with explicit restart dimension if return_best_only=False). If sequential=True, this is a q-dim tensor of expected acquisition values conditional on having observed candidates 0,1,…,i-1.

Return type:

Tuple[Tensor, Tensor]

Example

>>> # generate `q=2` candidates jointly using 20 random restarts
>>> # and 512 raw samples
>>> candidates, acq_value = optimize_acqf(qEI, bounds, 2, 20, 512)

>>> generate `q=3` candidates sequentially using 15 random restarts
>>> # and 256 raw samples
>>> qEI = qExpectedImprovement(model, best_f=0.2)
>>> bounds = torch.tensor([[0.], [1.]])
>>> candidates, acq_value_list = optimize_acqf(
>>>     qEI, bounds, 3, 15, 256, sequential=True
>>> )

botorch.optim.optimize.optimize_acqf_cyclic(acq_function, bounds, q, num_restarts, raw_samples=None, options=None, inequality_constraints=None, equality_constraints=None, fixed_features=None, post_processing_func=None, batch_initial_conditions=None, cyclic_options=None, *, ic_generator=None, timeout_sec=None, return_full_tree=False, retry_on_optimization_warning=True, **ic_gen_kwargs)[source]¶

Generate a set of q candidates via cyclic optimization.

Parameters:

• acq_function (AcquisitionFunction) – An AcquisitionFunction

• bounds (Tensor) – A 2 x d tensor of lower and upper bounds for each column of X (if inequality_constraints is provided, these bounds can be -inf and +inf, respectively).

• q (int) – The number of candidates.

• num_restarts (int) – Number of starting points for multistart acquisition function optimization.

• raw_samples (int | None) – Number of samples for initialization. This is required if batch_initial_conditions is not specified.

• options (Dict[str, bool | float | int | str] | None) – Options for candidate generation.

• constraints (equality) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs

• constraints – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) = rhs

• fixed_features (Dict[int, float] | None) – A map {feature_index: value} for features that should be fixed to a particular value during generation.

• post_processing_func (Callable[[Tensor], Tensor] | None) – A function that post-processes an optimization result appropriately (i.e., according to round-trip transformations).

• batch_initial_conditions (Tensor | None) – A tensor to specify the initial conditions. If no initial conditions are provided, the default initialization will be used.

• cyclic_options (Dict[str, bool | float | int | str] | None) – Options for stopping criterion for outer cyclic optimization.

• ic_generator (Callable[[qKnowledgeGradient, Tensor, int, int, int, Dict[int, float] | None, Dict[str, bool | float | int] | None, List[Tuple[Tensor, Tensor, float]] | None, List[Tuple[Tensor, Tensor, float]] | None], Tensor | None] | None) – Function for generating initial conditions. Not needed when batch_initial_conditions are provided. Defaults to gen_one_shot_kg_initial_conditions for qKnowledgeGradient acquisition functions and gen_batch_initial_conditions otherwise. Must be specified for nonlinear inequality constraints.

• timeout_sec (float | None) – Max amount of time optimization can run for.

• return_full_tree (bool) –

• retry_on_optimization_warning (bool) – Whether to retry candidate generation with a new set of initial conditions when it fails with an OptimizationWarning.

• ic_gen_kwargs (Any) – Additional keyword arguments passed to function specified by ic_generator

• inequality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

• equality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

Returns:

A two-element tuple containing

• a q x d-dim tensor of generated candidates.

• a q-dim tensor of expected acquisition values, where the value at

  index i is the acquisition value conditional on having observed all candidates except candidate i.

Return type:

Tuple[Tensor, Tensor]

Example

>>> # generate `q=3` candidates cyclically using 15 random restarts
>>> # 256 raw samples, and 4 cycles
>>>
>>> qEI = qExpectedImprovement(model, best_f=0.2)
>>> bounds = torch.tensor([[0.], [1.]])
>>> candidates, acq_value_list = optimize_acqf_cyclic(
>>>     qEI, bounds, 3, 15, 256, cyclic_options={"maxiter": 4}
>>> )

botorch.optim.optimize.optimize_acqf_list(acq_function_list, bounds, num_restarts, raw_samples=None, options=None, inequality_constraints=None, equality_constraints=None, nonlinear_inequality_constraints=None, fixed_features=None, fixed_features_list=None, post_processing_func=None, ic_generator=None, ic_gen_kwargs=None)[source]¶

Generate a list of candidates from a list of acquisition functions.

The acquisition functions are optimized in sequence, with previous candidates set as X_pending. This is also known as sequential greedy optimization.

Parameters:

• acq_function_list (List[AcquisitionFunction]) – A list of acquisition functions.

• bounds (Tensor) – A 2 x d tensor of lower and upper bounds for each column of X (if inequality_constraints is provided, these bounds can be -inf and +inf, respectively).

• num_restarts (int) – Number of starting points for multistart acquisition function optimization.

• raw_samples (int | None) – Number of samples for initialization. This is required if batch_initial_conditions is not specified.

• options (Dict[str, bool | float | int | str] | None) – Options for candidate generation.

• constraints (equality) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs

• constraints – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) = rhs

• nonlinear_inequality_constraints (List[Callable] | None) – A list of callables with that represent non-linear inequality constraints of the form callable(x) >= 0. Each callable is expected to take a (num_restarts) x q x d-dim tensor as an input and return a (num_restarts) x q-dim tensor with the constraint values. The constraints will later be passed to SLSQP. You need to pass in batch_initial_conditions in this case. Using non-linear inequality constraints also requires that batch_limit is set to 1, which will be done automatically if not specified in options.

• fixed_features (Dict[int, float] | None) – A map {feature_index: value} for features that should be fixed to a particular value during generation.

• fixed_features_list (List[Dict[int, float]] | None) – A list of maps {feature_index: value}. The i-th item represents the fixed_feature for the i-th optimization. If fixed_features_list is provided, optimize_acqf_mixed is invoked.

• post_processing_func (Callable[[Tensor], Tensor] | None) – A function that post-processes an optimization result appropriately (i.e., according to round-trip transformations).

• ic_generator (Callable[[qKnowledgeGradient, Tensor, int, int, int, Dict[int, float] | None, Dict[str, bool | float | int] | None, List[Tuple[Tensor, Tensor, float]] | None, List[Tuple[Tensor, Tensor, float]] | None], Tensor | None] | None) – Function for generating initial conditions. Not needed when batch_initial_conditions are provided. Defaults to gen_one_shot_kg_initial_conditions for qKnowledgeGradient acquisition functions and gen_batch_initial_conditions otherwise. Must be specified for nonlinear inequality constraints.

• ic_gen_kwargs (Dict | None) – Additional keyword arguments passed to function specified by ic_generator

• inequality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

• equality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

Returns:

A two-element tuple containing

• a q x d-dim tensor of generated candidates.

• a q-dim tensor of expected acquisition values, where the value at

  index i is the acquisition value conditional on having observed all candidates except candidate i.

Return type:

Tuple[Tensor, Tensor]

botorch.optim.optimize.optimize_acqf_mixed(acq_function, bounds, q, num_restarts, fixed_features_list, raw_samples=None, options=None, inequality_constraints=None, equality_constraints=None, nonlinear_inequality_constraints=None, post_processing_func=None, batch_initial_conditions=None, ic_generator=None, ic_gen_kwargs=None, **kwargs)[source]¶

Optimize over a list of fixed_features and returns the best solution.

This is useful for optimizing over mixed continuous and discrete domains. For q > 1 this function always performs sequential greedy optimization (with proper conditioning on generated candidates).

Parameters:

• acq_function (AcquisitionFunction) – An AcquisitionFunction

• bounds (Tensor) – A 2 x d tensor of lower and upper bounds for each column of X (if inequality_constraints is provided, these bounds can be -inf and +inf, respectively).

• q (int) – The number of candidates.

• num_restarts (int) – Number of starting points for multistart acquisition function optimization.

• raw_samples (int | None) – Number of samples for initialization. This is required if batch_initial_conditions is not specified.

• fixed_features_list (List[Dict[int, float]]) – A list of maps {feature_index: value}. The i-th item represents the fixed_feature for the i-th optimization.

• options (Dict[str, bool | float | int | str] | None) – Options for candidate generation.

• constraints (equality) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs

• constraints – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) = rhs

• nonlinear_inequality_constraints (List[Callable] | None) – A list of callables with that represent non-linear inequality constraints of the form callable(x) >= 0. Each callable is expected to take a (num_restarts) x q x d-dim tensor as an input and return a (num_restarts) x q-dim tensor with the constraint values. The constraints will later be passed to SLSQP. You need to pass in batch_initial_conditions in this case. Using non-linear inequality constraints also requires that batch_limit is set to 1, which will be done automatically if not specified in options.

• post_processing_func (Callable[[Tensor], Tensor] | None) – A function that post-processes an optimization result appropriately (i.e., according to round-trip transformations).

• batch_initial_conditions (Tensor | None) – A tensor to specify the initial conditions. Set this if you do not want to use default initialization strategy.

• ic_generator (Callable[[qKnowledgeGradient, Tensor, int, int, int, Dict[int, float] | None, Dict[str, bool | float | int] | None, List[Tuple[Tensor, Tensor, float]] | None, List[Tuple[Tensor, Tensor, float]] | None], Tensor | None] | None) – Function for generating initial conditions. Not needed when batch_initial_conditions are provided. Defaults to gen_one_shot_kg_initial_conditions for qKnowledgeGradient acquisition functions and gen_batch_initial_conditions otherwise. Must be specified for nonlinear inequality constraints.

• ic_gen_kwargs (Dict | None) – Additional keyword arguments passed to function specified by ic_generator

• kwargs (Any) – kwargs do nothing. This is provided so that the same arguments can be passed to different acquisition functions without raising an error.

• inequality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

• equality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

Returns:

A two-element tuple containing

• a q x d-dim tensor of generated candidates.

• an associated acquisition value.

Return type:

Tuple[Tensor, Tensor]

botorch.optim.optimize.optimize_acqf_discrete(acq_function, q, choices, max_batch_size=2048, unique=True, **kwargs)[source]¶

Optimize over a discrete set of points using batch evaluation.

For q > 1 this function generates candidates by means of sequential conditioning (rather than joint optimization), since for all but the smalles number of choices the set choices^q of discrete points to evaluate quickly explodes.

Parameters:

• acq_function (AcquisitionFunction) – An AcquisitionFunction.

• q (int) – The number of candidates.

• choices (Tensor) – A num_choices x d tensor of possible choices.

• max_batch_size (int) – The maximum number of choices to evaluate in batch. A large limit can cause excessive memory usage if the model has a large training set.

• unique (bool) – If True return unique choices, o/w choices may be repeated (only relevant if q > 1).

• kwargs (Any) – kwargs do nothing. This is provided so that the same arguments can be passed to different acquisition functions without raising an error.

Returns:

A three-element tuple containing

• a q x d-dim tensor of generated candidates.

• an associated acquisition value.

Return type:

Tuple[Tensor, Tensor]

botorch.optim.optimize.optimize_acqf_discrete_local_search(acq_function, discrete_choices, q, num_restarts=20, raw_samples=4096, inequality_constraints=None, X_avoid=None, batch_initial_conditions=None, max_batch_size=2048, unique=True, **kwargs)[source]¶

Optimize acquisition function over a lattice.

This is useful when d is large and enumeration of the search space isn’t possible. For q > 1 this function always performs sequential greedy optimization (with proper conditioning on generated candidates).

NOTE: While this method supports arbitrary lattices, it has only been thoroughly tested for {0, 1}^d. Consider it to be in alpha stage for the more general case.

Parameters:

• acq_function (AcquisitionFunction) – An AcquisitionFunction

• discrete_choices (List[Tensor]) – A list of possible discrete choices for each dimension. Each element in the list is expected to be a torch tensor.

• q (int) – The number of candidates.

• num_restarts (int) – Number of starting points for multistart acquisition function optimization.

• raw_samples (int) – Number of samples for initialization. This is required if batch_initial_conditions is not specified.

• inequality_constraints (List[Tuple[Tensor, Tensor, float]] | None) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs

• X_avoid (Tensor | None) – An n x d tensor of candidates that we aren’t allowed to pick.

• batch_initial_conditions (Tensor | None) – A tensor of size n x 1 x d to specify the initial conditions. Set this if you do not want to use default initialization strategy.

• max_batch_size (int) – The maximum number of choices to evaluate in batch. A large limit can cause excessive memory usage if the model has a large training set.

• unique (bool) – If True return unique choices, o/w choices may be repeated (only relevant if q > 1).

• kwargs (Any) – kwargs do nothing. This is provided so that the same arguments can be passed to different acquisition functions without raising an error.

Returns:

A two-element tuple containing

• a q x d-dim tensor of generated candidates.

• an associated acquisition value.

Return type:

Tuple[Tensor, Tensor]

Model Fitting Optimization¶

Tools for model fitting.

botorch.optim.fit.fit_gpytorch_mll_scipy(mll, parameters=None, bounds=None, closure=None, closure_kwargs=None, method='L-BFGS-B', options=None, callback=None, timeout_sec=None)[source]¶

Generic scipy.optimized-based fitting routine for GPyTorch MLLs.

The model and likelihood in mll must already be in train mode.

Parameters:

• mll (MarginalLogLikelihood) – MarginalLogLikelihood to be maximized.

• parameters (Dict[str, Tensor] | None) – Optional dictionary of parameters to be optimized. Defaults to all parameters of mll that require gradients.

• bounds (Dict[str, Tuple[float | None, float | None]] | None) – A dictionary of user-specified bounds for parameters. Used to update default parameter bounds obtained from mll.

• closure (Callable[[], Tuple[Tensor, Sequence[Tensor | None]]] | None) – Callable that returns a tensor and an iterable of gradient tensors. Responsible for setting the grad attributes of parameters. If no closure is provided, one will be obtained by calling get_loss_closure_with_grads.

• closure_kwargs (Dict[str, Any] | None) – Keyword arguments passed to closure.

• method (str) – Solver type, passed along to scipy.minimize.

• options (Dict[str, Any] | None) – Dictionary of solver options, passed along to scipy.minimize.

• callback (Callable[[Dict[str, Tensor], OptimizationResult], None] | None) – Optional callback taking parameters and an OptimizationResult as its sole arguments.

• timeout_sec (float | None) – Timeout in seconds after which to terminate the fitting loop (note that timing out can result in bad fits!).

Returns:

The final OptimizationResult.

Return type:

OptimizationResult

botorch.optim.fit.fit_gpytorch_mll_torch(mll, parameters=None, bounds=None, closure=None, closure_kwargs=None, step_limit=None, stopping_criterion=<class 'botorch.utils.types.DEFAULT'>, optimizer=<class 'torch.optim.adam.Adam'>, scheduler=None, callback=None, timeout_sec=None)[source]¶

Generic torch.optim-based fitting routine for GPyTorch MLLs.

Parameters:

• mll (MarginalLogLikelihood) – MarginalLogLikelihood to be maximized.

• parameters (Dict[str, Tensor] | None) – Optional dictionary of parameters to be optimized. Defaults to all parameters of mll that require gradients.

• bounds (Dict[str, Tuple[float | None, float | None]] | None) – A dictionary of user-specified bounds for parameters. Used to update default parameter bounds obtained from mll.

• closure (Callable[[], Tuple[Tensor, Sequence[Tensor | None]]] | None) – Callable that returns a tensor and an iterable of gradient tensors. Responsible for setting the grad attributes of parameters. If no closure is provided, one will be obtained by calling get_loss_closure_with_grads.

• closure_kwargs (Dict[str, Any] | None) – Keyword arguments passed to closure.

• step_limit (int | None) – Optional upper bound on the number of optimization steps.

• stopping_criterion (Callable[[Tensor], bool] | None) – A StoppingCriterion for the optimization loop.

• optimizer (Optimizer | Callable[[...], Optimizer]) – A torch.optim.Optimizer instance or a factory that takes a list of parameters and returns an Optimizer instance.

• scheduler (_LRScheduler | Callable[[...], _LRScheduler] | None) – A torch.optim.lr_scheduler._LRScheduler instance or a factory that takes an Optimizer instance and returns an _LRSchedule.

• callback (Callable[[Dict[str, Tensor], OptimizationResult], None] | None) – Optional callback taking parameters and an OptimizationResult as its sole arguments.

• timeout_sec (float | None) – Timeout in seconds after which to terminate the fitting loop (note that timing out can result in bad fits!).

Returns:

The final OptimizationResult.

Return type:

OptimizationResult

botorch.optim.fit.fit_gpytorch_scipy(mll, bounds=None, method='L-BFGS-B', options=None, track_iterations=False, approx_mll=False, scipy_objective=<function _scipy_objective_and_grad>, module_to_array_func=<function module_to_array>, module_from_array_func=<function set_params_with_array>, **kwargs)[source]¶

Legacy method for scipy-based fitting of gpytorch models.

The model and likelihood in mll must already be in train mode. This method requires that the model has train_inputs and train_targets.

Parameters:

• mll (MarginalLogLikelihood) – MarginalLogLikelihood to be maximized.

• bounds (Dict[str, Tuple[float | None, float | None]] | None) – A dictionary mapping parameter names to tuples of lower and upper bounds.

• method (str) – Solver type, passed along to scipy.optimize.minimize.

• options (Dict[str, Any] | None) – Dictionary of solver options, passed along to scipy.optimize.minimize.

• approx_mll (bool) – If True, use gpytorch’s approximate MLL computation. This is disabled by default since the stochasticity is an issue for determistic optimizers). Enabling this is only recommended when working with large training data sets (n>2000).

• track_iterations (bool) –

• scipy_objective (Callable[[ndarray, MarginalLogLikelihood, Dict[str, TorchAttr]], Tuple[float, ndarray]]) –

• module_to_array_func (Callable[[Module, Dict[str, Tuple[float | None, float | None]] | None, Set[str] | None], Tuple[ndarray, Dict[str, TorchAttr], ndarray | None]]) –

• module_from_array_func (Callable[[Module, ndarray, Dict[str, TorchAttr]], Module]) –

• kwargs (Any) –

Returns:

2-element tuple containing - MarginalLogLikelihood with parameters optimized in-place. - Dictionary with the following key/values: “fopt”: Best mll value. “wall_time”: Wall time of fitting. “iterations”: List of OptimizationResult objects with information on each iteration. If track_iterations is False, will be empty. “OptimizeResult”: The result returned by scipy.optim.minimize.

Return type:

Tuple[MarginalLogLikelihood, Dict[str, float | List[OptimizationResult]]]

botorch.optim.fit.fit_gpytorch_torch(mll, bounds=None, optimizer_cls=<class 'torch.optim.adam.Adam'>, options=None, track_iterations=False, approx_mll=False)[source]¶

Legacy method for torch-based fitting of gpytorch models.

The model and likelihood in mll must already be in train mode. Note: this method requires that the model has train_inputs and train_targets.

Parameters:

• mll (MarginalLogLikelihood) – MarginalLogLikelihood to be maximized.

• bounds (Dict[str, Tuple[float | None, float | None]] | None) – An optional dictionary mapping parameter names to tuples of lower and upper bounds. Bounds specified here take precedence over bounds on the same parameters specified in the constraints registered with the module.

• optimizer_cls (Optimizer) – Torch optimizer to use. Must not require a closure.

• options (Dict[str, Any] | None) – options for model fitting. Relevant options will be passed to the optimizer_cls. Additionally, options can include: “disp” to specify whether to display model fitting diagnostics and “maxiter” to specify the maximum number of iterations.

• track_iterations (bool) –

• approx_mll (bool) –

Returns:

2-element tuple containing - mll with parameters optimized in-place. - Dictionary with the following key/values: “fopt”: Best mll value. “wall_time”: Wall time of fitting. “iterations”: List of OptimizationResult objects with information on each iteration. If track_iterations is False, will be empty.

Return type:

Tuple[MarginalLogLikelihood, Dict[str, float | List[OptimizationResult]]]

Example

>>> gp = SingleTaskGP(train_X, train_Y)
>>> mll = ExactMarginalLogLikelihood(gp.likelihood, gp)
>>> mll.train()
>>> fit_gpytorch_torch(mll)
>>> mll.eval()

Initialization Helpers¶

References

botorch.optim.initializers.transform_constraints(constraints, q, d)[source]¶

Transform constraints to sample from a d*q-dimensional space instead of a d-dimensional state.

This function assumes that constraints are the same for each input batch, and broadcasts the constraints accordingly to the input batch shape.

Parameters:

• constraints (List[Tuple[Tensor, Tensor, float]] | None) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an (in-)equality constraint of the form sum_i (X[indices[i]] * coefficients[i]) (>)= rhs. If indices is a 2-d Tensor, this supports specifying constraints across the points in the q-batch (inter-point constraints). If None, this function is a nullop and simply returns None.

• q (int) – Size of the q-batch.

• d (int) – Dimensionality of the problem.

Returns:

List of transformed constraints.

Return type:

List[Tuple[Tensor, Tensor, float]]

botorch.optim.initializers.transform_intra_point_constraint(constraint, d, q)[source]¶

Transforms an intra-point/pointwise constraint from d-dimensional space to a d*q-dimesional space.

Parameters:

• constraints – A list of tuples (indices, coefficients, rhs), with each tuple encoding an (in-)equality constraint of the form sum_i (X[indices[i]] * coefficients[i]) (>)= rhs. Here indices must be one-dimensional, and the constraint is applied to all points within the q-batch.

• d (int) – Dimensionality of the problem.

• constraint (Tuple[Tensor, Tensor, float]) –

• q (int) –

Raises:

ValueError – If indices in the constraints are larger than the dimensionality d of the problem.

Returns:

List of transformed constraints.

Return type:

List[Tuple[Tensor, Tensor, float]]

botorch.optim.initializers.transform_inter_point_constraint(constraint, d)[source]¶

Transforms an inter-point constraint from d-dimensional space to a d*q dimesional space.

Parameters:

• constraints – A list of tuples (indices, coefficients, rhs), with each tuple encoding an (in-)equality constraint of the form sum_i (X[indices[i]] * coefficients[i]) (>)= rhs. indices must be a 2-d Tensor, where in each row indices[i] = (k_i, l_i) the first index k_i corresponds to the k_i-th element of the q-batch and the second index l_i corresponds to the l_i-th feature of that element.

• constraint (Tuple[Tensor, Tensor, float]) –

• d (int) –

Raises:

ValueError – If indices in the constraints are larger than the dimensionality d of the problem.

Returns:

Transformed constraint.

Return type:

List[Tuple[Tensor, Tensor, float]]

botorch.optim.initializers.sample_q_batches_from_polytope(n, q, bounds, n_burnin, thinning, seed, inequality_constraints=None, equality_constraints=None)[source]¶

Samples n q-baches from a polytope of dimension d.

Parameters:

• n (int) – Number of q-batches to sample.

• q (int) – Number of samples per q-batch

• bounds (Tensor) – A 2 x d tensor of lower and upper bounds for each column of X.

• n_burnin (int) – The number of burn-in samples for the Markov chain sampler. thinning: The amount of thinning (number of steps to take between returning samples).

• seed (int) – The random seed.

• constraints (equality) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs.

• constraints – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) = rhs.

• thinning (int) –

• inequality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

• equality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

Returns:

A n x q x d-dim tensor of samples.

Return type:

Tensor

botorch.optim.initializers.gen_batch_initial_conditions(acq_function, bounds, q, num_restarts, raw_samples, fixed_features=None, options=None, inequality_constraints=None, equality_constraints=None, generator=None)[source]¶

Generate a batch of initial conditions for random-restart optimziation.

TODO: Support t-batches of initial conditions.

Parameters:

• acq_function (AcquisitionFunction) – The acquisition function to be optimized.

• bounds (Tensor) – A 2 x d tensor of lower and upper bounds for each column of X.

• q (int) – The number of candidates to consider.

• num_restarts (int) – The number of starting points for multistart acquisition function optimization.

• raw_samples (int) – The number of raw samples to consider in the initialization heuristic. Note: if sample_around_best is True (the default is False), then 2 * raw_samples samples are used.

• fixed_features (Dict[int, float] | None) – A map {feature_index: value} for features that should be fixed to a particular value during generation.

• options (Dict[str, bool | float | int] | None) – Options for initial condition generation. For valid options see initialize_q_batch and initialize_q_batch_nonneg. If options contains a nonnegative=True entry, then acq_function is assumed to be non-negative (useful when using custom acquisition functions). In addition, an “init_batch_limit” option can be passed to specify the batch limit for the initialization. This is useful for avoiding memory limits when computing the batch posterior over raw samples.

• constraints (equality) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs.

• constraints – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) = rhs.

• generator (Callable[[int, int, int], Tensor] | None) – Callable for generating samples that are then further processed. It receives n, q and seed as arguments and returns a tensor of shape n x q x d.

• inequality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

• equality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

Returns:

A num_restarts x q x d tensor of initial conditions.

Return type:

Tensor

Example

>>> qEI = qExpectedImprovement(model, best_f=0.2)
>>> bounds = torch.tensor([[0.], [1.]])
>>> Xinit = gen_batch_initial_conditions(
>>>     qEI, bounds, q=3, num_restarts=25, raw_samples=500
>>> )

botorch.optim.initializers.gen_one_shot_kg_initial_conditions(acq_function, bounds, q, num_restarts, raw_samples, fixed_features=None, options=None, inequality_constraints=None, equality_constraints=None)[source]¶

Generate a batch of smart initializations for qKnowledgeGradient.

This function generates initial conditions for optimizing one-shot KG using the maximizer of the posterior objective. Intutively, the maximizer of the fantasized posterior will often be close to a maximizer of the current posterior. This function uses that fact to generate the initital conditions for the fantasy points. Specifically, a fraction of 1 - frac_random (see options) is generated by sampling from the set of maximizers of the posterior objective (obtained via random restart optimization) according to a softmax transformation of their respective values. This means that this initialization strategy internally solves an acquisition function maximization problem. The remaining frac_random fantasy points as well as all q candidate points are chosen according to the standard initialization strategy in gen_batch_initial_conditions.

Parameters:

• acq_function (qKnowledgeGradient) – The qKnowledgeGradient instance to be optimized.

• bounds (Tensor) – A 2 x d tensor of lower and upper bounds for each column of task features.

• q (int) – The number of candidates to consider.

• num_restarts (int) – The number of starting points for multistart acquisition function optimization.

• raw_samples (int) – The number of raw samples to consider in the initialization heuristic.

• fixed_features (Dict[int, float] | None) – A map {feature_index: value} for features that should be fixed to a particular value during generation.

• options (Dict[str, bool | float | int] | None) – Options for initial condition generation. These contain all settings for the standard heuristic initialization from gen_batch_initial_conditions. In addition, they contain frac_random (the fraction of fully random fantasy points), num_inner_restarts and raw_inner_samples (the number of random restarts and raw samples for solving the posterior objective maximization problem, respectively) and eta (temperature parameter for sampling heuristic from posterior objective maximizers).

• constraints (equality) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs.

• constraints – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) = rhs.

• inequality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

• equality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

Returns:

A num_restarts x q’ x d tensor that can be used as initial conditions for optimize_acqf(). Here q’ = q + num_fantasies is the total number of points (candidate points plus fantasy points).

Return type:

Tensor | None

Example

>>> qKG = qKnowledgeGradient(model, num_fantasies=64)
>>> bounds = torch.tensor([[0., 0.], [1., 1.]])
>>> Xinit = gen_one_shot_kg_initial_conditions(
>>>     qKG, bounds, q=3, num_restarts=10, raw_samples=512,
>>>     options={"frac_random": 0.25},
>>> )

botorch.optim.initializers.gen_value_function_initial_conditions(acq_function, bounds, num_restarts, raw_samples, current_model, fixed_features=None, options=None)[source]¶

Generate a batch of smart initializations for optimizing the value function of qKnowledgeGradient.

This function generates initial conditions for optimizing the inner problem of KG, i.e. its value function, using the maximizer of the posterior objective. Intutively, the maximizer of the fantasized posterior will often be close to a maximizer of the current posterior. This function uses that fact to generate the initital conditions for the fantasy points. Specifically, a fraction of 1 - frac_random (see options) of raw samples is generated by sampling from the set of maximizers of the posterior objective (obtained via random restart optimization) according to a softmax transformation of their respective values. This means that this initialization strategy internally solves an acquisition function maximization problem. The remaining raw samples are generated using draw_sobol_samples. All raw samples are then evaluated, and the initial conditions are selected according to the standard initialization strategy in ‘initialize_q_batch’ individually for each inner problem.

Parameters:

• acq_function (AcquisitionFunction) – The value function instance to be optimized.

• bounds (Tensor) – A 2 x d tensor of lower and upper bounds for each column of task features.

• num_restarts (int) – The number of starting points for multistart acquisition function optimization.

• raw_samples (int) – The number of raw samples to consider in the initialization heuristic.

• current_model (Model) – The model of the KG acquisition function that was used to generate the fantasy model of the value function.

• fixed_features (Dict[int, float] | None) – A map {feature_index: value} for features that should be fixed to a particular value during generation.

• options (Dict[str, bool | float | int] | None) – Options for initial condition generation. These contain all settings for the standard heuristic initialization from gen_batch_initial_conditions. In addition, they contain frac_random (the fraction of fully random fantasy points), num_inner_restarts and raw_inner_samples (the number of random restarts and raw samples for solving the posterior objective maximization problem, respectively) and eta (temperature parameter for sampling heuristic from posterior objective maximizers).

Returns:

A num_restarts x batch_shape x q x d tensor that can be used as initial conditions for optimize_acqf(). Here batch_shape is the batch shape of value function model.

Return type:

Tensor

Example

>>> fant_X = torch.rand(5, 1, 2)
>>> fantasy_model = model.fantasize(fant_X, SobolQMCNormalSampler(16))
>>> value_function = PosteriorMean(fantasy_model)
>>> bounds = torch.tensor([[0., 0.], [1., 1.]])
>>> Xinit = gen_value_function_initial_conditions(
>>>     value_function, bounds, num_restarts=10, raw_samples=512,
>>>     options={"frac_random": 0.25},
>>> )

botorch.optim.initializers.initialize_q_batch(X, Y, n, eta=1.0)[source]¶

Heuristic for selecting initial conditions for candidate generation.

This heuristic selects points from X (without replacement) with probability proportional to exp(eta * Z), where Z = (Y - mean(Y)) / std(Y) and eta is a temperature parameter.

When using an acquisiton function that is non-negative and possibly zero over large areas of the feature space (e.g. qEI), you should use initialize_q_batch_nonneg instead.

Parameters:

• X (Tensor) – A b x batch_shape x q x d tensor of b - batch_shape samples of q-batches from a d`-dim feature space. Typically, these are generated using qMC sampling.

• Y (Tensor) – A tensor of b x batch_shape outcomes associated with the samples. Typically, this is the value of the batch acquisition function to be maximized.

• n (int) – The number of initial condition to be generated. Must be less than b.

• eta (float) – Temperature parameter for weighting samples.

Returns:

A n x batch_shape x q x d tensor of n - batch_shape q-batch initial conditions, where each batch of n x q x d samples is selected independently.

Return type:

Tensor

Example

>>> # To get `n=10` starting points of q-batch size `q=3`
>>> # for model with `d=6`:
>>> qUCB = qUpperConfidenceBound(model, beta=0.1)
>>> Xrnd = torch.rand(500, 3, 6)
>>> Xinit = initialize_q_batch(Xrnd, qUCB(Xrnd), 10)

botorch.optim.initializers.initialize_q_batch_nonneg(X, Y, n, eta=1.0, alpha=0.0001)[source]¶

Heuristic for selecting initial conditions for non-neg. acquisition functions.

This function is similar to initialize_q_batch, but designed specifically for acquisition functions that are non-negative and possibly zero over large areas of the feature space (e.g. qEI). All samples for which Y < alpha * max(Y) will be ignored (assuming that Y contains at least one positive value).

Parameters:

• X (Tensor) – A b x q x d tensor of b samples of q-batches from a d-dim. feature space. Typically, these are generated using qMC.

• Y (Tensor) – A tensor of b outcomes associated with the samples. Typically, this is the value of the batch acquisition function to be maximized.

• n (int) – The number of initial condition to be generated. Must be less than b.

• eta (float) – Temperature parameter for weighting samples.

• alpha (float) – The threshold (as a fraction of the maximum observed value) under which to ignore samples. All input samples for which Y < alpha * max(Y) will be ignored.

Returns:

A n x q x d tensor of n q-batch initial conditions.

Return type:

Tensor

Example

>>> # To get `n=10` starting points of q-batch size `q=3`
>>> # for model with `d=6`:
>>> qEI = qExpectedImprovement(model, best_f=0.2)
>>> Xrnd = torch.rand(500, 3, 6)
>>> Xinit = initialize_q_batch(Xrnd, qEI(Xrnd), 10)

botorch.optim.initializers.sample_points_around_best(acq_function, n_discrete_points, sigma, bounds, best_pct=5.0, subset_sigma=0.1, prob_perturb=None)[source]¶

Find best points and sample nearby points.

Parameters:

• acq_function (AcquisitionFunction) – The acquisition function.

• n_discrete_points (int) – The number of points to sample.

• sigma (float) – The standard deviation of the additive gaussian noise for perturbing the best points.

• bounds (Tensor) – A 2 x d-dim tensor containing the bounds.

• best_pct (float) – The percentage of best points to perturb.

• subset_sigma (float) – The standard deviation of the additive gaussian noise for perturbing a subset of dimensions of the best points.

• prob_perturb (float | None) – The probability of perturbing each dimension.

Returns:

An optional n_discrete_points x d-dim tensor containing the

sampled points. This is None if no baseline points are found.

Return type:

Tensor | None

botorch.optim.initializers.sample_truncated_normal_perturbations(X, n_discrete_points, sigma, bounds, qmc=True)[source]¶

Sample points around X.

Sample perturbed points around X such that the added perturbations are sampled from N(0, sigma^2 I) and truncated to be within [0,1]^d.

Parameters:

• X (Tensor) – A n x d-dim tensor starting points.

• n_discrete_points (int) – The number of points to sample.

• sigma (float) – The standard deviation of the additive gaussian noise for perturbing the points.

• bounds (Tensor) – A 2 x d-dim tensor containing the bounds.

• qmc (bool) – A boolean indicating whether to use qmc.

Returns:

A n_discrete_points x d-dim tensor containing the sampled points.

Return type:

Tensor

botorch.optim.initializers.sample_perturbed_subset_dims(X, bounds, n_discrete_points, sigma=0.1, qmc=True, prob_perturb=None)[source]¶

Sample around X by perturbing a subset of the dimensions.

By default, dimensions are perturbed with probability equal to min(20 / d, 1). As shown in [Regis], perturbing a small number of dimensions can be beneificial. The perturbations are sampled from N(0, sigma^2 I) and truncated to be within [0,1]^d.

Parameters:

• X (Tensor) – A n x d-dim tensor starting points. X must be normalized to be within [0, 1]^d.

• bounds (Tensor) – The bounds to sample perturbed values from

• n_discrete_points (int) – The number of points to sample.

• sigma (float) – The standard deviation of the additive gaussian noise for perturbing the points.

• qmc (bool) – A boolean indicating whether to use qmc.

• prob_perturb (float | None) – The probability of perturbing each dimension. If omitted, defaults to min(20 / d, 1).

Returns:

A n_discrete_points x d-dim tensor containing the sampled points.

Return type:

Tensor

Stopping Criteria¶

class botorch.optim.stopping.ExpMAStoppingCriterion(maxiter=10000, minimize=True, n_window=10, eta=1.0, rel_tol=1e-05)[source]¶

Bases: StoppingCriterion

Exponential moving average stopping criterion.

Computes an exponentially weighted moving average over window length n_window and checks whether the relative decrease in this moving average between steps is less than a provided tolerance level. That is, in iteration i, it computes

v[i,j] := fvals[i - n_window + j] * w[j]

for all j = 0, …, n_window, where w[j] = exp(-eta * (1 - j / n_window)). Letting ma[i] := sum_j(v[i,j]), the criterion evaluates to True whenever

(ma[i-1] - ma[i]) / abs(ma[i-1]) < rel_tol (if minimize=True) (ma[i] - ma[i-1]) / abs(ma[i-1]) < rel_tol (if minimize=False)

Exponential moving average stopping criterion.

Parameters:

• maxiter (int) – Maximum number of iterations.

• minimize (bool) – If True, assume minimization.

• n_window (int) – The size of the exponential moving average window.

• eta (float) – The exponential decay factor in the weights.

• rel_tol (float) – Relative tolerance for termination.

evaluate(fvals)[source]¶

Evaluate the stopping criterion.

Parameters:

fvals (Tensor) – tensor containing function values for the current iteration. If fvals contains more than one element, then the stopping criterion is evaluated element-wise and True is returned if the stopping criterion is true for all elements.

Return type:

bool

TODO: add support for utilizing gradient information

Returns:

Stopping indicator (if True, stop the optimziation).

Parameters:

fvals (Tensor) –

Return type:

bool

Core¶

Core methods for building closures in torch and interfacing with numpy.

class botorch.optim.closures.core.ForwardBackwardClosure(forward, parameters, backward=<function Tensor.backward>, reducer=<built-in method sum of type object>, callback=None, context_manager=None)[source]¶

Bases: object

Wrapper for fused forward and backward closures.

Initializes a ForwardBackwardClosure instance.

Parameters:

• closure – Callable that returns a tensor.

• parameters (Dict[str, Tensor]) – A dictionary of tensors whose grad fields are to be returned.

• backward (Callable[[Tensor], None]) – Callable that takes the (reduced) output of forward and sets the grad attributes of tensors in parameters.

• reducer (Optional[Callable[[Tensor], Tensor]]) – Optional callable used to reduce the output of the forward pass.

• callback (Optional[Callable[[Tensor, Sequence[Optional[Tensor]]], None]]) – Optional callable that takes the reduced output of forward and the gradients of parameters as positional arguments.

• context_manager (Callable) – A ContextManager used to wrap each forward-backward call. When passed as None, context_manager defaults to a zero_grad_ctx that zeroes the gradients of parameters upon entry.

• forward (Callable[[], Tensor]) –

class botorch.optim.closures.core.NdarrayOptimizationClosure(closure, parameters, as_array=None, as_tensor=<built-in method as_tensor of type object>, get_state=None, set_state=None, fill_value=0.0, persistent=True)[source]¶

Bases: object

Adds stateful behavior and a numpy.ndarray-typed API to a closure with an expected return type Tuple[Tensor, Union[Tensor, Sequence[Optional[Tensor]]]].

Initializes a NdarrayOptimizationClosure instance.

Parameters:

• closure (Callable[[], Tuple[Tensor, Sequence[Optional[Tensor]]]]) – A ForwardBackwardClosure instance.

• parameters (Dict[str, Tensor]) – A dictionary of tensors representing the closure’s state. Expected to correspond with the first len(parameters) optional gradient tensors returned by closure.

• as_array (Callable[[Tensor], ndarray]) – Callable used to convert tensors to ndarrays.

• as_tensor (Callable[[ndarray], Tensor]) – Callable used to convert ndarrays to tensors.

• get_state (Callable[[], ndarray]) – Callable that returns the closure’s state as an ndarray. When passed as None, defaults to calling get_tensors_as_ndarray_1d on closure.parameters while passing as_array (if given by the user).

• set_state (Callable[[ndarray], None]) – Callable that takes a 1-dimensional ndarray and sets the closure’s state. When passed as None, set_state defaults to calling set_tensors_from_ndarray_1d with closure.parameters and a given ndarray while passing as_tensor.

• fill_value (float) – Fill value for parameters whose gradients are None. In most cases, fill_value should either be zero or NaN.

• persistent (bool) – Boolean specifying whether an ndarray should be retained as a persistent buffer for gradients.

property state: ndarray¶

Model Fitting Closures¶

Utilities for building model-based closures.

botorch.optim.closures.model_closures.get_loss_closure(mll, data_loader=None, **kwargs)[source]¶

Public API for GetLossClosure dispatcher.

This method, and the dispatcher that powers it, acts as a clearing house for factory functions that define how mll is evaluated.

Users may specify custom evaluation routines by registering a factory function with GetLossClosure. These factories should be registered using the type signature

Type[MarginalLogLikeLihood], Type[Likelihood], Type[Model], Type[DataLoader].

The final argument, Type[DataLoader], is optional. Evaluation routines that obtain training data from, e.g., mll.model should register this argument as type(None).

Parameters:

• mll (MarginalLogLikelihood) – A MarginalLogLikelihood instance whose negative defines the loss.

• data_loader (DataLoader | None) – An optional DataLoader instance for cases where training data is passed in rather than obtained from mll.model.

• kwargs (Any) –

Returns:

A closure that takes zero positional arguments and returns the negated value of mll.

Return type:

Callable[[], Tensor]

botorch.optim.closures.model_closures.get_loss_closure_with_grads(mll, parameters, data_loader=None, backward=<function Tensor.backward>, reducer=<method 'sum' of 'torch._C._TensorBase' objects>, context_manager=None, **kwargs)[source]¶

Public API for GetLossClosureWithGrads dispatcher.

In most cases, this method simply adds a backward pass to a loss closure obtained by calling get_loss_closure. For further details, see get_loss_closure.

Parameters:

• mll (MarginalLogLikelihood) – A MarginalLogLikelihood instance whose negative defines the loss.

• parameters (Dict[str, Tensor]) – A dictionary of tensors whose grad fields are to be returned.

• reducer (Callable[[Tensor], Tensor] | None) – Optional callable used to reduce the output of the forward pass.

• data_loader (DataLoader | None) – An optional DataLoader instance for cases where training data is passed in rather than obtained from mll.model.

• context_manager (Callable | None) – An optional ContextManager used to wrap each forward-backward pass. Defaults to a zero_grad_ctx that zeroes the gradients of parameters upon entry. None may be passed as an alias for nullcontext.

• backward (Callable[[Tensor], None]) –

• kwargs (Any) –

Returns:

A closure that takes zero positional arguments and returns the reduced and negated value of mll along with the gradients of parameters.

Return type:

Callable[[], Tuple[Tensor, Tuple[Tensor, …]]]

General Optimization Utilities¶

General-purpose optimization utilities.

Acquisition Optimization Utilities¶

Utilities for maximizing acquisition functions.

botorch.optim.utils.acquisition_utils.columnwise_clamp(X, lower=None, upper=None, raise_on_violation=False)[source]¶

Clamp values of a Tensor in column-wise fashion (with support for t-batches).

This function is useful in conjunction with optimizers from the torch.optim package, which don’t natively handle constraints. If you apply this after a gradient step you can be fancy and call it “projected gradient descent”. This funtion is also useful for post-processing candidates generated by the scipy optimizer that satisfy bounds only up to numerical accuracy.

Parameters:

• X (Tensor) – The b x n x d input tensor. If 2-dimensional, b is assumed to be 1.

• lower (float | Tensor | None) – The column-wise lower bounds. If scalar, apply bound to all columns.

• upper (float | Tensor | None) – The column-wise upper bounds. If scalar, apply bound to all columns.

• raise_on_violation (bool) – If True, raise an exception when the elments in X are out of the specified bounds (up to numerical accuracy). This is useful for post-processing candidates generated by optimizers that satisfy imposed bounds only up to numerical accuracy.

Returns:

The clamped tensor.

Return type:

Tensor

botorch.optim.utils.acquisition_utils.fix_features(X, fixed_features=None)[source]¶

Fix feature values in a Tensor.

The fixed features will have zero gradient in downstream calculations.

Parameters:

• X (Tensor) – input Tensor with shape … x p, where p is the number of features

• fixed_features (Dict[int, float | None] | None) – A dictionary with keys as column indices and values equal to what the feature should be set to in X. If the value is None, that column is just considered fixed. Keys should be in the range [0, p - 1].

Returns:

The tensor X with fixed features.

Return type:

Tensor

botorch.optim.utils.acquisition_utils.get_X_baseline(acq_function)[source]¶

Extract X_baseline from an acquisition function.

This tries to find the baseline set of points. First, this checks if the acquisition function has an X_baseline attribute. If it does not, then this method attempts to use the model’s train_inputs as X_baseline.

Parameters:

acq_function (AcquisitionFunction) – The acquisition function.

Return type:

Tensor | None

Returns

An optional n x d-dim tensor of baseline points. This is None if no

baseline points are found.

Model Fitting Utilities¶

Utilities for fitting and manipulating models.

class botorch.optim.utils.model_utils.TorchAttr(shape, dtype, device)[source]¶

Bases: tuple

Create new instance of TorchAttr(shape, dtype, device)

Parameters:

• shape (Size) –

• dtype (dtype) –

• device (device) –

shape: Size¶

Alias for field number 0

dtype: dtype¶

Alias for field number 1

device: device¶

Alias for field number 2

botorch.optim.utils.model_utils.get_data_loader(model, batch_size=1024, **kwargs)[source]¶

Parameters:

• model (GPyTorchModel) –

• batch_size (int) –

• kwargs (Any) –

Return type:

DataLoader

botorch.optim.utils.model_utils.get_parameters(module, requires_grad=None, name_filter=None)[source]¶

Helper method for obtaining a module’s parameters and their respective ranges.

Parameters:

• module (Module) – The target module from which parameters are to be extracted.

• requires_grad (bool | None) – Optional Boolean used to filter parameters based on whether or not their require_grad attribute matches the user provided value.

• name_filter (Callable[[str], bool] | None) – Optional Boolean function used to filter parameters by name.

Returns:

A dictionary of parameters.

Return type:

Dict[str, Tensor]

botorch.optim.utils.model_utils.get_parameters_and_bounds(module, requires_grad=None, name_filter=None, default_bounds=(-inf, inf))[source]¶

Helper method for obtaining a module’s parameters and their respective ranges.

Parameters:

• module (Module) – The target module from which parameters are to be extracted.

• name_filter (Callable[[str], bool] | None) – Optional Boolean function used to filter parameters by name.

• requires_grad (bool | None) – Optional Boolean used to filter parameters based on whether or not their require_grad attribute matches the user provided value.

• default_bounds (Tuple[float, float]) – Default lower and upper bounds for constrained parameters with None typed bounds.

Returns:

A dictionary of parameters and a dictionary of parameter bounds.

Return type:

Tuple[Dict[str, Tensor], Dict[str, Tuple[float | None, float | None]]]

botorch.optim.utils.model_utils.get_name_filter(patterns)[source]¶

Returns a binary function that filters strings (or iterables whose first element is a string) according to a bank of excluded patterns. Typically, used in conjunction with generators such as module.named_parameters().

Parameters:

patterns (Iterator[Union[Pattern, str]]) – A collection of regular expressions or strings that define the set of names to be excluded.

Returns:

A binary function indicating whether or not an item should be filtered.

Return type:

Callable[[Union[str, Tuple[str, Any, …]]], bool]

botorch.optim.utils.model_utils.sample_all_priors(model, max_retries=100)[source]¶

Sample from hyperparameter priors (in-place).

Parameters:

• model (GPyTorchModel) – A GPyTorchModel.

• max_retries (int) –

Return type:

None

Numpy - Torch Conversion Tools¶

Utilities for interfacing Numpy and Torch.

botorch.optim.utils.numpy_utils.as_ndarray(values, dtype=None, inplace=True)[source]¶

Helper for going from torch.Tensor to numpy.ndarray.

Parameters:

• values (Tensor) – Tensor to be converted to ndarray.

• dtype (dtype | None) – Optional numpy.dtype for the converted tensor.

• inplace (bool) – Boolean indicating whether memory should be shared if possible.

Returns:

An ndarray with the same data as values.

Return type:

ndarray

botorch.optim.utils.numpy_utils.get_tensors_as_ndarray_1d(tensors, out=None, dtype=None, as_array=<function as_ndarray>)[source]¶

Parameters:

• tensors (Iterator[Tensor] | Dict[str, Tensor]) –

• out (ndarray | None) –

• dtype (dtype | str | None) –

• as_array (Callable[[Tensor], ndarray]) –

Return type:

ndarray

botorch.optim.utils.numpy_utils.set_tensors_from_ndarray_1d(tensors, array, as_tensor=<built-in method as_tensor of type object>)[source]¶

Sets the values of one more tensors based off of a vector of assignments.

Parameters:

• tensors (Iterator[Tensor] | Dict[str, Tensor]) –

• array (ndarray) –

• as_tensor (Callable[[ndarray], Tensor]) –

Return type:

None

botorch.optim.utils.numpy_utils.get_bounds_as_ndarray(parameters, bounds)[source]¶

Helper method for converting bounds into an ndarray.

Parameters:

• parameters (Dict[str, Tensor]) – A dictionary of parameters.

• bounds (Dict[str, Tuple[float | Tensor | None, float | Tensor | None]]) – A dictionary of (optional) lower and upper bounds.

Returns:

An ndarray of bounds.

Return type:

ndarray | None

Optimization with Timeouts¶

botorch.optim.utils.timeout.minimize_with_timeout(fun, x0, args=(), method=None, jac=None, hess=None, hessp=None, bounds=None, constraints=(), tol=None, callback=None, options=None, timeout_sec=None)[source]¶

Wrapper around scipy.optimize.minimize to support timeout.

This method calls scipy.optimize.minimize with all arguments forwarded verbatim. The only difference is that if provided a timeout_sec argument, it will automatically stop the optimziation after the timeout is reached.

Internally, this is achieved by automatically constructing a wrapper callback method that is injected to the scipy.optimize.minimize call and that keeps track of the runtime and the optimization variables at the current iteration.

Parameters:

• fun (Callable[[np.ndarray, *Any], float]) –

• x0 (np.ndarray) –

• args (Tuple[Any, ...]) –

• method (Optional[str]) –

• jac (Optional[Union[str, Callable, bool]]) –

• hess (Optional[Union[str, Callable, optimize.HessianUpdateStrategy]]) –

• hessp (Optional[Callable]) –

• bounds (Optional[Union[Sequence[Tuple[float, float]], optimize.Bounds]]) –

• tol (Optional[float]) –

• callback (Optional[Callable]) –

• options (Optional[Dict[str, Any]]) –

• timeout_sec (Optional[float]) –

Return type:

optimize.OptimizeResult

Numpy - Torch Conversion Tools (OLD)¶

A converter that simplifies using numpy-based optimizers with generic torch nn.Module classes. This enables using a scipy.optim.minimize optimizer for optimizing module parameters.

botorch.optim.numpy_converter.module_to_array(module, bounds=None, exclude=None)[source]¶

Extract named parameters from a module into a numpy array.

Only extracts parameters with requires_grad, since it is meant for optimizing.

Parameters:

• module (Module) – A module with parameters. May specify parameter constraints in a named_parameters_and_constraints method.

• bounds (Dict[str, Tuple[float | None, float | None]] | None) – A dictionary mapping parameter names t lower and upper bounds. of lower and upper bounds. Bounds specified here take precedence over bounds on the same parameters specified in the constraints registered with the module.

• exclude (Set[str] | None) – A list of parameter names that are to be excluded from extraction.

Returns:

3-element tuple containing - The parameter values as a numpy array. - An ordered dictionary with the name and tensor attributes of each parameter. - A 2 x n_params numpy array with lower and upper bounds if at least one constraint is finite, and None otherwise.

Return type:

Tuple[ndarray, Dict[str, TorchAttr], ndarray | None]

Example

>>> mll = ExactMarginalLogLikelihood(model.likelihood, model)
>>> parameter_array, property_dict, bounds_out = module_to_array(mll)

botorch.optim.numpy_converter.set_params_with_array(module, x, property_dict)[source]¶

Set module parameters with values from numpy array.

Parameters:

• module (Module) – Module with parameters to be set

• x (ndarray) – Numpy array with parameter values

• property_dict (Dict[str, TorchAttr]) – Dictionary of parameter names and torch attributes as returned by module_to_array.

Returns:

module with parameters updated in-place.

Return type:

Module

Example

>>> mll = ExactMarginalLogLikelihood(model.likelihood, model)
>>> parameter_array, property_dict, bounds_out = module_to_array(mll)
>>> parameter_array += 0.1  # perturb parameters (for example only)
>>> mll = set_params_with_array(mll, parameter_array,  property_dict)

Parameter Constraint Utilities¶

Utility functions for constrained optimization.

botorch.optim.parameter_constraints.make_scipy_bounds(X, lower_bounds=None, upper_bounds=None)[source]¶

Creates a scipy Bounds object for optimziation

Parameters:

• X (Tensor) – … x d tensor

• lower_bounds (float | Tensor | None) – Lower bounds on each column (last dimension) of X. If this is a single float, then all columns have the same bound.

• upper_bounds (float | Tensor | None) – Lower bounds on each column (last dimension) of X. If this is a single float, then all columns have the same bound.

Returns:

A scipy Bounds object if either lower_bounds or upper_bounds is not None, and None otherwise.

Return type:

Bounds | None

Example

>>> X = torch.rand(5, 2)
>>> scipy_bounds = make_scipy_bounds(X, 0.1, 0.8)

botorch.optim.parameter_constraints.make_scipy_linear_constraints(shapeX, inequality_constraints=None, equality_constraints=None)[source]¶

Generate scipy constraints from torch representation.

Parameters:

• shapeX (Size) – The shape of the torch.Tensor to optimize over (i.e. (b) x q x d)

• constraints (equality) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs, where indices is a single-dimensional index tensor (long dtype) containing indices into the last dimension of X, coefficients is a single-dimensional tensor of coefficients of the same length, and rhs is a scalar.

• constraints – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) == rhs (with indices and coefficients of the same form as in inequality_constraints).

• inequality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

• equality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

Returns:

A list of dictionaries containing callables for constraint function values and Jacobians and a string indicating the associated constraint type (“eq”, “ineq”), as expected by scipy.minimize.

Return type:

List[Dict[str, str | Callable[[ndarray], float] | Callable[[ndarray], ndarray]]]

This function assumes that constraints are the same for each input batch, and broadcasts the constraints accordingly to the input batch shape. This function does support constraints across elements of a q-batch if the indices are a 2-d Tensor.

Example

The following will enforce that x[1] + 0.5 x[3] >= -0.1 for each x in both elements of the q-batch, and each of the 3 t-batches:

>>> constraints = make_scipy_linear_constraints(
>>>     torch.Size([3, 2, 4]),
>>>     [(torch.tensor([1, 3]), torch.tensor([1.0, 0.5]), -0.1)],
>>> )

The following will enforce that x[0, 1] + 0.5 x[1, 3] >= -0.1 where x[0, :] is the first element of the q-batch and x[1, :] is the second element of the q-batch, for each of the 3 t-batches:

>>> constraints = make_scipy_linear_constraints(
>>>     torch.size([3, 2, 4])
>>>     [(torch.tensor([[0, 1], [1, 3]), torch.tensor([1.0, 0.5]), -0.1)],
>>> )