botorch.optim¶
botorch.optim.fit¶
Tools for model fitting.

class
botorch.optim.fit.
OptimizationIteration
(itr, fun, time)[source]¶ Create new instance of OptimizationIteration(itr, fun, time)

property
fun
¶ Alias for field number 1

property
itr
¶ Alias for field number 0

property
time
¶ Alias for field number 2

property

botorch.optim.fit.
fit_gpytorch_scipy
(mll, bounds=None, method='LBFGSB', options=None, track_iterations=True)[source]¶ Fit a gpytorch model by maximizing MLL with a scipy optimizer.
The model and likelihood in mll must already be in train mode. Note: this method requires that the model has train_inputs and train_targets.
 Parameters
mll (
MarginalLogLikelihood
) – MarginalLogLikelihood to be maximized.bounds (
Optional
[Dict
[str
,Tuple
[Optional
[float
],Optional
[float
]]]]) – A dictionary mapping parameter names to tuples of lower and upper bounds.method (
str
) – Solver type, passed along to scipy.minimize.options (
Optional
[Dict
[str
,Any
]]) – Dictionary of solver options, passed along to scipy.minimize.track_iterations (
bool
) – Track the function values and wall time for each iteration.
 Return type
Tuple
[MarginalLogLikelihood
,Dict
[str
,Union
[float
,List
[OptimizationIteration
]]]] Returns
2element tuple containing  MarginalLogLikelihood with parameters optimized inplace.  Dictionary with the following key/values: “fopt”: Best mll value. “wall_time”: Wall time of fitting. “iterations”: List of OptimizationIteration objects with information on each iteration. If track_iterations is False, will be empty.
Example
>>> gp = SingleTaskGP(train_X, train_Y) >>> mll = ExactMarginalLogLikelihood(gp.likelihood, gp) >>> mll.train() >>> fit_gpytorch_scipy(mll) >>> mll.eval()

botorch.optim.fit.
fit_gpytorch_torch
(mll, bounds=None, optimizer_cls=<class 'torch.optim.adam.Adam'>, options=None, track_iterations=True)[source]¶ Fit a gpytorch model by maximizing MLL with a torch optimizer.
The model and likelihood in mll must already be in train mode. Note: this method requires that the model has train_inputs and train_targets.
 Parameters
mll (
MarginalLogLikelihood
) – MarginalLogLikelihood to be maximized.bounds (
Optional
[Dict
[str
,Tuple
[Optional
[float
],Optional
[float
]]]]) – A ParameterBounds dictionary mapping parameter names to tuples of lower and upper bounds. Bounds specified here take precedence over bounds on the same parameters specified in the constraints registered with the module.optimizer_cls (
Optimizer
) – Torch optimizer to use. Must not require a closure.options (
Optional
[Dict
[str
,Any
]]) – options for model fitting. Relevant options will be passed to the optimizer_cls. Additionally, options can include: “disp” to specify whether to display model fitting diagnostics and “maxiter” to specify the maximum number of iterations.track_iterations (
bool
) – Track the function values and wall time for each iteration.
 Return type
Tuple
[MarginalLogLikelihood
,Dict
[str
,Union
[float
,List
[OptimizationIteration
]]]] Returns
2element tuple containing  mll with parameters optimized inplace.  Dictionary with the following key/values: “fopt”: Best mll value. “wall_time”: Wall time of fitting. “iterations”: List of OptimizationIteration objects with information on each iteration. If track_iterations is False, will be empty.
Example
>>> gp = SingleTaskGP(train_X, train_Y) >>> mll = ExactMarginalLogLikelihood(gp.likelihood, gp) >>> mll.train() >>> fit_gpytorch_torch(mll) >>> mll.eval()
botorch.optim.initializers¶

botorch.optim.initializers.
gen_batch_initial_conditions
(acq_function, bounds, q, num_restarts, raw_samples, options=None)[source]¶ Generate a batch of initial conditions for randomrestart optimziation.
 Parameters
acq_function (
AcquisitionFunction
) – The acquisition function to be optimized.bounds (
Tensor
) – A 2 x d tensor of lower and upper bounds for each column of X.q (
int
) – The number of candidates to consider.num_restarts (
int
) – The number of starting points for multistart acquisition function optimization.raw_samples (
int
) – The number of raw samples to consider in the initialization heuristic.options (
Optional
[Dict
[str
,Union
[bool
,float
,int
]]]) – Options for initial condition generation. For valid options see initialize_q_batch and initialize_q_batch_nonneg. If options contains a nonnegative=True entry, then acq_function is assumed to be nonnegative (useful when using custom acquisition functions).
 Return type
Tensor
 Returns
A num_restarts x q x d tensor of initial conditions.
Example
>>> qEI = qExpectedImprovement(model, best_f=0.2) >>> bounds = torch.tensor([[0.], [1.]]) >>> Xinit = gen_batch_initial_conditions( >>> qEI, bounds, q=3, num_restarts=25, raw_samples=500 >>> )

botorch.optim.initializers.
gen_one_shot_kg_initial_conditions
(acq_function, bounds, q, num_restarts, raw_samples, options=None)[source]¶ Generate a batch of smart initializations for qKnowledgeGradient.
This function generates initial conditions for optimizing oneshot KG using the maximizer of the posterior objective. Intutively, the maximizer of the fantasized posterior will often be close to a maximizer of the current posterior. This function uses that fact to generate the initital conditions for the fantasy points. Specifically, a fraction of 1  frac_random (see options) is generated by sampling from the set of maximizers of the posterior objective (obtained via random restart optimization) according to a softmax transformation of their respective values. This means that this initialization strategy internally solves an acquisition function maximization problem. The remaining frac_random fantasy points as well as all q candidate points are chosen according to the standard initialization strategy in gen_batch_initial_conditions.
 Parameters
acq_function (
qKnowledgeGradient
) – The qKnowledgeGradient instance to be optimized.bounds (
Tensor
) – A 2 x d tensor of lower and upper bounds for each column of task features.q (
int
) – The number of candidates to consider.num_restarts (
int
) – The number of starting points for multistart acquisition function optimization.raw_samples (
int
) – The number of raw samples to consider in the initialization heuristic.options (
Optional
[Dict
[str
,Union
[bool
,float
,int
]]]) – Options for initial condition generation. These contain all settings for the standard heuristic initialization from gen_batch_initial_conditions. In addition, they contain frac_random (the fraction of fully random fantasy points), num_inner_restarts and raw_inner_samples (the number of random restarts and raw samples for solving the posterior objective maximization problem, respectively) and eta (temperature parameter for sampling heuristic from posterior objective maximizers).
 Return type
Optional
[Tensor
] Returns
A num_restarts x q’ x d tensor that can be used as initial conditions for optimize_acqf(). Here q’ = q + num_fantasies is the total number of points (candidate points plus fantasy points).
Example
>>> qKG = qKnowledgeGradient(model, num_fantasies=64) >>> bounds = torch.tensor([[0., 0.], [1., 1.]]) >>> Xinit = gen_one_shot_kg_initial_conditions( >>> qKG, bounds, q=3, num_restarts=10, raw_samples=512, >>> options={"frac_random": 0.25}, >>> )

botorch.optim.initializers.
initialize_q_batch
(X, Y, n, eta=1.0)[source]¶ Heuristic for selecting initial conditions for candidate generation.
This heuristic selects points from X (without replacement) with probability proportional to exp(eta * Z), where Z = (Y  mean(Y)) / std(Y) and eta is a temperature parameter.
When using an acquisiton function that is nonnegative and possibly zero over large areas of the feature space (e.g. qEI), you should use initialize_q_batch_nonneg instead.
 Parameters
X (
Tensor
) – A b x q x d tensor of b samples of qbatches from a ddim. feature space. Typically, these are generated using qMC sampling.Y (
Tensor
) – A tensor of b outcomes associated with the samples. Typically, this is the value of the batch acquisition function to be maximized.n (
int
) – The number of initial condition to be generated. Must be less than b.eta (
float
) – Temperature parameter for weighting samples.
 Return type
Tensor
 Returns
A n x q x d tensor of n qbatch initial conditions.
Example
>>> # To get `n=10` starting points of qbatch size `q=3` >>> # for model with `d=6`: >>> qUCB = qUpperConfidenceBound(model, beta=0.1) >>> Xrnd = torch.rand(500, 3, 6) >>> Xinit = initialize_q_batch(Xrnd, qUCB(Xrnd), 10)

botorch.optim.initializers.
initialize_q_batch_nonneg
(X, Y, n, eta=1.0, alpha=0.0001)[source]¶ Heuristic for selecting initial conditions for nonneg. acquisition functions.
This function is similar to initialize_q_batch, but designed specifically for acquisition functions that are nonnegative and possibly zero over large areas of the feature space (e.g. qEI). All samples for which Y < alpha * max(Y) will be ignored (assuming that Y contains at least one positive value).
 Parameters
X (
Tensor
) – A b x q x d tensor of b samples of qbatches from a ddim. feature space. Typically, these are generated using qMC.Y (
Tensor
) – A tensor of b outcomes associated with the samples. Typically, this is the value of the batch acquisition function to be maximized.n (
int
) – The number of initial condition to be generated. Must be less than b.eta (
float
) – Temperature parameter for weighting samples.alpha (
float
) – The threshold (as a fraction of the maximum observed value) under which to ignore samples. All input samples for which Y < alpha * max(Y) will be ignored.
 Return type
Tensor
 Returns
A n x q x d tensor of n qbatch initial conditions.
Example
>>> # To get `n=10` starting points of qbatch size `q=3` >>> # for model with `d=6`: >>> qEI = qExpectedImprovement(model, best_f=0.2) >>> Xrnd = torch.rand(500, 3, 6) >>> Xinit = initialize_q_batch(Xrnd, qEI(Xrnd), 10)
botorch.optim.numpy_converter¶
A converter that simplifies using numpybased optimizers with generic torch nn.Module classes. This enables using a scipy.optim.minimize optimizer for optimizing module parameters.

class
botorch.optim.numpy_converter.
TorchAttr
(shape, dtype, device)[source]¶ Create new instance of TorchAttr(shape, dtype, device)

property
device
¶ Alias for field number 2

property
dtype
¶ Alias for field number 1

property
shape
¶ Alias for field number 0

property

botorch.optim.numpy_converter.
module_to_array
(module, bounds=None, exclude=None)[source]¶ Extract named parameters from a module into a numpy array.
Only extracts parameters with requires_grad, since it is meant for optimizing.
 Parameters
module (
Module
) – A module with parameters. May specify parameter constraints in a named_parameters_and_constraints method.bounds (
Optional
[Dict
[str
,Tuple
[Optional
[float
],Optional
[float
]]]]) – A ParameterBounds dictionary mapping parameter names to tuples of lower and upper bounds. Bounds specified here take precedence over bounds on the same parameters specified in the constraints registered with the module.exclude (
Optional
[Set
[str
]]) – A list of parameter names that are to be excluded from extraction.
 Return type
Tuple
[ndarray
,Dict
[str
,TorchAttr
],Optional
[ndarray
]] Returns
3element tuple containing  The parameter values as a numpy array.  An ordered dictionary with the name and tensor attributes of each parameter.  A 2 x n_params numpy array with lower and upper bounds if at least one constraint is finite, and None otherwise.
Example
>>> mll = ExactMarginalLogLikelihood(model.likelihood, model) >>> parameter_array, property_dict, bounds_out = module_to_array(mll)

botorch.optim.numpy_converter.
set_params_with_array
(module, x, property_dict)[source]¶ Set module parameters with values from numpy array.
 Parameters
module (
Module
) – Module with parameters to be setx (
ndarray
) – Numpy array with parameter valuesproperty_dict (
Dict
[str
,TorchAttr
]) – Dictionary of parameter names and torch attributes as returned by module_to_array.
 Returns
module with parameters updated inplace.
 Return type
Module
Example
>>> mll = ExactMarginalLogLikelihood(model.likelihood, model) >>> parameter_array, property_dict, bounds_out = module_to_array(mll) >>> parameter_array += 0.1 # perturb parameters (for example only) >>> mll = set_params_with_array(mll, parameter_array, property_dict)
botorch.optim.optimize¶
Methods for optimizing acquisition functions.

botorch.optim.optimize.
joint_optimize
(acq_function, bounds, q, num_restarts, raw_samples, options=None, inequality_constraints=None, equality_constraints=None, fixed_features=None, post_processing_func=None, batch_initial_conditions=None, return_best_only=True)[source]¶ DEPRECATED  Use optimize_acqf instead.
 Return type
Tuple
[Tensor
,Tensor
]

botorch.optim.optimize.
optimize_acqf
(acq_function, bounds, q, num_restarts, raw_samples, options=None, inequality_constraints=None, equality_constraints=None, fixed_features=None, post_processing_func=None, batch_initial_conditions=None, return_best_only=True, sequential=False)[source]¶ Generate a set of candidates via multistart optimization.
 Parameters
acq_function (
AcquisitionFunction
) – An AcquisitionFunction.bounds (
Tensor
) – A 2 x d tensor of lower and upper bounds for each column of X.q (
int
) – The number of candidates.num_restarts (
int
) – The number of starting points for multistart acquisition function optimization.raw_samples (
int
) – The number of samples for initialization.options (
Optional
[Dict
[str
,Union
[bool
,float
,int
,str
]]]) – Options for candidate generation.constraints (equality) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs
constraints – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) = rhs
fixed_features (
Optional
[Dict
[int
,float
]]) – A map {feature_index: value} for features that should be fixed to a particular value during generation.post_processing_func (
Optional
[Callable
[[Tensor
],Tensor
]]) – A function that postprocesses an optimization result appropriately (i.e., according to roundtrip transformations).batch_initial_conditions (
Optional
[Tensor
]) – A tensor to specify the initial conditions. Set this if you do not want to use default initialization strategy.return_best_only (
bool
) – If False, outputs the solutions corresponding to all random restart initializations of the optimization.sequential (
bool
) – If False, uses joint optimization, otherwise uses sequential optimization.Returns –
A twoelement tuple containing
a (num_restarts) x q x ddim tensor of generated candidates.
a tensor of associated acquisiton values. If sequential=False, this is a (num_restarts)dim tensor of joint acquisition values (with explicit restart dimension if return_best_only=False). If sequential=True, this is a qdim tensor of expected acquisition values conditional on having observed canidates 0,1,…,i1.
Example –
>>> # generate `q=2` candidates jointly using 20 random restarts >>> # and 512 raw samples >>> candidates, acq_value = optimize_acqf(qEI, bounds, 2, 20, 512)
>>> generate `q=3` candidates sequentially using 15 random restarts >>> # and 256 raw samples >>> qEI = qExpectedImprovement(model, best_f=0.2) >>> bounds = torch.tensor([[0.], [1.]]) >>> candidates, acq_value_list = optimize_acqf( >>> qEI, bounds, 3, 15, 256, sequential=True >>> )
 Return type
Tuple
[Tensor
,Tensor
]

botorch.optim.optimize.
optimize_acqf_cyclic
(acq_function, bounds, q, num_restarts, raw_samples, options=None, inequality_constraints=None, equality_constraints=None, fixed_features=None, post_processing_func=None, batch_initial_conditions=None, cyclic_options=None)[source]¶ Generate a set of q candidates via cyclic optimization.
 Parameters
acq_function (
AcquisitionFunction
) – An AcquisitionFunctionbounds (
Tensor
) – A 2 x d tensor of lower and upper bounds for each column of X.q (
int
) – The number of candidates.num_restarts (
int
) – Number of starting points for multistart acquisition function optimization.raw_samples (
int
) – Number of samples for initializationoptions (
Optional
[Dict
[str
,Union
[bool
,float
,int
,str
]]]) – Options for candidate generation.constraints (equality) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs
constraints – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) = rhs
fixed_features (
Optional
[Dict
[int
,float
]]) – A map {feature_index: value} for features that should be fixed to a particular value during generation.post_processing_func (
Optional
[Callable
[[Tensor
],Tensor
]]) – A function that postprocesses an optimization result appropriately (i.e., according to roundtrip transformations).batch_initial_conditions (
Optional
[Tensor
]) – A tensor to specify the initial conditions. If no initial conditions are provided, the default initialization will be used.cyclic_options (
Optional
[Dict
[str
,Union
[bool
,float
,int
,str
]]]) – Options for convergence criterion for outer cyclic optimization.Returns –
A twoelement tuple containing
a q x ddim tensor of generated candidates.
a qdim tensor of expected acquisition values, where the i`th value is the acquistion value conditional on having observed all candidates except candidate `i.
Example –
>>> # generate `q=3` candidates cyclically using 15 random restarts >>> # 256 raw samples, and 4 cycles >>> >>> qEI = qExpectedImprovement(model, best_f=0.2) >>> bounds = torch.tensor([[0.], [1.]]) >>> candidates, acq_value_list = optimize_acqf_cyclic( >>> qEI, bounds, 3, 15, 256, cyclic_options={"maxiter": 4} >>> )
 Return type
float

botorch.optim.optimize.
sequential_optimize
(acq_function, bounds, q, num_restarts, raw_samples, options=None, inequality_constraints=None, equality_constraints=None, fixed_features=None, post_processing_func=None)[source]¶ DEPRECATED  Use optimize_acqf with sequential=True instead.
 Return type
Tuple
[Tensor
,Tensor
]
botorch.optim.parameter_constraints¶
Utility functions for constrained optimization.

botorch.optim.parameter_constraints.
eval_lin_constraint
(x, flat_idxr, coeffs, rhs)[source]¶ Evaluate a single linear constraint.
 Parameters
x (
ndarray
) – The input array.flat_idxr (
List
[int
]) – The indices in x to consider.coeffs (
ndarray
) – The coefficients corresponding to the indices.rhs (
float
) – The righthandside of the constraint.
 Returns
sum_i (coeffs[i] * x[i])  rhs
 Return type
The evaluted constraint

botorch.optim.parameter_constraints.
lin_constraint_jac
(x, flat_idxr, coeffs, n)[source]¶ Return the Jacobian associated with a linear constraint.
 Parameters
x (
ndarray
) – The input array.flat_idxr (
List
[int
]) – The indices for the elements of x that appear in the constraint.coeffs (
ndarray
) – The coefficients corresponding to the indices.n (
int
) – number of elements
 Return type
ndarray
 Returns
The Jacobian.

botorch.optim.parameter_constraints.
make_scipy_bounds
(X, lower_bounds=None, upper_bounds=None)[source]¶ Creates a scipy Bounds object for optimziation
 Parameters
X (
Tensor
) – … x d tensorlower_bounds (
Union
[float
,Tensor
,None
]) – Lower bounds on each column (last dimension) of X. If this is a single float, then all columns have the same bound.upper_bounds (
Union
[float
,Tensor
,None
]) – Lower bounds on each column (last dimension) of X. If this is a single float, then all columns have the same bound.
 Return type
Optional
[Bounds
] Returns
A scipy Bounds object if either lower_bounds or upper_bounds is not None, and None otherwise.
Example
>>> X = torch.rand(5, 2) >>> scipy_bounds = make_scipy_bounds(X, 0.1, 0.8)

botorch.optim.parameter_constraints.
make_scipy_linear_constraints
(shapeX, inequality_constraints=None, equality_constraints=None)[source]¶ Generate scipy constraints from torch representation.
 Parameters
shapeX (
Size
) – The shape of the torch.Tensor to optimize over (i.e. b x q x d)constraints (equality) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs, where indices is a singledimensional index tensor (long dtype) containing indices into the last dimension of X, coefficients is a singledimensional tensor of coefficients of the same length, and rhs is a scalar.
constraints – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) == rhs (with indices and coefficients of the same form as in inequality_constraints).
 Return type
List
[Dict
[str
,Union
[str
,Callable
[[ndarray
],float
],Callable
[[ndarray
],ndarray
]]]] Returns
A list of dictionaries containing callables for constraint function values and Jacobians and a string indicating the associated constraint type (“eq”, “ineq”), as expected by scipy.minimize.
This function assumes that constraints are the same for each input batch, and broadcasts the constraints accordingly to the input batch shape. This function does support constraints across elements of a qbatch if the indices are a 2d Tensor.
Example
The following will enforce that x[1] + 0.5 x[3] >= 0.1 for each x in both elements of the qbatch, and each of the 3 tbatches:
>>> constraints = make_scipy_linear_constraints( >>> torch.Size([3, 2, 4]), >>> [(torch.tensor([1, 3]), torch.tensor([1.0, 0.5]), 0.1)], >>> )
The following will enforce that x[0, 1] + 0.5 x[1, 3] >= 0.1 where x[0, :] is the first element of the qbatch and x[1, :] is the second element of the qbatch, for each of the 3 tbatches:
>>> constraints = make_scipy_linear_constraints( >>> torch.size([3, 2, 4]) >>> [(torch.tensor([[0, 1], [1, 3]), torch.tensor([1.0, 0.5]), 0.1)], >>> )
botorch.optim.utils¶
Utilities for optimization.

class
botorch.optim.utils.
ConvergenceCriterion
(maxiter=15000, ftol=2.220446049250313e09, minimize=True)[source]¶ Basic class for evaluating optimization convergence.
Constructor for ConvergenceCriterion.
 Parameters
maxiter (
int
) – maximum number of iterations.ftol (
float
) – Function value relative tolerance for termination.minimize (
bool
) – boolean indicating the optimization direction.

evaluate
(fvals)[source]¶ Evaluate convergence criterion.
 Parameters
fvals (
Tensor
) – tensor containing function values for the current iteration. If fvals contains more than one element, then the relative tolerance criterion is evaluated elementwise and True is returned if all elements have converged.
TODO: add support for utilizing gradient information
 Returns
convergence indicator
 Return type
bool

botorch.optim.utils.
columnwise_clamp
(X, lower=None, upper=None, raise_on_violation=False)[source]¶ Clamp values of a Tensor in columnwise fashion (with support for tbatches).
This function is useful in conjunction with optimizers from the torch.optim package, which don’t natively handle constraints. If you apply this after a gradient step you can be fancy and call it “projected gradient descent”. This funtion is also useful for postprocessing candidates generated by the scipy optimizer that satisfy bounds only up to numerical accuracy.
 Parameters
X (
Tensor
) – The b x n x d input tensor. If 2dimensional, b is assumed to be 1.lower (
Union
[float
,Tensor
,None
]) – The columnwise lower bounds. If scalar, apply bound to all columns.upper (
Union
[float
,Tensor
,None
]) – The columnwise upper bounds. If scalar, apply bound to all columns.raise_on_violation (
bool
) – If True, raise an exception when the elments in X are out of the specified bounds (up to numerical accuracy). This is useful for postprocessing candidates generated by optimizers that satisfy imposed bounds only up to numerical accuracy.
 Return type
Tensor
 Returns
The clamped tensor.

botorch.optim.utils.
fix_features
(X, fixed_features=None)[source]¶ Fix feature values in a Tensor.
The fixed features will have zero gradient in downstream calculations.
 Parameters
X (
Tensor
) – input Tensor with shape … x p, where p is the number of featuresfixed_features (
Optional
[Dict
[int
,Optional
[float
]]]) – A dictionary with keys as column indices and values equal to what the feature should be set to in X. If the value is None, that column is just considered fixed. Keys should be in the range [0, p  1].
 Return type
Tensor
 Returns
The tensor X with fixed features.

botorch.optim.utils.
sample_all_priors
(model)[source]¶ Sample from hyperparameter priors (inplace).
 Parameters
model (
GPyTorchModel
) – A GPyTorchModel. Return type
None