botorch.utils¶

Constraints¶

Helpers for handling input or outcome constraints.

botorch.utils.constraints.get_outcome_constraint_transforms(outcome_constraints)[source]¶

Create outcome constraint callables from outcome constraint tensors.

Parameters:

outcome_constraints (Tuple[Tensor, Tensor] | None) – A tuple of (A, b). For k outcome constraints and m outputs at f(x)`, A is k x m and b is k x 1 such that A f(x) <= b.

Returns:

A list of callables, each mapping a Tensor of size b x q x m to a tensor of size b x q, where m is the number of outputs of the model. Negative values imply feasibility. The callables support broadcasting (e.g. for calling on a tensor of shape mc_samples x b x q x m).

Return type:

List[Callable[[Tensor], Tensor]] | None

Example

>>> # constrain `f(x)[0] <= 0`
>>> A = torch.tensor([[1., 0.]])
>>> b = torch.tensor([[0.]])
>>> outcome_constraints = get_outcome_constraint_transforms((A, b))

botorch.utils.constraints.get_monotonicity_constraints(d, descending=False, dtype=None, device=None)[source]¶

Returns a system of linear inequalities (A, b) that generically encodes order constraints on the elements of a d-dimsensional space, i.e. A @ x < b implies x[i] < x[i + 1] for a d-dimensional vector x.

Idea: Could encode A as sparse matrix, if it is supported well.

Parameters:

• d (int) – Dimensionality of the constraint space, i.e. number of monotonic parameters.

• descending (bool) – If True, forces the elements of a vector to be monotonically de- creasing and be monotonically increasing otherwise.

• dtype (dtype | None) – The dtype of the returned Tensors.

• device (device | None) – The device of the returned Tensors.

Returns:

A tuple of Tensors (A, b) representing the monotonicity constraint as a system of linear inequalities A @ x < b. A is (d - 1) x d-dimensional and b is (d - 1) x 1-dimensional.

Return type:

Tuple[Tensor, Tensor]

Containers¶

Representations for different kinds of data.

class botorch.utils.containers.DenseContainer(values, event_shape)[source]¶

Bases: BotorchContainer

Basic representation of data stored as a dense Tensor.

Parameters:

• values (Tensor) –

• event_shape (Size) –

values: Tensor¶

event_shape: Size¶

property shape: Size¶

property device: device¶

property dtype: dtype¶

class botorch.utils.containers.SliceContainer(values, indices, event_shape)[source]¶

Bases: BotorchContainer

Represent data points formed by concatenating (n-1)-dimensional slices taken from the leading dimension of an n-dimensional source tensor.

Parameters:

• values (Tensor) –

• indices (LongTensor) –

• event_shape (Size) –

values: Tensor¶

indices: LongTensor¶

event_shape: Size¶

property shape: Size¶

property device: device¶

property dtype: dtype¶

Context Managers¶

Utilities for optimization.

class botorch.utils.context_managers.TensorCheckpoint(values, device, dtype)[source]¶

Bases: NamedTuple

Create new instance of TensorCheckpoint(values, device, dtype)

Parameters:

• values (Tensor) –

• device (device | None) –

• dtype (dtype | None) –

values: Tensor¶

Alias for field number 0

device: device | None¶

Alias for field number 1

dtype: dtype | None¶

Alias for field number 2

botorch.utils.context_managers.delattr_ctx(instance, *attrs, enforce_hasattr=False)[source]¶

Contextmanager for temporarily deleting attributes.

Parameters:

• instance (object) –

• attrs (str) –

• enforce_hasattr (bool) –

Return type:

Generator[None, None, None]

botorch.utils.context_managers.requires_grad_ctx(module, assignments)[source]¶

Contextmanager for temporarily setting the requires_grad field of a module’s parameters.

Parameters:

• module (Module) –

• assignments (Dict[str, bool]) –

Return type:

Generator[None, None, None]

botorch.utils.context_managers.parameter_rollback_ctx(parameters, checkpoint=None, **tkwargs)[source]¶

Contextmanager that exits by rolling back a module’s state_dict.

Parameters:

• module – Module instance.

• name_filter – Optional Boolean function used to filter items by name.

• checkpoint (Dict[str, TensorCheckpoint] | None) – Optional cache of values and tensor metadata specifying the rollback state for the module (or some subset thereof).

• **tkwargs (Any) – Keyword arguments passed to torch.Tensor.to when copying data from each tensor in module.state_dict() to the internally created checkpoint. Only adhered to when the checkpoint argument is None.

• parameters (Dict[str, Tensor]) –

Yields:

A dictionary of TensorCheckpoints for the module’s state_dict. Any in-places changes to the checkpoint will be observed at rollback time. If the checkpoint is cleared, no rollback will occur.

Return type:

Generator[Dict[str, TensorCheckpoint], None, None]

botorch.utils.context_managers.module_rollback_ctx(module, name_filter=None, checkpoint=None, **tkwargs)[source]¶

Contextmanager that exits by rolling back a module’s state_dict.

Parameters:

• module (Module) – Module instance.

• name_filter (Callable[[str], bool] | None) – Optional Boolean function used to filter items by name.

• checkpoint (Dict[str, TensorCheckpoint] | None) – Optional cache of values and tensor metadata specifying the rollback state for the module (or some subset thereof).

• **tkwargs (Any) – Keyword arguments passed to torch.Tensor.to when copying data from each tensor in module.state_dict() to the internally created checkpoint. Only adhered to when the checkpoint argument is None.

Yields:

A dictionary of TensorCheckpoints for the module’s state_dict. Any in-places changes to the checkpoint will be observed at rollback time. If the checkpoint is cleared, no rollback will occur.

Return type:

Generator[Dict[str, TensorCheckpoint], None, None]

botorch.utils.context_managers.zero_grad_ctx(parameters, zero_on_enter=True, zero_on_exit=False)[source]¶

Parameters:

• parameters (Dict[str, Tensor] | Iterable[Tensor]) –

• zero_on_enter (bool) –

• zero_on_exit (bool) –

Return type:

Generator[None, None, None]

Datasets¶

Representations for different kinds of datasets.

class botorch.utils.datasets.SupervisedDataset(X, Y, *, feature_names, outcome_names, Yvar=None, validate_init=True)[source]¶

Bases: object

Base class for datasets consisting of labelled pairs (X, Y) and an optional Yvar that stipulates observations variances so that Y[i] ~ N(f(X[i]), Yvar[i]).

Example:

X = torch.rand(16, 2)
Y = torch.rand(16, 1)
feature_names = ["learning_rate", "embedding_dim"]
outcome_names = ["neg training loss"]
A = SupervisedDataset(
    X=X,
    Y=Y,
    feature_names=feature_names,
    outcome_names=outcome_names,
)
B = SupervisedDataset(
    X=DenseContainer(X, event_shape=X.shape[-1:]),
    Y=DenseContainer(Y, event_shape=Y.shape[-1:]),
    feature_names=feature_names,
    outcome_names=outcome_names,
)
assert A == B

Constructs a SupervisedDataset.

Parameters:

• X (Union[BotorchContainer, Tensor]) – A Tensor or BotorchContainer representing the input features.

• Y (Union[BotorchContainer, Tensor]) – A Tensor or BotorchContainer representing the outcomes.

• feature_names (List[str]) – A list of names of the features in X.

• outcome_names (List[str]) – A list of names of the outcomes in Y.

• Yvar (Union[BotorchContainer, Tensor, None]) – An optional Tensor or BotorchContainer representing the observation noise.

• validate_init (bool) – If True, validates the input shapes.

property X: Tensor¶

property Y: Tensor¶

property Yvar: Tensor | None¶

class botorch.utils.datasets.FixedNoiseDataset(X, Y, Yvar, feature_names, outcome_names, validate_init=True)[source]¶

Bases: SupervisedDataset

A SupervisedDataset with an additional field Yvar that stipulates observations variances so that Y[i] ~ N(f(X[i]), Yvar[i]).

NOTE: This is deprecated. Use SupervisedDataset instead. Will be removed in a future release (~v0.11).

Initialize a FixedNoiseDataset – deprecated!

Parameters:

• X (Union[BotorchContainer, Tensor]) –

• Y (Union[BotorchContainer, Tensor]) –

• Yvar (Union[BotorchContainer, Tensor]) –

• feature_names (List[str]) –

• outcome_names (List[str]) –

• validate_init (bool) –

class botorch.utils.datasets.RankingDataset(X, Y, feature_names, outcome_names, validate_init=True)[source]¶

Bases: SupervisedDataset

A SupervisedDataset whose labelled pairs (x, y) consist of m-ary combinations x ∈ Z^{m} of elements from a ground set Z = (z_1, …) and ranking vectors y {0, …, m - 1}^{m} with properties:

1. Ranks start at zero, i.e. min(y) = 0.

2. Sorted ranks are contiguous unless one or more ties are present.

3. k ranks are skipped after a k-way tie.

Example:

X = SliceContainer(
    values=torch.rand(16, 2),
    indices=torch.stack([torch.randperm(16)[:3] for _ in range(8)]),
    event_shape=torch.Size([3 * 2]),
)
Y = DenseContainer(
    torch.stack([torch.randperm(3) for _ in range(8)]),
    event_shape=torch.Size([3])
)
feature_names = ["item_0", "item_1"]
outcome_names = ["ranking outcome"]
dataset = RankingDataset(
    X=X,
    Y=Y,
    feature_names=feature_names,
    outcome_names=outcome_names,
)

Construct a RankingDataset.

Parameters:

• X (SliceContainer) – A SliceContainer representing the input features being ranked.

• Y (Union[BotorchContainer, Tensor]) – A Tensor or BotorchContainer representing the rankings.

• feature_names (List[str]) – A list of names of the features in X.

• outcome_names (List[str]) – A list of names of the outcomes in Y.

• validate_init (bool) – If True, validates the input shapes.

class botorch.utils.datasets.MultiTaskDataset(datasets, target_outcome_name, task_feature_index=None)[source]¶

Bases: SupervisedDataset

This is a multi-task dataset that is constructed from the datasets of individual tasks. It offers functionality to combine parts of individual datasets to construct the inputs necessary for the MultiTaskGP models.

The datasets of individual tasks are allowed to represent different sets of features. When there are heterogeneous feature sets, calling MultiTaskDataset.X will result in an error.

Construct a MultiTaskDataset.

Parameters:

• datasets (List[SupervisedDataset]) – A list of the datasets of individual tasks. Each dataset is expected to contain data for only one outcome.

• target_outcome_name (str) – Name of the target outcome to be modeled.

• task_feature_index (Optional[int]) – If the task feature is included in the Xs of the individual datasets, this should be used to specify its index. If omitted, the task feature will be appended while concatenating Xs. If given, we sanity-check that the names of the task features match between all datasets.

classmethod from_joint_dataset(dataset, task_feature_index, target_task_value, outcome_names_per_task=None)[source]¶

Construct a MultiTaskDataset from a joint dataset that includes the data for all tasks with the task feature index.

This will break down the joint dataset into individual datasets by the value of the task feature. Each resulting dataset will have its outcome name set based on outcome_names_per_task, with the missing values defaulting to task_<task_feature> (except for the target task, which will retain the original outcome name from the dataset).

Parameters:

• dataset (SupervisedDataset) – The joint dataset.

• task_feature_index (int) – The column index of the task feature in dataset.X.

• target_task_value (int) – The value of the task feature for the target task in the dataset. The data for the target task is filtered according to dataset.X[task_feature_index] == target_task_value.

• outcome_names_per_task (Dict[int, str] | None) – Optional dictionary mapping task feature values to the outcome names for each task. If not provided, the auxiliary tasks will be named task_<task_feature> and the target task will retain the outcome name from the dataset.

Returns:

A MultiTaskDataset instance.

Return type:

MultiTaskDataset

property X: Tensor¶

Appends task features, if needed, and concatenates the Xs of datasets to produce the train_X expected by MultiTaskGP and subclasses.

If appending the task features, 0 is reserved for the target task and the remaining tasks are populated with 1, 2, …, len(datasets) - 1.

property Y: Tensor¶

Concatenates Ys of the datasets.

property Yvar: Tensor | None¶

Concatenates Yvars of the datasets if they exist.

get_dataset_without_task_feature(outcome_name)[source]¶

A helper for extracting the child datasets with their task features removed.

If the task feature index is None, the dataset will be returned as is.

Parameters:

outcome_name (str) – The outcome name for the dataset to extract.

Returns:

The dataset without the task feature.

Return type:

SupervisedDataset

class botorch.utils.datasets.ContextualDataset(datasets, parameter_decomposition, context_buckets, metric_decomposition=None)[source]¶

Bases: SupervisedDataset

This is a contextual dataset that is constructed from either a single dateset containing overall outcome or a list of datasets that each corresponds to a context breakdown.

Construct a ContextualDataset.

Parameters:

• datasets (List[SupervisedDataset]) – A list of the datasets of individual tasks. Each dataset is expected to contain data for only one outcome.

• parameter_decomposition (Dict[str, List[str]]) – Dict from context name to list of indices of X corresponding to that context.

• context_buckets (List[str]) – List of the context names in the order of dataset in datasets corresponding to each context outcome.

• metric_decomposition (Optional[Dict[str, List[str]]]) – Context breakdown metrics. Keys are context names. Values are the lists of metric names belonging to the context: {‘context1’: [‘m1_c1’], ‘context2’: [‘m1_c2’],}.

property X: Tensor¶

property Y: Tensor¶

Concatenates the Ys from the child datasets to create the Y expected by LCEM model if there are multiple datasets; Or return the Y expected by LCEA model if there is only one dataset.

property Yvar: Tensor¶

Concatenates the Yvars from the child datasets to create the Y expected by LCEM model if there are multiple datasets; Or return the Yvar expected by LCEA model if there is only one dataset.

Dispatcher¶

botorch.utils.dispatcher.type_bypassing_encoder(arg)[source]¶

Parameters:

arg (Any) –

Return type:

Type

class botorch.utils.dispatcher.Dispatcher(name, doc=None, encoder=<class 'type'>)[source]¶

Bases: Dispatcher

Clearing house for multiple dispatch functionality. This class extends <multipledispatch.Dispatcher> by: (i) generalizing the argument encoding convention during method lookup, (ii) implementing __getitem__ as a dedicated method lookup function.

Parameters:

• name (str) – A string identifier for the Dispatcher instance.

• doc (Optional[str]) – A docstring for the multiply dispatched method(s).

• encoder (Callable[Any, Type]) – A callable that individually transforms the arguments passed at runtime in order to construct the key used for method lookup as tuple(map(encoder, args)). Defaults to type.

dispatch(*types)[source]¶

Method lookup strategy. Checks for an exact match before traversing the set of registered methods according to the current ordering.

Parameters:

types (Type) – A tuple of types that gets compared with the signatures of registered methods to determine compatibility.

Returns:

The first method encountered with a matching signature.

Return type:

Callable

encode_args(args)[source]¶

Converts arguments into a tuple of types used during method lookup.

Parameters:

args (Any) –

Return type:

Tuple[Type]

help(*args, **kwargs)[source]¶

Prints the retrieved method’s docstring.

Parameters:

• args (Any) –

• kwargs (Any) –

Return type:

None

source(*args, **kwargs)[source]¶

Prints the retrieved method’s source types.

Return type:

None

property encoder: Callable[[Any], Type]¶

name¶

funcs¶

doc¶

Low-Rank Cholesky Update Utils¶

botorch.utils.low_rank.extract_batch_covar(mt_mvn)[source]¶

Extract a batched independent covariance matrix from an MTMVN.

Parameters:

mt_mvn (MultitaskMultivariateNormal) – A multi-task multivariate normal with a block diagonal covariance matrix.

Returns:

A lazy covariance matrix consisting of a batch of the blocks of

the diagonal of the MultitaskMultivariateNormal.

Return type:

LinearOperator

botorch.utils.low_rank.sample_cached_cholesky(posterior, baseline_L, q, base_samples, sample_shape, max_tries=6)[source]¶

Get posterior samples at the q new points from the joint multi-output posterior.

Parameters:

• posterior (GPyTorchPosterior) – The joint posterior is over (X_baseline, X).

• baseline_L (Tensor) – The baseline lower triangular cholesky factor.

• q (int) – The number of new points in X.

• base_samples (Tensor) – The base samples.

• sample_shape (Size) – The sample shape.

• max_tries (int) – The number of tries for computing the Cholesky decomposition with increasing jitter.

Returns:

A sample_shape x batch_shape x q x m-dim tensor of posterior

samples at the new points.

Return type:

Tensor

Objective¶

Helpers for handling objectives.

botorch.utils.objective.get_objective_weights_transform(weights)[source]¶

Create a linear objective callable from a set of weights.

Create a callable mapping a Tensor of size b x q x m and an (optional) Tensor of size b x q x d to a Tensor of size b x q, where m is the number of outputs of the model using scalarization via the objective weights. This callable supports broadcasting (e.g. for calling on a tensor of shape mc_samples x b x q x m). For m = 1, the objective weight is used to determine the optimization direction.

Parameters:

weights (Tensor | None) – a 1-dimensional Tensor containing a weight for each task. If not provided, the identity mapping is used.

Returns:

Transform function using the objective weights.

Return type:

Callable[[Tensor, Tensor | None], Tensor]

Example

>>> weights = torch.tensor([0.75, 0.25])
>>> transform = get_objective_weights_transform(weights)

botorch.utils.objective.apply_constraints_nonnegative_soft(obj, constraints, samples, eta)[source]¶

Applies constraints to a non-negative objective.

This function uses a sigmoid approximation to an indicator function for each constraint.

Parameters:

• obj (Tensor) – A n_samples x b x q (x m’)-dim Tensor of objective values.

• constraints (List[Callable[[Tensor], Tensor]]) – A list of callables, each mapping a Tensor of size b x q x m to a Tensor of size b x q, where negative values imply feasibility. This callable must support broadcasting. Only relevant for multi- output models (m > 1).

• samples (Tensor) – A n_samples x b x q x m Tensor of samples drawn from the posterior.

• eta (Tensor | float) – The temperature parameter for the sigmoid function. Can be either a float or a 1-dim tensor. In case of a float the same eta is used for every constraint in constraints. In case of a tensor the length of the tensor must match the number of provided constraints. The i-th constraint is then estimated with the i-th eta value.

Returns:

A n_samples x b x q (x m’)-dim tensor of feasibility-weighted objectives.

Return type:

Tensor

botorch.utils.objective.compute_feasibility_indicator(constraints, samples, marginalize_dim=None)[source]¶

Computes the feasibility of a list of constraints given posterior samples.

Parameters:

• constraints (List[Callable[[Tensor], Tensor]] | None) – A list of callables, each mapping a batch_shape x q x m`-dim Tensor to a batch_shape x q-dim Tensor, where negative values imply feasibility.

• samples (Tensor) – A batch_shape x q x m`-dim Tensor of posterior samples.

• marginalize_dim (int | None) – A batch dimension that should be marginalized. For example, this is useful when using a batched fully Bayesian model.

Returns:

A batch_shape x q-dim tensor of Boolean feasibility values.

Return type:

Tensor

botorch.utils.objective.compute_smoothed_feasibility_indicator(constraints, samples, eta, log=False, fat=False)[source]¶

Computes the smoothed feasibility indicator of a list of constraints.

Given posterior samples, using a sigmoid to smoothly approximate the feasibility indicator of each individual constraint to ensure differentiability and high gradient signal. The fat and log options improve the numerical behavior of the smooth approximation.

NOTE: Negative constraint values are associated with feasibility.

Parameters:

• constraints (List[Callable[[Tensor], Tensor]]) – A list of callables, each mapping a Tensor of size b x q x m to a Tensor of size b x q, where negative values imply feasibility. This callable must support broadcasting. Only relevant for multi- output models (m > 1).

• samples (Tensor) – A n_samples x b x q x m Tensor of samples drawn from the posterior.

• eta (Tensor | float) – The temperature parameter for the sigmoid function. Can be either a float or a 1-dim tensor. In case of a float the same eta is used for every constraint in constraints. In case of a tensor the length of the tensor must match the number of provided constraints. The i-th constraint is then estimated with the i-th eta value.

• log (bool) – Toggles the computation of the log-feasibility indicator.

• fat (bool) – Toggles the computation of the fat-tailed feasibility indicator.

Returns:

A n_samples x b x q-dim tensor of feasibility indicator values.

Return type:

Tensor

botorch.utils.objective.soft_eval_constraint(lhs, eta=0.001)[source]¶

Element-wise evaluation of a constraint in a ‘soft’ fashion

value(x) = 1 / (1 + exp(x / eta))

Parameters:

• lhs (Tensor) – The left hand side of the constraint lhs <= 0.

• eta (float) – The temperature parameter of the softmax function. As eta decreases, this approximates the Heaviside step function.

Returns:

Element-wise ‘soft’ feasibility indicator of the same shape as lhs. For each element x, value(x) -> 0 as x becomes positive, and value(x) -> 1 as x becomes negative.

Return type:

Tensor

botorch.utils.objective.apply_constraints(obj, constraints, samples, infeasible_cost, eta=0.001)[source]¶

Apply constraints using an infeasible_cost M for negative objectives.

This allows feasibility-weighting an objective for the case where the objective can be negative by using the following strategy: (1) Add M to make obj non-negative; (2) Apply constraints using the sigmoid approximation; (3) Shift by -M.

Parameters:

• obj (Tensor) – A n_samples x b x q (x m’)-dim Tensor of objective values.

• constraints (List[Callable[[Tensor], Tensor]]) – A list of callables, each mapping a Tensor of size b x q x m to a Tensor of size b x q, where negative values imply feasibility. This callable must support broadcasting. Only relevant for multi- output models (m > 1).

• samples (Tensor) – A n_samples x b x q x m Tensor of samples drawn from the posterior.

• infeasible_cost (float) – The infeasible value.

• eta (Tensor | float) – The temperature parameter of the sigmoid function. Can be either a float or a 1-dim tensor. In case of a float the same eta is used for every constraint in constraints. In case of a tensor the length of the tensor must match the number of provided constraints. The i-th constraint is then estimated with the i-th eta value.

Returns:

A n_samples x b x q (x m’)-dim tensor of feasibility-weighted objectives.

Return type:

Tensor

Rounding¶

Discretization (rounding) functions for acquisition optimization.

References

botorch.utils.rounding.approximate_round(X, tau=0.001)[source]¶

Diffentiable approximate rounding function.

This method is a piecewise approximation of a rounding function where each piece is a hyperbolic tangent function.

Parameters:

• X (Tensor) – The tensor to round to the nearest integer (element-wise).

• tau (float) – A temperature hyperparameter.

Returns:

The approximately rounded input tensor.

Return type:

Tensor

class botorch.utils.rounding.IdentitySTEFunction(*args, **kwargs)[source]¶

Bases: Function

Base class for functions using straight through gradient estimators.

This class approximates the gradient with the identity function.

static backward(ctx, grad_output)[source]¶

Use a straight-through estimator the gradient.

This uses the identity function.

Parameters:

grad_output (Tensor) – A tensor of gradients.

Returns:

The provided tensor.

Return type:

Tensor

class botorch.utils.rounding.RoundSTE(*args, **kwargs)[source]¶

Bases: IdentitySTEFunction

Round the input tensor and use a straight-through gradient estimator.

[Daulton2022bopr] proposes using this in acquisition optimization.

static forward(ctx, X)[source]¶

Round the input tensor element-wise.

Parameters:

X (Tensor) – The tensor to be rounded.

Returns:

A tensor where each element is rounded to the nearest integer.

Return type:

Tensor

class botorch.utils.rounding.OneHotArgmaxSTE(*args, **kwargs)[source]¶

Bases: IdentitySTEFunction

Discretize a continuous relaxation of a one-hot encoded categorical.

This returns a one-hot encoded categorical and use a straight-through gradient estimator via an identity function.

[Daulton2022bopr] proposes using this in acquisition optimization.

static forward(ctx, X)[source]¶

Discretize the input tensor.

This applies a argmax along the last dimensions of the input tensor and one-hot encodes the result.

Parameters:

X (Tensor) – The tensor to be rounded.

Returns:

A tensor where each element is rounded to the nearest integer.

Return type:

Tensor

Sampling¶

Utilities for MC and qMC sampling.

References

botorch.utils.sampling.manual_seed(seed=None)[source]¶

Contextmanager for manual setting the torch.random seed.

Parameters:

seed (int | None) – The seed to set the random number generator to.

Returns:

Generator

Return type:

Generator[None, None, None]

Example

>>> with manual_seed(1234):
>>>     X = torch.rand(3)

botorch.utils.sampling.draw_sobol_samples(bounds, n, q, batch_shape=None, seed=None)[source]¶

Draw qMC samples from the box defined by bounds.

Parameters:

• bounds (Tensor) – A 2 x d dimensional tensor specifying box constraints on a d-dimensional space, where bounds[0, :] and bounds[1, :] correspond to lower and upper bounds, respectively.

• n (int) – The number of (q-batch) samples. As a best practice, use powers of 2.

• q (int) – The size of each q-batch.

• batch_shape (Optional[Iterable[int], torch.Size]) – The batch shape of the samples. If given, returns samples of shape n x batch_shape x q x d, where each batch is an n x q x d-dim tensor of qMC samples.

• seed (Optional[int]) – The seed used for initializing Owen scrambling. If None (default), use a random seed.

Returns:

A n x batch_shape x q x d-dim tensor of qMC samples from the box defined by bounds.

Return type:

Tensor

Example

>>> bounds = torch.stack([torch.zeros(3), torch.ones(3)])
>>> samples = draw_sobol_samples(bounds, 16, 2)

botorch.utils.sampling.draw_sobol_normal_samples(d, n, device=None, dtype=None, seed=None)[source]¶

Draw qMC samples from a multi-variate standard normal N(0, I_d).

A primary use-case for this functionality is to compute an QMC average of f(X) over X where each element of X is drawn N(0, 1).

Parameters:

• d (int) – The dimension of the normal distribution.

• n (int) – The number of samples to return. As a best practice, use powers of 2.

• device (device | None) – The torch device.

• dtype (dtype | None) – The torch dtype.

• seed (int | None) – The seed used for initializing Owen scrambling. If None (default), use a random seed.

Returns:

A tensor of qMC standard normal samples with dimension n x d with device and dtype specified by the input.

Return type:

Tensor

Example

>>> samples = draw_sobol_normal_samples(2, 16)

botorch.utils.sampling.sample_hypersphere(d, n=1, qmc=False, seed=None, device=None, dtype=None)[source]¶

Sample uniformly from a unit d-sphere.

Parameters:

• d (int) – The dimension of the hypersphere.

• n (int) – The number of samples to return.

• qmc (bool) – If True, use QMC Sobol sampling (instead of i.i.d. uniform).

• seed (int | None) – If provided, use as a seed for the RNG.

• device (device | None) – The torch device.

• dtype (dtype | None) – The torch dtype.

Returns:

An n x d tensor of uniform samples from from the d-hypersphere.

Return type:

Tensor

Example

>>> sample_hypersphere(d=5, n=10)

botorch.utils.sampling.sample_simplex(d, n=1, qmc=False, seed=None, device=None, dtype=None)[source]¶

Sample uniformly from a d-simplex.

Parameters:

• d (int) – The dimension of the simplex.

• n (int) – The number of samples to return.

• qmc (bool) – If True, use QMC Sobol sampling (instead of i.i.d. uniform).

• seed (int | None) – If provided, use as a seed for the RNG.

• device (device | None) – The torch device.

• dtype (dtype | None) – The torch dtype.

Returns:

An n x d tensor of uniform samples from from the d-simplex.

Return type:

Tensor

Example

>>> sample_simplex(d=3, n=10)

botorch.utils.sampling.sample_polytope(A, b, x0, n=10000, n0=100, seed=None)[source]¶

Hit and run sampler from uniform sampling points from a polytope, described via inequality constraints A*x<=b.

Parameters:

• A (Tensor) – A Tensor describing inequality constraints so that all samples satisfy Ax<=b.

• b (Tensor) – A Tensor describing the inequality constraints so that all samples satisfy Ax<=b.

• x0 (Tensor) – A d-dim Tensor representing a starting point of the chain satisfying the constraints.

• n (int) – The number of resulting samples kept in the output.

• n0 (int) – The number of burn-in samples. The chain will produce n+n0 samples but the first n0 samples are not saved.

• seed (int | None) – The seed for the sampler. If omitted, use a random seed.

Returns:

(n, d) dim Tensor containing the resulting samples.

Return type:

Tensor

botorch.utils.sampling.batched_multinomial(weights, num_samples, replacement=False, generator=None, out=None)[source]¶

Sample from multinomial with an arbitrary number of batch dimensions.

Parameters:

• weights (Tensor) – A batch_shape x num_categories tensor of weights. For each batch index i, j, …, this functions samples from a multinomial with input weights[i, j, …, :]. Note that the weights need not sum to one, but must be non-negative, finite and have a non-zero sum.

• num_samples (int) – The number of samples to draw for each batch index. Must be smaller than num_categories if replacement=False.

• replacement (bool) – If True, samples are drawn with replacement.

• generator (Generator | None) – A a pseudorandom number generator for sampling.

• out (Tensor | None) – The output tensor (optional). If provided, must be of size batch_shape x num_samples.

Returns:

A batch_shape x num_samples tensor of samples.

Return type:

LongTensor

This is a thin wrapper around torch.multinomial that allows weight (input) tensors with an arbitrary number of batch dimensions (torch.multinomial only allows a single batch dimension). The calling signature is the same as for torch.multinomial.

Example

>>> weights = torch.rand(2, 3, 10)
>>> samples = batched_multinomial(weights, 4)  # shape is 2 x 3 x 4

botorch.utils.sampling.find_interior_point(A, b, A_eq=None, b_eq=None)[source]¶

Find an interior point of a polytope via linear programming.

Parameters:

• A (ndarray) – A n_ineq x d-dim numpy array containing the coefficients of the constraint inequalities.

• b (ndarray) – A n_ineq x 1-dim numpy array containing the right hand sides of the constraint inequalities.

• A_eq (ndarray | None) – A n_eq x d-dim numpy array containing the coefficients of the constraint equalities.

• b_eq (ndarray | None) – A n_eq x 1-dim numpy array containing the right hand sides of the constraint equalities.

Returns:

A d-dim numpy array containing an interior point of the polytope. This function will raise a ValueError if there is no such point.

Return type:

ndarray

This method solves the following Linear Program:

min -s subject to A @ x <= b - 2 * s, s >= 0, A_eq @ x = b_eq

In case the polytope is unbounded, then it will also constrain the slack variable s to s<=1.

class botorch.utils.sampling.HitAndRunPolytopeSampler(inequality_constraints=None, equality_constraints=None, bounds=None, interior_point=None, n_burnin=0)[source]¶

Bases: PolytopeSampler

A sampler for sampling from a polyope using a hit-and-run algorithm.

A sampler for sampling from a polyope using a hit-and-run algorithm.

Parameters:

• inequality_constraints (Optional[Tuple[Tensor, Tensor]]) – Tensors (A, b) describing inequality constraints A @ x <= b, where A is a n_ineq_con x d-dim Tensor and b is a n_ineq_con x 1-dim Tensor, with n_ineq_con the number of inequalities and d the dimension of the sample space.

• equality_constraints (Optional[Tuple[Tensor, Tensor]]) – Tensors (C, d) describing the equality constraints C @ x = d, where C is a n_eq_con x d-dim Tensor and d is a n_eq_con x 1-dim Tensor with n_eq_con the number of equalities.

• bounds (Optional[Tensor]) – A 2 x d-dim tensor of box bounds, where inf (-inf) means that the respective dimension is unbounded from above (below).

• interior_point (Optional[Tensor]) – A d x 1-dim Tensor representing a point in the (relative) interior of the polytope. If omitted, determined automatically by solving a Linear Program.

• n_burnin (int) – The number of burn in samples.

draw(n=1, seed=None)[source]¶

Draw samples from the polytope.

Parameters:

• n (int) – The number of samples.

• seed (int | None) – The random seed.

Returns:

A n x d Tensor of samples from the polytope.

Return type:

Tensor

class botorch.utils.sampling.DelaunayPolytopeSampler(inequality_constraints=None, equality_constraints=None, bounds=None, interior_point=None)[source]¶

Bases: PolytopeSampler

A polytope sampler using Delaunay triangulation.

This sampler first enumerates the vertices of the constraint polytope and then uses a Delaunay triangulation to tesselate its convex hull.

The sampling happens in two stages: 1. First, we sample from the set of hypertriangles generated by the Delaunay triangulation (i.e. which hyper-triangle to draw the sample from) with probabilities proportional to the triangle volumes. 2. Then, we sample uniformly from the chosen hypertriangle by sampling uniformly from the unit simplex of the appropriate dimension, and then computing the convex combination of the vertices of the hypertriangle according to that draw from the simplex.

The best reference (not exactly the same, but functionally equivalent) is [Trikalinos2014polytope]. A simple R implementation is available at https://github.com/gertvv/tesselample.

Initialize DelaunayPolytopeSampler.

Parameters:

• inequality_constraints (Optional[Tuple[Tensor, Tensor]]) – Tensors (A, b) describing inequality constraints A @ x <= b, where A is a n_ineq_con x d-dim Tensor and b is a n_ineq_con x 1-dim Tensor, with n_ineq_con the number of inequalities and d the dimension of the sample space.

• equality_constraints (Optional[Tuple[Tensor, Tensor]]) – Tensors (C, d) describing the equality constraints C @ x = d, where C is a n_eq_con x d-dim Tensor and d is a n_eq_con x 1-dim Tensor with n_eq_con the number of equalities.

• bounds (Optional[Tensor]) – A 2 x d-dim tensor of box bounds, where inf (-inf) means that the respective dimension is unbounded from above (below).

• interior_point (Optional[Tensor]) – A d x 1-dim Tensor representing a point in the (relative) interior of the polytope. If omitted, determined automatically by solving a Linear Program.

Warning: The vertex enumeration performed in this algorithm can become extremely costly if there are a large number of inequalities. Similarly, the triangulation can get very expensive in high dimensions. Only use this algorithm for moderate dimensions / moderately complex constraint sets. An alternative is the HitAndRunPolytopeSampler.

draw(n=1, seed=None)[source]¶

Draw samples from the polytope.

Parameters:

• n (int) – The number of samples.

• seed (int | None) – The random seed.

Returns:

A n x d Tensor of samples from the polytope.

Return type:

Tensor

botorch.utils.sampling.normalize_linear_constraints(bounds, constraints)[source]¶

Normalize linear constraints to the unit cube.

Parameters:

• bounds (Tensor) – A 2 x d-dim tensor containing the box bounds.

• constraints (List[Tuple[Tensor, Tensor, float]]) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs or sum_i (X[indices[i]] * coefficients[i]) = rhs.

Return type:

List[Tuple[Tensor, Tensor, float]]

botorch.utils.sampling.get_polytope_samples(n, bounds, inequality_constraints=None, equality_constraints=None, seed=None, thinning=32, n_burnin=10000)[source]¶

Sample from polytope defined by box bounds and (in)equality constraints.

This uses a hit-and-run Markov chain sampler.

TODO: make this method return the sampler object, to avoid doing burn-in every time we draw samples.

Parameters:

• n (int) – The number of samples.

• bounds (Tensor) – A 2 x d-dim tensor containing the box bounds.

• constraints (equality) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs.

• constraints – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) = rhs.

• seed (int | None) – The random seed.

• thinning (int) – The amount of thinning.

• n_burnin (int) – The number of burn-in samples for the Markov chain sampler.

• inequality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

• equality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

Returns:

A n x d-dim tensor of samples.

Return type:

Tensor

botorch.utils.sampling.sparse_to_dense_constraints(d, constraints)[source]¶

Convert parameter constraints from a sparse format into a dense format.

This method converts sparse triples of the form (indices, coefficients, rhs) to constraints of the form Ax >= b or Ax = b.

Parameters:

• d (int) – The input dimension.

• constraints (inequality) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an (in)equality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs or sum_i (X[indices[i]] * coefficients[i]) = rhs.

Returns:

• A: A n_constraints x d-dim tensor of coefficients.

• b: A n_constraints x 1-dim tensor of right hand sides.

Return type:

A two-element tuple containing

botorch.utils.sampling.optimize_posterior_samples(paths, bounds, candidates=None, raw_samples=1024, num_restarts=20, maximize=True, **kwargs)[source]¶

Cheaply maximizes posterior samples by random querying followed by vanilla gradient descent on the best num_restarts points.

Parameters:

• paths (SamplePath) – Random Fourier Feature-based sample paths from the GP

• bounds (Tensor) – The bounds on the search space.

• candidates (Optional[Tensor]) – A priori good candidates (typically previous design points) which acts as extra initial guesses for the optimization routine.

• raw_samples (Optional[int]) – The number of samples with which to query the samples initially.

• num_restarts (int) – The number of points selected for gradient-based optimization.

• maximize (bool) – Boolean indicating whether to maimize or minimize

• kwargs (Any) –

Returns:

• X_opt: A num_optima x [batch_size] x d-dim tensor of optimal inputs x*.

• f_opt: A num_optima x [batch_size] x 1-dim tensor of optimal outputs f*.

Return type:

A two-element tuple containing

Sampling from GP priors¶

class botorch.utils.gp_sampling.GPDraw(model, seed=None)[source]¶

Bases: Module

Convenience wrapper for sampling a function from a GP prior.

This wrapper implicitly defines the GP sample as a self-updating function by keeping track of the evaluated points and respective base samples used during the evaluation.

This does not yet support multi-output models.

Construct a GP function sampler.

Parameters:

• model (Model) – The Model defining the GP prior.

• seed (Optional[int]) –

property Xs: Tensor¶

A (batch_shape) x n_eval x d-dim tensor of locations at which the GP was evaluated (or None if the sample has never been evaluated).

property Ys: Tensor¶

A (batch_shape) x n_eval x d-dim tensor of associated function values (or None if the sample has never been evaluated).

forward(X)[source]¶

Evaluate the GP sample function at a set of points X.

Parameters:

X (Tensor) – A batch_shape x n x d-dim tensor of points

Returns:

The value of the GP sample at the n points.

Return type:

Tensor

training: bool¶

class botorch.utils.gp_sampling.RandomFourierFeatures(kernel, input_dim, num_rff_features, sample_shape=None)[source]¶

Bases: Module

A class that represents Random Fourier Features.

Initialize RandomFourierFeatures.

Parameters:

• kernel (Kernel) – The GP kernel.

• input_dim (int) – The input dimension to the GP kernel.

• num_rff_features (int) – The number of Fourier features.

• sample_shape (Optional[torch.Size]) – The shape of a single sample. For a single-element torch.Size object, this is simply the number of RFF draws.

forward(X)[source]¶

Get Fourier basis features for the provided inputs.

Note that the right-most subset of the batch shape of X should be (sample_shape) x (kernel_batch_shape) if using either the sample_shape argument or a batched kernel. In other words, X should be of shape (added_batch_shape) x (sample_shape) x (kernel_batch_shape) x n x input_dim, where parantheses denote that the given batch shape can be empty. X can always be a tensor of shape n x input_dim, in which case broadcasting will take care of the batch shape. This will raise a ValueError if the batch shapes are not compatible.

Parameters:

X (Tensor) – Input tensor of shape (batch_shape) x n x input_dim.

Returns:

A Tensor of shape (batch_shape) x n x rff. If X does not have a batch_shape, the output batch_shape will be (sample_shape) x (kernel_batch_shape).

Return type:

Tensor

training: bool¶

botorch.utils.gp_sampling.get_deterministic_model_multi_samples(weights, bases)[source]¶

Get a batched deterministic model that batch evaluates n_samples function samples. This supports multi-output models as well.

Parameters:

• weights (List[Tensor]) – A list of weights with num_outputs elements. Each weight is of shape (batch_shape_input) x n_samples x num_rff_features, where (batch_shape_input) is the batch shape of the inputs used to obtain the posterior weights.

• bases (List[RandomFourierFeatures]) – A list of RandomFourierFeatures with num_outputs elements. Each basis has a sample shape of n_samples.

• n_samples – The number of function samples.

Returns:

A batched GenericDeterministicModel`s that batch evaluates `n_samples function samples.

Return type:

GenericDeterministicModel

botorch.utils.gp_sampling.get_eval_gp_sample_callable(w, basis)[source]¶

Parameters:

• w (Tensor) –

• basis (RandomFourierFeatures) –

Return type:

Tensor

botorch.utils.gp_sampling.get_deterministic_model(weights, bases)[source]¶

Get a deterministic model using the provided weights and bases for each output.

Parameters:

• weights (List[Tensor]) – A list of weights with m elements.

• bases (List[RandomFourierFeatures]) – A list of RandomFourierFeatures with m elements.

Returns:

A deterministic model.

Return type:

GenericDeterministicModel

botorch.utils.gp_sampling.get_deterministic_model_list(weights, bases)[source]¶

Get a deterministic model list using the provided weights and bases for each output.

Parameters:

• weights (List[Tensor]) – A list of weights with m elements.

• bases (List[RandomFourierFeatures]) – A list of RandomFourierFeatures with m elements.

Returns:

A deterministic model.

Return type:

ModelList

botorch.utils.gp_sampling.get_weights_posterior(X, y, sigma_sq)[source]¶

Sample bayesian linear regression weights.

Parameters:

• X (Tensor) – A tensor of inputs with shape (*batch_shape, n num_rff_features).

• y (Tensor) – A tensor of outcomes with shape (*batch_shape, n).

• sigma_sq (Tensor) – The likelihood noise variance. This should be a tensor with shape kernel_batch_shape, 1, 1 if using a batched kernel. Otherwise, it should be a scalar tensor.

Returns:

The posterior distribution over the weights.

Return type:

MultivariateNormal

botorch.utils.gp_sampling.get_gp_samples(model, num_outputs, n_samples, num_rff_features=512)[source]¶

Sample functions from GP posterior using RFFs. The returned GenericDeterministicModel effectively wraps num_outputs models, each of which has a batch shape of n_samples. Refer get_deterministic_model_multi_samples for more details.

NOTE: If using input / outcome transforms, the gp samples must be accessed via the gp_sample.posterior(X) call. Otherwise, gp_sample(X) will produce bogus values that do not agree with the underlying model. It is also highly recommended to use outcome transforms to standardize the input data, since the gp samples do not work well when training outcomes are not zero-mean.

Parameters:

• model (Model) – The model.

• num_outputs (int) – The number of outputs.

• n_samples (int) – The number of functions to be sampled IID.

• num_rff_features (int) – The number of random Fourier features.

Returns:

A GenericDeterministicModel that evaluates n_samples sampled functions. If n_samples > 1, this will be a batched model.

Return type:

GenericDeterministicModel

Testing¶

class botorch.utils.testing.BotorchTestCase(methodName='runTest')[source]¶

Bases: TestCase

Basic test case for Botorch.

This

1. sets the default device to be torch.device(“cpu”)

2. ensures that no warnings are suppressed by default.

Create an instance of the class that will use the named test method when executed. Raises a ValueError if the instance does not have a method with the specified name.

device = device(type='cpu')¶

setUp(suppress_input_warnings=True)[source]¶

Hook method for setting up the test fixture before exercising it.

Parameters:

suppress_input_warnings (bool) –

Return type:

None

assertAllClose(input, other, rtol=1e-05, atol=1e-08, equal_nan=False)[source]¶

Calls torch.testing.assert_close, using the signature and default behavior of torch.allclose.

Example output:

AssertionError: Scalars are not close!

Absolute difference: 1.0000034868717194 (up to 0.0001 allowed) Relative difference: 0.8348668001940709 (up to 1e-05 allowed)

Parameters:

• input (Tensor) –

• other (Tensor | float) –

• rtol (float) –

• atol (float) –

• equal_nan (bool) –

Return type:

None

class botorch.utils.testing.BaseTestProblemTestCaseMixIn[source]¶

Bases: object

test_forward()[source]¶

abstract property functions: List[BaseTestProblem]¶

class botorch.utils.testing.SyntheticTestFunctionTestCaseMixin[source]¶

Bases: object

test_optimal_value()[source]¶

test_optimizer()[source]¶

class botorch.utils.testing.MultiObjectiveTestProblemTestCaseMixin[source]¶

Bases: object

test_attributes()[source]¶

test_max_hv()[source]¶

test_ref_point()[source]¶

class botorch.utils.testing.ConstrainedTestProblemTestCaseMixin[source]¶

Bases: object

test_num_constraints()[source]¶

test_evaluate_slack_true()[source]¶

class botorch.utils.testing.MockPosterior(mean=None, variance=None, samples=None, base_shape=None, batch_range=None)[source]¶

Bases: Posterior

Mock object that implements dummy methods and feeds through specified outputs

Parameters:

• mean – The mean of the posterior.

• variance – The variance of the posterior.

• samples – Samples to return from rsample, unless base_samples is provided.

• base_shape – If given, this is returned as base_sample_shape, and also used as the base of the _extended_shape.

• batch_range – If given, this is returned as batch_range. Defaults to (0, -2).

property device: device¶

The torch device of the distribution.

property dtype: dtype¶

The torch dtype of the distribution.

property batch_shape: Size¶

property base_sample_shape: Size¶

The base shape of the base samples expected in rsample.

Informs the sampler to produce base samples of shape sample_shape x base_sample_shape.

property batch_range: Tuple[int, int]¶

The t-batch range.

This is used in samplers to identify the t-batch component of the base_sample_shape. The base samples are expanded over the t-batches to provide consistency in the acquisition values, i.e., to ensure that a candidate produces same value regardless of its position on the t-batch.

property mean¶

property variance¶

rsample(sample_shape=None, base_samples=None)[source]¶

Mock sample by repeating self._samples. If base_samples is provided, do a shape check but return the same mock samples.

Parameters:

• sample_shape (Size | None) –

• base_samples (Tensor | None) –

Return type:

Tensor

rsample_from_base_samples(sample_shape, base_samples)[source]¶

Sample from the posterior (with gradients) using base samples.

This is intended to be used with a sampler that produces the corresponding base samples, and enables acquisition optimization via Sample Average Approximation.

Parameters:

• sample_shape (Size) – A torch.Size object specifying the sample shape. To draw n samples, set to torch.Size([n]). To draw b batches of n samples each, set to torch.Size([b, n]).

• base_samples (Tensor) – The base samples, obtained from the appropriate sampler. This is a tensor of shape sample_shape x base_sample_shape.

Returns:

Samples from the posterior, a tensor of shape self._extended_shape(sample_shape=sample_shape).

Return type:

Tensor

class botorch.utils.testing.MockModel(posterior)[source]¶

Bases: Model, FantasizeMixin

Mock object that implements dummy methods and feeds through specified outputs

Initializes internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

posterior (MockPosterior) –

posterior(X, output_indices=None, posterior_transform=None, observation_noise=False)[source]¶

Computes the posterior over model outputs at the provided points.

Note: The input transforms should be applied here using

self.transform_inputs(X) after the self.eval() call and before any model.forward or model.likelihood calls.

Parameters:

• X (Tensor) – A b x q x d-dim Tensor, where d is the dimension of the feature space, q is the number of points considered jointly, and b is the batch dimension.

• output_indices (List[int] | None) – A list of indices, corresponding to the outputs over which to compute the posterior (if the model is multi-output). Can be used to speed up computation if only a subset of the model’s outputs are required for optimization. If omitted, computes the posterior over all model outputs.

• observation_noise (bool) – For models with an inferred noise level, if True, include observation noise. For models with an observed noise level, this must be a model_batch_shape x 1 x m-dim tensor or a model_batch_shape x n’ x m-dim tensor containing the average noise for each batch and output. noise must be in the outcome-transformed space if an outcome transform is used.

• posterior_transform (PosteriorTransform | None) – An optional PosteriorTransform.

Returns:

A Posterior object, representing a batch of b joint distributions over q points and m outputs each.

Return type:

MockPosterior

property num_outputs: int¶

The number of outputs of the model.

property batch_shape: Size¶

The batch shape of the model.

This is a batch shape from an I/O perspective, independent of the internal representation of the model (as e.g. in BatchedMultiOutputGPyTorchModel). For a model with m outputs, a test_batch_shape x q x d-shaped input X to the posterior method returns a Posterior object over an output of shape broadcast(test_batch_shape, model.batch_shape) x q x m.

state_dict()[source]¶

Returns a dictionary containing references to the whole state of the module.

Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included.

Note

The returned object is a shallow copy. It contains references to the module’s parameters and buffers.

Warning

Currently state_dict() also accepts positional arguments for destination, prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases.

Warning

Please avoid the use of argument destination as it is not designed for end-users.

Parameters:

• destination (dict, optional) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None.

• prefix (str, optional) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: ''.

• keep_vars (bool, optional) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True, detaching will not be performed. Default: False.

Returns:

a dictionary containing a whole state of the module

Return type:

dict

Example:

>>> # xdoctest: +SKIP("undefined vars")
>>> module.state_dict().keys()
['bias', 'weight']

load_state_dict(state_dict=None, strict=False)[source]¶

Copies parameters and buffers from state_dict into this module and its descendants. If strict is True, then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function.

Warning

If assign is True the optimizer must be created after the call to load_state_dict.

Parameters:

• state_dict (dict) – a dict containing parameters and persistent buffers.

• strict (bool, optional) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True

• assign (bool, optional) – whether to assign items in the state dictionary to their corresponding keys in the module instead of copying them inplace into the module’s current parameters and buffers. When False, the properties of the tensors in the current module are preserved while when True, the properties of the Tensors in the state dict are preserved. Default: False

Returns:

• missing_keys is a list of str containing the missing keys

• unexpected_keys is a list of str containing the unexpected keys

Return type:

NamedTuple with missing_keys and unexpected_keys fields

Note

If a parameter or buffer is registered as None and its corresponding key exists in state_dict, load_state_dict() will raise a RuntimeError.

class botorch.utils.testing.MockAcquisitionFunction[source]¶

Bases: object

Mock acquisition function object that implements dummy methods.

set_X_pending(X_pending=None)[source]¶

Parameters:

X_pending (Tensor | None) –

Test Helpers¶

Dummy classes and other helpers that are used in multiple test files should be defined here to avoid relative imports.

botorch.utils.test_helpers.get_sample_moments(samples, sample_shape)[source]¶

Computes the mean and covariance of a set of samples.

Parameters:

• samples (Tensor) – A tensor of shape sample_shape x batch_shape x q.

• sample_shape (Size) – The sample_shape input used while generating the samples using the pathwise sampling API.

Return type:

Tuple[Tensor, Tensor]

botorch.utils.test_helpers.standardize_moments(transform, loc, covariance_matrix)[source]¶

Standardizes the loc and covariance_matrix using the mean and standard deviations from a Standardize transform.

Parameters:

• transform (Standardize) –

• loc (Tensor) –

• covariance_matrix (Tensor) –

Return type:

Tuple[Tensor, Tensor]

botorch.utils.test_helpers.gen_multi_task_dataset(yvar=None, **tkwargs)[source]¶

Constructs a multi-task dataset with two tasks, each with 10 data points.

Parameters:

yvar (float | None) –

Return type:

Tuple[MultiTaskDataset, Tuple[Tensor, Tensor, Tensor]]

botorch.utils.test_helpers.get_pvar_expected(posterior, model, X, m)[source]¶

Computes the expected variance of a posterior after adding the predictive noise from the likelihood.

Parameters:

• posterior (Posterior) –

• model (Model) –

• X (Tensor) –

• m (int) –

Return type:

Tensor

Torch¶

class botorch.utils.torch.BufferDict(buffers=None)[source]¶

Bases: Module

Holds buffers in a dictionary.

BufferDict can be indexed like a regular Python dictionary, but buffers it contains are properly registered, and will be visible by all Module methods.

BufferDict is an ordered dictionary that respects

• the order of insertion, and

• in update(), the order of the merged OrderedDict or another BufferDict (the argument to update()).

Note that update() with other unordered mapping types (e.g., Python’s plain dict) does not preserve the order of the merged mapping.

Parameters:

buffers (iterable, optional) – a mapping (dictionary) of (string : Tensor) or an iterable of key-value pairs of type (string, Tensor)

Example:

class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        self.buffers = nn.BufferDict({
                'left': torch.randn(5, 10),
                'right': torch.randn(5, 10)
        })

    def forward(self, x, choice):
        x = self.buffers[choice].mm(x)
        return x

Parameters:

buffers – A mapping (dictionary) from string to Tensor, or an iterable of key-value pairs of type (string, Tensor).

clear()[source]¶

Remove all items from the BufferDict.

pop(key)[source]¶

Remove key from the BufferDict and return its buffer.

Parameters:

key (string) – key to pop from the BufferDict

keys()[source]¶

Return an iterable of the BufferDict keys.

items()[source]¶

Return an iterable of the BufferDict key/value pairs.

values()[source]¶

Return an iterable of the BufferDict values.

update(buffers)[source]¶

Update the BufferDict with the key-value pairs from a mapping or an iterable, overwriting existing keys.

Note

If buffers is an OrderedDict, a BufferDict, or an iterable of key-value pairs, the order of new elements in it is preserved.

Parameters:

buffers (iterable) – a mapping (dictionary) from string to Tensor, or an iterable of key-value pairs of type (string, Tensor)

extra_repr()[source]¶

Set the extra representation of the module

To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable.

training: bool¶

Transformations¶

Some basic data transformation helpers.

botorch.utils.transforms.standardize(Y)[source]¶

Standardizes (zero mean, unit variance) a tensor by dim=-2.

If the tensor is single-dimensional, simply standardizes the tensor. If for some batch index all elements are equal (or if there is only a single data point), this function will return 0 for that batch index.

Parameters:

Y (Tensor) – A batch_shape x n x m-dim tensor.

Returns:

The standardized Y.

Return type:

Tensor

Example

>>> Y = torch.rand(4, 3)
>>> Y_standardized = standardize(Y)

botorch.utils.transforms.normalize(X, bounds)[source]¶

Min-max normalize X w.r.t. the provided bounds.

Parameters:

• X (Tensor) – … x d tensor of data

• bounds (Tensor) – 2 x d tensor of lower and upper bounds for each of the X’s d columns.

Returns:

A … x d-dim tensor of normalized data, given by

(X - bounds[0]) / (bounds[1] - bounds[0]). If all elements of X are contained within bounds, the normalized values will be contained within [0, 1]^d.

Return type:

Tensor

Example

>>> X = torch.rand(4, 3)
>>> bounds = torch.stack([torch.zeros(3), 0.5 * torch.ones(3)])
>>> X_normalized = normalize(X, bounds)

botorch.utils.transforms.unnormalize(X, bounds)[source]¶

Un-normalizes X w.r.t. the provided bounds.

Parameters:

• X (Tensor) – … x d tensor of data

• bounds (Tensor) – 2 x d tensor of lower and upper bounds for each of the X’s d columns.

Returns:

A … x d-dim tensor of unnormalized data, given by

X * (bounds[1] - bounds[0]) + bounds[0]. If all elements of X are contained in [0, 1]^d, the un-normalized values will be contained within bounds.

Return type:

Tensor

Example

>>> X_normalized = torch.rand(4, 3)
>>> bounds = torch.stack([torch.zeros(3), 0.5 * torch.ones(3)])
>>> X = unnormalize(X_normalized, bounds)

botorch.utils.transforms.normalize_indices(indices, d)[source]¶

Normalize a list of indices to ensure that they are positive.

Parameters:

• indices (List[int] | None) – A list of indices (may contain negative indices for indexing “from the back”).

• d (int) – The dimension of the tensor to index.

Returns:

A normalized list of indices such that each index is between 0 and d-1, or None if indices is None.

Return type:

List[int] | None

botorch.utils.transforms.is_fully_bayesian(model)[source]¶

Check if at least one model is a fully Bayesian model.

Parameters:

model (Model) – A BoTorch model (may be a ModelList or ModelListGP)

Returns:

True if at least one model is a fully Bayesian model.

Return type:

bool

botorch.utils.transforms.is_ensemble(model)[source]¶

Check if at least one model is an ensemble model.

Parameters:

model (Model) – A BoTorch model (may be a ModelList or ModelListGP)

Returns:

True if at least one model is an ensemble model.

Return type:

bool

botorch.utils.transforms.t_batch_mode_transform(expected_q=None, assert_output_shape=True)[source]¶

Factory for decorators enabling consistent t-batch behavior.

This method creates decorators for instance methods to transform an input tensor X to t-batch mode (i.e. with at least 3 dimensions). This assumes the tensor has a q-batch dimension. The decorator also checks the q-batch size if expected_q is provided, and the output shape if assert_output_shape is True.

Parameters:

• expected_q (Optional[int]) – The expected q-batch size of X. If specified, this will raise an AssertionError if X’s q-batch size does not equal expected_q.

• assert_output_shape (bool) – If True, this will raise an AssertionError if the output shape does not match either the t-batch shape of X, or the acqf.model.batch_shape for acquisition functions using batched models.

Returns:

The decorated instance method.

Return type:

Callable[[Callable[[AcquisitionFunction, Any], Any]], Callable[[AcquisitionFunction, Any], Any]]

Example

>>> class ExampleClass:
>>>     @t_batch_mode_transform(expected_q=1)
>>>     def single_q_method(self, X):
>>>         ...
>>>
>>>     @t_batch_mode_transform()
>>>     def arbitrary_q_method(self, X):
>>>         ...

botorch.utils.transforms.concatenate_pending_points(method)[source]¶

Decorator concatenating X_pending into an acquisition function’s argument.

This decorator works on the forward method of acquisition functions taking a tensor X as the argument. If the acquisition function has an X_pending attribute (that is not None), this is concatenated into the input X, appropriately expanding the pending points to match the batch shape of X.

Example

>>> class ExampleAcquisitionFunction:
>>>     @concatenate_pending_points
>>>     @t_batch_mode_transform()
>>>     def forward(self, X):
>>>         ...

Parameters:

method (Callable[[Any, Tensor], Any]) –

Return type:

Callable[[Any, Tensor], Any]

botorch.utils.transforms.match_batch_shape(X, Y)[source]¶

Matches the batch dimension of a tensor to that of another tensor.

Parameters:

• X (Tensor) – A batch_shape_X x q x d tensor, whose batch dimensions that correspond to batch dimensions of Y are to be matched to those (if compatible).

• Y (Tensor) – A batch_shape_Y x q’ x d tensor.

Returns:

A batch_shape_Y x q x d tensor containing the data of X expanded to the batch dimensions of Y (if compatible). For instance, if X is b’’ x b’ x q x d and Y is b x q x d, then the returned tensor is b’’ x b x q x d.

Return type:

Tensor

Example

>>> X = torch.rand(2, 1, 5, 3)
>>> Y = torch.rand(2, 6, 4, 3)
>>> X_matched = match_batch_shape(X, Y)
>>> X_matched.shape
torch.Size([2, 6, 5, 3])

botorch.utils.transforms.convert_to_target_pre_hook(module, *args)[source]¶

Pre-hook for automatically calling .to(X) on module prior to forward

Feasible Volume¶

botorch.utils.feasible_volume.get_feasible_samples(samples, inequality_constraints=None)[source]¶

Checks which of the samples satisfy all of the inequality constraints.

Parameters:

• samples (Tensor) – A sample size x d size tensor of feature samples, where d is a feature dimension.

• constraints (inequality) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs.

• inequality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

Returns:

2-element tuple containing

• Samples satisfying the linear constraints.

• Estimated proportion of samples satisfying the linear constraints.

Return type:

Tuple[Tensor, float]

botorch.utils.feasible_volume.get_outcome_feasibility_probability(model, X, outcome_constraints, threshold=0.1, nsample_outcome=1000, seed=None)[source]¶

Monte Carlo estimate of the feasible volume with respect to the outcome constraints.

Parameters:

• model (Model) – The model used for sampling the posterior.

• X (Tensor) – A tensor of dimension batch-shape x 1 x d, where d is feature dimension.

• outcome_constraints (List[Callable[[Tensor], Tensor]]) – A list of callables, each mapping a Tensor of dimension sample_shape x batch-shape x q x m to a Tensor of dimension sample_shape x batch-shape x q, where negative values imply feasibility.

• threshold (float) – A lower limit for the probability of posterior samples feasibility.

• nsample_outcome (int) – The number of samples from the model posterior.

• seed (int | None) – The seed for the posterior sampler. If omitted, use a random seed.

Returns:

Estimated proportion of features for which posterior samples satisfy given outcome constraints with probability above or equal to the given threshold.

Return type:

float

botorch.utils.feasible_volume.estimate_feasible_volume(bounds, model, outcome_constraints, inequality_constraints=None, nsample_feature=1000, nsample_outcome=1000, threshold=0.1, verbose=False, seed=None, device=None, dtype=None)[source]¶

Monte Carlo estimate of the feasible volume with respect to feature constraints and outcome constraints.

Parameters:

• bounds (Tensor) – A 2 x d tensor of lower and upper bounds for each column of X.

• model (Model) – The model used for sampling the outcomes.

• outcome_constraints (List[Callable[[Tensor], Tensor]]) – A list of callables, each mapping a Tensor of dimension sample_shape x batch-shape x q x m to a Tensor of dimension sample_shape x batch-shape x q, where negative values imply feasibility.

• constraints (inequality) – A list of tuples (indices, coefficients, rhs), with each tuple encoding an inequality constraint of the form sum_i (X[indices[i]] * coefficients[i]) >= rhs.

• nsample_feature (int) – The number of feature samples satisfying the bounds.

• nsample_outcome (int) – The number of outcome samples from the model posterior.

• threshold (float) – A lower limit for the probability of outcome feasibility

• seed (int | None) – The seed for both feature and outcome samplers. If omitted, use a random seed.

• verbose (bool) – An indicator for whether to log the results.

• inequality_constraints (List[Tuple[Tensor, Tensor, float]] | None) –

• device (device | None) –

• dtype (dtype | None) –

Returns:

• Estimated proportion of volume in feature space that is

  feasible wrt the bounds and the inequality constraints (linear).

• Estimated proportion of feasible features for which

  posterior samples (outcome) satisfies the outcome constraints with probability above the given threshold.

Return type:

2-element tuple containing

Types and Type Hints¶

class botorch.utils.types.DEFAULT¶

Bases: object

Constants¶

botorch.utils.constants.get_constants(values, device=None, dtype=None)[source]¶

Returns scalar-valued Tensors containing each of the given constants. Used to expedite tensor operations involving scalar arithmetic. Note that the returned Tensors should not be modified in-place.

Parameters:

• values (Number | Iterator[Number]) –

• device (device | None) –

• dtype (dtype | None) –

Return type:

Tensor | Tuple[Tensor, …]

botorch.utils.constants.get_constants_like(values, ref)[source]¶

Parameters:

• values (Number | Iterator[Number]) –

• ref (Tensor) –

Return type:

Tensor | Iterator[Tensor]

Safe Math¶

Special implementations of mathematical functions that solve numerical issues of naive implementations.

botorch.utils.safe_math.exp(x, **kwargs)[source]¶

Parameters:

x (Tensor) –

Return type:

Tensor

botorch.utils.safe_math.log(x, **kwargs)[source]¶

Parameters:

x (Tensor) –

Return type:

Tensor

botorch.utils.safe_math.add(a, b, **kwargs)[source]¶

Parameters:

• a (Tensor) –

• b (Tensor) –

Return type:

Tensor

botorch.utils.safe_math.sub(a, b)[source]¶

Parameters:

• a (Tensor) –

• b (Tensor) –

Return type:

Tensor

botorch.utils.safe_math.div(a, b)[source]¶

Parameters:

• a (Tensor) –

• b (Tensor) –

Return type:

Tensor

botorch.utils.safe_math.mul(a, b)[source]¶

Parameters:

• a (Tensor) –

• b (Tensor) –

Return type:

Tensor

botorch.utils.safe_math.log1mexp(x)[source]¶

Numerically accurate evaluation of log(1 - exp(x)) for x < 0. See [Maechler2012accurate] for details.

Parameters:

x (Tensor) –

Return type:

Tensor

botorch.utils.safe_math.log1pexp(x)[source]¶

Numerically accurate evaluation of log(1 + exp(x)). See [Maechler2012accurate] for details.

Parameters:

x (Tensor) –

Return type:

Tensor

botorch.utils.safe_math.logexpit(X)[source]¶

Computes the logarithm of the expit (a.k.a. sigmoid) function.

Parameters:

X (Tensor) –

Return type:

Tensor

botorch.utils.safe_math.logplusexp(a, b)[source]¶

Computes log(exp(a) + exp(b)) similar to logsumexp.

Parameters:

• a (Tensor) –

• b (Tensor) –

Return type:

Tensor