botorch.models<a class="headerlink" href="#module-botorch.models" title="Permalink to this headline">¶

fantasize(X, sampler, observation_noise=True, **kwargs)[source]¶

Construct a fantasy model.

Constructs a fantasy model in the following fashion: (1) compute the model posterior at X (including observation noise if observation_noise=True). (2) sample from this posterior (using sampler) to generate “fake” observations. (3) condition the model on the new fake observations.

Parameters

X (torch.Tensor) – A batch_shape x n’ x d-dim Tensor, where d is the dimension of the feature space, n’ is the number of points per batch, and batch_shape is the batch shape (must be compatible with the batch shape of the model).
sampler (botorch.sampling.samplers.MCSampler) – The sampler used for sampling from the posterior at X.
observation_noise (bool) – If True, include observation noise.
kwargs (Any) –

Returns

The constructed fantasy model.

Return type

botorch.posteriors.posterior.Posterior

classmethod construct_inputs(training_data, **kwargs)[source]¶

Construct kwargs for the Model from TrainingData and other options.

Parameters

training_data (botorch.utils.containers.TrainingData) –
kwargs (Any) –

Return type

Dict[str, Any]

transform_inputs(X, input_transform=None)[source]¶

Transform inputs.

Parameters

X (torch.Tensor) – A tensor of inputs
input_transform (Optional[torch.nn.modules.module.Module]) – A Module that performs the input transformation.

Returns

A tensor of transformed inputs

Return type

torch.Tensor

eval()[source]¶

Puts the model in eval mode and sets the transformed inputs.

Return type: botorch.models.model.Model

train(mode=True)[source]¶

Puts the model in train mode and reverts to the original inputs.

Parameters: mode (bool) – A boolean denoting whether to put in train or eval mode. If False, model is put in eval mode.
Return type: botorch.models.model.Model

class botorch.models.model.ModelList(*models)[source]¶

Bases: botorch.models.model.Model

Container for a list of models.

A multi-output Model represented by a list of independent models.

Parameters

*models – A variable number of models.
models (Model) –

Return type

None

Example

>>> m_1 = SingleTaskGP(train_X, train_Y
>>> m_2 = GenericDeterministicModel(lambda x: x.sum(dim=-1))
>>> m_12 = ModelList(m_1, m_2)
>>> m_12.predict(test_X)

posterior(X, output_indices=None, observation_noise=False, **kwargs)[source]¶

Computes the posterior over model outputs at the provided points.

Note: The input transforms should be applied here using: self.transform_inputs(X) after the self.eval() call and before any model.forward or model.likelihood calls.

Parameters

X (torch.Tensor) – A b x q x d-dim Tensor, where d is the dimension of the feature space, q is the number of points considered jointly, and b is the batch dimension.
output_indices (Optional[List[int]]) – A list of indices, corresponding to the outputs over which to compute the posterior (if the model is multi-output). Can be used to speed up computation if only a subset of the model’s outputs are required for optimization. If omitted, computes the posterior over all model outputs.
observation_noise (bool) – If True, add observation noise to the posterior.
kwargs (Any) –

Returns

A Posterior object, representing a batch of b joint distributions over q points and m outputs each.

Return type

property batch_shape: torch.Size¶

The batch shape of the model.

This is a batch shape from an I/O perspective, independent of the internal representation of the model (as e.g. in BatchedMultiOutputGPyTorchModel). For a model with m outputs, a test_batch_shape x q x d-shaped input X to the posterior method returns a Posterior object over an output of shape broadcast(test_batch_shape, model.batch_shape) x q x m.

property num_outputs: int¶

The number of outputs of the model.

Equal to the sum of the number of outputs of the individual models in the ModelList.

subset_output(idcs)[source]¶

Subset the model along the output dimension.

Parameters: idcs (List[int]) – The output indices to subset the model to. Relative to the overall number of outputs of the model.
Returns: A Model (either a ModelList or one of the submodels) with the outputs subset to the indices in idcs.
Return type: botorch.models.model.Model

Internally, this drops (if single-output) or subsets (if multi-output) the constitutent models and returns them as a ModelList. If the result is a single (possibly subset) model from the list, returns this model (instead of forming a degenerate singe-model ModelList). For instance, if m = ModelList(m1, m2) with m1 a two-output model and m2 a single-output model, then m.subset_output([1]) ` will return the model `m1 subset to its second output.

transform_inputs(X)[source]¶

Individually transform the inputs for each model.

Parameters: X (torch.Tensor) – A tensor of inputs.
Returns: A list of tensors of transformed inputs.
Return type: List[torch.Tensor]

GPyTorch Model API¶

Abstract model class for all GPyTorch-based botorch models.

To implement your own, simply inherit from both the provided classes and a GPyTorch Model class such as an ExactGP.

class botorch.models.gpytorch.GPyTorchModel[source]¶

Bases: botorch.models.model.Model, abc.ABC

Abstract base class for models based on GPyTorch models.

The easiest way to use this is to subclass a model from a GPyTorch model class (e.g. an ExactGP) and this GPyTorchModel. See e.g. SingleTaskGP.

Initializes internal Module state, shared by both nn.Module and ScriptModule.

Return type: None

property batch_shape: torch.Size¶

The batch shape of the model.

This is a batch shape from an I/O perspective, independent of the internal representation of the model (as e.g. in BatchedMultiOutputGPyTorchModel). For a model with m outputs, a test_batch_shape x q x d-shaped input X to the posterior method returns a Posterior object over an output of shape broadcast(test_batch_shape, model.batch_shape) x q x m.

property num_outputs: int¶: The number of outputs of the model.

posterior(X, observation_noise=False, posterior_transform=None, **kwargs)[source]¶

Computes the posterior over model outputs at the provided points.

Parameters

X (torch.Tensor) – A (batch_shape) x q x d-dim Tensor, where d is the dimension of the feature space and q is the number of points considered jointly.
observation_noise (Union[bool, torch.Tensor]) – If True, add the observation noise from the likelihood to the posterior. If a Tensor, use it directly as the observation noise (must be of shape (batch_shape) x q).
posterior_transform (Optional[botorch.acquisition.objective.PosteriorTransform]) – An optional PosteriorTransform.
kwargs (Any) –

Returns

A GPyTorchPosterior object, representing a batch of b joint distributions over q points. Includes observation noise if specified.

Return type

condition_on_observations(X, Y, **kwargs)[source]¶

Condition the model on new observations.

Parameters

X (torch.Tensor) – A batch_shape x n’ x d-dim Tensor, where d is the dimension of the feature space, n’ is the number of points per batch, and batch_shape is the batch shape (must be compatible with the batch shape of the model).
Y (torch.Tensor) – A batch_shape’ x n x m-dim Tensor, where m is the number of model outputs, n’ is the number of points per batch, and batch_shape’ is the batch shape of the observations. batch_shape’ must be broadcastable to batch_shape using standard broadcasting semantics. If Y has fewer batch dimensions than X, its is assumed that the missing batch dimensions are the same for all Y.
kwargs (Any) –

Returns

A Model object of the same type, representing the original model conditioned on the new observations (X, Y) (and possibly noise observations passed in via kwargs).

Return type

Example

>>> train_X = torch.rand(20, 2)
>>> train_Y = torch.sin(train_X[:, 0]) + torch.cos(train_X[:, 1])
>>> model = SingleTaskGP(train_X, train_Y)
>>> new_X = torch.rand(5, 2)
>>> new_Y = torch.sin(new_X[:, 0]) + torch.cos(new_X[:, 1])
>>> model = model.condition_on_observations(X=new_X, Y=new_Y)

class botorch.models.gpytorch.BatchedMultiOutputGPyTorchModel[source]¶

Bases: botorch.models.gpytorch.GPyTorchModel

Base class for batched multi-output GPyTorch models with independent outputs.

This model should be used when the same training data is used for all outputs. Outputs are modeled independently by using a different batch for each output.

Initializes internal Module state, shared by both nn.Module and ScriptModule.

Return type: None

static get_batch_dimensions(train_X, train_Y)[source]¶

Get the raw batch shape and output-augmented batch shape of the inputs.

Parameters

train_X (torch.Tensor) – A n x d or batch_shape x n x d (batch mode) tensor of training features.
train_Y (torch.Tensor) – A n x m or batch_shape x n x m (batch mode) tensor of training observations.

Returns

2-element tuple containing

The input_batch_shape
The output-augmented batch shape: input_batch_shape x (m)

Return type

Tuple[torch.Size, torch.Size]

property batch_shape: torch.Size¶

The batch shape of the model.

This is a batch shape from an I/O perspective, independent of the internal representation of the model (as e.g. in BatchedMultiOutputGPyTorchModel). For a model with m outputs, a test_batch_shape x q x d-shaped input X to the posterior method returns a Posterior object over an output of shape broadcast(test_batch_shape, model.batch_shape) x q x m.

posterior(X, output_indices=None, observation_noise=False, posterior_transform=None, **kwargs)[source]¶

Computes the posterior over model outputs at the provided points.

Parameters

X (torch.Tensor) – A (batch_shape) x q x d-dim Tensor, where d is the dimension of the feature space and q is the number of points considered jointly.
output_indices (Optional[List[int]]) – A list of indices, corresponding to the outputs over which to compute the posterior (if the model is multi-output). Can be used to speed up computation if only a subset of the model’s outputs are required for optimization. If omitted, computes the posterior over all model outputs.
observation_noise (Union[bool, torch.Tensor]) – If True, add the observation noise from the likelihood to the posterior. If a Tensor, use it directly as the observation noise (must be of shape (batch_shape) x q x m).
posterior_transform (Optional[botorch.acquisition.objective.PosteriorTransform]) – An optional PosteriorTransform.
kwargs (Any) –

Returns

A GPyTorchPosterior object, representing batch_shape joint distributions over q points and the outputs selected by output_indices each. Includes observation noise if specified.

Return type

botorch.models.gpytorch.BatchedMultiOutputGPyTorchModel

condition_on_observations(X, Y, **kwargs)[source]¶

Condition the model on new observations.

Parameters

X (torch.Tensor) – A batch_shape x n’ x d-dim Tensor, where d is the dimension of the feature space, m is the number of points per batch, and batch_shape is the batch shape (must be compatible with the batch shape of the model).
Y (torch.Tensor) – A batch_shape’ x n’ x m-dim Tensor, where m is the number of model outputs, n’ is the number of points per batch, and batch_shape’ is the batch shape of the observations. batch_shape’ must be broadcastable to batch_shape using standard broadcasting semantics. If Y has fewer batch dimensions than X, its is assumed that the missing batch dimensions are the same for all Y.
kwargs (Any) –

Returns

A BatchedMultiOutputGPyTorchModel object of the same type with n + n’ training examples, representing the original model conditioned on the new observations (X, Y) (and possibly noise observations passed in via kwargs).

Return type

Example

>>> train_X = torch.rand(20, 2)
>>> train_Y = torch.cat(
>>>     [torch.sin(train_X[:, 0]), torch.cos(train_X[:, 1])], -1
>>> )
>>> model = SingleTaskGP(train_X, train_Y)
>>> new_X = torch.rand(5, 2)
>>> new_Y = torch.cat([torch.sin(new_X[:, 0]), torch.cos(new_X[:, 1])], -1)
>>> model = model.condition_on_observations(X=new_X, Y=new_Y)

subset_output(idcs)[source]¶

Subset the model along the output dimension.

Parameters: idcs (List[int]) – The output indices to subset the model to.
Returns: The current model, subset to the specified output indices.
Return type: botorch.models.gpytorch.BatchedMultiOutputGPyTorchModel

class botorch.models.gpytorch.ModelListGPyTorchModel(*models)[source]¶

Bases: botorch.models.gpytorch.GPyTorchModel, botorch.models.model.ModelList, abc.ABC

Abstract base class for models based on multi-output GPyTorch models.

This is meant to be used with a gpytorch ModelList wrapper for independent evaluation of submodels.

A multi-output Model represented by a list of independent models.

Parameters

*models – A variable number of models.
models (Model) –

Return type

None

Example

>>> m_1 = SingleTaskGP(train_X, train_Y
>>> m_2 = GenericDeterministicModel(lambda x: x.sum(dim=-1))
>>> m_12 = ModelList(m_1, m_2)
>>> m_12.predict(test_X)

property batch_shape: torch.Size¶

The batch shape of the model.

This is a batch shape from an I/O perspective, independent of the internal representation of the model (as e.g. in BatchedMultiOutputGPyTorchModel). For a model with m outputs, a test_batch_shape x q x d-shaped input X to the posterior method returns a Posterior object over an output of shape broadcast(test_batch_shape, model.batch_shape) x q x m.

posterior(X, output_indices=None, observation_noise=False, posterior_transform=None, **kwargs)[source]¶

Computes the posterior over model outputs at the provided points.

Parameters

X (torch.Tensor) – A b x q x d-dim Tensor, where d is the dimension of the feature space, q is the number of points considered jointly, and b is the batch dimension.
output_indices (Optional[List[int]]) – A list of indices, corresponding to the outputs over which to compute the posterior (if the model is multi-output). Can be used to speed up computation if only a subset of the model’s outputs are required for optimization. If omitted, computes the posterior over all model outputs.
observation_noise (Union[bool, torch.Tensor]) – If True, add the observation noise from the respective likelihoods to the posterior. If a Tensor of shape (batch_shape) x q x m, use it directly as the observation noise (with observation_noise[…,i] added to the posterior of the i-th model).
posterior_transform (Optional[botorch.acquisition.objective.PosteriorTransform]) – An optional PosteriorTransform.
kwargs (Any) –

Returns

A GPyTorchPosterior object, representing batch_shape joint distributions over q points and the outputs selected by output_indices each. Includes measurement noise if observation_noise is specified.

Return type

condition_on_observations(X, Y, **kwargs)[source]¶

Condition the model on new observations.

Parameters

X (torch.Tensor) – A batch_shape x n’ x d-dim Tensor, where d is the dimension of the feature space, n’ is the number of points per batch, and batch_shape is the batch shape (must be compatible with the batch shape of the model).
Y (torch.Tensor) – A batch_shape’ x n x m-dim Tensor, where m is the number of model outputs, n’ is the number of points per batch, and batch_shape’ is the batch shape of the observations. batch_shape’ must be broadcastable to batch_shape using standard broadcasting semantics. If Y has fewer batch dimensions than X, its is assumed that the missing batch dimensions are the same for all Y.
kwargs (Any) –

Returns

A Model object of the same type, representing the original model conditioned on the new observations (X, Y) (and possibly noise observations passed in via kwargs).

Return type

Example

>>> train_X = torch.rand(20, 2)
>>> train_Y = torch.sin(train_X[:, 0]) + torch.cos(train_X[:, 1])
>>> model = SingleTaskGP(train_X, train_Y)
>>> new_X = torch.rand(5, 2)
>>> new_Y = torch.sin(new_X[:, 0]) + torch.cos(new_X[:, 1])
>>> model = model.condition_on_observations(X=new_X, Y=new_Y)

class botorch.models.gpytorch.MultiTaskGPyTorchModel[source]¶

Bases: botorch.models.gpytorch.GPyTorchModel, abc.ABC

Abstract base class for multi-task models based on GPyTorch models.

This class provides the posterior method to models that implement a “long-format” multi-task GP in the style of MultiTaskGP.

Initializes internal Module state, shared by both nn.Module and ScriptModule.

Return type: None

posterior(X, output_indices=None, observation_noise=False, posterior_transform=None, **kwargs)[source]¶

Computes the posterior over model outputs at the provided points.

Parameters

X (torch.Tensor) – A q x d or batch_shape x q x d (batch mode) tensor, where d is the dimension of the feature space (not including task indices) and q is the number of points considered jointly.
output_indices (Optional[List[int]]) – A list of indices, corresponding to the outputs over which to compute the posterior (if the model is multi-output). Can be used to speed up computation if only a subset of the model’s outputs are required for optimization. If omitted, computes the posterior over all model outputs.
observation_noise (Union[bool, torch.Tensor]) – If True, add observation noise from the respective likelihoods. If a Tensor, specifies the observation noise levels to add.
posterior_transform (Optional[botorch.acquisition.objective.PosteriorTransform]) – An optional PosteriorTransform.
kwargs (Any) –

Returns

A GPyTorchPosterior object, representing batch_shape joint distributions over q points and the outputs selected by output_indices. Includes measurement noise if observation_noise is specified.

Return type

botorch.posteriors.deterministic.DeterministicPosterior

Deterministic Model API¶

Deterministic Models. Simple wrappers that allow the usage of deterministic mappings via the BoTorch Model and Posterior APIs. Useful e.g. for defining known cost functions for cost-aware acquisition utilities.

class botorch.models.deterministic.DeterministicModel[source]¶

Bases: botorch.models.model.Model, abc.ABC

Abstract base class for deterministic models.

Initializes internal Module state, shared by both nn.Module and ScriptModule.

Return type: None

abstract forward(X)[source]¶

Compute the (deterministic) model output at X.

Parameters: X (torch.Tensor) – A batch_shape x n x d-dim input tensor X.
Returns: A batch_shape x n x m-dimensional output tensor (the outcome dimension m must be explicit if m=1).
Return type: torch.Tensor

property num_outputs: int¶: The number of outputs of the model.

posterior(X, output_indices=None, posterior_transform=None, **kwargs)[source]¶

Compute the (deterministic) posterior at X.

Parameters

X (torch.Tensor) – A batch_shape x n x d-dim input tensor X.
output_indices (Optional[List[int]]) – A list of indices, corresponding to the outputs over which to compute the posterior. If omitted, computes the posterior over all model outputs.
posterior_transform (Optional[botorch.acquisition.objective.PosteriorTransform]) – An optional PosteriorTransform.
kwargs (Any) –

Returns

A DeterministicPosterior object, representing batch_shape joint posteriors over n points and the outputs selected by output_indices.

Return type

class botorch.models.deterministic.GenericDeterministicModel(f, num_outputs=1)[source]¶

Bases: botorch.models.deterministic.DeterministicModel

A generic deterministic model constructed from a callable.

Parameters

f (Callable[[Tensor], Tensor]) – A callable mapping a batch_shape x n x d-dim input tensor X to a batch_shape x n x m-dimensional output tensor (the outcome dimension m must be explicit, even if m=1).
num_outputs (int) – The number of outputs m.

Return type

None

Example

>>> f = lambda x: x.sum(dim=-1, keep_dims=True)
>>> model = GenericDeterministicModel(f)

subset_output(idcs)[source]¶

Subset the model along the output dimension.

Parameters: idcs (List[int]) – The output indices to subset the model to.
Returns: The current model, subset to the specified output indices.
Return type: botorch.models.deterministic.GenericDeterministicModel

forward(X)[source]¶

Compute the (deterministic) model output at X.

Parameters: X (torch.Tensor) – A batch_shape x n x d-dim input tensor X.
Returns: A batch_shape x n x m-dimensional output tensor.
Return type: torch.Tensor

class botorch.models.deterministic.AffineDeterministicModel(a, b=0.01)[source]¶

Bases: botorch.models.deterministic.DeterministicModel

An affine deterministic model.

Affine deterministic model from weights and offset terms.

A simple model of the form

y[…, m] = b[m] + sum_{i=1}^d a[i, m] * X[…, i]

Parameters

a (Tensor) – A d x m-dim tensor of linear weights, where m is the number of outputs (must be explicit if m=1)
b (Union[Tensor, float]) – The affine (offset) term. Either a float (for single-output models or if the offset is shared), or a m-dim tensor (with different offset values for for the m different outputs).

Return type

None

subset_output(idcs)[source]¶

Subset the model along the output dimension.

Parameters: idcs (List[int]) – The output indices to subset the model to.
Returns: The current model, subset to the specified output indices.
Return type: botorch.models.deterministic.AffineDeterministicModel

forward(X)[source]¶

Compute the (deterministic) model output at X.

Parameters: X (torch.Tensor) – A batch_shape x n x d-dim input tensor X.
Returns: A batch_shape x n x m-dimensional output tensor (the outcome dimension m must be explicit if m=1).
Return type: torch.Tensor

class botorch.models.deterministic.PosteriorMeanModel(model)[source]¶

Bases: botorch.models.deterministic.DeterministicModel

A deterministic model that always return the posterior mean.

Parameters: model (Model) – The base model.
Return type: None

forward(X)[source]¶

Compute the (deterministic) model output at X.

Parameters: X (torch.Tensor) – A batch_shape x n x d-dim input tensor X.
Returns: A batch_shape x n x m-dimensional output tensor (the outcome dimension m must be explicit if m=1).
Return type: torch.Tensor

class botorch.models.deterministic.FixedSingleSampleModel(model, w=None)[source]¶

Bases: botorch.models.deterministic.DeterministicModel

A deterministic model defined by a single sample w.

Given a base model f and a fixed sample w, the model always outputs

y = f_mean(x) + f_stddev(x) * w

We assume the outcomes are uncorrelated here.

Parameters

model (Model) – The base model.
w (Optional[Tensor]) – A 1-d tensor with length model.num_outputs. If None, draw it from a standard normal distribution.

Return type

None

forward(X)[source]¶

Compute the (deterministic) model output at X.

Parameters: X (torch.Tensor) – A batch_shape x n x d-dim input tensor X.
Returns: A batch_shape x n x m-dimensional output tensor (the outcome dimension m must be explicit if m=1).
Return type: torch.Tensor

Models¶

Cost Models (for cost-aware optimization)¶

Cost models to be used with multi-fidelity optimization.

class botorch.models.cost.AffineFidelityCostModel(fidelity_weights=None, fixed_cost=0.01)[source]¶

Bases: botorch.models.deterministic.DeterministicModel

Affine cost model operating on fidelity parameters.

For each (q-batch) element of a candidate set X, this module computes a cost of the form

cost = fixed_cost + sum_j weights[j] * X[fidelity_dims[j]]

Affine cost model operating on fidelity parameters.

Parameters

fidelity_weights (Optional[Dict[int, float]]) – A dictionary mapping a subset of columns of X (the fidelity parameters) to it’s associated weight in the affine cost expression. If omitted, assumes that the last column of X is the fidelity parameter with a weight of 1.0.
fixed_cost (float) – The fixed cost of running a single candidate point (i.e. an element of a q-batch).

Return type

None

forward(X)[source]¶

Evaluate the cost on a candidate set X.

Computes a cost of the form

cost = fixed_cost + sum_j weights[j] * X[fidelity_dims[j]]

for each element of the q-batch

Parameters: X (torch.Tensor) – A batch_shape x q x d’-dim tensor of candidate points.
Returns: A batch_shape x q x 1-dim tensor of costs.
Return type: torch.Tensor

GP Regression Models¶

Gaussian Process Regression models based on GPyTorch models.

class botorch.models.gp_regression.SingleTaskGP(train_X, train_Y, likelihood=None, covar_module=None, outcome_transform=None, input_transform=None)[source]¶

Bases: botorch.models.gpytorch.BatchedMultiOutputGPyTorchModel, gpytorch.models.exact_gp.ExactGP

A single-task exact GP model.

A single-task exact GP using relatively strong priors on the Kernel hyperparameters, which work best when covariates are normalized to the unit cube and outcomes are standardized (zero mean, unit variance).

This model works in batch mode (each batch having its own hyperparameters). When the training observations include multiple outputs, this model will use batching to model outputs independently.

Use this model when you have independent output(s) and all outputs use the same training data. If outputs are independent and outputs have different training data, use the ModelListGP. When modeling correlations between outputs, use the MultiTaskGP.

A single-task exact GP model.

Parameters

train_X (Tensor) – A batch_shape x n x d tensor of training features.
train_Y (Tensor) – A batch_shape x n x m tensor of training observations.
likelihood (Optional[Likelihood]) – A likelihood. If omitted, use a standard GaussianLikelihood with inferred noise level.
covar_module (Optional[Module]) – The module computing the covariance (Kernel) matrix. If omitted, use a MaternKernel.
outcome_transform (Optional[OutcomeTransform]) – An outcome transform that is applied to the training data during instantiation and to the posterior during inference (that is, the Posterior obtained by calling .posterior on the model will be on the original scale).
input_transform (Optional[InputTransform]) – An input transform that is applied in the model’s forward pass.

Return type

None

Example

>>> train_X = torch.rand(20, 2)
>>> train_Y = torch.sin(train_X).sum(dim=1, keepdim=True)
>>> model = SingleTaskGP(train_X, train_Y)

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters: x (torch.Tensor) –
Return type: gpytorch.distributions.multivariate_normal.MultivariateNormal

classmethod construct_inputs(training_data, **kwargs)[source]¶

Construct kwargs for the Model from TrainingData and other options.

Parameters

training_data (botorch.utils.containers.TrainingData) – TrainingData container with data for single outcome or for multiple outcomes for batched multi-output case.
**kwargs – None expected for this class.
kwargs (Any) –

Return type

Dict[str, Any]

class botorch.models.gp_regression.FixedNoiseGP(train_X, train_Y, train_Yvar, covar_module=None, outcome_transform=None, input_transform=None, **kwargs)[source]¶

Bases: botorch.models.gpytorch.BatchedMultiOutputGPyTorchModel, gpytorch.models.exact_gp.ExactGP

A single-task exact GP model using fixed noise levels.

A single-task exact GP that uses fixed observation noise levels. This model also uses relatively strong priors on the Kernel hyperparameters, which work best when covariates are normalized to the unit cube and outcomes are standardized (zero mean, unit variance).

This model works in batch mode (each batch having its own hyperparameters).

A single-task exact GP model using fixed noise levels.

Parameters

train_X (Tensor) – A batch_shape x n x d tensor of training features.
train_Y (Tensor) – A batch_shape x n x m tensor of training observations.
train_Yvar (Tensor) – A batch_shape x n x m tensor of observed measurement noise.
outcome_transform (Optional[OutcomeTransform]) – An outcome transform that is applied to the training data during instantiation and to the posterior during inference (that is, the Posterior obtained by calling .posterior on the model will be on the original scale).
input_transform (Optional[InputTransform]) – An input transfrom that is applied in the model’s forward pass.
covar_module (Optional[Module]) –
kwargs (Any) –

Return type

None

Example

>>> train_X = torch.rand(20, 2)
>>> train_Y = torch.sin(train_X).sum(dim=1, keepdim=True)
>>> train_Yvar = torch.full_like(train_Y, 0.2)
>>> model = FixedNoiseGP(train_X, train_Y, train_Yvar)

fantasize(X, sampler, observation_noise=True, **kwargs)[source]¶

Construct a fantasy model.

Constructs a fantasy model in the following fashion: (1) compute the model posterior at X (if observation_noise=True, this includes observation noise taken as the mean across the observation noise in the training data. If observation_noise is a Tensor, use it directly as the observation noise to add). (2) sample from this posterior (using sampler) to generate “fake” observations. (3) condition the model on the new fake observations.

Parameters

X (torch.Tensor) – A batch_shape x n’ x d-dim Tensor, where d is the dimension of the feature space, n’ is the number of points per batch, and batch_shape is the batch shape (must be compatible with the batch shape of the model).
sampler (botorch.sampling.samplers.MCSampler) – The sampler used for sampling from the posterior at X.
observation_noise (Union[bool, torch.Tensor]) – If True, include the mean across the observation noise in the training data as observation noise in the posterior from which the samples are drawn. If a Tensor, use it directly as the specified measurement noise.
kwargs (Any) –

Returns

The constructed fantasy model.

Return type

botorch.models.gp_regression.FixedNoiseGP

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters: x (torch.Tensor) –
Return type: gpytorch.distributions.multivariate_normal.MultivariateNormal

subset_output(idcs)[source]¶

Subset the model along the output dimension.

Parameters: idcs (List[int]) – The output indices to subset the model to.
Returns: The current model, subset to the specified output indices.
Return type: botorch.models.gpytorch.BatchedMultiOutputGPyTorchModel

classmethod construct_inputs(training_data, **kwargs)[source]¶

Construct kwargs for the Model from TrainingData and other options.

Parameters

training_data (botorch.utils.containers.TrainingData) – TrainingData container with data for single outcome or for multiple outcomes for batched multi-output case.
**kwargs – None expected for this class.
kwargs (Any) –

Return type

Dict[str, Any]

class botorch.models.gp_regression.HeteroskedasticSingleTaskGP(train_X, train_Y, train_Yvar, outcome_transform=None, input_transform=None)[source]¶

botorch.models.gp_regression.HeteroskedasticSingleTaskGP

A single-task exact GP model using a heteroskeastic noise model.

This model internally wraps another GP (a SingleTaskGP) to model the observation noise. This allows the likelihood to make out-of-sample predictions for the observation noise levels.

A single-task exact GP model using a heteroskedastic noise model.

Parameters

train_X (Tensor) – A batch_shape x n x d tensor of training features.
train_Y (Tensor) – A batch_shape x n x m tensor of training observations.
train_Yvar (Tensor) – A batch_shape x n x m tensor of observed measurement noise.
outcome_transform (Optional[OutcomeTransform]) – An outcome transform that is applied to the training data during instantiation and to the posterior during inference (that is, the Posterior obtained by calling .posterior on the model will be on the original scale). Note that the noise model internally log-transforms the variances, which will happen after this transform is applied.
input_transform (Optional[InputTransform]) – An input transfrom that is applied in the model’s forward pass.

Return type

None

Example

>>> train_X = torch.rand(20, 2)
>>> train_Y = torch.sin(train_X).sum(dim=1, keepdim=True)
>>> se = torch.norm(train_X, dim=1, keepdim=True)
>>> train_Yvar = 0.1 + se * torch.rand_like(train_Y)
>>> model = HeteroskedasticSingleTaskGP(train_X, train_Y, train_Yvar)

condition_on_observations(X, Y, **kwargs)[source]¶

Condition the model on new observations.

Parameters

X (torch.Tensor) – A batch_shape x n’ x d-dim Tensor, where d is the dimension of the feature space, m is the number of points per batch, and batch_shape is the batch shape (must be compatible with the batch shape of the model).
Y (torch.Tensor) – A batch_shape’ x n’ x m-dim Tensor, where m is the number of model outputs, n’ is the number of points per batch, and batch_shape’ is the batch shape of the observations. batch_shape’ must be broadcastable to batch_shape using standard broadcasting semantics. If Y has fewer batch dimensions than X, its is assumed that the missing batch dimensions are the same for all Y.
kwargs (Any) –

Returns

A BatchedMultiOutputGPyTorchModel object of the same type with n + n’ training examples, representing the original model conditioned on the new observations (X, Y) (and possibly noise observations passed in via kwargs).

Return type

Example

>>> train_X = torch.rand(20, 2)
>>> train_Y = torch.cat(
>>>     [torch.sin(train_X[:, 0]), torch.cos(train_X[:, 1])], -1
>>> )
>>> model = SingleTaskGP(train_X, train_Y)
>>> new_X = torch.rand(5, 2)
>>> new_Y = torch.cat([torch.sin(new_X[:, 0]), torch.cos(new_X[:, 1])], -1)
>>> model = model.condition_on_observations(X=new_X, Y=new_Y)

subset_output(idcs)[source]¶

Subset the model along the output dimension.

Parameters: idcs (List[int]) – The output indices to subset the model to.
Returns: The current model, subset to the specified output indices.
Return type: botorch.models.gp_regression.HeteroskedasticSingleTaskGP

Multi-Fidelity GP Regression Models¶

Gaussian Process Regression models based on GPyTorch models.

Wu2019mf(1,2): J. Wu, S. Toscano-Palmerin, P. I. Frazier, and A. G. Wilson. Practical multi-fidelity bayesian optimization for hyperparameter tuning. ArXiv 2019.

class botorch.models.gp_regression_fidelity.SingleTaskMultiFidelityGP(train_X, train_Y, iteration_fidelity=None, data_fidelity=None, linear_truncated=True, nu=2.5, likelihood=None, outcome_transform=None, input_transform=None)[source]¶

A single task multi-fidelity GP model.

A SingleTaskGP model using a DownsamplingKernel for the data fidelity parameter (if present) and an ExponentialDecayKernel for the iteration fidelity parameter (if present).

This kernel is described in [Wu2019mf].

Parameters

train_X (Tensor) – A batch_shape x n x (d + s) tensor of training features, where s is the dimension of the fidelity parameters (either one or two).
train_Y (Tensor) – A batch_shape x n x m tensor of training observations.
iteration_fidelity (Optional[int]) – The column index for the training iteration fidelity parameter (optional).
data_fidelity (Optional[int]) – The column index for the downsampling fidelity parameter (optional).
linear_truncated (bool) – If True, use a LinearTruncatedFidelityKernel instead of the default kernel.
nu (float) – The smoothness parameter for the Matern kernel: either 1/2, 3/2, or 5/2. Only used when linear_truncated=True.
likelihood (Optional[Likelihood]) – A likelihood. If omitted, use a standard GaussianLikelihood with inferred noise level.
outcome_transform (Optional[OutcomeTransform]) – An outcome transform that is applied to the training data during instantiation and to the posterior during inference (that is, the Posterior obtained by calling .posterior on the model will be on the original scale).
input_transform (Optional[InputTransform]) – An input transform that is applied in the model’s forward pass.

Return type

None

Example

>>> train_X = torch.rand(20, 4)
>>> train_Y = train_X.pow(2).sum(dim=-1, keepdim=True)
>>> model = SingleTaskMultiFidelityGP(train_X, train_Y, data_fidelity=3)

A single-task exact GP model.

Parameters

train_X (Tensor) – A batch_shape x n x d tensor of training features.
train_Y (Tensor) – A batch_shape x n x m tensor of training observations.
likelihood (Optional[Likelihood]) – A likelihood. If omitted, use a standard GaussianLikelihood with inferred noise level.
covar_module – The module computing the covariance (Kernel) matrix. If omitted, use a MaternKernel.
outcome_transform (Optional[OutcomeTransform]) – An outcome transform that is applied to the training data during instantiation and to the posterior during inference (that is, the Posterior obtained by calling .posterior on the model will be on the original scale).
input_transform (Optional[InputTransform]) – An input transform that is applied in the model’s forward pass.
iteration_fidelity (Optional[int]) –
data_fidelity (Optional[int]) –
linear_truncated (bool) –
nu (float) –

Return type

None

Example

>>> train_X = torch.rand(20, 2)
>>> train_Y = torch.sin(train_X).sum(dim=1, keepdim=True)
>>> model = SingleTaskGP(train_X, train_Y)

classmethod construct_inputs(training_data, **kwargs)[source]¶

Construct kwargs for the Model from TrainingData and other options.

Parameters

training_data (botorch.utils.containers.TrainingData) – TrainingData container with data for single outcome or for multiple outcomes for batched multi-output case.
**kwargs – Options, expected for this class: - fidelity_features: List of columns of X that are fidelity parameters.

Return type

Dict[str, Any]

class botorch.models.gp_regression_fidelity.FixedNoiseMultiFidelityGP(train_X, train_Y, train_Yvar, iteration_fidelity=None, data_fidelity=None, linear_truncated=True, nu=2.5, outcome_transform=None, input_transform=None)[source]¶

Bases: botorch.models.gp_regression.FixedNoiseGP

A single task multi-fidelity GP model using fixed noise levels.

A FixedNoiseGP model analogue to SingleTaskMultiFidelityGP, using a DownsamplingKernel for the data fidelity parameter (if present) and an ExponentialDecayKernel for the iteration fidelity parameter (if present).

This kernel is described in [Wu2019mf].

Parameters

train_X (Tensor) – A batch_shape x n x (d + s) tensor of training features, where s is the dimension of the fidelity parameters (either one or two).
train_Y (Tensor) – A batch_shape x n x m tensor of training observations.
train_Yvar (Tensor) – A batch_shape x n x m tensor of observed measurement noise.
iteration_fidelity (Optional[int]) – The column index for the training iteration fidelity parameter (optional).
data_fidelity (Optional[int]) – The column index for the downsampling fidelity parameter (optional).
linear_truncated (bool) – If True, use a LinearTruncatedFidelityKernel instead of the default kernel.
nu (float) – The smoothness parameter for the Matern kernel: either 1/2, 3/2, or 5/2. Only used when linear_truncated=True.
outcome_transform (Optional[OutcomeTransform]) – An outcome transform that is applied to the training data during instantiation and to the posterior during inference (that is, the Posterior obtained by calling .posterior on the model will be on the original scale).
input_transform (Optional[InputTransform]) – An input transform that is applied in the model’s forward pass.

Return type

None

Example

>>> train_X = torch.rand(20, 4)
>>> train_Y = train_X.pow(2).sum(dim=-1, keepdim=True)
>>> train_Yvar = torch.full_like(train_Y) * 0.01
>>> model = FixedNoiseMultiFidelityGP(
>>>     train_X,
>>>     train_Y,
>>>     train_Yvar,
>>>     data_fidelity=3,
>>> )

A single-task exact GP model using fixed noise levels.

Parameters

train_X (Tensor) – A batch_shape x n x d tensor of training features.
train_Y (Tensor) – A batch_shape x n x m tensor of training observations.
train_Yvar (Tensor) – A batch_shape x n x m tensor of observed measurement noise.
outcome_transform (Optional[OutcomeTransform]) – An outcome transform that is applied to the training data during instantiation and to the posterior during inference (that is, the Posterior obtained by calling .posterior on the model will be on the original scale).
input_transform (Optional[InputTransform]) – An input transfrom that is applied in the model’s forward pass.
iteration_fidelity (Optional[int]) –
data_fidelity (Optional[int]) –
linear_truncated (bool) –
nu (float) –

Return type

None

Example

>>> train_X = torch.rand(20, 2)
>>> train_Y = torch.sin(train_X).sum(dim=1, keepdim=True)
>>> train_Yvar = torch.full_like(train_Y, 0.2)
>>> model = FixedNoiseGP(train_X, train_Y, train_Yvar)

classmethod construct_inputs(training_data, **kwargs)[source]¶

Construct kwargs for the Model from TrainingData and other options.

Parameters

training_data (botorch.utils.containers.TrainingData) – TrainingData container with data for single outcome or for multiple outcomes for batched multi-output case.
**kwargs – Options, expected for this class: - fidelity_features: List of columns of X that are fidelity parameters.

Return type

Dict[str, Any]

GP Regression Models for Mixed Parameter Spaces¶

class botorch.models.gp_regression_mixed.MixedSingleTaskGP(train_X, train_Y, cat_dims, cont_kernel_factory=None, likelihood=None, outcome_transform=None, input_transform=None)[source]¶

botorch.models.model_list_gp_regression.ModelListGP

A single-task exact GP model for mixed search spaces.

This model uses a kernel that combines a CategoricalKernel (based on Hamming distances) and a regular kernel into a kernel of the form

K((x1, c1), (x2, c2)) =
K_cont_1(x1, x2) + K_cat_1(c1, c2) + K_cont_2(x1, x2) * K_cat_2(c1, c2)

where xi and ci are the continuous and categorical features of the input, respectively. The suffix _i indicates that we fit different lengthscales for the kernels in the sum and product terms.

Since this model does not provide gradients for the categorical features, optimization of the acquisition function will need to be performed in a mixed fashion, i.e., treating the categorical features properly as discrete optimization variables.

A single-task exact GP model supporting categorical parameters.

Parameters

train_X (Tensor) – A batch_shape x n x d tensor of training features.
train_Y (Tensor) – A batch_shape x n x m tensor of training observations.
cat_dims (List[int]) – A list of indices corresponding to the columns of the input X that should be considered categorical features.
cont_kernel_factory (Optional[Callable[[int, List[int]], Kernel]]) – A method that accepts ard_num_dims and active_dims arguments and returns an instatiated GPyTorch Kernel object to be used as the ase kernel for the continuous dimensions. If omitted, this model uses a Matern-2.5 kernel as the kernel for the ordinal parameters.
likelihood (Optional[Likelihood]) – A likelihood. If omitted, use a standard GaussianLikelihood with inferred noise level.
outcome_transform (Optional[OutcomeTransform]) – An outcome transform that is applied to the training data during instantiation and to the posterior during inference (that is, the Posterior obtained by calling .posterior on the model will be on the original scale).
input_transform (Optional[InputTransform]) – An input transform that is applied in the model’s forward pass. Only input transforms are allowed which do not transform the categorical dimensions. This can be achieved by using the indices argument when constructing the transform.

Return type

None

Example

>>> train_X = torch.cat(
        [torch.rand(20, 2), torch.randint(3, (20, 1))], dim=-1)
    )
>>> train_Y = (
        torch.sin(train_X[..., :-1]).sum(dim=1, keepdim=True)
        + train_X[..., -1:]
    )
>>> model = MixedSingleTaskGP(train_X, train_Y, cat_dims=[-1])

classmethod construct_inputs(training_data, **kwargs)[source]¶

Construct kwargs for the Model from TrainingData and other options.

Parameters

training_data (botorch.utils.containers.TrainingData) – TrainingData container with data for single outcome or for multiple outcomes for batched multi-output case.
**kwargs – None expected for this class.
kwargs (Any) –

Return type

Dict[str, Any]

Model List GP Regression Models¶

Model List GP Regression models.

class botorch.models.model_list_gp_regression.ModelListGP(*gp_models)[source]¶

Bases: gpytorch.models.model_list.IndependentModelList, botorch.models.gpytorch.ModelListGPyTorchModel

A multi-output GP model with independent GPs for the outputs.

This model supports different-shaped training inputs for each of its sub-models. It can be used with any BoTorch models.

Internally, this model is just a list of individual models, but it implements the same input/output interface as all other BoTorch models. This makes it very flexible and convenient to work with. The sequential evaluation comes at a performance cost though - if you are using a block design (i.e. the same number of training example for each output, and a similar model structure, you should consider using a batched GP model instead).

A multi-output GP model with independent GPs for the outputs.

Parameters

*gp_models – An variable number of single-output BoTorch models. If models have input/output transforms, these are honored individually for each model.
gp_models (GPyTorchModel) –

Return type

None

Example

>>> model1 = SingleTaskGP(train_X1, train_Y1)
>>> model2 = SingleTaskGP(train_X2, train_Y2)
>>> model = ModelListGP(model1, model2)

condition_on_observations(X, Y, **kwargs)[source]¶

Condition the model on new observations.

Parameters

X (torch.Tensor) – A batch_shape x n’ x d-dim Tensor, where d is the dimension of the feature space, n’ is the number of points per batch, and batch_shape is the batch shape (must be compatible with the batch shape of the model).
Y (torch.Tensor) – A batch_shape’ x n’ x m-dim Tensor, where m is the number of model outputs, n’ is the number of points per batch, and batch_shape’ is the batch shape of the observations. batch_shape’ must be broadcastable to batch_shape using standard broadcasting semantics. If Y has fewer batch dimensions than X, its is assumed that the missing batch dimensions are the same for all Y.
kwargs (Any) –

Returns

A ModelListGPyTorchModel representing the original model conditioned on the new observations (X, Y) (and possibly noise observations passed in via kwargs). Here the i-th model has n_i + n’ training examples, where the n’ training examples have been added and all test-time caches have been updated.

Return type

subset_output(idcs)[source]¶

Subset the model along the output dimension.

Parameters: idcs (List[int]) – The output indices to subset the model to.
Returns: The current model, subset to the specified output indices.
Return type: botorch.models.model_list_gp_regression.ModelListGP

Multitask GP Models¶

Multi-Task GP models.

References

Doucet2010sampl: A. Doucet. A Note on Efficient Conditional Simulation of Gaussian Distributions. http://www.stats.ox.ac.uk/~doucet/doucet_simulationconditionalgaussian.pdf, Apr 2010.
Maddox2021bohdo: W. Maddox, M. Balandat, A. Wilson, and E. Bakshy. Bayesian Optimization with High-Dimensional Outputs. https://arxiv.org/abs/2106.12997, Jun 2021.

class botorch.models.multitask.MultiTaskGP(train_X, train_Y, task_feature, covar_module=None, task_covar_prior=None, output_tasks=None, rank=None, input_transform=None, outcome_transform=None)[source]¶

Bases: gpytorch.models.exact_gp.ExactGP, botorch.models.gpytorch.MultiTaskGPyTorchModel

Multi-Task GP model using an ICM kernel, inferring observation noise.

Multi-task exact GP that uses a simple ICM kernel. Can be single-output or multi-output. This model uses relatively strong priors on the base Kernel hyperparameters, which work best when covariates are normalized to the unit cube and outcomes are standardized (zero mean, unit variance).

This model infers the noise level. WARNING: It currently does not support different noise levels for the different tasks. If you have known observation noise, please use FixedNoiseMultiTaskGP instead.

Multi-Task GP model using an ICM kernel, inferring observation noise.

Parameters

train_X (Tensor) – A n x (d + 1) or b x n x (d + 1) (batch mode) tensor of training data. One of the columns should contain the task features (see task_feature argument).
train_Y (Tensor) – A n x 1 or b x n x 1 (batch mode) tensor of training observations.
task_feature (int) – The index of the task feature (-d <= task_feature <= d).
output_tasks (Optional[List[int]]) – A list of task indices for which to compute model outputs for. If omitted, return outputs for all task indices.
rank (Optional[int]) – The rank to be used for the index kernel. If omitted, use a full rank (i.e. number of tasks) kernel.
task_covar_prior (Optional[Prior]) – A Prior on the task covariance matrix. Must operate on p.s.d. matrices. A common prior for this is the LKJ prior.
input_transform (Optional[InputTransform]) – An input transform that is applied in the model’s forward pass.
covar_module (Optional[Module]) –
outcome_transform (Optional[OutcomeTransform]) –

Return type

None

Example

>>> X1, X2 = torch.rand(10, 2), torch.rand(20, 2)
>>> i1, i2 = torch.zeros(10, 1), torch.ones(20, 1)
>>> train_X = torch.cat([
>>>     torch.cat([X1, i1], -1), torch.cat([X2, i2], -1),
>>> ])
>>> train_Y = torch.cat(f1(X1), f2(X2)).unsqueeze(-1)
>>> model = MultiTaskGP(train_X, train_Y, task_feature=-1)

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters: x (torch.Tensor) –
Return type: gpytorch.distributions.multivariate_normal.MultivariateNormal

classmethod get_all_tasks(train_X, task_feature, output_tasks=None)[source]¶

Parameters

train_X (torch.Tensor) –
task_feature (int) –
output_tasks (Optional[List[int]]) –

Return type

Tuple[List[int], int, int]

classmethod construct_inputs(training_data, **kwargs)[source]¶

Construct kwargs for the Model from TrainingData and other options.

Parameters

training_data (botorch.utils.containers.TrainingData) – TrainingData container with data for single outcome or for multiple outcomes for batched multi-output case.
**kwargs –
Additional options for the model that pertain to the training data, including:
- task_features: Indices of the input columns containing the task features (expected list of length 1),
- task_covar_prior: A GPyTorch Prior object to use as prior on the cross-task covariance matrix,
- prior_config: A dict representing a prior config, should only be used if prior is not passed directly. Should contain: use_LKJ_prior (whether to use LKJ prior) and eta (eta value, float),
- rank: The rank of the cross-task covariance matrix.

Return type

Dict[str, Any]

class botorch.models.multitask.FixedNoiseMultiTaskGP(train_X, train_Y, train_Yvar, task_feature, covar_module=None, task_covar_prior=None, output_tasks=None, rank=None, input_transform=None)[source]¶

Bases: botorch.models.multitask.MultiTaskGP

Multi-Task GP model using an ICM kernel, with known observation noise.

Multi-task exact GP that uses a simple ICM kernel. Can be single-output or multi-output. This model uses relatively strong priors on the base Kernel hyperparameters, which work best when covariates are normalized to the unit cube and outcomes are standardized (zero mean, unit variance).

This model requires observation noise data (specified in train_Yvar).

Multi-Task GP model using an ICM kernel and known observation noise.

Parameters

train_X (Tensor) – A n x (d + 1) or b x n x (d + 1) (batch mode) tensor of training data. One of the columns should contain the task features (see task_feature argument).
train_Y (Tensor) – A n x 1 or b x n x 1 (batch mode) tensor of training observations.
train_Yvar (Tensor) – A n or b x n (batch mode) tensor of observation noise standard errors.
task_feature (int) – The index of the task feature (-d <= task_feature <= d).
task_covar_prior (Optional[Prior]) – A Prior on the task covariance matrix. Must operate on p.s.d. matrices. A common prior for this is the LKJ prior.
output_tasks (Optional[List[int]]) – A list of task indices for which to compute model outputs for. If omitted, return outputs for all task indices.
rank (Optional[int]) – The rank to be used for the index kernel. If omitted, use a full rank (i.e. number of tasks) kernel.
input_transform (Optional[InputTransform]) – An input transform that is applied in the model’s forward pass.
covar_module (Optional[Module]) –

Return type

None

Example

>>> X1, X2 = torch.rand(10, 2), torch.rand(20, 2)
>>> i1, i2 = torch.zeros(10, 1), torch.ones(20, 1)
>>> train_X = torch.cat([
>>>     torch.cat([X1, i1], -1), torch.cat([X2, i2], -1),
>>> ], dim=0)
>>> train_Y = torch.cat(f1(X1), f2(X2))
>>> train_Yvar = 0.1 + 0.1 * torch.rand_like(train_Y)
>>> model = FixedNoiseMultiTaskGP(train_X, train_Y, train_Yvar, -1)

classmethod construct_inputs(training_data, **kwargs)[source]¶

Construct kwargs for the Model from TrainingData and other options.

Parameters

training_data (botorch.utils.containers.TrainingData) – TrainingData container with data for single outcome or for multiple outcomes for batched multi-output case.
**kwargs –
Additional options for the model that pertain to the training data, including:
- task_features: Indices of the input columns containing the task features (expected list of length 1),
- task_covar_prior: A GPyTorch Prior object to use as prior on the cross-task covariance matrix,
- prior_config: A dict representing a prior config, should only be used if prior is not passed directly. Should contain: use_LKJ_prior` (whether to use LKJ prior) and eta (eta value, float),
- rank: The rank of the cross-task covariance matrix.

Return type

Dict[str, Any]

class botorch.models.multitask.KroneckerMultiTaskGP(train_X, train_Y, likelihood=None, data_covar_module=None, task_covar_prior=None, rank=None, input_transform=None, outcome_transform=None, **kwargs)[source]¶

Bases: gpytorch.models.exact_gp.ExactGP, botorch.models.gpytorch.GPyTorchModel

Multi-task GP with Kronecker structure, using an ICM kernel.

This model assumes the “block design” case, i.e., it requires that all tasks are observed at all data points.

For posterior sampling, this model uses Matheron’s rule [Doucet2010sampl] to compute the posterior over all tasks as in [Maddox2021bohdo] by exploiting Kronecker structure.

Multi-task GP with Kronecker structure, using a simple ICM kernel.

Parameters

train_X (Tensor) – A batch_shape x n x d tensor of training features.
train_Y (Tensor) – A batch_shape x n x m tensor of training observations.
likelihood (Optional[MultitaskGaussianLikelihood]) – A MultitaskGaussianLikelihood. If omitted, uses a MultitaskGaussianLikelihood with a GammaPrior(1.1, 0.05) noise prior.
data_covar_module (Optional[Module]) – The module computing the covariance (Kernel) matrix in data space. If omitted, use a MaternKernel.
task_covar_prior (Optional[Prior]) – A Prior on the task covariance matrix. Must operate on p.s.d. matrices. A common prior for this is the LKJ prior. If omitted, uses LKJCovariancePrior with eta parameter as specified in the keyword arguments (if not specified, use eta=1.5).
rank (Optional[int]) – The rank of the ICM kernel. If omitted, use a full rank kernel.
kwargs (Any) – Additional arguments to override default settings of priors, including: - eta: The eta parameter on the default LKJ task_covar_prior. A value of 1.0 is uninformative, values <1.0 favor stronger correlations (in magnitude), correlations vanish as eta -> inf. - sd_prior: A scalar prior over nonnegative numbers, which is used for the default LKJCovariancePrior task_covar_prior. - likelihood_rank: The rank of the task covariance matrix to fit. Defaults to 0 (which corresponds to a diagonal covariance matrix).
input_transform (Optional[InputTransform]) –
outcome_transform (Optional[OutcomeTransform]) –

Return type

None

Example

>>> train_X = torch.rand(10, 2)
>>> train_Y = torch.cat([f_1(X), f_2(X)], dim=-1)
>>> model = KroneckerMultiTaskGP(train_X, train_Y)

forward(X)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters: X (torch.Tensor) –
Return type: gpytorch.distributions.multitask_multivariate_normal.MultitaskMultivariateNormal

property train_full_covar¶

property predictive_mean_cache¶

posterior(X, output_indices=None, observation_noise=False, posterior_transform=None, **kwargs)[source]¶

Computes the posterior over model outputs at the provided points.

Parameters

X (torch.Tensor) – A (batch_shape) x q x d-dim Tensor, where d is the dimension of the feature space and q is the number of points considered jointly.
observation_noise (Union[bool, torch.Tensor]) – If True, add the observation noise from the likelihood to the posterior. If a Tensor, use it directly as the observation noise (must be of shape (batch_shape) x q).
posterior_transform (Optional[botorch.acquisition.objective.PosteriorTransform]) – An optional PosteriorTransform.
output_indices (Optional[List[int]]) –
kwargs (Any) –

Returns

A GPyTorchPosterior object, representing a batch of b joint distributions over q points. Includes observation noise if specified.

Return type

botorch.posteriors.multitask.MultitaskGPPosterior

train(val=True, *args, **kwargs)[source]¶

Puts the model in train mode and reverts to the original inputs.

Parameters: mode – A boolean denoting whether to put in train or eval mode. If False, model is put in eval mode.

Higher Order GP Models¶

References

Zhe2019hogp: S. Zhe, W. Xing, and R. M. Kirby. Scalable high-order gaussian process regression. Proceedings of Machine Learning Research, volume 89, Apr 2019.

class botorch.models.higher_order_gp.FlattenedStandardize(output_shape, batch_shape=None, min_stdv=1e-08)[source]¶

Bases: botorch.models.transforms.outcome.Standardize

Standardize outcomes in a structured multi-output settings by reshaping the batched output dimensions to be a vector. Specifically, an output dimension of [a x b x c] will be squeezed to be a vector of [a * b * c].

Standardize outcomes (zero mean, unit variance).

Parameters

m – The output dimension.
outputs – Which of the outputs to standardize. If omitted, all outputs will be standardized.
batch_shape (torch.Size) – The batch_shape of the training targets.
min_stddv – The minimum standard deviation for which to perform standardization (if lower, only de-mean the data).
output_shape (torch.Size) –
min_stdv (float) –

forward(Y, Yvar=None)[source]¶

Standardize outcomes.

If the module is in train mode, this updates the module state (i.e. the mean/std normalizing constants). If the module is in eval mode, simply applies the normalization using the module state.

Parameters

Y (torch.Tensor) – A batch_shape x n x m-dim tensor of training targets.
Yvar (Optional[torch.Tensor]) – A batch_shape x n x m-dim tensor of observation noises associated with the training targets (if applicable).

Returns

The transformed outcome observations.
The transformed observation noise (if applicable).

Return type

A two-tuple with the transformed outcomes

untransform(Y, Yvar=None)[source]¶

Un-standardize outcomes.

Parameters

Y (torch.Tensor) – A batch_shape x n x m-dim tensor of standardized targets.
Yvar (Optional[torch.Tensor]) – A batch_shape x n x m-dim tensor of standardized observation noises associated with the targets (if applicable).

Returns

The un-standardized outcome observations.
The un-standardized observation noise (if applicable).

Return type

A two-tuple with the un-standardized outcomes

untransform_posterior(posterior)[source]¶

Un-standardize the posterior.

Parameters: posterior (botorch.posteriors.higher_order.HigherOrderGPPosterior) – A posterior in the standardized space.
Returns: The un-standardized posterior. If the input posterior is a MVN, the transformed posterior is again an MVN.
Return type: botorch.posteriors.transformed.TransformedPosterior

training: bool¶

class botorch.models.higher_order_gp.HigherOrderGP(train_X, train_Y, likelihood=None, covar_modules=None, num_latent_dims=None, learn_latent_pars=True, latent_init='default', outcome_transform=None, input_transform=None)[source]¶

Bases: botorch.models.gpytorch.BatchedMultiOutputGPyTorchModel, gpytorch.models.exact_gp.ExactGP

A Higher order Gaussian process model (HOGP) (predictions are matrices/tensors) as described in [Zhe2019hogp]. The posterior uses Matheron’s rule [Doucet2010sampl] as described in [Maddox2021bohdo].

A HigherOrderGP model for high-dim output regression.

Parameters

train_X (Tensor) – A batch_shape x n x d-dim tensor of training inputs.
train_Y (Tensor) – A batch_shape x n x output_shape-dim tensor of training targets.
likelihood (Optional[Likelihood]) – Gaussian likelihood for the model.
covar_modules (Optional[List[Kernel]]) – List of kernels for each output structure.
num_latent_dims (Optional[List[int]]) – Sizes for the latent dimensions.
learn_latent_pars (bool) – If true, learn the latent parameters.
latent_init (str) – [default or gp] how to initialize the latent parameters.
outcome_transform (Optional[OutcomeTransform]) –
input_transform (Optional[InputTransform]) –

forward(X)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters: X (torch.Tensor) –
Return type: gpytorch.distributions.multivariate_normal.MultivariateNormal

get_fantasy_model(inputs, targets, **kwargs)[source]¶

Returns a new GP model that incorporates the specified inputs and targets as new training data.

Using this method is more efficient than updating with set_train_data when the number of inputs is relatively small, because any computed test-time caches will be updated in linear time rather than computed from scratch.

Note

If targets is a batch (e.g. b x m), then the GP returned from this method will be a batch mode GP. If inputs is of the same (or lesser) dimension as targets, then it is assumed that the fantasy points are the same for each target batch.

Parameters

inputs (torch.Tensor) – (b1 x … x bk x m x d or f x b1 x … x bk x m x d) Locations of fantasy observations.
targets (torch.Tensor) – (b1 x … x bk x m or f x b1 x … x bk x m) Labels of fantasy observations.

Returns

An ExactGP model with n + m training examples, where the m fantasy examples have been added and all test-time caches have been updated.

Return type

ExactGP

condition_on_observations(X, Y, **kwargs)[source]¶

Condition the model on new observations.

Parameters

X (torch.Tensor) – A batch_shape x n’ x d-dim Tensor, where d is the dimension of the feature space, m is the number of points per batch, and batch_shape is the batch shape (must be compatible with the batch shape of the model).
Y (torch.Tensor) – A batch_shape’ x n’ x m_d-dim Tensor, where m_d is the shaping of the model outputs, n’ is the number of points per batch, and batch_shape’ is the batch shape of the observations. batch_shape’ must be broadcastable to batch_shape using standard broadcasting semantics. If Y has fewer batch dimensions than X, its is assumed that the missing batch dimensions are the same for all Y.
kwargs (Any) –

Returns

A BatchedMultiOutputGPyTorchModel object of the same type with n + n’ training examples, representing the original model conditioned on the new observations (X, Y) (and possibly noise observations passed in via kwargs).

Return type

botorch.models.higher_order_gp.HigherOrderGP

posterior(X, output_indices=None, observation_noise=False, posterior_transform=None, **kwargs)[source]¶

Computes the posterior over model outputs at the provided points.

Parameters

X (torch.Tensor) – A (batch_shape) x q x d-dim Tensor, where d is the dimension of the feature space and q is the number of points considered jointly.
output_indices (Optional[List[int]]) – A list of indices, corresponding to the outputs over which to compute the posterior (if the model is multi-output). Can be used to speed up computation if only a subset of the model’s outputs are required for optimization. If omitted, computes the posterior over all model outputs.
observation_noise (Union[bool, torch.Tensor]) – If True, add the observation noise from the likelihood to the posterior. If a Tensor, use it directly as the observation noise (must be of shape (batch_shape) x q x m).
posterior_transform (Optional[botorch.acquisition.objective.PosteriorTransform]) – An optional PosteriorTransform.
kwargs (Any) –

Returns

A GPyTorchPosterior object, representing batch_shape joint distributions over q points and the outputs selected by output_indices each. Includes observation noise if specified.

Return type

botorch.posteriors.posterior.Posterior

make_posterior_variances(joint_covariance_matrix)[source]¶

Computes the posterior variances given the data points X. As currently implemented, it computes another forwards call with the stacked data to get out the joint covariance across all data points.

Parameters: joint_covariance_matrix (gpytorch.lazy.lazy_tensor.LazyTensor) –
Return type: torch.Tensor

Pairwise GP Models¶

Preference Learning with Gaussian Process

Chu2005preference(1,2,3): Wei Chu, and Zoubin Ghahramani. Preference learning with Gaussian processes. Proceedings of the 22nd international conference on Machine learning. 2005.
Brochu2010tutorial: Eric Brochu, Vlad M. Cora, and Nando De Freitas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599 (2010).

class botorch.models.pairwise_gp.PairwiseGP(datapoints, comparisons, covar_module=None, input_transform=None, **kwargs)[source]¶

Bases: botorch.models.model.Model, gpytorch.models.gp.GP

Probit GP for preference learning with Laplace approximation

Implementation is based on [Chu2005preference]. Also see [Brochu2010tutorial] for additional reference.

Note that in [Chu2005preference] the likelihood of a pairwise comparison is \(\left(\frac{f(x_1) - f(x_2)}{\sqrt{2}\sigma}\right)\), i.e. a scale is used in the denominator. To maintain consistency with usage of kernels elsewhere in botorch, we instead do not include \(\sigma\) in the code (implicitly setting it to 1) and use ScaleKernel to scale the function.

A probit-likelihood GP with Laplace approximation model that learns via: pairwise comparison data. By default it uses a scaled RBF kernel.

Parameters

datapoints (Tensor) – A batch_shape x n x d tensor of training features.
comparisons (Tensor) – A batch_shape x m x 2 training comparisons; comparisons[i] is a noisy indicator suggesting the utility value of comparisons[i, 0]-th is greater than comparisons[i, 1]-th.
covar_module (Optional[Module]) – Covariance module.
input_transform (Optional[InputTransform]) – An input transform that is applied in the model’s forward pass.

Return type

None

property num_outputs: int¶: The number of outputs of the model.

property batch_shape: torch.Size¶

The batch shape of the model.

This is a batch shape from an I/O perspective, independent of the internal representation of the model (as e.g. in BatchedMultiOutputGPyTorchModel). For a model with m outputs, a test_batch_shape x q x d-shaped input X to the posterior method returns a Posterior object over an output of shape broadcast(test_batch_shape, model.batch_shape) x q x m.

set_train_data(datapoints=None, comparisons=None, strict=False, update_model=True)[source]¶

Set datapoints and comparisons and update model properties if needed

Parameters

datapoints (Optional[torch.Tensor]) – A batch_shape x n x d dimension tensor X. If there are input transformations, assume the datapoints are not transformed
comparisons (Optional[torch.Tensor]) – A tensor of size batch_shape x m x 2. (i, j) means f_i is preferred over f_j.
strict (bool) – strict argument as in gpytorch.models.exact_gp for compatibility when using fit_gpytorch_model with input_transform.
update_model (bool) – True if we want to refit the model (see _update) after re-setting the data.

Return type

None

forward(datapoints)[source]¶

Calculate a posterior or prior prediction.

During training mode, forward implemented solely for gradient-based hyperparam opt. Essentially what it does is to re-calculate the utility f using its analytical form at f_map so that we are able to obtain gradients of the hyperparameters.

Parameters

datapoints (torch.Tensor) – A batch_shape x n x d Tensor, should be the same as self.datapoints during training

Returns

Posterior centered at MAP points for training data (training mode)
Prior predictions (prior mode)
Predictive posterior (eval mode)

Return type

A MultivariateNormal object, being one of the followings

posterior(X, output_indices=None, observation_noise=False, posterior_transform=None, **kwargs)[source]¶

Computes the posterior over model outputs at the provided points.

Parameters

X (torch.Tensor) – A batch_shape x q x d-dim Tensor, where d is the dimension of the feature space and q is the number of points considered jointly.
output_indices (Optional[List[int]]) – As defined in parent Model class, not used for this model.
observation_noise (bool) – Ignored (since noise is not identifiable from scale in probit models).
posterior_transform (Optional[botorch.acquisition.objective.PosteriorTransform]) – An optional PosteriorTransform.
kwargs (Any) –

Returns

A Posterior object, representing joint: distributions over q points.

Return type

condition_on_observations(X, Y, **kwargs)[source]¶

Condition the model on new observations.

Note that unlike other BoTorch models, PairwiseGP requires Y to be pairwise comparisons

Parameters

X (torch.Tensor) – A batch_shape x n x d dimension tensor X
Y (torch.Tensor) – A tensor of size batch_shape x m x 2. (i, j) means f_i is preferred over f_j
kwargs (Any) –

Returns

A (deepcopied) Model object of the same type, representing the original model conditioned on the new observations (X, Y).

Return type

class botorch.models.pairwise_gp.PairwiseLaplaceMarginalLogLikelihood(model)[source]¶

Bases: gpytorch.mlls.marginal_log_likelihood.MarginalLogLikelihood

Laplace-approximated marginal log likelihood/evidence for PairwiseGP

See (12) from [Chu2005preference].

Parameters: model (PairwiseGP) – A model using laplace approximation (currently only supports PairwiseGP)
Return type: None

forward(post, comp)[source]¶

Calculate approximated log evidence, i.e., log(P(D|theta))

Parameters

post (botorch.posteriors.posterior.Posterior) – training posterior distribution from self.model
comp (torch.Tensor) – Comparisons pairs, see PairwiseGP.__init__ for more details

Returns

The approximated evidence, i.e., the marginal log likelihood

Return type

torch.Tensor

training: bool¶

Contextual GP Models with Aggregate Rewards¶

class botorch.models.contextual.SACGP(train_X, train_Y, train_Yvar, decomposition)[source]¶

Bases: botorch.models.gp_regression.FixedNoiseGP

The GP uses Structural Additive Contextual(SAC) kernel.

Parameters

train_X (torch.Tensor) – (n x d) X training data.
train_Y (torch.Tensor) – (n x 1) Y training data.
train_Yvar (torch.Tensor) – (n x 1) Noise variances of each training Y.
decomposition (Dict[str, List[int]]) – Keys are context names. Values are the indexes of parameters belong to the context. The parameter indexes are in the same order across contexts.

Return type

None

A single-task exact GP model using fixed noise levels.

Parameters

train_X (torch.Tensor) – A batch_shape x n x d tensor of training features.
train_Y (torch.Tensor) – A batch_shape x n x m tensor of training observations.
train_Yvar (torch.Tensor) – A batch_shape x n x m tensor of observed measurement noise.
outcome_transform – An outcome transform that is applied to the training data during instantiation and to the posterior during inference (that is, the Posterior obtained by calling .posterior on the model will be on the original scale).
input_transform – An input transfrom that is applied in the model’s forward pass.
decomposition (Dict[str, List[int]]) –

Return type

None

Example

>>> train_X = torch.rand(20, 2)
>>> train_Y = torch.sin(train_X).sum(dim=1, keepdim=True)
>>> train_Yvar = torch.full_like(train_Y, 0.2)
>>> model = FixedNoiseGP(train_X, train_Y, train_Yvar)

class botorch.models.contextual.LCEAGP(train_X, train_Y, train_Yvar, decomposition, train_embedding=True, cat_feature_dict=None, embs_feature_dict=None, embs_dim_list=None, context_weight_dict=None)[source]¶

Bases: botorch.models.gp_regression.FixedNoiseGP

The GP with Latent Context Embedding Additive (LCE-A) Kernel. Note that the model does not support batch training. Input training data sets should have dim = 2.

Parameters

train_X (torch.Tensor) – (n x d) X training data.
train_Y (torch.Tensor) – (n x 1) Y training data.
train_Yvar (torch.Tensor) – (n x 1) Noise variance of Y.
decomposition (Dict[str, List[int]]) – Keys are context names. Values are the indexes of parameters belong to the context. The parameter indexes are in the same order across contexts.
cat_feature_dict (Optional[Dict]) – Keys are context names and values are list of categorical features i.e. {“context_name” : [cat_0, …, cat_k]}. k equals to number of categorical variables. If None, we use context names in the decomposition as the only categorical feature i.e. k = 1
embs_feature_dict (Optional[Dict]) – Pre-trained continuous embedding features of each context.
embs_dim_list (Optional[List[int]]) – Embedding dimension for each categorical variable. The length equals to num of categorical features k. If None, emb dim is set to 1 for each categorical variable.
context_weight_dict (Optional[Dict]) – Known population Weights of each context.
train_embedding (bool) –

Return type

None

A single-task exact GP model using fixed noise levels.

Parameters

train_X (torch.Tensor) – A batch_shape x n x d tensor of training features.
train_Y (torch.Tensor) – A batch_shape x n x m tensor of training observations.
train_Yvar (torch.Tensor) – A batch_shape x n x m tensor of observed measurement noise.
outcome_transform – An outcome transform that is applied to the training data during instantiation and to the posterior during inference (that is, the Posterior obtained by calling .posterior on the model will be on the original scale).
input_transform – An input transfrom that is applied in the model’s forward pass.
decomposition (Dict[str, List[int]]) –
train_embedding (bool) –
cat_feature_dict (Optional[Dict]) –
embs_feature_dict (Optional[Dict]) –
embs_dim_list (Optional[List[int]]) –
context_weight_dict (Optional[Dict]) –

Return type

None

Example

>>> train_X = torch.rand(20, 2)
>>> train_Y = torch.sin(train_X).sum(dim=1, keepdim=True)
>>> train_Yvar = torch.full_like(train_Y, 0.2)
>>> model = FixedNoiseGP(train_X, train_Y, train_Yvar)

Contextual GP Models with Context Rewards¶

class botorch.models.contextual_multioutput.LCEMGP(train_X, train_Y, task_feature, context_cat_feature=None, context_emb_feature=None, embs_dim_list=None, output_tasks=None, input_transform=None, outcome_transform=None)[source]¶

Bases: botorch.models.multitask.MultiTaskGP

The Multi-Task GP with the latent context embedding multioutput (LCE-M) kernel.

Parameters

train_X (torch.Tensor) – (n x d) X training data.
train_Y (torch.Tensor) – (n x 1) Y training data.
task_feature (int) – column index of train_X to get context indices.
context_cat_feature (Optional[torch.Tensor]) – (n_contexts x k) one-hot encoded context features. Rows are ordered by context indices. k equals to number of categorical variables. If None, task indices will be used and k = 1
context_emb_feature (Optional[torch.Tensor]) – (n_contexts x m) pre-given continuous embedding features. Rows are ordered by context indices.
embs_dim_list (Optional[List[int]]) – Embedding dimension for each categorical variable. The length equals to k. If None, emb dim is set to 1 for each categorical variable.
output_tasks (Optional[List[int]]) – A list of task indices for which to compute model outputs for. If omitted, return outputs for all task indices.
input_transform (Optional[botorch.models.transforms.input.InputTransform]) –
outcome_transform (Optional[botorch.models.transforms.outcome.OutcomeTransform]) –

Return type

None

Multi-Task GP model using an ICM kernel, inferring observation noise.

Parameters

train_X (torch.Tensor) – A n x (d + 1) or b x n x (d + 1) (batch mode) tensor of training data. One of the columns should contain the task features (see task_feature argument).
train_Y (torch.Tensor) – A n x 1 or b x n x 1 (batch mode) tensor of training observations.
task_feature (int) – The index of the task feature (-d <= task_feature <= d).
output_tasks (Optional[List[int]]) – A list of task indices for which to compute model outputs for. If omitted, return outputs for all task indices.
rank – The rank to be used for the index kernel. If omitted, use a full rank (i.e. number of tasks) kernel.
task_covar_prior – A Prior on the task covariance matrix. Must operate on p.s.d. matrices. A common prior for this is the LKJ prior.
input_transform (Optional[botorch.models.transforms.input.InputTransform]) – An input transform that is applied in the model’s forward pass.
context_cat_feature (Optional[torch.Tensor]) –
context_emb_feature (Optional[torch.Tensor]) –
embs_dim_list (Optional[List[int]]) –
outcome_transform (Optional[botorch.models.transforms.outcome.OutcomeTransform]) –

Return type

None

Example

>>> X1, X2 = torch.rand(10, 2), torch.rand(20, 2)
>>> i1, i2 = torch.zeros(10, 1), torch.ones(20, 1)
>>> train_X = torch.cat([
>>>     torch.cat([X1, i1], -1), torch.cat([X2, i2], -1),
>>> ])
>>> train_Y = torch.cat(f1(X1), f2(X2)).unsqueeze(-1)
>>> model = MultiTaskGP(train_X, train_Y, task_feature=-1)

task_covar_matrix(task_idcs)[source]¶

compute covariance matrix of a list of given context

Parameters: task_idcs (torch.Tensor) – (n x 1) or (b x n x 1) task indices tensor
Return type: torch.Tensor

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Parameters: x (torch.Tensor) –
Return type: gpytorch.distributions.multivariate_normal.MultivariateNormal

class botorch.models.contextual_multioutput.FixedNoiseLCEMGP(train_X, train_Y, train_Yvar, task_feature, context_cat_feature=None, context_emb_feature=None, embs_dim_list=None, output_tasks=None)[source]¶

Bases: botorch.models.contextual_multioutput.LCEMGP

The Multi-Task GP the latent context embedding multioutput (LCE-M) kernel, with known observation noise.

Parameters

train_X (torch.Tensor) – (n x d) X training data.
train_Y (torch.Tensor) – (n x 1) Y training data.
train_Yvar (torch.Tensor) – (n x 1) Noise variances of each training Y.
task_feature (int) – column index of train_X to get context indices.
context_cat_feature (Optional[torch.Tensor]) – (n_contexts x k) one-hot encoded context features. Rows are ordered by context indices. k equals to number of categorical variables. If None, task indices will be used and k = 1.
context_emb_feature (Optional[torch.Tensor]) – (n_contexts x m) pre-given continuous embedding features. Rows are ordered by context indices.
embs_dim_list (Optional[List[int]]) – Embedding dimension for each categorical variable. The length equals to k. If None, emb dim is set to 1 for each categorical variable.
output_tasks (Optional[List[int]]) – A list of task indices for which to compute model outputs for. If omitted, return outputs for all task indices.

Return type

None

Multi-Task GP model using an ICM kernel, inferring observation noise.

Parameters

train_X (torch.Tensor) – A n x (d + 1) or b x n x (d + 1) (batch mode) tensor of training data. One of the columns should contain the task features (see task_feature argument).
train_Y (torch.Tensor) – A n x 1 or b x n x 1 (batch mode) tensor of training observations.
task_feature (int) – The index of the task feature (-d <= task_feature <= d).
output_tasks (Optional[List[int]]) – A list of task indices for which to compute model outputs for. If omitted, return outputs for all task indices.
rank – The rank to be used for the index kernel. If omitted, use a full rank (i.e. number of tasks) kernel.
task_covar_prior – A Prior on the task covariance matrix. Must operate on p.s.d. matrices. A common prior for this is the LKJ prior.
input_transform – An input transform that is applied in the model’s forward pass.
train_Yvar (torch.Tensor) –
context_cat_feature (Optional[torch.Tensor]) –
context_emb_feature (Optional[torch.Tensor]) –
embs_dim_list (Optional[List[int]]) –

Return type

None

Example

>>> X1, X2 = torch.rand(10, 2), torch.rand(20, 2)
>>> i1, i2 = torch.zeros(10, 1), torch.ones(20, 1)
>>> train_X = torch.cat([
>>>     torch.cat([X1, i1], -1), torch.cat([X2, i2], -1),
>>> ])
>>> train_Y = torch.cat(f1(X1), f2(X2)).unsqueeze(-1)
>>> model = MultiTaskGP(train_X, train_Y, task_feature=-1)

Variational GP Models¶

References

burt2020svgp(1,2): David R. Burt and Carl Edward Rasmussen and Mark van der Wilk, Convergence of Sparse Variational Inference in Gaussian Process Regression, Journal of Machine Learning Research, 2020, http://jmlr.org/papers/v21/19-1015.html.
chen2018dpp: Laming Chen and Guoxin Zhang and Hanning Zhou, Fast greedy MAP inference for determinantal point process to improve recommendation diversity, Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018, https://arxiv.org/abs/1709.05135.
hensman2013svgp(1,2): James Hensman and Nicolo Fusi and Neil D. Lawrence, Gaussian Processes for Big Data, Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence, 2013, https://arxiv.org/abs/1309.6835.

class botorch.models.approximate_gp.ApproximateGPyTorchModel(model=None, likelihood=None, num_outputs=1, *args, **kwargs)[source]¶

Bases: botorch.models.gpytorch.GPyTorchModel

Botorch wrapper class for various (variational) approximate GP models in gpytorch. This can either include stochastic variational GPs (SVGPs) or variational implementations of weight space approximate GPs.

Parameters

model (Optional[ApproximateGP]) – Instance of gpytorch.approximate GP models. If omitted, constructs a _SingleTaskVariationalGP.
likelihood (Optional[Likelihood]) – Instance of a GPyYorch likelihood. If omitted, uses a either a GaussianLikelihood (if num_outputs=1) or a MultitaskGaussianLikelihood`(if `num_outputs>1).
num_outputs (int) – Number of outputs expected for the GP model.
args – Optional positional arguments passed to the _SingleTaskVariationalGP constructor if no model is provided.
kwargs – Optional keyword arguments passed to the _SingleTaskVariationalGP constructor if no model is provided.

Return type

None

property num_outputs¶: The number of outputs of the model.

posterior(X, output_indices=None, observation_noise=False, *args, **kwargs)[source]¶

Computes the posterior over model outputs at the provided points.

Parameters

X – A (batch_shape) x q x d-dim Tensor, where d is the dimension of the feature space and q is the number of points considered jointly.
observation_noise – If True, add the observation noise from the likelihood to the posterior. If a Tensor, use it directly as the observation noise (must be of shape (batch_shape) x q).
posterior_transform – An optional PosteriorTransform.

Returns

A GPyTorchPosterior object, representing a batch of b joint distributions over q points. Includes observation noise if specified.

Return type

forward(X, *args, **kwargs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Return type: gpytorch.distributions.multivariate_normal.MultivariateNormal

fantasize(X, sampler=<class 'botorch.sampling.samplers.MCSampler'>, observation_noise=True, *args, **kwargs)[source]¶

Construct a fantasy model.

Constructs a fantasy model in the following fashion: (1) compute the model posterior at X (including observation noise if observation_noise=True). (2) sample from this posterior (using sampler) to generate “fake” observations. (3) condition the model on the new fake observations.

Parameters

X – A batch_shape x n’ x d-dim Tensor, where d is the dimension of the feature space, n’ is the number of points per batch, and batch_shape is the batch shape (must be compatible with the batch shape of the model).
sampler – The sampler used for sampling from the posterior at X.
observation_noise – If True, include observation noise.

Returns

The constructed fantasy model.

class botorch.models.approximate_gp.SingleTaskVariationalGP(train_X, train_Y=None, likelihood=None, num_outputs=1, learn_inducing_points=True, covar_module=None, mean_module=None, variational_distribution=None, variational_strategy=<class 'gpytorch.variational.variational_strategy.VariationalStrategy'>, inducing_points=None, outcome_transform=None, input_transform=None)[source]¶

Bases: botorch.models.approximate_gp.ApproximateGPyTorchModel

A single-task variational GP model following [hensman2013svgp] with pivoted cholesky initialization following [chen2018dpp] and [burt2020svgp].

A single-task variational GP using relatively strong priors on the Kernel hyperparameters, which work best when covariates are normalized to the unit cube and outcomes are standardized (zero mean, unit variance).

This model works in batch mode (each batch having its own hyperparameters). When the training observations include multiple outputs, this model will use batching to model outputs independently. However, batches of multi-output models are not supported at this time, if you need to use those, please use a ModelListGP.

Use this model if you have a lot of data or if your responses are non-Gaussian.

To train this model, you should use gpytorch.mlls.VariationalELBO and not the exact marginal log likelihood. Example mll:

mll = VariationalELBO(model.likelihood, model.model, num_data=train_X.shape[-2])

A single task stochastic variational Gaussian process model (SVGP) as described by [hensman2013svgp]. We use pivoted cholesky initialization [burt2020svgp] to initialize the inducing points of the model.

Parameters

train_X (Tensor) – Training inputs (due to the ability of the SVGP to sub-sample this does not have to be all of the training inputs).
train_Y (Optional[Tensor]) – Training targets (optional).
likelihood (Optional[Likelihood]) – Instance of a GPyYorch likelihood. If omitted, uses a either a GaussianLikelihood (if num_outputs=1) or a MultitaskGaussianLikelihood`(if `num_outputs>1).
num_outputs (int) – Number of output responses per input (default: 1).
covar_module (Optional[Kernel]) – Kernel function. If omitted, uses a MaternKernel.
mean_module (Optional[Mean]) – Mean of GP model. If omitted, uses a ConstantMean.
variational_distribution (Optional[_VariationalDistribution]) – Type of variational distribution to use (default: CholeskyVariationalDistribution), the properties of the variational distribution will encourage scalability or ease of optimization.
variational_strategy (Type[_VariationalStrategy]) – Type of variational strategy to use (default: VariationalStrategy). The default setting uses “whitening” of the variational distribution to make training easier.
inducing_points (Optional[Union[Tensor, int]]) – The number or specific locations of the inducing points.
learn_inducing_points (bool) –
outcome_transform (Optional[OutcomeTransform]) –
input_transform (Optional[InputTransform]) –

Return type

None

init_inducing_points(inputs)[source]¶

Reinitialize the inducing point locations in-place with the current kernel applied to inputs. The variational distribution and variational strategy caches are reset.

Parameters: inputs (torch.Tensor) – (*batch_shape, n, d)-dim input data tensor.
Returns: (*batch_shape, m, d)-dim tensor of selected inducing point locations.
Return type: torch.Tensor

Fully Bayesian GP Models¶

Gaussian Process Regression models with fully Bayesian inference.

We use a lightweight PyTorch implementation of a Matern-5/2 kernel as there are some performance issues with running NUTS on top of standard GPyTorch models. The resulting hyperparameter samples are loaded into a batched GPyTorch model after fitting.

References:

Eriksson2021saasbo(1,2): D. Eriksson, M. Jankowiak. High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces. Proceedings of the Thirty- Seventh Conference on Uncertainty in Artificial Intelligence, 2021.

botorch.models.fully_bayesian.matern52_kernel(X, lengthscale)[source]¶

Matern-5/2 kernel.

Parameters

X (torch.Tensor) –
lengthscale (torch.Tensor) –

Return type

torch.Tensor

botorch.models.fully_bayesian.compute_dists(X, lengthscale)[source]¶

Compute kernel distances.

Parameters

X (torch.Tensor) –
lengthscale (torch.Tensor) –

Return type

torch.Tensor

class botorch.models.fully_bayesian.PyroModel[source]¶

Bases: object

Abstract base class for a Pyro model.

abstract sample()[source]¶

Sample from the model.

Return type: None

abstract postprocess_mcmc_samples(mcmc_samples, **kwargs)[source]¶

Post-process the final MCMC samples.

Parameters

mcmc_samples (Dict[str, torch.Tensor]) –
kwargs (Any) –

Return type

Dict[str, torch.Tensor]

class botorch.models.fully_bayesian.SaasPyroModel(train_X, train_Y, train_Yvar=None)[source]¶

Bases: botorch.models.fully_bayesian.PyroModel

Implementation of the sparse axis-aligned subspace priors (SAAS) model.

The SAAS model uses sparsity-inducing priors to identift the most important parameters. This model is suitable for high-dimensional BO with potentially hundreds of tunable parameters. See [Eriksson2021saasbo] for more details.

Initialize the SAAS model.

Parameters

train_X (torch.Tensor) – Training inputs (n x d)
train_Y (torch.Tensor) – Training targets (n x 1)
train_Yvar (Optional[torch.Tensor]) – Observed noise variance (n x 1). Inferred if None.

sample()[source]¶

Sample from the SAAS model.

This samples the mean, noise variance, outputscale, and lengthscales according to the SAAS prior.

Return type: None

sample_outputscale(concentration=2.0, rate=0.15, **tkwargs)[source]¶

Sample the outputscale.

Parameters

concentration (float) –
rate (float) –
tkwargs (Any) –

Return type

torch.Tensor

sample_mean(**tkwargs)[source]¶

Sample the mean constant.

Parameters: tkwargs (Any) –
Return type: torch.Tensor

sample_noise(**tkwargs)[source]¶

Sample the noise variance.

Parameters: tkwargs (Any) –
Return type: torch.Tensor

sample_lengthscale(dim, alpha=0.1, **tkwargs)[source]¶

Sample the lengthscale.

Parameters

dim (int) –
alpha (float) –
tkwargs (Any) –

Return type

torch.Tensor

postprocess_mcmc_samples(mcmc_samples)[source]¶

Post-process the MCMC samples.

This computes the true lengthscales and removes the inverse lengthscales and tausq (global shrinkage).

Parameters: mcmc_samples (Dict[str, torch.Tensor]) –
Return type: Dict[str, torch.Tensor]

class botorch.models.fully_bayesian.SaasFullyBayesianSingleTaskGP(train_X, train_Y, train_Yvar=None, outcome_transform=None, input_transform=None)[source]¶