In this tutorial, we illustrate how to use a custom BoTorch model within Ax's botorch_modular
API. This allows us to harness the convenience of Ax for running Bayesian Optimization loops, while at the same time maintaining full flexibility in terms of the modeling.
Acquisition functions and strategies for optimizing acquisitions can be swapped out in much the same fashion. See for example the tutorial for Implementing a custom acquisition function.
If you want to do something non-standard, or would like to have full insight into every aspect of the implementation, please see this tutorial for how to write your own full optimization loop in BoTorch.
Next cell sets up a decorator solely to speed up the testing of the notebook. You can safely ignore this cell and the use of the decorator throughout the tutorial.
import os
from contextlib import contextmanager
from ax.utils.testing.mock import fast_botorch_optimize_context_manager
import plotly.io as pio
# Ax uses Plotly to produce interactive plots. These are great for viewing and analysis,
# though they also lead to large file sizes, which is not ideal for files living in GH.
# Changing the default to `png` strips the interactive components to get around this.
pio.renderers.default = "png"
SMOKE_TEST = os.environ.get("SMOKE_TEST")
NUM_EVALS = 10 if SMOKE_TEST else 30
@contextmanager
def dummy_context_manager():
yield
if SMOKE_TEST:
fast_smoke_test = fast_botorch_optimize_context_manager
else:
fast_smoke_test = dummy_context_manager
For this tutorial, we implement a very simple gpytorch Exact GP Model that uses an RBF kernel (with ARD) and infers a (homoskedastic) noise level.
Model definition is straightforward - here we implement a gpytorch ExactGP
that also inherits from GPyTorchModel
-- this adds all the api calls that botorch expects in its various modules.
Note: botorch also allows implementing other custom models as long as they follow the minimal Model
API. For more information, please see the Model Documentation.
from typing import Optional
from botorch.models.gpytorch import GPyTorchModel
from botorch.utils.datasets import SupervisedDataset
from gpytorch.distributions import MultivariateNormal
from gpytorch.kernels import RBFKernel, ScaleKernel
from gpytorch.likelihoods import GaussianLikelihood
from gpytorch.means import ConstantMean
from gpytorch.models import ExactGP
from torch import Tensor
class SimpleCustomGP(ExactGP, GPyTorchModel):
_num_outputs = 1 # to inform GPyTorchModel API
def __init__(self, train_X, train_Y, train_Yvar: Optional[Tensor] = None):
# NOTE: This ignores train_Yvar and uses inferred noise instead.
# squeeze output dim before passing train_Y to ExactGP
super().__init__(train_X, train_Y.squeeze(-1), GaussianLikelihood())
self.mean_module = ConstantMean()
self.covar_module = ScaleKernel(
base_kernel=RBFKernel(ard_num_dims=train_X.shape[-1]),
)
self.to(train_X) # make sure we're on the right device/dtype
def forward(self, x):
mean_x = self.mean_module(x)
covar_x = self.covar_module(x)
return MultivariateNormal(mean_x, covar_x)
BoTorchModel
in Ax¶A BoTorchModel
in Ax encapsulates both the surrogate (commonly referred to as Model
in BoTorch) and an acquisition function. Here, we will only specify the custom surrogate and let Ax choose the default acquisition function.
Most models should work with the base Surrogate
in Ax, except for BoTorch ModelListGP
, which works with ListSurrogate
.
Note that the Model
(e.g., the SimpleCustomGP
) must implement construct_inputs
, as this is used to construct the inputs required for instantiating a Model
instance from the experiment data.
from ax.models.torch.botorch_modular.model import BoTorchModel
from ax.models.torch.botorch_modular.surrogate import Surrogate
ax_model = BoTorchModel(
surrogate=Surrogate(
# The model class to use
botorch_model_class=SimpleCustomGP,
# Optional, MLL class with which to optimize model parameters
# mll_class=ExactMarginalLogLikelihood,
# Optional, dictionary of keyword arguments to model constructor
# model_options={}
),
# Optional, acquisition function class to use - see custom acquisition tutorial
# botorch_acqf_class=qExpectedImprovement,
)
ModelBridge
¶Model
s in Ax require a ModelBridge
to interface with Experiment
s. A ModelBridge
takes the inputs supplied by the Experiment
and converts them to the inputs expected by the Model
. For a BoTorchModel
, we use TorchModelBridge
. The usage is as follows:
from ax.modelbridge import TorchModelBridge
model_bridge = TorchModelBridge(
experiment: Experiment,
search_space: SearchSpace,
data: Data,
model: TorchModel,
transforms: List[Type[Transform]],
# And additional optional arguments.
)
# To generate a trial
trial = model_bridge.gen(1)
For Modular BoTorch interface, we can combine the creation of the BoTorchModel
and the TorchModelBridge
into a single step as follows:
from ax.modelbridge.registry import Models
model_bridge = Models.BOTORCH_MODULAR(
experiment=experiment,
data=data,
surrogate=Surrogate(SimpleCustomGP), # Optional, will use default if unspecified
# Optional, will use default if unspecified
# botorch_acqf_class=qNoisyExpectedImprovement,
)
# To generate a trial
trial = model_bridge.gen(1)
We will demonstrate this with both the Service API (simpler, easier to use) and the Developer API (advanced, more customizable).
from ax.modelbridge.generation_strategy import GenerationStep, GenerationStrategy
from ax.modelbridge.registry import Models
gs = GenerationStrategy(
steps=[
# Quasi-random initialization step
GenerationStep(
model=Models.SOBOL,
num_trials=5, # How many trials should be produced from this generation step
),
# Bayesian optimization step using the custom acquisition function
GenerationStep(
model=Models.BOTORCH_MODULAR,
num_trials=-1, # No limitation on how many trials should be produced from this step
# For `BOTORCH_MODULAR`, we pass in kwargs to specify what surrogate or acquisition function to use.
model_kwargs={
"surrogate": Surrogate(SimpleCustomGP),
},
),
]
)
In order to use the GenerationStrategy
we just created, we will pass it into the AxClient
.
import torch
from ax.service.ax_client import AxClient
from ax.service.utils.instantiation import ObjectiveProperties
from botorch.test_functions import Branin
# Initialize the client - AxClient offers a convenient API to control the experiment
ax_client = AxClient(generation_strategy=gs)
# Setup the experiment
ax_client.create_experiment(
name="branin_test_experiment",
parameters=[
{
"name": "x1",
"type": "range",
# It is crucial to use floats for the bounds, i.e., 0.0 rather than 0.
# Otherwise, the parameter would be inferred as an integer range.
"bounds": [-5.0, 10.0],
},
{
"name": "x2",
"type": "range",
"bounds": [0.0, 15.0],
},
],
objectives={
"branin": ObjectiveProperties(minimize=True),
},
)
# Setup a function to evaluate the trials
branin = Branin()
def evaluate(parameters):
x = torch.tensor([[parameters.get(f"x{i+1}") for i in range(2)]])
# The GaussianLikelihood used by our model infers an observation noise level,
# so we pass an sem value of NaN to indicate that observation noise is unknown
return {"branin": (branin(x).item(), float("nan"))}
[INFO 05-10 20:27:10] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 6 decimal points. [INFO 05-10 20:27:10] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x1. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict. [INFO 05-10 20:27:10] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x2. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict. [INFO 05-10 20:27:10] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[RangeParameter(name='x1', parameter_type=FLOAT, range=[-5.0, 10.0]), RangeParameter(name='x2', parameter_type=FLOAT, range=[0.0, 15.0])], parameter_constraints=[]).
with fast_smoke_test():
for i in range(NUM_EVALS):
parameters, trial_index = ax_client.get_next_trial()
# Local evaluation here can be replaced with deployment to external system.
ax_client.complete_trial(trial_index=trial_index, raw_data=evaluate(parameters))
[INFO 05-10 20:27:10] ax.service.ax_client: Generated new trial 0 with parameters {'x1': -0.175425, 'x2': 9.532596}. [INFO 05-10 20:27:10] ax.service.ax_client: Completed trial 0 with data: {'branin': (30.013487, nan)}. [INFO 05-10 20:27:10] ax.service.ax_client: Generated new trial 1 with parameters {'x1': 7.182822, 'x2': 10.563368}. [INFO 05-10 20:27:10] ax.service.ax_client: Completed trial 1 with data: {'branin': (103.023666, nan)}. [INFO 05-10 20:27:10] ax.service.ax_client: Generated new trial 2 with parameters {'x1': 0.337159, 'x2': 9.081922}. [INFO 05-10 20:27:10] ax.service.ax_client: Completed trial 2 with data: {'branin': (32.049171, nan)}. [INFO 05-10 20:27:10] ax.service.ax_client: Generated new trial 3 with parameters {'x1': 1.347757, 'x2': 12.714035}. [INFO 05-10 20:27:10] ax.service.ax_client: Completed trial 3 with data: {'branin': (86.504211, nan)}. [INFO 05-10 20:27:10] ax.service.ax_client: Generated new trial 4 with parameters {'x1': 4.097963, 'x2': 1.033034}. [INFO 05-10 20:27:10] ax.service.ax_client: Completed trial 4 with data: {'branin': (4.841832, nan)}. [INFO 05-10 20:27:11] ax.service.ax_client: Generated new trial 5 with parameters {'x1': -5.0, 'x2': 3.747547}. [INFO 05-10 20:27:11] ax.service.ax_client: Completed trial 5 with data: {'branin': (193.352325, nan)}. [INFO 05-10 20:27:11] ax.service.ax_client: Generated new trial 6 with parameters {'x1': 6.677599, 'x2': 0.311248}. [INFO 05-10 20:27:11] ax.service.ax_client: Completed trial 6 with data: {'branin': (19.539585, nan)}. [INFO 05-10 20:27:11] ax.service.ax_client: Generated new trial 7 with parameters {'x1': 5.77183, 'x2': 2.719129}. [INFO 05-10 20:27:11] ax.service.ax_client: Completed trial 7 with data: {'branin': (20.939034, nan)}. [INFO 05-10 20:27:12] ax.service.ax_client: Generated new trial 8 with parameters {'x1': 0.935724, 'x2': 0.0}. [INFO 05-10 20:27:12] ax.service.ax_client: Completed trial 8 with data: {'branin': (37.076393, nan)}. [INFO 05-10 20:27:13] ax.service.ax_client: Generated new trial 9 with parameters {'x1': 4.760404, 'x2': 0.0}. [INFO 05-10 20:27:13] ax.service.ax_client: Completed trial 9 with data: {'branin': (12.286317, nan)}. [INFO 05-10 20:27:13] ax.service.ax_client: Generated new trial 10 with parameters {'x1': 2.542748, 'x2': 3.052277}. [INFO 05-10 20:27:13] ax.service.ax_client: Completed trial 10 with data: {'branin': (2.138437, nan)}. [INFO 05-10 20:27:14] ax.service.ax_client: Generated new trial 11 with parameters {'x1': 10.0, 'x2': 0.0}. [INFO 05-10 20:27:14] ax.service.ax_client: Completed trial 11 with data: {'branin': (10.960894, nan)}. [INFO 05-10 20:27:15] ax.service.ax_client: Generated new trial 12 with parameters {'x1': -4.30473, 'x2': 15.0}. [INFO 05-10 20:27:15] ax.service.ax_client: Completed trial 12 with data: {'branin': (6.253198, nan)}. [INFO 05-10 20:27:17] ax.service.ax_client: Generated new trial 13 with parameters {'x1': 10.0, 'x2': 4.229326}. [INFO 05-10 20:27:17] ax.service.ax_client: Completed trial 13 with data: {'branin': (3.447122, nan)}. [INFO 05-10 20:27:18] ax.service.ax_client: Generated new trial 14 with parameters {'x1': -2.999801, 'x2': 15.0}. [INFO 05-10 20:27:18] ax.service.ax_client: Completed trial 14 with data: {'branin': (9.877211, nan)}. [INFO 05-10 20:27:19] ax.service.ax_client: Generated new trial 15 with parameters {'x1': 2.89182, 'x2': 2.038139}. [INFO 05-10 20:27:19] ax.service.ax_client: Completed trial 15 with data: {'branin': (0.889195, nan)}. [INFO 05-10 20:27:19] ax.service.ax_client: Generated new trial 16 with parameters {'x1': 10.0, 'x2': 15.0}. [INFO 05-10 20:27:19] ax.service.ax_client: Completed trial 16 with data: {'branin': (145.872208, nan)}. [INFO 05-10 20:27:20] ax.service.ax_client: Generated new trial 17 with parameters {'x1': 10.0, 'x2': 2.606601}. [INFO 05-10 20:27:20] ax.service.ax_client: Completed trial 17 with data: {'branin': (2.10024, nan)}. [INFO 05-10 20:27:21] ax.service.ax_client: Generated new trial 18 with parameters {'x1': -3.631172, 'x2': 13.098937}. [INFO 05-10 20:27:21] ax.service.ax_client: Completed trial 18 with data: {'branin': (1.672988, nan)}. [INFO 05-10 20:27:22] ax.service.ax_client: Generated new trial 19 with parameters {'x1': -3.572242, 'x2': 13.594542}. [INFO 05-10 20:27:22] ax.service.ax_client: Completed trial 19 with data: {'branin': (1.342538, nan)}. [INFO 05-10 20:27:22] ax.service.ax_client: Generated new trial 20 with parameters {'x1': 3.012768, 'x2': 2.204223}. [INFO 05-10 20:27:22] ax.service.ax_client: Completed trial 20 with data: {'branin': (0.507518, nan)}. [INFO 05-10 20:27:24] ax.service.ax_client: Generated new trial 21 with parameters {'x1': 2.936445, 'x2': 1.718755}. [INFO 05-10 20:27:24] ax.service.ax_client: Completed trial 21 with data: {'branin': (1.120039, nan)}. [INFO 05-10 20:27:25] ax.service.ax_client: Generated new trial 22 with parameters {'x1': -5.0, 'x2': 13.278294}. [INFO 05-10 20:27:25] ax.service.ax_client: Completed trial 22 with data: {'branin': (28.004551, nan)}. [INFO 05-10 20:27:27] ax.service.ax_client: Generated new trial 23 with parameters {'x1': 9.371581, 'x2': 2.663384}. [INFO 05-10 20:27:27] ax.service.ax_client: Completed trial 23 with data: {'branin': (0.46571, nan)}. [INFO 05-10 20:27:31] ax.service.ax_client: Generated new trial 24 with parameters {'x1': 9.533112, 'x2': 2.615868}. [INFO 05-10 20:27:31] ax.service.ax_client: Completed trial 24 with data: {'branin': (0.456481, nan)}. [INFO 05-10 20:27:32] ax.service.ax_client: Generated new trial 25 with parameters {'x1': 9.511004, 'x2': 2.932077}. [INFO 05-10 20:27:32] ax.service.ax_client: Completed trial 25 with data: {'branin': (0.580544, nan)}. [INFO 05-10 20:27:33] ax.service.ax_client: Generated new trial 26 with parameters {'x1': 3.127995, 'x2': 2.48469}. [INFO 05-10 20:27:33] ax.service.ax_client: Completed trial 26 with data: {'branin': (0.4384, nan)}. [INFO 05-10 20:27:34] ax.service.ax_client: Generated new trial 27 with parameters {'x1': -3.155126, 'x2': 12.550524}. [INFO 05-10 20:27:34] ax.service.ax_client: Completed trial 27 with data: {'branin': (0.457804, nan)}. [INFO 05-10 20:27:36] ax.service.ax_client: Generated new trial 28 with parameters {'x1': -3.296565, 'x2': 13.088491}. [INFO 05-10 20:27:36] ax.service.ax_client: Completed trial 28 with data: {'branin': (0.704763, nan)}. [INFO 05-10 20:27:37] ax.service.ax_client: Generated new trial 29 with parameters {'x1': 9.465665, 'x2': 2.27895}. [INFO 05-10 20:27:37] ax.service.ax_client: Completed trial 29 with data: {'branin': (0.459161, nan)}.
ax_client.get_trials_data_frame()
branin | trial_index | arm_name | x1 | x2 | trial_status | generation_method | |
---|---|---|---|---|---|---|---|
0 | 30.013487 | 0 | 0_0 | -0.175425 | 9.532596 | COMPLETED | Sobol |
3 | 103.023666 | 1 | 1_0 | 7.182822 | 10.563368 | COMPLETED | Sobol |
15 | 32.049171 | 2 | 2_0 | 0.337159 | 9.081922 | COMPLETED | Sobol |
23 | 86.504211 | 3 | 3_0 | 1.347757 | 12.714035 | COMPLETED | Sobol |
24 | 4.841832 | 4 | 4_0 | 4.097963 | 1.033034 | COMPLETED | Sobol |
25 | 193.352325 | 5 | 5_0 | -5.000000 | 3.747547 | COMPLETED | BoTorch |
26 | 19.539585 | 6 | 6_0 | 6.677599 | 0.311248 | COMPLETED | BoTorch |
27 | 20.939034 | 7 | 7_0 | 5.771830 | 2.719129 | COMPLETED | BoTorch |
28 | 37.076393 | 8 | 8_0 | 0.935724 | 0.000000 | COMPLETED | BoTorch |
29 | 12.286317 | 9 | 9_0 | 4.760404 | 0.000000 | COMPLETED | BoTorch |
1 | 2.138437 | 10 | 10_0 | 2.542748 | 3.052277 | COMPLETED | BoTorch |
2 | 10.960894 | 11 | 11_0 | 10.000000 | 0.000000 | COMPLETED | BoTorch |
4 | 6.253198 | 12 | 12_0 | -4.304730 | 15.000000 | COMPLETED | BoTorch |
5 | 3.447122 | 13 | 13_0 | 10.000000 | 4.229326 | COMPLETED | BoTorch |
6 | 9.877211 | 14 | 14_0 | -2.999801 | 15.000000 | COMPLETED | BoTorch |
7 | 0.889195 | 15 | 15_0 | 2.891820 | 2.038139 | COMPLETED | BoTorch |
8 | 145.872208 | 16 | 16_0 | 10.000000 | 15.000000 | COMPLETED | BoTorch |
9 | 2.100240 | 17 | 17_0 | 10.000000 | 2.606601 | COMPLETED | BoTorch |
10 | 1.672988 | 18 | 18_0 | -3.631172 | 13.098937 | COMPLETED | BoTorch |
11 | 1.342538 | 19 | 19_0 | -3.572242 | 13.594542 | COMPLETED | BoTorch |
12 | 0.507518 | 20 | 20_0 | 3.012768 | 2.204223 | COMPLETED | BoTorch |
13 | 1.120039 | 21 | 21_0 | 2.936445 | 1.718755 | COMPLETED | BoTorch |
14 | 28.004551 | 22 | 22_0 | -5.000000 | 13.278294 | COMPLETED | BoTorch |
16 | 0.465710 | 23 | 23_0 | 9.371581 | 2.663384 | COMPLETED | BoTorch |
17 | 0.456481 | 24 | 24_0 | 9.533112 | 2.615868 | COMPLETED | BoTorch |
18 | 0.580544 | 25 | 25_0 | 9.511004 | 2.932077 | COMPLETED | BoTorch |
19 | 0.438400 | 26 | 26_0 | 3.127995 | 2.484690 | COMPLETED | BoTorch |
20 | 0.457804 | 27 | 27_0 | -3.155126 | 12.550524 | COMPLETED | BoTorch |
21 | 0.704763 | 28 | 28_0 | -3.296565 | 13.088491 | COMPLETED | BoTorch |
22 | 0.459161 | 29 | 29_0 | 9.465665 | 2.278950 | COMPLETED | BoTorch |
parameters, values = ax_client.get_best_parameters()
print(f"Best parameters: {parameters}")
print(f"Corresponding mean: {values[0]}, covariance: {values[1]}")
Best parameters: {'x1': 9.53311206835773, 'x2': 2.615867701958974} Corresponding mean: {'branin': 0.39199786406546977}, covariance: {'branin': {'branin': 0.04835207801718367}}
from ax.utils.notebook.plotting import render
render(ax_client.get_contour_plot())
[INFO 05-10 20:27:37] ax.service.ax_client: Retrieving contour plot with parameter 'x1' on X-axis and 'x2' on Y-axis, for metric 'branin'. Remaining parameters are affixed to the middle of their range.
best_parameters, values = ax_client.get_best_parameters()
best_parameters, values[0]
({'x1': 9.53311206835773, 'x2': 2.615867701958974}, {'branin': 0.39199786406546977})
render(ax_client.get_optimization_trace(objective_optimum=0.397887))
A detailed tutorial on the Service API can be found here.
We need 3 inputs for an Ax Experiment
:
import pandas as pd
import torch
from ax import (
Data,
Experiment,
Metric,
Objective,
OptimizationConfig,
ParameterType,
RangeParameter,
Runner,
SearchSpace,
)
from ax.utils.common.result import Ok
from botorch.test_functions import Branin
branin_func = Branin()
# For our purposes, the metric is a wrapper that structures the function output.
class BraninMetric(Metric):
def fetch_trial_data(self, trial):
records = []
for arm_name, arm in trial.arms_by_name.items():
params = arm.parameters
tensor_params = torch.tensor([params["x1"], params["x2"]])
records.append(
{
"arm_name": arm_name,
"metric_name": self.name,
"trial_index": trial.index,
"mean": branin_func(tensor_params),
"sem": float(
"nan"
), # SEM (observation noise) - NaN indicates unknown
}
)
return Ok(value=Data(df=pd.DataFrame.from_records(records)))
# Search space defines the parameters, their types, and acceptable values.
search_space = SearchSpace(
parameters=[
RangeParameter(
name="x1", parameter_type=ParameterType.FLOAT, lower=-5, upper=10
),
RangeParameter(
name="x2", parameter_type=ParameterType.FLOAT, lower=0, upper=15
),
]
)
optimization_config = OptimizationConfig(
objective=Objective(
metric=BraninMetric(name="branin_metric", lower_is_better=True),
minimize=True, # This is optional since we specified `lower_is_better=True`
)
)
class MyRunner(Runner):
def run(self, trial):
trial_metadata = {"name": str(trial.index)}
return trial_metadata
exp = Experiment(
name="branin_experiment",
search_space=search_space,
optimization_config=optimization_config,
runner=MyRunner(),
)
First, we use the Sobol generator to create 5 (quasi-) random initial point in the search space. Ax controls objective evaluations via Trial
s.
Trial
using a generator run, e.g., Sobol
below. A Trial
specifies relevant metadata as well as the parameters to be evaluated. At this point, the Trial
is at the CANDIDATE
stage.Trial
using Trial.run()
. In our example, this serves to mark the Trial
as RUNNING
. In an advanced application, this can be used to dispatch the Trial
for evaluation on a remote server.Trial
is done running, we mark it as COMPLETED
. This tells the Experiment
that it can fetch the Trial
data.A Trial
supports evaluation of a single parameterization. For parallel evaluations, see BatchTrial
.
from ax.modelbridge.registry import Models
sobol = Models.SOBOL(exp.search_space)
for i in range(5):
trial = exp.new_trial(generator_run=sobol.gen(1))
trial.run()
trial.mark_completed()
Once the initial (quasi-) random stage is completed, we can use our SimpleCustomGP
with the default acquisition function chosen by Ax
to run the BO loop.
with fast_smoke_test():
for i in range(NUM_EVALS - 5):
model_bridge = Models.BOTORCH_MODULAR(
experiment=exp,
data=exp.fetch_data(),
surrogate=Surrogate(SimpleCustomGP),
)
trial = exp.new_trial(generator_run=model_bridge.gen(1))
trial.run()
trial.mark_completed()
/Users/balandat/Code/gpytorch/gpytorch/utils/cholesky.py:40: NumericalWarning: A not p.d., added jitter of 1.0e-08 to the diagonal /Users/balandat/Code/gpytorch/gpytorch/utils/cholesky.py:40: NumericalWarning: A not p.d., added jitter of 1.0e-08 to the diagonal
View the trials attached to the Experiment
.
exp.trials
{0: Trial(experiment_name='branin_experiment', index=0, status=TrialStatus.COMPLETED, arm=Arm(name='0_0', parameters={'x1': 0.380820631980896, 'x2': 6.721179485321045})), 1: Trial(experiment_name='branin_experiment', index=1, status=TrialStatus.COMPLETED, arm=Arm(name='1_0', parameters={'x1': 6.710590762086213, 'x2': 10.495372856967151})), 2: Trial(experiment_name='branin_experiment', index=2, status=TrialStatus.COMPLETED, arm=Arm(name='2_0', parameters={'x1': 5.014360551722348, 'x2': 1.2309261644259095})), 3: Trial(experiment_name='branin_experiment', index=3, status=TrialStatus.COMPLETED, arm=Arm(name='3_0', parameters={'x1': -1.7846458964049816, 'x2': 12.460853182710707})), 4: Trial(experiment_name='branin_experiment', index=4, status=TrialStatus.COMPLETED, arm=Arm(name='4_0', parameters={'x1': -4.388935035094619, 'x2': 2.6336864847689867})), 5: Trial(experiment_name='branin_experiment', index=5, status=TrialStatus.COMPLETED, arm=Arm(name='5_0', parameters={'x1': 0.04038039984453867, 'x2': 10.98190323903202})), 6: Trial(experiment_name='branin_experiment', index=6, status=TrialStatus.COMPLETED, arm=Arm(name='6_0', parameters={'x1': 2.747621414161279, 'x2': 3.1034759051289638})), 7: Trial(experiment_name='branin_experiment', index=7, status=TrialStatus.COMPLETED, arm=Arm(name='7_0', parameters={'x1': -4.674046512049014, 'x2': 15.0})), 8: Trial(experiment_name='branin_experiment', index=8, status=TrialStatus.COMPLETED, arm=Arm(name='8_0', parameters={'x1': -4.94897707613822, 'x2': 11.813659813422815})), 9: Trial(experiment_name='branin_experiment', index=9, status=TrialStatus.COMPLETED, arm=Arm(name='9_0', parameters={'x1': 2.5443960760675486, 'x2': 0.0})), 10: Trial(experiment_name='branin_experiment', index=10, status=TrialStatus.COMPLETED, arm=Arm(name='10_0', parameters={'x1': 10.0, 'x2': 0.0})), 11: Trial(experiment_name='branin_experiment', index=11, status=TrialStatus.COMPLETED, arm=Arm(name='11_0', parameters={'x1': -2.9836712046975786, 'x2': 15.0})), 12: Trial(experiment_name='branin_experiment', index=12, status=TrialStatus.COMPLETED, arm=Arm(name='12_0', parameters={'x1': 10.0, 'x2': 3.9875118645697})), 13: Trial(experiment_name='branin_experiment', index=13, status=TrialStatus.COMPLETED, arm=Arm(name='13_0', parameters={'x1': 8.94042569724171, 'x2': 2.6190964592481505})), 14: Trial(experiment_name='branin_experiment', index=14, status=TrialStatus.COMPLETED, arm=Arm(name='14_0', parameters={'x1': 2.0203520206471293, 'x2': 2.487196832202605})), 15: Trial(experiment_name='branin_experiment', index=15, status=TrialStatus.COMPLETED, arm=Arm(name='15_0', parameters={'x1': 10.0, 'x2': 2.589179064663685})), 16: Trial(experiment_name='branin_experiment', index=16, status=TrialStatus.COMPLETED, arm=Arm(name='16_0', parameters={'x1': 9.198024870857061, 'x2': 3.353211160431645})), 17: Trial(experiment_name='branin_experiment', index=17, status=TrialStatus.COMPLETED, arm=Arm(name='17_0', parameters={'x1': 9.468271046921094, 'x2': 2.6549499120312157})), 18: Trial(experiment_name='branin_experiment', index=18, status=TrialStatus.COMPLETED, arm=Arm(name='18_0', parameters={'x1': 3.1253628896318197, 'x2': 2.0367803716917807})), 19: Trial(experiment_name='branin_experiment', index=19, status=TrialStatus.COMPLETED, arm=Arm(name='19_0', parameters={'x1': 10.0, 'x2': 15.0})), 20: Trial(experiment_name='branin_experiment', index=20, status=TrialStatus.COMPLETED, arm=Arm(name='20_0', parameters={'x1': -3.3863733829154663, 'x2': 13.132108286757118})), 21: Trial(experiment_name='branin_experiment', index=21, status=TrialStatus.COMPLETED, arm=Arm(name='21_0', parameters={'x1': 3.1502184692763517, 'x2': 2.613641616480066})), 22: Trial(experiment_name='branin_experiment', index=22, status=TrialStatus.COMPLETED, arm=Arm(name='22_0', parameters={'x1': 9.368931369226562, 'x2': 2.1812933650946675})), 23: Trial(experiment_name='branin_experiment', index=23, status=TrialStatus.COMPLETED, arm=Arm(name='23_0', parameters={'x1': -3.014857554031104, 'x2': 11.693571615730272})), 24: Trial(experiment_name='branin_experiment', index=24, status=TrialStatus.COMPLETED, arm=Arm(name='24_0', parameters={'x1': -3.2133666348611305, 'x2': 12.557484080096193})), 25: Trial(experiment_name='branin_experiment', index=25, status=TrialStatus.COMPLETED, arm=Arm(name='25_0', parameters={'x1': 3.3049358815827254, 'x2': 3.0506205758770633})), 26: Trial(experiment_name='branin_experiment', index=26, status=TrialStatus.COMPLETED, arm=Arm(name='26_0', parameters={'x1': 9.519548235670394, 'x2': 2.576731508576853})), 27: Trial(experiment_name='branin_experiment', index=27, status=TrialStatus.COMPLETED, arm=Arm(name='27_0', parameters={'x1': 3.1108091675566403, 'x2': 2.2963116481878485})), 28: Trial(experiment_name='branin_experiment', index=28, status=TrialStatus.COMPLETED, arm=Arm(name='28_0', parameters={'x1': 9.525041431874874, 'x2': 2.8161634816383123})), 29: Trial(experiment_name='branin_experiment', index=29, status=TrialStatus.COMPLETED, arm=Arm(name='29_0', parameters={'x1': -3.1073710202692113, 'x2': 12.159577025343067}))}
View the evaluation data about these trials.
exp.fetch_data().df
arm_name | metric_name | mean | sem | trial_index | |
---|---|---|---|---|---|
0 | 0_0 | branin_metric | 20.626492 | NaN | 0 |
1 | 1_0 | branin_metric | 106.313690 | NaN | 1 |
2 | 2_0 | branin_metric | 12.857041 | NaN | 2 |
3 | 3_0 | branin_metric | 18.260235 | NaN | 3 |
4 | 4_0 | branin_metric | 171.812851 | NaN | 4 |
5 | 5_0 | branin_metric | 45.055992 | NaN | 5 |
6 | 6_0 | branin_metric | 1.384670 | NaN | 6 |
7 | 7_0 | branin_metric | 11.222628 | NaN | 7 |
8 | 8_0 | branin_metric | 39.571274 | NaN | 8 |
9 | 9_0 | branin_metric | 9.826130 | NaN | 9 |
10 | 10_0 | branin_metric | 10.960894 | NaN | 10 |
11 | 11_0 | branin_metric | 10.135442 | NaN | 11 |
12 | 12_0 | branin_metric | 2.912488 | NaN | 12 |
13 | 13_0 | branin_metric | 1.775204 | NaN | 13 |
14 | 14_0 | branin_metric | 6.507253 | NaN | 14 |
15 | 15_0 | branin_metric | 2.114354 | NaN | 15 |
16 | 16_0 | branin_metric | 1.773315 | NaN | 16 |
17 | 17_0 | branin_metric | 0.427422 | NaN | 17 |
18 | 18_0 | branin_metric | 0.462108 | NaN | 18 |
19 | 19_0 | branin_metric | 145.872208 | NaN | 19 |
20 | 20_0 | branin_metric | 0.752295 | NaN | 20 |
21 | 21_0 | branin_metric | 0.517516 | NaN | 21 |
22 | 22_0 | branin_metric | 0.473867 | NaN | 22 |
23 | 23_0 | branin_metric | 0.552698 | NaN | 23 |
24 | 24_0 | branin_metric | 0.434562 | NaN | 24 |
25 | 25_0 | branin_metric | 1.334905 | NaN | 25 |
26 | 26_0 | branin_metric | 0.441401 | NaN | 26 |
27 | 27_0 | branin_metric | 0.402444 | NaN | 27 |
28 | 28_0 | branin_metric | 0.511283 | NaN | 28 |
29 | 29_0 | branin_metric | 0.404620 | NaN | 29 |
We can use convenient Ax utilities for plotting the results.
import numpy as np
from ax.plot.trace import optimization_trace_single_method
# `plot_single_method` expects a 2-d array of means, because it expects to average means from multiple
# optimization runs, so we wrap out best objectives array in another array.
objective_means = np.array([[trial.objective_mean for trial in exp.trials.values()]])
best_objective_plot = optimization_trace_single_method(
y=np.minimum.accumulate(objective_means, axis=1),
optimum=0.397887, # Known minimum objective for Branin function.
)
render(best_objective_plot)