Getting started with Advanced HPO Algorithms

Loading libraries

# Basic utils for folder manipulations etc
import time
import multiprocessing # to count the number of CPUs available

# External tools to load and process data
import numpy as np
import pandas as pd

# MXNet (NeuralNets)
import mxnet as mx
from mxnet import gluon, autograd
from mxnet.gluon import nn

# AutoGluon and HPO tools
import autogluon as ag
from autogluon.utils import load_and_split_openml_data

Check the version of MxNet, you should be fine with version >= 1.5

mx.__version__
'1.7.0'

You can also check the version of AutoGluon and the specific commit and check that it matches what you want.

ag.__version__
'0.0.14b20200919'

Hyperparameter Optimization of a 2-layer MLP

Setting up the context

Here we declare a few “environment variables” setting the context for what we’re doing

OPENML_TASK_ID = 6                # describes the problem we will tackle
RATIO_TRAIN_VALID = 0.33          # split of the training data used for validation
RESOURCE_ATTR_NAME = 'epoch'      # how do we measure resources   (will become clearer further)
REWARD_ATTR_NAME = 'objective'    # how do we measure performance (will become clearer further)

NUM_CPUS = multiprocessing.cpu_count()

Preparing the data

We will use a multi-way classification task from OpenML. Data preparation includes:

  • Missing values are imputed, using the ‘mean’ strategy of sklearn.impute.SimpleImputer

  • Split training set into training and validation

  • Standardize inputs to mean 0, variance 1

X_train, X_valid, y_train, y_valid, n_classes = load_and_split_openml_data(
    OPENML_TASK_ID, RATIO_TRAIN_VALID, download_from_openml=False)
n_classes
Downloading ./org/openml/www/datasets/6/dataset.arff from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.arff...
100%|██████████| 704/704 [00:00<00:00, 46769.46KB/s]
Downloading ./org/openml/www/datasets/6/dataset.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.pkl.py3...
100%|██████████| 2521/2521 [00:00<00:00, 22504.19KB/s]
Downloading ./org/openml/www/datasets/6/description.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/description.xml...
3KB [00:00, 3966.87KB/s]
Downloading ./org/openml/www/datasets/6/features.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/features.xml...
8KB [00:00, 10628.58KB/s]
Downloading ./org/openml/www/datasets/6/qualities.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/qualities.xml...
15KB [00:00, 17481.12KB/s]
Downloading ./org/openml/www/tasks/6/datasplits.arff from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.arff...
2998KB [00:00, 38828.35KB/s]
Downloading ./org/openml/www/tasks/6/datasplits.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.pkl.py3...
881KB [00:00, 43574.23KB/s]
Downloading ./org/openml/www/tasks/6/task.xml from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/task.xml...
3KB [00:00, 3799.19KB/s]
Data pickle file already exists and is up to date.
Data pickle file already exists and is up to date.
26

The problem has 26 classes.

Declaring a model specifying a hyperparameter space with AutoGluon

Two layer MLP where we optimize over:

  • the number of units on the first layer

  • the number of units on the second layer

  • the dropout rate after each layer

  • the learning rate

  • the scaling

  • the @ag.args decorator allows us to specify the space we will optimize over, this matches the ConfigSpace syntax

The body of the function run_mlp_openml is pretty simple:

  • it reads the hyperparameters given via the decorator

  • it defines a 2 layer MLP with dropout

  • it declares a trainer with the ‘adam’ loss function and a provided learning rate

  • it trains the NN with a number of epochs (most of that is boilerplate code from mxnet)

  • the reporter at the end is used to keep track of training history in the hyperparameter optimization

Note: The number of epochs and the hyperparameter space are reduced to make for a shorter experiment

@ag.args(n_units_1=ag.space.Int(lower=16, upper=128),
         n_units_2=ag.space.Int(lower=16, upper=128),
         dropout_1=ag.space.Real(lower=0, upper=.75),
         dropout_2=ag.space.Real(lower=0, upper=.75),
         learning_rate=ag.space.Real(lower=1e-6, upper=1, log=True),
         batch_size=ag.space.Int(lower=8, upper=128),
         scale_1=ag.space.Real(lower=0.001, upper=10, log=True),
         scale_2=ag.space.Real(lower=0.001, upper=10, log=True),
         epochs=9)
def run_mlp_openml(args, reporter, **kwargs):
    # Time stamp for elapsed_time
    ts_start = time.time()
    # Unwrap hyperparameters
    n_units_1 = args.n_units_1
    n_units_2 = args.n_units_2
    dropout_1 = args.dropout_1
    dropout_2 = args.dropout_2
    scale_1 = args.scale_1
    scale_2 = args.scale_2
    batch_size = args.batch_size
    learning_rate = args.learning_rate

    ctx = mx.cpu()
    net = nn.Sequential()
    with net.name_scope():
        # Layer 1
        net.add(nn.Dense(n_units_1, activation='relu',
                         weight_initializer=mx.initializer.Uniform(scale=scale_1)))
        # Dropout
        net.add(gluon.nn.Dropout(dropout_1))
        # Layer 2
        net.add(nn.Dense(n_units_2, activation='relu',
                         weight_initializer=mx.initializer.Uniform(scale=scale_2)))
        # Dropout
        net.add(gluon.nn.Dropout(dropout_2))
        # Output
        net.add(nn.Dense(n_classes))
    net.initialize(ctx=ctx)

    trainer = gluon.Trainer(net.collect_params(), 'adam',
                            {'learning_rate': learning_rate})

    for epoch in range(args.epochs):
        ts_epoch = time.time()

        train_iter = mx.io.NDArrayIter(
                        data={'data': X_train},
                        label={'label': y_train},
                        batch_size=batch_size,
                        shuffle=True)
        valid_iter = mx.io.NDArrayIter(
                        data={'data': X_valid},
                        label={'label': y_valid},
                        batch_size=batch_size,
                        shuffle=False)

        metric = mx.metric.Accuracy()
        loss = gluon.loss.SoftmaxCrossEntropyLoss()

        for batch in train_iter:
            data = batch.data[0].as_in_context(ctx)
            label = batch.label[0].as_in_context(ctx)
            with autograd.record():
                output = net(data)
                L = loss(output, label)
            L.backward()
            trainer.step(data.shape[0])
            metric.update([label], [output])

        name, train_acc = metric.get()

        metric = mx.metric.Accuracy()
        for batch in valid_iter:
            data = batch.data[0].as_in_context(ctx)
            label = batch.label[0].as_in_context(ctx)
            output = net(data)
            metric.update([label], [output])

        name, val_acc = metric.get()

        print('Epoch %d ; Time: %f ; Training: %s=%f ; Validation: %s=%f' % (
            epoch + 1, time.time() - ts_start, name, train_acc, name, val_acc))

        ts_now = time.time()
        eval_time = ts_now - ts_epoch
        elapsed_time = ts_now - ts_start

        # The resource reported back (as 'epoch') is the number of epochs
        # done, starting at 1
        reporter(
            epoch=epoch + 1,
            objective=float(val_acc),
            eval_time=eval_time,
            time_step=ts_now,
            elapsed_time=elapsed_time)

Note: The annotation epochs=9 specifies the maximum number of epochs for training. It becomes available as args.epochs. Importantly, it is also processed by HyperbandScheduler below in order to set its max_t attribute.

Recommendation: Whenever writing training code to be passed as train_fn to a scheduler, if this training code reports a resource (or time) attribute, the corresponding maximum resource value should be included in train_fn.args:

  • If the resource attribute (time_attr of scheduler) in train_fn is epoch, make sure to include epochs=XYZ in the annotation. This allows the scheduler to read max_t from train_fn.args.epochs. This case corresponds to our example here.

  • If the resource attribute is something else than epoch, you can also include the annotation max_t=XYZ, which allows the scheduler to read max_t from train_fn.args.max_t.

Annotating the training function by the correct value for max_t simplifies scheduler creation (since max_t does not have to be passed), and avoids inconsistencies between train_fn and the scheduler.

Running the Hyperparameter Optimization

You can use the following schedulers:

  • FIFO (fifo)

  • Hyperband (either the stopping (hbs) or promotion (hbp) variant)

And the following searchers:

  • Random search (random)

  • Gaussian process based Bayesian optimization (bayesopt)

  • SkOpt Bayesian optimization (skopt; only with FIFO scheduler)

Note that the method known as (asynchronous) Hyperband is using random search. Combining Hyperband scheduling with the bayesopt searcher uses a novel method called asynchronous BOHB.

Pick the combination you’re interested in (doing the full experiment takes around 120 seconds, see the time_out parameter), running everything with multiple runs can take a fair bit of time. In real life, you will want to choose a larger time_out in order to obtain good performance.

SCHEDULER = "hbs"
SEARCHER = "bayesopt"
def compute_error(df):
    return 1.0 - df["objective"]

def compute_runtime(df, start_timestamp):
        return df["time_step"] - start_timestamp

def process_training_history(task_dicts, start_timestamp,
                             runtime_fn=compute_runtime,
                             error_fn=compute_error):
    task_dfs = []
    for task_id in task_dicts:
        task_df = pd.DataFrame(task_dicts[task_id])
        task_df = task_df.assign(task_id=task_id,
                                 runtime=runtime_fn(task_df, start_timestamp),
                                 error=error_fn(task_df),
                                 target_epoch=task_df["epoch"].iloc[-1])
        task_dfs.append(task_df)

    result = pd.concat(task_dfs, axis="index", ignore_index=True, sort=True)
    # re-order by runtime
    result = result.sort_values(by="runtime")
    # calculate incumbent best -- the cumulative minimum of the error.
    result = result.assign(best=result["error"].cummin())
    return result

resources = dict(num_cpus=NUM_CPUS, num_gpus=0)
search_options = {
    'num_init_random': 2,
    'debug_log': True}
if SCHEDULER == 'fifo':
    myscheduler = ag.scheduler.FIFOScheduler(
        run_mlp_openml,
        resource=resources,
        searcher=SEARCHER,
        search_options=search_options,
        time_out=120,
        time_attr=RESOURCE_ATTR_NAME,
        reward_attr=REWARD_ATTR_NAME)

else:
    # This setup uses rung levels at 1, 3, 9 epochs. We just use a single
    # bracket, so this is in fact successive halving (Hyperband would use
    # more than 1 bracket).
    # Also note that since we do not use the max_t argument of
    # HyperbandScheduler, this value is obtained from train_fn.args.epochs.
    sch_type = 'stopping' if SCHEDULER == 'hbs' else 'promotion'
    myscheduler = ag.scheduler.HyperbandScheduler(
        run_mlp_openml,
        resource=resources,
        searcher=SEARCHER,
        search_options=search_options,
        time_out=120,
        time_attr=RESOURCE_ATTR_NAME,
        reward_attr=REWARD_ATTR_NAME,
        type=sch_type,
        grace_period=1,
        reduction_factor=3,
        brackets=1)

# run tasks
myscheduler.run()
myscheduler.join_jobs()

results_df = process_training_history(
                myscheduler.training_history.copy(),
                start_timestamp=myscheduler._start_time)
max_t = 9, as inferred from train_fn.args
scheduler_options: Key 'resume': Imputing default value False
scheduler_options: Key 'keep_size_ratios': Imputing default value False
scheduler_options: Key 'maxt_pending': Imputing default value False
scheduler_options: Key 'searcher_data': Imputing default value rungs
scheduler_options: Key 'do_snapshots': Imputing default value False

scheduler_options: Key 'visualizer': Imputing default value none
scheduler_options: Key 'training_history_callback_delta_secs': Imputing default value 60
scheduler_options: Key 'delay_get_config': Imputing default value True

search_options: Key 'random_seed': Imputing default value 31415927
search_options: Key 'opt_skip_init_length': Imputing default value 150
search_options: Key 'opt_skip_period': Imputing default value 1
search_options: Key 'profiler': Imputing default value False
search_options: Key 'opt_maxiter': Imputing default value 50
search_options: Key 'opt_nstarts': Imputing default value 2
search_options: Key 'opt_warmstart': Imputing default value False
search_options: Key 'opt_verbose': Imputing default value False
search_options: Key 'opt_debug_writer': Imputing default value False
search_options: Key 'num_fantasy_samples': Imputing default value 20
search_options: Key 'num_init_candidates': Imputing default value 250
search_options: Key 'initial_scoring': Imputing default value thompson_indep
search_options: Key 'first_is_default': Imputing default value True
search_options: Key 'opt_skip_num_max_resource': Imputing default value False
search_options: Key 'gp_resource_kernel': Imputing default value matern52
search_options: Key 'resource_acq': Imputing default value bohb

[GPMultiFidelitySearcher.__init__]
- acquisition_class = <class 'autogluon.searcher.bayesopt.models.nphead_acqfunc.EIAcquisitionFunction'>
- local_minimizer_class = <class 'autogluon.searcher.bayesopt.tuning_algorithms.bo_algorithm_components.LBFGSOptimizeAcquisition'>
- num_initial_candidates = 250
- num_initial_random_choices = 2
- initial_scoring = thompson_indep
- first_is_default = True
Starting Experiments
Num of Finished Tasks is 0
Time out (secs) is 120
Starting get_config[random] for config_id 0
Start with default config:
{'batch_size': 68, 'dropout_1': 0.375, 'dropout_2': 0.375, 'learning_rate': 0.001, 'n_units_1': 72, 'n_units_2': 72, 'scale_1': 0.1, 'scale_2': 0.1}
[0: random]
batch_size: 68
dropout_1: 0.375
dropout_2: 0.375
learning_rate: 0.001
n_units_1: 72
n_units_2: 72
scale_1: 0.1
scale_2: 0.1
/var/lib/jenkins/miniconda3/envs/autogluon_docs/lib/python3.7/site-packages/distributed/worker.py:3382: UserWarning: Large object of size 1.30 MB detected in task graph:
  (<function run_mlp_openml at 0x7f758f6eac20>, {'ar ... sReporter}, [])
Consider scattering large objects ahead of time
with client.scatter to reduce scheduler burden and
keep data on workers

    future = client.submit(func, big_data)    # bad

    big_future = client.scatter(big_data)     # good
    future = client.submit(func, big_future)  # good
  % (format_bytes(len(b)), s)
Epoch 1 ; Time: 0.466086 ; Training: accuracy=0.260079 ; Validation: accuracy=0.531250
Update for config_id 0:1: reward = 0.53125, crit_val = 0.46875
config_id 0: Reaches 1, continues to 3
Epoch 2 ; Time: 0.884446 ; Training: accuracy=0.496365 ; Validation: accuracy=0.655247
Epoch 3 ; Time: 1.290279 ; Training: accuracy=0.559650 ; Validation: accuracy=0.694686
Update for config_id 0:3: reward = 0.6946858288770054, crit_val = 0.30531417112299464
config_id 0: Reaches 3, continues to 9
Epoch 4 ; Time: 1.697857 ; Training: accuracy=0.588896 ; Validation: accuracy=0.711063
Epoch 5 ; Time: 2.178183 ; Training: accuracy=0.609385 ; Validation: accuracy=0.726939
Epoch 6 ; Time: 2.585163 ; Training: accuracy=0.628139 ; Validation: accuracy=0.745321
Epoch 7 ; Time: 2.991122 ; Training: accuracy=0.641193 ; Validation: accuracy=0.750501
Epoch 8 ; Time: 3.407490 ; Training: accuracy=0.653751 ; Validation: accuracy=0.763202
Epoch 9 ; Time: 3.819573 ; Training: accuracy=0.665482 ; Validation: accuracy=0.766043
config_id 0: Terminating evaluation at 9
Update for config_id 0:9: reward = 0.766042780748663, crit_val = 0.23395721925133695
Starting get_config[random] for config_id 1
[1: random]
batch_size: 49
dropout_1: 0.6118880110946948
dropout_2: 0.5692033531564655
learning_rate: 0.00010310483923558212
n_units_1: 126
n_units_2: 93
scale_1: 0.17664689253248303
scale_2: 0.5396711634456218
Epoch 1 ; Time: 0.691406 ; Training: accuracy=0.048005 ; Validation: accuracy=0.111743
config_id 1: Terminating evaluation at 1
Update for config_id 1:1: reward = 0.11174305787888926, crit_val = 0.8882569421211107
Starting get_config[BO] for config_id 2
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4740695831238606
- self.std = 0.25382661537049356
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 4
Current best is [0.46875]
[2: BO] (1 evaluations)
batch_size: 50
dropout_1: 0.21593234798980557
dropout_2: 0.2842099148996821
learning_rate: 0.12426792150054146
n_units_1: 61
n_units_2: 79
scale_1: 9.791854554777688
scale_2: 4.705654410614348
Started BO from (top scorer):
batch_size: 50
dropout_1: 0.21593234798980557
dropout_2: 0.2842099148996821
learning_rate: 0.12426792150054146
n_units_1: 61
n_units_2: 79
scale_1: 9.791854554777688
scale_2: 4.705654410614348
Top score values: [-0.27440679  0.04673507  0.04701008  0.10393977  0.10463477]
Labeled: 0:1, 0:3, 0:9, 1:1. Pending:
Targets: [-0.02095755 -0.66484522 -0.94597     1.63177277]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.7572453035969392, 'kernel_inv_bw1': 0.9003213708321405, 'kernel_inv_bw2': 0.9671934120293091, 'kernel_inv_bw3': 3.935156155481897, 'kernel_inv_bw4': 26.153730997155336, 'kernel_inv_bw5': 1.6599386110717549, 'kernel_inv_bw6': 4.378455276759085, 'kernel_inv_bw7': 35.45306859775246, 'kernel_inv_bw8': 1.6274469437655676, 'kernel_covariance_scale': 0.8177941757954839, 'mean_mean_value': 0.3168762072734543}
Epoch 1 ; Time: 0.602260 ; Training: accuracy=0.072975 ; Validation: accuracy=0.038319
config_id 2: Terminating evaluation at 1
Update for config_id 2:1: reward = 0.03831932773109244, crit_val = 0.9616806722689075
Starting get_config[BO] for config_id 3
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.57159180095287
- self.std = 0.2993070203278022
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 5
Current best is [0.46875]
[3: BO] (1 evaluations)
batch_size: 22
dropout_1: 0.6341443733804044
dropout_2: 0.04086583670464042
learning_rate: 0.12818863950369533
n_units_1: 19
n_units_2: 101
scale_1: 0.026860146662962745
scale_2: 4.431236744525342
Started BO from (top scorer):
batch_size: 22
dropout_1: 0.6341443733804044
dropout_2: 0.04086583670464042
learning_rate: 0.12818863950369533
n_units_1: 19
n_units_2: 101
scale_1: 0.026860146662962745
scale_2: 4.431236744525342
Top score values: [-0.02497327  0.00036035  0.04139247  0.12358631  0.12397302]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1. Pending:
Targets: [-0.3435997  -0.88964712 -1.12805433  1.05799437  1.30330679]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 7.917350904102372, 'kernel_inv_bw1': 1.7688291458396213, 'kernel_inv_bw2': 1.9520232274466205, 'kernel_inv_bw3': 1.4782426762258778, 'kernel_inv_bw4': 96.92689951596101, 'kernel_inv_bw5': 4.225187089919363, 'kernel_inv_bw6': 100.00000000000004, 'kernel_inv_bw7': 99.70951885667674, 'kernel_inv_bw8': 1.339282496524658, 'kernel_covariance_scale': 0.7384889758580241, 'mean_mean_value': 0.38059821696758545}
Epoch 1 ; Time: 1.315001 ; Training: accuracy=0.041812 ; Validation: accuracy=0.039562
config_id 3: Terminating evaluation at 1
Update for config_id 3:1: reward = 0.03956228956228956, crit_val = 0.9604377104377104
Starting get_config[BO] for config_id 4
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.6363994525336767
- self.std = 0.30927991991886417
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 6
Current best is [0.46875]
[4: BO] (6 evaluations)
batch_size: 110
dropout_1: 0.5744977554712186
dropout_2: 0.7047747941055491
learning_rate: 3.6223201928225404e-05
n_units_1: 92
n_units_2: 92
scale_1: 0.06664854483482407
scale_2: 0.022478620206744602
Started BO from (top scorer):
batch_size: 110
dropout_1: 0.5744977553473402
dropout_2: 0.7047748951791846
learning_rate: 3.6221732543522435e-05
n_units_1: 92
n_units_2: 92
scale_1: 0.06664854527483652
scale_2: 0.017643835128767108
Top score values: [-0.02863203  0.00599656  0.042477    0.05395405  0.06651693]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1. Pending:
Targets: [-0.54206381 -1.07050364 -1.30122328  0.81433508  1.05173727  1.04771838]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00015112858442299117, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.0011999688107008975, 'kernel_inv_bw3': 0.009268380795486905, 'kernel_inv_bw4': 0.00010293036296093569, 'kernel_inv_bw5': 0.0001746162036548292, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 4.272493402660988, 'kernel_inv_bw8': 1.3164718962193263, 'kernel_covariance_scale': 0.7976260528193392, 'mean_mean_value': 0.02335758824826675}
Epoch 1 ; Time: 0.321528 ; Training: accuracy=0.042066 ; Validation: accuracy=0.109933
config_id 4: Terminating evaluation at 1
Update for config_id 4:1: reward = 0.10993265993265994, crit_val = 0.89006734006734
Starting get_config[BO] for config_id 5
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.6726377221813429
- self.std = 0.2997807115320194
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 7
Current best is [0.46875]
[5: BO] (8 evaluations)
batch_size: 106
dropout_1: 0.38750926166961597
dropout_2: 0.04489600208822606
learning_rate: 0.002679472509181979
n_units_1: 93
n_units_2: 20
scale_1: 0.48508438196776654
scale_2: 0.03480880867031308
Started BO from (top scorer):
batch_size: 106
dropout_1: 0.38750926119776946
dropout_2: 0.04489600246106559
learning_rate: 0.002265665895170696
n_units_1: 93
n_units_2: 20
scale_1: 0.4850843918016997
scale_2: 0.03481291816391773
Top score values: [0.18489824 0.20828588 0.23691933 0.2416328  0.24992744]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1. Pending:
Targets: [-0.68012288 -1.22530749 -1.46333799  0.71925648  0.96418128  0.96003504
  0.72529556]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00011518069880241772, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 7.5553647632413945, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 0.00017086338991248112, 'kernel_inv_bw7': 0.03572986667956557, 'kernel_inv_bw8': 1.311175980985117, 'kernel_covariance_scale': 0.6932464483370223, 'mean_mean_value': 0.12225736946212887}
Epoch 1 ; Time: 0.337694 ; Training: accuracy=0.355842 ; Validation: accuracy=0.601787
Update for config_id 5:1: reward = 0.6017874875868917, crit_val = 0.3982125124131083
config_id 5: Reaches 1, continues to 3
Epoch 2 ; Time: 0.650713 ; Training: accuracy=0.574892 ; Validation: accuracy=0.681397
Epoch 3 ; Time: 0.920556 ; Training: accuracy=0.622559 ; Validation: accuracy=0.727574
Update for config_id 5:3: reward = 0.7275736511089044, crit_val = 0.2724263488910956
config_id 5: Reaches 3, continues to 9
Epoch 4 ; Time: 1.204922 ; Training: accuracy=0.648047 ; Validation: accuracy=0.750083
Epoch 5 ; Time: 1.478106 ; Training: accuracy=0.672459 ; Validation: accuracy=0.760344
Epoch 6 ; Time: 1.744974 ; Training: accuracy=0.684542 ; Validation: accuracy=0.766137
Epoch 7 ; Time: 2.012398 ; Training: accuracy=0.701258 ; Validation: accuracy=0.792453
Epoch 8 ; Time: 2.278716 ; Training: accuracy=0.709782 ; Validation: accuracy=0.798080
Epoch 9 ; Time: 2.547311 ; Training: accuracy=0.713009 ; Validation: accuracy=0.812479
config_id 5: Terminating evaluation at 9
Update for config_id 5:9: reward = 0.8124793114862628, crit_val = 0.18752068851373715
Starting get_config[BO] for config_id 6
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5566623605087341
- self.std = 0.3107077976680413
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 10
Current best is [0.39821251]
[6: BO] (8 evaluations)
batch_size: 100
dropout_1: 0.4870263486471881
dropout_2: 0.59113405302611
learning_rate: 0.00513747238186704
n_units_1: 105
n_units_2: 107
scale_1: 0.2432746016644627
scale_2: 3.384577528513164
Started BO from (top scorer):
batch_size: 100
dropout_1: 0.4870263426644337
dropout_2: 0.5911340539952046
learning_rate: 0.009911435239249514
n_units_1: 105
n_units_2: 107
scale_1: 0.24327462238842057
scale_2: 3.38457748137845
Top score values: [0.19986271 0.27163003 0.37278358 0.37794283 0.37953738]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9. Pending:
Targets: [-0.28294224 -0.80895359 -1.03861295  1.06722324  1.30353443  1.29953401
  1.07304993 -0.50996418 -0.91480167 -1.18806697]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0005635679216323996, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 5.72011418892912, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.002059714008881696, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.1165087347384535, 'kernel_covariance_scale': 0.6850617763692147, 'mean_mean_value': 0.3838207699127142}
Epoch 1 ; Time: 0.348299 ; Training: accuracy=0.281405 ; Validation: accuracy=0.649833
Update for config_id 6:1: reward = 0.6498333333333334, crit_val = 0.3501666666666666
config_id 6: Reaches 1, continues to 3
Epoch 2 ; Time: 0.638623 ; Training: accuracy=0.446942 ; Validation: accuracy=0.693167
Epoch 3 ; Time: 0.918685 ; Training: accuracy=0.491240 ; Validation: accuracy=0.712833
config_id 6: Terminating evaluation at 3
Update for config_id 6:3: reward = 0.7128333333333333, crit_val = 0.2871666666666667
Starting get_config[BO] for config_id 7
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5169964115350562
- self.std = 0.297458835498879
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 12
Current best is [0.35016667]
[7: BO] (5 evaluations)
batch_size: 70
dropout_1: 0.6131746934906157
dropout_2: 0.02296778309071037
learning_rate: 0.008313446396583152
n_units_1: 24
n_units_2: 99
scale_1: 0.0028554704441241347
scale_2: 0.01197150439687674
Started BO from (top scorer):
batch_size: 70
dropout_1: 0.6131746914586562
dropout_2: 0.022967791041196395
learning_rate: 0.007721593613613116
n_units_1: 24
n_units_2: 99
scale_1: 0.0028554705455487786
scale_2: 0.011971504865511351
Top score values: [0.25898139 0.30766019 0.30984522 0.31771857 0.32951799]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3. Pending:
Targets: [-0.16219525 -0.71163541 -0.9515239   1.24810725  1.49494386  1.49076526
  1.25419347 -0.39932886 -0.82219801 -1.10763468 -0.56084986 -0.77264387]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.000184988308049362, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00017831600435311493, 'kernel_inv_bw3': 6.785443344162829, 'kernel_inv_bw4': 0.0008660651295677859, 'kernel_inv_bw5': 0.00029962703298209927, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.0001574295344670858, 'kernel_inv_bw8': 1.1136085785503336, 'kernel_covariance_scale': 0.6935746028745766, 'mean_mean_value': 0.4582469893961607}
Epoch 1 ; Time: 0.510453 ; Training: accuracy=0.270933 ; Validation: accuracy=0.543866
Update for config_id 7:1: reward = 0.5438655462184874, crit_val = 0.4561344537815126
config_id 7: Reaches 1, continues to 3
Epoch 2 ; Time: 0.919471 ; Training: accuracy=0.385632 ; Validation: accuracy=0.572269
Epoch 3 ; Time: 1.320536 ; Training: accuracy=0.422048 ; Validation: accuracy=0.567899
config_id 7: Terminating evaluation at 3
Update for config_id 7:3: reward = 0.5678991596638655, crit_val = 0.4321008403361345
Starting get_config[BO] for config_id 8
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5065851594670229
- self.std = 0.27660893273101
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 14
Current best is [0.35016667]
[8: BO] (3 evaluations)
batch_size: 112
dropout_1: 0.4125805230977853
dropout_2: 0.07535360976307452
learning_rate: 1.0000000000000004e-06
n_units_1: 40
n_units_2: 33
scale_1: 0.19181973129262858
scale_2: 0.10218151730880085
Started BO from (top scorer):
batch_size: 112
dropout_1: 0.4125805242468248
dropout_2: 0.07535360980344002
learning_rate: 2.48539902503145e-06
n_units_1: 40
n_units_2: 33
scale_1: 0.1918197310941
scale_2: 0.1021815173022151
Top score values: [0.30025592 0.30262276 0.3117204  0.3394005  0.34197217]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3. Pending:
Targets: [-0.13678213 -0.72763734 -0.98560787  1.3798245   1.64526687  1.6407733
  1.38636947 -0.39179012 -0.8465338  -1.15348578 -0.56548605 -0.79324442
 -0.18239001 -0.26927662]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.0009286905298238828, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 7.992950577667862, 'kernel_inv_bw4': 0.00500631918082955, 'kernel_inv_bw5': 0.0009410828273667761, 'kernel_inv_bw6': 0.0004477638682613567, 'kernel_inv_bw7': 0.00011420460011308597, 'kernel_inv_bw8': 1.0592287834123066, 'kernel_covariance_scale': 0.7461873568270722, 'mean_mean_value': 0.6137628308851325}
Epoch 1 ; Time: 0.312589 ; Training: accuracy=0.037616 ; Validation: accuracy=0.032738
config_id 8: Terminating evaluation at 1
Update for config_id 8:1: reward = 0.03273809523809524, crit_val = 0.9672619047619048
Starting get_config[BO] for config_id 9
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5372969424866817
- self.std = 0.290889427187723
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 15
Current best is [0.35016667]
[9: BO] (16 evaluations)
batch_size: 16
dropout_1: 0.4938297203182308
dropout_2: 0.17576536414296595
learning_rate: 0.00429168229252887
n_units_1: 75
n_units_2: 24
scale_1: 0.0011204811934850463
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 16
dropout_1: 0.4938284333967552
dropout_2: 0.17577032689555921
learning_rate: 0.0044596526917615315
n_units_1: 75
n_units_2: 24
scale_1: 0.0011206396260318283
scale_2: 0.01320113262596052
Top score values: [0.34235655 0.34256671 0.34447978 0.34541647 0.34632746]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1. Pending:
Targets: [-0.23564604 -0.79749468 -1.04280079  1.20650655  1.45891769  1.45464472
  1.21273022 -0.47813505 -0.91055421 -1.20243715 -0.64330381 -0.85988095
 -0.27901491 -0.36163605  1.47810447]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 7.870357811798808, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.016733381670634786, 'kernel_inv_bw8': 1.0231829282362996, 'kernel_covariance_scale': 0.698218329573643, 'mean_mean_value': 0.6913674933513881}
Epoch 1 ; Time: 1.831780 ; Training: accuracy=0.352371 ; Validation: accuracy=0.595766
Update for config_id 9:1: reward = 0.5957661290322581, crit_val = 0.4042338709677419
config_id 9: Reaches 1, continues to 3
Epoch 2 ; Time: 3.575926 ; Training: accuracy=0.527851 ; Validation: accuracy=0.699429
Epoch 3 ; Time: 5.439312 ; Training: accuracy=0.578664 ; Validation: accuracy=0.737063
Update for config_id 9:3: reward = 0.7370631720430108, crit_val = 0.26293682795698925
config_id 9: Reaches 3, continues to 9
Epoch 4 ; Time: 7.173787 ; Training: accuracy=0.602951 ; Validation: accuracy=0.746472
Epoch 5 ; Time: 8.889302 ; Training: accuracy=0.610660 ; Validation: accuracy=0.744624
Epoch 6 ; Time: 10.627008 ; Training: accuracy=0.628979 ; Validation: accuracy=0.774194
Epoch 7 ; Time: 12.345299 ; Training: accuracy=0.630637 ; Validation: accuracy=0.775538
Epoch 8 ; Time: 14.070482 ; Training: accuracy=0.630471 ; Validation: accuracy=0.762937
Epoch 9 ; Time: 15.792049 ; Training: accuracy=0.644148 ; Validation: accuracy=0.784946
config_id 9: Terminating evaluation at 9
Update for config_id 9:9: reward = 0.7849462365591398, crit_val = 0.21505376344086025
Starting get_config[BO] for config_id 10
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4967599222036566
- self.std = 0.2824979156649466
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 18
Current best is [0.35016667]
[10: BO] (13 evaluations)
batch_size: 82
dropout_1: 0.1455727987618754
dropout_2: 0.5231235410428672
learning_rate: 0.003859510358291107
n_units_1: 40
n_units_2: 101
scale_1: 5.981988219333076
scale_2: 10.0
Started BO from (top scorer):
batch_size: 82
dropout_1: 0.14557280856507585
dropout_2: 0.5231231614760518
learning_rate: 0.0033261781877540063
n_units_1: 40
n_units_2: 101
scale_1: 5.981988181743532
scale_2: 6.375505195149974
Top score values: [0.32961825 0.34897599 0.34944606 0.35127473 0.35325755]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9. Pending:
Targets: [-0.0991509  -0.67768907 -0.93028192  1.38584038  1.64574931  1.64134941
  1.39224892 -0.34884296 -0.794107   -1.09466023 -0.51891801 -0.7419285
 -0.14380803 -0.22888339  1.66550603 -0.32752826 -0.82769848 -0.9971973 ]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.0005885942367902979, 'kernel_inv_bw3': 7.9443254148791445, 'kernel_inv_bw4': 0.0005758773902456391, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.2689542400990702, 'kernel_inv_bw8': 1.3350609218557346, 'kernel_covariance_scale': 0.7164309577766733, 'mean_mean_value': 0.7058448222147075}
Epoch 1 ; Time: 0.399611 ; Training: accuracy=0.257746 ; Validation: accuracy=0.483628
config_id 10: Terminating evaluation at 1
Update for config_id 10:1: reward = 0.4836284664216505, crit_val = 0.5163715335783494
Starting get_config[BO] for config_id 11
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.49779211227600884
- self.std = 0.2749981506905145
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 19
Current best is [0.35016667]
[11: BO] (14 evaluations)
batch_size: 79
dropout_1: 0.75
dropout_2: 0.4233827385525278
learning_rate: 0.00468292399852034
n_units_1: 123
n_units_2: 57
scale_1: 0.4942438135442485
scale_2: 5.998665338357913
Started BO from (top scorer):
batch_size: 79
dropout_1: 0.6243451100425594
dropout_2: 0.42338274050078417
learning_rate: 0.002139116508757636
n_units_1: 119
n_units_2: 57
scale_1: 0.49430045483873486
scale_2: 5.998675112100447
Top score values: [0.31483642 0.32896679 0.3468756  0.35287739 0.36168769]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1. Pending:
Targets: [-0.10560839 -0.69992449 -0.95940606  1.41988166  1.68687883  1.68235894
  1.42646497 -0.36211007 -0.81951738 -1.12826731 -0.53682341 -0.76591586
 -0.15148341 -0.23887896  1.70717436 -0.34021407 -0.85402496 -1.02814636
  0.06756199]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.4763779717508467, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 5.2678425643977915, 'kernel_inv_bw4': 1.1712606543513226, 'kernel_inv_bw5': 0.00046897355421093135, 'kernel_inv_bw6': 0.00195378084467982, 'kernel_inv_bw7': 0.00036277729241632173, 'kernel_inv_bw8': 1.1829355130286097, 'kernel_covariance_scale': 0.7578611012181969, 'mean_mean_value': 0.8107491624272125}
Epoch 1 ; Time: 0.426325 ; Training: accuracy=0.111194 ; Validation: accuracy=0.353098
config_id 11: Terminating evaluation at 1
Update for config_id 11:1: reward = 0.3530979347101932, crit_val = 0.6469020652898068
Starting get_config[BO] for config_id 12
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5052476099266987
- self.std = 0.2699979410999501
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 20
Current best is [0.35016667]
[12: BO] (15 evaluations)
batch_size: 51
dropout_1: 0.0
dropout_2: 0.6946818865220364
learning_rate: 0.00024034828518000115
n_units_1: 48
n_units_2: 128
scale_1: 0.09785505736236708
scale_2: 0.002830752329069545
Started BO from (top scorer):
batch_size: 51
dropout_1: 0.12842037118675856
dropout_2: 0.6946818859317343
learning_rate: 2.843859105310109e-06
n_units_1: 48
n_units_2: 127
scale_1: 0.09785507387618159
scale_2: 0.00283098271302883
Top score values: [0.23055987 0.24472849 0.27706981 0.34549748 0.35325744]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1. Pending:
Targets: [-0.13517736 -0.74049986 -1.00478689  1.4185639   1.69050571  1.68590212
  1.42526913 -0.39642931 -0.86230754 -1.17677535 -0.57437824 -0.80771336
 -0.18190197 -0.27091603  1.7111771  -0.37412781 -0.89745418 -1.07480022
  0.04120003  0.52465013]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.004378402490547834, 'kernel_inv_bw1': 1.7658950013971273, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 5.22388204390642, 'kernel_inv_bw4': 0.00024488573982846734, 'kernel_inv_bw5': 0.48906916665098116, 'kernel_inv_bw6': 0.0004108235867804542, 'kernel_inv_bw7': 0.006695837949954032, 'kernel_inv_bw8': 1.1479935466198072, 'kernel_covariance_scale': 0.7680910660672096, 'mean_mean_value': 1.0021929943741534}
Epoch 1 ; Time: 0.606386 ; Training: accuracy=0.208158 ; Validation: accuracy=0.244847
config_id 12: Terminating evaluation at 1
Update for config_id 12:1: reward = 0.24484665661136248, crit_val = 0.7551533433886375
Starting get_config[BO] for config_id 13
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5171478829486957
- self.std = 0.26881190920360515
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 21
Current best is [0.35016667]
[13: BO] (29 evaluations)
batch_size: 63
dropout_1: 0.40467054657502044
dropout_2: 0.193233018417109
learning_rate: 0.0033587042163634663
n_units_1: 16
n_units_2: 128
scale_1: 2.4918306209674417
scale_2: 0.007557821373477355
Started BO from (top scorer):
batch_size: 65
dropout_1: 0.4169220946335453
dropout_2: 0.1932436849600857
learning_rate: 0.0035468887104147884
n_units_1: 57
n_units_2: 60
scale_1: 2.4972005561729547
scale_2: 0.007559210365656552
Top score values: [0.32381977 0.32830845 0.35974254 0.36522478 0.36561045]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1. Pending:
Targets: [-0.18004367 -0.78803693 -1.05349002  1.38055289  1.65369455  1.64907064
  1.38728771 -0.44244829 -0.91038204 -1.22623732 -0.62118236 -0.85554698
 -0.22697443 -0.31638123  1.67445714 -0.4200484  -0.94568375 -1.12381226
 -0.00288808  0.48269507  0.88539775]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00376181918524633, 'kernel_inv_bw1': 1.753529736399951, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 5.243399418045996, 'kernel_inv_bw4': 0.017268110945428267, 'kernel_inv_bw5': 0.5115883450182885, 'kernel_inv_bw6': 0.0005624318230775499, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.1434737686195484, 'kernel_covariance_scale': 0.7399708613829901, 'mean_mean_value': 1.0149452792025901}
Epoch 1 ; Time: 0.504410 ; Training: accuracy=0.268519 ; Validation: accuracy=0.565079
Update for config_id 13:1: reward = 0.5650793650793651, crit_val = 0.43492063492063493
config_id 13: Reaches 1, continues to 3
Epoch 2 ; Time: 0.953655 ; Training: accuracy=0.369048 ; Validation: accuracy=0.635422
Epoch 3 ; Time: 1.393476 ; Training: accuracy=0.425678 ; Validation: accuracy=0.636090
config_id 13: Terminating evaluation at 3
Update for config_id 13:3: reward = 0.6360902255639098, crit_val = 0.3639097744360902
Starting get_config[BO] for config_id 14
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5069102587512755
- self.std = 0.2592035733038758
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 23
Current best is [0.35016667]
[14: BO] (12 evaluations)
batch_size: 74
dropout_1: 0.4290572893211159
dropout_2: 0.5073198541610456
learning_rate: 0.0046715551446154
n_units_1: 64
n_units_2: 73
scale_1: 0.019328883880041194
scale_2: 0.0011645755004720119
Started BO from (top scorer):
batch_size: 74
dropout_1: 0.49303188555081023
dropout_2: 0.5073198355850157
learning_rate: 0.005174423901175618
n_units_1: 64
n_units_2: 47
scale_1: 0.0193288857882071
scale_2: 0.0011645753928866541
Top score values: [0.29631371 0.31674666 0.31686626 0.3334686  0.33618963]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 13:1, 13:3. Pending:
Targets: [-0.14722119 -0.77775196 -1.05304505  1.47122464  1.7544913   1.74969599
  1.4782091  -0.41935281 -0.90463224 -1.23219586 -0.60471231 -0.84776452
 -0.19589161 -0.2886126   1.77602353 -0.39612258 -0.94124254 -1.12597404
  0.03650133  0.5400844   0.95771475 -0.27773392 -0.55169179]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 1.7362936891444805, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 5.131987092324683, 'kernel_inv_bw4': 0.002053380743302223, 'kernel_inv_bw5': 0.8653123957646578, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.0182979062244049, 'kernel_covariance_scale': 0.7722930639907186, 'mean_mean_value': 1.1448384610804114}
Epoch 1 ; Time: 0.444210 ; Training: accuracy=0.334024 ; Validation: accuracy=0.623457
Update for config_id 14:1: reward = 0.6234567901234568, crit_val = 0.3765432098765432
config_id 14: Reaches 1, continues to 3
Epoch 2 ; Time: 0.834786 ; Training: accuracy=0.523711 ; Validation: accuracy=0.704204
Epoch 3 ; Time: 1.211267 ; Training: accuracy=0.573454 ; Validation: accuracy=0.735235
Update for config_id 14:3: reward = 0.7352352352352353, crit_val = 0.26476476476476474
config_id 14: Reaches 3, continues to 9
Epoch 4 ; Time: 1.632802 ; Training: accuracy=0.599569 ; Validation: accuracy=0.757424
Epoch 5 ; Time: 2.007355 ; Training: accuracy=0.627839 ; Validation: accuracy=0.769269
Epoch 6 ; Time: 2.382748 ; Training: accuracy=0.641768 ; Validation: accuracy=0.788288
Epoch 7 ; Time: 2.757718 ; Training: accuracy=0.649975 ; Validation: accuracy=0.793627
Epoch 8 ; Time: 3.134140 ; Training: accuracy=0.655364 ; Validation: accuracy=0.797464
Epoch 9 ; Time: 3.509736 ; Training: accuracy=0.667800 ; Validation: accuracy=0.806473
config_id 14: Terminating evaluation at 9
Update for config_id 14:9: reward = 0.8064731398064732, crit_val = 0.19352686019352683
Starting get_config[BO] for config_id 15
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4805296456197758
- self.std = 0.25578191824386487
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 26
Current best is [0.35016667]
[15: BO] (22 evaluations)
batch_size: 55
dropout_1: 0.42762622837718267
dropout_2: 0.6805103726352633
learning_rate: 0.003931314287417092
n_units_1: 85
n_units_2: 128
scale_1: 0.053293300662071295
scale_2: 10.0
Started BO from (top scorer):
batch_size: 55
dropout_1: 0.534180671962703
dropout_2: 0.6805056958385995
learning_rate: 0.003939276585290293
n_units_1: 37
n_units_2: 82
scale_1: 0.053293349524855406
scale_2: 4.228969909955284
Top score values: [0.27709614 0.33799434 0.38042335 0.38085229 0.38325832]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9. Pending:
Targets: [-0.04605347 -0.685019   -0.96399475  1.59404269  1.88109867  1.87623921
  1.60112058 -0.32182546 -0.81359659 -1.14554211 -0.50966456 -0.75596813
 -0.09537497 -0.18933631  1.90291895 -0.29828447 -0.85069664 -1.03789933
  0.14012675  0.65044637  1.07366345 -0.1783121  -0.45593477 -0.40654334
 -0.84355017 -1.12206049]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0008102460975812275, 'kernel_inv_bw1': 1.5233880553857808, 'kernel_inv_bw2': 0.0008833887440650072, 'kernel_inv_bw3': 5.160620777051369, 'kernel_inv_bw4': 0.7832626419399352, 'kernel_inv_bw5': 0.280131080081185, 'kernel_inv_bw6': 0.00015583387491305457, 'kernel_inv_bw7': 0.22597802996135352, 'kernel_inv_bw8': 1.135056397037835, 'kernel_covariance_scale': 0.7333046175267716, 'mean_mean_value': 1.1956539724861677}
Epoch 1 ; Time: 0.676336 ; Training: accuracy=0.298099 ; Validation: accuracy=0.642256
Update for config_id 15:1: reward = 0.6422558922558923, crit_val = 0.3577441077441077
config_id 15: Reaches 1, continues to 3
Epoch 2 ; Time: 1.189129 ; Training: accuracy=0.427107 ; Validation: accuracy=0.671549
Epoch 3 ; Time: 1.697465 ; Training: accuracy=0.453058 ; Validation: accuracy=0.680303
config_id 15: Terminating evaluation at 3
Update for config_id 15:3: reward = 0.6803030303030303, crit_val = 0.3196969696969697
Starting get_config[BO] for config_id 16
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.47040042369840174
- self.std = 0.24922055194579654
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 28
Current best is [0.26293683]
[16: BO] (30 evaluations)
batch_size: 80
dropout_1: 0.4506887925316827
dropout_2: 0.6719571863210274
learning_rate: 0.0035717166316857614
n_units_1: 128
n_units_2: 59
scale_1: 0.5413260512980093
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 80
dropout_1: 0.42660095528057573
dropout_2: 0.671954869672605
learning_rate: 0.0024691812805164395
n_units_1: 26
n_units_2: 35
scale_1: 0.5413411182074122
scale_2: 9.766007104880408
Top score values: [0.22764364 0.23267679 0.23485306 0.23830916 0.24878035]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3. Pending:
Targets: [-0.00662234 -0.66241027 -0.94873076  1.67665353  1.971267    1.9662796
  1.68391777 -0.28965473 -0.79437299 -1.13505782 -0.48243917 -0.73522731
 -0.05724235 -0.15367747  1.99366175 -0.26549397 -0.83244979 -1.02458107
  0.18445955  0.70821463  1.14257399 -0.14236301 -0.42729481 -0.37660303
 -0.82511517 -1.11095799 -0.45203461 -0.60469914]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 1.8301893134231024, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 5.681015262023943, 'kernel_inv_bw4': 0.2699791411074525, 'kernel_inv_bw5': 0.5061216656978157, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.16522408078152165, 'kernel_inv_bw8': 1.178841550227455, 'kernel_covariance_scale': 0.7582392011573784, 'mean_mean_value': 1.2312231584310094}
Epoch 1 ; Time: 0.407867 ; Training: accuracy=0.328974 ; Validation: accuracy=0.631000
Update for config_id 16:1: reward = 0.631, crit_val = 0.369
config_id 16: Reaches 1, continues to 3
Epoch 2 ; Time: 0.765634 ; Training: accuracy=0.470116 ; Validation: accuracy=0.692333
Epoch 3 ; Time: 1.117356 ; Training: accuracy=0.508195 ; Validation: accuracy=0.714833
config_id 16: Terminating evaluation at 3
Update for config_id 16:3: reward = 0.7148333333333333, crit_val = 0.2851666666666667
Starting get_config[BO] for config_id 17
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.46084595100739717
- self.std = 0.2436500010642932
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 30
Current best is [0.26293683]
[17: BO] (22 evaluations)
batch_size: 9
dropout_1: 0.41908997488829114
dropout_2: 0.07488155004342928
learning_rate: 0.006257236846515682
n_units_1: 94
n_units_2: 16
scale_1: 2.041686994448543
scale_2: 0.007527342543043179
Started BO from (top scorer):
batch_size: 9
dropout_1: 0.37045365149070153
dropout_2: 0.07488158908445264
learning_rate: 0.00841937723801848
n_units_1: 76
n_units_2: 78
scale_1: 2.0416895489325446
scale_2: 0.007527334022259543
Top score values: [0.22939282 0.23173621 0.23307707 0.23641874 0.26439591]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 16:1, 16:3. Pending:
Targets: [ 0.03244018 -0.63834098 -0.9312076   1.75420065  2.05554984  2.05044842
  1.76163098 -0.25706316 -0.77332075 -1.12179463 -0.45425522 -0.71282283
 -0.01933715 -0.11797706  2.0784566  -0.23235001 -0.8122681  -1.00879206
  0.22789075  0.76362041  1.20791049 -0.10640392 -0.3978501  -0.34599935
 -0.80476579 -1.09714381 -0.42315552 -0.57931041 -0.37695855 -0.72103133]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 1.730294061320088, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 6.439164292618701, 'kernel_inv_bw4': 0.3511376127951275, 'kernel_inv_bw5': 0.3602583892952772, 'kernel_inv_bw6': 0.0001306211639428832, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.4138001601172758, 'kernel_covariance_scale': 0.8040910580989261, 'mean_mean_value': 1.193575207423106}
Epoch 1 ; Time: 3.151741 ; Training: accuracy=0.435240 ; Validation: accuracy=0.656902
Update for config_id 17:1: reward = 0.6569023569023569, crit_val = 0.3430976430976431
config_id 17: Reaches 1, continues to 3
Epoch 2 ; Time: 6.329521 ; Training: accuracy=0.549171 ; Validation: accuracy=0.688215
Epoch 3 ; Time: 9.401875 ; Training: accuracy=0.576368 ; Validation: accuracy=0.714646
config_id 17: Terminating evaluation at 3
Update for config_id 17:3: reward = 0.7146464646464646, crit_val = 0.28535353535353536
Starting get_config[BO] for config_id 18
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4516821783960342
- self.std = 0.23867700135430228
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 32
Current best is [0.26293683]
[18: BO] (28 evaluations)
batch_size: 29
dropout_1: 0.43125922144917483
dropout_2: 0.14232697437371072
learning_rate: 0.0017320946859465094
n_units_1: 16
n_units_2: 16
scale_1: 0.012816079985121014
scale_2: 0.03762779559898151
Started BO from (top scorer):
batch_size: 29
dropout_1: 0.6420664042353282
dropout_2: 0.14232698216264267
learning_rate: 0.0015986669988118994
n_units_1: 49
n_units_2: 98
scale_1: 0.012848630503362952
scale_2: 0.037627741428044
Top score values: [0.23803555 0.24386695 0.26410476 0.26495884 0.27797719]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 16:1, 16:3, 17:1, 17:3. Pending:
Targets: [ 0.07151012 -0.61324722 -0.91221591  1.82914467  2.13677267  2.13156496
  1.8367298  -0.22402521 -0.75103939 -1.10677396 -0.42532591 -0.68928096
  0.01865398 -0.08204116  2.16015671 -0.19879715 -0.79079823 -0.99141691
  0.27103305  0.817925    1.27147217 -0.07022689 -0.36774555 -0.31481445
 -0.78313961 -1.08160953 -0.39357823 -0.55298671 -0.34641871 -0.69766048
 -0.45494344 -0.69687755]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 1.7727174225446345, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 6.782888824305436, 'kernel_inv_bw4': 0.4238247058940885, 'kernel_inv_bw5': 0.45710443243792287, 'kernel_inv_bw6': 0.00395596942024941, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.5179122090884987, 'kernel_covariance_scale': 0.7829745981348675, 'mean_mean_value': 1.1908179846813767}
Epoch 1 ; Time: 1.006335 ; Training: accuracy=0.142158 ; Validation: accuracy=0.299748
config_id 18: Terminating evaluation at 1
Update for config_id 18:1: reward = 0.29974768713204375, crit_val = 0.7002523128679563
Starting get_config[BO] for config_id 19
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.45921460671336517
- self.std = 0.23886407484155286
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 33
Current best is [0.26293683]
[19: BO] (23 evaluations)
batch_size: 11
dropout_1: 0.45691507437324347
dropout_2: 0.397776297677671
learning_rate: 0.0038417306644828067
n_units_1: 89
n_units_2: 56
scale_1: 0.0025456088371450395
scale_2: 0.04677716386344848
Started BO from (top scorer):
batch_size: 11
dropout_1: 0.27773647423520337
dropout_2: 0.3977763768050508
learning_rate: 0.0011784010178056816
n_units_1: 117
n_units_2: 92
scale_1: 0.0025456093439067047
scale_2: 0.04677717904168564
Top score values: [0.20490766 0.24055197 0.24854143 0.27172953 0.27970916]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 16:1, 16:3, 17:1, 17:3, 18:1. Pending:
Targets: [ 0.03991975 -0.64430131 -0.94303586  1.79617775  2.10356482  2.09836119
  1.80375694 -0.25538413 -0.78198556 -1.13744153 -0.45652717 -0.7202755
 -0.012895   -0.11351128  2.12693055 -0.23017583 -0.82171326 -1.02217482
  0.23928641  0.78575005  1.23894201 -0.10170626 -0.39899191 -0.34610226
 -0.81406064 -1.1122968  -0.42480435 -0.58408799 -0.37768177 -0.72864846
 -0.48612151 -0.72786614  1.00909987]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 1.7982124325939244, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 6.4714081782141495, 'kernel_inv_bw4': 0.9000793709544641, 'kernel_inv_bw5': 0.677626658826137, 'kernel_inv_bw6': 0.0001249857957162385, 'kernel_inv_bw7': 0.00011029213858002101, 'kernel_inv_bw8': 1.1281900019436804, 'kernel_covariance_scale': 0.7903757989592313, 'mean_mean_value': 1.3220158144002192}
Epoch 1 ; Time: 2.650803 ; Training: accuracy=0.444187 ; Validation: accuracy=0.682155
Update for config_id 19:1: reward = 0.6821548821548822, crit_val = 0.3178451178451178
config_id 19: Reaches 1, continues to 3
Epoch 2 ; Time: 5.212583 ; Training: accuracy=0.583326 ; Validation: accuracy=0.733165
Epoch 3 ; Time: 7.750841 ; Training: accuracy=0.619292 ; Validation: accuracy=0.775926
Update for config_id 19:3: reward = 0.7759259259259259, crit_val = 0.2240740740740741
config_id 19: Reaches 3, continues to 9
Epoch 4 ; Time: 10.307248 ; Training: accuracy=0.635784 ; Validation: accuracy=0.789057
Epoch 5 ; Time: 12.823853 ; Training: accuracy=0.652109 ; Validation: accuracy=0.795286
Epoch 6 ; Time: 15.310953 ; Training: accuracy=0.658573 ; Validation: accuracy=0.804882
Epoch 7 ; Time: 17.853411 ; Training: accuracy=0.675230 ; Validation: accuracy=0.808418
Epoch 8 ; Time: 20.377344 ; Training: accuracy=0.681611 ; Validation: accuracy=0.805219
Epoch 9 ; Time: 23.066664 ; Training: accuracy=0.682688 ; Validation: accuracy=0.816498
config_id 19: Terminating evaluation at 9
Update for config_id 19:9: reward = 0.8164983164983165, crit_val = 0.1835016835016835
Starting get_config[BO] for config_id 20
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4410973026933868
- self.std = 0.23701424144784278
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 36
Current best is [0.22407407]
[20: BO] (28 evaluations)
batch_size: 21
dropout_1: 0.4985657438263086
dropout_2: 0.31596390970399413
learning_rate: 0.002291434153237027
n_units_1: 81
n_units_2: 74
scale_1: 0.007192848484107947
scale_2: 0.28479200588957154
Started BO from (top scorer):
batch_size: 21
dropout_1: 0.4529491928048109
dropout_2: 0.3159639174162398
learning_rate: 0.004439547486620908
n_units_1: 97
n_units_2: 35
scale_1: 0.007192848067690455
scale_2: 0.2847919284324264
Top score values: [0.23964188 0.26306247 0.26378048 0.26475681 0.26544197]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 16:1, 16:3, 17:1, 17:3, 18:1, 19:1, 19:3, 19:9. Pending:
Targets: [ 0.11667104 -0.57289018 -0.87395627  1.88663616  2.19642232  2.19117807
  1.89427452 -0.18093761 -0.71164902 -1.06987923 -0.38365052 -0.64945733
  0.06344408 -0.03795748  2.21997041 -0.15553256 -0.75168679 -0.9537129
  0.3175937   0.86832235  1.32505135 -0.02606032 -0.3256662  -0.27236377
 -0.74397444 -1.04453826 -0.35168011 -0.51220691 -0.30418975 -0.65789564
 -0.41347583 -0.65710721  1.09341535 -0.52002016 -0.9156548  -1.08683604]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000036402100007, 'kernel_inv_bw1': 1.6916244552301682, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 5.90930417022225, 'kernel_inv_bw4': 1.1476021642343148, 'kernel_inv_bw5': 0.6431100740024973, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.0001202020448337597, 'kernel_inv_bw8': 1.2026781628407233, 'kernel_covariance_scale': 0.788759501489613, 'mean_mean_value': 1.4005925756949087}
Epoch 1 ; Time: 1.532699 ; Training: accuracy=0.434865 ; Validation: accuracy=0.674743
Update for config_id 20:1: reward = 0.6747433955914521, crit_val = 0.32525660440854787
config_id 20: Reaches 1, continues to 3
Epoch 2 ; Time: 2.865534 ; Training: accuracy=0.586253 ; Validation: accuracy=0.725728
Epoch 3 ; Time: 4.186523 ; Training: accuracy=0.627495 ; Validation: accuracy=0.762578
Update for config_id 20:3: reward = 0.762577822648494, crit_val = 0.23742217735150595
config_id 20: Reaches 3, continues to 9
Epoch 4 ; Time: 5.504053 ; Training: accuracy=0.655404 ; Validation: accuracy=0.785967
Epoch 5 ; Time: 6.820409 ; Training: accuracy=0.675197 ; Validation: accuracy=0.799260
Epoch 6 ; Time: 8.153004 ; Training: accuracy=0.686957 ; Validation: accuracy=0.801279
Epoch 7 ; Time: 9.480141 ; Training: accuracy=0.697640 ; Validation: accuracy=0.820293
Epoch 8 ; Time: 10.824791 ; Training: accuracy=0.710228 ; Validation: accuracy=0.819956
Epoch 9 ; Time: 12.151117 ; Training: accuracy=0.700621 ; Validation: accuracy=0.836615
config_id 20: Terminating evaluation at 9
Update for config_id 20:9: reward = 0.8366145044590274, crit_val = 0.1633854955409726
Starting get_config[BO] for config_id 21
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.42578377369905
- self.std = 0.234532149270782
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 39
Current best is [0.22407407]
[21: BO] (34 evaluations)
batch_size: 84
dropout_1: 0.24405649768055715
dropout_2: 0.5682353905292297
learning_rate: 0.003866301823213872
n_units_1: 118
n_units_2: 68
scale_1: 0.110676890904691
scale_2: 2.7198656549274713
Started BO from (top scorer):
batch_size: 84
dropout_1: 0.29045243854599945
dropout_2: 0.5681299155477846
learning_rate: 0.007410577045318821
n_units_1: 111
n_units_2: 123
scale_1: 0.11063448065875608
scale_2: 2.7172159701453054
Top score values: [0.12732899 0.23285289 0.25453678 0.2555106  0.2596913 ]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 16:1, 16:3, 17:1, 17:3, 18:1, 19:1, 19:3, 19:9, 20:1, 20:3, 20:9. Pending:
Targets: [ 0.18319973 -0.51365923 -0.81791155  1.97189669  2.28496136  2.27966161
  1.97961588 -0.11755856 -0.65388658 -1.01590799 -0.32241681 -0.5910367
  0.12940947  0.02693476  2.30875866 -0.09188464 -0.69434807 -0.89851225
  0.38624879  0.94280589  1.40436853  0.03895782 -0.26381884 -0.2099523
 -0.6865541  -0.99029883 -0.29010806 -0.45233374 -0.2421151  -0.59956431
 -0.35255777 -0.59876754  1.17028109 -0.46022968 -0.86005138 -1.03304426
 -0.42862853 -0.80313764 -1.11881582]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 1.716071542684936, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 6.198905369468684, 'kernel_inv_bw4': 1.1335804458357337, 'kernel_inv_bw5': 0.6718895443895254, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.0001376933543024904, 'kernel_inv_bw8': 1.249468973408061, 'kernel_covariance_scale': 0.7814018374928079, 'mean_mean_value': 1.4380276235477927}
Epoch 1 ; Time: 0.404428 ; Training: accuracy=0.400794 ; Validation: accuracy=0.707243
Update for config_id 21:1: reward = 0.7072434607645876, crit_val = 0.29275653923541245
config_id 21: Reaches 1, continues to 3
Epoch 2 ; Time: 0.815481 ; Training: accuracy=0.551587 ; Validation: accuracy=0.772971
Epoch 3 ; Time: 1.150014 ; Training: accuracy=0.599124 ; Validation: accuracy=0.783870
Update for config_id 21:3: reward = 0.783869885982562, crit_val = 0.21613011401743798
config_id 21: Reaches 3, continues to 9
Epoch 4 ; Time: 1.489168 ; Training: accuracy=0.631862 ; Validation: accuracy=0.790912
Epoch 5 ; Time: 1.821591 ; Training: accuracy=0.636905 ; Validation: accuracy=0.799463
Epoch 6 ; Time: 2.152489 ; Training: accuracy=0.652530 ; Validation: accuracy=0.814554
Epoch 7 ; Time: 2.484596 ; Training: accuracy=0.670800 ; Validation: accuracy=0.818243
Epoch 8 ; Time: 2.822749 ; Training: accuracy=0.677249 ; Validation: accuracy=0.823944
Epoch 9 ; Time: 3.153155 ; Training: accuracy=0.687335 ; Validation: accuracy=0.829645
config_id 21: Terminating evaluation at 9
Update for config_id 21:9: reward = 0.829644533869886, crit_val = 0.17035546613011399
Starting get_config[BO] for config_id 22
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4115430784201409
- self.std = 0.232152703448289
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 42
Current best is [0.21613011]
[22: BO] (15 evaluations)
batch_size: 84
dropout_1: 0.21649828569529656
dropout_2: 0.313857981977794
learning_rate: 0.00718899438320498
n_units_1: 128
n_units_2: 73
scale_1: 0.001930120035655991
scale_2: 3.033856422086701
Started BO from (top scorer):
batch_size: 84
dropout_1: 0.2595079228943481
dropout_2: 0.31385802404939867
learning_rate: 0.006424520709483749
n_units_1: 117
n_units_2: 57
scale_1: 0.0019301220923606107
scale_2: 3.033857530350639
Top score values: [0.20201621 0.2096012  0.23044524 0.23770343 0.23837534]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 16:1, 16:3, 17:1, 17:3, 18:1, 19:1, 19:3, 19:9, 20:1, 20:3, 20:9, 21:1, 21:3, 21:9. Pending:
Targets: [ 0.24641936 -0.45758204 -0.76495279  2.05344955  2.36972297  2.3643689
  2.06124785 -0.05742154 -0.59924665 -0.9649786  -0.26437948 -0.53575259
  0.19207778  0.08855276  2.39376418 -0.03148448 -0.64012285 -0.84637961
  0.45154958  1.0138111   1.48010452  0.10069905 -0.20518091 -0.15076227
 -0.632249   -0.93910695 -0.23173958 -0.395628   -0.18325472 -0.54436761
 -0.29482937 -0.54356267  1.2436178  -0.40360487 -0.80752454 -0.9822905
 -0.37167982 -0.75002745 -1.06894117 -0.51167416 -0.84174322 -1.03891796]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 1.250517126456807, 'kernel_inv_bw2': 0.00010657533207863419, 'kernel_inv_bw3': 6.297044180860336, 'kernel_inv_bw4': 1.2032534939760378, 'kernel_inv_bw5': 0.7016469768419507, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.153947413974426, 'kernel_covariance_scale': 0.8783083934298492, 'mean_mean_value': 1.4325264625109173}
Epoch 1 ; Time: 0.390454 ; Training: accuracy=0.540923 ; Validation: accuracy=0.762408
Update for config_id 22:1: reward = 0.7624077800134138, crit_val = 0.2375922199865862
config_id 22: Reaches 1, continues to 3
Epoch 2 ; Time: 0.738169 ; Training: accuracy=0.670387 ; Validation: accuracy=0.809021
Epoch 3 ; Time: 1.076316 ; Training: accuracy=0.704282 ; Validation: accuracy=0.816734
Update for config_id 22:3: reward = 0.8167337357478203, crit_val = 0.18326626425217973
config_id 22: Reaches 3, continues to 9
Epoch 4 ; Time: 1.424338 ; Training: accuracy=0.725860 ; Validation: accuracy=0.833166
Epoch 5 ; Time: 1.759171 ; Training: accuracy=0.741815 ; Validation: accuracy=0.858484
Epoch 6 ; Time: 2.118161 ; Training: accuracy=0.762401 ; Validation: accuracy=0.862005
Epoch 7 ; Time: 2.556355 ; Training: accuracy=0.765708 ; Validation: accuracy=0.870389
Epoch 8 ; Time: 2.907164 ; Training: accuracy=0.774884 ; Validation: accuracy=0.871563
Epoch 9 ; Time: 3.237925 ; Training: accuracy=0.780671 ; Validation: accuracy=0.884474
config_id 22: Terminating evaluation at 9
Update for config_id 22:9: reward = 0.8844735077129443, crit_val = 0.1155264922870557
Starting get_config[BO] for config_id 23
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.3960265393371497
- self.std = 0.2320318730089979
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 45
Current best is [0.11552649]
[23: BO] (17 evaluations)
batch_size: 21
dropout_1: 0.03762080001384937
dropout_2: 0.2021987139075742
learning_rate: 0.009913217688194184
n_units_1: 128
n_units_2: 94
scale_1: 0.033972802638160327
scale_2: 5.408674968860422
Started BO from (top scorer):
batch_size: 21
dropout_1: 0.1261911981363594
dropout_2: 0.20226308957482592
learning_rate: 0.00873054833781275
n_units_1: 126
n_units_2: 43
scale_1: 0.033972847054251624
scale_2: 5.408679213821764
Top score values: [0.0741922  0.10136904 0.11617398 0.11693578 0.12863733]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 16:1, 16:3, 17:1, 17:3, 18:1, 19:1, 19:3, 19:9, 20:1, 20:3, 20:9, 21:1, 21:3, 21:9, 22:1, 22:3, 22:9. Pending:
Targets: [ 0.31342013 -0.39094788 -0.6984787   2.12139133  2.43782945  2.43247259
  2.12919369  0.009421   -0.53268626 -0.89860866 -0.19764471 -0.46915913
  0.25905025  0.15547132  2.46188318  0.03537157 -0.57358375 -0.77994792
  0.51865717  1.08121148  1.54774773  0.16762393 -0.13841532 -0.08396833
 -0.56570579 -0.87272355 -0.16498781 -0.32896157 -0.1164777  -0.47777864
 -0.22811046 -0.47697328  1.31113786 -0.3369426  -0.74107261 -0.91592958
 -0.30500092 -0.68354558 -1.00262538 -0.44506817 -0.77530911 -0.97258653
 -0.68281274 -0.91694418 -1.20888585]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00034807084826716573, 'kernel_inv_bw1': 1.253347846818683, 'kernel_inv_bw2': 0.003374360874893918, 'kernel_inv_bw3': 5.913845749459688, 'kernel_inv_bw4': 1.2164746656023504, 'kernel_inv_bw5': 0.6540702525797073, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.0872460625797382, 'kernel_covariance_scale': 1.0068227498860507, 'mean_mean_value': 1.3887842517335733}
Epoch 1 ; Time: 1.400241 ; Training: accuracy=0.585921 ; Validation: accuracy=0.718661
Update for config_id 23:1: reward = 0.7186606091199731, crit_val = 0.2813393908800269
config_id 23: Reaches 1, continues to 3
Epoch 2 ; Time: 2.717346 ; Training: accuracy=0.686294 ; Validation: accuracy=0.747434
Epoch 3 ; Time: 4.031124 ; Training: accuracy=0.702857 ; Validation: accuracy=0.776207
Update for config_id 23:3: reward = 0.7762073027090695, crit_val = 0.22379269729093054
config_id 23: Reaches 3, continues to 9
Epoch 4 ; Time: 5.379706 ; Training: accuracy=0.724141 ; Validation: accuracy=0.783443
Epoch 5 ; Time: 6.720272 ; Training: accuracy=0.729524 ; Validation: accuracy=0.801279
Epoch 6 ; Time: 8.081211 ; Training: accuracy=0.742774 ; Validation: accuracy=0.806159
Epoch 7 ; Time: 9.489575 ; Training: accuracy=0.744099 ; Validation: accuracy=0.807673
Epoch 8 ; Time: 10.907590 ; Training: accuracy=0.759917 ; Validation: accuracy=0.812384
Epoch 9 ; Time: 12.347323 ; Training: accuracy=0.762899 ; Validation: accuracy=0.822985
config_id 23: Terminating evaluation at 9
Update for config_id 23:9: reward = 0.822985024398452, crit_val = 0.177014975601548
Starting get_config[BO] for config_id 24
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.38548627779050504
- self.std = 0.2285915478585371
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 48
Current best is [0.11552649]
[24: BO] (24 evaluations)
batch_size: 85
dropout_1: 0.27689159685958553
dropout_2: 0.46081970204967376
learning_rate: 0.008677550325246837
n_units_1: 128
n_units_2: 114
scale_1: 0.001028911642767248
scale_2: 0.031194392220863357
Started BO from (top scorer):
batch_size: 85
dropout_1: 0.5566114422886048
dropout_2: 0.46081974429643935
learning_rate: 0.011724856654620737
n_units_1: 91
n_units_2: 118
scale_1: 0.001029001714917599
scale_2: 0.031194407051080224
Top score values: [0.14810636 0.16308574 0.1650346  0.17160118 0.18089965]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 16:1, 16:3, 17:1, 17:3, 18:1, 19:1, 19:3, 19:9, 20:1, 20:3, 20:9, 21:1, 21:3, 21:9, 22:1, 22:3, 22:9, 23:1, 23:3, 23:9. Pending:
Targets: [ 0.36424672 -0.3507221  -0.66288128  2.19942806  2.52062861  2.51519113
  2.20734785  0.05567238 -0.49459365 -0.86602322 -0.1545097  -0.43011044
  0.30905857  0.20392076  2.54504435  0.0820135  -0.53610665 -0.74557662
  0.5725726   1.14359341  1.61715107  0.21625628 -0.09438889 -0.03912248
 -0.52811013 -0.83974854 -0.12136131 -0.28780289 -0.07212112 -0.43885967
 -0.18543395 -0.43804219  1.3769802  -0.29590403 -0.70611624 -0.88360482
 -0.26348163 -0.64772343 -0.9716054  -0.4056569  -0.740868   -0.94111446
 -0.64697955 -0.88463469 -1.18097011 -0.45560253 -0.70734715 -0.91198167]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 1.1920758703731407, 'kernel_inv_bw2': 0.00013113702036106052, 'kernel_inv_bw3': 5.997126585991591, 'kernel_inv_bw4': 1.2443123441783879, 'kernel_inv_bw5': 0.6554128420181277, 'kernel_inv_bw6': 0.0014667988988133025, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.0664138320943437, 'kernel_covariance_scale': 0.993960434141025, 'mean_mean_value': 1.476977261728907}
Epoch 1 ; Time: 0.404573 ; Training: accuracy=0.506711 ; Validation: accuracy=0.726555
Update for config_id 24:1: reward = 0.7265546218487395, crit_val = 0.27344537815126047
config_id 24: Reaches 1, continues to 3
Epoch 2 ; Time: 0.787543 ; Training: accuracy=0.683761 ; Validation: accuracy=0.803025
Epoch 3 ; Time: 1.111934 ; Training: accuracy=0.732891 ; Validation: accuracy=0.831092
Update for config_id 24:3: reward = 0.83109243697479, crit_val = 0.16890756302521004
config_id 24: Reaches 3, continues to 9
Epoch 4 ; Time: 1.443772 ; Training: accuracy=0.757415 ; Validation: accuracy=0.852773
Epoch 5 ; Time: 1.777350 ; Training: accuracy=0.769594 ; Validation: accuracy=0.865378
Epoch 6 ; Time: 2.127753 ; Training: accuracy=0.780282 ; Validation: accuracy=0.865210
Epoch 7 ; Time: 2.470772 ; Training: accuracy=0.791632 ; Validation: accuracy=0.880672
Epoch 8 ; Time: 2.828913 ; Training: accuracy=0.798923 ; Validation: accuracy=0.879664
Epoch 9 ; Time: 3.165489 ; Training: accuracy=0.797100 ; Validation: accuracy=0.886387
config_id 24: Terminating evaluation at 9
Update for config_id 24:9: reward = 0.8863865546218488, crit_val = 0.11361344537815121
Starting get_config[BO] for config_id 25
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.373711916088213
- self.std = 0.22728148396976072
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 51
Current best is [0.11361345]
[25: BO] (25 evaluations)
batch_size: 66
dropout_1: 0.2680066434652472
dropout_2: 0.694221361589957
learning_rate: 0.012997643499152702
n_units_1: 128
n_units_2: 87
scale_1: 0.058650373989879726
scale_2: 0.2276666190017815
Started BO from (top scorer):
batch_size: 66
dropout_1: 0.12832840740272275
dropout_2: 0.6942082513852165
learning_rate: 0.014601379220547455
n_units_1: 90
n_units_2: 99
scale_1: 0.05865040201561297
scale_2: 0.22766679901993808
Top score values: [-0.01342452  0.06791953  0.13432537  0.14354376  0.14544903]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 16:1, 16:3, 17:1, 17:3, 18:1, 19:1, 19:3, 19:9, 20:1, 20:3, 20:9, 21:1, 21:3, 21:9, 22:1, 22:3, 22:9, 23:1, 23:3, 23:9, 24:1, 24:3, 24:9. Pending:
Targets: [ 0.41815146 -0.30093848 -0.61489697  2.26391089  2.58696285  2.58149403
  2.27187633  0.10779847 -0.44563933 -0.81920984 -0.10359511 -0.38078443
  0.36264519  0.25690137  2.61151933  0.13429143 -0.48739161 -0.69806898
  0.62767813  1.20199034  1.67827762  0.26930799 -0.04312776  0.01245721
 -0.479349   -0.7927837  -0.07025565 -0.23765661 -0.02073163 -0.38958409
 -0.13469761 -0.3887619   1.43672239 -0.24580444 -0.65838114 -0.83689278
 -0.21319516 -0.59965175 -0.9254006  -0.35618993 -0.69333322 -0.89473391
 -0.59890359 -0.83792858 -1.1359721  -0.40642345 -0.65961915 -0.86543319
 -0.44115577 -0.90110443 -1.14438918]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 1.182166163041795, 'kernel_inv_bw2': 0.003996326511339676, 'kernel_inv_bw3': 5.791137776952736, 'kernel_inv_bw4': 1.2386444150261766, 'kernel_inv_bw5': 0.6589419037343729, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00010129412212643454, 'kernel_inv_bw8': 1.1217358498821017, 'kernel_covariance_scale': 1.0082971588353606, 'mean_mean_value': 1.507407085777986}
Epoch 1 ; Time: 0.489510 ; Training: accuracy=0.414638 ; Validation: accuracy=0.699832
Update for config_id 25:1: reward = 0.6998316498316498, crit_val = 0.3001683501683502
config_id 25: Reaches 1, continues to 3
Epoch 2 ; Time: 0.917078 ; Training: accuracy=0.537837 ; Validation: accuracy=0.760101
Epoch 3 ; Time: 1.350923 ; Training: accuracy=0.580642 ; Validation: accuracy=0.767677
Update for config_id 25:3: reward = 0.7676767676767676, crit_val = 0.23232323232323238
config_id 25: Reaches 3, continues to 9
Epoch 4 ; Time: 1.784789 ; Training: accuracy=0.590909 ; Validation: accuracy=0.767003
Epoch 5 ; Time: 2.256382 ; Training: accuracy=0.606889 ; Validation: accuracy=0.793266
Epoch 6 ; Time: 2.682153 ; Training: accuracy=0.611442 ; Validation: accuracy=0.786532
Epoch 7 ; Time: 3.099258 ; Training: accuracy=0.625849 ; Validation: accuracy=0.796633
Epoch 8 ; Time: 3.517808 ; Training: accuracy=0.620301 ; Validation: accuracy=0.815825
Epoch 9 ; Time: 3.935841 ; Training: accuracy=0.632969 ; Validation: accuracy=0.811616
config_id 25: Terminating evaluation at 9
Update for config_id 25:9: reward = 0.8116161616161616, crit_val = 0.18838383838383843

Analysing the results

The training history is stored in the results_df, the main fields are the runtime and 'best' (the objective).

Note: You will get slightly different curves for different pairs of scheduler/searcher, the time_out here is a bit too short to really see the difference in a significant way (it would be better to set it to >1000s). Generally speaking though, hyperband stopping / promotion + model will tend to significantly outperform other combinations given enough time.

results_df.head()
bracket elapsed_time epoch error eval_time objective runtime searcher_data_size searcher_params_kernel_covariance_scale searcher_params_kernel_inv_bw0 ... searcher_params_kernel_inv_bw7 searcher_params_kernel_inv_bw8 searcher_params_mean_mean_value searcher_params_noise_variance target_epoch task_id time_since_start time_step time_this_iter best
0 0 0.468420 1 0.468750 0.463765 0.531250 0.558362 NaN 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 0.560135 1.600500e+09 0.496789 0.468750
1 0 0.885907 2 0.344753 0.411108 0.655247 0.975849 1.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 0.978528 1.600500e+09 0.417470 0.344753
2 0 1.291719 3 0.305314 0.402126 0.694686 1.381661 1.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.382431 1.600500e+09 0.405813 0.305314
3 0 1.699280 4 0.288937 0.403800 0.711063 1.789222 2.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.790066 1.600500e+09 0.407560 0.288937
4 0 2.179495 5 0.273061 0.478450 0.726939 2.269437 2.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 2.270132 1.600500e+09 0.480215 0.273061

5 rows × 26 columns

import matplotlib.pyplot as plt

plt.figure(figsize=(12, 8))

runtime = results_df['runtime'].values
objective = results_df['best'].values

plt.plot(runtime, objective, lw=2)
plt.xticks(fontsize=12)
plt.xlim(0, 120)
plt.ylim(0, 0.5)
plt.yticks(fontsize=12)
plt.xlabel("Runtime [s]", fontsize=14)
plt.ylabel("Objective", fontsize=14)
Text(0, 0.5, 'Objective')
../../_images/output_mlp_cb387f_18_1.png

Diving Deeper

Now, you are ready to try HPO on your own machine learning models (if you use PyTorch, have a look at MNIST Training in PyTorch). While AutoGluon comes with well-chosen defaults, it can pay off to tune it to your specific needs. Here are some tips which may come useful.

Logging the Search Progress

First, it is a good idea in general to switch on debug_log, which outputs useful information about the search progress. This is already done in the example above.

The outputs show which configurations are chosen, stopped, or promoted. For BO and BOHB, a range of information is displayed for every get_config decision. This log output is very useful in order to figure out what is going on during the search.

Configuring HyperbandScheduler

The most important knobs to turn with HyperbandScheduler are max_t, grace_period, reduction_factor, brackets, and type. The first three determine the rung levels at which stopping or promotion decisions are being made.

  • The maximum resource level max_t (usually, resource equates to epochs, so max_t is the maximum number of training epochs) is typically hardcoded in train_fn passed to the scheduler (this is run_mlp_openml in the example above). As already noted above, the value is best fixed in the ag.args decorator as epochs=XYZ, it can then be accessed as args.epochs in the train_fn code. If this is done, you do not have to pass max_t when creating the scheduler.

  • grace_period and reduction_factor determine the rung levels, which are grace_period, grace_period * reduction_factor, grace_period * (reduction_factor ** 2), etc. All rung levels must be less or equal than max_t. It is recommended to make max_t equal to the largest rung level. For example, if grace_period = 1, reduction_factor = 3, it is in general recommended to use max_t = 9, max_t = 27, or max_t = 81. Choosing a max_t value “off the grid” works against the successive halving principle that the total resources spent in a rung should be roughly equal between rungs. If in the example above, you set max_t = 10, about a third of configurations reaching 9 epochs are allowed to proceed, but only for one more epoch.

  • With reduction_factor, you tune the extent to which successive halving filtering is applied. The larger this integer, the fewer configurations make it to higher number of epochs. Values 2, 3, 4 are commonly used.

  • Finally, grace_period should be set to the smallest resource (number of epochs) for which you expect any meaningful differentiation between configurations. While grace_period = 1 should always be explored, it may be too low for any meaningful stopping decisions to be made at the first rung.

  • brackets sets the maximum number of brackets in Hyperband (make sure to study the Hyperband paper or follow-ups for details). For brackets = 1, you are running successive halving (single bracket). Higher brackets have larger effective grace_period values (so runs are not stopped until later), yet are also chosen with less probability. We recommend to always consider successive halving (brackets = 1) in a comparison.

  • Finally, with type (values stopping, promotion) you are choosing different ways of extending successive halving scheduling to the asynchronous case. The method for the default stopping is simpler and seems to perform well, but promotion is more careful promoting configurations to higher resource levels, which can work better in some cases.

Asynchronous BOHB

Finally, here are some ideas for tuning asynchronous BOHB, apart from tuning its HyperbandScheduling component. You need to pass these options in search_options.

  • We support a range of different surrogate models over the criterion functions across resource levels. All of them are jointly dependent Gaussian process models, meaning that data collected at all resource levels are modelled together. The surrogate model is selected by gp_resource_kernel, values are matern52, matern52-res-warp, exp-decay-sum, exp-decay-combined, exp-decay-delta1. These are variants of either a joint Matern 5/2 kernel over configuration and resource, or the exponential decay model. Details about the latter can be found here.

  • Fitting a Gaussian process surrogate model to data encurs a cost which scales cubically with the number of datapoints. When applied to expensive deep learning workloads, even multi-fidelity asynchronous BOHB is rarely running up more than 100 observations or so (across all rung levels and brackets), and the GP computations are subdominant. However, if you apply it to cheaper train_fn and find yourself beyond 2000 total evaluations, the cost of GP fitting can become painful. In such a situation, you can explore the options opt_skip_period and opt_skip_num_max_resource. The basic idea is as follows. By far the most expensive part of a get_config call (picking the next configuration) is the refitting of the GP model to past data (this entails re-optimizing hyperparameters of the surrogate model itself). The options allow you to skip this expensive step for most get_config calls, after some initial period. Check the docstrings for details about these options. If you find yourself in such a situation and gain experience with these skipping features, make sure to contact the AutoGluon developers – we would love to learn about your use case.